Data storage method, device and equipment, readable storage medium and program product

Abstract

The invention relates to a data storage method and device, equipment, a readable storage medium and a program product. The method comprises the steps of obtaining a predicted cold and hot degree corresponding to to-be-written data, then storing the to-be-written data into a first storage space corresponding to the predicted cold and hot degree according to the predicted cold and hot degree so as to obtain attribute parameters of the data stored in the first storage space, and storing the to-be-written data into a second storage space according to the attribute parameters. Data stored in the first storage space can be transferred to the data blocks included in the target data group in the second storage space. By adopting the method, the write amplification problem can be relieved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data storage method, apparatus, device, readable storage medium and program product. Background Art

[0002] In storage media (such as disks), data is usually stored in different locations in the form of data blocks, and multiple data blocks can logically form a data group.

[0003] In a traditional data storage method, data is usually randomly written to all data blocks in a data group at the same time. After all data blocks in the data group are filled, data is randomly written to all data blocks in other data groups.

[0004] However, the above data storage method may lead to write amplification problem. Summary of the invention

[0005] Based on this, it is necessary to provide a data storage method, device, equipment, readable storage medium and program product that can alleviate the write amplification problem in response to the above technical problems.

[0006] In a first aspect, the present application provides a data storage method, comprising:

[0007] Get the predicted hotness or coldness of the data to be written;

[0008] According to the predicted degree of coldness or heat, the data to be written is stored in a first storage space corresponding to the predicted degree of coldness or heat;

[0009] Attribute parameters of the data stored in the first storage space are acquired, and according to the attribute parameters, the data stored in the first storage space are transferred to data blocks included in the target data group in the second storage space.

[0010] In this way, for all the acquired data to be written, their hotness or coldness is first predicted, and the data to be written is stored in the first storage space corresponding to the predicted hotness or coldness according to the predicted hotness or coldness, so that the predicted hotness or coldness of the data stored in each first storage space is the same, and subsequently the attribute parameters corresponding to the first storage space are continuously detected, and according to the attribute parameters, the data with the same hotness or coldness stored in the first storage space are stored in the same data block in the target data group, so that the predicted hotness or coldness of the data stored in a data block is the same, which avoids the write amplification problem caused by the simultaneous storage of valid data and invalid data in the data blocks due to the different hotness or coldness of the data stored in the data blocks when the target data group is subsequently garbage collected. The technical solution provided by the present application makes the predicted hotness or coldness of the data stored in different data blocks in the second storage space the same, which greatly alleviates the write amplification problem in the subsequent garbage collection process.

[0011] In one embodiment, the attribute parameter includes a data amount, and according to the attribute parameter, transferring the data stored in the first storage space to a data block included in the target data group in the second storage space includes:

[0012] Check whether the data volume is greater than the data volume threshold;

[0013] If the data volume is greater than or equal to the data volume threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0014] In this way, the above method can make the data in each data block belong to the same predicted hotness or coldness level, and the life cycle of each data may be the same, and the time required to become invalid data is the same, that is, the data in the data block may become invalid data at the same time. At this time, when garbage collection is performed on the target data group, there is no need to move the data block, and direct garbage collection can be performed, which alleviates the write amplification problem.

[0015] In one embodiment, the method further comprises:

[0016] When all data blocks included in the target data group become invalid data blocks, garbage collection is performed on the data blocks included in the target data group, wherein the invalidation time of the data of each data block is the same.

[0017] In this way, when the predicted hotness and coldness corresponding to the data contained in the target data group are the same, since the data invalidation time of each data block is the same, the entire target data group can be directly garbage collected without moving the internal data blocks, alleviating the problem of write amplification.

[0018] In one embodiment, the method further comprises:

[0019] If the data volume is less than the data volume threshold, determining whether the time difference between the time when the to-be-written data was last stored in the first storage space and the current time is greater than the time difference threshold;

[0020] If the time difference value is greater than the time difference threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0021] In this way, when the server transfers the data stored in the first storage space to the data blocks included in the target data group, since the data cannot fill up the target data group, the server can simultaneously write the data in other first storage spaces into other data blocks of the target data group. Although the data blocks included in the final target data group have different predicted hot and cold levels, the predicted hot and cold levels corresponding to the data in each data block are the same, which alleviates the write amplification problem in the subsequent garbage collection process.

[0022] In one embodiment, the method further comprises:

[0023] When the number of invalid data blocks included in the target data group reaches a number threshold, determining valid data blocks included in the target data group as data blocks to be reorganized, and the data invalidation time of the valid data blocks is different from the data invalidation time of the invalid data blocks included in the target data group;

[0024] The to-be-reorganized data blocks included in the target data group are recombined with the to-be-reorganized data blocks included in other data groups to obtain a recombined data group, and the invalid data blocks included in the target data group are garbage collected.

[0025] In this way, when performing garbage collection, you only need to change the logical relationship between the data blocks to be reorganized (valid data blocks) in the target data group, reorganize the data blocks to be reorganized with the data blocks to be reorganized in other data groups, and perform garbage collection on invalid data blocks to complete the garbage collection of the target data group. The entire process does not require data movement of data blocks, alleviating the write amplification problem.

[0026] In one embodiment, obtaining the predicted hotness or coldness corresponding to the data to be written includes:

[0027] The data to be written is input into the coldness and heatness prediction model to obtain the predicted coldness and heatness output by the coldness and heatness prediction model.

[0028] In this way, the hotness or coldness of the data to be written is automatically predicted through the hotness or coldness prediction model, and the prediction accuracy is high.

[0029] In one embodiment, the method further comprises:

[0030] Obtaining a target data invalidation time of a target data block storing data to be written, and determining an actual hotness or coldness corresponding to the data to be written according to the target data invalidation time;

[0031] According to the actual degree of coldness and heat and the predicted degree of coldness and heat, the model parameters of the coldness and heat prediction model are updated.

[0032] In this way, for the data to be written, after it is transferred to the data block of the target data group in the second storage space, the invalid time required for it to become invalid data can be monitored, so as to determine its corresponding actual degree of coldness and hotness, and then adjust the degree of coldness and hotness prediction model based on the actual degree of coldness and hotness, so that the model can be more adapted to the current application scenario.

[0033] In a second aspect, the present application also provides a data storage device, comprising:

[0034] An acquisition module, used to obtain the predicted hotness or coldness corresponding to the data to be written;

[0035] A writing module, used for storing the data to be written into a first storage space corresponding to the predicted degree of coldness or heat according to the predicted degree of coldness or heat;

[0036] The storage module is used to obtain attribute parameters of the data stored in the first storage space, and transfer the data stored in the first storage space to the data blocks included in the target data group in the second storage space according to the attribute parameters.

[0037] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of the first aspect described above.

[0039] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method in the first aspect are implemented.

[0040] The above-mentioned data storage method, device, equipment, readable storage medium and program product obtain the predicted degree of hotness or coldness corresponding to the data to be written, and then store the data to be written in the first storage space corresponding to the predicted degree of hotness or coldness according to the predicted degree of hotness or coldness, thereby obtaining the attribute parameters of the data stored in the first storage space, and according to the attribute parameters, the data stored in the first storage space can be transferred to the data block included in the target data group in the second storage space. In this way, for all the acquired data to be written, their hotness or coldness is first predicted, and the data to be written is stored in the first storage space corresponding to the predicted hotness or coldness according to the predicted hotness or coldness, so that the predicted hotness or coldness of the data stored in each first storage space is the same, and subsequently the attribute parameters corresponding to the first storage space are continuously detected, and according to the attribute parameters, the data with the same hotness or coldness stored in the first storage space are stored in the same data block in the target data group, so that the predicted hotness or coldness of the data stored in a data block is the same, which avoids the write amplification problem caused by the simultaneous storage of valid data and invalid data in the data blocks due to the different hotness or coldness of the data stored in the data blocks when the target data group is subsequently garbage collected. The technical solution provided by the present application makes the predicted hotness or coldness of the data stored in different data blocks in the second storage space the same, which greatly alleviates the write amplification problem in the subsequent garbage collection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a structural diagram of a traditional data set;

[0043] Figure 2 is the structure of another traditional data group;

[0044] Figure 3 is a schematic diagram of the garbage collection process in traditional technology;

[0045] Figure 4 An application environment diagram of a data storage method in one embodiment;

[0046] Figure 5 A schematic diagram of a flow chart of a data storage method in one embodiment;

[0047] Figure 6 is a schematic structural diagram of an exemplary first storage space in another embodiment;

[0048] Figure 7 is a schematic flow chart of step 503 in another embodiment;

[0049] Figure 8 is a flowchart of a data storage process in another embodiment when the amount of data in the first storage space is less than a data amount threshold;

[0050] Fig. 9 A schematic diagram of a target data group filled with data in another embodiment;

[0051] Fig.10 is a flowchart of a garbage collection process in another embodiment;

[0052] Fig.11 is a flow chart of an exemplary data storage method in another embodiment;

[0053] Fig.12 is a structural block diagram of a data storage device in one embodiment;

[0054] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] With the development of storage technology, massive data storage is usually solved by using storage systems, which store the data to be stored in storage media.

[0057] Storage media (such as disks) usually store data in the form of data blocks in different locations. Multiple data blocks can logically form a data group. In order to achieve high concurrency, data blocks in the same data group will be written to at the same time. Therefore, traditional data storage methods usually randomly write data to all data blocks in a data group at the same time. After all data blocks in the data group are full, data are randomly written to all data blocks in other data groups.

[0058] However, the above data storage method may cause the same data block to be written with data of different hot and cold degrees. Figure 1 , is a 4+2 data group (including 4 normal data blocks and two metadata blocks). It can be seen that each normal data block will have data pages of different hot and cold degrees due to random data writing at the same time. Figure 1 When garbage collection is performed on the data group in Figure 2 As shown in the figure, it can be seen that due to the different hot and cold data in different data blocks, when the data group needs to be garbage collected (when the amount of invalid data reaches the threshold), each data block stores both invalid data and valid data. At this time, it is necessary to first move the valid data in each data block, and then garbage collect the data block after the version is completed. The specific process can be referred to Figure 3 ,This process requires frequent reading and writing of each data block, resulting in the ,write amplification problem.

[0059] In view of this, the present application provides a data storage method, apparatus, device, readable storage medium and program product, which obtains the predicted degree of hotness or coldness corresponding to the data to be written, and then stores the data to be written in a first storage space corresponding to the predicted degree of hotness or coldness according to the predicted degree of hotness or coldness, thereby obtaining the attribute parameters of the data stored in the first storage space, and according to the attribute parameters, the data stored in the first storage space can be transferred to the data blocks included in the target data group in the second storage space. In this way, for all the acquired data to be written, their hotness or coldness is first predicted, and the data to be written is stored in the first storage space corresponding to the predicted hotness or coldness according to the predicted hotness or coldness, so that the predicted hotness or coldness of the data stored in each first storage space is the same, and subsequently the attribute parameters corresponding to the first storage space are continuously detected, and according to the attribute parameters, the data with the same hotness or coldness stored in the first storage space are stored in the same data block in the target data group, so that the predicted hotness or coldness of the data stored in a data block is the same, which avoids the write amplification problem caused by the simultaneous storage of valid data and invalid data in the data blocks due to the different hotness or coldness of the data stored in the data blocks when the target data group is subsequently garbage collected. The technical solution provided by the present application makes the predicted hotness or coldness of the data stored in different data blocks in the second storage space the same, which greatly alleviates the write amplification problem in the subsequent garbage collection process.

[0060] The data storage method provided in the embodiment of the present application can be applied to Figure 3 In the application environment shown. Among them, the data storage system can store the data that the server 401 needs to process. The data storage system can be integrated on the server 401, or it can be placed on the cloud or other network servers. The server 401 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0061] In an exemplary embodiment, Figure 5 As shown, a data storage method is provided, which is applied to Figure 4 The server 401 in the example is used as an example to illustrate, including the following steps 501 to 503. Among them:

[0062] Step 501, obtaining the predicted hotness or coldness corresponding to the data to be written.

[0063] The data to be written may be data waiting to be written into a storage medium, which may be a disk or a solid state drive, etc. The degree of hotness or coldness may characterize the life cycle of the data to be written being updated, and may reflect the time required for the data to be written to become invalid data.

[0064] In the embodiment of the present application, after the server obtains the data to be written, it can obtain the predicted degree of hotness and coldness corresponding to the data to be written. Optionally, the server can directly obtain the predicted degree of hotness and coldness corresponding to the data to be written through an external device; optionally, the server can predict the degree of hotness and coldness of the data to be written according to a preset prediction strategy to obtain the predicted degree of hotness and coldness of the data to be written.

[0065] In a possible implementation, the server may obtain the write space of the data to be written, and predict the hotness or coldness of the data to be written according to the update frequency of the write space, thereby obtaining the predicted hotness or coldness of the data to be written.

[0066] In another possible implementation, the server may input the data to be written into the coldness / heatness prediction model to obtain the predicted coldness / heatness output by the coldness / heatness prediction model.

[0067] Optionally, the hotness / coldness prediction model can be a neural network model trained based on historical written data and the hotness / coldness labels corresponding to the historical written data; optionally, the hotness / coldness prediction model can be a self-training model, so that the hotness / coldness prediction model can continuously adjust model parameters according to newly written data to be written, so as to be more adaptable to different application scenarios.

[0068] In this way, the hotness or coldness of the data to be written is automatically predicted through the hotness or coldness prediction model, and the prediction accuracy is high.

[0069] Step 502 , based on the predicted degree of coldness or heat, the data to be written is stored in a first storage space corresponding to the predicted degree of coldness or heat.

[0070] The first storage space is a data storage space that does not belong to the storage medium. Optionally, the first storage space may be located in the memory. Optionally, the first storage space may be located in the cache.

[0071] In the embodiment of the present application, the first storage space can be regarded as a transition space before the data to be written is written into the disk or storage medium. The number of first storage spaces can be multiple, and different first storage spaces correspond to different predicted coldness and heatness levels.

[0072] Reference Figure 6 , is a schematic diagram of the structure of an exemplary first storage space, where different first storage spaces store data to be written with different predicted coldness or hotness levels.

[0073] In this way, the data to be written with the same predicted hotness or coldness can be aggregated into the same first storage space to facilitate subsequent processing. In addition, for the data to be written with higher hotness, its reading and writing can be completed in the first storage space, avoiding the problem of frequent reading and writing of the disk caused by the data being updated immediately after being written to the disk.

[0074] Step 503: Acquire attribute parameters of the data stored in the first storage space, and transfer the data stored in the first storage space to the data blocks included in the target data group in the second storage space according to the attribute parameters.

[0075] For each first storage space, the server can detect the attribute parameters of the data stored therein in real time, so as to determine a suitable transfer timing for transferring the data in the first storage space to the second storage space.

[0076] Optionally, the attribute parameter may be the amount of data stored in the first storage space, so that when the data amount reaches a preset condition, the server may transfer the data stored in the first storage space to the second storage space; optionally, the attribute parameter may be the length of time from the time when the first storage space last stored data to the current time, so that when the length of time reaches a preset threshold, it means that the first storage space has not written new data to be written for a long time. In order to avoid the invalidation of the data in the first storage space, the server may transfer the data in the first storage space to the second storage space.

[0077] In the embodiment of the present application, the second storage space may be a data storage space different from the first storage space, and the second storage space may be deployed in a storage medium.

[0078] In this embodiment, the second storage space may include multiple data groups, and a data group may include multiple logically associated data blocks. The corresponding positions of each data block are relatively scattered, and each data block can be deployed in the solid-state storage space of each node in the storage system, or deployed on a disk.

[0079] In one possible implementation, for the data to be written, after it is transferred to the second storage space, the server can monitor the invalid time required for it to become invalid data, thereby determining its corresponding actual degree of coldness or hotness, and then adjust the degree of coldness or hotness prediction model based on the actual degree of coldness or hotness, so that the model can be more adapted to the current application scenario.

[0080] In an embodiment of the present application, the server can obtain the target data invalidation time of the target data block storing the data to be written, and determine the actual hotness or coldness corresponding to the data to be written based on the target data invalidation time, and then update the model parameters of the hotness or coldness prediction model based on the actual hotness or coldness and the predicted hotness or coldness.

[0081] Regarding the process of the server obtaining the target data invalidation time of the target data block storing the data to be written, optionally, the server can use a timer to count the time required from receiving the data to be written to the time when the data to be written becomes invalid data, and determine the target data invalidation time based on the statistical time; optionally, the server can obtain the metadata corresponding to the target data block, and determine the target data invalidation time based on the time required for the data to be written included in the source data to become invalid data.

[0082] After obtaining the target data invalid time, the server can determine the actual hotness or coldness of the data to be written based on the target data invalid time. Regarding the determination process, optionally, different hotness or coldness corresponds to different time value ranges, and the server can monitor the time value range to which the target data invalid time belongs, thereby determining the corresponding actual hotness or coldness; optionally, the server can map the target data invalid time according to preset mapping conditions, thereby determining the actual hotness or coldness corresponding to the target data invalid time.

[0083] In an embodiment of the present application, after determining the actual degree of hotness or coldness, the server can calculate the loss value based on a preset loss function, according to the preset degree of hotness or coldness and the actual degree of hotness or coldness, and use the loss value to update the model parameters of the hotness or coldness prediction model, thereby making the hotness or coldness prediction model more suitable for the current application environment.

[0084] In the above embodiment, the hotness or coldness of all acquired data to be written is first predicted, and the data to be written is stored in the first storage space corresponding to the predicted hotness or coldness according to the predicted hotness or coldness, so that the predicted hotness or coldness of the data stored in each first storage space is the same. Subsequently, the attribute parameters corresponding to the first storage space are continuously detected, and the data with the same hotness or coldness stored in the first storage space are stored in the same data block in the target data group according to the attribute parameters, so that the predicted hotness or coldness of the data stored in one data block is the same, thereby avoiding the write amplification problem caused by the simultaneous storage of valid data and invalid data in the data blocks due to the different hotness or coldness of the data stored in the data blocks when garbage collection is performed on the target data group. The technical solution provided by the present application ensures that the predicted hotness or coldness of the data stored in different data blocks in the second storage space is the same, which greatly alleviates the write amplification problem in the subsequent garbage collection process.

[0085] In one embodiment, based on the above Figure 5 The embodiment shown, referring to Figure 7 This embodiment relates to a process of transferring data stored in a first storage space to a data block included in a target data group in a second storage space according to attribute parameters. In this embodiment, the attribute parameters include the amount of data. Figure 7As shown, in this embodiment, step 503 includes Figure 7 Step 701 and step 702 are shown.

[0086] Step 701, detecting whether the data volume is greater than a data volume threshold.

[0087] Step 702: If the data volume is greater than or equal to the data volume threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0088] In an embodiment of the present application, the attribute parameters may include the data volume in the first storage space. For each first storage space, the server may detect whether the data volume therein is greater than a data volume threshold. The data volume threshold may be set according to an actual application scenario. Optionally, the data volume threshold may be the total capacity of the first storage space; optionally, the data volume threshold may be the data volume constituting a data group.

[0089] In a possible implementation, if the data volume threshold can fill the amount of data required for a data block, when the data volume is greater than the data volume threshold, it represents that the amount of data in the current first storage space is sufficient to fill a data block, and the server can transfer the data stored in the first storage space to any data block included in the target data group, so that data with the same predicted coldness or hotness can be written into the same data block.

[0090] In another possible implementation, if the data volume threshold may be the amount of data constituting a data group, when the data volume is greater than the data volume threshold, it represents that the amount of data in the current first storage space is sufficient to fill a data group, then the server may transfer the data stored in the first storage space to all data blocks included in the target data group, so that the data contained in the same data group have the same predicted hotness or coldness.

[0091] When the predicted hotness or coldness corresponding to the data included in the target data group is the same, the time point at which the data becomes invalid data may be the same. Therefore, in an embodiment of the present application, when all data blocks included in the target data group become invalid data blocks, the server can perform garbage collection on the data blocks included in the target data group. Since the invalidation time of each data block is the same, garbage collection can be performed directly on the entire target data group without moving the internal data blocks, thereby alleviating the problem of write amplification.

[0092] In this way, the above method can make the data in each data block belong to the same predicted hotness or coldness level, and the life cycle of each data may be the same, and the time required to become invalid data is the same, that is, the data in the data block may become invalid data at the same time. At this time, when garbage collection is performed on the target data group, there is no need to move the data block, and direct garbage collection can be performed, which alleviates the write amplification problem.

[0093] In one embodiment, based on the above Figure 7 The embodiment shown, referring to Figure 8 , this embodiment relates to a process of storing data when the amount of data in the first storage space is less than a data amount threshold, such as Figure 8 As shown, in this embodiment, the process includes Figure 8 Step 801 and step 802 are shown.

[0094] Step 801: If the data volume is less than the data volume threshold, determine whether the time difference between the time when the to-be-written data was last stored in the first storage space and the current time is greater than the time difference threshold.

[0095] Step 802: If the time difference value is greater than the time difference threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0096] In an embodiment of the present application, the data volume threshold may be the amount of data that a data group can store. When the data volume is less than the data volume threshold, it represents that the data in the current first storage space cannot fill the entire data group. In order to avoid the situation where the data in the first storage space has not been solidified into the second storage space due to the long-term inability of the data volume to reach the data volume threshold, in a possible implementation manner, the server may monitor the time difference between the moment when the data to be written was last stored in the first storage space and the current moment. If the time difference is greater than the time difference threshold, it represents that the time for the existing data in the first storage space to be stored in the second storage space is too long. The server may directly transfer the data stored in the first storage space to the data blocks included in the target data group.

[0097] In an embodiment of the present application, when the server transfers the data stored in the first storage space to the data blocks included in the target data group, since the data cannot fill the target data group, the server can simultaneously write the data in other first storage spaces into other data blocks of the target data group. In a possible implementation, the data in the same first storage space cannot fill a data block. At this time, the server can use default data to fill the remaining storage space of the data block. The default data can be 0 or other preset data.

[0098] In this way, although the data blocks included in the target data group have different predicted hotness and coldness, the predicted hotness and coldness corresponding to the data in each data block are the same, which alleviates the write amplification problem in the subsequent garbage collection process. Fig. 9 , which is a schematic diagram of a target data group filled with data using the above method.

[0099] Based on the above embodiment, the process of garbage collection for the target data group can refer to Fig.10 , including: when the number of invalid data blocks included in the target data group reaches a quantity threshold, the server can determine the valid data blocks included in the target data group as the data blocks to be reorganized, and then recombine the data blocks to be reorganized included in the target data group with the data blocks to be reorganized included in other data groups to obtain a reorganized data group, and perform garbage collection on the invalid data blocks included in the target data group.

[0100] The data invalidation time of the valid data block is different from the data invalidation time of the invalid data block included in the target data group.

[0101] In an embodiment of the present application, the invalid data in the target data group 0 reaches a quantity threshold. At this time, the server needs to perform garbage collection on the target data group 0. It can be seen that the invalid data blocks are data blocks 0 and 1, and the valid data blocks are data blocks 2 and 3. The server can determine that the valid data blocks 2 and 3 are the data blocks to be reorganized, and combine them with the data blocks to be reorganized (valid data blocks 2 and 3) in other data groups (data group 1) to form a new reorganized data group (data group 2), and then perform garbage collection on the invalid data blocks.

[0102] In this way, when performing garbage collection, you only need to change the logical relationship between the data blocks to be reorganized (valid data blocks) in the target data group, reorganize the data blocks to be reorganized with the data blocks to be reorganized in other data groups, and perform garbage collection on invalid data blocks to complete the garbage collection of the target data group. The entire process does not require data movement of data blocks, alleviating the write amplification problem.

[0103] In one embodiment, please refer to Fig.11 , which shows a flowchart of an exemplary data storage method provided by an embodiment of the present application, which can be applied to Figure 4 In the implementation environment shown.

[0104] Step 1101, input the data to be written into the coldness and heatness prediction model to obtain the predicted coldness and heatness output by the coldness and heatness prediction model.

[0105] Step 1102 , based on the predicted degree of coldness or heat, store the data to be written into a first storage space corresponding to the predicted degree of coldness or heat.

[0106] Step 1103: Acquire attribute parameters of the data stored in the first storage space.

[0107] Among them, the attribute parameters include data volume.

[0108] Step 1104, detecting whether the data volume is greater than a data volume threshold.

[0109] Step 1105: If the data volume is greater than or equal to the data volume threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0110] Step 1106: When all data blocks included in the target data group become invalid data blocks, garbage collection is performed on the data blocks included in the target data group.

[0111] The data invalidation time of each data block is the same.

[0112] Step 1107: If the data volume is less than the data volume threshold, determine whether the time difference between the time when the to-be-written data was last stored in the first storage space and the current time is greater than the time difference threshold.

[0113] Step 1108: If the time difference value is greater than the time difference threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0114] Step 1109: When the number of invalid data blocks included in the target data group reaches a number threshold, determine the valid data blocks included in the target data group as data blocks to be reorganized.

[0115] The data invalidation time of the valid data block is different from the data invalidation time of the invalid data block included in the target data group.

[0116] Step 1110: recombining the to-be-reorganized data blocks included in the target data group with the to-be-reorganized data blocks included in other data groups to obtain a recombined data group, and performing garbage collection on the invalid data blocks included in the target data group.

[0117] Step 1111 , obtaining a target data invalidation time of a target data block storing data to be written, and determining an actual hotness or coldness corresponding to the data to be written according to the target data invalidation time.

[0118] Step 1112, updating the model parameters of the coldness and heatness prediction model according to the actual coldness and heatness and the predicted coldness and heatness.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides a data storage device for implementing the data storage method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data storage device embodiments provided below can refer to the limitations on the data storage method above, and will not be repeated here.

[0121] In an exemplary embodiment, Fig.12 As shown, a data storage device is provided, including: an acquisition module 1201, a writing module 1202 and a storage module 1203, wherein:

[0122] An acquisition module 1201 is used to acquire the predicted hotness or coldness corresponding to the data to be written;

[0123] A writing module 1202 is used to store the data to be written into a first storage space corresponding to the predicted degree of coldness or heat according to the predicted degree of coldness or heat;

[0124] The storage module 1203 is used to obtain attribute parameters of the data stored in the first storage space, and transfer the data stored in the first storage space to the data blocks included in the target data group in the second storage space according to the attribute parameters.

[0125] In one embodiment, the attribute parameter includes data volume, and the storage module 1203 includes:

[0126] A detection unit, used to detect whether the data volume is greater than a data volume threshold;

[0127] The first transfer unit is configured to transfer the data stored in the first storage space to the data blocks included in the target data group if the data amount is greater than or equal to the data amount threshold.

[0128] In one embodiment, the storage module 1203 further includes:

[0129] The first garbage collection unit is used to perform garbage collection on each data block included in the target data group when each data block included in the target data group becomes an invalid data block, wherein the data invalidation time of each data block is the same.

[0130] In one embodiment, the storage module 1203 further includes:

[0131] a determination unit, configured to determine, if the data volume is less than the data volume threshold, whether a time difference between a time when the data to be written was last stored in the first storage space and a current time is greater than a time difference threshold;

[0132] The second transfer unit is used to transfer the data stored in the first storage space to the data block included in the target data group if the time difference value is greater than the time difference threshold.

[0133] In one embodiment, the storage module 1203 further includes:

[0134] a reorganization unit, configured to, when the number of invalid data blocks included in the target data group reaches a number threshold, determine valid data blocks included in the target data group as data blocks to be reorganized, wherein the data invalidation time of the valid data blocks is different from the data invalidation time of the invalid data blocks included in the target data group;

[0135] The second garbage collection unit is used to recombine the to-be-reorganized data blocks included in the target data group with the to-be-reorganized data blocks included in other data groups to obtain a reorganized data group, and to perform garbage collection on invalid data blocks included in the target data group.

[0136] In one embodiment, the acquisition module 1201 includes:

[0137] The prediction unit is used to input the data to be written into the coldness and heatness prediction model to obtain the predicted coldness and heatness output by the coldness and heatness prediction model.

[0138] In one embodiment, the acquisition module 1201 further includes:

[0139] a timing acquisition unit, configured to acquire a target data invalidation time of a target data block storing the data to be written, and determine an actual hotness or coldness corresponding to the data to be written according to the target data invalidation time;

[0140] An adjustment unit is used to update the model parameters of the coldness and heatness prediction model according to the actual coldness and heatness and the predicted coldness and heatness.

[0141] Each module in the above data storage device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0142] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.13 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data storage data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data storage method is implemented.

[0143] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0145] Get the predicted hotness or coldness of the data to be written;

[0146] According to the predicted degree of coldness or heat, storing the data to be written into a first storage space corresponding to the predicted degree of coldness or heat;

[0147] Attribute parameters of the data stored in the first storage space are acquired, and according to the attribute parameters, the data stored in the first storage space are transferred to data blocks included in the target data group in the second storage space.

[0148] In one embodiment, the attribute parameter includes data volume, and the processor further implements the following steps when executing the computer program:

[0149] Detecting whether the data volume is greater than a data volume threshold;

[0150] If the data volume is greater than or equal to the data volume threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] In the case that all data blocks included in the target data group become invalid data blocks, garbage collection is performed on the data blocks included in the target data group, wherein the data invalidation time of each data block is the same.

[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0154] If the data volume is less than the data volume threshold, determining whether the time difference between the time when the to-be-written data was last stored in the first storage space and the current time is greater than the time difference threshold;

[0155] If the time difference value is greater than the time difference threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0156] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0157] When the number of invalid data blocks included in the target data group reaches a number threshold, determining valid data blocks included in the target data group as data blocks to be reorganized, and the data invalidation time of the valid data blocks is different from the data invalidation time of the invalid data blocks included in the target data group;

[0158] The to-be-reorganized data blocks included in the target data group are recombined with the to-be-reorganized data blocks included in other data groups to obtain a recombined data group, and garbage collection is performed on the invalid data blocks included in the target data group.

[0159] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0160] The data to be written is input into a coldness / heatness prediction model to obtain the predicted coldness / heatness output by the coldness / heatness prediction model.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] Obtaining a target data invalidation time of a target data block storing the data to be written, and determining an actual hotness or coldness corresponding to the data to be written according to the target data invalidation time;

[0163] According to the actual degree of coldness and the predicted degree of coldness and heat, model parameters of the degree of coldness and heat prediction model are updated.

[0164] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0165] Get the predicted hotness or coldness of the data to be written;

[0166] According to the predicted degree of coldness or heat, storing the data to be written into a first storage space corresponding to the predicted degree of coldness or heat;

[0167] Attribute parameters of the data stored in the first storage space are acquired, and according to the attribute parameters, the data stored in the first storage space are transferred to data blocks included in the target data group in the second storage space.

[0168] In one embodiment, the attribute parameter includes data volume, and when the computer program is executed by a processor, the following steps are further implemented:

[0169] Detecting whether the data volume is greater than a data volume threshold;

[0170] If the data volume is greater than or equal to the data volume threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0172] In the case that all data blocks included in the target data group become invalid data blocks, garbage collection is performed on the data blocks included in the target data group, wherein the data invalidation time of each data block is the same.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] If the data volume is less than the data volume threshold, determining whether the time difference between the time when the to-be-written data was last stored in the first storage space and the current time is greater than the time difference threshold;

[0175] If the time difference value is greater than the time difference threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0177] When the number of invalid data blocks included in the target data group reaches a number threshold, determining valid data blocks included in the target data group as data blocks to be reorganized, and the data invalidation time of the valid data blocks is different from the data invalidation time of the invalid data blocks included in the target data group;

[0178] The to-be-reorganized data blocks included in the target data group are recombined with the to-be-reorganized data blocks included in other data groups to obtain a recombined data group, and garbage collection is performed on the invalid data blocks included in the target data group.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0180] The data to be written is input into a coldness / heatness prediction model to obtain the predicted coldness / heatness output by the coldness / heatness prediction model.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0182] Obtaining a target data invalidation time of a target data block storing the data to be written, and determining an actual hotness or coldness corresponding to the data to be written according to the target data invalidation time;

[0183] According to the actual degree of coldness and the predicted degree of coldness and heat, model parameters of the degree of coldness and heat prediction model are updated.

[0184] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0185] Get the predicted hotness or coldness of the data to be written;

[0186] According to the predicted degree of coldness or heat, storing the data to be written into a first storage space corresponding to the predicted degree of coldness or heat;

[0187] Attribute parameters of the data stored in the first storage space are acquired, and according to the attribute parameters, the data stored in the first storage space are transferred to data blocks included in the target data group in the second storage space.

[0188] In one embodiment, the attribute parameter includes data volume, and when the computer program is executed by a processor, the following steps are further implemented:

[0189] Detecting whether the data volume is greater than a data volume threshold;

[0190] If the data volume is greater than or equal to the data volume threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0191] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0192] In the case that all data blocks included in the target data group become invalid data blocks, garbage collection is performed on the data blocks included in the target data group, wherein the data invalidation time of each data block is the same.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0194] If the data volume is less than the data volume threshold, determining whether the time difference between the time when the to-be-written data was last stored in the first storage space and the current time is greater than the time difference threshold;

[0195] If the time difference value is greater than the time difference threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0197] When the number of invalid data blocks included in the target data group reaches a number threshold, determining valid data blocks included in the target data group as data blocks to be reorganized, and the data invalidation time of the valid data blocks is different from the data invalidation time of the invalid data blocks included in the target data group;

[0198] The to-be-reorganized data blocks included in the target data group are recombined with the to-be-reorganized data blocks included in other data groups to obtain a recombined data group, and garbage collection is performed on the invalid data blocks included in the target data group.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] The data to be written is input into a coldness / heatness prediction model to obtain the predicted coldness / heatness output by the coldness / heatness prediction model.

[0201] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0202] Obtaining a target data invalidation time of a target data block storing the data to be written, and determining an actual hotness or coldness corresponding to the data to be written according to the target data invalidation time;

[0203] According to the actual degree of coldness and the predicted degree of coldness and heat, model parameters of the degree of coldness and heat prediction model are updated.

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0205] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0206] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0207] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data storage method, characterized in that: The method comprises: Get the predicted hotness or coldness of the data to be written; According to the predicted degree of coldness or heat, storing the data to be written into a first storage space corresponding to the predicted degree of coldness or heat; Attribute parameters of the data stored in the first storage space are acquired, and according to the attribute parameters, the data stored in the first storage space are transferred to data blocks included in the target data group in the second storage space.

2. The method according to claim 1, characterized in that The attribute parameter includes a data amount, and transferring the data stored in the first storage space to a data block included in a target data group in the second storage space according to the attribute parameter includes: Detecting whether the data volume is greater than a data volume threshold; If the data volume is greater than or equal to the data volume threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

3. The method according to claim 2, characterized in that The method further comprises: In the case that all data blocks included in the target data group become invalid data blocks, garbage collection is performed on the data blocks included in the target data group, wherein the data invalidation time of each data block is the same.

4. The method according to claim 2, characterized in that: The method further comprises: If the data volume is less than the data volume threshold, determining whether the time difference between the time when the to-be-written data was last stored in the first storage space and the current time is greater than the time difference threshold; If the time difference value is greater than the time difference threshold, the data stored in the first storage space is transferred to the data block included in the target data group.

5. The method according to claim 4, characterized in that The method further comprises: When the number of invalid data blocks included in the target data group reaches a number threshold, determining valid data blocks included in the target data group as data blocks to be reorganized, and the data invalidation time of the valid data blocks is different from the data invalidation time of the invalid data blocks included in the target data group; The to-be-reorganized data blocks included in the target data group are recombined with the to-be-reorganized data blocks included in other data groups to obtain a recombined data group, and garbage collection is performed on the invalid data blocks included in the target data group.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining of the predicted hotness or coldness corresponding to the data to be written includes: The data to be written is input into a coldness / heatness prediction model to obtain the predicted coldness / heatness output by the coldness / heatness prediction model.

7. The method according to claim 6, characterized in that The method further comprises: Obtaining a target data invalidation time of a target data block storing the data to be written, and determining an actual hotness or coldness corresponding to the data to be written according to the target data invalidation time; According to the actual degree of coldness and the predicted degree of coldness and heat, model parameters of the degree of coldness and heat prediction model are updated.

8. A data storage device, characterized in that: The device comprises: An acquisition module, used to obtain the predicted hotness or coldness corresponding to the data to be written; A writing module, configured to store the data to be written into a first storage space corresponding to the predicted degree of coldness or heat according to the predicted degree of coldness or heat; The storage module is used to obtain attribute parameters of the data stored in the first storage space, and transfer the data stored in the first storage space to the data blocks included in the target data group in the second storage space according to the attribute parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images