Data processing method and device, equipment and computer readable storage medium

Abstract

The present disclosure relates to the technical field of converged storage, and particularly relates to a data processing method and device, equipment and a computer readable storage medium. The present disclosure dynamically adjusts the heat value of a data block based on the access condition of the data block, and when the first storage space meets a preset data fall condition, determines a target migration data block according to the dynamically adjusted heat value, and migrates the target migration data block to a second storage space. The heat value is used to dynamically manage the data block, which reduces the increase of read-write delay and the slow response of requests caused by the division of fixed areas in the first storage space, and improves the flexibility of data storage in the converged storage system.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of converged storage technology, and particularly relates to a data processing method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] As a converged storage system, a hybrid flash converged system uses a solid state drive (SSD) as a cache layer and a hard disk drive (HDD) as a capacity layer for hierarchical storage. Since the read-write efficiency of the solid state drive is higher than that of the hard disk drive, in order to realize data read-write of the upper-layer application through the solid state drive as much as possible, a fixed proportion of two spaces in the solid state drive is usually divided, one part of the space is exclusively used for data read request to write data in the hard disk drive to the solid state drive, and then the data read request is realized through the solid state drive; the other part of the space is exclusively used for data write request to store data of the data write request.

[0003] However, such fixed proportion of storage space division has a great limitation on the data processing capability of the converged storage system, for example, different upper-layer applications have different read-write I / O proportions; or when a certain type of IO (read or write) operation is concentrated, due to the limitation of the exclusive space of the solid state drive, frequent interaction with the hard disk drive is required, which increases the read-write latency and slows down the request response. Therefore, the existing data processing method in the converged storage system lacks rationality. SUMMARY

[0004] In order to solve the above technical problems, the present disclosure provides a data processing method, device, equipment and computer readable storage medium to improve the rationality of the data processing method in the converged storage system.

[0005] In a first aspect, the present disclosure provides a data processing method, the converged storage system includes a first storage space and a second storage space, the read-write latency of the first storage space is less than that of the second storage space, and the data processing method includes:

[0006] updating the hotness value of the data block based on the access information of the data block;

[0007] when the data storage condition in the first storage space meets a preset data fall disk condition, determining a target migration data block in the first storage space according to the hotness value of the data block;

[0008] migrating the target migration data block to the second storage space.

[0009] Optionally, the updating the hotness value of the data block based on the access information of the data block includes:

[0010] in response to an access operation on the data block, increasing a hotness value of the data block, and updating a response time of the data block in the access information as a time when the access operation occurs;

[0011] based on a time difference between the response time and a current time, decreasing the hotness value of the data block.

[0012] Optionally, the hotness value of the data block comprises a reference hotness value, and the increasing the hotness value of the data block in response to the access operation on the data block comprises:

[0013] increasing the hotness value of the data block by a first hotness adjustment value each time the access operation on the data block occurs.

[0014] Optionally, the decreasing the hotness value of the data block based on the time difference between the response time and the current time comprises:

[0015] determining a second hotness adjustment value based on the time difference, the second hotness adjustment value being proportional to the time difference;

[0016] decreasing the hotness value of the data block by the second hotness adjustment value.

[0017] Optionally, the preset data flushing condition comprises:

[0018] a remaining storage capacity in the first storage space is less than a first storage capacity threshold; and / or,

[0019] a size of a key data block in the first storage space is greater than a second storage capacity threshold, the key data block being at least one data block in the first storage space whose hotness value is higher than a first hotness value threshold.

[0020] Optionally, the determining the target migration data block in the first storage space according to the hotness value of the data block comprises:

[0021] selecting a plurality of data blocks in the first storage space whose hotness values are the lowest as candidate data blocks;

[0022] selecting a target migration data block from the candidate data blocks, so that a size of the target migration data block is greater than a third storage capacity threshold, and a number of the target migration data block is the least.

[0023] Optionally, the migrating the target migration data block into the second storage space comprises:

[0024] determining backup data blocks in the target migration data blocks, the backup data blocks being data blocks with highest hotness values in the target migration data blocks, and a size of the backup data blocks being less than a fourth storage capacity threshold value than a sum of sizes of non-target migration data blocks in the first storage space;

[0025] copying the target migration data blocks into a second storage space;

[0026] retaining the backup data blocks in the first storage space, and deleting target migration data blocks other than the backup data blocks in the first storage space.

[0027] Optionally, the method further comprises:

[0028] in response to a data write event trigger of to-be-written data, obtaining a preset priority of the to-be-written data;

[0029] calculating a hotness value of the to-be-written data based on a hotness value calculation parameter, the hotness value calculation parameter comprising the preset priority, a preset hotness step, and a reference hotness value;

[0030] constructing a to-be-written data block based on the to-be-written data and the hotness value of the to-be-written data;

[0031] storing the to-be-written data block into the converged storage system.

[0032] Optionally, the calculating the hotness value of the to-be-written data block based on the preset priority, the preset hotness step, and the reference hotness value comprises:

[0033] calculating a hotness bonus value based on the preset priority and the preset hotness step, the hotness bonus value being proportional to the preset priority;

[0034] calculating a sum of the reference hotness value and the priority bonus hotness value to obtain the hotness value of the to-be-written data block.

[0035] Optionally, the method further comprises:

[0036] counting a hit rate of a data access request in the converged storage system, the hit rate being a probability that the first storage space stores target data corresponding to the data access request;

[0037] when the hit rate is lower than a preset hit rate threshold value, updating the hotness value calculation parameter to make the hit rate higher than the preset hit rate threshold value.

[0038] Optionally, the method comprises:

[0039] In response to a data write request of to-be-written data, if the to-be-written data exists in the first storage space, a response time of a data block corresponding to the to-be-written data is updated as a current time; or,

[0040] If the to-be-written data does not exist in the first storage space, and a size of the to-be-written data is not greater than a remaining storage capacity in the first storage space, the to-be-written data is written into the first storage space, and a response time of a data block corresponding to the to-be-written data is recorded as the current time.

[0041] Optionally, the method further includes:

[0042] In response to a data read request for a target data block, if the target data block is only located in the second storage space, and a size of the target data block is not greater than a remaining storage capacity in the first storage space, the target data block is migrated into the first storage space, so as to complete the data read request based on the first storage space.

[0043] In a second aspect, an embodiment of the present disclosure provides a data processing apparatus, applied to a converged storage system, the converged storage system including a first storage space and a second storage space, a read-write latency of the first storage space being less than a read-write latency of the second storage space, and the data processing apparatus including:

[0044] An updating module, configured to update a heat value of a data block based on access information of the data block;

[0045] A determining module, configured to determine a target migration data block in the first storage space according to the heat value of the data block when a data storage condition in the first storage space meets a preset data fall-to-disk condition;

[0046] A migration module, configured to migrate the target migration data block into the second storage space.

[0047] In a third aspect, an embodiment of the present disclosure provides a converged storage system management device, including:

[0048] A memory;

[0049] A processor; and

[0050] A computer program;

[0051] The computer program is stored in the memory and configured to be executed by the processor to implement the method in the first aspect.

[0052] In a fourth aspect, the embodiments of the present disclosure provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method in the first aspect.

[0053] In a fifth aspect, the embodiments of the present disclosure also provide a computer program product, which includes a computer program or instructions. The computer program or instructions are executed by a processor to implement the data processing method as described above.

[0054] The data processing method, device, equipment and computer readable storage medium provided by the embodiments of the present disclosure dynamically adjust the heat value of the data block based on the access condition of the data block, and when the first storage space meets the preset data landing condition, the target migration data block is determined according to the dynamically adjusted heat value, and the target migration data block is migrated to the second storage space. The heat value is used to dynamically manage the data block, which reduces the increase of read-write delay and slow response caused by the division of fixed area of the first storage space, and improves the flexibility of data storage in the converged storage system. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A data processing method flowchart is provided for the embodiments of the present disclosure.

[0056] Figure 2 A converged storage system structure schematic diagram is provided for the embodiments of the present disclosure.

[0057] Figure 3 A data block structure schematic diagram is provided for the embodiments of the present disclosure.

[0058] Figure 4 A data processing device schematic diagram is provided for the embodiments of the present disclosure.

[0059] Figure 5 A structure schematic diagram of a converged storage system management equipment is provided for the embodiments of the present disclosure. DETAILED DESCRIPTION

[0060] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the schemes of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0061] In the following description, many specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some of the embodiments of the present disclosure, not all the embodiments.

[0062] This disclosure provides a data processing method, which will be described below with reference to specific embodiments.

[0063] Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present disclosure. This method can be applied to, for example... Figure 2 The converged storage system shown. For example... Figure 2 As shown, the converged storage space includes a first storage space 21 and a second storage space 22. The second storage space 22 includes multiple second storage subspaces. The read / write latency of the first storage space is less than that of the second storage space. Optionally, the first storage space is an SSD and the second storage space is an HDD; when the second storage space includes multiple second storage subspaces, each second storage subspace is an HDD.

[0064] The following is combined with Figure 2 The fused storage system shown is for Figure 1 The data processing methods shown will be introduced. For example... Figure 1 As shown, the data processing method includes the following steps:

[0065] S101. Update the popularity value of the data block based on the access information of the data block.

[0066] Data blocks are a data storage format in converged storage systems. A data block includes data content, data attribute information, and log information.

[0067] Figure 3 This is a schematic diagram of a data block structure provided in an embodiment of this disclosure. Figure 3 As shown, in addition to the data content, the data block also includes attribute information such as popularity value, priority flag, and response time, as well as log information. The log information of the data block includes access records, status, and error and exception information.

[0068] The response time of a data block is updated as data block read and write operations occur, while the access record contains specific information about the read and write operations. The access record and the response time of the data block together constitute the access information of the data block.

[0069] The popularity value of a data block reflects how frequently it is accessed, including both reading and writing. A higher popularity value indicates that the data block is accessed more frequently and is considered "hot data"; a lower popularity value indicates that the data block is accessed less frequently and is considered "cold data".

[0070] In this step, based on the access information of the data block, the hotness value of the data block is updated, including increasing the hotness value of the data block and reducing the hotness value of the data block. For example, when the data block has a read-write operation, the hotness value of the data block is increased; when the data block has not had a read-write operation for a long time, the hotness value of the data block is reduced.

[0071] S102, when the data storage condition in the first storage space meets the preset data landing condition, determining a target migration data block in the first storage space according to the hotness value of the data block.

[0072] When the data storage condition in the first storage space meets the preset data landing condition, it indicates that the current remaining storage space of the first storage space may not be able to support subsequent data normal storage, or the data block in the first storage space occupies the storage space in an unreasonable manner, and part of the data block in the first storage space needs to be migrated to the second storage space to improve the data storage condition in the first storage space.

[0073] Specifically, the target migration data block in the first storage space is determined according to the hotness value of the data block, including sorting the data blocks in a manner from high to low according to the hotness value, and selecting a plurality of data blocks at the back of the sorting as the target migration data block, so that after the target migration data block is migrated to the second storage space, the first storage space no longer meets the preset data landing condition.

[0074] S103, migrating the target migration data block to the second storage space.

[0075] The embodiments of the present disclosure update the hotness value of the data block based on the access information of the data block; when the data storage condition in the first storage space meets the preset data landing condition, the target migration data block in the first storage space is determined according to the hotness value of the data block; the target migration data block is migrated to the second storage space; the hotness value of the data block is dynamically adjusted based on the access condition of the data block, and when the first storage space meets the preset data landing condition, the target migration data block is determined according to the dynamically adjusted hotness value, and the target migration data block is migrated to the second storage space. The data block is dynamically managed based on the hotness value, which reduces the problems of increased read-write delay, slow request response and the like caused by fixed region division of the first storage space, and improves the flexibility of data storage in the converged storage system.

[0076] In some embodiments, the hotness value of the data block is updated based on the access information of the data block, including: in response to an access operation on the data block, increasing the hotness value of the data block, and updating the response time of the data block in the access information to the time when the access operation occurs; based on the time difference between the response time and the current time, reducing the hotness value of the data block.

[0077] The access operation of the data block includes reading the data block or writing the data block. When the data block in the fusion storage space is accessed, the hotness value of the data block is recalculated.

[0078] Specifically, the hotness value of the data block includes a reference hotness value, and the increasing the hotness value of the data block in response to the access operation of the data block includes: increasing the hotness value of the data block by a first hotness adjustment value each time the access operation of the data block occurs.

[0079] The hotness value of the data block includes a reference hotness value and a hotness addition value. The reference hotness value of each data block in the fusion storage system is the same, which is a reasonable preset value. The hotness addition value is related to the priority mark of the data block. The higher the priority represented by the priority mark, the higher the hotness addition value of the data block.

[0080] The hotness value of the data block is increased by a first hotness adjustment value each time the data block is accessed. The first hotness adjustment value is a preset parameter value, but its value range should be controlled within a reasonable range, such as the ratio of the first hotness adjustment value to the reference hotness value being less than a first preset ratio, so as to avoid the first hotness adjustment value being too large or too small to cause the hotness value of the data block in the fusion storage system to be invalid. The value range of the first preset ratio is (0, 1).

[0081] For example, let the first hotness adjustment value be M, the reference hotness value be I, and the first preset ratio be 0.2. The relationship between the first hotness adjustment value and the reference hotness value can be expressed as:

[0082]

[0083] In addition, when the data block in the fusion storage space is not accessed within a certain preset time, the hotness value of the data block is also recalculated.

[0084] Specifically, based on the time difference between the response time and the current time, the hotness value of the data block is reduced, including: determining a second hotness adjustment value based on the time difference, the second hotness adjustment value being proportional to the time difference; and reducing the hotness value of the data block by the second hotness adjustment value.

[0085] The response time of the data block is the time when the data block was last accessed. The larger the time difference between the response time and the current time, the longer the data block has not been accessed, and the larger the corresponding second hotness adjustment value. The time difference is an integer, and the unit is second.

[0086] Optionally, the second hotness adjustment value is the product of the time difference and a preset adjustment value.

[0087] Optionally, the preset adjustment value is in a preset proportion to the first hotness adjustment value.

[0088] For example, the preset adjustment value is 0.1 times the first hotness adjustment value, and the second hotness adjustment value X is expressed as:

[0089]

[0090] Optionally, a hotness value update period is preset, and the second hotness adjustment value of the data block is calculated according to the hotness value update period, and the hotness value of the data block is reduced by the second hotness adjustment value.

[0091] For example, the hotness value update period is 60 seconds, the response time of the data block is called every 60 seconds, the second hotness adjustment value of the data block is calculated based on the time difference between the response time and the current time, and the hotness value of the data block is reduced by the second hotness adjustment value.

[0092] Optionally, the increase and decrease of the hotness value of the same data block at the same time are avoided to avoid confusion of the hotness value of the data block.

[0093] The embodiments of the present disclosure increase the hotness value of the data block according to the access situation of the data block, and reduce the hotness value of the data block based on the time when the data block is not accessed, thereby achieving dynamic adjustment of the hotness value of the data block. Meanwhile, in the embodiments of the present disclosure, the proportion of the first hotness adjustment value and the reference hotness value is controlled to avoid the first hotness adjustment value being too large or too small to cause invalidation of the hotness value of the data block in the fusion storage system, so that the adjustment of the hotness value is more reasonable.

[0094] In some embodiments, the preset data flushing condition includes: the remaining storage capacity in the first storage space is less than a first storage capacity threshold; and / or, the size of a key data block in the first storage space is greater than a second storage capacity threshold, the key data block being at least one data block in the first storage space whose hotness value is higher than a first hotness value threshold.

[0095] When the remaining storage capacity in the first storage space is less than the first storage capacity threshold, it indicates that the remaining storage capacity in the first storage space is insufficient, which may cause that subsequent data blocks cannot be directly written into the first storage space but need to be directly written into the second storage space, so that the high-speed read-write advantage of the first storage space cannot be fully utilized. At this time, part of the data blocks in the first storage space need to be migrated to the second storage space to increase the remaining storage capacity in the first storage space.

[0096] Optionally, the first storage capacity threshold is 30% of the total capacity of the first storage space.

[0097] The key data block is a data block with a high hotness value in the first storage space, that is, a frequently accessed data block.

[0098] Optionally, the first hotness threshold is determined according to the hotness value ranking of the data blocks in the first storage space, and the minimum value of the hotness values of a preset proportion of data blocks ranked at the front of the hotness value ranking is the first hotness threshold. For example, the minimum value of the hotness values of the data blocks ranked at the front of 20% of the hotness value ranking is the first hotness threshold, and the key data blocks are the data blocks ranked at the front of 20% of the hotness value ranking in the first storage space.

[0099] When the size of the key data blocks in the first storage space is greater than the second storage capacity threshold, it indicates that there are more large key data blocks in the first storage space, and the remaining storage capacity is relatively tight. Alternatively, there can be some potential problems in the system, such as unreasonable data access mode, abnormal business traffic, etc. At this time, part of the data blocks in the first storage space need to be migrated to the second storage space to avoid the failure of the entire cache mechanism due to the excessive occupation of the first storage space by a small number of key data blocks.

[0100] Optionally, the remaining storage capacity in the first storage space is detected in real time to be less than the first storage capacity threshold; whether the size of the key data blocks in the first storage space is greater than the second storage capacity threshold is detected according to a hotness value update period; or, whenever the hotness value of a data block is updated, whether the size of the key data blocks in the first storage space is greater than the second storage capacity threshold is detected.

[0101] Correspondingly, when the data storage condition in the first storage space meets the preset data landing condition, the target migration data block in the first storage space is determined according to the hotness value of the data block, including: selecting a plurality of data blocks with the lowest hotness value in the first storage space as candidate data blocks; and selecting a target migration data block from the candidate data blocks, so that the size of the target migration data block is greater than a third storage capacity threshold, and the number of the target migration data blocks is the least.

[0102] The third storage capacity threshold is greater than the first storage capacity threshold.

[0103] The data blocks in the first storage space are sorted according to the hotness value from high to low, and a plurality of data blocks ranked at the back of the hotness value are selected as candidate data blocks. Further, at least one target migration data block is selected from the candidate data blocks, so that the size of the target migration data block is greater than the third storage capacity threshold, so that after the target migration data block is migrated to the second storage space, the remaining storage capacity of the first storage space is increased by at least the third storage capacity threshold.

[0104] Meanwhile, under the condition that the size of the target migration data block meets the third storage capacity threshold, the number of the selected target migration data blocks should be as small as possible, that is, the larger data blocks in the candidate data blocks are selected. In the converged storage system, data access is in units of data blocks, when there is a data block corresponding to the access request in the first storage space, it is considered that the access request is hit, at this time the access request can be completed based on the first storage space. Therefore, the smaller the number of target migration data blocks migrated to the second storage space, the more data blocks can be retained in the first storage space to make the access request hit rate of the first storage space higher after the data blocks are migrated.

[0105] Optionally, the candidate data blocks are sorted based on the order from large to small, and a plurality of candidate data blocks at the top of the order are selected as the target migration data blocks, so that the size of the selected target migration data blocks is greater than the third storage capacity threshold.

[0106] For example, when the remaining storage capacity of the first storage space is less than 30% of the total capacity of the first storage space, and / or the size of the data blocks with the top 20% of the hot degree value (i.e. the key data blocks) in the first storage space exceeds 30% of the total capacity of the first storage space, it is considered that the storage condition of the first storage space meets the preset fall disk condition, at this time the data fall disk operation is triggered, and the target migration data blocks are selected to be migrated to the second storage space. Among them, a plurality of data blocks with the lowest hot degree value are selected as candidate data blocks, and a plurality of target migration data blocks are selected from the candidate data blocks, so that the size of the target migration data blocks is greater than 30% of the total capacity of the first storage space, and the number of the target migration data blocks is as small as possible.

[0107] The embodiments of the present disclosure can ensure that the remaining storage capacity of the first storage space increases to meet the expectation while migrating as few data blocks as possible when selecting the target migration data blocks, so that the data blocks stored in the first storage space can meet the access demand as much as possible after the data blocks are migrated, and the access request hit rate of the first storage space is ensured.

[0108] In some embodiments, the migrating the target migration data blocks to the second storage space includes: determining a backup data block in the target migration data blocks, the backup data block being a plurality of data blocks with the highest hot degree value in the target migration data blocks, and the size of the backup data block and the size of the non-target migration data blocks in the first storage space being less than a fourth storage capacity threshold; copying the target migration data blocks to the second storage space; retaining the backup data blocks in the first storage space, and deleting the target migration data blocks in the first storage space except the backup data blocks.

[0109] After the target migration data block is migrated to the second storage space, the corresponding storage space in the first storage space is released, and the remaining storage capacity of the first storage space is greatly increased. Under the premise of meeting the demand of the remaining storage capacity of the first storage space, part of the data blocks in the target migration data block is selected as backup data blocks, and the copies of the backup data blocks are retained in the first storage space. That is, after the data block migration, the backup data blocks are saved in the first storage space and the second storage space.

[0110] Wherein, in order to meet the demand of the remaining storage capacity of the first storage space, it is necessary to control the sum of the sizes of the backup data blocks and the non-target migration data blocks in the first storage space to be less than the fourth storage capacity value. The sum of the sizes of the backup data blocks and the non-target migration data blocks in the first storage space is also the size of the data storage space occupied in the first storage space after the data block migration.

[0111] Optionally, the sum of the sizes of the backup data blocks and the non-target migration data blocks in the first storage space is controlled to be less than the fourth storage capacity value, while the number of the selected backup data blocks is maximized. Specifically, a plurality of data block combinations are determined based on the target migration data blocks, the sum of the sizes of the target migration data blocks and the non-target migration data blocks in the first storage space in each data block combination is less than the fourth storage capacity value, the sum of the hotness values of the target migration data blocks in each data block combination and the number of data blocks are calculated, the first score of the data block combination is determined based on the sum of the hotness values, and the sum of the hotness values is proportional to the first score; the second score of the data block combination is determined based on the number of data blocks, the number of data blocks is proportional to the second score, and the sum of the first score and the second score is calculated to obtain the data block combination score; finally, the plurality of target migration data blocks in the data block combination with the highest data block combination score are selected as the backup data blocks.

[0112] The above operation can backup as many data blocks as possible in the first storage space to improve the data access request hit rate of the first storage space after the data block migration.

[0113] After the non-backup data blocks in the target migration data block are migrated to the second storage space, the corresponding data in the first storage space is deleted, and the storage space previously occupied by the non-backup data blocks is released. Since the backup data blocks have high hotness values, they will be accessed with a high probability in a short time. Therefore, after the backup data blocks are migrated to the second storage space, the first storage space still retains the data copies of the backup data blocks, so that the data access requests corresponding to the backup data blocks can be responded by the backup data blocks in the first storage space.

[0114] For example, a plurality of data blocks with the highest heat values in the target migration data blocks are selected as backup data blocks, so that the sum of the sizes of the backup data blocks and the non-target migration data blocks in the first storage space is less than 60% of the total capacity of the first storage space, and the number of the backup data blocks is the largest. Correspondingly, after the data block migration, the remaining storage capacity in the first storage space is not less than 40% of the total capacity of the first storage space.

[0115] The embodiments of the present disclosure select backup data blocks in the target migration data blocks, meet the requirement of the remaining storage capacity of the first storage space, and enable the most data blocks to be backed up in the first storage space when the target migration data blocks are migrated, so as to improve the data access request hit rate of the first storage space after the data block migration.

[0116] In some embodiments, the method further includes: in response to a data write event trigger of the to-be-written data, obtaining a preset priority of the to-be-written data; calculating a heat value of the to-be-written data based on a heat value calculation parameter, the heat value calculation parameter including the preset priority, a preset heat step, and a reference heat value; constructing a to-be-written data block based on the to-be-written data and the heat value of the to-be-written data; and storing the to-be-written data block into the converged storage system.

[0117] The data write event trigger of the to-be-written data includes: in response to a write request of the to-be-written data, determining whether the to-be-written data exists in the first storage space. If the to-be-written data does not exist in the first storage space, the data write event of the to-be-written data is triggered.

[0118] At this time, since the to-be-written data is new data in the first storage space, there is no corresponding heat value yet, which needs to be calculated.

[0119] Specifically, the calculation of the heat value of the to-be-written data block based on the preset priority, the preset heat step, and the reference heat value includes: calculating a heat addition value based on the preset priority and the preset heat step, the heat addition value being proportional to the preset priority; and calculating the sum of the reference heat value and the priority addition heat value to obtain the heat value of the to-be-written data block.

[0120] The to-be-written data has a priority identifier indicating the preset priority of the to-be-written data. The higher the preset priority, the higher the access frequency of the to-be-written data. The lower the preset priority, the lower the access frequency of the to-be-written data.

[0121] The corresponding heat addition value is calculated based on the preset priority of the to-be-written data. The heat addition value is proportional to the preset priority. The higher the preset priority, the higher the heat addition value. The lower the preset priority, the lower the heat addition value.

[0122] The preset heat degree step is a difference of heat degree addition values of different preset priorities, that is, the heat degree addition value increases by the preset heat degree step for each increase of the preset priority of the data to be written.

[0123] Optionally, the priority identifier is in a digital form, and the higher the number, the higher the preset priority corresponding to the priority identifier; correspondingly, the heat degree addition value is a product of the preset priority and the preset heat degree step.

[0124] Optionally, the optional number of priority identifiers is less than the predetermined number, and if the optional number of priority identifiers is too large, the distinguishing effect of the heat degree value will be reduced, and the complexity of the data block management will be caused.

[0125] For example, let the heat degree value be PV, the reference heat degree value be I, the preset priority be PF, and the preset heat degree step be N, and the heat degree value calculation process of the data to be written is as follows:

[0126] PV = I + PF * N

[0127] Further, a data block to be written is constructed based on the data to be written and the corresponding heat degree value, wherein the data content is the data to be written, and the response time is the writing time of the data to be written.

[0128] On the basis of the above embodiment, the method further includes: counting a hit rate of a data access request in the fusion storage system, the hit rate being a probability that the first storage space stores target data corresponding to the data access request; and updating the heat degree value calculation parameter when the hit rate is lower than a preset hit rate threshold, so that the hit rate is higher than the preset hit rate threshold.

[0129] The data access request includes a read data request and a write data request. When the first storage space stores the target data corresponding to the data access request, the data access request is hit. The hit rate of the data access request is the probability that the first storage space stores the target data corresponding to the data access request.

[0130] The higher the hit rate of the data access request, the more ideal the data block storage in the first storage space is; and when the hit rate of the data access request is lower than the preset hit rate threshold, it means that the data block storage in the first storage space is unreasonable, and the heat degree calculation parameter of the data block needs to be updated.

[0131] Specifically, the heat degree calculation parameter of the data block includes the preset priority, the preset heat degree step, and the reference heat degree value, which are mainly used to calculate the initial heat degree value of the new data block; and the heat degree calculation parameter of the data block also includes the first heat degree adjustment value and the second heat degree adjustment value in the above embodiment.

[0132] When the first hotness adjustment value and / or the second hotness adjustment value has a proportional relationship with the reference hotness value, updating the reference hotness value is equivalent to updating the first hotness adjustment value and the second hotness adjustment value.

[0133] Optionally, the hotness calculation parameter of the data block further includes a first hotness value threshold.

[0134] Taking updating of the reference hotness value, the first hotness adjustment value and the preset hotness step length as an example, the hotness calculation parameter is updated, including: setting a reasonable range and an adjustment step length of each parameter, fixing any one of the parameters, determining an optimal combination of the other two parameters based on the data access request hit rate of the first storage space, further changing the fixed parameter, and gradually determining the optimal combination of the reference hotness value, the first hotness adjustment value and the preset hotness step length, so that the hit rate is higher than the preset hit rate threshold.

[0135] For example, the value range of the reference hotness value is set to 1-5, the value range of the first hotness adjustment value is set to 0.1-0.5, the value range of the preset hotness step length is set to 0.5-1, and the adjustment step length is 0.5.

[0136] After updating the hotness calculation parameter, the hit rate of the data access request is higher than the hit rate threshold, and the updating of the hotness calculation parameter can be stopped; or the optimal hotness calculation parameter is further searched, and the hotness calculation parameter is updated whenever a hotness calculation parameter with a higher hit rate of the data access request is found.

[0137] Optionally, when all possible hotness calculation parameter combinations are explored, the hit rate of the data access request is still less than the preset hit rate threshold, an error report is generated and the user is prompted for manual intervention to check whether the parameters and the preset hit rate threshold are reasonable.

[0138] The embodiments of the present disclosure calculate the initial hotness value of the new data through the preset priority, the preset hotness step length and the reference hotness value, and provide a reasonable data basis for updating of the data block hotness value in the subsequent process; meanwhile, whether the hotness calculation parameter is reasonable is determined through monitoring of the hit rate of the data access request, and the hotness calculation parameter is updated to improve the hit rate of the data access request.

[0139] In some embodiments, the method comprises: in response to a data write request of to-be-written data, if the to-be-written data exists in the first storage space, updating a response time of a data block corresponding to the to-be-written data to a current time.

[0140] Specifically, if the data write request involves data update, the to-be-written data is written into the corresponding data block, and the response time of the data block is updated to the current time.

[0141] In some embodiments, the method further comprises: in response to a data write request of to-be-written data, if the to-be-written data does not exist in the first storage space and a size of the to-be-written data is not greater than a remaining storage capacity in the first storage space, writing the to-be-written data into the first storage space and recording a response time of a data block corresponding to the to-be-written data as a current time.

[0142] When the to-be-written data does not exist in the first storage space and the remaining storage capacity of the first storage space is sufficient to support storage of the to-be-written data, the to-be-written data is directly written into the first storage space.

[0143] Specifically, referring to the above embodiments, a hotness value of to-be-written data is calculated, a to-be-written data block is formed based on the to-be-written data and the hotness value corresponding to the to-be-written data, and the to-be-written data block is stored into the first storage space. The response time of the to-be-written data block is the current time.

[0144] In some embodiments, the method further comprises: in response to a data write request of to-be-written data, if the to-be-written data does not exist in the first storage space and a size of the to-be-written data is greater than a remaining storage capacity in the first storage space, writing the to-be-written data into the second storage space and recording a response time of a data block corresponding to the to-be-written data as a current time.

[0145] When the to-be-written data does not exist in the first storage space and the remaining storage capacity of the first storage space is not sufficient to support storage of the to-be-written data, the to-be-written data is written into the second storage space.

[0146] Specifically, referring to the above embodiments, a hotness value of to-be-written data is calculated, a to-be-written data block is formed based on the to-be-written data and the hotness value corresponding to the to-be-written data, and the to-be-written data block is stored into the second storage space. The response time of the to-be-written data block is the current time.

[0147] In some embodiments, the method further comprises: in response to a data read request of a target data block, if the target data block only exists in the second storage space and a size of the target data block is not greater than a remaining storage capacity in the first storage space, migrating the target data block to the first storage space to complete the data read request based on the first storage space. Meanwhile, a response time of the target data block is updated as a current time.

[0148] The process of migrating the target data to the first storage space involves data transmission between the first storage space and the second storage space, which has a relatively small cost compared to directly completing the data read request based on the second storage space. Therefore, when there is sufficient remaining storage capacity in the first storage space, the target data block is preferentially migrated to the first storage space, and the data read request is completed based on the first storage space.

[0149] In some embodiments, the method further includes: in response to a data read request for a target data block, if the target data block is located in the first storage space, completing the data read request based on the first storage space. At the same time, updating the response time of the target data block to the current time.

[0150] In some embodiments, the target data block in the first storage space can be a backup data block, and in this case, the first storage space and the second storage space both contain the target data block. The data read request is preferentially completed based on the first storage space.

[0151] In some embodiments, the method further includes: in response to a data read request for a target data block, if the target data block is only located in the second storage space and the size of the target data block is greater than the remaining storage capacity in the first storage space, completing the data read request based on the second storage space. At the same time, updating the response time of the target data block to the current time.

[0152] When the remaining storage capacity of the first storage space is insufficient, the target data block in the second storage space cannot be migrated to the first storage space, and in this case, the data read request is completed based on the second storage space.

[0153] The embodiments of the present disclosure set the corresponding target data block read-write mode for various cases, so that the converged storage system can meet various data access requests, and preferentially meet the data access request based on the first storage space, thereby improving the efficiency of data read-write while ensuring the stability of data read-write. In addition, the embodiments of the present disclosure update the response time of the target data block according to the data access request, which facilitates dynamic management of the heat value and further improves the reliability of data processing.

[0154] Figure 4 The data processing device provided by the embodiments of the present disclosure is applied to a converged storage system, and the converged storage system includes a first storage space and a second storage space. The read-write latency of the first storage space is less than the read-write latency of the second storage space. Figure 4As shown, the data processing apparatus 40 comprises: an updating module 41, a determining module 42, and a migration module 43; the updating module 41 is configured to update a hotness value of a data block based on access information of the data block; the determining module 42 is configured to determine a target migration data block in a first storage space according to the hotness value of the data block when a data storage condition in the first storage space meets a preset data flushing condition; and the migration module 43 is configured to migrate the target migration data block to a second storage space.

[0155] Optionally, the updating module 41 comprises a hotness value increasing unit 411 and a hotness value decreasing unit 412; the hotness value increasing unit 411 is configured to increase the hotness value of the data block in response to an access operation on the data block, and update a response time of the data block in the access information as a time when the access operation occurs; and the hotness value decreasing unit 412 is configured to decrease the hotness value of the data block based on a time difference between the response time and a current time.

[0156] Optionally, the hotness value increasing unit 411 is configured to increase the hotness value of the data block by a first hotness adjustment value each time an access operation on the data block occurs.

[0157] Optionally, the hotness value decreasing unit 412 is configured to determine a second hotness adjustment value based on the time difference, the second hotness adjustment value being proportional to the time difference; and decrease the hotness value of the data block by the second hotness adjustment value.

[0158] Optionally, the preset data flushing condition comprises: a remaining storage capacity in the first storage space is less than a first storage capacity threshold; and / or, a size of a key data block in the first storage space is greater than a second storage capacity threshold, the key data block being at least one data block in the first storage space whose hotness value is higher than a first hotness value threshold.

[0159] Optionally, the determining module 42 is specifically configured to select a plurality of data blocks with the lowest hotness values in the first storage space as candidate data blocks; and select a target migration data block from the candidate data blocks, so that a size of the target migration data block is greater than a third storage capacity threshold, and a number of the target migration data blocks is the least.

[0160] Optionally, the migration module 43 comprises a first determining unit 431, a copying unit 432, and a backup unit 433. The first determining unit 431 is configured to determine backup data blocks in the target migration data blocks, the backup data blocks being a plurality of data blocks with the highest hotness values in the target migration data blocks, and a size of the backup data blocks and a size of non-target migration data blocks in the first storage space being less than a fourth storage capacity threshold. The copying unit 432 is configured to copy the target migration data blocks to the second storage space. The backup unit 433 is configured to retain the backup data blocks in the first storage space and delete target migration data blocks other than the backup data blocks in the first storage space.

[0161] Optionally, the data processing apparatus 40 comprises a data block acquisition module 44, which comprises an acquisition unit 441, a calculation unit 442, a construction unit 443, and a storage unit 444. The acquisition unit 441 is configured to acquire a preset priority of the to-be-written data in response to a data write event trigger. The calculation unit 442 is configured to calculate a hotness value of the to-be-written data based on hotness value calculation parameters, the hotness value calculation parameters comprising the preset priority, a preset hotness step, and a reference hotness value. The construction unit 443 is configured to construct a to-be-written data block based on the to-be-written data and the hotness value of the to-be-written data. The storage unit 444 is configured to store the to-be-written data block in the converged storage system.

[0162] Optionally, the calculation unit 442 is configured to calculate a hotness bonus value based on the preset priority and the preset hotness step, the hotness bonus value being proportional to the preset priority, and calculate a sum of the reference hotness value and the priority bonus hotness value to obtain the hotness value of the to-be-written data block.

[0163] Optionally, the data processing apparatus 40 further comprises a parameter optimization module 45, which comprises a statistics unit 451 and a parameter updating unit 452. The statistics unit 451 is configured to count a hit rate of a data access request in the converged storage system, the hit rate being a probability that the first storage space stores target data corresponding to the data access request. The parameter updating unit 452 is configured to update the hotness value calculation parameters when the hit rate is lower than a preset hit rate threshold, so as to make the hit rate higher than the preset hit rate threshold.

[0164] Optionally, the data processing apparatus 40 further comprises a request processing module 46, the request processing module 46 comprising a write request processing unit 461; the write request processing unit 461 is configured to, in response to a data write request of to-be-written data, update a response time of a data block corresponding to the to-be-written data to a current time if the to-be-written data exists in the first storage space; or, if the to-be-written data does not exist in the first storage space and a size of the to-be-written data is not greater than a remaining storage capacity in the first storage space, write the to-be-written data into the first storage space and record the response time of the data block corresponding to the to-be-written data as the current time.

[0165] Optionally, the request processing module 46 comprises a read request processing unit 462; the read request processing unit 462 is configured to, in response to a data read request of a target data block, if the target data block only exists in the second storage space and a size of the target data block is not greater than a remaining storage capacity in the first storage space, migrate the target data block to the first storage space to complete the data read request based on the first storage space.

[0166] Optionally, the data processing apparatus 40 further comprises an exception processing module 47 configured to process an exception condition in a read-write process to guarantee consistency of data. Specifically, the exception processing module 47 is configured to, after completing all possible explorations of combinations of heat calculation parameters, if a hit rate of a data access request is still less than a preset hit rate threshold, generate an error report and prompt a user to perform manual intervention.

[0167] Figure 4 The data apparatus of the illustrated embodiment can be used to execute the technical solutions of the above-mentioned method embodiments, and the implementation principles and technical effects are similar, which will not be described here again.

[0168] Figure 5 A structural schematic diagram of a fusion storage system management device provided by the embodiments of the present disclosure is provided. The fusion storage system management device can be used to manage the fusion storage system as described in the above embodiments. The electronic device provided by the embodiments of the present disclosure can execute the processing flow provided by the kernel upgrade method embodiments, such as Figure 5 As shown, the fusion storage system management device 50 comprises a storage 51, a processor 52, a computer program and a communication interface 53; wherein the computer program is stored in the storage 51 and is configured to be executed by the processor 52 to execute the data processing method as described above.

[0169] In addition, the embodiments of the present disclosure further provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data processing method described in the above embodiments.

[0170] It should be noted that the computer-readable medium in the above disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) link, and the like, or any suitable combination of the above.

[0171] In some embodiments, the client, server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communications (e.g., communications networks) of any form or medium, including the Internet. Examples of communications networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0172] In addition, the embodiments of the present disclosure also provide a computer program product, which includes a computer program or instructions, and the computer program or instructions are executed by a processor to realize the data processing method as described above.

[0173] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0174] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized by, The method is applied to a fusion storage system including a first storage space and a second storage space, a read-write latency of the first storage space being less than a read-write latency of the second storage space, and the data processing method comprising: updating a hotness value of a data block based on access information of the data block; when a data storage condition in the first storage space meets a preset data writing condition, determining a target migration data block in the first storage space according to the hotness value of the data block; migrating the target migration data block to the second storage space; the migrating the target migration data block to the second storage space comprises: determining a backup data block in the target migration data block, the backup data block being a plurality of data blocks with the highest hotness value in the target migration data block, and a size of the backup data block and a size of a non-target migration data block in the first storage space being less than a fourth storage capacity threshold; copying the target migration data block to the second storage space; retaining the backup data block in the first storage space and deleting the target migration data block in the first storage space except the backup data block; wherein the determining the backup data block in the target migration data block comprises: determining a plurality of data block combinations based on the target migration data block, a size of each data block combination being less than a fourth storage capacity value, calculating a hotness value sum and a data block quantity of the target migration data block in each data block combination, determining a first score of the data block combination based on the hotness value sum, the hotness value sum being proportional to the first score, determining a second score of the data block combination based on the data block quantity, the data block quantity being proportional to the second score, and calculating a sum of the first score and the second score to obtain a data block combination score; and selecting a plurality of target migration data blocks in a data block combination with the highest data block combination score as the backup data block.

2. The data processing method according to claim 1, characterized in that, The updating the hotness value of the data block based on the access information of the data block comprises: in response to an access operation on the data block, increasing the hotness value of the data block, and updating a response time of the data block in the access information to a time when the access operation occurs; based on a time difference between the response time and a current time, decreasing the hotness value of the data block.

3. The data processing method according to claim 2, characterized in that, The hotness value of the data block comprises a reference hotness value, and the increasing the hotness value of the data block in response to the access operation on the data block comprises: increasing the hotness value of the data block by a first hotness adjustment value each time the access operation on the data block occurs.

4. The data processing method according to claim 2, characterized in that, The decreasing the hotness value of the data block based on the time difference between the response time and the current time comprises: determining a second hotness adjustment value based on the time difference, the second hotness adjustment value being proportional to the time difference; decreasing the hotness value of the data block by the second hotness adjustment value.

5. The data processing method of claim 1, wherein, The preset data writing condition comprises: a remaining storage capacity in the first storage space being less than a first storage capacity threshold; and / or, The size of the key data block in the first storage space is greater than a second storage capacity threshold, and the key data block is at least one data block in the first storage space with a hotness value higher than a first hotness value threshold.

6. The data processing method of claim 1, wherein, The method further comprises: selecting a plurality of data blocks with the lowest hotness value in the first storage space as candidate data blocks; selecting a target migration data block from the candidate data blocks, so that the size of the target migration data block is greater than a third storage capacity threshold, and the number of the target migration data block is the least.

7. The data processing method of claim 1, wherein, The method further comprises: in response to a data write event trigger of to-be-written data, obtaining a preset priority of the to-be-written data; based on a hotness value calculation parameter, calculating a hotness value of the to-be-written data, the hotness value calculation parameter comprising the preset priority, a preset hotness step and a reference hotness value; based on the to-be-written data and the hotness value of the to-be-written data, constructing a to-be-written data block; storing the to-be-written data block into the fusion storage system.

8. The data processing method according to claim 7, characterized in that, The method further comprises: based on the preset priority and the preset hotness step, calculating a hotness addition value, the hotness addition value being proportional to the preset priority; calculating the sum of the reference hotness value and the priority addition hotness value to obtain the hotness value of the to-be-written data block.

9. The data processing method according to claim 7, characterized in that, The method further comprises: statistically obtaining a hit rate of a data access request in the fusion storage system, the hit rate being a probability that the first storage space stores target data corresponding to the data access request; when the hit rate is lower than a preset hit rate threshold, updating the hotness value calculation parameter so that the hit rate is higher than the preset hit rate threshold.

10. The data processing method of claim 1, wherein, The method comprises: in response to a data write request of to-be-written data, if the to-be-written data exists in the first storage space, updating the response time of the data block corresponding to the to-be-written data to the current time; or, if the to-be-written data does not exist in the first storage space, and the size of the to-be-written data is not greater than the remaining storage capacity in the first storage space, writing the to-be-written data into the first storage space, and recording the response time of the data block corresponding to the to-be-written data as the current time.

11. The data processing method of claim 1, wherein, The method further comprises: in response to a data read request for a target data block, if the target data block is only located in the second storage space, and the size of the target data block is not greater than the remaining storage capacity in the first storage space, migrating the target data block to the first storage space to complete the data read request based on the first storage space.

12. A data processing apparatus, characterized by The device is applied to a fusion storage system, the fusion storage system comprising a first storage space and a second storage space, the read-write latency of the first storage space being less than the read-write latency of the second storage space, and the data processing device comprising: an updating module configured to update the hotness value of a data block based on access information of the data block; The determining module is configured to determine target migration data blocks in the first storage space according to the heat values of the data blocks when the data storage condition in the first storage space meets a preset data flushing condition. The migration module is configured to migrate the target migration data blocks to a second storage space. The migration module includes a first determining unit, a copying unit, and a backup unit. The first determining unit is configured to determine backup data blocks in the target migration data blocks. The backup data blocks are a plurality of data blocks with the highest heat values in the target migration data blocks, and the size of the backup data blocks and the size of non-target migration data blocks in the first storage space are less than a fourth storage capacity threshold. The copying unit is configured to copy the target migration data blocks to the second storage space. The backup unit is configured to retain the backup data blocks in the first storage space and delete target migration data blocks in the first storage space other than the backup data blocks. The method includes determining backup data blocks in the target migration data blocks, including: determining a plurality of data block combinations based on the target migration data blocks, the size of the target migration data blocks and non-target migration data blocks in the first storage space in each data block combination being less than a fourth storage capacity value, calculating the heat value sum and the data block quantity of the target migration data blocks in each data block combination, determining a first score of the data block combination based on the heat value sum, the heat value sum being directly proportional to the first score, determining a second score of the data block combination based on the data block quantity, the data block quantity being directly proportional to the second score, and calculating the sum of the first score and the second score to obtain a data block combination score; and selecting a plurality of target migration data blocks in the data block combination with the highest data block combination score as the backup data blocks.

13. A converged storage system management appliance, comprising: The method includes: a memory; a processor; and a computer program; The computer program is stored in the memory and is configured to be executed by the processor to implement the method of any one of claims 1-11. The computer program is executed by the processor to implement the method of any one of claims 1-11.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images