Data processing method and system

Abstract

The embodiment of the invention relates to a data processing method and system, and relates to the technical field of storage. The method comprises the following steps: receiving an access request for a data block, wherein data corresponding to the data block is stored in a first storage pool; in response to the access request, determining a second storage pool according to the data popularity of the data block and the bandwidth required by the data block; and writing the data corresponding to the data block into a second storage pool. Therefore, the overall access efficiency of the storage pool in the cloud computing platform can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of storage technology, and more particularly to a data processing method and system. Background Art

[0002] As technology develops, cloud computing platforms will eliminate poorly performing or old storage pools and add higher-performing or new storage pools, thereby continuously upgrading, replacing and building cloud storage software and hardware arrays, and continuously backing up and migrating data. Summary of the invention

[0003] According to a first aspect of an embodiment of the present application, a data processing method is provided, the method comprising: receiving an access request for a data block, the data corresponding to the data block being stored in a first storage pool; in response to the access request, determining a second storage pool based on the data heat of the data block and the bandwidth required by the data block; and writing the data corresponding to the data block into the second storage pool.

[0004] In the above technical solution, an access request for a data block is received, and a second storage pool is determined based on the data heat of the data block and the bandwidth required by the data block, and the data corresponding to the data block is written from the first storage pool to the second storage pool. Data migration is performed in units of data blocks, and the data corresponding to the data block is migrated to a storage pool with corresponding performance based on the data heat of the data block and the required bandwidth, thereby avoiding the uneven migration of hot and cold data in multiple data of the same file to the same storage pool, and data can be migrated accurately. In this way, the cloud storage data migration strategy is redefined, and the cloud storage space management capability is improved. For example, the overall efficiency of accessing a large amount of data in the storage pool and the bandwidth utilization efficiency of the storage pool are improved, thereby reducing storage costs.

[0005] In a possible implementation of the first aspect, determining the second storage pool according to the data heat of the data block and the bandwidth required by the data block includes: determining the second storage pool when the heat difference between the data heat of the data block and the reference heat is within the heat difference threshold range between the heat of the data block supported by the second storage pool and the reference heat, and the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the bandwidth difference threshold range between the second storage pool and the first storage pool. In the above possible implementation, according to the heat difference between the data heat of the data block and the reference heat, the heat change of the data block targeted by the access request can be obtained. If the heat difference is within the heat difference threshold range between the heat of the data block supported by the second storage pool and the reference heat, then the data block targeted by the access request meets the temperature condition for migration to the second storage pool. According to the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool, the bandwidth of the data block targeted by the access request can be obtained. If the bandwidth difference is within the bandwidth difference threshold range between the second storage pool and the first storage pool, then the data block targeted by the access request meets the bandwidth condition for migration to the second storage pool. When the data block targeted by the access request meets both the heat condition and the bandwidth condition for migration to the second storage pool, the data corresponding to the data block is migrated to the second storage pool. This can accurately migrate the data, improve the overall efficiency when accessing a large amount of data in the storage pool, and improve the bandwidth utilization efficiency of the storage pool, thereby reducing storage costs.

[0006] In a possible implementation of the first aspect, the performance of the second storage pool is different from that of the first storage pool, and the performance includes access speed and / or access delay. In the above possible implementation, data corresponding to data blocks with different heat and bandwidth changes are migrated to storage pools with corresponding performance, and the performance of the storage pool includes access speed and / or access delay, so that the overall efficiency of accessing a large amount of data in the storage pool and the bandwidth utilization efficiency of the storage pool can be improved, thereby reducing storage costs.

[0007] In a possible implementation of the first aspect, the data heat of the data block is determined based on the number of accesses to the data block and the access request delay of the data block, and the access request delay of the data block is: the time difference between the time of accessing the storage area storing the data corresponding to the data block in the first storage pool and the time of receiving the response from the first storage pool. In the above possible implementation, the data heat is determined based on the number of accesses to the data block and the access request delay of the data block, and the access request delay of the data block is: the time difference between the time of accessing the storage area storing the data corresponding to the data block in the first storage pool and the time of receiving the response from the first storage pool. Considering the number of data accesses and the access request delay at the same time, so as to formulate a data block migration strategy, the overall data access rate can be improved and the overall data access time can be reduced.

[0008] In a possible implementation of the first aspect, the reference temperature is the temperature of the data block supported by the first storage pool for storage. In the above possible implementation, the reference temperature is determined according to the temperature of the data block supported by the first storage pool for storage, and the temperature change of the data block targeted by the access request can be obtained, providing a basis for accurate data migration.

[0009] In a possible implementation of the first aspect, the reference heat is the historical access heat of the data block. In the above possible implementation, the reference heat is determined according to the historical access heat of the data block, and the heat change of the data block can be obtained, providing a basis for accurate data migration.

[0010] In a possible implementation of the first aspect, the method further includes: receiving first indication information, the first indication information being used to indicate the remaining bandwidth and reference heat of the first storage pool. In the above possible implementation, the first indication information indicating the remaining bandwidth and reference heat of the first storage pool can be received to provide a basis for analyzing the heat gap and the bandwidth gap.

[0011] In a possible implementation of the first aspect, writing the data corresponding to the data block to the second storage pool includes: writing the data corresponding to the data block to the second storage pool within a preset time, the preset time is determined based on the access network delay of the data block, and the access network delay of the data block is: the time difference between the sending time of the access request and the receiving time of the access request. In the above possible implementation, the preset time is determined based on the access network delay of the data block, and the data corresponding to the data block is written to the second storage pool within the preset time. The data corresponding to the data block can be migrated in real time based on the access network delay of the data block, or the data corresponding to the data block can be migrated when the device is relatively idle, so that the cloud storage space management is more flexible.

[0012] In a possible implementation of the first aspect, the method further includes: receiving second indication information, the second indication information being used to indicate an access network delay of the data block. In the above possible implementation, the access network delay of the data block is stored in other devices, and when the access network delay is needed, receiving the second indication information for indicating the access network delay of the data block can save storage space and provide a basis for determining the preset time.

[0013] In a possible implementation of the first aspect, the data block is one of multiple data blocks of the target file, and data corresponding to at least two of the multiple data blocks are stored in different storage pools. In the above possible implementation, since data migration is performed in units of data blocks, data corresponding to different data blocks of the same file can be migrated to different storage pools, without being restricted to storing data corresponding to data blocks of the same file in the same storage pool. In this way, data can be migrated accurately, improving the overall efficiency of accessing a large amount of data in the storage pool and the bandwidth utilization efficiency of the storage pool, thereby reducing storage costs.

[0014] According to a second aspect of an embodiment of the present application, a data processing system is provided, which includes a computing node, and a first storage pool and a second storage pool coupled to the computing node, wherein the computing node is configured to: receive an access request for a data block, the data corresponding to the data block being stored in the first storage pool; determine a second storage pool in response to the access request based on the data heat of the data block and the bandwidth required by the data block; and write the data corresponding to the data block into the second storage pool.

[0015] In a possible implementation of the second aspect, the computing node is specifically configured to determine the second storage pool when the temperature difference between the data temperature of the data block and a reference temperature is within a temperature difference threshold range between the temperature of the data block supported for storage by the second storage pool and the reference temperature, and the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within a bandwidth difference threshold range between the second storage pool and the first storage pool.

[0016] In a possible implementation manner of the second aspect, performance of the second storage pool is different from performance of the first storage pool, and the performance includes access speed and / or access delay.

[0017] In a possible implementation of the second aspect, the data heat of the data block is determined based on the number of times the data block is accessed and the access request delay of the data block, and the access request delay of the data block is: the time difference between the time of accessing the storage area storing the data corresponding to the data block in the first storage pool and the time of receiving the response from the first storage pool.

[0018] In a possible implementation manner of the second aspect, the reference temperature is the temperature of the data blocks supported for storage by the first storage pool.

[0019] In a possible implementation manner of the second aspect, the reference temperature is a historical access temperature of the data block.

[0020] In a possible implementation manner of the second aspect, the computing node is further configured to: receive first indication information, where the first indication information is used to indicate a remaining bandwidth and a reference heat of the first storage pool.

[0021] In a possible implementation of the second aspect, the computing node is specifically configured to write the data corresponding to the data block into the second storage pool within a preset time, where the preset time is determined based on the access network delay of the data block, and the access network delay of the data block is the time difference between the sending time of the access request and the receiving time of the access request.

[0022] In a possible implementation manner of the second aspect, the computing node is further configured to: receive second indication information, where the second indication information is used to indicate an access network delay of the data block.

[0023] In a possible implementation manner of the second aspect, the data block is one of multiple data blocks of the target file, and data corresponding to at least two of the multiple data blocks are stored in different storage pools.

[0024] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a program code is stored. The program code can be called by a processor to execute the method provided by the first aspect or any possible implementation of the first aspect.

[0025] According to another aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the method provided by the first aspect or any possible implementation manner of the first aspect.

[0026] It can be understood that any of the data processing systems, computer storage media or computer program products provided above are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A cloud platform architecture provided for the embodiment of the present application Figure 1 ;

[0028] Figure 2 A cloud platform architecture provided for the embodiment of the present application Figure 2 ;

[0029] Figure 3 A schematic diagram of the structure of a cloud platform provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of a data processing method provided in an embodiment of the present application Figure 1 ;

[0031] Figure 5 A schematic diagram of a data processing method provided in an embodiment of the present application Figure 2 ;

[0032] Figure 6 A schematic diagram of a data processing method provided in an embodiment of the present application Figure 3 ;

[0033] Figure 7 A schematic diagram of a data processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in some embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments provided by the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present application.

[0035] In the description of the present application, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0036] When describing some embodiments, the term "coupled" may be used to indicate that two or more components are in direct physical or electrical contact. However, the term "coupled" may also refer to two or more components that are not in direct contact with each other but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the contents of this document.

[0037] "At least one of A, B, and C" has the same meaning as "at least one of A, B, or C", and both include the following combinations of A, B, and C: only A, only B, only C, a combination of A and B, a combination of A and C, a combination of B and C, and a combination of A, B, and C. "A and / or B" includes the following three combinations: only A, only B, and a combination of A and B. The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps.

[0038] Before introducing the embodiments of the present application, the application scenarios of the embodiments of the present application are first introduced and explained. The technical solution provided in the embodiments of the present application can be applied to a cloud computing platform. A cloud computing platform, also referred to as a cloud platform, refers to a service based on hardware resources and software resources, providing computing, network and storage capabilities. Computing nodes and storage nodes can be nodes in a cloud platform. The computing node is used to perform the function of data computing, and the hardware resources of the computing node may include computing devices such as servers, desktop computers or processors. The storage node is used to perform the function of data storage, and the hardware resources of the storage node may include a storage pool. Exemplarily, the hardware resources of the computing node include a communication interface and a processor, and the hardware resources of the storage node include a communication interface and a storage pool. There can be multiple computing nodes and storage nodes in the cloud platform, and the computing nodes and storage nodes can communicate through the communication interface.

[0039] In one embodiment, the cloud platform adopts a storage-computing integrated structure, that is, the computing nodes and storage nodes are set in the same device. Figure 1 As shown, the device 100 includes multiple storage nodes (storage nodes 111, 112, 113, ...) and a computing node 120. Each of the multiple storage nodes is coupled to the computing node 120, and the computing node 120 can store data in any one of the multiple storage nodes.

[0040] In another embodiment, the cloud platform adopts a storage-computing separation structure, that is, the computing nodes and storage nodes are set in different devices. Figure 2 As shown, device 210 includes storage node 211, device 220 includes storage node 221, device 230 includes computing node 231, and the cloud platform may further include more devices and storage nodes therein. Device 240 includes computing node 241, and device 240 may communicate with devices 210, 220, 230, ... through a communication network, so that computing node 241 may store data in any one of storage nodes 211, 221, 231, ...

[0041] As technology develops, cloud platforms will eliminate poor-performing or old storage pools in storage nodes, and add high-performing or new storage pools, so as to continuously upgrade, replace and build cloud storage hardware and software arrays, and continuously perform data backup and migration. Data migration can be performed in units of files. Files often include multiple data blocks. The data corresponding to each of the multiple data blocks can be accessed independently. According to the access situation of the data corresponding to each data block, the heat of each data block can be determined. The data blocks with higher heat are usually a small number of data blocks in a file. If the heat of some data blocks in a file is higher, the heat of the file will increase accordingly. Exemplarily, the heat of a file is the number of accesses to the file per unit time, and the number of accesses to a file is the sum of the number of accesses to the data corresponding to all data blocks in the file. According to the migration strategy, the files with increased heat are migrated to storage pools with higher performance.

[0042] For example, Figure 3 As shown, the cloud platform includes computing nodes 310, storage nodes 320, storage nodes 330 and other storage nodes, storage nodes 320 include storage pools 321, 322 and other storage pools, and storage nodes 330 include storage pools 331, 332 and other storage pools. The computing node 310 in the cloud platform can communicate with terminals 351, 352, 353 and other terminals through cloud interfaces 340. File 311 is stored in the storage pool 332 of the storage node 330, and the file 311 includes multiple data blocks. Since the popularity of file 311 is high, the computing node 310 writes the file 311 from the storage pool 332 of the storage node 330 to the storage pool 321 of the storage node 320, wherein the file 311 includes not only the data blocks with higher popularity, but also other data blocks, such as the data blocks with lower popularity, and the data migration efficiency and flexibility of the cloud platform are low.

[0043] Please refer to Figure 4 , the embodiment of the present application provides a data processing method, including:

[0044] S110: Receive an access request for a data block, data corresponding to the data block is stored in a first storage pool.

[0045] S120: In response to the access request, determine a second storage pool according to the data popularity of the data block and the bandwidth required by the data block.

[0046] S130: Writing data corresponding to the data block into the second storage pool.

[0047] The access request may be a read request. The data heat may be the current heat of the data block targeted by the access request. The bandwidth required by the data block is related to the data heat. If the heat of the data block is higher and the storage pool needs to transmit more data corresponding to the data block at the same time, the bandwidth required by the data block is greater.

[0048] Exemplarily, the method is applied to a cloud platform, and the storage system of the cloud platform includes an Internet application, a public cloud storage interface, a cloud storage file system, a data block temperature server, a data block temperature client, etc. The Internet application can be run on a terminal, the cloud storage file system and the data block temperature server can be run on a computing node, and the data block temperature client can be run on a storage node. The Internet application can communicate with the cloud storage file system through the public cloud storage interface.

[0049] Exemplarily, the terminal sends an access request for the data block to the computing node. Prior to this, the computing node allocates a storage pool for the data corresponding to the data block, and stores the data corresponding to the data block in a corresponding storage pool, such as the first storage pool.

[0050] The computing node receives the access request for the data block, and sends a read command to the first storage pool according to the access request. The first storage pool receives the read command and reads the corresponding data, and sends the data corresponding to the data block to the computing node.

[0051] The computing node receives data corresponding to the data block, and determines a second storage pool according to data popularity of the data block and bandwidth required by the data block.

[0052] The computing node sends a write command involving the data corresponding to the data block to the second storage pool. The second storage pool receives the data corresponding to the data block and completes the writing of the data corresponding to the data block.

[0053] In some embodiments, after the computing node sends the data corresponding to the data block to the second storage pool, the first storage pool may delete the data corresponding to the data block or mark it as invalid.

[0054] Exemplarily, the first storage pool and the second storage pool may be located in the same storage node or in different storage nodes, that is, the data corresponding to the data block targeted by the access request may be migrated in the same storage node or between different storage nodes. For multiple data in the storage pool, they may be migrated in the same storage node or between different storage nodes in units of data blocks.

[0055] Exemplarily, the data block to be migrated may be a gradually hot (increasing temperature) data block or a gradually cold (decreasing temperature) data block. For example, in response to an access request, based on the data heat of the data block and the bandwidth required by the data block, it is determined that the data block is a gradually hot data block, and the data corresponding to the data block is migrated to the second storage pool, and the performance of the second storage pool is higher than that of the first storage pool. For another example, in response to an access request, based on the data heat of the data block and the bandwidth required by the data block, it is determined that the data block is a gradually cold data block, and the data corresponding to the data block is migrated to the second storage pool, and the performance of the second storage pool is lower than that of the first storage pool.

[0056] Exemplarily, a constant-hot (hotness does not change much) data block may not be migrated. For example, in response to an access request, based on the data heat of the data block and the bandwidth required by the data block, if the data block is determined to be a constant-hot data block, the data corresponding to the data block will not be migrated, and the data corresponding to the data block will be stored in the first storage pool.

[0057] When storing massive amounts of data in a storage pool, as time goes by, the value stability of most data blocks decreases, and it is necessary to formulate a migration strategy according to certain rules, such as the heat of the data blocks. The embodiment of the present application identifies the gradually hot data blocks, gradually cold data blocks, and constant hot data blocks in the storage pool, determines the migration strategy according to the heat of the data blocks, and migrates the data corresponding to the data blocks to the corresponding storage pools. For example, according to the quality of service (QoS) strategy, the data corresponding to the gradually hot data blocks are migrated to the storage pools with higher performance, higher bandwidth, and higher cost, and the data corresponding to the gradually cold data blocks are migrated to the storage pools with lower performance, lower bandwidth, and lower cost, and the data corresponding to the constant hot data blocks are not migrated. In this way, it meets the current development needs of cloud computing. New higher-performance storage pools and computing nodes are added every day, but the data blocks will change in heat every second. The computing nodes continuously migrate the data corresponding to the data blocks to the corresponding storage pools according to the heat of the data, and complete the migration of the data corresponding to the data blocks in the process of hot and cold stratification of the data blocks, so that the data can be accurately migrated. In addition, the cloud storage file system running on the computing nodes is compatible with storage pools of various performances (such as hard disks), which improves the utilization rate of existing storage pools.

[0058] In the method provided in the embodiment of the present application, an access request for a data block is received, and a second storage pool is determined based on the data heat of the data block and the bandwidth required by the data block, and the data corresponding to the data block is migrated from the first storage pool to the second storage pool. Data migration is performed in units of data blocks, and the data corresponding to the data block is migrated to a storage pool of corresponding performance based on the data heat of the data block and the required bandwidth, thereby avoiding the uneven migration of hot and cold data in multiple data of the same file to the same storage pool, and data can be migrated accurately. In this way, the cloud storage data migration strategy is redefined, and the cloud storage space management capability is improved. For example, the overall efficiency of accessing a large amount of data in the storage pool and the bandwidth utilization efficiency of the storage pool are improved, thereby reducing storage costs.

[0059] In a possible implementation, S120 (determining the second storage pool based on the data heat of the data block and the bandwidth required by the data block) may include: determining the second storage pool when the heat gap between the data heat of the data block and a reference heat is within a heat difference threshold range between the heat of the data block supported for storage by the second storage pool and the reference heat, and the bandwidth gap between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within a bandwidth difference threshold range between the second storage pool and the first storage pool.

[0060] The heat difference between the data heat of the data block and the reference heat may refer to: a value obtained by subtracting the reference heat from the data heat of the data block.

[0061] In one example, the reference temperature can be used to indicate the temperature of the data block targeted by the access request. For example, the reference temperature is the historical access temperature of the data block. Each time the data corresponding to the data block is accessed, the temperature of the data block will change, and the historical access temperature of the data block includes the temperature after each change. The reference temperature can be any historical access temperature of the data block. For example, the reference temperature is the temperature that changed when the data corresponding to the data block was last accessed. In this way, by determining the reference temperature based on the historical access temperature of the data block, the temperature change of the data block can be obtained, providing a basis for accurate data migration.

[0062] In another example, the reference heat can also be used to indicate the heat of other data blocks other than the data block targeted by the access request. For example, the heat of all data blocks stored in multiple storage pools of the storage node is divided into a preset number of levels, the performance of multiple storage pools of the storage node is divided into a preset number of levels, and the performance of each storage pool is matched with the heat of each level of data blocks. Among them, the preset number may be equal to the number of storage pools, or may not be equal to the number of storage pools, and the embodiment of the present application does not limit this. The reference heat is the heat of the data blocks supported by the first storage pool for storage, that is, the heat of the data blocks of the level corresponding to the performance of the first storage pool. Since the heat of the data blocks in the storage node will change, the above steps of classifying the heat of all data blocks in the storage node and matching the performance of each storage pool with the heat of each level of data blocks can be performed once every first period. In this example, the heat of the data blocks supported by the first storage pool for storage may be a first heat range. If the data heat of the data block is greater than the maximum value of the first heat range, the heat gap may be: the value obtained by subtracting the data heat of the data block from the maximum value of the first heat range. If the data heat of the data block is less than the minimum value of the first heat range, the heat difference can be: the value obtained by subtracting the minimum value of the first heat range from the data heat of the data block. In this way, the reference heat is determined according to the heat of the data block supported by the first storage pool, and the heat change of the data block targeted by the access request can be obtained, providing a basis for accurate data migration.

[0063] Furthermore, the temperature of the data blocks supported for storage by the second storage pool may be a second temperature range, and the temperature difference threshold range between the temperature of the data blocks supported for storage by the second storage pool and the reference temperature may refer to: a range obtained by subtracting the reference temperature from the second temperature range.

[0064] In one example, the reference heat is the historical access heat of the data block. According to the performance of the first storage pool, the first storage pool is set to support the storage of data blocks in the first heat range; according to the performance of the second storage pool, the second storage pool is set to support the storage of data blocks in the second heat range. If the historical access heat of the data block is less than the minimum value of the second heat range, the minimum value of the heat difference threshold range may be: the value obtained by subtracting the historical access heat of the data block from the minimum value of the second heat range; the maximum value of the heat difference threshold range may be: the value obtained by subtracting the historical access heat of the data block from the maximum value of the second heat range. If the historical access heat of the data block is greater than the maximum value of the second heat range, the minimum value of the heat difference threshold range may be: the value obtained by subtracting the historical access heat of the data block from the minimum value of the second heat range; the maximum value of the heat difference threshold range may be: the value obtained by subtracting the historical access heat of the data block from the maximum value of the second heat range.

[0065] In another example, the reference heat is the heat of the data blocks supported for storage by the first storage pool. According to the performance of the first storage pool, the first storage pool is set to support the storage of data blocks in the first heat range; according to the performance of the second storage pool, the second storage pool is set to support the storage of data blocks in the second heat range. If the maximum value of the first heat range is less than the minimum value of the second heat range, the minimum value of the heat difference threshold range may be: the value obtained by subtracting the maximum value of the first heat range from the minimum value of the second heat range; the maximum value of the heat difference threshold range may be: the value obtained by subtracting the maximum value of the first heat range from the maximum value of the second heat range. If the minimum value of the first heat range is greater than the maximum value of the second heat range, the minimum value of the heat difference threshold range may be: the value obtained by subtracting the minimum value of the first heat range from the minimum value of the second heat range; the maximum value of the heat difference threshold range may be: the value obtained by subtracting the minimum value of the first heat range from the maximum value of the second heat range.

[0066] Next, combine Figure 5 Taking the reference heat as the historical access heat of the data block as an example, the situation where the heat difference is within the heat difference threshold range is illustrated. The following specific heat values ​​are all examples. In practice, the heat value can be other values, and the embodiments of the present application do not limit this. Computing node 410 can communicate with storage pool 421, storage pool 422, and storage pool 423. The performance of storage pool 421, storage pool 422, and storage pool 423 gradually increases. The first storage pool is storage pool 422, and the data corresponding to data block 411 is stored in storage pool 422. Computing node 410 reads the data corresponding to data block 411 from storage pool 422.

[0067] In the first case, the data block is a gradually hot data block. The data temperature of the data block is 70 degrees, the historical access temperature of the data block is 50 degrees, the first temperature range is 31 to 60 degrees, and the second temperature range is 61 to 90 degrees. The temperature difference can be: the value obtained by subtracting the historical access temperature of the data block from the data temperature of the data block, that is, +20 degrees. The minimum value of the temperature difference threshold range can be: the value obtained by subtracting the historical access temperature of the data block from the minimum value of the second temperature range, that is, +11 degrees. The maximum value of the temperature difference threshold range can be: the value obtained by subtracting the historical access temperature of the data block from the maximum value of the second temperature range, that is, +40 degrees. The temperature difference (+20 degrees) between the data temperature of the data block and the reference temperature is within the temperature difference threshold range (+11 to +40 degrees) between the temperature of the data block supported for storage by the second storage pool and the reference temperature, and the second storage pool can be determined to be storage pool 423. The computing node 410 writes the data corresponding to the data block 411 into the storage pool 423 .

[0068] In the second case, the data block is a gradually cooling data block. The data temperature of the data block is 20 degrees, the historical access temperature of the data block is 50 degrees, the first temperature range is 31 to 60 degrees, and the second temperature range is 1 to 30 degrees. The temperature difference can be: the value obtained by subtracting the historical access temperature of the data block from the data temperature of the data block, that is, -30 degrees. The minimum value of the temperature difference threshold range can be: the value obtained by subtracting the historical access temperature of the data block from the minimum value of the second temperature range, that is, -49 degrees. The maximum value of the temperature difference threshold range can be: the value obtained by subtracting the historical access temperature of the data block from the maximum value of the second temperature range, that is, -20 degrees. The temperature difference (-30 degrees) between the data temperature of the data block and the reference temperature is within the temperature difference threshold range (-49 to -20 degrees) between the temperature of the data block supported for storage by the second storage pool and the reference temperature, and the second storage pool can be determined to be storage pool 421. The computing node 410 writes the data corresponding to the data block 411 into the storage pool 421 .

[0069] In the third case, the data block is a constant heat data block. The data heat of the data block is 60 degrees, the historical access heat of the data block is 50 degrees, the first heat range is 31 to 60 degrees, the data heat of the data block is within the first heat range, and the data corresponding to the data block does not need to be migrated.

[0070] Next, combine Figure 5 Taking the reference heat as the heat of the data block supported by the first storage pool for storage (first heat range) as an example, the situation where the heat difference is within the heat difference threshold range is illustrated. The following specific heat values ​​are all examples. In practice, the heat values ​​can be other values, and the embodiments of the present application do not limit this. Computing node 410 can communicate with storage pool 421, storage pool 422, and storage pool 423. The performance of storage pool 421, storage pool 422, and storage pool 423 gradually increases. The first storage pool is storage pool 422, and the data corresponding to data block 411 is stored in storage pool 422. Computing node 410 reads the data corresponding to data block 411 from storage pool 422.

[0071] In the first example, the data block is a gradually heated data block. The data temperature of the data block is 70 degrees, the first temperature range is 31 to 60 degrees, and the second temperature range is 61 to 90 degrees. The temperature difference can be: the value obtained by subtracting the maximum value of the first temperature range from the data temperature of the data block, that is, +10 degrees. The minimum value of the temperature difference threshold range can be: the value obtained by subtracting the maximum value of the first temperature range from the minimum value of the second temperature range, that is, +1 degree. The maximum value of the temperature difference threshold range can be: the value obtained by subtracting the maximum value of the first temperature range from the maximum value of the second temperature range, that is, +30 degrees. The temperature difference (+10 degrees) between the data temperature of the data block and the reference temperature is within the temperature difference threshold range (+1 to +30 degrees) between the temperature of the data block supported for storage by the second storage pool and the reference temperature, and the second storage pool can be determined to be storage pool 423. Computing node 410 writes the data corresponding to data block 411 into storage pool 423.

[0072] In the second example, the data block is a gradually cooling data block. The data temperature of the data block is 20 degrees, the first temperature range is 31 to 60 degrees, and the second temperature range is 1 to 30 degrees. The temperature difference can be: the value obtained by subtracting the minimum value of the first temperature range from the data temperature of the data block, that is, -11 degrees. The minimum value of the temperature difference threshold range can be: the value obtained by subtracting the minimum value of the first temperature range from the minimum value of the second temperature range, that is, -30 degrees. The maximum value of the temperature difference threshold range can be: the value obtained by subtracting the minimum value of the first temperature range from the maximum value of the second temperature range, that is, -1 degree. The temperature difference (-11 degrees) between the data temperature of the data block and the reference temperature is within the temperature difference threshold range (-1 to -30 degrees) between the temperature of the data block supported for storage by the second storage pool and the reference temperature, and the second storage pool can be determined to be storage pool 421. Computing node 410 writes the data corresponding to data block 411 into storage pool 421.

[0073] In the third example, the data block is a constant-heat data block. The data heat of the data block is 60 degrees, the first heat range is 31 to 60 degrees, the data heat of the data block is within the first heat range, and the data corresponding to the data block does not need to be migrated.

[0074] In addition, the remaining bandwidth of the first storage pool refers to the bandwidth that the first storage pool can provide. Since the hardware devices of the storage pool may age during use, the bandwidth that the first storage pool can provide may decrease with the use time. The bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool can be: the value obtained by subtracting the remaining bandwidth of the first storage pool from the bandwidth required by the data block. The bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the bandwidth difference threshold range between the second storage pool and the first storage pool, which can be: the bandwidth difference is less than the bandwidth difference threshold range.

[0075] In the method provided in the embodiment of the present application, the temperature change of the data block targeted by the access request can be obtained according to the temperature difference between the data temperature of the data block and the reference temperature. If the temperature difference is within the temperature difference threshold range between the temperature of the data block supported by the second storage pool and the reference temperature, then the data block targeted by the access request meets the temperature condition for migration to the second storage pool. According to the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool, the bandwidth of the data block targeted by the access request can be obtained. If the bandwidth difference is within the bandwidth difference threshold range between the second storage pool and the first storage pool, then the data block targeted by the access request meets the bandwidth condition for migration to the second storage pool. When the data block targeted by the access request meets both the temperature condition for migration to the second storage pool and the bandwidth condition for migration to the second storage pool, the data corresponding to the data block is migrated to the second storage pool, which can accurately migrate the data, improve the overall efficiency when accessing a large amount of data in the storage pool, and the bandwidth utilization efficiency of the storage pool, and reduce storage costs.

[0076] In a possible implementation, the performance of the second storage pool is different from that of the first storage pool, and the performance includes access speed and / or access delay. In the method provided in the embodiment of the present application, data corresponding to data blocks with different heat and bandwidth changes are migrated to storage pools with corresponding performance, and the performance of the storage pool includes access speed and / or access delay, so that the overall efficiency of accessing a large amount of data in the storage pool and the bandwidth utilization efficiency of the storage pool can be improved, thereby reducing storage costs.

[0077] In one possible implementation, the data heat of the data block is determined based on the number of times the data block is accessed and the access request delay of the data block, where the access request delay of the data block is the time difference between the time of accessing the storage area storing the data corresponding to the data block in the first storage pool and the time of receiving the response from the first storage pool.

[0078] Among them, the calculation formula for the data heat of the data block can be: data heat = number of access times of the data block * access request delay of the data block. Exemplarily, the access request is a read request, and the access request delay of the data block is: the time difference between the time when the computing node sends a read request to the storage pool and the time when the computing node receives the data block from the first storage pool. In the method provided in the embodiment of the present application, the data heat is determined according to the number of access times of the data block and the access request delay of the data block. The access request delay of the data block is: the time difference between the time to access the storage area storing the data corresponding to the data block in the first storage pool and the time to receive the response from the first storage pool. Considering the number of data accesses and the access request delay at the same time, a data block migration strategy is formulated, which can improve the overall data access rate and reduce the overall data access time.

[0079] In a possible implementation, the method further includes: receiving first indication information, the first indication information being used to indicate the remaining bandwidth and reference heat of the first storage pool. The first indication information may come from the storage pool. Exemplarily, the storage pool may send the remaining bandwidth of each storage pool and the reference heat of all data stored in the storage pool to the computing node every second period. In the method provided in an embodiment of the present application, first indication information indicating the remaining bandwidth and reference heat of the first storage pool may be received, providing a basis for analyzing heat gaps and bandwidth gaps.

[0080] In one possible implementation, S130 (writing the data corresponding to the data block into the second storage pool) may include: writing the data corresponding to the data block into the second storage pool within a preset time, the preset time is determined based on the access network delay of the data block, and the access network delay of the data block is: the time difference between the sending time of the access request and the receiving time of the access request.

[0081] Among them, the greater the access network delay, the shorter the preset time, and it is necessary to migrate to the second storage pool as soon as possible after receiving the access request. The greater the access network delay, the longer the preset time, and the migration can be performed within the preset time after receiving the access request, when the device is relatively idle. In the method provided in the embodiment of the present application, the preset time is determined according to the access network delay of the data block, and the data corresponding to the data block is written to the second storage pool within the preset time. It can be determined based on the access network delay of the data block to migrate the data corresponding to the data block in real time, or migrate the data corresponding to the data block when the device is relatively idle, so that the cloud storage space management is more flexible.

[0082] In a possible implementation, the method further includes: receiving second indication information, where the second indication information is used to indicate a network delay for accessing the data block.

[0083] Among them, the second indication information can come from the storage pool. Exemplarily, when receiving an access request, the computing node will also receive the communication time from the terminal signal to the computing node, that is, the access network delay. Each time the computing node accesses the data corresponding to the data block in the storage pool, the access network delay will be sent to the storage pool for storage. The storage pool calculates the heat of the data block based on the access network delay, which can be used as a reference heat when the data block is accessed later. When the computing node needs to use the access network delay, the storage pool sends the second indication information to the computing node. In the method provided in an embodiment of the present application, the access network delay of the data block is stored in other devices. When the access network delay is needed, receiving the second indication information for indicating the access network delay of the data block can save storage space and provide a basis for determining the preset time.

[0084] In a possible implementation, the data block is one of multiple data blocks of the target file, and data corresponding to at least two of the multiple data blocks are stored in different storage pools. Data blocks belonging to the same file can be determined based on file identifiers carried by the data blocks.

[0085] For example, Figure 6 As shown, the cloud platform includes computing nodes 510, storage nodes 520, storage nodes 530 and other storage nodes, storage node 520 includes storage pool 521, storage pool 522 and other storage pools, and storage node 530 includes storage pool 531, storage pool 532 and other storage pools. The computing node 510 in the cloud platform can communicate with terminal 551, terminal 552, terminal 553 and other terminals through cloud interface 540. The computing node 510 can communicate with storage node 520 and storage node 530. File 511 includes data blocks 1, 2, 3, ..., m and other data blocks. The computing node can migrate the data corresponding to data block 1 to storage pool 522, migrate the data corresponding to data block 2 to storage pool 531, migrate the data corresponding to data block 3 and the data corresponding to data block m to storage pool 521, and migrate the data corresponding to other data blocks to the corresponding storage pools according to the heat of each data block in file 511. In this way, data corresponding to at least two data blocks among the multiple data blocks of the file 511 are respectively stored in different storage pools.

[0086] If the migration is performed in units of files, the data corresponding to the data blocks of the same file should be stored in the same storage pool. In the method provided in the embodiment of the present application, since data migration is performed in units of data blocks, the data corresponding to different data blocks of the same file can be migrated to different storage pools, without being restricted to storing the data corresponding to the data blocks of the same file in the same storage pool. In this way, data can be accurately migrated, the overall efficiency of accessing a large amount of data in the storage pool and the bandwidth utilization efficiency of the storage pool can be improved, and the storage cost can be reduced.

[0087] The above mainly introduces the method provided by the embodiment of the present application. The embodiment of the present application also provides a system, which includes a hardware structure and / or software module corresponding to each function in order to realize the above functions. It should be easily appreciated by those skilled in the art that, in combination with the structures and method steps of each example described in the embodiments disclosed herein, the system provided by the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0088] Next, we will take the data processing system as an example to explain the system. Figure 7 , the system 600 includes a computing node 610, and a first storage pool 620 and a second storage pool 630 coupled to the computing node 610, the computing node 610 is configured to: receive an access request for a data block, and the data corresponding to the data block is stored in the first storage pool 620. In response to the access request, the second storage pool 630 is determined according to the data heat of the data block and the bandwidth required by the data block. The data corresponding to the data block is written into the second storage pool 630. Exemplarily, the system 600 may include multiple computing nodes 610 and multiple storage pools, and each computing node 610 is coupled to multiple storage pools.

[0089] In one possible implementation, the computing node 610 is specifically configured to determine the second storage pool 630 when the temperature difference between the data temperature of the data block and a reference temperature is within a temperature difference threshold range between the temperature of the data block supported for storage by the second storage pool 630 and the reference temperature, and when the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool 620 is within a bandwidth difference threshold range between the second storage pool 630 and the first storage pool 620.

[0090] In a possible implementation, the performance of the second storage pool 630 is different from the performance of the first storage pool 620 , and the performance includes access speed and / or access delay.

[0091] In one possible implementation, the data heat of the data block is determined based on the number of times the data block is accessed and the access request delay of the data block, where the access request delay of the data block is the time difference between the time of accessing the storage area storing the data corresponding to the data block in the first storage pool 620 and the time of receiving the response from the first storage pool 620.

[0092] In a possible implementation, the reference temperature is the temperature of the data blocks supported for storage by the first storage pool 620 .

[0093] In a possible implementation, the reference temperature is the historical access temperature of the data block.

[0094] In a possible implementation, the computing node 610 is further configured to: receive first indication information, where the first indication information is used to indicate the remaining bandwidth and reference heat of the first storage pool 620 .

[0095] In one possible implementation, the computing node 610 is specifically configured to write the data corresponding to the data block to the second storage pool 630 within a preset time, where the preset time is determined based on the access network delay of the data block, and the access network delay of the data block is the time difference between the sending time of the access request and the receiving time of the access request.

[0096] In a possible implementation, the computing node 610 is further configured to: receive second indication information, where the second indication information is used to indicate a network delay for accessing the data block.

[0097] In a possible implementation, the data block is one of multiple data blocks of the target file, and data corresponding to at least two data blocks of the multiple data blocks are stored in different storage pools.

[0098] The above-mentioned system 600 provided in the embodiment of the present application is used to implement the functions of the corresponding steps of the above-mentioned method. Since each step has been described in detail in the above-mentioned method embodiment, it will not be repeated here.

[0099] The embodiment of the present application further provides a server, wherein both the first storage pool and the second storage pool are coupled to the server, and the server is configured to: receive an access request for a data block, and the data corresponding to the data block is stored in the first storage pool. In response to the access request, determine the second storage pool according to the data popularity of the data block and the bandwidth required by the data block. Write the data corresponding to the data block into the second storage pool.

[0100] In one possible implementation, the server is specifically configured to determine the second storage pool when the temperature difference between the data temperature of the data block and a reference temperature is within a temperature difference threshold range between the temperature of the data block supported for storage by the second storage pool and the reference temperature, and the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within a bandwidth difference threshold range between the second storage pool and the first storage pool.

[0101] In a possible implementation, the performance of the second storage pool is different from the performance of the first storage pool, and the performance includes access speed and / or access delay.

[0102] In one possible implementation, the data heat of the data block is determined based on the number of times the data block is accessed and the access request delay of the data block, where the access request delay of the data block is the time difference between the time of accessing the storage area storing the data corresponding to the data block in the first storage pool and the time of receiving the response from the first storage pool.

[0103] In a possible implementation, the reference temperature is the temperature of the data block supported for storage by the first storage pool.

[0104] In a possible implementation, the reference temperature is the historical access temperature of the data block.

[0105] In a possible implementation manner, the server is further configured to: receive first indication information, where the first indication information is used to indicate a remaining bandwidth and a reference heat of the first storage pool.

[0106] In one possible implementation, the server is specifically configured to write the data corresponding to the data block into the second storage pool within a preset time, where the preset time is determined based on a network access delay of the data block, where the network access delay of the data block is the time difference between a sending time of an access request and a receiving time of the access request.

[0107] In a possible implementation manner, the server is further configured to: receive second indication information, where the second indication information is used to indicate a network access delay of the data block.

[0108] In a possible implementation, the data block is one of multiple data blocks of the target file, and data corresponding to at least two data blocks of the multiple data blocks are stored in different storage pools.

[0109] The above-mentioned server provided in the embodiment of the present application is used to implement the functions of the corresponding steps of the aforementioned method. Since each step has been described in detail in the aforementioned method embodiment, it will not be repeated here.

[0110] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores program code, and when it is run on a device (for example, the device can be a single-chip microcomputer, a chip, a computer or a processor, etc.), the program code therein can be called by the processor to execute one or more steps in the above method embodiment.

[0111] Based on this understanding, the embodiments of the present application also provide a computer program product comprising instructions. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor therein to execute all or part of the steps of the methods described in the embodiments of the present application.

[0112] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: include: Receiving an access request for a data block, data corresponding to the data block is stored in a first storage pool; In response to the access request, determining a second storage pool according to data popularity of the data block and bandwidth required by the data block; The data corresponding to the data block is written into the second storage pool.

2. The method according to claim 1, characterized in that The determining the second storage pool according to the data heat of the data block and the bandwidth required by the data block includes: When the temperature difference between the data temperature of the data block and the reference temperature is within the temperature difference threshold range between the temperature of the data block supported by the second storage pool for storage and the reference temperature, and the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the bandwidth difference threshold range between the second storage pool and the first storage pool, the second storage pool is determined.

3. The method according to claim 2, characterized in that The performance of the second storage pool is different from the performance of the first storage pool, and the performance includes access speed and / or access delay.

4. The method according to claim 2, characterized in that: The data heat of the data block is determined based on the number of times the data block is accessed and the access request delay of the data block. The access request delay of the data block is: the time difference between the time of accessing the storage area storing the data corresponding to the data block in the first storage pool and the time of receiving the response from the first storage pool.

5. The method according to claim 2, characterized in that: The reference temperature is the temperature of the data blocks supported for storage by the first storage pool.

6. The method according to claim 2, characterized in that The reference temperature is the historical access temperature of the data block.

7. The method according to claim 2, characterized in that The method further comprises: First indication information is received, where the first indication information is used to indicate the remaining bandwidth of the first storage pool and the reference heat.

8. The method according to claim 1, characterized in that Writing the data corresponding to the data block into the second storage pool includes: The data corresponding to the data block is written into the second storage pool within a preset time, wherein the preset time is determined based on a network delay for accessing the data block, and the network delay for accessing the data block is the time difference between a sending time of the access request and a receiving time of the access request.

9. The method according to claim 8, characterized in that The method further comprises: Second indication information is received, where the second indication information is used to indicate a network delay for accessing the data block.

10. The method according to claim 1, characterized in that The data block is one of multiple data blocks of the target file, and data corresponding to at least two of the multiple data blocks are stored in different storage pools.

11. A data processing system, characterized in that: The system includes a computing node, and a first storage pool and a second storage pool coupled to the computing node, wherein the computing node is configured to: Receiving an access request for a data block, data corresponding to the data block is stored in a first storage pool; In response to the access request, determining a second storage pool according to data popularity of the data block and bandwidth required by the data block; The data corresponding to the data block is written into the second storage pool.

12. The system according to claim 11, characterized in that The computing node is specifically configured as follows: When the temperature difference between the data temperature of the data block and the reference temperature is within the temperature difference threshold range between the temperature of the data block supported by the second storage pool for storage and the reference temperature, and the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the bandwidth difference threshold range between the second storage pool and the first storage pool, the second storage pool is determined.

13. The system according to claim 12, characterized in that The performance of the second storage pool is different from the performance of the first storage pool, and the performance includes access speed and / or access delay.

14. The system according to claim 12, characterized in that The data heat of the data block is determined based on the number of times the data block is accessed and the access request delay of the data block. The access request delay of the data block is: the time difference between the time of accessing the storage area storing the data corresponding to the data block in the first storage pool and the time of receiving the response from the first storage pool.

15. The system according to claim 12, characterized in that The reference temperature is the temperature of the data blocks supported for storage by the first storage pool.

16. The system according to claim 12, characterized in that The reference temperature is the historical access temperature of the data block.

17. The system according to claim 12, characterized in that The computing node is further configured to: First indication information is received, where the first indication information is used to indicate the remaining bandwidth of the first storage pool and the reference heat.

18. The system according to claim 11, characterized in that The computing node is specifically configured as follows: The data corresponding to the data block is written into the second storage pool within a preset time, wherein the preset time is determined based on a network delay for accessing the data block, and the network delay for accessing the data block is the time difference between a sending time of the access request and a receiving time of the access request.

19. The system according to claim 18, characterized in that The computing node is further configured to: Second indication information is received, where the second indication information is used to indicate a network delay for accessing the data block.

20. The system according to claim 11, characterized in that The data block is one of multiple data blocks of the target file, and data corresponding to at least two of the multiple data blocks are stored in different storage pools.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method described in any one of claims 1 to 10.

Images