A data processing method and system

Abstract

Embodiments of the present application relate to a data processing method and system, and relate to the technical field of storage. The method comprises: receiving an access request for a data block, the data corresponding to the data block being stored in a first storage pool; in response to the access request, determining a second storage pool according to a data heat of the data block and a bandwidth required by the data block; and writing the data corresponding to the data block into the second storage pool. In this way, the overall access efficiency of the storage pool in the cloud computing platform can be improved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data processing method and system. Background Technology

[0002] As technology advances, cloud computing platforms will phase out poor-performing or outdated storage pools and add higher-performing or new storage pools, thereby continuously upgrading, replacing, and building cloud storage hardware and software arrays, and continuously backing up and migrating data. Summary of the Invention

[0003] A first aspect of this application provides a data processing method, the method comprising: receiving an access request for a data block, the data corresponding to the data block being stored in a first storage pool; responding to the access request, determining a second storage pool based on the data popularity of the data block and the bandwidth required by the data block; and writing the data corresponding to the data block into the second storage pool.

[0004] In the above technical solution, an access request for a data block is received. Based on the data block's popularity and required bandwidth, a second storage pool is determined, and the data corresponding to that data block is written from the first storage pool to the second storage pool. Data migration is performed on a block-by-block basis. Based on the data block's popularity and required bandwidth, the data corresponding to that data block is migrated to a storage pool with corresponding performance. This avoids migrating unevenly popular data from multiple data points within the same file to the same storage pool, enabling precise data migration. Thus, cloud storage data migration strategies are redefined, improving cloud storage space management capabilities. For example, it improves the overall efficiency when accessing large amounts of data in a storage pool, as well as the bandwidth utilization efficiency of the storage pool, and reduces storage costs.

[0005] In one possible implementation of the first aspect, determining the second storage pool based on the data popularity of the data block and the bandwidth required by the data block includes: determining the second storage pool when the difference between the data popularity of the data block and the reference popularity is within a threshold range of the difference between the popularity of data blocks supported by the second storage pool and the reference popularity, and when the difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within a threshold range of the bandwidth difference between the second storage pool and the first storage pool. In the above possible implementation, based on the difference between the data popularity of the data block and the reference popularity, the popularity change of the data block targeted by the access request can be obtained. If the popularity difference is within a threshold range of the difference between the popularity of data blocks supported by the second storage pool and the reference popularity, then the data block targeted by the access request meets the temperature conditions for migration to the second storage pool. Based on the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool, the bandwidth of the data block targeted by the access request can be obtained. If the bandwidth difference is within a threshold range of the bandwidth difference between the second storage pool and the first storage pool, then the data block targeted by the access request meets the bandwidth conditions for migration to the second storage pool. When the data block targeted by the access request meets both the heat conditions and the bandwidth conditions for migration to the second storage pool, the data corresponding to that data block can be migrated to the second storage pool. This allows for precise data migration, improves the overall efficiency when accessing large amounts of data in the storage pool, enhances the bandwidth utilization efficiency of the storage pool, and reduces storage costs.

[0006] In one possible implementation of the first aspect, the performance of the second storage pool differs from that of the first storage pool, including access speed and / or access latency. In the aforementioned possible implementation, data corresponding to data blocks with varying popularity and bandwidth are migrated to storage pools with corresponding performance levels, where the storage pool's performance includes access speed and / or access latency. This improves the overall efficiency of accessing large amounts of data in the storage pool, as well as the bandwidth utilization efficiency of the storage pool, thereby reducing storage costs.

[0007] In one possible implementation of the first aspect, the data popularity of the data block is determined based on the number of times the data block is accessed and the access request latency of the data block. The access request latency of the data block is the time difference between accessing the storage area in the first storage pool that stores the data corresponding to the data block and receiving the response from the first storage pool. In the above possible implementation, determining the data popularity based on the number of times the data block is accessed and the access request latency of the data block, where the access request latency is the time difference between accessing the storage area in the first storage pool that stores the data corresponding to the data block and receiving the response from the first storage pool, and simultaneously considering both the number of data accesses and the access request latency to formulate a data block migration strategy, can improve the overall data access rate and reduce the overall data access time.

[0008] In one possible implementation of the first aspect, the reference heat is the heat of the data blocks supported by the first storage pool. In the above possible implementations, determining the reference heat based on the heat of the data blocks supported by the first storage pool allows for obtaining information on the heat changes of the data blocks targeted by access requests, providing a basis for accurate data migration.

[0009] In one possible implementation of the first aspect, the reference popularity is the historical access popularity of the data block. In the above possible implementations, determining the reference popularity based on the historical access popularity of the data block allows for the acquisition of the data block's popularity changes, providing a basis for accurate data migration.

[0010] In one possible implementation of the first aspect, the method further includes: receiving first indication information, the first indication information being used to indicate the remaining bandwidth and reference heat of the first storage pool. In the above possible implementations, receiving the first indication information indicating the remaining bandwidth and reference heat of the first storage pool provides a basis for analyzing heat and bandwidth differences.

[0011] In one possible implementation of the first aspect, writing the data corresponding to the data block to the second storage pool includes: writing the data corresponding to the data block to the second storage pool within a preset time period. The preset time period is determined based on the network access latency of the data block, which is the time difference between the sending time of the access request and the receiving time of the access request. In the above possible implementation, the preset time is determined based on the network access latency of the data block, and the data corresponding to the data block is written to the second storage pool within the preset time period. This allows for real-time migration of the data corresponding to the data block, or migration of the data corresponding to the data block when the device is relatively idle, based on the network access latency of the data block, making cloud storage space management more flexible.

[0012] In one possible implementation of the first aspect, the method further includes: receiving second indication information, the second indication information being used to indicate the network access latency of the data block. In the above possible implementation, the network access latency of the data block is stored in another device. Receiving the second indication information indicating the network access latency of the data block when the network access latency is needed can save storage space and provide a basis for determining the preset time.

[0013] In one possible implementation of the first aspect, the data block is one of multiple data blocks in the target file, and at least two of the multiple data blocks have their corresponding data stored in different storage pools. In the above possible implementation, because data migration is performed on a block-by-block basis, data corresponding to different data blocks of the same file can be migrated to different storage pools, without being restricted from storing data blocks of the same file in the same storage pool. This allows for precise data migration, improving the overall efficiency when accessing large amounts of data in the storage pool, as well as the bandwidth utilization efficiency of the storage pool, and reducing storage costs.

[0014] A second aspect of this application provides a data processing system, which includes a computing node and a first storage pool and a second storage pool coupled to the computing node. The computing node is configured to: receive an access request for a data block, the data corresponding to the data block being stored in the first storage pool; in response to the access request, determine the second storage pool based on the data popularity of the data block and the bandwidth required by the data block; and write the data corresponding to the data block into the second storage pool.

[0015] In one possible implementation of the second aspect, the computing node is specifically configured to: determine the second storage pool when the difference between the data heat of the data block and the reference heat is within the heat difference threshold range between the heat of the data block supported by the second storage pool and the reference heat, and the difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the bandwidth difference threshold range between the second storage pool and the first storage pool.

[0016] In one possible implementation of the second aspect, the performance of the second storage pool differs from that of the first storage pool, including access speed and / or access latency.

[0017] In one possible implementation of the second aspect, the data popularity of the data block is determined based on the number of times the data block is accessed and the access request latency of the data block. The access request latency of the data block is the time difference between the time to access the storage area in the first storage pool that stores the data corresponding to the data block and the time to receive the response from the first storage pool.

[0018] In one possible implementation of the second aspect, the reference heat is the heat of the data blocks supported by the first storage pool.

[0019] In one possible implementation of the second aspect, the reference popularity is the historical access popularity of the data block.

[0020] In one possible implementation of the second aspect, the computing node is further configured to receive first indication information, which indicates the remaining bandwidth and reference heat of the first storage pool.

[0021] In one possible implementation of the second aspect, the computing node is specifically configured to write the data corresponding to the data block into a second storage pool within a preset time. The preset time is determined based on the access network latency of the data block, which is the time difference between the sending time of the access request and the receiving time of the access request.

[0022] In one possible implementation of the second aspect, the computing node is further configured to receive second indication information, which is used to indicate the network latency for accessing the data block.

[0023] In one possible implementation of the second aspect, the data block is one of a plurality of data blocks in the target file, and at least two of the data blocks are stored in different storage pools.

[0024] A third aspect of the embodiments of this application provides a computer-readable storage medium storing program code that can be invoked by a processor to execute the method provided by the first aspect or any possible implementation thereof.

[0025] Another aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform the method provided by the first aspect or any possible implementation thereof.

[0026] Understandably, any of the data processing systems, computer storage media, or computer program products provided above are used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0027] Figure 1 An architecture of a cloud platform provided in this application embodiment Figure 1 ;

[0028] Figure 2 An architecture of a cloud platform provided in this application embodiment Figure 2 ;

[0029] Figure 3 This application provides a schematic diagram of the structure of a cloud platform according to an embodiment of the present application.

[0030] Figure 4 An illustration of a data processing method provided in an embodiment of this application. Figure 1 ;

[0031] Figure 5 An illustration of a data processing method provided in an embodiment of this application. Figure 2 ;

[0032] Figure 6 An illustration of a data processing method provided in an embodiment of this application. Figure 3 ;

[0033] Figure 7 This is a schematic diagram of a data processing system provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions in some embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application are within the scope of protection of this application.

[0035] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0036] In describing some embodiments, the term "coupled" may be used to indicate that two or more components have direct physical or electrical contact. However, the term "coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the content of this document.

[0037] "At least one of A, B, and C" has the same meaning as "at least one of A, B, or C," both including the following combinations of A, B, and C: A only, B only, C only, a combination of A and B, a combination of A and C, a combination of B and C, and a combination of A, B, and C. "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B. The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps.

[0038] Before introducing the embodiments of this application, the application scenarios of these embodiments will first be described. The technical solutions provided by the embodiments of this application can be applied to cloud computing platforms. A cloud computing platform, also known as a cloud platform, refers to a service that provides computing, networking, and storage capabilities based on hardware and software resources. Computing nodes and storage nodes can be nodes in a cloud platform. Computing nodes are used to perform data computation functions, and the hardware resources of computing nodes can include computing devices, such as servers, desktop computers, or processors. Storage nodes are used to perform data storage functions, and the hardware resources of storage nodes can include storage pools. For example, the hardware resources of computing nodes include communication interfaces and processors, and the hardware resources of storage nodes include communication interfaces and storage pools. There can be multiple computing nodes and storage nodes in a cloud platform, and the computing nodes and storage nodes can communicate with each other through the communication interfaces.

[0039] In one implementation, the cloud platform adopts a compute-in-store architecture, meaning that compute nodes and storage nodes are located in the same device. For example, Figure 1 As shown, device 100 includes multiple storage nodes (storage nodes 111, 112, 113, ...) and compute node 120. Each of the multiple storage nodes is coupled to compute node 120, and compute node 120 can store data to any one of the multiple storage nodes.

[0040] In another implementation, the cloud platform employs a compute-storage separation architecture, where compute nodes and storage nodes are located on different devices. For example, Figure 2 As shown, device 210 includes storage node 211, device 220 includes storage node 221, and device 230 includes compute node 231. The cloud platform may also include more devices and their storage nodes. Device 240 includes compute node 241. Device 240 can communicate with devices 210, 220, 230, ... through a communication network, so that compute node 241 can store data to any one of the storage nodes 211, 221, 231, ...

[0041] As technology advances, cloud platforms will phase out underperforming or outdated storage pools and add higher-performance or newer storage pools, continuously upgrading, replacing, and building cloud storage hardware and software arrays, and continuously backing up and migrating data. Data migration can be performed on a file-by-file basis. Files often consist of multiple data blocks, each of which can be accessed independently. Based on the access patterns of each data block, its "popularity" can be determined. High-popularity data blocks are typically a small subset of data blocks within a file. If some data blocks within a file are highly popular, the file's overall popularity increases. For example, a file's popularity can be represented by the number of accesses per unit of time, where the total number of accesses is the sum of the access counts for all data blocks within the file. According to the migration strategy, files with increased popularity are migrated to higher-performance storage pools.

[0042] For example, such as Figure 3 As shown, the cloud platform includes compute nodes 310, storage nodes 320, storage nodes 330, and other storage nodes. Storage node 320 includes storage pools 321, 322, and others, while storage node 330 includes storage pools 331, 332, and others. Compute node 310 in the cloud platform can communicate with terminals 351, 352, 353, and others via cloud interface 340. File 311 is stored in storage pool 332 of storage node 330. File 311 includes multiple data blocks. Due to the high frequency of file 311, compute node 310 writes file 311 from storage pool 332 of storage node 330 to storage pool 321 of storage node 320. File 311 includes not only high-frequency data blocks but also other data blocks, such as low-frequency data blocks. This results in low data migration efficiency and flexibility for the cloud platform.

[0043] Please refer to Figure 4 This application provides a data processing method, including:

[0044] S110: Receive an access request for a data block whose data is stored in the first storage pool.

[0045] S120: In response to the access request, determine the second storage pool based on the data popularity of the data block and the bandwidth required by the data block.

[0046] S130: Write the data corresponding to the data block to the second storage pool.

[0047] The access request can be a read request. Data popularity refers to the current popularity of the data block targeted by the access request. The bandwidth required for this data block is related to its popularity; the higher the popularity of the data block and the more data corresponding to that data block the storage pool needs to transfer at the same time, the greater the bandwidth required for that data block.

[0048] For example, this method is applied to a cloud platform, whose storage system includes internet applications, a public cloud storage interface, a cloud storage file system, a data block temperature server, and a data block temperature client. The internet applications can run on terminals, the cloud storage file system and the data block temperature server can run on compute nodes, and the data block temperature client can run on storage nodes. The internet applications can communicate with the cloud storage file system through the public cloud storage interface.

[0049] For example, the terminal sends an access request for the data block to the computing node. Before this, the computing node allocates a storage pool for the data corresponding to the data block and stores the data corresponding to the data block in the appropriate storage pool, such as the first storage pool.

[0050] The compute node receives an access request for the data block and sends a read command to the first storage pool based on the access request. The first storage pool receives the read command, reads the corresponding data, and sends the data corresponding to the data block to the compute node.

[0051] The compute node receives the data corresponding to the data block and determines the second storage pool based on the data popularity of the data block and the bandwidth required by the data block.

[0052] The compute node sends a write command to the second storage pool containing the data corresponding to the data block. The second storage pool receives the data corresponding to the data block and completes the write operation.

[0053] In some embodiments, after a compute node sends data corresponding to a data block to a second storage pool, the first storage pool can delete or mark the data corresponding to the data block as invalid.

[0054] For example, the first storage pool and the second storage pool can be located on the same storage node or on different storage nodes. That is, the data corresponding to the data block targeted by the access request can be migrated within the same storage node or between different storage nodes. For multiple data in a storage pool, they can be migrated on a data block basis within the same storage node or between different storage nodes.

[0055] For example, the data block to be migrated can be a gradually warming (increasing popularity) data block or a gradually cooling (decreasing popularity) data block. For instance, in response to an access request, if the data block is determined to be a gradually warming data block based on its popularity and the bandwidth required, the data corresponding to that data block is migrated to a second storage pool, where the performance of the second storage pool is higher than that of the first storage pool. As another example, in response to an access request, if the data block is determined to be a gradually cooling data block based on its popularity and the bandwidth required, the data corresponding to that data block is migrated to a second storage pool, where the performance of the second storage pool is lower than that of the first storage pool.

[0056] For example, data blocks with constant heat (little temperature change) may not need to be migrated. For instance, in response to an access request, if the data block is determined to be a constant heat data block based on its data heat and the bandwidth required by the data block, then the data corresponding to that data block will not be migrated, and the data corresponding to that data block will be stored in the first storage pool.

[0057] When storing massive amounts of data in a storage pool, the value stability of most data blocks decreases over time, necessitating migration strategies based on certain rules, such as data block popularity. This application identifies gradually warming, cooling, and constant-hot data blocks in the storage pool, determines migration strategies based on data block popularity, and migrates the data corresponding to each data block to its corresponding storage pool. For example, based on a Quality of Service (QoS) policy, data corresponding to gradually warming data blocks is migrated to a storage pool with higher performance, higher bandwidth, and higher cost; data corresponding to cooling data blocks is migrated to a storage pool with lower performance, lower bandwidth, and lower cost; and data corresponding to constant-hot data blocks is not migrated. This aligns with the current needs of cloud computing development. New, higher-performance storage pools and compute nodes are added daily, but the popularity of data blocks changes every second. Compute nodes continuously migrate the data corresponding to data blocks to their corresponding storage pools based on data popularity, completing the migration of data corresponding to data blocks during the data block hot / cold stratification process, enabling precise data migration. Furthermore, the cloud storage file system running on the compute nodes is compatible with storage pools of various performance levels (such as hard drives), improving the utilization rate of existing storage pools.

[0058] In the method provided in this application embodiment, an access request for a data block is received. Based on the data popularity and bandwidth required for the data block, a second storage pool is determined, and the data corresponding to the data block is migrated from the first storage pool to the second storage pool. Data migration is performed on a block-by-block basis. By migrating data corresponding to a data block to a storage pool with corresponding performance based on its popularity and required bandwidth, it avoids migrating unevenly popular data from multiple data sets within the same file to the same storage pool, enabling precise data migration. This redefines cloud storage data migration strategies, improves cloud storage space management capabilities, for example, increasing the overall efficiency when accessing large amounts of data in a storage pool, improving the bandwidth utilization efficiency of the storage pool, and reducing storage costs.

[0059] In one possible implementation, S120 (determining a second storage pool based on the data popularity of the data block and the bandwidth required by the data block) may include: determining a second storage pool when the popularity difference between the data popularity of the data block and the reference popularity is within the popularity difference threshold range between the popularity of the data block supported by the second storage pool and the reference popularity, and the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the bandwidth difference threshold range between the second storage pool and the first storage pool.

[0060] The difference between the data popularity of this data block and the reference popularity can be defined as the value obtained by subtracting the reference popularity from the data popularity of this data block.

[0061] In one example, reference popularity can be used to indicate the popularity of a data block targeted by an access request. For instance, the reference popularity could be the historical access popularity of that data block. The popularity of a data block changes each time the data corresponding to it is accessed, and the historical access popularity includes the popularity after each change. The reference popularity can be any historical access popularity of that data block; for example, the reference popularity could be the popularity that changed after the last time the data corresponding to the data block was accessed. Thus, by determining the reference popularity based on the historical access popularity of a data block, the popularity changes of that data block can be obtained, providing a basis for accurate data migration.

[0062] In another example, reference heat can also be used to indicate the heat of other data blocks besides the data block targeted by the access request. For example, the heat of all data blocks stored in multiple storage pools of a storage node is divided into a preset number of levels, the performance of multiple storage pools of the storage node is divided into a preset number of levels, and the performance of each storage pool is mapped to the heat of data blocks in each level. The preset number may or may not be equal to the number of storage pools; this embodiment does not impose any limitation on this. The reference heat is the heat of data blocks supported by the first storage pool, that is, the heat of data blocks in the level corresponding to the performance of the first storage pool. Since the heat of data blocks in a storage node changes, the steps of dividing the heat of all data blocks in the storage node into levels and mapping the performance of each storage pool to the heat of data blocks in each level can be performed once every first cycle. In this example, the heat of data blocks supported by the first storage pool can be a first heat range. If the heat of a data block is greater than the maximum value of the first heat range, the heat difference can be the heat of the data block minus the maximum value of the first heat range. If the data popularity of a data block is less than the minimum value of the first popularity range, the popularity difference can be the data popularity of the data block minus the minimum value of the first popularity range. Thus, by determining the reference popularity based on the popularity of the data blocks supported by the first storage pool, the popularity changes of the data blocks targeted by access requests can be obtained, providing a basis for accurate data migration.

[0063] Furthermore, the popularity of the data blocks supported by the second storage pool can be a second popularity range, and the popularity difference threshold range between the popularity of the data blocks supported by the second storage pool and the reference popularity can refer to the range obtained by subtracting the reference popularity from the second popularity range.

[0064] In one example, the reference popularity is the historical access popularity of the data block. Based on the performance of the first storage pool, the first storage pool is configured to support storing data blocks within a first popularity range; based on the performance of the second storage pool, the second storage pool is configured to support storing data blocks within a second popularity range. If the historical access popularity of the data block is less than the minimum value of the second popularity range, the minimum value of the popularity difference threshold range can be: the minimum value of the second popularity range minus the historical access popularity of the data block; the maximum value of the popularity difference threshold range can be: the maximum value of the second popularity range minus the historical access popularity of the data block. If the historical access popularity of the data block is greater than the maximum value of the second popularity range, the minimum value of the popularity difference threshold range can be: the minimum value of the second popularity range minus the historical access popularity of the data block; the maximum value of the popularity difference threshold range can be: the maximum value of the second popularity range minus the historical access popularity of the data block.

[0065] In another example, the reference heat is the heat of data blocks that the first storage pool supports storing. Based on the performance of the first storage pool, it is configured to support storing data blocks within a first heat range; based on the performance of the second storage pool, it is configured to support storing data blocks within a second heat range. If the maximum value of the first heat range is less than the minimum value of the second heat range, then the minimum value of the heat difference threshold range can be: the minimum value of the second heat range minus the maximum value of the first heat range; the maximum value of the heat difference threshold range can be: the maximum value of the second heat range minus the maximum value of the first heat range. If the minimum value of the first heat range is greater than the maximum value of the second heat range, then the minimum value of the heat difference threshold range can be: the minimum value of the second heat range minus the minimum value of the first heat range; the maximum value of the heat difference threshold range can be: the maximum value of the second heat range minus the minimum value of the first heat range.

[0066] Next, combine Figure 5 Taking the historical access popularity of the data block as a reference popularity as an example, this paper illustrates the situation where the popularity difference is within the popularity difference threshold range. The specific popularity values ​​below are examples; in practice, the popularity values ​​can be other values, and this application embodiment does not impose any restrictions on them. Compute node 410 can communicate with storage pools 421, 422, and 423. The performance of storage pools 421, 422, and 423 gradually increases. The first storage pool is storage pool 422; the data corresponding to data block 411 is stored in storage pool 422, and compute node 410 reads the data corresponding to data block 411 from storage pool 422.

[0067] In the first scenario, the data block is a gradually warming data block. The data block's data popularity is 70 degrees, its historical access popularity is 50 degrees, the first popularity range is 31 to 60 degrees, and the second popularity range is 61 to 90 degrees. The popularity difference can be: the data block's data popularity minus its historical access popularity, i.e., +20 degrees. The minimum value of the popularity difference threshold range can be: the minimum value of the second popularity range minus the data block's historical access popularity, i.e., +11 degrees. The maximum value of the popularity difference threshold range can be: the maximum value of the second popularity range minus the data block's historical access popularity, i.e., +40 degrees. The popularity difference (+20 degrees) between the data block's data popularity and the reference popularity falls within the popularity difference threshold range (+11 to +40 degrees) between the data blocks supported by the second storage pool and the reference popularity, thus confirming the second storage pool as storage pool 423. Compute node 410 writes the data corresponding to data block 411 into storage pool 423.

[0068] In the second scenario, the data block is a gradually cooling data block. The data block's data popularity is 20 degrees, its historical access popularity is 50 degrees, the first popularity range is 31 to 60 degrees, and the second popularity range is 1 to 30 degrees. The popularity difference can be: the data block's data popularity minus its historical access popularity, i.e., -30 degrees. The minimum value of the popularity difference threshold range can be: the minimum value of the second popularity range minus the data block's historical access popularity, i.e., -49 degrees. The maximum value of the popularity difference threshold range can be: the maximum value of the second popularity range minus the data block's historical access popularity, i.e., -20 degrees. The popularity difference (-30 degrees) between the data block's data popularity and the reference popularity falls within the popularity difference threshold range (-49 to -20 degrees) between the data blocks supported by the second storage pool and the reference popularity, thus confirming the second storage pool as storage pool 421. Compute node 410 writes the data corresponding to data block 411 into storage pool 421.

[0069] In the third scenario, the data block is a constant-hot data block. The data block's hotness is 60 degrees, its historical access hotness is 50 degrees, and the first hotness range is 31 to 60 degrees. Since the data block's hotness falls within the first hotness range, the data corresponding to this data block does not need to be migrated.

[0070] Next, combine Figure 5 Taking the reference heat value as the heat value (first heat value range) of the data blocks supported for storage in the first storage pool as an example, this paper illustrates the case where the heat value difference is within the heat value difference threshold range. The specific heat value values ​​below are examples; in practice, the heat value can be other values, and this application embodiment does not impose any restrictions on this. Compute node 410 can communicate with storage pools 421, 422, and 423. The performance of storage pools 421, 422, and 423 gradually increases. The first storage pool is storage pool 422; the data corresponding to data block 411 is stored in storage pool 422, and compute node 410 reads the data corresponding to data block 411 from storage pool 422.

[0071] In the first example, the data block is a gradually warming data block. The data block's heat value is 70 degrees, with a first heat value range of 31 to 60 degrees and a second heat value range of 61 to 90 degrees. The heat value difference can be the data block's heat value minus the maximum value of the first heat value range, i.e., +10 degrees. The minimum value of the heat value difference threshold range can be the minimum value of the second heat value range minus the maximum value of the first heat value range, i.e., +1 degree. The maximum value of the heat value difference threshold range can be the maximum value of the second heat value range minus the maximum value of the first heat value range, i.e., +30 degrees. The heat value difference (+10 degrees) between the data block's heat value and the reference heat value falls within the heat value difference threshold range (+1 to +30 degrees) between the heat value of data blocks supported by the second storage pool and the reference heat value, thus identifying the second storage pool as storage pool 423. Compute node 410 writes the data corresponding to data block 411 into storage pool 423.

[0072] In the second example, the data block is a gradually cooling data block. The data block's heat value is 20 degrees, with a first heat value range of 31 to 60 degrees and a second heat value range of 1 to 30 degrees. The heat value difference can be the data block's heat value minus the minimum value of the first heat value range, i.e., -11 degrees. The minimum value of the heat value difference threshold range can be the minimum value of the second heat value range minus the minimum value of the first heat value range, i.e., -30 degrees. The maximum value of the heat value difference threshold range can be the maximum value of the second heat value range minus the minimum value of the first heat value range, i.e., -1 degree. The heat value difference (-11 degrees) between the data block's heat value and the reference heat value falls within the heat value difference threshold range (-1 to -30 degrees) between the heat value of data blocks supported by the second storage pool and the reference heat value, thus identifying the second storage pool as storage pool 421. Compute node 410 writes the data corresponding to data block 411 into storage pool 421.

[0073] In the third example, the data block is a constant-temperature data block. The data temperature of this data block is 60 degrees, and the first temperature range is 31 to 60 degrees. Since the data temperature of this data block is within the first temperature range, the data corresponding to this data block does not need to be migrated.

[0074] Furthermore, the remaining bandwidth of the first storage pool refers to the bandwidth that the first storage pool can provide. Due to potential aging of the hardware devices in the storage pool during use, the bandwidth provided by the first storage pool may decrease over time. The bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool can be the value obtained by subtracting the remaining bandwidth of the first storage pool from the bandwidth required by the data block. If the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool falls within the bandwidth difference threshold range between the second and first storage pools, it can be that the bandwidth difference is less than the bandwidth difference threshold range.

[0075] In the method provided in this application embodiment, the heat change of the data block targeted by the access request can be obtained based on the heat difference between the data heat of the data block and the reference heat. If the heat difference is within the heat difference threshold range between the heat of the data block supported by the second storage pool and the reference heat, then the data block targeted by the access request meets the temperature condition for migration to the second storage pool. Based on the bandwidth difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool, the bandwidth of the data block targeted by the access request can be obtained. If the bandwidth difference is within the bandwidth difference threshold range between the second storage pool and the first storage pool, then the data block targeted by the access request meets the bandwidth condition for migration to the second storage pool. When the data block targeted by the access request meets both the heat condition and the bandwidth condition for migration to the second storage pool, the data corresponding to the data block is migrated to the second storage pool. This allows for precise data migration, improves the overall efficiency when accessing large amounts of data in the storage pool, enhances the bandwidth utilization efficiency of the storage pool, and reduces storage costs.

[0076] In one possible implementation, the performance of the second storage pool differs from that of the first storage pool, including access speed and / or access latency. In the method provided in this application embodiment, data corresponding to data blocks with varying popularity and bandwidth are migrated to storage pools with corresponding performance levels. The performance of the storage pool includes access speed and / or access latency. This improves the overall efficiency of accessing large amounts of data in the storage pool, as well as the bandwidth utilization efficiency of the storage pool, thereby reducing storage costs.

[0077] In one possible implementation, the data popularity of the data block is determined based on the number of times the data block is accessed and the access request latency of the data block. The access request latency of the data block is the time difference between the time to access the storage area in the first storage pool that stores the data corresponding to the data block and the time to receive the response from the first storage pool.

[0078] The formula for calculating the data popularity of a data block can be: Data Popularity = Number of Accesses to the Data Block * Access Request Latency of the Data Block. For example, the access request is a read request, and the access request latency of the data block is the time difference between the time it takes for the compute node to send a read request to the storage pool and the time it takes for the compute node to receive the data block from the first storage pool. In the method provided in this application embodiment, data popularity is determined based on the number of accesses to the data block and the access request latency of the data block. The access request latency of the data block is the time difference between the time it takes to access the storage area in the first storage pool that stores the data corresponding to the data block and the time it takes to receive the response from the first storage pool. By simultaneously considering the number of data accesses and the access request latency, a data block migration strategy can be formulated, which can improve the overall data access rate and reduce the overall data access time.

[0079] In one possible implementation, the method further includes: receiving first indication information, the first indication information being used to indicate the remaining bandwidth and reference heat of a first storage pool. The first indication information may originate from the storage pool. Exemplarily, the storage pool may send the remaining bandwidth of each storage pool, and the reference heat of all data stored in the storage pool, to the computing node every second cycle. In the method provided in this application embodiment, receiving the first indication information indicating the remaining bandwidth and reference heat of the first storage pool provides a basis for analyzing heat and bandwidth differences.

[0080] In one possible implementation, S130 (writing the data corresponding to the data block to the second storage pool) may include: writing the data corresponding to the data block to the second storage pool within a preset time, wherein the preset time is determined based on the access network latency of the data block, and the access network latency of the data block is the time difference between the sending time of the access request and the receiving time of the access request.

[0081] The higher the network latency, the shorter the preset time, requiring the data to be migrated to the second storage pool as quickly as possible after receiving the access request. Conversely, the higher the network latency, the longer the preset time, allowing migration to occur within a preset timeframe after receiving the access request, when the device is relatively idle. In the method provided in this application embodiment, the preset time is determined based on the network latency of the data block, and the data corresponding to the data block is written to the second storage pool within the preset time. This allows for more flexible cloud storage space management by determining whether to migrate the data corresponding to the data block in real-time or when the device is relatively idle, based on the network latency of the data block.

[0082] In one possible implementation, the method further includes receiving second indication information, the second indication information being used to indicate the network latency for accessing the data block.

[0083] The second indication information can originate from the storage pool. For example, when a compute node receives an access request, it also receives the communication time from the terminal signal to the compute node, i.e., the access network latency. Each time the compute node accesses data corresponding to a data block in the storage pool, it sends the access network latency to the storage pool for storage. The storage pool calculates the "heat" of the data block based on the access network latency, which can be used as a reference heat when the data block is accessed subsequently. When the compute node needs to use the access network latency, the storage pool sends the second indication information to the compute node. In the method provided in this application embodiment, the access network latency of the data block is stored in other devices. Receiving the second indication information indicating the access network latency of the data block when it is needed saves storage space and provides a basis for determining the preset time.

[0084] In one possible implementation, the data block is one of multiple data blocks in the target file, and at least two data blocks are stored in different storage pools. Data blocks belonging to the same file can be identified based on the file identifier carried by the data block.

[0085] For example, such as Figure 6 As shown, the cloud platform includes compute nodes 510, storage nodes 520, storage nodes 530, and other storage nodes. Storage node 520 includes storage pools 521, 522, and other storage pools, and storage node 530 includes storage pools 531, 532, and other storage pools. Compute node 510 in the cloud platform can communicate with terminals 551, 552, 553, and other terminals through cloud interface 540. Compute node 510 can communicate with storage nodes 520 and 530. File 511 includes data blocks 1, 2, 3, ..., m. Compute nodes can migrate data corresponding to data block 1 to storage pool 522, data corresponding to data block 2 to storage pool 531, data corresponding to data block 3 and data corresponding to data block m to storage pool 521, and data corresponding to other data blocks to their respective storage pools, based on the popularity of each data block in file 511. Thus, at least two data blocks in file 511 are stored in different storage pools.

[0086] If migration is performed on a file-by-file basis, data corresponding to data blocks of the same file should be stored in the same storage pool. In the method provided in this application, because data migration is performed on a data block-by-data block basis, data corresponding to different data blocks of the same file can be migrated to different storage pools, without being restricted from storing data blocks of the same file in the same storage pool. This allows for precise data migration, improving the overall efficiency when accessing large amounts of data in the storage pool, as well as the bandwidth utilization efficiency of the storage pool, and reducing storage costs.

[0087] The foregoing mainly describes the methods provided in the embodiments of this application. This application also provides a system that, in order to achieve the above functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the structures and method steps of the various examples described in the embodiments disclosed herein, the system provided in this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0088] The following explanation will use a data processing system as an example. Please refer to... Figure 7 The system 600 includes a compute node 610 and a first storage pool 620 and a second storage pool 630 coupled to the compute node 610. The compute node 610 is configured to: receive an access request for a data block, the data corresponding to which the data block is stored in the first storage pool 620; in response to the access request, determine the second storage pool 630 based on the data popularity of the data block and the bandwidth required by the data block; and write the data corresponding to the data block to the second storage pool 630. Exemplarily, the system 600 may include multiple compute nodes 610 and multiple storage pools, each compute node 610 being coupled to multiple storage pools.

[0089] In one possible implementation, the computing node 610 is specifically configured to: determine the second storage pool 630 when the difference between the data popularity of the data block and the reference popularity is within the threshold range of the difference between the popularity of the data block supported by the second storage pool 630 and the reference popularity, and the difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool 620 is within the threshold range of the difference between the bandwidth of the second storage pool 630 and the first storage pool 620.

[0090] In one possible implementation, the performance of the second storage pool 630 differs from that of the first storage pool 620, including access speed and / or access latency.

[0091] In one possible implementation, the data popularity of the data block is determined based on the number of times the data block is accessed and the access request delay of the data block. The access request delay of the data block is the time difference between the time to access the storage area in the first storage pool 620 that stores the data corresponding to the data block and the time to receive the response from the first storage pool 620.

[0092] In one possible implementation, the reference heat is the heat of the data blocks supported by the first storage pool 620.

[0093] In one possible implementation, the reference popularity is the historical access popularity of the data block.

[0094] In one possible implementation, the computing node 610 is further configured to receive first indication information, which indicates the remaining bandwidth and reference heat of the first storage pool 620.

[0095] In one possible implementation, the computing node 610 is specifically configured to write the data corresponding to the data block into the second storage pool 630 within a preset time. The preset time is determined based on the access network latency of the data block, which is the time difference between the sending time of the access request and the receiving time of the access request.

[0096] In one possible implementation, the computing node 610 is further configured to receive second indication information, which indicates the network latency for accessing the data block.

[0097] In one possible implementation, the data block is one of a plurality of data blocks in the target file, and at least two of the data blocks are stored in different storage pools.

[0098] The system 600 provided in this application embodiment is used to implement the functions of each step of the corresponding aforementioned method. Since each step has been described in detail in the aforementioned method embodiment, it will not be repeated here.

[0099] This application embodiment also provides a server, with both a first storage pool and a second storage pool coupled to the server. The server is configured to: receive access requests for data blocks, the data corresponding to which the data blocks are stored in the first storage pool; in response to the access requests, determine the second storage pool based on the data popularity of the data block and the bandwidth required by the data block; and write the data corresponding to the data block into the second storage pool.

[0100] In one possible implementation, the server is specifically configured to: determine the second storage pool when the difference between the data popularity of the data block and the reference popularity is within the threshold range of the difference between the popularity of the data block supported by the second storage pool and the reference popularity, and the difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the threshold range of the difference between the bandwidth of the second storage pool and the first storage pool.

[0101] In one possible implementation, the performance of the second storage pool differs from that of the first storage pool, including access speed and / or access latency.

[0102] In one possible implementation, the data popularity of the data block is determined based on the number of times the data block is accessed and the access request latency of the data block. The access request latency of the data block is the time difference between the time to access the storage area in the first storage pool that stores the data corresponding to the data block and the time to receive the response from the first storage pool.

[0103] In one possible implementation, the reference heat is the heat of the data blocks supported by the first storage pool.

[0104] In one possible implementation, the reference popularity is the historical access popularity of the data block.

[0105] In one possible implementation, the server is further configured to receive first indication information, which indicates the remaining bandwidth and reference heat of the first storage pool.

[0106] In one possible implementation, the server is specifically configured to write the data corresponding to the data block into a second storage pool within a preset time. The preset time is determined based on the network access latency of the data block, which is the time difference between the sending time of the access request and the receiving time of the access request.

[0107] In one possible implementation, the server is further configured to receive second indication information, which indicates the network latency for accessing the data block.

[0108] In one possible implementation, the data block is one of a plurality of data blocks in the target file, and at least two of the data blocks are stored in different storage pools.

[0109] The server provided in this application embodiment is used to implement the functions of each step of the corresponding method. Since each step has been described in detail in the foregoing method embodiment, it will not be repeated here.

[0110] This application also provides a computer-readable storage medium. The computer-readable storage medium stores program code, which, when run on a device (e.g., a microcontroller, chip, computer, or processor), can be invoked by the processor to execute one or more steps in the above method embodiments.

[0111] Based on this understanding, this application also provides a computer program product containing instructions. The technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or its processor to execute all or part of the steps of the methods described in the various embodiments of this application.

[0112] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, include: Receive an access request for a data block, wherein the data block corresponds to data stored in a first storage pool; In response to the access request, a second storage pool is determined based on the data popularity of the data block and the bandwidth required by the data block; Write the data corresponding to the data block into the second storage pool; The step of determining the second storage pool based on the data popularity and bandwidth required by the data block includes: The second storage pool is determined when the difference between the data popularity of the data block and the reference popularity is within the threshold range of the difference between the popularity of the data block supported by the second storage pool and the reference popularity, and the difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the threshold range of the difference between the bandwidth of the second storage pool and the first storage pool.

2. The method according to claim 1, characterized in that, The performance of the second storage pool differs from that of the first storage pool, and the performance includes access speed and / or access latency.

3. The method according to claim 1, characterized in that, The data popularity of the data block is determined based on the number of times the data block is accessed and the access request latency of the data block. The access request latency of the data block is the time difference between the time to access the storage area in the first storage pool that stores the data corresponding to the data block and the time to receive the response from the first storage pool.

4. The method according to claim 1, characterized in that, The reference heat is the heat of the data blocks that the first storage pool supports storing.

5. The method according to claim 1, characterized in that, The reference popularity is the historical access popularity of the data block.

6. The method according to claim 1, characterized in that, The method further includes: Receive first indication information, which is used to indicate the remaining bandwidth of the first storage pool and the reference heat.

7. The method according to claim 1, characterized in that, The step of writing the data corresponding to the data block into the second storage pool includes: The data corresponding to the data block is written to the second storage pool within a preset time. The preset time is determined based on the access network latency of the data block, which is the time difference between the sending time of the access request and the receiving time of the access request.

8. The method according to claim 7, characterized in that, The method further includes: Receive a second indication message, which is used to indicate the network latency for accessing the data block.

9. The method according to claim 1, characterized in that, The data block is one of a plurality of data blocks in the target file, and at least two of the plurality of data blocks are stored in different storage pools.

10. A data processing system, characterized in that, The system includes compute nodes and a first storage pool and a second storage pool coupled to the compute nodes. The compute nodes are configured as follows: Receive an access request for a data block, wherein the data block corresponds to data stored in a first storage pool; In response to the access request, a second storage pool is determined based on the data popularity of the data block and the bandwidth required by the data block; Write the data corresponding to the data block into the second storage pool; The computing node is specifically configured as follows: The second storage pool is determined when the difference between the data popularity of the data block and the reference popularity is within the threshold range of the difference between the popularity of the data block supported by the second storage pool and the reference popularity, and the difference between the bandwidth required by the data block and the remaining bandwidth of the first storage pool is within the threshold range of the difference between the bandwidth of the second storage pool and the first storage pool.

11. The system according to claim 10, characterized in that, The performance of the second storage pool differs from that of the first storage pool, and the performance includes access speed and / or access latency.

12. The system according to claim 10, characterized in that, The data popularity of the data block is determined based on the number of times the data block is accessed and the access request latency of the data block. The access request latency of the data block is the time difference between the time to access the storage area in the first storage pool that stores the data corresponding to the data block and the time to receive the response from the first storage pool.

13. The system according to claim 10, characterized in that, The reference heat is the heat of the data blocks that the first storage pool supports storing.

14. The system according to claim 10, characterized in that, The reference popularity is the historical access popularity of the data block.

15. The system according to claim 10, characterized in that, The computing node is also configured to: Receive first indication information, which is used to indicate the remaining bandwidth of the first storage pool and the reference heat.

16. The system according to claim 10, characterized in that, The computing node is specifically configured as follows: The data corresponding to the data block is written to the second storage pool within a preset time. The preset time is determined based on the access network latency of the data block, which is the time difference between the sending time of the access request and the receiving time of the access request.

17. The system according to claim 16, characterized in that, The computing node is also configured to: Receive a second indication message, which is used to indicate the network latency for accessing the data block.

18. The system according to claim 10, characterized in that, The data block is one of a plurality of data blocks in the target file, and at least two of the plurality of data blocks are stored in different storage pools.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method according to any one of claims 1 to 9.

Images