Data Processing Method, Apparatus, Device, and Storage Medium Applied to Storage Nodes

Abstract

The present disclosure provides a data processing method, apparatus, device and storage medium applicable to a storage node, which can be applied to the field of big data technology. The storage node includes a local node and a remote node, and each of the local node and the remote node includes a first storage unit and a second storage unit with a slower read / write speed than the first storage unit. The local node is associated with a processor. The method includes: determining target data accessed by the processor according to the current process of the processor; and in the case where it is determined that the target data is stored at a first predetermined position, migrating the target data from the first predetermined position to the first storage unit of the local node; wherein the first predetermined position includes at least one of the following: the second storage unit of the local node, the first storage unit of the remote node, and the second storage unit of the remote node.

Description

Technical Field

[0001] The present disclosure relates to the field of big data, and in particular, to a data processing method, apparatus, device, medium, and program product applied to a storage node. Background Art

[0002] The non-uniform memory access architecture (NUMA) is a multi-processor computer architecture. For a computing device with a NUMA structure, each processor (CPU) can be equipped with multiple layers of heterogeneous memory.

[0003] Currently, in a NUMA-based computing device, the advantages of each memory in the multiple layers of heterogeneous memory cannot be fully utilized, resulting in lower access performance. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a data processing method, apparatus, device, medium, and program product applied to a storage node.

[0005] According to a first aspect of the present disclosure, there is provided a data processing method applied to a storage node. The storage node includes a local node and a remote node. The local node and the remote node each include a first storage unit and a second storage unit with a slower read / write speed than the first storage unit. The local node is associated with a processor. The method includes: determining target data accessed by the processor according to the current process of the processor; and when it is determined that the target data is stored at a first predetermined location, migrating the target data from the first predetermined location to the first storage unit of the local node; where the first predetermined location includes at least one of the following: the second storage unit of the local node, the first storage unit of the remote node, and the second storage unit of the remote node.

[0006] According to another embodiment of the present disclosure, the migrating the target data from the first predetermined location to the first storage unit of the local node includes: in response to detecting that the remaining storage space of the first storage unit of the local node is less than the occupied space of the target data, determining cold data in the first storage unit of the local node according to the access frequency of the data in the first storage unit of the local node; migrating the cold data from the first storage unit of the local node to a second predetermined location; and migrating the target data from the first predetermined location to the first storage unit of the local node.

[0007] According to another embodiment of the present disclosure, the second predetermined location includes N storage areas, where N is an integer greater than or equal to 2; the migrating the cold data from the first storage unit of the local node to the second predetermined location includes: determining a target storage area from the N storage areas according to the priorities of the N storage areas, the occupied space of the cold data, and the remaining storage space of each of the N storage areas; and migrating the cold data from the first storage unit of the local node to the target storage area.

[0008] According to another embodiment of the present disclosure, the determining a target storage area from the N storage areas according to the priorities of the N storage areas, the occupied space of the cold data, and the remaining storage space of each of the N storage areas includes: in the case where it is determined that the remaining storage space of at least one of the N storage areas is greater than or equal to the occupied space of the cold data, repeatedly performing the following operations until a target storage area is obtained: determining the storage area with the highest priority among the N storage areas as the current storage area; determining whether the remaining storage space of the current storage area is greater than or equal to the occupied space of the cold data; if so, determining the current storage area as the target storage area; and if not, deleting the current storage area from the N storage areas.

[0009] According to another embodiment of the present disclosure, the determining a target storage area from the N storage areas according to the priorities of the N storage areas, the occupied space of the cold data, and the remaining storage space of each of the N storage areas includes: in the case where it is determined that the remaining storage space of each of the N storage areas is less than the occupied space of the cold data, performing block processing on the cold data according to the priorities of the N storage areas, the occupied space of the cold data, and the remaining space of each of the N storage areas to obtain at least two data blocks; and determining at least two target storage areas corresponding to the at least two data blocks from the N storage areas.

[0010] According to another embodiment of the present disclosure, the migrating the target data from the first predetermined location to the first storage unit of the local node includes: in response to detecting that the remaining storage space of the first storage unit of the local node is greater than or equal to the occupied space of the target data, migrating the target data from the first predetermined location to the first storage unit of the local node.

[0011] According to another embodiment of the present disclosure, the number of the target data is multiple, and the multiple target data includes migrated data and non-migrated data; the migrating the target data from the first predetermined location to the first storage unit of the local node includes: in response to detecting that the ratio between the data volume of the migrated data and the data volume of the multiple target data is greater than or equal to a ratio threshold, migrating the non-migrated data from the first predetermined location to the first storage unit of the local node.

[0012] According to another embodiment of the present disclosure, migrating the un-migrated data from the first predetermined location to the first storage unit of the local node includes: determining current candidate data from the un-migrated data according to the access order of the plurality of target data; migrating the current candidate data to the first storage unit of the local node; deleting the current candidate data from the un-migrated data; and in response to detecting that the current candidate data is accessed, returning to the operation of determining the current candidate data until the number of the un-migrated data is 0.

[0013] A second aspect of the present disclosure provides a data processing apparatus applied to a storage node. The storage node includes a local node and a remote node. The local node and the remote node each include a first storage unit and a second storage unit with a slower read / write speed than the first storage unit. The local node is associated with a processor. The apparatus includes: a determination module and a migration module. The determination module is configured to determine target data accessed by the processor according to the current process of the processor. The migration module is configured to migrate the target data from the first predetermined location to the first storage unit of the local node when it is determined that the target data is stored in the first predetermined location. The first predetermined location includes at least one of the following: the second storage unit of the local node, the first storage unit of the remote node, and the second storage unit of the remote node.

[0014] A third aspect of the present disclosure provides an electronic device, including: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the above method.

[0015] A fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, and when the instructions are executed by a processor, the processor is caused to execute the above method.

[0016] A fifth aspect of the present disclosure further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0017] According to the data processing method, apparatus, device, medium, and program product provided by the present disclosure, according to the technical solution provided by the embodiments of the present disclosure, since when a process runs on a first processor and the accessed target data is at a first predetermined location, the target data can be migrated from the first predetermined location to the first local storage unit, the system access performance can be improved. Description of the Drawings

[0018] Through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, the above content and other objects, features, and advantages of the present disclosure will become clearer. In the drawings:

[0019] Figure 1 Schematically shows a structural diagram of a storage node according to an embodiment of the present disclosure;

[0020] Figure 2 Schematically shows a flowchart of a data processing method according to an embodiment of the present disclosure;

[0021] Figure 3 Schematically shows a flowchart of a data processing method according to another embodiment of the present disclosure;

[0022] Figure 4 Schematically shows a block diagram of a data processing device according to an embodiment of the present disclosure; and

[0023] Figure 5 Schematically shows a block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. Detailed Description of the Embodiments

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

[0025] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] In the case of using expressions such as "at least one of A, B, and C", generally, it should be interpreted according to the meaning that those skilled in the art usually understand this expression (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0028] In the technical solutions of the present disclosure, the processing of the data involved (such as including but not limited to user personal information) in aspects of collection, storage, use, processing, transmission, provision, disclosure, and application, etc., all comply with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and it does not violate public order and good customs.

[0029] The computer system adopts a NUMA architecture, and the NUMA architecture includes multiple NUMA nodes. Each NUMA node includes its own memory. The process on each NUMA node accessing its own memory is called accessing local memory, and accessing memory across NUMA nodes is called accessing remote memory.

[0030] The NUMA architecture can adopt a memory with a multi-layer heterogeneous architecture. For example, a NUMA node can include a first storage unit and a second storage unit. The performance of the first storage unit is better than that of the second storage unit, and the second storage unit is used to increase the memory capacity of a single node.

[0031] In a multi-layer heterogeneous architecture based on NUMA, the CPU can directly access the first storage unit and the second storage unit, and data can also freely migrate between the first storage unit and the second storage unit.

[0032] In the related art, the memory management method is designed based on the first storage unit, only considering the locality of the program and not considering the performance characteristics of the second storage unit. In practical applications, data with a low access frequency in the first storage unit will not be actively migrated to the second storage unit with relatively poor performance. In addition, data with a high access frequency in the second storage unit will not be actively migrated to the first storage unit with better performance.

[0033] Since the above technical solution does not migrate data, in a computer system with a multi-layer heterogeneous memory architecture, the memory management method cannot fully utilize the advantages of the high performance of the first storage unit and the large capacity of the second storage unit, thereby resulting in a low access performance.

[0034] The embodiments of the present disclosure aim to provide a data processing method. For a multi-layer heterogeneous memory architecture, this method can migrate data to a suitable location according to factors such as the storage location of the data, the space usage of the first storage unit, and the access frequency of the data, so as to reduce system overhead and improve the system access performance.

[0035] Figure 1 Schematically shows a structural diagram of a storage node according to an embodiment of the present disclosure.

[0036] The data processing method provided by the embodiments of the present disclosure can be applied to a storage node. The storage node may include at least two nodes. The nodes may be, for example, the NUMA nodes described above. Each node includes a first storage unit and a second storage unit. Both the first storage unit and the second storage unit may be memories. The performance of the first storage unit in the same node is better than that of the second storage unit. For example, the read / write speed of the first storage unit is faster than that of the second storage unit. The first storage unit may be, for example, DRAM, and the second storage unit may be, for example, persistent memory. At least two nodes are each associated with a corresponding CPU.

[0037] As Figure 1 shown, for the convenience of distinguishing the above at least two nodes, one of the nodes is called the local node 110, and the other nodes are called remote nodes 120. The number of remote nodes 120 is at least one. The processor associated with the local node 110 is called the first processor 130, the processor associated with the remote node 120 is called the second processor 140, the first storage unit in the local node 110 is called the first local storage unit 111, the second storage unit in the local node 110 is called the second local storage unit 112, the first storage unit in each remote node 120 is called the first remote storage unit 121, and the second storage unit in the remote node 120 is called the second remote storage unit 122.

[0038] The CPU can access the data stored in the node associated with itself. For example, the first processor 130 can access the first local storage unit 111 and the second local storage unit 112, and the second processor 140 can access the first remote storage unit 121 and the second remote storage unit 122. In addition, the two CPUs can be connected by a bus, and the two connected CPUs can also access the data stored in the node associated with the other party. For example, the first processor 130 can also access the first remote storage unit 121 and the second remote storage unit 122. The second processor 140 can also access the first local storage unit 111 and the second local storage unit 112.

[0039] The following will be based on Figure 1 the described storage node structure, and will describe in detail the data processing method of the disclosed embodiments through Figures 2 to 3 the following.

[0040] Figure 2 Schematically shows a flowchart of the data processing method according to an embodiment of the present disclosure.

[0041] As Figure 2 shown, the data processing method 200 of this embodiment may include operation S210 to operation S240.

[0042] In operation S210, according to the current process of the processor, determine the target data accessed by the processor.

[0043] For example, the processor in this operation may refer to the first processor described above. The first processor may run at least one process, and the first processor needs to access data to implement the running process of the process. The target data represents the data that the first processor needs to access to run the current process. The number of target data is at least one.

[0044] In operation S220, determine whether the target data is stored at the first predetermined location. If not, operation S230 can be executed; if so, operation S240 can be executed.

[0045] For example, the first predetermined location may represent other locations outside the first local storage unit. For example, the first predetermined location may include at least one of a second local storage unit, a first remote storage unit, and a second remote storage unit.

[0046] It can be understood that when the number of target data is multiple, different target data may be stored at different first predetermined locations. For example, one target data is stored in the second local storage unit, and another target data is stored in the first remote storage unit.

[0047] In operation S230, there is no need to migrate the target data.

[0048] For example, in the case where it is determined that the target data is stored in the first local storage unit, there is no need to migrate the target data, and the first processor can directly access the target data from the first local storage unit.

[0049] In operation S240, migrate the target data from the first predetermined location to the first storage unit of the local node.

[0050] In some embodiments, the target data may be divided into high-frequency target data and low-frequency target data according to the access frequency. For example, the migration operation may be performed only on the high-frequency target data, and the migration operation is not performed on the low-frequency target data.

[0051] In other embodiments, the target data may not be divided into high-frequency target data and low-frequency target data, and in the case where any target data is accessed, the migration operation is performed on the accessed target data.

[0052] According to the technical solution provided by the embodiments of the present disclosure, since when a process runs on a first processor and the target data to be accessed is at a first predetermined location, the target data can be migrated from the first predetermined location to a first local storage unit, the system access performance can be improved.

[0053] Figure 3 Schematically shows a flowchart of a data processing method according to another embodiment of the present disclosure.

[0054] As Figure 3 shown, the data processing method 300 of this embodiment may include operation S310 to operation S330, and may further include operation S341 to operation S345.

[0055] In operation S310, according to the current process of the first processor, determine the target data accessed by the first processor.

[0056] In operation S320, determine whether the target data is stored at the first predetermined location. If not, operation S330 can be executed; if so, operation S341 can be executed.

[0057] In operation S330, for example, without migrating the target data, the first processor can directly access the target data from the first local storage unit.

[0058] For example, the above operations S310 to S330 can refer to the above operations S210 to S230, and details are not described herein for this embodiment.

[0059] In operation S341, determine whether the remaining storage space of the first local storage unit is greater than or equal to the occupied space of the target data. If so, operation S342 can be executed; if not, operation S343 can be executed.

[0060] In operation S342, migrate the target data from the first predetermined location to the first storage unit of the local node.

[0061] For example, if the accessed data is in the first remote storage unit, migrate the target data in the first remote storage unit to the first local storage unit. For Figure 1 example, if the process runs on CPU1, when the first storage unit of CPU1 has sufficient space and the target data to be accessed is in the first storage unit of CPU2, migrate the target data in the first storage unit of CPU2 to the first storage unit of CPU1.

[0062] For another example, if the accessed data is in the second remote storage unit, migrate the target data in the second remote storage unit to the first local storage unit. For Figure 1For example, if the process is running on CPU1, when the first storage unit of CPU1 has sufficient space and the target data to be accessed is in the second storage unit of CPU2, then migrate the target data in the second storage unit of CPU2 to the first storage unit of CPU1.

[0063] For another example, if the accessed data is in the second local storage unit, then migrate the target data in the second local storage unit to the first local storage unit. Figure 1 For example, if the process is running on CPU1, when the first storage unit of CPU1 has sufficient space and the target data to be accessed is in the second storage unit of CPU1, then migrate the target data in the second storage unit of CPU1 to the first storage unit of CPU1.

[0064] In operation S343, determine the cold data in the first storage unit of the local node according to the access frequency of the data in the first storage unit of the local node.

[0065] For example, the data with the number of accesses less than the quantity threshold within a past predetermined time period can be determined as cold data.

[0066] In operation S344, migrate the cold data from the first storage unit of the local node to the second predetermined location.

[0067] For example, the second predetermined location may include at least one of the first storage unit of the target remote node, the second storage unit of the target remote node, and the second local storage unit. It should be noted that the above-mentioned target remote node and the remote node currently storing the target data may be the same remote node or different remote nodes. Therefore, the first predetermined location and the second predetermined location may be the same or different.

[0068] In operation S345, migrate the target data from the first predetermined location to the first storage unit of the local node.

[0069] Specific examples of the above operation S344 and operation S345 are as follows:

[0070] In one example, if the target data to be accessed is in the first remote storage unit. For example, the cold data with a low access frequency in the first local storage unit can be migrated to the first storage unit of the target remote node first, and then the target data in the first remote storage unit can be migrated to the first local storage unit. For another example, the cold data with a low access frequency in the first local storage unit can be migrated to the second storage unit of the target remote node first, and then the target data in the first remote storage unit can be migrated to the first local storage unit. For another example, the cold data with a low access frequency in the first local storage unit can be migrated to the second local storage unit first, and then the target data in the first remote storage unit can be migrated to the first local storage unit.

[0071] In another example, if the target data to be accessed is in the second remote storage unit. For example, cold data with a low access frequency in the first local storage unit can be migrated to the first storage unit of the target remote node first, and then the target data in the second remote storage unit can be migrated to the first local storage unit. Another example is that cold data with a low access frequency in the first local storage unit can be migrated to the second storage unit of the target remote node first, and then the target data in the second remote storage unit can be migrated to the first local storage unit. Another example is that cold data with a low access frequency in the first local storage unit can be migrated to the second local storage unit first, and then the target data in the second remote storage unit can be migrated to the first local storage unit.

[0072] In another example, if the target data to be accessed is in the second local storage unit. For example, cold data with a low access frequency in the first local storage unit can be migrated to the first storage unit of the target remote node first, and then the target data in the second local storage unit can be migrated to the first local storage unit. Another example is that cold data with a low access frequency in the first local storage unit can be migrated to the second storage unit of the target remote node first, and then the target data in the second local storage unit can be migrated to the first local storage unit. Another example is that cold data with a low access frequency in the first local storage unit can be migrated to the second local storage unit first, and then the target data in the second local storage unit can be migrated to the first local storage unit.

[0073] According to another embodiment of the present disclosure, the second predetermined location includes N storage areas, where N is an integer greater than or equal to 2, and the storage area can represent any one of the first storage unit of the target remote node, the second storage unit of the target remote node, and the second local storage unit. The method of migrating cold data from the first storage unit of the local node to the second predetermined location can perform the following operations: determine the target storage area from the N storage areas according to the priorities of the N storage areas, the occupied space of the cold data, and the remaining storage spaces of the N storage areas respectively. Then migrate the cold data from the first storage unit of the local node to the target storage area.

[0074] According to the technical solution provided by the embodiment of the present disclosure, since the target storage area is selected based on the priority and then the cold data is migrated to the target storage area, the migration efficiency and the access performance of the data can be ensured.

[0075] Exemplarily, it can be determined first whether the remaining storage space of at least one storage area among the N storage areas is greater than or equal to the occupied space of the cold data, and different methods can be adopted to determine the target storage area according to the determination result.

[0076] In one example, when it is determined that the remaining storage space of at least one of the N storage areas is greater than or equal to the occupied space of the cold data, it means that a certain storage area can store all the cold data. Then, all the cold data is preferentially stored in one storage area without chunking the cold data, thereby avoiding reducing the efficiency of migrating the cold data due to data chunking.

[0077] In this case, the target storage area can be determined in the following way. For example, the following operations are repeatedly executed until the target storage area is obtained: The storage area with the highest priority among the N storage areas is determined as the current storage area. It is determined whether the remaining storage space of the current storage area is greater than or equal to the occupied space of the cold data. If so, the current storage area is determined as the target storage area; if not, the current storage area is deleted from the N storage areas.

[0078] The storage units included in the local node and the target remote node can be sorted according to performance, and the performance can characterize the speed at which the first processor accesses the target data. The performance of the first storage unit of the target remote and the second local storage unit needs to be determined according to the actual situation. Therefore, the above priorities can be that the priorities of the first storage unit of the target remote, the second local storage unit, and the second storage unit of the target remote decrease in turn, or the priorities of the second local storage unit, the first storage unit of the target remote, and the second storage unit of the target remote decrease in turn.

[0079] Adopting the technical solution provided in this example, the storage units included in the local node and the target remote node are sorted based on the priority, and all the cold data is preferentially migrated to the storage area with a high priority. When the current memory cannot store all the cold data, it is determined whether to migrate all the cold data to the next-level storage area. Through this solution, the migration efficiency is improved and the access performance of the cold data in the subsequent process is ensured.

[0080] In another example, when it is determined that the remaining storage space of each of the N storage areas is less than the occupied space of the cold data, it means that any storage area cannot store all the cold data. Then, the cold data can be chunked to ensure the completion of the migration process of the cold data.

[0081] In this case, the target storage area can be determined in the following way. For example, according to the priorities of the N storage areas, the occupied space of the cold data, and the respective remaining spaces of the N storage areas, the cold data is chunked to obtain at least two data blocks. Then, at least two target storage areas corresponding to the at least two data blocks are determined from the N storage areas.

[0082] For example, according to the remaining space of the storage area with the highest priority, a data block that can be stored in the remaining space is determined from the cold data. Then, according to the remaining space of the storage area with the next highest priority, a data block that can be stored in the remaining space is determined from the cold data. If there is still remaining cold data, a data block is determined again according to the remaining space of the storage area with the next highest priority after that. The above operations are repeatedly executed until the quantity of the cold data is 0.

[0083] For example, after chunking, one data block is migrated to the second local storage unit, another data block is migrated to the first storage unit of the target remote node, and another data block is migrated to the second storage unit of the target remote node.

[0084] Adopting the technical solution provided in this example, the storage units included in the local node and the target remote node are sorted based on the priority, and the data blocks in the cold data are preferentially migrated to the storage area with a higher priority. The higher the priority, the faster the migration speed. Therefore, through this solution, it can be ensured that the time taken to migrate all the cold data is short, and the access performance of the subsequent cold data is ensured.

[0085] According to another embodiment of the present disclosure, the quantity of the target data is multiple, and the multiple target data includes migrated data and un - migrated data. It can be understood that the migrated data is the target data that has been migrated from the first predetermined position to the first local storage unit among the multiple target data, and the un - migrated data is the target data that has not been migrated from the first predetermined position to the first local storage unit among the multiple target data.

[0086] Correspondingly, the method of migrating the target data from the first predetermined position to the first storage unit of the local node may include the following operations: in response to detecting that the ratio between the data volume of the migrated data and the data volume of the multiple target data is greater than or equal to the ratio threshold, the un - migrated data is migrated from the first predetermined position to the first storage unit of the local node.

[0087] For example, a process or a thread runs on the processor of the local node, and the target data to be accessed is in the storage unit of the remote node. If the target data to be accessed is 100M, and currently 10M of the target data has been migrated from the storage unit of the remote node to the first local storage unit, that is, the migrated data is 10M, then the remaining 90M of un - migrated data can be pre - migrated to the first local storage unit.

[0088] It can be understood that a process needs to be executed according to a time slice. After the time slice of a certain process is executed, other processes need to be executed. During the execution of other processes, in the embodiments of the present disclosure, the remaining unmigrated data can be pre-migrated to the first local storage unit during the execution of other processes. When the time slice of other threads ends and the process is executed again, the process can directly access the target data from the first local storage unit without waiting to migrate the target data again, thereby improving the processing efficiency.

[0089] In some embodiments, during the actual execution of pre-migrating the remaining unmigrated data, all the unmigrated data can be migrated to the first local storage unit at one time.

[0090] In other embodiments, during the actual execution of pre-migrating the remaining unmigrated data, all the unmigrated data can also be migrated to the first local storage unit in multiple batches. For example, according to the access order of multiple target data, the current candidate data can be determined from the unmigrated data; the current candidate data is migrated to the first storage unit of the local node; the current candidate data is deleted from the unmigrated data; in response to detecting that the current candidate data is accessed, the operation of determining the current candidate data is returned until the number of unmigrated data is 0.

[0091] For example, if the target data to be accessed is 100M, the 100M data can be divided into multiple candidate data, for example, each 10M is a candidate data. After accessing the first 10M candidate data, the second 10M candidate data can be migrated to the first local storage unit. After accessing the second 10M candidate data, the third 10M candidate data can be migrated to the first local storage unit, thus forming a sliding window.

[0092] The embodiments of the present disclosure adopt the method of migrating target data in batches, which can migrate the target data required by multiple threads in time when multiple threads are running, thereby improving the processing efficiency.

[0093] In practical applications, a process or a thread can be pre-bound to a fixed NUMA node core, so that the bound thread or process cannot run on other NUMA nodes. Thus, when the process and / or the thread is not executed, the above operation of pre-migrating the target data is performed.

[0094] According to another embodiment of the present disclosure, the process can also be automatically scheduled according to the load overhead. For example, if a process runs on the first NUMA node and the accessed data is on the second NUMA node. If the migration overhead of the data is too large, the process or the thread can be scheduled to the second NUMA node for execution.

[0095] According to another embodiment of the present disclosure, considering that migrating data incurs performance overhead. For example, when the space of the first local storage unit is insufficient and cold data needs to be migrated out, this process incurs an overhead, and then when migrating the target data to the first local storage unit, another overhead is incurred. Therefore, a predetermined space can be reserved in the first local storage unit.

[0096] For example, to avoid performance overhead caused by insufficient remaining space in the first local storage unit when frequently accessed data needs to be scheduled from the first predetermined location to the first local storage unit, the space of the first local storage unit can be reserved during memory management, so that the target data can be directly scheduled to the first local storage unit, avoiding the performance overhead caused by first scheduling out the cold data in the first local storage unit. In addition, by reserving space in the first local storage unit, while migrating the target data in, the cold data in the first local storage unit can also be migrated outwards, realizing parallel operations, thereby improving the data migration efficiency.

[0097] For another example, a certain space can also be reserved for the first storage unit of the remote node to avoid rescheduling overhead caused by lack of space when cold data is preferentially scheduled to the first storage unit of the remote node.

[0098] According to another embodiment of the present disclosure, the data processing module may include a basic resource identification sub-module, a memory management sub-module, a data access frequency statistics sub-module, and a data scheduling sub-module.

[0099] The basic resource identification sub-module is responsible for identifying the number of CPUs in the NUMA structure, the space size of the first storage unit owned by each CPU, the address range of the first storage unit, the space size of the second storage unit, and the address range of the second storage unit due to the local NUMA Node memory area.

[0100] The memory management sub-module is used to identify the total space capacity, the used space capacity, the free space capacity, etc. of the first local storage unit, the second local storage unit, the first remote storage unit, and the second remote storage unit respectively.

[0101] The data access frequency statistics sub-module is used to count the access frequencies of the data in the first local storage unit, the second local storage unit, the first remote storage unit, and the second remote storage unit.

[0102] The data scheduling sub-module is used to schedule processes and threads to the NUMA node to which the target data belongs, and to schedule and migrate data between each storage unit.

[0103] Based on the above data processing method, the present disclosure also provides a data processing device applied to a storage node. The following will be combined with Figure 4Describe the device in detail.

[0104] Figure 4 A schematic block diagram of a data processing device according to an embodiment of the present disclosure is shown.

[0105] As Figure 4 shown, the data processing device 400 of this embodiment can be applied to a storage node. The storage node includes a local node and a remote node. The local node and the remote node each include a first storage unit and a second storage unit with a slower read / write speed than the first storage unit. The local node is associated with a processor. The data processing device 400 includes: a determination module 410 and a migration module 420.

[0106] The determination module 410 is configured to determine target data accessed by the processor according to the current process of the processor. In one embodiment, the determination module 410 can be used to perform the operation S210 described above, which will not be elaborated here.

[0107] The migration module 420 is configured to, when it is determined that the target data is stored in a first predetermined location, migrate the target data from the first predetermined location to the first storage unit of the local node. The first predetermined location includes at least one of the following: the second storage unit of the local node, the first storage unit of the remote node, and the second storage unit of the remote node. In one embodiment, the migration module 420 can be used to perform the operation S220 described above, which will not be elaborated here.

[0108] According to another embodiment of the present disclosure, the migration module includes: a cold data determination sub-module, a first migration sub-module, and a second migration sub-module. The cold data determination sub-module is configured to, in response to detecting that the remaining storage space of the first storage unit of the local node is less than the occupied space of the target data, determine cold data in the first storage unit of the local node according to the access frequency of the data in the first storage unit of the local node. The first migration sub-module is configured to migrate the cold data from the first storage unit of the local node to a second predetermined location. The second migration sub-module is configured to migrate the target data from the first predetermined location to the first storage unit of the local node.

[0109] According to another embodiment of the present disclosure, the second predetermined location includes N storage areas, where N is an integer greater than or equal to 2. The first migration sub-module includes: a determination unit and a first migration unit. The determination unit is configured to determine a target storage area from the N storage areas according to the priorities of the N storage areas, the occupied space of the cold data, and the remaining storage space of each of the N storage areas. The first migration unit is configured to migrate the cold data from the first storage unit of the local node to the target storage area.

[0110] According to another embodiment of the present disclosure, the determination unit includes an execution subunit configured to repeatedly perform the following operations until a target storage area is obtained when it is determined that the remaining storage space of at least one of the N storage areas is greater than or equal to the occupied space of the cold data: determine the storage area with the highest priority among the N storage areas as the current storage area; determine whether the remaining storage space of the current storage area is greater than or equal to the occupied space of the cold data; if so, determine the current storage area as the target storage area; and if not, delete the current storage area from the N storage areas.

[0111] According to another embodiment of the present disclosure, the determination unit includes a chunking subunit and a determination subunit. The chunking subunit is configured to perform chunking processing on the cold data to obtain at least two data blocks according to the priorities of the N storage areas, the occupied space of the cold data, and the respective remaining spaces of the N storage areas when it is determined that the remaining storage space of each of the N storage areas is less than the occupied space of the cold data; the determination subunit is configured to determine at least two target storage areas corresponding to the at least two data blocks one by one from the N storage areas.

[0112] According to another embodiment of the present disclosure, the migration module includes a third migration sub-module configured to migrate the target data from a first predetermined position to a first storage unit of the local node in response to detecting that the remaining storage space of the first storage unit of the local node is greater than or equal to the occupied space of the target data.

[0113] According to another embodiment of the present disclosure, the number of target data is multiple, and the multiple target data includes migrated data and non-migrated data; the migration module includes a fourth migration sub-module configured to migrate the non-migrated data from the first predetermined position to the first storage unit of the local node in response to detecting that the ratio between the data volume of the migrated data and the data volume of the multiple target data is greater than or equal to a ratio threshold.

[0114] According to another embodiment of the present disclosure, the fourth migration sub-module includes: a candidate determination unit, a second migration unit, a deletion unit, and a return unit. The candidate determination unit is configured to determine a current candidate data from the non-migrated data according to the access order of the multiple target data; the second migration unit is configured to migrate the current candidate data to the first storage unit of the local node; the deletion unit is configured to delete the current candidate data from the non-migrated data; the return unit is configured to return the operation of determining the current candidate data in response to detecting that the current candidate data is accessed until the number of non-migrated data is 0.

[0115] According to an embodiment of the present disclosure, any of the determination module 410 and the migration module 420 may be combined and implemented in one module, or any one of them may be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the determination module 410 and the migration module 420 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on a substrate, a system in a package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging circuits, etc., implemented by hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, at least one of the determination module 410 and the migration module 420 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0116] Figure 5 FIG. schematically shows a block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0117] As Figure 5 shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage section 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 501 may also include on-board memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0118] In the RAM 503, various programs and data required for the operation of the electronic device 500 are stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. The processor 501 performs various operations of the method flow according to an embodiment of the present disclosure by executing the program in the ROM 502 and / or the RAM 503. It should be noted that the program may also be stored in one or more memories other than the ROM 502 and the RAM 503. The processor 501 may also perform various operations of the method flow according to an embodiment of the present disclosure by executing the program stored in the one or more memories.

[0119] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, and the input / output (I / O) interface 505 is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. The drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom can be installed into the storage portion 508 as needed.

[0120] The present disclosure also provides a computer-readable storage medium, which may be included in the device / device / system described in the above embodiments; or may exist separately without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0121] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device, or device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503.

[0122] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to cause the computer system to implement the item recommendation method provided by the embodiments of the present disclosure.

[0123] When the computer program is executed by the processor 501, the above functions defined in the system / apparatus of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0124] In one embodiment, the computer program can rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program can also be transmitted and distributed in the form of signals on a network medium, and be downloaded and installed through the communication part 509, and / or be installed from the removable medium 511. The program code included in the computer program can be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0125] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or be installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the above-described systems, devices, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0126] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedures and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include but are not limited to, such as Java, C++, python, the "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0128] Those skilled in the art will appreciate that the features recited in the various embodiments and / or claims of the present disclosure may be combined in various ways and / or combinations, even if such combinations or combinations are not explicitly recited in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features recited in the various embodiments and / or claims of the present disclosure may be combined in various ways and / or combinations. All such combinations and / or combinations fall within the scope of the present disclosure.

[0129] The embodiments of the present disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present disclosure.

Claims

1. A data processing method applied to a storage node, the storage node including a local node and a remote node, the local node and the remote node each including a first storage unit and a second storage unit with a slower read / write speed than the first storage unit, the local node being associated with a processor; The method includes: Determining target data accessed by the processor according to the current process of the processor; and When it is determined that the target data is stored at a first predetermined position, in response to detecting that the remaining storage space of the first storage unit of the local node is less than the occupied space of the target data, determining cold data in the first storage unit of the local node according to the access frequency of the data in the first storage unit of the local node; When it is determined that the remaining storage space of at least one of the N storage areas included in the second predetermined position is greater than or equal to the occupied space of the cold data, repeatedly perform the following operations until a target storage area is obtained: determining the storage area with the highest priority among the N storage areas as the current storage area; determining whether the remaining storage space of the current storage area is greater than or equal to the occupied space of the cold data; if so, determining the current storage area as the target storage area; if not, deleting the current storage area from the N storage areas; N is an integer greater than or equal to 2; Migrating the cold data from the first storage unit of the local node to the target storage area; and Migrating the target data from the first predetermined position to the first storage unit of the local node; Wherein, the first predetermined position includes at least one of the following: the second storage unit of the local node, the first storage unit of the remote node, and the second storage unit of the remote node.

2. The method according to claim 1, further comprising: When it is determined that the remaining storage space of each of the N storage areas is less than the occupied space of the cold data, performing block processing on the cold data according to the priorities of the N storage areas, the occupied space of the cold data, and the respective remaining spaces of the N storage areas to obtain at least two data blocks; And Determining at least two target storage areas corresponding to the at least two data blocks one by one from the N storage areas.

3. The method according to claim 1, wherein, The migrating the target data from the first predetermined position to the first storage unit of the local node includes: In response to detecting that the remaining storage space of the first storage unit of the local node is greater than or equal to the occupied space of the target data, migrating the target data from the first predetermined position to the first storage unit of the local node.

4. The method according to any one of claims 1 to 3, wherein The number of the target data is multiple, and the multiple target data includes migrated data and un - migrated data; The migrating the target data from the first predetermined position to the first storage unit of the local node includes: In response to detecting that the ratio between the data volume of the migrated data and the data volume of the multiple target data is greater than or equal to a ratio threshold, migrating the un - migrated data from the first predetermined position to the first storage unit of the local node.

5. The method according to claim 4, wherein, The migrating the un - migrated data from the first predetermined position to the first storage unit of the local node includes: Determining current candidate data from the un - migrated data according to the access order of the multiple target data; Migrating the current candidate data to the first storage unit of the local node; Deleting the current candidate data from the un - migrated data; and In response to detecting that the current candidate data is accessed, return the operation of determining the current candidate data until the number of the non-migrated data is 0.

6. A data processing device applied to a storage node, the storage node including a local node and a remote node, the local node and the remote node each including a first storage unit and a second storage unit with a slower read / write speed than the first storage unit, the local node being associated with a processor; The device includes: a determination module, configured to determine target data accessed by the processor according to the current process of the processor; and a cold data determination sub-module, configured to, when it is determined that the target data is stored in a first predetermined location, in response to detecting that the remaining storage space of the first storage unit of the local node is less than the occupied space of the target data, determine cold data in the first storage unit of the local node according to the access frequency of the data in the first storage unit of the local node; an execution sub-unit, configured to, when it is determined that the remaining storage space of at least one of the N storage areas included in the second predetermined location is greater than or equal to the occupied space of the cold data, repeatedly execute the following operations until a target storage area is obtained: determine the storage area with the highest priority among the N storage areas as the current storage area; determine whether the remaining storage space of the current storage area is greater than or equal to the occupied space of the cold data; if so, determine the current storage area as the target storage area; if not, delete the current storage area from the N storage areas; N is an integer greater than or equal to 2; a first migration unit, configured to migrate the cold data from the first storage unit of the local node to the target storage area; and a migration module, configured to migrate the target data from the first predetermined location to the first storage unit of the local node; wherein, the first predetermined location includes at least one of the following: the second storage unit of the local node, the first storage unit of the remote node, and the second storage unit of the remote node.

7. An electronic device, including: one or more processors; a storage device, configured to store one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 5.

9. A computer program product, including a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images