Data migration method, system and device, computer equipment and storage medium
By adopting the continuous and continuous processing mechanism of list transfer and data transfer interaction concurrent and uninterrupted in the chip, the problem of low data transfer efficiency is solved and the chip performance is improved.
Patent Information
- Application Number
- CN202411080428.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, data transfer scheduling consumes a lot of time, resulting in low data transfer efficiency and inability to meet the high-performance requirements of chips.
Adopt a continuous and continuous processing mechanism of list transfer and data transfer interactive concurrent and uninterrupted interaction, data transfer is carried out while address transfer, and when the number of local addresses and host addresses in the list cache space is less than the preset threshold, the next list transfer is started to ensure data Moving is uninterrupted.
It improves data transfer efficiency, improves chip performance, and solves the problem of long data transfer scheduling time.
Smart Images

Figure CN119065603A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of chip design technology, and in particular to a data migration method, system, device, computer equipment and storage medium. Background Art
[0002] With the rise of computational storage technology in recent years, computational storage architecture has unloaded data calculations from the host CPU (Central Processing Unit) to a local data processing acceleration unit close to the storage unit, reducing the movement of data between the CPU and the storage unit, thereby maximizing the performance of the host system.
[0003] To offload data calculation from the host CPU to the data processing acceleration unit, the relevant data needs to be quickly moved to the local data processing acceleration unit. Currently, when moving data, the scheduling of data movement consumes a lot of time, resulting in very low data movement efficiency, which does not meet the high performance requirements of the chip.
[0004] Therefore, the related technology has the problem that the scheduling of data movement consumes a lot of time, resulting in low data movement efficiency and failure to meet the high performance requirements of the chip. Summary of the invention
[0005] In view of this, the present invention provides a data migration method, system, apparatus, computer device and storage medium to solve the problem that scheduling data migration consumes a lot of time, resulting in low data migration efficiency and failure to meet the high performance requirements of the chip.
[0006] In a first aspect, the present invention provides a data migration method, which is applied to a task processing module and includes:
[0007] When task information of the data migration task is obtained, a first local list pointer and a first host list pointer in the task information are obtained, wherein the first local list pointer is used to determine a local list including multiple local addresses, and the first host list pointer is used to determine a host list including multiple host addresses;
[0008] According to the first local list pointer and the first host list pointer, at least part of the local addresses in the local list and at least part of the host addresses in the host list are moved to the list cache space;
[0009] If there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, at least part of the local addresses of the next local list and at least part of the host addresses of the next host list are moved to the list cache space;
[0010] While the address is being moved, the target local address and the target host address are extracted from the list cache space, and the data in the target local address is moved to the target local address.
[0011] The data migration method provided in this embodiment adopts a processing mechanism of interactive concurrent continuous uninterrupted list migration and data migration, performs data migration at the same time as address migration, extracts the target local address and the target host address from the list cache space, and moves the data in the target local address to the target local address, thereby improving the efficiency of data migration. The method adopts a processing mechanism of interactive concurrent continuous uninterrupted host list migration and data migration, and starts the next list migration when the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, ensuring uninterrupted data migration, improving data migration efficiency, and improving chip performance. The method solves the problem that the scheduling of data migration consumes a lot of time, resulting in low data migration efficiency and failure to meet the high performance requirements of the chip.
[0012] In an optional implementation, extracting the target local address and the target host address from the list cache space and moving the data in the target local address to the target local address includes:
[0013] Extract the target local address and the target host address from the list cache space;
[0014] The target local address is used as the destination address, the target host address is used as the source address, and the offset between the destination address and the source address and the data size of the data to be moved are determined;
[0015] When the data size is smaller than a second preset threshold, generating a data move command including a destination address, a source address, an offset and a data size;
[0016] The data migration command is sent to the migration module, instructing the migration module to move the data to be migrated from the target host address to the target local address.
[0017] In this embodiment, the target local address is used as the destination address, the target host address is used as the source address, a data move command including the destination address, the source address, the offset and the data size is generated, and the data move command is sent to the move module, and a new data move command can be continuously generated. The move module is only used to move the data to be moved from the target host address to the target local address, so as to realize the step-by-step concurrency mechanism and make the data move command uninterrupted. In addition, it is ensured that the data size of the data to be moved is less than the second preset threshold value to avoid the data being moved at a single time being too large, resulting in a long move time and the inability to process other data move commands in time.
[0018] In an optional implementation, sending a data migration command to the migration module to instruct the migration module to migrate the data to be migrated from the target host address to the target local address includes:
[0019] Convert the data transfer command according to the data receiving format of the transfer module, and send the converted data transfer command to the transfer module, instructing the transfer module to transfer the data to be transferred from the target host address to the target local address;
[0020] A first interrupt signal is obtained from the migration module, and first completion information of the data migration command is determined according to the first interrupt signal.
[0021] In an optional implementation, the list cache space includes a local list sub-cache space and a host list sub-cache space; according to the first local list pointer and the first host list pointer, at least part of the local addresses in the local list and at least part of the host addresses in the host list are moved to the list cache space, including:
[0022] Generate a first local list moving command including a first local list pointer, convert the format of the first local list moving command according to a data receiving format of the moving module, send the converted first local list moving command to the moving module, and instruct the moving module to move at least part of the local addresses in the local list to the local list sub-cache space;
[0023] Obtaining a second local list pointer in the local list sub-cache space, and caching the second local list pointer, wherein the second local list pointer is used to determine a next local list;
[0024] Acquire a second interrupt signal from the moving module, and determine second completion information of the first local list moving command according to the second interrupt signal;
[0025] Generate a first host list moving command including a first host list pointer, convert the format of the first host list moving command according to a data receiving format of the moving module, send the converted first host list moving command to the moving module, and instruct the moving module to move at least part of the host addresses in the host list to the host list sub-cache space;
[0026] Obtaining a second host list pointer in the host list sub-cache space, and caching the second host list pointer, wherein the second host list pointer is used to determine a next host list;
[0027] A third interrupt signal is obtained from the migration module, and third completion information of the first host list migration command is determined according to the third interrupt signal.
[0028] In this embodiment, a first host list move command and a first local list move command are generated to instruct the move module to move at least part of the local addresses in the local list to the local list sub-cache space, and to move at least part of the host addresses in the host list to the host list sub-cache space, so as to facilitate subsequent data movement. In addition, a second local list pointer and a second host list pointer are obtained and cached, so as to facilitate the subsequent movement of the local addresses of the next local list and the host addresses of the next host list.
[0029] In an optional implementation, when there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, moving at least part of the local addresses of the next local list and at least part of the host addresses of the next host list to the list cache space includes:
[0030] Determine whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold;
[0031] If there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, generating a second local list move command including a preset number and a second local list pointer;
[0032] Convert the format of the second local list transfer command according to the data receiving format of the transfer module, send the converted second local list transfer command to the transfer module, and instruct the transfer module to transfer a preset number of local addresses in the next local list to the local list sub-cache space;
[0033] Acquire a fourth interrupt signal from the moving module, and determine fourth completion information of the second local list moving command according to the fourth interrupt signal;
[0034] Generate a second host list move command including a preset number and a second host list pointer;
[0035] Convert the format of the second host list transfer command according to the data receiving format of the transfer module, send the converted second host list transfer command to the transfer module, and instruct the transfer module to transfer a preset number of host addresses in the next host list to the host list sub-cache space;
[0036] Acquire a fifth interrupt signal from the migration module, and determine fifth completion information of the second host list migration command according to the fifth interrupt signal;
[0037] Starting from determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, executing subsequent steps, until the next local list and the next host list are both empty, then terminating the moving process of the next local list and the next host list, determining whether a third local list pointer can be read from the local list sub-cache space, and determining whether a third host list pointer can be read from the host list sub-cache space;
[0038] If the third local list pointer and the third host list pointer can be read, the third local list pointer is used as the second local list pointer, and the third host list pointer is used as the second host list pointer, and subsequent steps are performed from determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, until the local address and host address are moved.
[0039] In this embodiment, when there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, a preset number of local addresses in the next local list are migrated to the local list sub-cache space, and a preset number of host addresses in the next host list are migrated to the host list sub-cache space so that the local address and the host address can be obtained from the list cache space each time data is moved, thereby ensuring that the address information in the list cache space is continuous, the data movement process is continuous and uninterrupted, and high-speed operation of the system can be achieved, thereby improving the efficiency of data movement of the entire system.
[0040] In an optional embodiment, the method further includes:
[0041] Obtaining first completion information, second completion information, third completion information, fourth completion information, and fifth completion information;
[0042] Acquire process information when generating a migration command, wherein the migration command includes: a data migration command, a first local list migration command, a second local list migration command, a first host list migration command, and a second host list migration command;
[0043] Generate total information to be reported including first completion information, second completion information, third completion information, fourth completion information, fifth completion information and process information, and upload the total information to be reported to the task issuing module, wherein the task issuing module is used to issue data migration tasks.
[0044] In this embodiment, the total information to be reported, including the completion information of the move command and the process information when the move command is generated, is uploaded to the task issuing module, so that the task issuing module can monitor the entire data migration process and handle fault problems in the data migration process in a timely manner.
[0045] In a second aspect, the present invention provides a data migration system, the system comprising: a task issuing module, a task receiving and processing node, and a first preset number of task processing modules;
[0046] The task sending module is connected to the task receiving and processing node, and is used to send the data migration task to the task receiving and processing node;
[0047] The task receiving processing node is connected to the task processing module, and is used to obtain task information according to the data migration task, determine a target task processing module from a first preset number of task processing modules, and send the task information to the target task processing module;
[0048] The target task processing module is used to execute the data migration method of the first aspect or any corresponding implementation method thereof.
[0049] The data migration system provided in this embodiment adopts a processing mechanism of interactive concurrent continuous uninterrupted list migration and data migration, performs data migration at the same time as address migration, extracts the target local address and the target host address from the list cache space, and moves the data in the target local address to the target local address, thereby improving the efficiency of data migration. The system adopts a processing mechanism of interactive concurrent continuous uninterrupted host list migration and data migration, and starts the next list migration when the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, ensuring uninterrupted data migration, improving data migration efficiency, and improving chip performance. The system solves the problem that the scheduling of data migration consumes a lot of time, resulting in low data migration efficiency and failure to meet the high performance requirements of the chip.
[0050] In an optional embodiment, the system further includes: a moving module, the task processing module includes: a task parsing submodule, a data moving task management submodule, a response reporting submodule, a list moving task management submodule and a second preset number of task processing submodules;
[0051] The task parsing submodule is connected to the task processing submodule, and is used to obtain the first local list pointer and the first host list pointer in the task information, determine the target task processing submodule from the second preset number of task processing submodules, and send the first local list pointer and the first host list pointer to the target task processing submodule;
[0052] The target task processing submodule is used to generate a first local list moving command including a first local list pointer and a first host list moving command including a first host list pointer, and send the first local list moving command and the first host list moving command to the list moving task management submodule;
[0053] The list migration task management submodule is used to perform format conversion on the first local list migration command and the first host list migration command according to the data receiving format of the migration module, send the converted first local list migration command to the migration module, instruct the migration module to migrate at least part of the local addresses in the local list to the list cache space, and send the converted first host list migration command to the migration module, instructing the migration module to migrate at least part of the host addresses in the host list to the list cache space;
[0054] The target task processing submodule is further used to extract the target local address and the target host address from the list cache space, use the target local address as the destination address, use the target host address as the source address, and determine the offset between the destination address and the source address and the data size of the data to be moved, and when the data size is less than a second preset threshold, generate a data moving command including the destination address, the source address, the offset and the data size, and send the data moving command to the data moving task management submodule;
[0055] The data migration task management submodule is used to convert the format of the data migration command according to the data receiving format of the migration module, send the converted data migration command to the migration module, and instruct the migration module to move the data to be migrated from the target host address to the target local address;
[0056] The response reporting submodule is used to obtain the completion information of the move command and the process information when generating the move command, generate the total information to be reported including the completion information and the process information, and upload the total information to be reported to the task issuing module, wherein the move command includes the first local list move command, the first host list move command and the data move command.
[0057] In this embodiment, the system utilizes a migration module, a task analysis submodule, a data migration task management submodule, a response reporting submodule, and a list migration task management submodule corresponding to a second preset number of task processing submodules, which can ensure that the chip can efficiently migrate data while minimizing the chip area and reducing the cost of the chip.
[0058] In a third aspect, the present invention provides a data migration device, the device comprising:
[0059] an acquiring unit, configured to acquire, when task information of the data migration task is acquired, a first local list pointer and a first host list pointer in the task information, wherein the first local list pointer is used to determine a local list including a plurality of local addresses, and the first host list pointer is used to determine a host list including a plurality of host addresses;
[0060] A first moving unit, configured to move at least part of the local addresses in the local list and at least part of the host addresses in the host list to the list cache space according to the first local list pointer and the first host list pointer;
[0061] A second moving unit is used to move at least part of the local addresses of the next local list and at least part of the host addresses of the next host list to the list cache space when there are local addresses and host addresses in the list cache space and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold;
[0062] The third moving unit is used to extract the target local address and the target host address from the list cache space while the address is being moved, and to move the data in the target local address to the target local address.
[0063] In a fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data migration method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0064] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data migration method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0066] Figure 1 is a flow chart of a data migration method according to an embodiment of the present invention;
[0067] Figure 2 is a block diagram of the hardware structure of the data migration high-speed processing control device according to an embodiment of the present invention;
[0068] Figure 3 It is a schematic diagram of the process of concurrent host list migration, local list migration and data migration according to an embodiment of the present invention;
[0069] Figure 4 is a structural schematic diagram of a data migration system according to an embodiment of the present invention;
[0070] Figure 5 is a schematic diagram of the structure of a traditional task processing module according to an embodiment of the present invention;
[0071] Figure 6 is a comparison chart of the execution time of a single data migration task of the present application according to an embodiment of the present invention and the execution time of a single data migration task of a traditional solution;
[0072] Figure 7 is a schematic diagram of hierarchical concurrent structural tasks of a data migration method according to an embodiment of the present invention;
[0073] Figure 8 is a comparison chart of multi-task execution efficiency between the present application scheme according to an embodiment of the present invention and the traditional scheme;
[0074] Fig. 9 is a flow chart of a data migration high-speed processing control method according to an embodiment of the present invention;
[0075] Fig.10 is a structural block diagram of a data migration device according to an embodiment of the present invention;
[0076] Fig.11 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0078] With the rise of computing storage technology in recent years, computing storage architecture has reduced the movement of data between CPU and storage unit by offloading data calculation from host CPU to data processing acceleration unit close to storage unit, thus releasing the host system performance as much as possible. Computing acceleration architecture is a newly defined computing storage architecture. Its core idea is to innovate micro-architecture, use microcode drive to optimize the division of software and hardware interfaces, and have high flexibility and scalability of software programmability while taking into account hardware high performance. The computing storage architecture adopts NVMe (non-volatile memory host controller interface specification) storage technology, and its solution is to implement data movement function in the software layer. In this way, when data is moved, the scheduling of the software layer will consume a lot of time and the efficiency is low. Therefore, in the computing storage architecture, it is becoming more and more important to hardwareize software operations, localize remote operations, and release CPU resources to the greatest extent to improve system performance. The hardware circuit related to data movement is also a very important part of it.
[0079] In today's chip design, with the increasing integration, chip complexity, chip size, integrated modules, overall performance requirements, and application scenarios changing faster and faster, how to efficiently and quickly complete data migration has become a key point in chip design.
[0080] Based on the above content, an embodiment of the present invention provides a data migration method, which provides a processing mechanism for interactive concurrent continuous uninterrupted list migration and data migration. While the address is being moved, the data is moved according to the host address and local address being moved, thereby improving the efficiency of data migration. A highly configurable step-by-step concurrency mechanism is provided, which can ensure the high performance of the chip while maximally reducing the chip area and reducing the cost of the chip. It is applicable to various chip designs that require data migration processing. In order to achieve the effect of interactive concurrent continuous uninterrupted list migration and data migration, and improve the performance of chip data migration.
[0081] According to an embodiment of the present invention, a data migration method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a task processing module with data processing capabilities, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0082] In this embodiment, a data migration method is provided. Figure 1 is a flow chart of a data migration method according to an embodiment of the present invention, which is applied to a task processing module, such as Figure 1 As shown, the process includes the following steps:
[0083] Step S101, when task information of a data migration task is obtained, obtain a first local list pointer and a first host list pointer in the task information, wherein the first local list pointer is used to determine a local list including multiple local addresses, and the first host list pointer is used to determine a host list including multiple host addresses.
[0084] Specifically, combined Figure 2 This embodiment is described. Figure 2 As shown, the task processing module includes a task processing engine, and the number of task processing modules can be set according to actual needs. The upper-level processing engine in the task issuing module is used to issue data movement tasks and provide upstream and downstream related information. The task receiving processing node is used to read the command into the local cache through the AXI (Advanced eXtensible Interface, on-chip bus protocol) read operation according to the address information of the received data movement task, and obtain the task information of the data movement task. Check which task processing module is idle and send the task information to the idle task processing module. The task processing module contains circuits that do not affect the performance bottleneck and the function of parsing the CB (flag control command block) task, and uses the task processing engine for unified management and is responsible for managing the reception of the first-level tasks.
[0085] The task processing module includes a task parsing submodule. The task parsing submodule parses the task information according to the system scenario, and obtains the source address first pointer, i.e., the first host list pointer, the destination address first pointer, i.e., the first local list pointer, the amount of data moved, the control mode, the reporting content, and other information in the task information. After the task information is parsed, check which task processing submodule is idle among the multiple task processing submodules included in the task processing module, and send the first host list pointer, the first local list pointer, and other information to the idle task processing submodule.
[0086] Step S102: according to the first local list pointer and the first host list pointer, at least part of the local addresses in the local list and at least part of the host addresses in the host list are moved to the list cache space.
[0087] Specifically, the task processing submodule in the task processing module uses its own subtask processing engine to determine a local list including multiple local addresses according to the first local list pointer, and determines a host list including multiple host addresses according to the first host list pointer. In addition, the task processing submodule includes a task management unit, which is responsible for receiving and verifying the task information sent by the superior, sending the tasks that pass the verification to the subordinate modules for processing, and is also responsible for controlling the subordinate modules, controlling their start, how to interact and end, etc. The subordinate modules of the task management unit are the list transfer conversion unit, the local list unit and the host list unit included in the task processing submodule.
[0088] The list cache space is composed of a local list unit and a host list unit, and the circuit of the host list unit and the circuit of the local list unit are independently triggered in parallel. The list transfer conversion unit, according to the first local list pointer and the first host list pointer, gradually transfers at least part of the host addresses in the host list and at least part of the local addresses in the local list to the host list unit and the local list unit in the list cache space. For example: the list transfer conversion unit generates a local list transfer command containing the first local list pointer, sends the local list transfer command to the list transfer task management submodule, and the list transfer task management submodule forwards the local list transfer command to the transfer module, indicating that the transfer module can transfer at least part of the local addresses in the local list to the local list unit in the list cache space; the list transfer conversion unit generates a host list transfer command containing the first host list pointer, sends the host list transfer command to the list transfer task management submodule, and the list transfer task management submodule forwards the host list transfer command to the transfer module, indicating that the transfer module can transfer at least part of the host addresses in the host list to the host list unit in the list cache space.
[0089] Step S103, when there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, at least part of the local addresses of the next local list and at least part of the host addresses of the next host list are moved to the list cache space.
[0090] Specifically, the traditional NVMe address list needs to find the pointer of the last level list each time before moving to the next list. When moving data, the address will be extracted from the list and the addresses in the list will be consumed. After each consumption of the current list, it is necessary to wait for a while before the address in the next list can be moved. Data movement also requires waiting for a while, which affects the efficiency of data movement and is not conducive to performance improvement.
[0091] This embodiment adopts a transfer processing method. When the local addresses and host addresses in the list cache space are consumed to a number less than a first preset threshold, at least part of the local addresses of the next local list and at least part of the host addresses of the next host list are moved. For example, half of the local addresses in the next local list and half of the host addresses in the next host list can be moved. Each list move is based on half of the cache. In this way, the bubbles in the middle of the entire list reading are eliminated, so that each data move can obtain the local address and host address from the list cache space. The efficiency of data movement of the entire system is improved.
[0092] When there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, the list move conversion unit generates a new host list move command and a new local list move command respectively, and the list move task management submodule forwards the new host list move command and the new local list move command to the move module, instructing the move module to move at least part of the local addresses of the next local list and at least part of the host addresses of the next host list to the list cache space.
[0093] Step S104, while the address is being moved, extract the target local address and the target host address from the list cache space, and move the data in the target local address to the target local address.
[0094] Specifically, the data movement starts at the same time as the address is moved. The subtask processing engine of the task processing submodule includes a data movement conversion unit. When the list cache space has a local address and a host address, the data movement conversion unit is synchronously started to move the data. The data movement conversion unit is responsible for consuming the local list and the host list in the list cache space, extracting the target local address and the target host address from the list cache space, and generating a data movement command containing the target local address and the target host address. The data movement task management submodule sends the data movement command to the moving module, instructing the moving module to move the data in the target local address to the target local address.
[0095] The data migration method provided in this embodiment adopts a processing mechanism of interactive concurrent continuous uninterrupted list migration and data migration, performs data migration at the same time as address migration, extracts the target local address and the target host address from the list cache space, and moves the data in the target local address to the target local address, thereby improving the efficiency of data migration. The method adopts a processing mechanism of interactive concurrent continuous uninterrupted host list migration and data migration, and starts the next list migration when the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, ensuring uninterrupted data migration, improving data migration efficiency, and improving chip performance. The method solves the problem that the scheduling of data migration consumes a lot of time, resulting in low data migration efficiency and failure to meet the high performance requirements of the chip.
[0096] In some optional implementations, extracting the target local address and the target host address from the list cache space and moving the data in the target local address to the target local address includes:
[0097] Extract the target local address and the target host address from the list cache space;
[0098] The target local address is used as the destination address, the target host address is used as the source address, and the offset between the destination address and the source address and the data size of the data to be moved are determined;
[0099] When the data size is smaller than a second preset threshold, generating a data move command including a destination address, a source address, an offset and a data size;
[0100] The data migration command is sent to the migration module, instructing the migration module to move the data to be migrated from the target host address to the target local address.
[0101] Specifically, when the list cache space has the local address and the host address, the data migration conversion unit is synchronously started to perform data migration.
[0102] The data migration conversion unit consumes the local list and host list in the list cache space, extracts the target local address from the local list unit in the list cache space and the target host address from the host list unit in the list cache space. The local list unit is responsible for caching and verifying the local list. The host list unit is responsible for caching and verifying the host list.
[0103] The data migration conversion unit uses the target local address as the destination address and the target host address as the source address, and determines the offset between the destination address and the source address and the data size of the data to be migrated.
[0104] If the size of the data to be moved at one time is too large, each move will take too long, and if it is too small, computing resources will be wasted. Therefore, the data move command of this embodiment is a source address and destination address command that does not span 4KB, that is, the data size of the data to be moved does not exceed 4KB, and the second preset threshold is 4KB. The size of the second preset threshold can be modified according to actual needs. When the data size is less than the second preset threshold, a data move command including the destination address, source address, offset and data size is generated. When the data size is greater than or equal to the second preset threshold, the data size is adjusted to be less than the second preset threshold.
[0105] The data migration command is sent to the migration module, instructing the migration module to move the data to be migrated from the target host address to the target local address.
[0106] The above process is as follows Figure 3 As shown, the data migration conversion unit migrates data, obtains the host address from the host list unit in the list cache space, and obtains the local address from the local list unit in the list cache space to perform data migration. Figure 3Step 2 indicates that the data migration conversion unit takes the source address and the destination address from the host list and the local list at the same time and performs a data migration. The data migration conversion unit continuously consumes the host list and the local list until a CB task is completed. The unit completes the execution of this task and starts the next task.
[0107] In this embodiment, the target local address is used as the destination address, the target host address is used as the source address, a data move command including the destination address, the source address, the offset and the data size is generated, and the data move command is sent to the move module, and a new data move command can be continuously generated. The move module is only used to move the data to be moved from the target host address to the target local address, so as to realize the step-by-step concurrency mechanism and make the data move command uninterrupted. In addition, it is ensured that the data size of the data to be moved is less than the second preset threshold value to avoid the data being moved at a single time being too large, resulting in a long move time and the inability to process other data move commands in time.
[0108] In some optional implementations, sending a data migration command to a migration module to instruct the migration module to migrate the data to be migrated from a target host address to a target local address includes:
[0109] Convert the data transfer command according to the data receiving format of the transfer module, and send the converted data transfer command to the transfer module, instructing the transfer module to transfer the data to be transferred from the target host address to the target local address;
[0110] A first interrupt signal is obtained from the migration module, and first completion information of the data migration command is determined according to the first interrupt signal.
[0111] Specifically, the data migration conversion unit sends the generated data migration command to the data migration task management submodule.
[0112] The data migration task management submodule is responsible for receiving the data migration commands generated by the data migration conversion unit in each task processing submodule, and processing the data migration commands into a format acceptable to the lower-level module, sending them to the lower-level module for processing, completing the information transmission and interacting in the form of pulse interruption to improve the execution efficiency of the system. The lower-level module of the data migration task management submodule is the migration module.
[0113] The data migration task management submodule converts the format of the data migration command according to the data receiving format of the migration module, and sends the converted data migration command to the migration module. At this time, the migration module can receive and identify the converted data migration command, and according to the converted data migration command, move the data to be migrated from the target host address to the target local address. The data receiving format is, for example, JSON (JavaScript Object Notation) format, CSV (Comma-Separated Values) format, etc., among which JSON format is a lightweight data exchange format, and CSV format is a general table data format.
[0114] In addition, the data migration task management submodule obtains a first interrupt signal from the migration module, and determines first completion information of the data migration command according to the first interrupt signal. The first completion information may be, for example, data migration command execution success, data migration command execution failure, etc.
[0115] The above process is as follows Figure 2 As shown, the data moving task management submodule obtains the data moving command from the data moving conversion unit in the task processing submodule by means of Dma (Direct Memory Access)_wr (write). After the data moving command is formatted according to the data receiving format of the moving module, the data in the converted data moving command is sent to the moving module. An interrupt signal is obtained from the moving module, and the completion information of the data moving command is determined according to the interrupt signal.
[0116] In some optional implementations, the process of the migration module migrating the data to be migrated from the target host address to the target local address includes steps A1 to A4.
[0117] Step A1, establishing a mapping between the memory space of the local device and the memory space of the host device to form a shared memory between the local device and the host device.
[0118] Specifically, the local device is a local device whose address is the target local address, and the host device is a host device whose address is the target host address. By mapping outwards on the host device and mapping inwards on the local device, the reading, writing and access of the shared memory space between the host device and the local device are realized. Shared memory is used to transfer data to be moved, and the amount of data to be moved is generally small. As mentioned above, it does not exceed 4KB. Therefore, data movement can be achieved by mapping the host device. This embodiment is not limited to this. It is also within the protection scope of the present invention to realize shared memory by the local device through outward mapping. Specifically, the shared memory is at least used to transmit the data to be moved. In some preferred examples, the shared memory can also realize the transmission of flag information and address information.
[0119] In step A2, the address information of the shared memory is mapped by outbound mapping to perform data migration from the host device to the local device and from the local device to the host device.
[0120] Specifically, by outward mapping, a mapping of the memory space is established based on the address information of the shared memory to achieve data movement. Compared with the inward mapping method, there is no limit on the amount of data in the mapping window. After the mapping of the memory space is established by outward mapping based on the address information of the shared memory, the data movement step from the host device to the local device can be performed through direct memory access of the local device, reducing the performance impact of data movement on the central processor and further improving data transmission performance.
[0121] In an optional implementation, step A2 includes steps A21 to A23.
[0122] Step A21, receiving a first interrupt message sent by the host device, the first interrupt message indicating that the host device has written address information of the data to be moved in the memory space of the host device into the shared memory.
[0123] Step A22: The local device obtains address information of the memory space of the host device from the shared memory.
[0124] In step A23, the local device maps the address information of the shared memory by outward mapping to move the data to be moved from the memory space corresponding to the address information, that is, to move the data to be moved from the memory space of the host device to the local device.
[0125] Therefore, data migration from the host device to the local device is achieved by transmitting the address information of the shared memory and performing outward mapping by the local device.
[0126] In the data migration method applied to the local device provided by the present application, data migration between the local device and the host device is achieved by mapping the local device outward, thereby breaking through the limitation of bar on data transmission between the host device and the local device; data migration from the host device to the local device is achieved through the local device, thereby distributing the system performance loss to the local device, reducing the CPU occupancy rate of the host device, and improving the scalability of the entire system.
[0127] In some optional implementations, the list cache space includes a local list sub-cache space and a host list sub-cache space; and according to the first local list pointer and the first host list pointer, at least part of the local addresses in the local list and at least part of the host addresses in the host list are moved to the list cache space, including:
[0128] Generate a first local list moving command including a first local list pointer, convert the format of the first local list moving command according to a data receiving format of the moving module, send the converted first local list moving command to the moving module, and instruct the moving module to move at least part of the local addresses in the local list to the local list sub-cache space;
[0129] Obtaining a second local list pointer in the local list sub-cache space, and caching the second local list pointer, wherein the second local list pointer is used to determine a next local list;
[0130] Acquire a second interrupt signal from the moving module, and determine second completion information of the first local list moving command according to the second interrupt signal;
[0131] Generate a first host list moving command including a first host list pointer, convert the format of the first host list moving command according to a data receiving format of the moving module, send the converted first host list moving command to the moving module, and instruct the moving module to move at least part of the host addresses in the host list to the host list sub-cache space;
[0132] Obtaining a second host list pointer in the host list sub-cache space, and caching the second host list pointer, wherein the second host list pointer is used to determine a next host list;
[0133] A third interrupt signal is obtained from the migration module, and third completion information of the first host list migration command is determined according to the third interrupt signal.
[0134] Specifically, the list cache space includes a local list sub-cache space and a host list sub-cache space. The local list sub-cache space is a cache space in a local list unit, and the host list sub-cache space is a cache space in a host list unit.
[0135] The list transfer conversion unit generates a first local list transfer command including a first local list pointer, and uses the first local list transfer command to transfer the host addresses in the host list to the local list sub-cache space step by step. The list transfer conversion unit sends the first local list transfer command to the list transfer task management submodule.
[0136] The list move task management submodule is responsible for receiving the first local list move command generated by the list move conversion unit in each task processing submodule, and processing the first local list move command into a format acceptable to the move module, and sending it to the move module for processing, including: format conversion of the first local list move command according to the data receiving format of the move module, sending the converted first local list move command to the move module, instructing the move module to move at least part of the local addresses in the local list to the local list sub-cache space.
[0137] The list migration task management submodule obtains the second local list pointer in the local list sub-cache space and caches the second local list pointer, which is used to determine the next local list. Figure 3 As shown, the local address in the first local list is moved to the local list sub-cache space of the local list unit. The last bit of the local list sub-cache space stores the next local list pointer, i.e., the second local list pointer. Figure 3 On the local list migration side, step 1 indicates that the list migration task management submodule obtains the second local list pointer from the last bit of the local list sub-cache space, and backs up the second local list pointer at the interface and caches it in the buffer area (Buffer) of the host list unit. Buffer is a memory composed of RAM (Random Access Memory).
[0138] The list migration task management submodule obtains the second interrupt signal from the migration module, and determines the second completion information of the first local list migration command according to the second interrupt signal. Figure 2 As shown, the list migration task management submodule obtains the first local list migration command from the list migration conversion unit in the task processing submodule by means of Dma_wr. After the format of the first local list migration command is converted according to the data receiving format of the migration module, the data in the converted first local list migration command is sent to the migration module. An interrupt signal is obtained from the migration module, and the completion information of the first local list migration command is determined according to the interrupt signal.
[0139] The list transfer conversion unit generates a first local list transfer command including a first local list pointer, and uses the first local list transfer command to transfer the host addresses in the host list to the local list sub-cache space step by step. The list transfer conversion unit sends the first local list transfer command to the list transfer task management submodule.
[0140] The list migration task management submodule is responsible for receiving the first host list migration command generated by the list migration conversion unit in each task processing submodule, and processing the first host list migration command into a format acceptable to the migration module, and sending it to the migration module for processing, including: format conversion of the first host list migration command according to the data receiving format of the migration module, sending the converted first host list migration command to the migration module, instructing the migration module to move at least part of the host addresses in the host list to the host list sub-cache space.
[0141] The list migration task management submodule obtains the second host list pointer in the host list sub-cache space and caches the second host list pointer, which is used to determine the next host list. Figure 3 As shown, the host address in the first host list is moved to the host list sub-cache space of the host list unit. The last bit of the host list sub-cache space stores the next host list pointer, i.e., the second host list pointer. Figure 3 On the host list migration side, step 1 indicates that the list migration task management submodule obtains the second host list pointer from the last bit of the host list sub-cache space, and backs up the second host list pointer at the interface and caches it in the buffer of the host list unit.
[0142] The list migration task management submodule obtains the second interrupt signal from the migration module, and determines the second completion information of the first host list migration command according to the second interrupt signal. Figure 2 As shown, the list migration task management submodule obtains the first host list migration command from the list migration conversion unit in the task processing submodule by means of Dma_wr. After the first host list migration command is format-converted according to the data receiving format of the migration module, the data in the converted first host list migration command is sent to the migration module. An interrupt signal is obtained from the migration module, and the completion information of the first host list migration command is determined according to the interrupt signal.
[0143] In this embodiment, a first host list move command and a first local list move command are generated to instruct the move module to move at least part of the local addresses in the local list to the local list sub-cache space, and to move at least part of the host addresses in the host list to the host list sub-cache space, so as to facilitate subsequent data movement. In addition, a second local list pointer and a second host list pointer are obtained and cached, so as to facilitate the subsequent movement of the local addresses of the next local list and the host addresses of the next host list.
[0144] In some optional implementations, when there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, moving at least part of the local addresses of the next local list and at least part of the host addresses of the next host list to the list cache space includes:
[0145] Determine whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold;
[0146] If there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, generating a second local list move command including a preset number and a second local list pointer;
[0147] Convert the format of the second local list transfer command according to the data receiving format of the transfer module, send the converted second local list transfer command to the transfer module, and instruct the transfer module to transfer a preset number of local addresses in the next local list to the local list sub-cache space;
[0148] Acquire a fourth interrupt signal from the moving module, and determine fourth completion information of the second local list moving command according to the fourth interrupt signal;
[0149] Generate a second host list move command including a preset number and a second host list pointer;
[0150] Convert the format of the second host list transfer command according to the data receiving format of the transfer module, send the converted second host list transfer command to the transfer module, and instruct the transfer module to transfer a preset number of host addresses in the next host list to the host list sub-cache space;
[0151] Acquire a fifth interrupt signal from the migration module, and determine fifth completion information of the second host list migration command according to the fifth interrupt signal;
[0152] Starting from determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, executing subsequent steps, until the next local list and the next host list are both empty, then terminating the moving process of the next local list and the next host list, determining whether a third local list pointer can be read from the local list sub-cache space, and determining whether a third host list pointer can be read from the host list sub-cache space;
[0153] If the third local list pointer and the third host list pointer can be read, the third local list pointer is used as the second local list pointer, and the third host list pointer is used as the second host list pointer, and subsequent steps are performed from determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, until the local address and host address are moved.
[0154] Specifically, the traditional NVMe address list needs to find the pointer of the last level list each time before the next list can be moved. When moving data, addresses will be extracted from the list and the addresses in the list will be consumed. After each consumption of the current list, it is necessary to wait for a period of time before the addresses in the next list can be moved. Data movement also requires waiting for a period of time, which affects the efficiency of data movement and is not conducive to performance improvement. In this embodiment, when the local addresses and host addresses in the list cache space are consumed to a number less than the first preset threshold, at least part of the local addresses of the next local list and at least part of the host addresses of the next host list are started to be moved, so that each data move can obtain local addresses and host addresses from the list cache space. Improve the efficiency of data movement of the entire system. The first preset threshold is, for example, half of the total number of local addresses in the local list, or other values that meet actual needs.
[0155] It is determined whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold.
[0156] If there are local addresses and host addresses in the list cache space and the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, the list move conversion unit generates a second local list move command containing a preset number and a second local list pointer. The preset number is, for example, half of the total number of local addresses in the local list, or other values that meet actual needs. The host addresses in the next host list are moved to the local list sub-cache space step by step using the second local list move command. The list move conversion unit sends the second local list move command to the list move task management submodule. The list move task management submodule is responsible for receiving the second local list move command generated by the list move conversion unit in each task processing submodule, and processing the second local list move command into a format acceptable to the move module, and sending it to the move module for processing, instructing the move module to migrate the preset number of local addresses in the next local list to the local list sub-cache space. The list move task management submodule obtains a fourth interrupt signal from the move module, and determines the fourth completion information of the second local list move command according to the fourth interrupt signal.
[0157] The above process is as follows Figure 3 As shown, Figure 3 Local list migration side, Figure 3 Step 3 indicates determining the next local list according to the local list pointer, and migrating a preset number of local addresses in the next local list to the local list sub-cache space, such as Figure 3 As shown, the local address of the next local list is moved to the local list sub-cache space above the local address of the first local list. At the same time, the data moving module continues to obtain the local address from the local list sub-cache space to perform data moving operations. Steps 4 and 5 also indicate that while the address is being moved, the data moving module continues to obtain the local address from the local list sub-cache space to perform data moving operations.
[0158] The list transfer conversion unit generates a second host list transfer command including a preset number and a second host list pointer. The preset number is, for example, half of the total number of host addresses in the host list, or other values that meet actual needs. The second host list transfer command is used to gradually transfer the host addresses in the next host list to the host list sub-cache space. The list transfer conversion unit sends the second host list transfer command to the list transfer task management submodule. The list transfer task management submodule is responsible for receiving the second host list transfer command generated by the list transfer conversion unit in each task processing submodule, and processing the second host list transfer command into a format acceptable to the transfer module, and sending it to the transfer module for processing, instructing the transfer module to transfer the preset number of host addresses in the next host list to the host list sub-cache space. The list transfer task management submodule obtains a fourth interrupt signal from the transfer module, and determines the fourth completion information of the second host list transfer command according to the fourth interrupt signal.
[0159] The above process is as follows Figure 3 As shown, Figure 3 Host list migration side, Figure 3 Step 3 in the above example indicates determining the next host list according to the host list pointer, and migrating a preset number of host addresses in the next host list to the host list sub-cache space. Figure 3 As shown, the host address of the next host list is moved to the host list sub-cache space above the host address of the first host list. At the same time, the data moving module continues to obtain the host address from the host list sub-cache space to perform data moving operations. Steps 4 and 5 also indicate that while the address is being moved, the data moving module continues to obtain the host address from the host list sub-cache space to perform data moving operations.
[0160] Re-execute the step of "determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold" and subsequent steps, and continue to move the local addresses in the next local list and the host addresses in the next host list until the next local list and the next host list are both empty, then end the process of moving the next local list and the next host list. Determine whether the third local list pointer can be read from the local list sub-cache space, and determine whether the third host list pointer can be read from the host list sub-cache space. If the third local list pointer and the third host list pointer can be read, it means that the third local list and the third host list need to be moved. It is necessary to repeat the above list moving operation, use the third local list pointer as the second local list pointer, and use the third host list pointer as the second host list pointer, and re-execute the step of "determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold" and subsequent steps until the local address and host address are moved. Figure 3 As shown, step 5 indicates that after the address migration is completed, the data migration module will continue to obtain the host address from the host list sub-cache space and the local address from the local list sub-cache space to perform data migration operations. When all list addresses are consumed, the entire data migration task is completed.
[0161] In this embodiment, when there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, a preset number of local addresses in the next local list are migrated to the local list sub-cache space, and a preset number of host addresses in the next host list are migrated to the host list sub-cache space so that the local address and the host address can be obtained from the list cache space each time data is moved, thereby ensuring that the address information in the list cache space is continuous, the data movement process is continuous and uninterrupted, and high-speed operation of the system can be achieved, thereby improving the efficiency of data movement of the entire system.
[0162] In some optional embodiments, the method further comprises:
[0163] Obtaining first completion information, second completion information, third completion information, fourth completion information, and fifth completion information;
[0164] Acquire process information when generating a migration command, wherein the migration command includes: a data migration command, a first local list migration command, a second local list migration command, a first host list migration command, and a second host list migration command;
[0165] Generate total information to be reported including first completion information, second completion information, third completion information, fourth completion information, fifth completion information and process information, and upload the total information to be reported to the task issuing module, wherein the task issuing module is used to issue data migration tasks.
[0166] Specifically, the task aggregation processing unit in the task processing submodule aggregates the completion information of the move command and the process information when generating the move command, and sends the completion information and constant information to the response reporting submodule. The move command includes a data move command, a first local list move command, a second local list move command, a first host list move command, and a second host list move command. The completion information includes a first completion information, a second completion information, a third completion information, a fourth completion information, and a fifth completion information, which are respectively the completion information of the above-mentioned move commands including: a data move command, a first local list move command, a second local list move command, a first host list move command, and a second host list move command.
[0167] The response reporting submodule is responsible for obtaining the completion information and process information sent by the task aggregation processing unit in all task processing submodules, and sending the completion information and process information to the bus processing node. In addition, the response reporting submodule can obtain the response information of the task parsing submodule from the task parsing submodule through the cb (callback, callback) _resp (response, response) method.
[0168] The bus processing node is responsible for receiving the completion information and process information sent by the response reporting submodule in all task processing modules, and converting the completion information and process information into AXI protocol to send to the upper engine in the task issuing module for processing.
[0169] In this embodiment, the total information to be reported, including the completion information of the move command and the process information when the move command is generated, is uploaded to the task issuing module, so that the task issuing module can monitor the entire data migration process and handle fault problems in the data migration process in a timely manner.
[0170] In this embodiment, a data migration system is provided. Figure 6 As shown, the system includes: a task issuing module, a task receiving and processing node, and a first preset number of task processing modules;
[0171] The task sending module is connected to the task receiving and processing node, and is used to send the data migration task to the task receiving and processing node;
[0172] The task receiving processing node is connected to the task processing module, and is used to obtain task information according to the data migration task, determine a target task processing module from a first preset number of task processing modules, and send the task information to the target task processing module;
[0173] The target task processing module is used to execute the data migration method of step S101 to step S104 or any corresponding implementation method thereof.
[0174] Specifically, Figure 4 As shown, the data migration system includes: a task issuing module, a task receiving and processing node, and a first preset number of task processing modules.
[0175] The task sending module is connected to the task receiving and processing node, and sends the data moving task to the task receiving and processing node.
[0176] The task receiving processing node is connected to a first preset number of task processing modules. The first preset number is set according to actual needs, and no specific number limit is made here. The task receiving processing node reads the data moving task into the local cache through the AXI read operation to obtain the data moving task. Read the task information according to the data moving task. Check which task processing module is idle among the first preset number of task processing modules, take the idle task processing module as the target task processing module, and send the task information to the target task processing module.
[0177] The target task processing module is used to execute the data migration method of steps S101 to S104 or any corresponding implementation method thereof. For details, please refer to the above embodiments and will not be repeated here.
[0178] The data migration system provided in this embodiment adopts a processing mechanism of interactive concurrent continuous uninterrupted list migration and data migration, performs data migration at the same time as address migration, extracts the target local address and the target host address from the list cache space, and moves the data in the target local address to the target local address, thereby improving the efficiency of data migration. The system adopts a processing mechanism of interactive concurrent continuous uninterrupted host list migration and data migration, and starts the next list migration when the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, ensuring uninterrupted data migration, improving data migration efficiency, and improving chip performance. The system solves the problem that the scheduling of data migration consumes a lot of time, resulting in low data migration efficiency and failure to meet the high performance requirements of the chip.
[0179] In some optional embodiments, the system further comprises: a moving module, the task processing module comprises: a task parsing submodule, a data moving task management submodule, a response reporting submodule, a list moving task management submodule and a second preset number of task processing submodules;
[0180] The task parsing submodule is connected to the task processing submodule, and is used to obtain the first local list pointer and the first host list pointer in the task information, determine the target task processing submodule from the second preset number of task processing submodules, and send the first local list pointer and the first host list pointer to the target task processing submodule;
[0181] The target task processing submodule is used to generate a first local list moving command including a first local list pointer and a first host list moving command including a first host list pointer, and send the first local list moving command and the first host list moving command to the list moving task management submodule;
[0182] The list migration task management submodule is used to perform format conversion on the first local list migration command and the first host list migration command according to the data receiving format of the migration module, send the converted first local list migration command to the migration module, instruct the migration module to migrate at least part of the local addresses in the local list to the list cache space, and send the converted first host list migration command to the migration module, instructing the migration module to migrate at least part of the host addresses in the host list to the list cache space;
[0183] The target task processing submodule is further used to extract the target local address and the target host address from the list cache space, use the target local address as the destination address, use the target host address as the source address, and determine the offset between the destination address and the source address and the data size of the data to be moved, and when the data size is less than a second preset threshold, generate a data moving command including the destination address, the source address, the offset and the data size, and send the data moving command to the data moving task management submodule;
[0184] The data migration task management submodule is used to convert the format of the data migration command according to the data receiving format of the migration module, send the converted data migration command to the migration module, and instruct the migration module to move the data to be migrated from the target host address to the target local address;
[0185] The response reporting submodule is used to obtain the completion information of the move command and the process information when generating the move command, generate the total information to be reported including the completion information and the process information, and upload the total information to be reported to the task issuing module, wherein the move command includes the first local list move command, the first host list move command and the data move command.
[0186] Specifically, Figure 4 As shown, the data migration system also includes: a migration module, and the task processing module includes: a task analysis sub-module, a data migration task management sub-module, a response reporting sub-module, a list migration task management sub-module and a second preset number of task processing sub-modules, the second preset number is n, which means multiple, and no specific number limit is made here.
[0187] The task parsing submodule is connected to the task processing submodule. The task parsing submodule parses the task information according to the system scenario, and obtains the source address first pointer, i.e., the first host list pointer, the destination address first pointer, i.e., the first local list pointer, the amount of data moved, the control mode, the reporting content, etc. in the task information. After the task information is parsed, it is checked which task processing submodule is idle among the second preset number of task processing submodules, and the idle task processing submodule is used as the target task processing submodule, and the first local list pointer and the first host list pointer are sent to the target task processing submodule.
[0188] The target task processing submodule is used to generate a first local list moving command including a first local list pointer and a first host list moving command including a first host list pointer, and send the first local list moving command and the first host list moving command to the list moving task management submodule.
[0189] The list migration task management submodule converts the format of the first local list migration command and the first host list migration command according to the data receiving format of the migration module, sends the converted first local list migration command to the migration module, instructs the migration module to move the local addresses in the local list to the list cache space, and sends the converted first host list migration command to the migration module, instructs the migration module to move the host addresses in the host list to the list cache space. The data receiving format is, for example, JSON (JavaScript Object Notation) format, CSV (Comma-Separated Values) format, etc., wherein the JSON format is a lightweight data exchange format, and the CSV format is a general table data format.
[0190] The target task processing submodule is used to extract the target local address and the target host address from the list cache space, use the target local address as the destination address, use the target host address as the source address, and determine the offset between the destination address and the source address and the data size of the data to be moved. If the size of the data moved at one time is too large, it will cause each move to take too long, and if it is too small, it will waste computing resources. Therefore, the data move command of this embodiment is a source address and destination address command that does not span 4KB, that is, the data size of the data to be moved does not exceed 4KB, and the second preset threshold is 4KB. The size of the second preset threshold can be modified according to actual needs. In the case where the data size is less than the second preset threshold, a data move command including the destination address, source address, offset and data size is generated, and the data move command is sent to the data move task management submodule.
[0191] The data migration task management submodule converts the format of the data migration command according to the data receiving format of the migration module, and sends the converted data migration command to the migration module. At this time, the migration module can receive and identify the converted data migration command, and according to the converted data migration command, move the data to be moved from the target host address to the target local address.
[0192] The response reporting submodule is used to obtain the completion information of the move command and the process information when the move command is generated, generate the total information to be reported including the completion information and the process information, and upload the total information to be reported to the task issuing module.
[0193] In this embodiment, the system utilizes a migration module, a task analysis submodule, a data migration task management submodule, a response reporting submodule, and a list migration task management submodule corresponding to a second preset number of task processing submodules, which can ensure that the chip can efficiently migrate data while minimizing the chip area and reducing the cost of the chip.
[0194] Figure 5 It is a schematic diagram of the structure of the traditional task processing module. Figure 4 and Figure 5 By comparing the task processing module in this embodiment with the traditional task processing module, it can be seen that the scheme of the present invention adopts a hierarchical concurrency mechanism, separates the units that consume less time in a single stage and have little impact on performance from the subtask processing engine, and sets them into the task processing engine. Its overall execution efficiency does not change much from the internal settings. When the number of subtask processing engines increases, the number of instantiated unit modules will be greatly reduced. Compared with the full concurrency scheme, this scheme can reduce the area consumption of (n-1)*(data movement task management unit + response reporting unit + list movement task management unit + task parsing unit) while achieving the same functions and performance.
[0195] Figure 6 This is a comparison chart of the execution time of the single data migration task of this application and the execution time of the single data migration task of the traditional solution. Figure 6 As shown, the traditional solution uses a serial mechanism to process the host list and the local list, that is, each time the data migration module consumes the last list address, a new list migration is performed, and the overall execution cannot be completely parallelized, and its single task execution time = t1 + t2 + t3 + t4. This solution uses a fully concurrent mechanism to process the host list and the local list. When the data migration module consumes half of the list address, it starts the migration of the next list task, which can ensure that the list space address is continuous. In this way, the host execution does not need to be interrupted due to waiting for the list address, and the list migration and data migration can be executed concurrently all the time, and its single task execution time = t1 + t2 + t4. From the above analysis, it can be seen that the single task execution time of this solution is t3 times faster than that of the traditional solution, and its task execution efficiency is better.
[0196] Figure 7 This is a diagram of the hierarchical concurrent structure of the data migration method. The overall concurrent flow consists of 4 concurrent tasks in the parsing task phase, 16 concurrent tasks in the data migration phase, and 4 concurrent tasks in the response reporting phase. The concurrency granularity is configurable according to the execution time of each stage of the system.
[0197] Figure 8This is a comparison chart of the multi-task execution efficiency of the present application scheme and the traditional scheme, which compares the time required for the traditional scheme and the present scheme in the task parsing stage, data movement stage, response reporting stage and overall task execution speed stage. The traditional scheme is a serial mechanism, and the present scheme is a multi-level concurrent mechanism. Now we will briefly describe each execution stage. In the task parsing stage, this scheme adopts a multi-concurrency mechanism, and its concurrency is 4 for example. The execution time of this scheme is t6=t5 / 4. In the data movement stage, this scheme adopts a multi-concurrency mechanism, and its concurrency is 16 for example. The execution time of this scheme is t8=t7 / 16. In the response reporting stage, this scheme adopts a multi-concurrency mechanism, and its concurrency is 4 for example. The execution time of this scheme is t10=t9 / 4. The time consumed by the same number of tasks is t12=t11 / 16 of the execution time of this scheme.
[0198] From the above analysis, it can be seen that this solution is superior to the traditional solution in both single-task execution performance and multi-task execution performance.
[0199] In this embodiment, a data migration high-speed processing control method is provided, which can solve the same technical problems as steps S101 to S104 and produce the same technical effects, such as Fig. 9 As shown, the process includes the following steps:
[0200] Start; the upper processing engine sends the address information of the task to the task receiving processing unit; the task receiving processing unit reads the task information from the storage space according to the received address information. After receiving the task information, the task processing engine distributes the task to the lower-level task processing engine; the task parsing unit performs command parsing after receiving the task, reads the local list header pointer, host list header pointer, and transfer length and other information, and distributes it to the lower-level subtask processing engine; the task management unit receives the task issued by the task parsing unit, and manages the execution of the data transfer conversion unit and the list transfer conversion unit; after receiving the start command, the list transfer conversion unit synchronously and concurrently starts the mechanism of reading the local list and the host list, and the local list is executed through the list request bus. The host list is processed by the list transfer task management unit; the list transfer task management unit is mainly responsible for the management of the host column transfer command, which is based on RR (Round Robin, round-robin) polling arbitration method, is responsible for efficiently receiving the list migration conversion migration tasks of each subtask engine, and recombining them internally to generate streamlined control information and send them to the lower-level engine for execution; the local list unit caches the list read from the local; the host list unit caches the list read from the host; the data migration conversion unit reads the host address and the local address from the local list unit and the host list unit respectively, splits and reorganizes them internally according to whether they are offset, and generates data migration control commands to be executed in the data migration management unit; the data migration task management unit is mainly responsible for the management of data migration commands, which is based on RR polling arbitration It is responsible for receiving the moving tasks of the data moving and conversion unit of each subtask engine in an efficient manner, and recombining it internally to generate streamlined control information and send it to the lower-level engine for execution; the task aggregation processing unit is responsible for unifying the completion information of the data conversion unit, the list conversion unit and the task management unit and sending it to the response reporting unit for processing; the response reporting unit receives the aggregated task information of the task processing engine according to the RR polling arbitration mechanism, and then classifies, judges and assembles it internally, and sends it to the bus processing unit for processing according to different purposes; the bus processing unit processes the information received from the task processing engine and converts it into the AXI protocol and sends it to the upper-level engine for processing; end.
[0201] Specifically, the upper processing engine sends the task address information to the task receiving processing node. The task receiving processing node reads the task information according to the address and distributes it to n task processing engines. The task parsing submodule parses the source address first pointer, the destination address first pointer, the amount of data to be moved, the control mode and the reporting content, and sends it to the n subtask processing engines. The list migration conversion unit starts the host list and local list migration. When the list unit has the address information, it starts the data migration conversion unit synchronously to move the data. The data migration command of the multi-subtask processing engine will enter the data migration task management submodule, process the command into a format that the lower module can receive, and send it to the lower engine for processing. The list migration command of the multi-subtask processing engine will enter the list migration task management submodule, process the command into a format that the lower module can receive, and send it to the lower engine for processing. When the task migration is completed, all subtask processing engines send the information to be reported to the lower processing engine through the response reporting submodule.
[0202] In this embodiment, by utilizing its highly configurable level-by-level concurrency mechanism, the high performance of the chip can be ensured while minimizing the chip area and reducing the chip cost to the maximum extent; by utilizing its host list migration and data migration interactive concurrent continuous and uninterrupted processing mechanism, the chip performance is further greatly improved. In the current situation of significant competition, more efficient and lower-cost chip design is becoming increasingly important to seize the opportunity to occupy a larger market. The design concept of the present invention is described in detail below.
[0203] In this embodiment, a data migration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0204] This embodiment provides a data migration device, such as Fig.10 As shown, including:
[0205] The acquiring unit 1001 is used to acquire, when task information of the data migration task is acquired, a first local list pointer and a first host list pointer in the task information, wherein the first local list pointer is used to determine a local list including a plurality of local addresses, and the first host list pointer is used to determine a host list including a plurality of host addresses;
[0206] A first moving unit 1002 is used to move at least part of the local addresses in the local list and at least part of the host addresses in the host list to the list cache space according to the first local list pointer and the first host list pointer;
[0207] A second moving unit 1003 is used to move at least part of the local addresses of the next local list and at least part of the host addresses of the next host list to the list cache space when there are local addresses and host addresses in the list cache space and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold;
[0208] The third moving unit 1004 is used to extract the target local address and the target host address from the list cache space while moving the address, and move the data in the target local address to the target local address.
[0209] In some optional implementations, the third moving unit 1004 extracts the target local address and the target host address from the list cache space, and moves the data in the target local address to the target local address, including:
[0210] Extract the target local address and the target host address from the list cache space;
[0211] The target local address is used as the destination address, the target host address is used as the source address, and the offset between the destination address and the source address and the data size of the data to be moved are determined;
[0212] When the data size is smaller than a second preset threshold, generating a data move command including a destination address, a source address, an offset and a data size;
[0213] The data migration command is sent to the migration module, instructing the migration module to move the data to be migrated from the target host address to the target local address.
[0214] In some optional implementations, the third moving unit 1004 sends a data moving command to the moving module, instructing the moving module to move the data to be moved from the target host address to the target local address, including:
[0215] Convert the data transfer command according to the data receiving format of the transfer module, and send the converted data transfer command to the transfer module, instructing the transfer module to transfer the data to be transferred from the target host address to the target local address;
[0216] A first interrupt signal is obtained from the migration module, and first completion information of the data migration command is determined according to the first interrupt signal.
[0217] In some optional implementations, the list cache space includes a local list sub-cache space and a host list sub-cache space; the first moving unit 1002 moves at least part of the local addresses in the local list and at least part of the host addresses in the host list to the list cache space according to the first local list pointer and the first host list pointer, including:
[0218] Generate a first local list moving command including a first local list pointer, convert the format of the first local list moving command according to a data receiving format of the moving module, send the converted first local list moving command to the moving module, and instruct the moving module to move at least part of the local addresses in the local list to the local list sub-cache space;
[0219] Obtaining a second local list pointer in the local list sub-cache space, and caching the second local list pointer, wherein the second local list pointer is used to determine a next local list;
[0220] Acquire a second interrupt signal from the moving module, and determine second completion information of the first local list moving command according to the second interrupt signal;
[0221] Generate a first host list moving command including a first host list pointer, convert the format of the first host list moving command according to a data receiving format of the moving module, send the converted first host list moving command to the moving module, and instruct the moving module to move at least part of the host addresses in the host list to the host list sub-cache space;
[0222] Obtaining a second host list pointer in the host list sub-cache space, and caching the second host list pointer, wherein the second host list pointer is used to determine a next host list;
[0223] A third interrupt signal is obtained from the migration module, and third completion information of the first host list migration command is determined according to the third interrupt signal.
[0224] In some optional implementations, the second moving unit 1003 moves at least part of the local addresses of the next local list and at least part of the host addresses of the next host list to the list cache space when the list cache space has local addresses and host addresses, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, including:
[0225] Determine whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold;
[0226] If there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, generating a second local list move command including a preset number and a second local list pointer;
[0227] Convert the format of the second local list transfer command according to the data receiving format of the transfer module, send the converted second local list transfer command to the transfer module, and instruct the transfer module to transfer a preset number of local addresses in the next local list to the local list sub-cache space;
[0228] Acquire a fourth interrupt signal from the moving module, and determine fourth completion information of the second local list moving command according to the fourth interrupt signal;
[0229] Generate a second host list move command including a preset number and a second host list pointer;
[0230] Convert the format of the second host list transfer command according to the data receiving format of the transfer module, send the converted second host list transfer command to the transfer module, and instruct the transfer module to transfer a preset number of host addresses in the next host list to the host list sub-cache space;
[0231] Acquire a fifth interrupt signal from the migration module, and determine fifth completion information of the second host list migration command according to the fifth interrupt signal;
[0232] Starting from determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, executing subsequent steps, until the next local list and the next host list are both empty, then terminating the moving process of the next local list and the next host list, determining whether a third local list pointer can be read from the local list sub-cache space, and determining whether a third host list pointer can be read from the host list sub-cache space;
[0233] If the third local list pointer and the third host list pointer can be read, the third local list pointer is used as the second local list pointer, and the third host list pointer is used as the second host list pointer, and subsequent steps are performed from determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, until the local address and host address are moved.
[0234] In some optional embodiments, the device is used to:
[0235] Obtaining first completion information, second completion information, third completion information, fourth completion information, and fifth completion information;
[0236] Acquire process information when generating a migration command, wherein the migration command includes: a data migration command, a first local list migration command, a second local list migration command, a first host list migration command, and a second host list migration command;
[0237] Generate total information to be reported including first completion information, second completion information, third completion information, fourth completion information, fifth completion information and process information, and upload the total information to be reported to the task issuing module, wherein the task issuing module is used to issue data migration tasks.
[0238] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0239] The data migration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0240] The embodiment of the present invention also provides a computer device having the above Fig.10 The data moving device shown.
[0241] See also Fig.11 , Fig.11 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.11 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 10 is taken as an example.
[0242] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0243] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0244] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0245] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0246] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0247] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0248] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A data migration method, characterized in that: The method is applied to a task processing module, and the method comprises: When task information of the data migration task is obtained, a first local list pointer and a first host list pointer in the task information are obtained, wherein the first local list pointer is used to determine a local list including multiple local addresses, and the first host list pointer is used to determine a host list including multiple host addresses; According to the first local list pointer and the first host list pointer, move at least part of the local addresses in the local list and at least part of the host addresses in the host list to the list cache space; If there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, at least part of the local addresses of the next local list and at least part of the host addresses of the next host list are moved to the list cache space; While the address is being moved, the target local address and the target host address are extracted from the list cache space, and the data in the target local address is moved to the target local address.
2. The method according to claim 1, characterized in that The step of extracting the target local address and the target host address from the list cache space and moving the data in the target local address to the target local address includes: Extracting the target local address and the target host address from the list cache space; Taking the target local address as the destination address and the target host address as the source address, and determining the offset between the destination address and the source address and the data size of the data to be moved; When the data size is smaller than a second preset threshold, generating a data move command including the destination address, the source address, the offset and the data size; The data migration command is sent to the migration module, instructing the migration module to migrate the data to be migrated from the target host address to the target local address.
3. The method according to claim 2, characterized in that The step of sending the data migration command to the migration module to instruct the migration module to migrate the data to be migrated from the target host address to the target local address includes: Convert the data transfer command according to the data receiving format of the transfer module, send the converted data transfer command to the transfer module, and instruct the transfer module to transfer the data to be transferred from the target host address to the target local address; A first interrupt signal is obtained from the migration module, and first completion information of the data migration command is determined according to the first interrupt signal.
4. The method according to claim 1, characterized in that: The list cache space includes a local list sub-cache space and a host list sub-cache space; and the step of moving at least part of the local addresses in the local list and at least part of the host addresses in the host list to the list cache space according to the first local list pointer and the first host list pointer includes: generating a first local list moving command including the first local list pointer, converting the format of the first local list moving command according to a data receiving format of a moving module, and sending the converted first local list moving command to the moving module, instructing the moving module to move at least part of the local addresses in the local list to the local list sub-cache space; Obtaining a second local list pointer in the local list sub-cache space, and caching the second local list pointer, wherein the second local list pointer is used to determine the next local list; Acquire a second interrupt signal from the migration module, and determine second completion information of the first local list migration command according to the second interrupt signal; generating a first host list moving command including the first host list pointer, converting the format of the first host list moving command according to a data receiving format of a moving module, sending the converted first host list moving command to the moving module, and instructing the moving module to move at least part of the host addresses in the host list to the host list sub-cache space; Obtaining a second host list pointer in the host list sub-cache space, and caching the second host list pointer, wherein the second host list pointer is used to determine the next host list; A third interrupt signal is obtained from the migration module, and third completion information of the first host list migration command is determined according to the third interrupt signal.
5. The method according to claim 4, characterized in that The step of moving at least part of the local addresses of a next local list and at least part of the host addresses of a next host list to the list cache space when the list cache space contains local addresses and host addresses, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, comprises: Determine whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold; If there are local addresses and host addresses in the list cache space, and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, generating a second local list move command including a preset number and the second local list pointer; Performing format conversion on the second local list move command according to the data receiving format of the move module, sending the converted second local list move command to the move module, and instructing the move module to migrate the preset number of local addresses in the next local list to the local list sub-cache space; Acquire a fourth interrupt signal from the moving module, and determine fourth completion information of the second local list moving command according to the fourth interrupt signal; generating a second host list moving command including a preset number and a pointer to the second host list; Performing format conversion on the second host list moving command according to the data receiving format of the moving module, sending the converted second host list moving command to the moving module, and instructing the moving module to migrate the preset number of host addresses in the next host list to the host list sub-cache space; Acquire a fifth interrupt signal from the migration module, and determine fifth completion information of the second host list migration command according to the fifth interrupt signal; Starting from the step of determining whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold, the subsequent steps are executed until the next local list and the next host list are both empty, then the moving process of the next local list and the next host list is terminated, and it is determined whether a third local list pointer can be read from the local list sub-cache space, and whether a third host list pointer can be read from the host list sub-cache space; If the third local list pointer and the third host list pointer can be read, the third local list pointer is used as the second local list pointer, and the third host list pointer is used as the second host list pointer, and subsequent steps are executed from the judgment whether there are local addresses and host addresses in the list cache space, and whether the number of local addresses and the number of host addresses in the list cache space are both less than the first preset threshold, until the local address and the host address are moved.
6. The method according to claim 1, characterized in that The method further comprises: Obtaining first completion information, second completion information, third completion information, fourth completion information, and fifth completion information; Acquire process information when generating a migration command, wherein the migration command includes: a data migration command, a first local list migration command, a second local list migration command, a first host list migration command, and a second host list migration command; Generate total information to be reported including the first completion information, the second completion information, the third completion information, the fourth completion information, the fifth completion information and the process information, and upload the total information to be reported to the task issuing module, wherein the task issuing module is used to issue the data migration task.
7. A data migration system, characterized in that: The system comprises: a task issuing module, a task receiving and processing node and a first preset number of task processing modules; The task sending module is connected to the task receiving and processing node, and is used to send the data migration task to the task receiving and processing node; The task receiving processing node is connected to the task processing module, and is used to obtain task information according to the data migration task, determine a target task processing module from the first preset number of task processing modules, and send the task information to the target task processing module; The target task processing module is used to execute the data migration method described in any one of claims 1 to 6.
8. The system according to claim 7, characterized in that The system further comprises: a moving module, wherein the task processing module comprises: a task parsing submodule, a data moving task management submodule, a response reporting submodule, a list moving task management submodule and a second preset number of task processing submodules; The task parsing submodule is connected to the task processing submodule, and is used to obtain the first local list pointer and the first host list pointer in the task information, determine the target task processing submodule from the second preset number of task processing submodules, and send the first local list pointer and the first host list pointer to the target task processing submodule; The target task processing submodule is used to generate a first local list moving command including the first local list pointer and a first host list moving command including the first host list pointer, and send the first local list moving command and the first host list moving command to the list moving task management submodule; The list migration task management submodule is used to perform format conversion on the first local list migration command and the first host list migration command according to the data receiving format of the migration module, send the converted first local list migration command to the migration module, instruct the migration module to migrate at least part of the local addresses in the local list to the list cache space, and send the converted first host list migration command to the migration module, instructing the migration module to migrate at least part of the host addresses in the host list to the list cache space; The target task processing submodule is further used to extract a target local address and a target host address from the list cache space, use the target local address as the destination address, use the target host address as the source address, and determine the offset between the destination address and the source address and the data size of the data to be moved, and when the data size is less than a second preset threshold, generate a data moving command including the destination address, the source address, the offset and the data size, and send the data moving command to the data moving task management submodule; The data migration task management submodule is used to convert the format of the data migration command according to the data receiving format of the migration module, send the converted data migration command to the migration module, and instruct the migration module to move the data to be migrated from the target host address to the target local address; The response reporting submodule is used to obtain the completion information of the move command and the process information when generating the move command, generate the total information to be reported including the completion information and the process information, and upload the total information to be reported to the task issuing module, wherein the move command includes the first local list move command, the first host list move command and the data move command.
9. A data migration device, characterized in that: The device comprises: an acquiring unit, configured to acquire, when task information of the data migration task is acquired, a first local list pointer and a first host list pointer in the task information, wherein the first local list pointer is used to determine a local list including a plurality of local addresses, and the first host list pointer is used to determine a host list including a plurality of host addresses; A first moving unit, configured to move at least part of the local addresses in the local list and at least part of the host addresses in the host list to a list cache space according to the first local list pointer and the first host list pointer; A second moving unit is used to move at least part of the local addresses of the next local list and at least part of the host addresses of the next host list to the list cache space if there are local addresses and host addresses in the list cache space and the number of local addresses and the number of host addresses in the list cache space are both less than a first preset threshold; The third moving unit is used to extract the target local address and the target host address from the list cache space while the address is being moved, and move the data in the target local address to the target local address.
10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data migration method according to any one of claims 1 to 6 by executing the computer instructions.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data migration method according to any one of claims 1 to 6.
