Data moving method, device and equipment and computer readable storage medium

Abstract

The invention discloses a data moving method, device and equipment and a computer readable storage medium, which are applied to the technical field of data processing, and the method comprises the following steps: determining a target class according to the data type of to-be-moved data; determining a target flow according to the processing flow corresponding to each target class; each stream represents a processing flow which can be processed in parallel by using the same class of processing equipment in one class; determining a processing resource of each processing device corresponding to the target class and the target flow; and after the first processing device processes the streams under the target class in the first processing device according to the allocated processing resources, the streams are directly sent to the next processing device to be directly processed, and after all the processing devices finish processing, target mobile data are obtained. According to the method, the corresponding processing equipment resource is determined according to the target class and the target flow, the data can be directly processed according to the allocated processing resource during data processing, and a memory bus and a peripheral bus do not need to be used for transmission, so that the data moving efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data moving method, device, equipment and computer-readable storage medium. Background Art

[0002] In order to reduce the system memory copy on the IO (input and output) path, many technologies have made contributions in this regard. RDMA (remote direct data access) technology can reduce the number of data copies, but it cannot achieve zero copy. If the data needs to be processed in the DPU (data processor unit) and then stored on the NVMe (a high-performance interface standard designed for non-volatile storage devices) disk. It needs to copy the data from the system memory to the DPU through the memory bus and the peripheral bus. After the DPU processing is completed, it needs to copy the data to the system memory again, and then store the data from the system memory to the NVMe disk. During the whole process, the system memory must pass through the memory bus and the peripheral bus many times, which will seriously affect the performance of the system.

[0003] It can be seen that how to improve the efficiency of data movement is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a data movement method, device, equipment and computer-readable storage medium, which solve the technical problem of slow data movement in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a data movement method, comprising:

[0006] Acquire different types of data to be moved, and determine a target class according to the data type of the data to be moved; wherein each class represents a same type of operation, and each class corresponds to the same processing device;

[0007] Determine a target flow according to the processing flow corresponding to each target class; wherein each flow represents a processing flow in a class that can be processed in parallel using the same type of processing equipment;

[0008] Determine the processing resources of each processing device corresponding to the target class and the target flow;

[0009] The data to be moved is directly sent to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then sends the processed data to be moved directly to the next processing device for direct processing. When all processing devices have completed the processing, the target mobile data is obtained.

[0010] On the one hand, when the target class is a text class, an image class, and a video class, determining the target flow according to the processing flow corresponding to each target class includes:

[0011] Determine a first target flow according to the processing flow corresponding to the text class; wherein the first target flow includes text cleaning, word segmentation processing and vocabulary building;

[0012] Determine a second target stream according to the processing flow corresponding to the image class; wherein the second target stream includes a first format conversion and a first resolution adjustment;

[0013] According to the processing flow corresponding to the video class, a third target stream is determined; wherein the third target stream includes frame extraction, second format conversion, second resolution adjustment and array storage operation; the array storage operation refers to storing all extracted video frames in chronological order in the form of an array; the number of processing devices corresponding to each target stream is determined according to demand.

[0014] On the one hand, after determining the processing resources of each processing device corresponding to the target class and the target flow, the method further includes:

[0015] When it is determined that the processing resources corresponding to the current processing device cannot meet the resource requirements for processing the target class and the target flow, determining to increase processing on the processing device;

[0016] When it is determined that there is an unused processing device in the current processing devices, it is determined to delete the unused processing device.

[0017] On the one hand, determining the processing resources of each processing device corresponding to the target class and the target flow includes:

[0018] Determine the processing device corresponding to the current target class;

[0019] According to the time taken by each processing device to process the flow under the current target class, the processing time ratio corresponding to each processing device is obtained;

[0020] Determine the number of hardware resource shares for each flow under the current target class on each processing device based on the processing time ratio;

[0021] The processing resources of each processing device are determined based on the number of hardware resource shares of each flow under the current target class.

[0022] On the one hand, when the processing device includes a network card, a data processor and a storage device, the data to be moved is directly sent to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then directly sends the processed data to be moved to the next processing device for direct processing. When all processing devices complete the processing, the target mobile data is obtained, including:

[0023] Sending the data to be moved directly to the network card;

[0024] Using the network card according to the processing resources allocated to the network card, the flows under the current target class are processed in parallel to obtain first processed data;

[0025] sending the first processed data directly to the data processor;

[0026] Using the data processor according to the processing resources allocated to the data processor, processing the flow under the current target class in the current data processor to obtain second processed data;

[0027] sending the second processed data to the storage medium; wherein the storage medium is a physical device or medium for recording and storing data;

[0028] The storage medium is used to store the flow under the current target class in the storage medium according to the processing resources allocated to the storage medium, so as to obtain the target mobile data.

[0029] On the one hand, the above data movement method further includes:

[0030] Monitor the entire data movement process, and when it is determined that the data movement is complete, determine whether the movement is successful;

[0031] When the movement is unsuccessful, an error code, an error location and an error cause are determined; wherein the error location includes an error type, an error flow and an error processing device.

[0032] On the one hand, after determining the error code, error location and error cause, it also includes:

[0033] Determine whether there is a corresponding repair strategy for the current problem according to the error location and the error cause;

[0034] When a repair strategy exists, determining to perform repair based on the repair strategy, and when the repair is successful, re-executing the data movement request;

[0035] When no recovery strategy exists, an error code is sent to the user.

[0036] An embodiment of the present invention provides a data movement method, including:

[0037] A target class determination module is used to obtain different types of data to be moved and determine the target class according to the data type of the data to be moved; wherein each class represents a same type of operation and each class corresponds to the same processing device;

[0038] A target flow determination module, used to determine the target flow according to the processing flow corresponding to each target class; wherein each flow represents a processing flow in a class that can be processed in parallel using the same type of processing equipment;

[0039] A processing resource determination module, used to determine the processing resources of each processing device corresponding to the target class and the target flow;

[0040] The target mobile data determination module is used to send the data to be moved directly to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then sends the processed data to be moved directly to the next processing device for direct processing. When all processing devices have completed the processing, the target mobile data is obtained.

[0041] An embodiment of the present invention further provides a data mobile device, including:

[0042] Memory for storing computer programs;

[0043] A processor is used to execute the computer program to implement the steps of the above data moving method.

[0044] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data moving method are implemented.

[0045] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above data moving method when executed by a processor.

[0046] The purpose of the embodiments of the present invention is to provide a data movement method, device, equipment and computer-readable storage medium, which can solve the technical problem of low data movement efficiency.

[0047] To solve the above technical problems, an embodiment of the present invention provides a data movement method which may include:

[0048] Obtain different types of data to be moved, and determine the target class according to the data type of the data to be moved; wherein each class represents a class of identical operations, and each class corresponds to the same processing device; determine the target flow according to the processing flow corresponding to each target class; wherein each flow represents a processing flow in a class that can be processed in parallel using the same class of processing devices; determine the processing resources of each processing device corresponding to the target class and the target flow; send the data to be moved directly to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then sends the processed data to be moved directly to the next processing device for direct processing, and when all processing devices have completed processing, the target mobile data is obtained.

[0049] It can be seen from the above technical scheme that the beneficial effect of the present invention is that compared with the current need to use the memory bus and the peripheral bus for transmission processing, the present invention determines the target class and the target stream corresponding to the data to be moved, and then determines the corresponding processing device resources according to the target class and the target stream. Therefore, when processing data, the data can be directly processed according to the allocated processing device resources, without the need to use the memory bus and the peripheral bus for transmission, thereby reducing resource usage and improving the efficiency of data movement and the performance of the entire storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flow chart of a data movement method provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a class and a flow provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a first type of processing resource allocation provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a second processing resource allocation provided by an embodiment of the present invention;

[0055] Figure 5 An existing data movement process provided by an embodiment of the present invention;

[0056] Figure 6 An example flowchart of a data movement method provided by an embodiment of the present invention;

[0057] Figure 7 An architectural diagram of a data movement method provided by an embodiment of the present invention;

[0058] Figure 8 A schematic diagram of class and flow division provided in an embodiment of the present invention;

[0059] Fig. 9 A schematic diagram of a structure of a processing flow of correctly corresponding data movement provided by an embodiment of the present invention;

[0060] Fig.10 A structural schematic diagram of a processing flow corresponding to a data movement error provided by an embodiment of the present invention;

[0061] Fig.11 A schematic diagram of the structure of a data moving device provided by an embodiment of the present invention;

[0062] Fig.12 A schematic diagram of the structure of a data mobile device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0065] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0066] Next, a flow chart of a data moving method provided by an embodiment of the present invention is described in detail. Figure 1 A data migration method provided in an embodiment of the present invention may include:

[0067] S101, obtaining different types of data to be moved, and determining a target class according to the data type of the data to be moved; wherein each class represents a same type of operation, and each class corresponds to the same processing device.

[0068] The execution subject of this embodiment is an electronic device, which can be a computer, a mobile phone, etc. This embodiment does not limit the specific data to be moved. The data to be moved in this embodiment can be text data; or the data to be moved in this embodiment can also be image data; or the data to be moved in this embodiment can also be video data; or the data to be moved in this embodiment can also be voice data. This embodiment does not limit the specific target class, and different target classes can be divided according to different data types, as long as different types of data to be moved can be distinguished. For example, the target class in this embodiment can be a first target class, a second target class, a third target class, and a fourth target class; or the target class in this embodiment can also be a text class, a data class, etc. The target class (class) in this embodiment represents a class of the same operations, which require the same hardware resources to process and have the same processing flow.

[0069] It should be further explained that the above-mentioned acquisition of different types of data to be moved and determination of the target class according to the data type of the data to be moved may include: acquiring different types of data to be moved in at least one of an AIGC scenario, a distributed file system, a distributed database, a distributed object system, and a distributed block storage. The AIGC (artificial intelligence generated content) scenario in this embodiment covers a variety of applications, including text, audio, image, and video processing. The distributed file system (Distributed File System, DFS) in this embodiment is a file system that stores data in a distributed manner on multiple machines, aiming to improve the reliability, availability, and access efficiency of the system. The distributed database in this embodiment is a database system that stores and manages data distributed on multiple servers, aiming to improve the availability, reliability, and scalability of the system. The distributed database in this embodiment is a database system that stores and manages data distributed on multiple servers, aiming to improve the availability, reliability, and scalability of the system. The distributed object system in this embodiment is a distributed system that organizes and interacts application components in the form of objects, aiming to achieve communication and collaboration between objects through message passing. The distributed block storage in this embodiment is a storage architecture that distributes data in the form of blocks on multiple physical servers, aiming to improve the scalability, reliability, and performance of the system. The data movement method in this embodiment can be applied to a variety of scenarios, thereby improving the diversification of applications.

[0070] S102, determining a target flow according to a processing flow corresponding to each target class; wherein each flow represents a processing flow in a class that can be processed in parallel using a processing device of the same class.

[0071] In this embodiment, the specific target stream is not limited. The target stream (stream) represents a hardware pipeline that can be processed in parallel in a class. For example, the target stream in this embodiment may include text cleaning, word segmentation processing, and vocabulary building; or the target stream in this embodiment may include format conversion and resolution adjustment. The same type of processing equipment in this embodiment refers to the same type of required processing equipment. For example, the processing equipment required for stream 1 is a network card, a DPU, and a storage device, and the processing equipment required for stream 2 is also a network card, a DPU, and a storage device, then it is determined that stream 1 and stream 2 belong to the same class and belong to tasks (processing processes) that can be processed in parallel using the same type of processing equipment; or when the processing equipment required for stream 1 is a network card, a DPU, and a storage device, and the processing equipment required for stream 2 is a network card and a storage device, then it is determined that stream 1 and stream 2 do not belong to the same class and do not belong to tasks that can be processed in parallel using the same type of processing equipment. For ease of understanding, please refer to Figure 2 , Figure 2 A schematic diagram of a class and flow provided by an embodiment of the present invention, from Figure 2 It can be seen that class 1 requires the network card, DPU1, DPU2, and data storage to do corresponding work, so hardware resources need to be allocated from the four of them. Class 1 requires 5 streams, so five hardware resources need to be allocated from the above four hardwares to complete five concurrencies; class 2 requires the network card and data storage to do corresponding work, so hardware resources need to be allocated from two of them. Class 2 requires 3 streams, so three hardware resources need to be allocated from the above two hardwares to complete three concurrencies; class 3 requires the network card, DPU2, and data storage to do corresponding work, so hardware resources need to be allocated from the three of them. Class 3 requires 2 streams, so two hardware resources need to be allocated from the above three hardwares to complete two concurrencies.

[0072] It should be further explained that, when the target class is a text class, an image class, and a video class, determining the target flow according to the processing flow corresponding to each target class may include:

[0073] S1021, determining a first target flow according to the processing flow corresponding to the text class; wherein the first target flow includes text cleaning, word segmentation processing, and vocabulary building;

[0074] S1022, determining a second target stream according to a processing flow corresponding to the image class; wherein the second target stream includes a first format conversion and a first resolution adjustment;

[0075] S1023, determine the third target stream according to the processing flow corresponding to the video class; wherein the third target stream includes frame extraction, second format conversion, second resolution adjustment and array storage operation; the array storage operation refers to storing all extracted video frames in chronological order in the form of an array; the number of processing devices corresponding to each target stream is determined according to demand.

[0076] The text cleaning in this embodiment can be to remove HTML (Hypertext Markup Language) tags, special characters and noise text, and retain pure text data. The word segmentation processing in this embodiment can be to segment the text according to vocabulary units to form a vocabulary sequence. The vocabulary building in this embodiment can be to sort all the words according to frequency and build a vocabulary for converting text into numerical representation. Through data processing, AIGC (Artificial Intelligence Generated Content) can better understand and process input text data in model training. The format conversion in this embodiment can be that there are various image modes. In order to speed up the processing, we unify them into one format, such as JPEG (joint photographic experts group, Joint Photographic Experts Group (static image compression standard)) format. The resolution adjustment in this embodiment can be that considering that the resolution of the image is different, it is not conducive to improving the efficiency of model training, so the resolution of all images is adjusted to the same resolution. The frame extraction in this embodiment can be that three types of frames are defined in the H.264 protocol, the fully encoded frame is called I frame, the frame generated by referring to the previous I frame and only containing the difference part encoding is called P frame, and there is also a frame encoded by referring to the previous and next frames called B frame. The simplest way to extract frames is to find and extract all key frames (I frames). This embodiment stores all extracted frames in chronological order in an array. It is important to store them in order because the training will analyze their internal relationships in this order. This data movement method can be used in different types of data, which improves the comprehensiveness of data movement.

[0077] S103, determining the processing resources of each processing device corresponding to the target class and the target flow.

[0078] This embodiment does not limit the specific scheme for determining the processing resources of each processing device corresponding to the target class and the target flow. For example, this embodiment can determine the processing resources based on the time required for the storage resources to process each flow. For example, if the time spent by the network card, DPU2 and storage resources is 1:1:1, the processing resources of the hardware applied are also 1:1:1.

[0079] It should be further explained that the processing resources of each processing device corresponding to the target class and the target flow may include:

[0080] S1031, determining a processing device corresponding to the current target class;

[0081] S1032, obtaining a processing time ratio corresponding to each processing device according to the time taken by each processing device to process the flow under the current target class;

[0082] S1033 determines the number of hardware resource shares for each flow under the current target class on each processing device based on the processing time ratio;

[0083] S1034: Determine the processing resources of each processing device based on the number of hardware resources of each flow under the current target class.

[0084] This embodiment can determine the processing resources based on the processing time of the flow. For ease of understanding, for example, the number of hardware resources applied is related to the processing time of each step. If the processing time of each step is the same, the number of hardware resources applied will be allocated at a ratio of 1:1. If the time spent by the network card, DPU2 and storage resources is 1:1:1, the hardware resources applied are also 1:1:1. Figure 3 As shown, Figure 3 This is a schematic diagram of the first processing resource allocation provided by an embodiment of the present invention. There are two streams, each with three hardware resources, so six tasks can be processed in parallel. If the time spent by the network card, DPU2, and storage resources is 1:2:1, the hardware resources applied are also 1:2:1. Figure 4 As shown, Figure 4 A schematic diagram of the second processing resource allocation provided by an embodiment of the present invention. It has two streams, each stream has four hardware resources, so a maximum of 8 tasks can be run in parallel. If a class has n components involved in processing, the processing time of each component is T1, T2, ..., Tn, Tmin = min (T1, T2, ..., Tn). Then the number of hardware resources that need to be allocated to the n components is: [T1 / Tmin], [T2 / Tmin], ..., [Tn / Tmin]. This embodiment determines the processing resources of each processing device in advance, so that relevant resources can be reserved, the accuracy of data movement is provided, and it is prevented from being unable to move.

[0085] It should be further explained that the above-mentioned determination of the processing resources of each processing device corresponding to the target class and the target flow may include: determining the processing resources of each processing device corresponding to the target class and the target flow based on the current load resources and the future load resources, as well as the resources required by the target class and the target flow. In this embodiment, the current load resources may be to collect the load data of the current system, including CPU usage, memory occupancy, disk I / O, etc. These data can be obtained in real time through the system monitoring tool. The future load resource prediction in this embodiment may be based on business development trends and market demand, and the load in the future period of time is predicted. This can be done through historical data analysis, market research and expert consultation. This embodiment clarifies the specific needs of the target class and the target flow, including the required computing power, storage space, network bandwidth, etc. This embodiment determines the processing resources based on the current load resources and the future load resources, as well as the resources required by the target class and the target flow, and can provide the accuracy of the processing resource determination, so that the resources of the processing device corresponding to the processing resources can be released in time, or expansion processing can be performed, so that the target class and the target flow can be processed.

[0086] It should be further explained that after determining the processing resources of each processing device corresponding to the target class and the target flow, the above may also include: when it is determined that the processing resources corresponding to the current processing device cannot meet the resource requirements for processing the target class and the target flow, determining to increase the processing of the processing device; when it is determined that there are unused processing devices in the current processing device, determining to delete the unused processing devices. In this embodiment, according to the amount and type of data to be processed, the client and the front-end processing server can be dynamically added and reduced, and the hardware resources inside the front-end processing server can also be dynamically added and reduced. It does not use or uses a very small amount of CPU and system memory, so that the computationally intensive training phase can have exclusive use of them, improving its computing efficiency.

[0087] S104, directly sending the data to be moved to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then directly sends the processed data to be moved to the next processing device for direct processing. When all processing devices have completed the processing, the target mobile data is obtained.

[0088] The first processing device in this embodiment is the first processing device corresponding to the target flow under the target class, which means that the flow under the target class corresponds to multiple processing devices, and the multiple processing devices corresponding to the flow under the target class can be determined according to the processing flow. At this time, the first device that processes the data to be moved can be used as the first processing device, and then the next processing device can be determined. This embodiment does not limit the specific first processing device and other processing devices, as long as these processing devices can process the target class and target flow of the data to be moved. For example, the processing device in this embodiment can be a network card, or the processing device in this embodiment can also be a DPU; or the processing device in this embodiment can also be a storage device; or the processing device in this embodiment can also be a graphics processing unit (GPU) or a field programmable gate array (FPGA).

[0089] It should be further explained that, when the processing device includes a network card, a data processor and a storage device, the data to be moved is directly sent to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then directly sends the processed data to be moved to the next processing device for direct processing. When all processing devices have completed the processing, the target mobile data is obtained, which may include:

[0090] S1041, directly sending the data to be moved to the network card;

[0091] S1042, using the network card according to the processing resources allocated to the network card, performing parallel processing on the flows under the current target class to obtain first processed data;

[0092] S1043, directly sending the first processed data to the data processor;

[0093] S1044, using the data processor to process the flow under the current target class in the current data processor according to the processing resources allocated to the data processor, to obtain second processed data;

[0094] S1045, sending the second processed data to a storage medium; wherein the storage medium is a physical device or medium for recording and storing data.

[0095] S1046: Using the storage medium according to the processing resources allocated to the storage medium, the flow under the current target class in the storage medium is stored to obtain the target mobile data.

[0096] In this embodiment, the data to be moved of the local server can be directly sent to the network card of the remote server, and then sent directly to the DPU by the network card, and then sent to the data storage by the DPU. In this embodiment, the data to be moved is sent directly to the network card: This step refers to transferring the data to be processed directly from its original location (which may be a memory or other storage device) to the network card. The network card (network interface card) is a hardware interface between a computer and a network, responsible for sending and receiving data. Here, it serves as the starting point for data transmission. The network card is used to process the streams under the current target class in parallel according to the processing resources allocated to the network card to obtain the first processed data: After receiving the data, the network card will use the processor resources allocated to it to perform preliminary processing on the data. Parallel processing means that the network card can process multiple data streams at the same time, thereby improving processing efficiency. After the processing is completed, the so-called "first processed data" is generated. The first processed data is sent directly to the data processor: the data (first processed data) preliminarily processed by the network card is sent to the data processor. The data processor is usually a more powerful computing device that can perform more complex data processing tasks. The data processor is used to process the flow under the current target class in the current data processor according to the processing resources allocated to the data processor, and the second processed data is obtained: After the data processor receives the data from the network card, it will use its allocated processor resources to further process the data. This may include data analysis, conversion, encryption / decryption and other operations. After the processing is completed, the second processed data is generated. The second processed data is sent to the storage medium: The data processed by the data processor (the second processed data) is sent to the storage medium for storage. The storage medium can be a physical device or medium such as a hard disk, a solid-state drive, a tape library, etc., which is used to store data for a long time. The storage medium is used to store the flow under the current target class in the storage medium according to the processing resources allocated to the storage medium, and the target mobile data is obtained: After receiving the data, the storage medium will use its allocated resources to perform storage operations on the data. This may include data writing, management and maintenance. Finally, the data movement process is completed and the target mobile data is obtained. In general, the entire process describes a series of steps from the initial transmission of data to the final storage, involving the collaborative work of the network card, the data processor and the storage medium. This process is very common in big data processing, cloud computing, and distributed systems, which aim to efficiently process and store large amounts of data.

[0097] It should be further explained that it can also include: monitoring the entire data movement process, and when it is determined that the data movement is completed, determining whether the movement is successful; when the movement is not successful, determining the error code, error location and error cause; wherein the error location includes error class, error flow and error processing device. In this embodiment, monitoring the entire data movement process is a key step to ensure data security, integrity and efficient transmission. The following is a detailed explanation of monitoring the entire data movement process: Determine monitoring indicators: It is necessary to clarify which indicators are critical to the data movement process. These indicators may include data movement speed, error rate, number of re-movements, network delay, etc. By monitoring these indicators, the status and performance of data movement can be understood in real time. Select monitoring tools: In order to effectively monitor the data movement process, it is necessary to select appropriate monitoring tools. These tools can be professional monitoring software, such as Zabbix (multi-level monitoring and alarm functions), Nagios (supporting multiple monitoring tasks), Prometheus (multi-dimensional data model), etc., or they can be self-built monitoring systems. These tools can provide real-time monitoring, alarm notification and data analysis functions to help timely discover and solve problems. Set monitoring rules: According to business needs and actual conditions, reasonable monitoring rules need to be set. These rules may include threshold settings (such as triggering an alarm when the data transmission speed is lower than a certain value), monitoring frequency (such as how often to check the data status). By setting these rules, you can ensure that you are notified in time when an abnormal situation occurs and take appropriate measures. Real-time monitoring and alarm processing: During the data movement process, it is necessary to continuously monitor the system operation status and handle abnormal situations in a timely manner. Once a problem with data transmission is found or the preset threshold is reached, the monitoring system should immediately trigger an alarm and notify relevant personnel through email, SMS, phone, etc. At the same time, it is necessary to analyze and process the alarm information to find out the cause of the problem and take appropriate measures to repair it. Determine whether the move is successful: When it is determined that the data has been moved, it is necessary to verify whether the data has been successfully moved to the target location. This can be achieved by checking the integrity and correctness of the data in the target storage medium. If the data is complete and correct, the data can be considered to have been moved successfully; otherwise, further troubleshooting and processing are required. Determine the error code, error location, and error cause: If the data has not been moved successfully, the error code, error location, and error cause need to be determined. The error code is a code or number used to identify a specific error; the error location refers to the specific location or link where the error occurred; the error cause is the direct cause or root cause of the error. By analyzing this information, you can more accurately locate the problem and take appropriate solutions. Segmentation of error location: When determining the error location, it can be further subdivided into error class, error flow, and error handling device. The error class refers to the target class where the error occurred; the error flow refers to the target flow where the error occurred; and the error handling device refers to the device involved in the processing.By subdividing the error location, the source of the problem can be more accurately located and a targeted solution can be formulated. This embodiment improves the accuracy of data movement error location by determining the error type, error flow and error processing device.

[0098] It should be further explained that after determining the error code, error location and error cause, the method may further include: determining whether there is a corresponding repair strategy for the current problem according to the error location and error cause; when a repair strategy exists, determining to perform repair based on the repair strategy, and when the repair is successful, re-executing the data movement request; when no repair strategy exists, sending the error code to the user. The error location and error cause in this embodiment can directly establish a corresponding relationship with the repair strategy, and the repair strategy can be determined directly according to the corresponding relationship. This embodiment can directly repair the error location according to the repair strategy, thereby improving the efficiency of repair.

[0099] A data movement method provided by an embodiment of the present invention may include: S101, obtaining different types of data to be moved, and determining a target class according to the data type of the data to be moved; wherein each class represents a class of the same operation, and each class corresponds to the same processing device; S102, determining a target flow according to the processing flow corresponding to each target class; wherein each flow represents a processing flow in a class that can be processed in parallel using the same class of processing devices; S103, determining the processing resources of each processing device corresponding to the target class and the target flow; S104, sending the data to be moved directly to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then sends the processed data to be moved directly to the next processing device for direct processing, and when all processing devices have completed processing, the target mobile data is obtained. Compared with the current need to use the memory bus and the peripheral bus for transmission processing, the present invention determines the target class and the target stream corresponding to the data to be moved, and then determines the corresponding processing device resources according to the target class and the target stream. Therefore, when processing data, the data can be directly processed according to the allocated processing device resources without using the memory bus and the peripheral bus for transmission, thereby reducing resource usage and improving the efficiency of data movement and the performance of the entire storage system.

[0100] The current fastest commercial network (infiniband) has a latency of 0.6 microseconds, and a request and a response take a total of 1.2 microseconds. This is just the network overhead. With the software stack overhead, an RPC (remote procedure call) request can generally take tens of microseconds. After performance analysis, memory copy accounts for a large proportion, and the time it takes will range from 0.5 microseconds to hundreds of microseconds depending on the amount of data copied. For example, 4KB of data memory copy takes 0.6 microseconds, and 1MB of data memory copy takes more than 100 microseconds. In order to reduce system memory copy on the IO path, many technologies have made contributions in this regard. The unilateral operation of RDMA (remote direct memory access) can directly copy data to the other party's system memory without the intervention of the other party's CPU. It reduces the copy from kernel memory to user memory, but still uses the memory bus to copy data to system memory. If the data needs to be encrypted in the DPU, it is then stored on the NVMe disk. It needs to copy data from system memory to DPU. After DPU encryption is completed, it needs to copy the encrypted data to system memory, and then store the encrypted data from system memory to NVMe disk. Figure 5 As shown, Figure 5 An existing data movement process provided by an embodiment of the present invention may include: a local server transmits data to the network card of a remote server through network transmission via a network card, the network card directly writes the data into the memory using RDMA, the data in the system memory passes through the memory bus, CPU, and peripheral bus to the DPU, the DPU encrypts the data, the encrypted data is written into the system memory, and the data in the system memory passes through the memory bus, CPU, and peripheral bus to the data storage. Therefore, the system memory needs to be read twice and written twice (step 2-step 5) during the entire process, which will seriously affect the performance of the system.

[0101] The present invention avoids using system memory by directly transmitting data between peripheral devices (network card, DPU (data processing unit), data storage), thereby reducing the usage frequency of memory bus and system memory and improving the performance of the entire distributed storage system. Figure 6 , Figure 6 An example flow chart of a data movement method provided by an embodiment of the present invention may specifically include:

[0102] S201. Divide the data processing tasks to obtain processing flows for different types of data; the processing flow for text data includes text cleaning, word segmentation, and vocabulary building; the processing flow for image data includes format conversion and resolution adjustment; the processing flow for video data includes frame extraction, format conversion, resolution adjustment, and the extracted data is stored sequentially in chronological order in the form of an array.

[0103] It is understandable that if Figure 7 As shown, Figure 7 A data movement method architecture diagram provided by an embodiment of the present invention; Figure 7 It can be seen that the data is written to the processing device (such as DPU, NVMe (a high-speed storage protocol based on PCIe (peripheral component interconnect express), SSD (storage device based on flash memory technology), SCM (non-volatile memory technology)) through the network card, and is directly written to the data storage device after the processing device is processed, which simplifies the entire data processing path and improves the efficiency of storage (0 reads and 0 writes in this invention). The whole process consists of three parts: definition and establishment of the flow, transmission and processing of data, and notification of data processing results.

[0104] S202. According to the processing flow of different types of data, the class and stream corresponding to each type of data are obtained, wherein the class represents a type of identical operations; the stream represents a hardware device processing pipeline that can be processed in parallel in a class; the text stream of text data is regarded as one class, the image stream of image data is regarded as one class, and the video stream of video data is regarded as one class.

[0105] In this embodiment, the local server can tell the remote server what the remote server needs to do. The remote server decomposes the task and tells each peripheral device what it needs to do and reserves relevant resources. Two concepts are introduced here: class and stream. A class represents a class of identical operations, which require the same hardware resources to process and have the same processing flow (1G data class and stream); a stream represents a hardware pipeline that can be run in parallel in a class. The classes and streams in this embodiment correspond to the target classes and target streams mentioned above.

[0106] For easier understanding, please refer to Figure 8 , Figure 8 A schematic diagram of a class and flow division provided by an embodiment of the present invention, from Figure 8It can be seen that text stream 1 and text stream 2 can be used as a target class, image stream 1 and image stream 2 can be used as a target class, and video stream 1 and video stream 2 can be used as a target class, where target stream 1 can include text retention, text cleaning and word segmentation processing, vocabulary building, and storage resources. The processing devices used by the video stream under the first target class may include network cards, DPU1, DPU2, and storage media. Correspondingly, it can be seen from the figure that text stream 2, image stream 1, image stream 2, video stream 1, and video stream 2 correspond to the processing devices and the target classes they belong to. Figure 8 It is a multimodal AIGC distributed system, and its preprocessing stage and training stage will share a set of hardware. It has n clients and m servers. Each client can publish tasks to m servers in parallel. Generally speaking, n is much larger than m. In order to speed up the processing of tasks, each client will distribute tasks to m servers in parallel and evenly. The server includes a network card, a text cleaning and word segmentation processing DPU, a vocabulary building DPU, an image format conversion DPU, a resolution adjustment DPU, a frame extraction DPU1, a frame extraction DPU2, a storage medium, a CPU, a GPU, and a system memory, which are all interconnected through a PCIE bus. The allocation of resources is related to the usage scenario. Text processing is relatively simple, and each server is equipped with hardware that supports two text streams; video processing also uses image streams after frame extraction, so each server must be equipped with hardware that supports four image streams; video processing is relatively complex, and at least two DPUs are required to support its parallelism. Depending on the amount and type of data to be processed, clients and front-end processing servers can be dynamically added and reduced, and the hardware resources inside the front-end processing server can also be dynamically added and reduced. It does not use or uses very little CPU, GPU and system memory, so that the computationally intensive training stage can exclusively use them, improving its computing efficiency. In summary, the data preprocessing stage will mainly use the PCIE (PCIE) bus and the components on the PCIE, and the CPU, GPU and memory resources will be mainly used in the training stage with intensive computing power requirements. The overall hardware resources will be better utilized, thereby improving the performance of the overall AIGC (artificial intelligence generated content, generative artificial intelligence) system. In a multimodal AIGC environment (text, image and sound), the data preparation stage needs to convert a large amount of raw data into data that is easy to train. At this time, a large amount of calculation and IO operations are involved, which is often the most time-consuming stage in the entire process. Using the data movement technology of the present invention, the occupation of system resources can be greatly reduced, and the CPU and memory resources can be left to the training stage of the model, thereby greatly improving the overall performance of AIGC training.

[0107] S203. Determine the number of divided resources for the flows in each class according to the number of flows in each class and the number of processing devices and processing devices; wherein the processing devices include a network card, a DPU, and a storage medium.

[0108] S204: When dividing resources, obtain the processing time ratio between all processing devices according to the time taken by the processing devices to process data.

[0109] S205. Determine the processing resource size of each flow on each device based on the processing time ratio and the number of divided resources, process the task data according to the divided processing resource size, and process data belonging to the same category but different flows in parallel.

[0110] In order to improve the execution efficiency, in this embodiment, data can be processed in parallel at the hardware level between streams; pipeline technology can be used to perform component-level parallelism on one stream, thereby improving processing efficiency.

[0111] S206. Notify the error code, error location and error cause through the network card. The error location includes the class, flow and error component.

[0112] S207. Determine whether the error can be repaired based on the error location and error cause; if the error can be repaired, the request is re-executed after the error is repaired locally. If the error cannot be repaired locally, the error code is sent to the user program for processing.

[0113] For easier understanding, please refer to Fig. 9 , Fig. 9 A schematic diagram of a processing flow for correctly corresponding data movement provided by an embodiment of the present invention, and a notification of data processing results. During the entire execution process, if there is no problem with the processing, the CPU or DPU is notified through the network card to proceed to the next step.

[0114] Please refer to Fig.10 , Fig.10 A structural diagram of a processing flow corresponding to a data movement error provided by an embodiment of the present invention. During the entire execution process, if there is a problem in the processing, the CPU is notified of the error code and the location of the error through the network card, which includes the class, flow, error component and the cause of the error. The reason why the error condition is notified to the CPU is that it has strong logical processing capabilities and can handle complex fault problems. After the local CPU is notified, it decides whether it can be repaired. If it can be repaired, it will re-execute the request after repairing itself; if the local CPU cannot handle it, it will send the relevant error code to the user program for processing.

[0115] The embodiment of the present invention proposes a data movement method without system memory copy, thereby completely avoiding the use of system memory and memory bus during data processing and storage, which not only reduces resource usage and IO latency, but also improves the performance of the entire system.

[0116] The following is an introduction to a data moving device provided by an embodiment of the present invention. The data moving device described below and the data moving method described above can be referred to each other.

[0117] Fig.11 A structural diagram of a data moving device provided in an embodiment of the present invention may include:

[0118] The target class determination module 100 is used to obtain different types of data to be moved and determine the target class according to the data type of the data to be moved; wherein each class represents a same type of operation and each class corresponds to the same processing device;

[0119] The target flow determination module 200 is used to determine the target flow according to the processing flow corresponding to each target class; wherein each flow represents a processing flow in a class that can be processed in parallel using the same type of processing equipment;

[0120] A processing resource determination module 300, configured to determine the processing resources of each processing device corresponding to the target class and the target flow;

[0121] The target mobile data determination module 400 is used to send the data to be moved directly to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then sends the processed data to be moved directly to the next processing device for direct processing. When all processing devices have completed the processing, the target mobile data is obtained.

[0122] Further, based on the above embodiment, when the target class is a text class, an image class, and a video class, the target flow determination module 200 may include:

[0123] The first target flow determination module is used to determine the first target flow according to the processing flow corresponding to the text class.

[0124] Target flow; wherein the first target flow includes text cleaning, word segmentation processing and vocabulary building;

[0125] A second target stream determination module, configured to determine a second target stream according to a processing flow corresponding to the image class; wherein the second target stream includes a first format conversion and a first resolution adjustment;

[0126] The third target stream determination module is used to determine the third target stream according to the processing flow corresponding to the video class; wherein the third target stream includes frame extraction, second format conversion, second resolution adjustment and array storage operation; the array storage operation refers to storing all extracted video frames in chronological order in the form of an array; the number of processing devices corresponding to each target stream is determined according to demand.

[0127] Further, based on any of the above embodiments, the above data moving device may include:

[0128] A processing device adding processing module, used for determining to add processing to the processing device when it is determined that the processing resources corresponding to the current processing device cannot meet the resource requirements for processing the target class and the target flow;

[0129] The processing device deletion module is used to determine to delete the unused processing device when it is determined that there is an unused processing device in the current processing device.

[0130] Further, based on any of the above embodiments, the processing resource determination module 300 may include:

[0131] A processing device determination module unit is used to determine the processing device corresponding to the current target class;

[0132] A processing time ratio determination unit, configured to obtain a processing time ratio corresponding to each processing device according to the time taken by each processing device to process the flow under the current target class;

[0133] A resource share determination unit, configured to determine the number of hardware resource shares for each flow under the current target class on each processing device based on the processing time ratio;

[0134] The processing resource determination unit is used to determine the processing resources of each processing device based on the number of hardware resource shares of each flow under the current target class.

[0135] Further, based on any of the above embodiments, the target mobile data determination module 400 may include:

[0136] A first sending unit, used for sending the data to be moved directly to the network card;

[0137] A first processing data determining unit, configured to perform parallel processing on flows under the current target class using the network card according to the processing resources allocated to the network card, to obtain first processing data;

[0138] a second sending unit, configured to send the first processed data directly to the data processor;

[0139] A second processing data determining unit, configured to process the stream under the current target class in the current data processor using the data processor according to the processing resources allocated to the data processor, to obtain second processing data;

[0140] a third sending unit, configured to send the second processed data to the storage medium; wherein the storage medium is a physical device or medium for recording and storing data;

[0141] The target mobile data storage unit is used to store the flow under the current target class in the storage medium by using the storage medium according to the processing resources allocated to the storage medium to obtain the target mobile data.

[0142] Further, based on any of the above embodiments, the above data moving device may further include:

[0143] The move completion judgment module is used to monitor the entire data move process and determine whether the move is successful when it is determined that the data move is completed;

[0144] The error location positioning module is used to determine the error code, error location and error cause when the movement is unsuccessful; wherein the error location includes the error type, error flow and error processing device.

[0145] Further, based on any of the above embodiments, the above data moving device may further include:

[0146] A repair strategy determination module, used to determine whether there is a corresponding repair strategy for the current problem according to the error location and the error cause;

[0147] A repair module, used for determining to perform repair based on a repair strategy when a repair strategy exists, and re-executing the data movement request when the repair is successful;

[0148] The error code sending module is used to send the error code to the user when there is no repair strategy.

[0149] It should be noted that the order of the modules and units in the above data moving device can be changed without affecting the logic.

[0150] Fig.11 The description of the features in the corresponding embodiments can be found in Fig.11 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0151] The data moving device provided by the embodiment of the present invention may include: a target class determination module 100, which is used to obtain different types of data to be moved, and determine the target class according to the data type of the data to be moved; wherein each class represents a class of the same operation, and each class corresponds to the same processing device; a target flow determination module 200, which is used to determine the target flow according to the processing flow corresponding to each of the target classes; wherein each flow represents a processing flow in a class that can be processed in parallel using the same class of processing devices; a processing resource determination module 300, which is used to determine the processing resources of each processing device corresponding to the target class and the target flow; a target mobile data determination module 400, which is used to send the data to be moved directly to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then sends the processed data to be moved directly to the next processing device for direct processing, and when all processing devices have completed processing, the target mobile data is obtained. Compared with the current need to use the memory bus and the peripheral bus for transmission processing, the present invention determines the target class and the target stream corresponding to the data to be moved, and then determines the corresponding processing device resources according to the target class and the target stream. Therefore, when processing data, the data can be directly processed according to the allocated processing device resources without using the memory bus and the peripheral bus for transmission, thereby reducing resource usage and improving the efficiency of data movement and the performance of the entire storage system.

[0152] A data moving device provided in an embodiment of the present invention is introduced below. The data moving device described below and the data moving method described above can refer to each other.

[0153] Fig.12 A schematic diagram of the structure of a data mobile device provided by an embodiment of the present invention, such as Fig.12 As shown, the data mobile device may include: a memory 60 for storing a computer program;

[0154] The processor 61 is used to implement the steps of the data moving method in the above embodiment when executing a computer program.

[0155] The data mobile device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0156] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0157] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the data movement method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include but is not limited to data required for data movement, etc.

[0158] In some embodiments, the data mobile device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0159] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the data mobile device, and may include more or fewer components than shown in the figure.

[0160] It is understandable that if the data moving method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0161] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data moving method are implemented.

[0162] The above is a detailed introduction to a data movement method, device, equipment and computer-readable storage medium provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0163] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0164] The above is a detailed introduction to a data movement method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A data migration method, characterized in that: include: Acquire different types of data to be moved, and determine a target class according to the data type of the data to be moved; wherein each class represents a same processing flow, and each class corresponds to the same processing device; Determine the target flow according to the processing flow corresponding to each of the target classes; wherein each flow represents a processing flow in a class that can be processed in parallel using the same type of processing equipment; Determine the processing resources of each processing device corresponding to the target class and the target flow; The data to be moved is directly sent to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then sends the processed data to be moved directly to the next processing device for direct processing. When all processing devices have completed the processing, the target mobile data is obtained.

2. The data migration method according to claim 1, characterized in that: When the target class is a text class, an image class, and a video class, determining the target flow according to the processing flow corresponding to each target class includes: Determine a first target flow according to the processing flow corresponding to the text class; wherein the first target flow includes text cleaning, word segmentation processing and vocabulary building; Determine a second target stream according to the processing flow corresponding to the image class; wherein the second target stream includes a first format conversion and a first resolution adjustment; According to the processing flow corresponding to the video class, a third target stream is determined; wherein the third target stream includes frame extraction, second format conversion, second resolution adjustment and array storage operation; the array storage operation refers to storing all extracted video frames in chronological order in the form of an array; the number of processing devices corresponding to each target stream is determined according to demand.

3. The data migration method according to claim 1, characterized in that: After determining the processing resources of each processing device corresponding to the target class and the target flow, the method further includes: When it is determined that the processing resources corresponding to the current processing device cannot meet the resource requirements for processing the target class and the target flow, determining to increase processing on the processing device; When it is determined that there is an unused processing device in the current processing devices, it is determined to delete the unused processing device.

4. The data migration method according to any one of claims 1 to 3, characterized in that: Determining the processing resources of each processing device corresponding to the target class and the target flow includes: Determine the processing device corresponding to the current target class; According to the time taken by each processing device to process the flow under the current target class, the processing time ratio corresponding to each processing device is obtained; Determine the number of hardware resource shares for each flow under the current target class on each processing device based on the processing time ratio; The processing resources of each processing device are determined based on the number of hardware resource shares of each flow under the current target class.

5. The data migration method according to claim 1, characterized in that: When the processing device includes a network card, a data processor and a storage device, the data to be moved is directly sent to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then directly sends the processed data to be moved to the next processing device for direct processing. When all processing devices complete the processing, the target mobile data is obtained, including: Sending the data to be moved directly to the network card; Using the network card according to the processing resources allocated to the network card, the flows under the current target class are processed in parallel to obtain first processed data; sending the first processed data directly to the data processor; Using the data processor according to the processing resources allocated to the data processor, processing the flow under the current target class in the current data processor to obtain second processed data; sending the second processed data to the storage medium; wherein the storage medium is a physical device or medium for recording and storing data; The storage medium is used to store the flow under the current target class in the storage medium according to the processing resources allocated to the storage medium, so as to obtain the target mobile data.

6. The data migration method according to claim 1, characterized in that: Also includes: Monitor the entire data movement process, and when it is determined that the data movement is complete, determine whether the movement is successful; When the movement is unsuccessful, an error code, an error location and an error cause are determined; wherein the error location includes an error type, an error flow and an error processing device.

7. The data migration method according to claim 6, characterized in that: After determining the error code, error location and error cause, it also includes: Determine whether there is a corresponding repair strategy for the current problem according to the error location and the error cause; When a repair strategy exists, determining to perform repair based on the repair strategy, and when the repair is successful, re-executing the data movement request; When no recovery strategy exists, an error code is sent to the user.

8. A data moving device, characterized in that: include: A target class determination module is used to obtain different types of data to be moved and determine the target class according to the data type of the data to be moved; wherein each class represents a same type of operation and each class corresponds to the same processing device; A target flow determination module, used to determine the target flow according to the processing flow corresponding to each target class; wherein each flow represents a processing flow in a class that can be processed in parallel using the same type of processing equipment; A processing resource determination module, used to determine the processing resources of each processing device corresponding to the target class and the target flow; The target mobile data determination module is used to send the data to be moved directly to the first processing device corresponding to the target flow under the target class, so that the first processing device processes the data to be moved according to the allocated processing resources, and then sends the processed data to be moved directly to the next processing device for direct processing. When all processing devices have completed the processing, the target mobile data is obtained.

9. A data mobile device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the data moving method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data moving method according to any one of claims 1 to 7 are implemented.

Images