A data migration processing method and system in a data center

Abstract

The application discloses a data migration processing method and system in a data center. The method comprises the following steps: determining a migration task containing to-be-migrated data, network topology, a target terminal and a processing task; evaluating a transmission time according to the migration task, and entering a migration optimization mode if the transmission time is predicted to exceed a time limit; dividing processing subtasks based on the network topology in the migration task and distributing the processing subtasks to intermediate link devices by the target terminal; intercepting the to-be-migrated data flowing through by the intermediate link devices and processing the to-be-migrated data according to the subtasks, and returning results; and receiving and merging the processing results by the target terminal. According to the judgment of the to-be-migrated data and the actual situation of the network topology in the data migration task, the processing task is divided into multiple subtasks and sent to the network topology for distributed execution of data processing, so that the data is processed while being transmitted, the data migration amount and the total time consumption are significantly reduced, the migration efficiency is improved, and the centralized processing load on the target terminal is reduced.

Description

Technical Field

[0001] This application belongs to the field of big data migration technology, and specifically relates to a data migration processing method and system in a data center. Background Technology

[0002] With the acceleration of enterprise digitalization, the volume of data is exploding. In the lifecycle of a data cluster, data migration has become a routine and critical operation, with common application scenarios including: data processing, cluster expansion or contraction, business system reconstruction, cross-cloud deployment, and data archiving.

[0003] The mainstream data migration solutions in the industry mainly follow these modes: 1. Source-end processing-transmission mode: In this mode, data is processed in whole or in part on the source storage system side (such as data filtering, format conversion, compression, etc.), but terminal computing power is not effectively utilized, making it unsuitable for edge computing scenarios, while source-end computing power is often limited and difficult to scale. 2. Transmission-destination processing mode: In this mode, the original data is first transmitted completely from the source to the destination, and then the required processing operations are performed on the destination system. This is the simplest migration method, such as using rsync, scp, or simple copy functionality of object storage. Although this mode avoids interference with the source, its most significant drawback is that it transmits all data indiscriminately, including a large amount of invalid or unnecessary data content, resulting in a huge waste of network bandwidth. Especially when the cost of backbone network bandwidth across regions and clouds is high, the economic and time costs of migration become unacceptable. 3. Parallel processing mode based on fixed computing nodes: In order to improve efficiency, existing technologies also use parallel processing frameworks (such as Apache Spark and Flink) for data migration. This approach defines the migration task as an ETL job, executing a complete pipeline from reading, processing, to writing on a dedicated, fixed computing cluster. While this model improves throughput through parallel computing, its computation location (i.e., the fixed computing cluster) is decoupled from the actual data transmission path. Data still needs to be pulled from the source to this fixed computing cluster, processed, and then pushed to the target, which can cause traffic detours (the "triangular transmission" problem) and does not optimize the actual data flow path in the network. Furthermore, the resource configuration of the dedicated computing cluster is static and cannot be dynamically adjusted or elastically scaled according to the characteristics of each migration task and real-time network conditions.

[0004] In summary, existing data migration technologies cannot dynamically optimize the data transmission path and processing location in a coordinated manner. The migration process either interferes with production systems, generates redundant network traffic and latency, or fails to fully utilize potentially idle distributed computing resources along the path. Ultimately, large-scale data migration is time-consuming, costly, and fails to meet the stringent requirement of minimizing business impact. Therefore, there is an urgent need for an innovative data migration solution that can break down the barriers between computing and transmission, achieving network-aware, dynamically scheduled intelligent migration, thereby significantly improving migration efficiency and reducing business impact and overall costs.

[0005] The information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] To address the problems existing in the prior art, the present invention provides a data migration processing method and system in a data center.

[0007] Specifically, this application claims a data migration processing method in a data center, the method comprising the following steps:

[0008] S100: Determine the migration task from the data center. The migration task includes the data to be migrated, the migration network topology, the target terminal, and the data processing task to be executed. Determine the estimated transmission time based on the migration network topology configuration. When the estimated transmission time exceeds the time requirement of the data processing task to be executed, configure the system to enter the migration optimization mode.

[0009] S110, the target terminal determines the distribution of intermediate link devices based on the migration network topology, divides the data processing task to be executed into subtasks based on the time difference to be optimized, obtains multiple subtasks corresponding to the distribution of intermediate link devices, and sends them to the corresponding intermediate link devices.

[0010] S120, each intermediate link device filters and extracts the data to be migrated according to the received sub-tasks, and processes the extracted data to be migrated according to the requirements of the received sub-tasks.

[0011] S130, the intermediate link device sends the processing result to the target terminal, and the target terminal receives the processing result and performs the merging operation.

[0012] Further, the step S100 of determining the estimated transmission time based on the migration network topology configuration specifically involves:

[0013] T_real = D / (B * U_real)

[0014] U_real = U_topology * U_protocol * U_contention * U_distance

[0015] Where D represents the total amount of valid data to be migrated, B represents the bottleneck link bandwidth in the migration path, and U_real represents the actual bandwidth utilization rate, which is jointly determined by the topology factor U_topology, the protocol and transmission factor U_protocol, the congestion contention factor U_contention, and the distance delay factor U_distance. The topology factor U_topology is the maximum available bandwidth ratio determined by the physical and logical topology; the protocol and transmission factor U_protocol is determined by the migration tool and the transmission protocol; the congestion contention factor U_contention refers to the discount when migration traffic shares the link with production business traffic; and the distance delay factor U_distance is a correction factor for estimating the migration of a large number of small files or database transactions that require real-time ACK.

[0016] Further, step S110 includes the following sub-steps:

[0017] S111: Identify computable nodes in the migration network topology;

[0018] S112: Quantify the optimization objective and parse the operation primitives of the data processing task to be executed based on the computable nodes.

[0019] S113: Based on the parsed operation primitives, the data processing task to be executed is divided into subtasks, and the subtasks are sent to the computable nodes in the corresponding intermediate link devices.

[0020] Further, in step S120, the task engine is configured to execute subtasks: S121, the task engine is configured via a protocol. When the computeable node receives a subtask request from the target terminal, the task engine parses the subtask, extracts the data source address and the identifier of the data to be processed; S122, the task engine creates an isolated and secure execution environment; S123, based on the extracted data source address and identifier of the data to be processed, the task engine obtains the raw data to be processed for this subtask from the network traffic flowing through the device and imports it into the execution environment; S124, based on the imported raw data to be processed for this subtask and the received processing logic, the task engine restores the intercepted data into application layer or storage layer data objects and executes the processing logic; S125, the task engine packages the processing results and prepares to send them to the target terminal.

[0021] Furthermore, the specific steps of obtaining the raw data to be processed by this subtask from the network traffic flowing through the device are as follows:

[0022] (1) Translate the “data identifier to be processed” in the subtask into a matching rule at the network layer. (2) Copy or redirect the matched data traffic to the local task engine.

[0023] Furthermore, the processing result is the parallel training result of multiple sub-neural network models, where the training result is the sub-model parameters. During the merging operation, the parameters are merged based on weights and biases, or based on the parameter values ​​of each layer of the neural network. Alternatively, the processing result is the prediction result of multiple sub-neural network models, and the prediction results of multiple sub-neural network models are balanced during the merging operation. Or, the processing result is key-value merging, and the processing result files sorted by user_id are merged using a "multi-way merge sort" algorithm during the merging operation.

[0024] A data migration processing system in a data center is also provided, the system comprising:

[0025] The task evaluation module is configured to determine migration tasks from the data center. The migration tasks include data to be migrated, migration network topology, target terminal, and data processing tasks to be executed. The estimated transmission time is determined based on the migration network topology configuration. When the estimated transmission time exceeds the time requirement of the data processing tasks to be executed, the module is configured to enter migration optimization mode.

[0026] The task partitioning module is configured to enable the target terminal to determine the distribution of intermediate link devices based on the migration network topology, divide the data processing task to be executed into subtasks based on the time difference to be optimized, obtain multiple subtasks corresponding to the distribution of intermediate link devices, and send them to the corresponding intermediate link devices.

[0027] The distributed data processing module is configured to enable each intermediate link device to filter and extract the data to be migrated based on the received sub-tasks, and process the extracted data to be migrated according to the requirements of the received sub-tasks.

[0028] The result merging module is configured to enable intermediate link devices to send processing results to the target terminal, and the target terminal receives the processing results and performs the merging operation.

[0029] Furthermore, the calculation method for determining the estimated transmission time T_real based on the migration network topology configuration in the task evaluation module is as follows:

[0030] T_real = D / (B * U_real)

[0031] U_real = U_topology * U_protocol * U_contention * U_distance

[0032] Where D represents the total amount of valid data to be migrated, B represents the bottleneck link bandwidth in the migration path, and U_real represents the actual bandwidth utilization, which is jointly determined by the topology factor U_topology, the protocol and transmission factor U_protocol, the congestion contention factor U_contention, and the distance delay factor U_distance. The topology factor U_topology is the maximum available bandwidth ratio determined by the physical and logical topology; the protocol and transmission factor U_protocol is determined by the migration tool and the transmission protocol; the congestion contention factor U_contention refers to the correction factor when migration traffic shares the link with production business traffic; and the distance delay factor U_distance refers to the correction factor for estimating the migration of a large number of small files or database transactions that require real-time confirmation.

[0033] Furthermore, the task partitioning module specifically includes:

[0034] A node identification unit is used to identify computable nodes in the migrating network topology.

[0035] The task parsing unit is used to quantify the optimization objective and parse the operation primitives of the data processing task to be executed based on the computable nodes;

[0036] The subtask generation unit is used to divide the data processing task to be executed into subtasks based on the parsed operation primitives, and send the divided subtasks to the computable nodes in the corresponding intermediate link devices.

[0037] Furthermore, the distributed data processing module deploys a task engine in the intermediate link device, and this module specifically includes:

[0038] The information extraction unit is used to parse the subtask and extract the data source address and the identifier of the data to be processed when the computable node receives the subtask request.

[0039] The environment creation unit is used to create an isolated and secure execution environment for task execution;

[0040] The data extraction unit is used to match and extract the required raw data from the network traffic flowing through the device based on the extracted data source address and identifier, and import it into the execution environment;

[0041] The data execution unit is used to restore the intercepted data into data objects in the application layer or storage layer, and to perform data processing based on the received processing logic;

[0042] The result packaging unit is used to package the processing results and prepare them for sending to the target terminal.

[0043] Furthermore, the data interception unit specifically performs the following:

[0044] (1) Generate network-level traffic matching rules based on the "data identifier to be processed" in the subtask;

[0045] (2) Apply the matching rules to the network traffic that flows through it, and copy or redirect the successfully matched data traffic to the execution environment of the local task engine.

[0046] Furthermore, the processing result is the parallel training result of multiple sub-neural network models, where the training result is the sub-model parameters. During the merging operation, the parameters are merged based on weights and biases, or based on the parameter values ​​of each layer of the neural network. Alternatively, the processing result is the prediction result of multiple sub-neural network models, and the prediction results of multiple sub-neural network models are balanced during the merging operation. Or, the processing result is key-value merging, and the processing result files sorted by user_id are merged using a "multi-way merge sort" algorithm during the merging operation.

[0047] This invention provides a data migration processing method and system in a data center. The method innovates the traditional end-to-end data migration mode by introducing transmission time estimation and dynamic task partitioning, and transforms it into an in-transit computing mode based on the network topology in the migration task. The system has at least the following effects: (1) By decomposing the data processing task and distributing it to intermediate nodes on the transmission path, deep parallelism between computing and transmission is achieved, which greatly reduces the total time of massive data migration from transmission time to centralized processing time, and significantly improves migration efficiency. (2) The computing load of the target terminal is effectively distributed, and the utilization rate and scalability of the entire system resources are improved by utilizing the idle computing power of the intermediate nodes in the network. (3) Based on a multi-factor (topology, protocol, congestion, distance) accurate transmission time estimation model, a scientific decision-making basis is provided for whether to start the optimization mode, and combined with dynamic task partitioning, the processing capacity is adaptively matched with the network conditions, which enhances the success rate and reliability of the migration task. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only used to illustrate some embodiments of this application. For those skilled in the art, without creative effort, they can obtain other technical features, connection relationships, and even method steps not mentioned in the accompanying drawings based on these drawings.

[0049] Figure 1 This is a flowchart of a data migration processing method in a data center provided by an embodiment of the present invention.

[0050] Figure 2 This is a structural diagram of a data migration processing system in a data center provided by an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] Example 1: As Figure 1 As shown, the present invention provides a data migration processing method in a data center, the method comprising the following steps:

[0055] S100: Determine the migration task from the data center. The migration task includes the data to be migrated, the migration network topology, the target terminal, and the data processing task to be executed. Determine the estimated transmission time based on the migration network topology configuration. When the estimated transmission time exceeds the time requirement of the data processing task to be executed, configure the system to enter the migration optimization mode.

[0056] In data center scenarios such as data backup or archiving, edge computing, environment deployment and upgrades, and model training, data migration is a routine task. Key challenges include data consistency, compatibility, downtime, data cleaning and transformation, and migration processing time. The migration network topology directly determines the migration path, cost, and speed. Its core configuration includes network bandwidth, latency, congestion points, and path reliability. Specifically, network bandwidth can be understood as the combined bandwidth between all devices. The narrowest link in the topology (the bottleneck link) determines the maximum theoretical migration speed. For example, migrating data from a gigabit LAN on device A to the cloud will be limited to 100Mbps if it passes through a 100Mbps line. Latency is the time consumed by hardware and software data processing during transmission. For example, the hop count and link type in the topology can directly affect latency. For scenarios involving the migration of large numbers of small files or requiring real-time synchronization, high latency can severely impact speed. Congestion points typically refer to speed bottlenecks in core switches, routers, firewalls, and other devices within a complex topology. If migration traffic shares paths with production traffic, network congestion may occur. Once the data to be migrated and the configuration of the migration network topology in the migration task are determined, the estimated transmission time can be calculated. If the estimated transmission time meets the time requirements of the current data processing task, the data migration processing is performed directly based on the current migration network topology. If the estimated transmission time does not meet the time requirements of the current data processing task, the system is configured to enter the migration optimization mode.

[0057] In a preferred embodiment, the step S100 of determining the estimated transmission time based on the migration network topology configuration specifically involves:

[0058] T_real = D / (B * U_real)

[0059] U_real = U_topology * U_protocol * U_contention * U_distance

[0060] Where D represents the total amount of valid data to be migrated, B represents the bottleneck link bandwidth in the migration path, and U_real represents the actual bandwidth utilization rate, which is jointly determined by the topology factor U_topology, the protocol and transmission factor U_protocol, the congestion contention factor U_contention, and the distance delay factor U_distance. The topology factor U_topology is the maximum available bandwidth ratio determined by the physical and logical topology; the protocol and transmission factor U_protocol is determined by the migration tool and the transmission protocol; the congestion contention factor U_contention refers to the discount when migration traffic shares the link with production business traffic; and the distance delay factor U_distance is a correction factor for estimating the migration of a large number of small files or database transactions that require real-time ACK.

[0061] In a preferred embodiment, the step S100 of determining the estimated transmission time based on the migration network topology configuration specifically involves: during the planned migration period, using an actual migration tool, transmitting a representative data sample (such as a 100GB large file + a directory containing 10,000 files), recording the actual average transmission rate, and determining the estimated transmission time based on the average transmitted data and the total amount of valid data D to be migrated.

[0062] In a preferred embodiment, the time requirement of the data processing task in step S100 can be an absolute time value, such as the data center downtime window or the task completion deadline (including data training, data processing, and other tasks). Similarly, the time requirement of the data processing task can also be a relative time value, such as the user's optimized value or optimization ratio for the expected transmission time.

[0063] S110, the target terminal determines the distribution of intermediate link devices based on the migration network topology, divides the data processing task to be executed into subtasks based on the time difference to be optimized, obtains multiple subtasks corresponding to the distribution of intermediate link devices, and sends them to the corresponding intermediate link devices.

[0064] This step is used to segment the tasks of the target terminal. This segmentation step is not similar to the static task segmentation and distributed processing in existing technologies, but rather a dynamic perception and decision-making control process determined based on network topology, task status, transmission configuration, and other configuration conditions. Existing technologies typically segment the total task into sub-tasks based on the target device's load and capabilities, thereby integrating the capabilities of multiple computing devices to meet the requirements of a single total task. This application innovatively uses information perception of the distribution of intermediate link devices in the current migration network topology to subdivide the data processing tasks to be executed based on the time difference to be optimized, thus determining the optimal migration speed and data processing speed. Specifically, step S110 includes the following sub-steps:

[0065] S111: Identify computable nodes in the migration network topology;

[0066] Migrating network topologies can be simple point-to-point topologies, bus (chain) topologies, star topologies, ring topologies, etc., or complex mesh topologies, such as extended tree topologies, partial mesh topologies, cross topologies, fat tree topologies, etc. In the above topology types, data migration involves various types of intermediate link devices, including: (1) computing devices that can directly provide computing power, such as host devices; (2) computing devices that can provide computing power, such as core switches, border routers, WAN optimization devices, etc. These devices can be pre-installed with lightweight agents or support a management protocol similar to NETCONF to report their ability to "provide computing services" (such as the number of CPU cores, available memory, and whether they have specific processing acceleration units such as GPUs / FPGAs); (3) intermediate devices that can indirectly provide computing power, such as conventional routers, switches, etc. These devices can forward data to other nearby computing devices for processing, thereby indirectly providing the computing power of nearby computing devices to the system. In the process of identifying the above-mentioned computable nodes, the comprehensive computing power that the computable nodes can provide should be determined, so as to clearly divide the data processing tasks to be performed into subtasks.

[0067] S112: Quantify the optimization objective and parse the operation primitives of the data processing task to be executed based on the computable nodes.

[0068] The optimization objective is a value (absolute or relative) related to the optimization time, which is quantified in relation to the task time. Parsing the "data processing task to be executed" and decomposing it into distributed execution primitives is the core step of the solution, requiring an evaluation of the computational complexity of each primitive (e.g., compressing 1GB of data requires approximately X CPU clock cycles). Operation primitives are decomposable intermediate steps in the data processing process. Depending on the data processing task, the form of the operation primitives varies; for example, they can be multiple parallel computation operations, sample training sub-steps in big data models, data cleaning, archiving, etc. In this embodiment, taking model sample training as a data processing task (such as unsupervised learning in an autoencoder VAE) as an example, large-batch training particularly requires data parallelism. Based on the quantified optimization objective and the comprehensive computing power of the computable nodes, the sample training task is parsed into multiple sub-sample set training tasks as multiple operation primitives. Taking the quantified optimization objective of "reducing time by 600 seconds" as an example, if the overall computing power of computable node A is "the unsupervised learning inference speed of this autoencoder VAE is 200 sample frames / second", then computable node A can save the target terminal 100 seconds by consuming 20,000 sample frames. Furthermore, since these 20,000 sample frames do not need to be processed by the target terminal, ideally, data migration time for 20,000 sample data can also be saved. Considering the total time saved by all computable nodes, we can determine how to parse the sample training task into multiple operational primitives. In practice, the above analysis should also consider the resource and time consumption generated during the task division process (and the subsequent merging of calculation results).

[0069] S113: Based on the parsed operation primitives, the data processing task to be executed is divided into subtasks, and the subtasks are sent to the computable nodes in the corresponding intermediate link devices.

[0070] This step distributes the task load across the distributed network topology, improving data processing efficiency from a data processing perspective. At the same time, since the target terminal does not need to process the data to be migrated corresponding to the divided subtasks, it further improves the efficiency of data migration.

[0071] S120, each intermediate link device filters and extracts the data to be migrated according to the received sub-tasks, and processes the extracted data to be migrated according to the requirements of the received sub-tasks.

[0072] Step S120 establishes a local data processing pipeline within the computed node of the intermediate link device. In a preferred embodiment, a task engine is configured to execute subtasks. Specifically, firstly, the task engine is configured via a protocol. When the computed node receives a subtask request from the target terminal, the task engine parses the subtask, including decryption and signature verification, to ensure the instruction is legitimate, and extracts the data source address and data identifier to be processed (such as a file list, data block ID range, and data query conditions). Secondly, the task engine creates an isolated and secure execution environment. In this embodiment, the processing logic is a Python task script. The task execution engine starts a dedicated Python interpreter container to prepare for executing the subtask. Thirdly, based on the extracted data source address and data identifier of the subtask, the task engine obtains the raw data to be processed from the network traffic flowing through the device and imports it into the execution environment, ensuring that the normal forwarding of unrelated data is not affected. Fourthly, based on the imported raw data to be processed by the subtask and the received processing logic, the task engine restores the intercepted data into application-layer or storage-layer data objects and executes the processing logic. Finally, the task engine packages the processing results and prepares to send them to the final target.

[0073] In a preferred embodiment, obtaining the raw data to be processed by this subtask from the network traffic flowing through the device specifically involves:

[0074] (1) Translate the "data identifier to be processed" in the subtask into a network-level matching rule. For example, if the subtask is to process an object with a source IP of 10.0.1.100 that accesses the / logs / pos-sample prefix under bucket-a, then generate the corresponding five-tuple (source IP, source port, destination IP, destination port, data filtering condition) matching rule. (2) Copy or redirect the matched data traffic to the local task engine, including pointing the next hop of the matched traffic to a local virtual interface, or mirroring the traffic of the relevant port to the port connected to the computing module on a switch that supports the commodity analyzer SPAN function, or on an OpenFlow switch, having the target terminal or dedicated scheduler issue flow table entries to perform an output-to-local action on the matched traffic. The above method realizes the filtering and interception of the original migration data.

[0075] S130, the intermediate link device sends the processing result to the target terminal, and the target terminal receives the processing result and performs the merging operation.

[0076] The target terminal establishes an authenticated, encrypted connection (such as TLS) with each intermediate link device to receive processing results. A resident service program on the target terminal parses the received data stream. Each processing result should be encapsulated in an enhanced result packet, which should at least contain a global task ID identifying which migration task it belongs to, a unique subtask ID, a data range descriptor precisely describing which part of the original data this result corresponds to (e.g., [file A: byte offset], [table T: primary key range 'a'-'m']), a checksum, and other metadata (e.g., timestamp, version number, etc.). In a preferred embodiment, the target terminal manages a temporary buffer to maintain a result cache for each subtask ID in memory or high-speed temporary storage (such as SSD). The "data range descriptor" parsed by the resident service program on the target terminal synchronously processes each asynchronous result and places it in the logically correct location.

[0077] In a preferred embodiment, the authenticity of the result data is verified through digital signatures and authentication. Verifying the authenticity of the results ensures the security of the data source and prevents data from malicious intermediate link devices from adversely affecting the results. The digital signature information is recorded in the protocol of the connection established between the target terminal and each intermediate link device.

[0078] In a preferred embodiment, the processing result is the parallel training result of multiple sub-neural network models. This training result can be sub-model parameters. During the merging operation, the parameters are merged based on weights (w) and biases (b), or based on the parameter values ​​of each layer of the neural network. In another preferred embodiment, the processing result is the prediction result of multiple sub-neural network models. During the merging operation, the prediction results of the multiple sub-neural network models are balanced. In a preferred embodiment, the processing result is key-value merging. During the merging operation, the processing result files sorted by user_id are merged using a multi-way merge sort algorithm.

[0079] Example 2: Figure 2 As shown, the present invention provides a data migration processing system in a data center, comprising:

[0080] The task evaluation module is configured to determine migration tasks from the data center. The migration tasks include data to be migrated, migration network topology, target terminal, and data processing tasks to be executed. The estimated transmission time is determined based on the migration network topology configuration. When the estimated transmission time exceeds the time requirement of the data processing tasks to be executed, the module is configured to enter migration optimization mode.

[0081] The task partitioning module is configured to enable the target terminal to determine the distribution of intermediate link devices based on the migration network topology, divide the data processing task to be executed into subtasks based on the time difference to be optimized, obtain multiple subtasks corresponding to the distribution of intermediate link devices, and send them to the corresponding intermediate link devices.

[0082] The distributed data processing module is configured to enable each intermediate link device to filter and extract the data to be migrated based on the received sub-tasks, and process the extracted data to be migrated according to the requirements of the received sub-tasks.

[0083] The result merging module is configured to enable intermediate link devices to send processing results to the target terminal, and the target terminal receives the processing results and performs the merging operation.

[0084] In a preferred embodiment, the calculation method for determining the estimated transmission time T_real based on the migration network topology configuration in the task evaluation module is as follows:

[0085] T_real = D / (B * U_real)

[0086] U_real = U_topology * U_protocol * U_contention * U_distance

[0087] Where D represents the total amount of valid data to be migrated, B represents the bottleneck link bandwidth in the migration path, and U_real represents the actual bandwidth utilization, which is jointly determined by the topology factor U_topology, the protocol and transmission factor U_protocol, the congestion contention factor U_contention, and the distance delay factor U_distance. The topology factor U_topology is the maximum available bandwidth ratio determined by the physical and logical topology; the protocol and transmission factor U_protocol is determined by the migration tool and the transmission protocol; the congestion contention factor U_contention refers to the correction factor when migration traffic shares the link with production business traffic; and the distance delay factor U_distance refers to the correction factor for estimating the migration of a large number of small files or database transactions that require real-time confirmation.

[0088] In a preferred embodiment, the task partitioning module specifically includes:

[0089] A node identification unit is used to identify computable nodes in the migrating network topology.

[0090] The task parsing unit is used to quantify the optimization objective and parse the operation primitives of the data processing task to be executed based on the computable nodes;

[0091] The subtask generation unit is used to divide the data processing task to be executed into subtasks based on the parsed operation primitives, and send the divided subtasks to the computable nodes in the corresponding intermediate link devices.

[0092] In a preferred embodiment, the distributed data processing module has a task engine deployed in the intermediate link device, and the module specifically includes:

[0093] The information extraction unit is used to parse the subtask and extract the data source address and the identifier of the data to be processed when the computable node receives the subtask request.

[0094] The environment creation unit is used to create an isolated and secure execution environment for task execution;

[0095] The data extraction unit is used to match and extract the required raw data from the network traffic flowing through the device based on the extracted data source address and identifier, and import it into the execution environment;

[0096] The data execution unit is used to restore the intercepted data into data objects in the application layer or storage layer, and to perform data processing based on the received processing logic;

[0097] The result packaging unit is used to package the processing results and prepare them for sending to the target terminal.

[0098] In a preferred embodiment, the data interception unit specifically performs the following:

[0099] (1) Generate network-level traffic matching rules based on the "data identifier to be processed" in the subtask;

[0100] (2) Apply the matching rules to the network traffic that flows through it, and copy or redirect the successfully matched data traffic to the execution environment of the local task engine.

[0101] In a preferred embodiment, the processing result is the parallel training result of multiple sub-neural network models, where the training result is the sub-model parameters. During the merging operation, the parameters are merged based on weights and biases, or based on the parameter values ​​of each layer of the neural network. Alternatively, the processing result is the prediction result of multiple sub-neural network models, and during the merging operation, the prediction results of the multiple sub-neural network models are balanced. Alternatively, the processing result is key-value merging, and during the merging operation, the processing result files sorted by user_id are merged using a multi-way merge sort algorithm.

[0102] This invention provides a data migration processing method and system in a data center. The method innovates the traditional end-to-end data migration mode by introducing transmission time estimation and dynamic task partitioning, and transforms it into an in-transit computing mode based on the network topology in the migration task. The system has at least the following effects: (1) By decomposing the data processing task and distributing it to intermediate nodes on the transmission path, deep parallelism between computing and transmission is achieved, which greatly reduces the total time of massive data migration from transmission time to centralized processing time, and significantly improves migration efficiency. (2) The computing load of the target terminal is effectively distributed, and the utilization rate and scalability of the entire system resources are improved by utilizing the idle computing power of the intermediate nodes in the network. (3) Based on a multi-factor (topology, protocol, congestion, distance) accurate transmission time estimation model, a scientific decision-making basis is provided for whether to start the optimization mode, and combined with dynamic task partitioning, the processing capacity is adaptively matched with the network conditions, which enhances the success rate and reliability of the migration task.

[0103] According to one embodiment, a program product of a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations of the various embodiments of this specification. Figure 1 The various operations and functions described. Specifically, a system or apparatus equipped with a readable storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium.

[0104] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0105] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0106] Those skilled in the art will understand that the various embodiments disclosed above can be modified and varied without departing from the spirit of the invention. Therefore, the scope of protection of this specification should be defined by the appended claims.

[0107] It should be noted that not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or they may be jointly implemented by certain components in multiple independent devices.

[0108] In the above embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit or processor may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.

[0109] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0110] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles applicable herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A data migration processing method in a data center, characterized in that, The method comprises the steps of: S100, determining a migration task from a data center, the migration task including data to be migrated, a migration network topology, a target terminal, and a data processing task to be executed; determining a predicted transmission time according to a configuration of the migration network topology, and configuring a migration optimization mode when the predicted transmission time exceeds a time requirement of the data processing task to be executed; S110, determining an intermediate link device distribution based on the migration network topology by the target terminal, dividing the data processing task to be executed into subtasks based on a time difference to be optimized, obtaining a plurality of subtasks corresponding to the intermediate link device distribution, and sending the subtasks to corresponding intermediate link devices; S120, each intermediate link device filters and intercepts the data to be migrated according to the received subtasks, and processes the intercepted data to be migrated according to the requirements of the received subtasks; S130, the intermediate link device sends the processing result to the target terminal, and the target terminal receives the processing result and performs a merging operation; In the step S100, the calculation method of the predicted transmission time T_real according to the configuration of the migration network topology is: T_real = D / (B * U_real) U_real = U_topology * U_protocol * U_contention * U_distance Where D is the total amount of effective data to be migrated, B is the bottleneck link bandwidth in the migration path, U_real is the actual bandwidth utilization, which is determined by the topology factor U_topology, the protocol and transmission factor U_protocol, the congestion contention factor U_contention, and the distance delay factor U_distance, wherein the topology factor U_topology is the maximum available bandwidth ratio determined by the physical and logical topology; the protocol and transmission factor U_protocol is determined by the migration tool and the transmission protocol; the congestion contention factor U_contention is a correction factor when the migration traffic and the production business traffic share the link; and the distance delay factor U_distance is a correction factor for the estimation of a large number of small files or database transactions requiring real-time confirmation.

2. The method of claim 1, wherein, The step S110 comprises the following sub-steps: S111: identifying a calculable node in the migration network topology; S112: quantifying the optimization target, and analyzing the operation primitives of the data processing task to be executed based on the calculable node; S113: dividing the data processing task to be executed into subtasks based on the analyzed operation primitives, and sending the divided subtasks to the calculable nodes in the corresponding intermediate link devices.

3. The method of claim 1, wherein the data migration processing method in the data center is characterized by, In the step S120, the task engine executes the subtasks, including: S121, configuring a task engine through a protocol, the task engine analyzing the subtask when the calculable node receives the subtask request sent by the target terminal, and extracting the data source address and the data identifier to be processed of the subtask; S122, the task engine creates an isolated and secure execution environment; S123, the task engine acquires the original data required by the subtask from the network traffic flowing through the device according to the data source address of the extracted subtask and the to-be-processed data identifier, and imports the original data into the execution environment; S124, the task engine restores the intercepted data to a data object at the application layer or the storage layer based on the imported original data required by the subtask and the received processing logic, and executes the processing logic; S125, the task engine packs the processing result and prepares to send it to the target terminal.

4. The method of claim 3, wherein, The original data required by the subtask is acquired from the network traffic flowing through the device, and the specific method is: (1) According to the "to-be-processed data identifier" in the subtask, it is translated into a matching rule at the network layer; (2) The matched data traffic is copied or redirected to the local task engine.

5. A data migration processing system in a data center, the system comprising: It includes: The task evaluation module is configured to determine the migration task from the data center, the migration task including to-be-migrated data, migration network topology, target terminal, and to-be-executed data processing task; according to the migration network topology configuration, the expected transmission time is determined, and when the expected transmission time exceeds the time requirement of the to-be-executed data processing task, the migration optimization mode is entered; The task division module is configured to determine the intermediate link device distribution based on the migration network topology, divide the to-be-executed data processing task into subtasks based on the to-be-optimized time difference, obtain a plurality of subtasks corresponding to the intermediate link device distribution, and send them to the corresponding intermediate link device; The distributed data processing module is configured to filter and intercept the to-be-migrated data according to the received subtask for each intermediate link device, and process the intercepted to-be-migrated data according to the requirements of the received subtask; The result merging module is configured to send the processing result to the target terminal, and the target terminal receives the processing result and performs merging operation; The calculation method of the task evaluation module for determining the expected transmission time T_real according to the migration network topology configuration is: T_real = D / (B * U_real) U_real = U_topology * U_protocol * U_contention * U_distance Wherein D is the total amount of effective data to be migrated, B is the bottleneck link bandwidth in the migration path, U_real is the actual bandwidth utilization, which is determined by the topology factor U_topology, the protocol and transmission factor U_protocol, the congestion contention factor U_contention, and the distance delay factor U_distance, wherein the topology factor U_topology is the maximum available bandwidth ratio determined by the physical and logical topology; the protocol and transmission factor U_protocol is determined by the migration tool and the transmission protocol; the congestion contention factor U_contention refers to the correction factor when the migration traffic and the production service traffic share the link; the distance delay factor U_distance refers to the estimated correction factor for a large number of small files or database transaction migration requiring real-time confirmation.

6. The system of claim 5, wherein, The task division module specifically comprises: A node identification unit, configured to identify a calculable node in a migration network topology; A task analysis unit, configured to quantify the optimization target, and analyze operation primitives of a to-be-executed data processing task based on the calculable node; A subtask generation unit, configured to divide the to-be-executed data processing task into subtasks based on the analyzed operation primitives, and send the divided subtasks to the calculable node in the corresponding intermediate link device.

7. The system of claim 5, wherein, The distributed data processing module is deployed with a task engine in the intermediate link device, and the module specifically comprises: An information extraction unit, configured to analyze the subtask and extract a data source address and a to-be-processed data identifier when the calculable node receives a subtask request; An environment creation unit, configured to create an isolated and secure execution environment for task execution; A data interception unit, configured to match and intercept the required original data from network traffic flowing through the device according to the extracted data source address and identifier, and import the original data into the execution environment; A data execution unit, configured to restore the intercepted data into a data object at the application layer or the storage layer, and execute data processing based on the received processing logic; A result packaging unit, configured to package the processing result and prepare to send the result to a target terminal.

8. The system of claim 7, wherein, The data interception unit specifically performs: (1) generating a network-level traffic matching rule according to the "to-be-processed data identifier" in the subtask; (2) applying the matching rule to the flowing network traffic, and copying or redirecting the data traffic matched successfully to the execution environment of the local task engine.

Images