An industrial data migration state priority adjustment method of a productivity middle platform

Abstract

The application provides an industrial data migration state priority adjustment method of a production force middle platform, belongs to the technical field of industrial data migration, and solves the problem that key business data migration is delayed due to the inability to dynamically adjust the task priority according to the real-time state in the industrial data migration process.

Description

Technical Field

[0001] This invention belongs to the field of industrial data migration technology, and more specifically, relates to a method for dynamically adjusting the priority of industrial data migration in a productivity platform. Background Technology

[0002] In industrial data migration scenarios within a productivity platform, traditional technologies employ static priority queues to manage data migration tasks. These queues pre-allocate bandwidth resources based on business importance and execute the migration process in a fixed order. However, existing technologies are hampered by network bandwidth fluctuations, storage device performance changes, and resource contention among concurrent tasks. Static priority schemes cannot detect real-time migration progress deviations. When a high-priority task faces deadline risks due to reduced transmission rates, the system continues to occupy bandwidth resources according to the initial allocation scheme, preventing the timely migration of critical business data. In other words, existing technologies suffer from the inability to dynamically adjust task priorities based on real-time status during industrial data migration, leading to delays in the migration of critical business data. Summary of the Invention

[0003] In view of this, the present invention provides a method for dynamically adjusting the priority of industrial data migration in a productivity platform, which can solve the technical problem in the prior art that the inability to dynamically adjust the task priority according to the real-time status during the industrial data migration process leads to the delay of critical business data migration.

[0004] This invention is implemented as follows: It provides a method for dynamically prioritizing industrial data migration in a productivity platform. This includes collecting basic attribute information of all datasets to be migrated in the productivity platform; performing dependency scanning on the datasets to be migrated to construct a directed data dependency graph and using the Tarjan algorithm to detect strongly connected components and identify cyclic dependencies; using micro-benchmarking tools to collect performance characteristics of various storage media at the source and target ends to establish a storage device performance characteristic library; logically partitioning the datasets to be migrated according to the topological hierarchy of the directed data dependency graph and business importance scores, and calculating the data access affinity matrix between each logical partition; and using a data partitioning optimization migration algorithm based on dynamic programming to determine the optimal data access affinity matrix. The initial migration order is determined by inputting the total data volume of each logical shard, business importance score, migration deadline, sequential read / write rate, random read / write rate, access latency, and current network bandwidth status into the migration time estimation model. The model outputs the estimated migration time and resource consumption. A multi-dimensional state space is constructed, and the bandwidth allocation strategy generator outputs dynamic bandwidth quotas. Combined with traffic shaping parameters, burst traffic is controlled. The migration progress of each logical shard is monitored in real time. When the deviation between the actual migration rate and the estimated rate exceeds a threshold, a priority reassessment process is triggered. The migration queue is adjusted according to the urgency index, and the position of each logical shard in the migration queue is dynamically adjusted through periodic global state refresh.

[0005] The basic attribute information includes total data volume, business importance score, migration deadline, data access frequency, and storage medium type.

[0006] The data dependency directed graph uses an adjacency list storage structure to represent foreign key constraints, trigger dependencies, and view associations between data tables. Each node in the graph represents a data table or view, the directed edges represent the direction of the dependency, and the edge weights represent the strength of the dependency.

[0007] The Tarjan algorithm identifies strongly connected components in a directed graph in linear time complexity using depth-first search and a stack structure. Nodes within strongly connected components that form circular dependencies need to be migrated in stages by temporarily disabling constraints.

[0008] The micro-benchmark tool measures the number of input / output operations per second and the average latency of the storage medium in 4KB random read, 4KB random write, 1MB sequential read, and 1MB sequential write modes by initiating input / output request sequences with different block sizes and queue depths.

[0009] The storage device performance characteristic library is stored in a key-value pair structure. The key is a combination of storage medium type identifier and input / output mode, and the value is a quadruple containing sequential read / write rate, random read / write rate, access latency, and standard deviation.

[0010] The data access affinity matrix is ​​a symmetric matrix, and the matrix elements represent the frequency at which two logical shards are accessed simultaneously in a business query. This frequency is obtained by analyzing the table association patterns in the historical query logs.

[0011] The data sharding optimization migration algorithm based on dynamic programming aims to minimize the total migration time and maximize resource utilization. It uses a variant model of the knapsack problem to calculate the optimal sharding scheduling scheme within each time window.

[0012] The migration time prediction model receives a 73-dimensional feature vector as its input layer, followed by three fully connected network layers with 128, 64, and 32 neurons in each layer, and uses a modified linear unit as the activation function.

[0013] The migration time prediction model utilizes the coupling between topology-preserving projection based on self-organizing maps and deep clustering, maintains the neighborhood relationship of high-dimensional data on the low-dimensional grid through competitive learning, and optimizes the clustering objective using deep embedding networks.

[0014] The migration time prediction model introduces a self-organizing map layer after the hidden layer, which contains a 10x10 two-dimensional neuron grid, with each grid neuron maintaining a weight vector with the same dimension as the output of the hidden layer.

[0015] The multi-dimensional state space includes the remaining data volume of each logical shard, the remaining time until the migration deadline, the current network bandwidth utilization rate, and the storage device load rate. Normalization is used to map the values ​​of each dimension to the range of 0 to 1.

[0016] The bandwidth allocation strategy generator adopts a deep Q network architecture. The network input is a multi-dimensional state space vector, and the output is the bandwidth quota allocation ratio of each data migration flow. After normalization processing, it ensures that the sum of the quotas of all data migration flows is equal to the current total available bandwidth.

[0017] The urgency index is equal to the quotient of the product of the remaining data volume of the logical shard divided by the standard data volume, the remaining time before the migration deadline divided by the standard time, and the current available throughput in the storage device performance feature library divided by the standard throughput.

[0018] For cross-data center migration tasks, the BBR transmission protocol is used instead of the standard TCP protocol, and multi-path transmission technology is enabled to utilize multiple physical links in parallel, dynamically adjusting the retransmission timeout parameters according to the packet loss mode.

[0019] Every 5 minutes, a global state refresh is performed to update the remaining data volume of all logical shards, the remaining time until the migration deadline, the current network bandwidth utilization, and the storage device load rate. The migration time estimation model is then re-entered to calculate the updated estimated migration time and resource consumption.

[0020] This invention constructs a multi-dimensional state space to monitor in real time the remaining data volume of each logical shard, the remaining time before the migration deadline, the current network bandwidth utilization, and the storage device load rate. It uses a migration time prediction model to calculate the estimated migration time. When the deviation between the actual migration rate and the predicted rate exceeds a threshold, a priority reassessment process is triggered. The migration queue is dynamically adjusted and bandwidth quotas are reallocated based on urgency indicators. This invention captures migration progress deviations through real-time status monitoring, quantifies task delay risks using urgency indicators, and dynamically adjusts resource allocation through a bandwidth allocation strategy generator. This ensures that high-risk tasks receive additional bandwidth to guarantee timely migration, while low-risk tasks proactively relinquish resources to improve overall resource utilization efficiency. In summary, this invention solves the technical problem mentioned in the background art where the inability to dynamically adjust task priorities based on real-time status during industrial data migration leads to delays in the migration of critical business data. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention.

[0022] Figure 2 This is a schematic diagram of a data-dependent directed graph structure.

[0023] Figure 3 This is a graph showing the relationship between the logically partitioned data volume distribution and affinity.

[0024] Figure 4 This is a flowchart of the multi-dimensional state space normalization process.

[0025] Figure 5 Network performance monitoring graph for cross-data center migration tasks. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0027] like Figure 1 The diagram shown is a flowchart of a method for dynamically adjusting the priority of industrial data migration in a productivity platform provided by this invention. This method includes the following steps:

[0028] S01. Collect basic attribute information of all datasets to be migrated in the productivity platform. The basic attribute information includes total data volume, business importance score, migration deadline, data access frequency, and storage medium type.

[0029] S02. Perform dependency scanning on the dataset to be migrated, construct a directed graph of data dependencies, use the Tarjan algorithm to detect strongly connected components to identify cyclic dependencies, and mark the detected cyclic dependencies as candidate sets for constraint disabling.

[0030] S03. Use micro-benchmark testing tools to collect performance characteristics of various storage media at the source and target ends, record sequential read / write rates, random read / write rates and access latency under different input / output modes, and establish a storage device performance characteristic library.

[0031] S04. Logically partition the dataset to be migrated according to the topological hierarchy of the data dependency directed graph and the business importance score, calculate the data access affinity matrix between each logical partition, and solve the initial migration order by using a data partitioning optimization migration algorithm based on dynamic programming.

[0032] S05. Input the total amount of data, business importance score, migration deadline, sequential read / write rate, random read / write rate, access latency, and current network bandwidth status of each logical shard into the migration time estimation model. The migration time estimation model outputs the estimated migration time and resource consumption for each logical shard.

[0033] S06. Construct a multi-dimensional state space based on the expected migration time and resource consumption. The multi-dimensional state space includes the remaining data volume of each logical shard, the remaining time before the migration deadline, the current network bandwidth utilization rate, and the storage device load rate. Input the multi-dimensional state space into the bandwidth allocation strategy generator. The bandwidth allocation strategy generator outputs the dynamic bandwidth quota of each data migration stream and controls burst traffic in combination with traffic shaping parameters.

[0034] S07. Start the data migration task and monitor the migration progress of each logical shard in real time. When the actual migration rate of a logical shard deviates from the estimated rate corresponding to the expected migration time by more than 15%, the priority re-evaluation process is triggered.

[0035] S08. In the priority re-evaluation process, the urgency index of each logical shard is recalculated. The urgency index is obtained by calculating the remaining data volume, the remaining time before the migration deadline, and the current available throughput in the storage device performance feature library. The migration queue is adjusted in descending order according to the urgency index.

[0036] S09. For cross-data center migration tasks, the BBR transmission protocol is used instead of the standard TCP protocol, and multi-path transmission technology is enabled to utilize multiple physical links in parallel, and the retransmission timeout parameters are dynamically adjusted according to the packet loss mode.

[0037] S10. When a new high-priority migration task is added or the business importance score of an existing task changes, the updated multi-dimensional state space is input again into the bandwidth allocation strategy generator to redistribute the dynamic bandwidth quota, and the initial migration order is recalculated through the data sharding optimization migration algorithm based on dynamic programming.

[0038] S11. During the data migration task, a global status refresh is performed every 5 minutes to update the remaining data volume of all logical shards, the remaining time until the migration deadline, the current network bandwidth utilization, and the storage device load rate. The migration time estimation model is re-entered to calculate the updated estimated migration time and resource consumption. The position of each logical shard in the migration queue is dynamically adjusted according to the urgency index.

[0039] The data dependency directed graph is a directed graph structure that uses an adjacency list storage structure to represent foreign key constraints, trigger dependencies, and view associations between data tables. Each node in the graph represents a data table or view, directed edges represent the direction of dependency, and edge weights represent the strength of dependency. The Tarjan algorithm identifies strongly connected components in the directed graph in linear time complexity using depth-first search and a stack structure. Nodes within strongly connected components form circular dependencies, requiring phased migration by temporarily disabling constraints. The constraint disabling candidate set contains all foreign key constraints and triggers involved in circular dependencies. During the data migration task, constraints in the constraint disabling candidate set are disabled first to complete data transmission, and then re-enabled and data integrity is verified.

[0040] The micro-benchmark tool measures the number of input / output operations per second and average latency of the storage medium in 4KB random read, 4KB random write, 1MB sequential read, and 1MB sequential write modes by initiating input / output request sequences with different block sizes and queue depths. The storage device performance characteristic library uses a key-value pair structure, where the key is a combination of the storage medium type identifier and the input / output mode, and the value is a quadruple containing sequential read / write rate, random read / write rate, access latency, and standard deviation. Performance characteristic collection needs to be repeated under different system load levels to establish a mapping curve between load rate and performance indicators.

[0041] The data access affinity matrix is ​​a symmetric matrix, where each element represents the frequency at which two logical shards are accessed simultaneously in a business query. This matrix is ​​obtained by analyzing table join patterns in historical query logs. The dynamic programming-based data sharding optimization migration algorithm divides the dataset to be migrated into several logical shards and constructs a two-dimensional state transition matrix to represent the migration cost of different combinations of logical shards. Dynamic programming is then used to solve for the optimal sharding granularity and initial migration order. The algorithm first calculates the data access affinity matrix between logical shards based on data access frequency. Then, using a variant of the knapsack problem as the objective function, it calculates the optimal sharding scheduling scheme within each time window, and adapts to changes in business importance scores and resource fluctuations during runtime through a rolling optimization mechanism.

[0042] The specific structure of the migration time prediction model is as follows: the input layer receives a 73-dimensional feature vector, which includes the total amount of data in logical fragments, data compression rate, source storage medium type encoding, target storage medium type encoding, network bandwidth, network latency, number of concurrent migration tasks, historical average transmission rate, sequential read / write rate, random read / write rate, and access latency. After the input layer, three fully connected network layers are connected, with 128, 64, and 32 neurons in each layer, respectively, using modified linear units as activation functions. The output layer contains two neurons that output the estimated migration time and resource consumption, respectively. The migration time prediction model utilizes the coupling between topology-preserving projection and deep clustering based on self-organizing maps, maintains the neighborhood relationship of high-dimensional data on the low-dimensional grid through competitive learning, and optimizes the clustering objective using deep embedding networks to obtain clustering results that are both topologically sound and compact.

[0043] The steps for establishing the training dataset for the migration time prediction model specifically include: collecting data migration task records completed in the past 6 months from the production environment, with each record containing the task's input features and actual completion time and actual resource usage as labels; performing min-max normalization on continuous values ​​in the feature vector; performing one-hot encoding on categorical features; and dividing the training set and validation set into an 8:2 ratio after removing outliers.

[0044] The specific steps for training the transfer time prediction model include: using mean squared error as the loss function, using an adaptive moment estimate optimizer to update parameters, setting the initial learning rate to 0.001, decreasing the learning rate to 0.9 times the original value after every 20 iterations, setting the batch size to 64, evaluating the performance of the transfer time prediction model on the validation set every 5 rounds during training, stopping training early when the validation set loss does not decrease for 10 consecutive rounds, and saving the model parameters with the minimum validation set loss.

[0045] The specific implementation of the coupling between topology-preserving projection and deep clustering based on self-organizing maps in the migration time prediction model is as follows: A self-organizing map layer is introduced after the hidden layer of the migration time prediction model. The self-organizing map layer contains a 10x10 two-dimensional neuron grid. Each neuron in the grid maintains a weight vector with the same dimension as the output of the hidden layer. For each input sample, the best-matching winning neuron is found by calculating the Euclidean distance. The weight vectors of the winning neuron and its neighboring neurons are updated to make them closer to the input sample. The neighborhood range and learning rate gradually decay as the training process progresses. The output of the self-organizing map layer is used as the input of the deep embedding network. The deep embedding network optimizes the clustering structure by minimizing the distance from the sample to the cluster center and maximizing the separation between different clusters. The number of cluster centers is adaptively determined according to the data distribution. The two networks are jointly trained by alternating optimization. The self-organizing map layer ensures that similar data migration tasks are spatially close on the two-dimensional grid, while the deep embedding network aggregates tasks with similar migration characteristics into the same category.

[0046] The technical benefits of coupling topology-preserving projection and deep clustering based on self-organizing maps for migration time prediction models are as follows: The topology-preserving property of self-organizing maps ensures that logical fragments with similar migration characteristics maintain proximity in the low-dimensional projection space, avoiding local structural distortions caused by traditional dimensionality reduction methods. This allows the migration time prediction model to capture migration performance patterns under different storage media type combinations and network conditions. The deep clustering mechanism automatically groups massive historical data migration tasks according to performance characteristics, with each cluster representing a typical migration scenario. When a new data migration task arrives, the migration time prediction model first determines its cluster and then uses historical samples within that cluster for refined prediction. Compared to simple regression models, the coupled structure of topology-preserving projection and deep clustering based on self-organizing maps exhibits stronger generalization ability and prediction stability when facing heterogeneous storage environments and dynamic network conditions. In particular, for edge scenarios not fully covered in the training set, the migration time prediction model can infer reasonable prediction results through topological neighborhood relationships, thereby providing more accurate time estimation support for dynamic priority adjustment and reducing the risk of task delays caused by prediction errors.

[0047] The multi-dimensional state space employs normalization to map the values ​​of each dimension to the range of 0 to 1. The remaining data volume is divided by the initial total data volume of the logical shards, the remaining time until the migration deadline is divided by the total time limit of the task, and the current network bandwidth utilization and storage device load rate are directly calculated as percentages divided by 100. The number of dimensions in the multi-dimensional state space affects the decision quality of the bandwidth allocation strategy generator; too few dimensions lead to insufficient decision-making basis, while too many dimensions increase the computational burden and introduce noise.

[0048] The bandwidth allocation strategy generator employs a deep Q-network architecture. The network input is a multi-dimensional state space vector, and the output is the bandwidth quota allocation ratio for each data migration flow. This bandwidth quota allocation ratio is normalized to ensure that the sum of the quotas for all data migration flows equals the currently available total bandwidth. The attention mechanism weight parameters of the bandwidth allocation strategy generator are dynamically adjusted based on three parameters: the business importance score of the logical shard, the remaining time before the migration deadline, and the current migration completion rate. Specifically, the three parameters are normalized separately and then weighted and summed to calculate the attention weight, with weight coefficients set to 0.5, 0.3, and 0.2, respectively. Traffic shaping parameters include the token bucket capacity and the token generation rate. The token bucket capacity is set to 1.5 times the dynamic bandwidth quota in bytes, and the token generation rate is equal to the dynamic bandwidth quota.

[0049] The urgency index is calculated as follows: The urgency index equals the quotient of the remaining data volume of the logical shard divided by the standard data volume, the remaining time before the migration deadline divided by the standard time, and the current available throughput in the storage device performance characteristic database divided by the standard throughput. Here, the standard data volume is the median of the remaining data volume of all logical shards, the standard time is the median of the remaining time before the migration deadline for all tasks, and the standard throughput is the average throughput of the corresponding storage medium type in the storage device performance characteristic database. An urgency index greater than 2 indicates that the logical shard faces a high risk of latency and needs to be prioritized. An urgency index less than 0.5 indicates that the logical shard has sufficient time to lower its priority and make room for other tasks.

[0050] The BBR transmission protocol periodically probes the network's bottleneck bandwidth and round-trip time (RTD), dynamically adjusting the transmission rate and congestion window size based on the measurement results. This prevents a significant drop in throughput due to packet loss in high-bandwidth, high-delay networks compared to the standard TCP protocol. Multipath transmission technology establishes multiple sub-streams at the transport layer, each transmitting data through different network interfaces or routing paths. The receiving end is responsible for reassembling and out-of-order processing of these sub-streams. Packet loss pattern recognition statistically analyzes the temporal and spatial distribution characteristics of packet loss. Continuous packet loss patterns are identified as congestion-related, requiring a reduction in the transmission rate; discrete, random packet loss is identified as a link error and does not trigger congestion control mechanisms. The retransmission timeout parameter is dynamically adjusted based on the packet loss pattern: it is set to four times the RTD for congestion-related packet loss and twice the RTD for random packet loss.

[0051] The state transition equation of the dynamic programming-based data sharding optimization migration algorithm is expressed as follows: The state function is defined as the minimum cost required to migrate a set of logical shards within a time window. The state transition equation is the minimum sum of the state function of the current time window and the cost of migrating newly added logical shards in the current time window. The cost of adding a new logical shard includes migration time cost and inter-shard affinity loss cost. The migration time cost equals the total data volume of the logical shard divided by the allocated bandwidth. The inter-shard affinity loss cost equals the sum of the elements of the data access affinity matrix between the migrated logical shards and the logical shard to be migrated multiplied by a penalty coefficient. Each cost term is normalized by dividing by its corresponding maximum value. The rolling optimization mechanism re-executes the dynamic programming solution at the end of each time window, incorporating the latest multi-dimensional state space information and changes in business importance scores into the calculation, and updating the initial migration order for subsequent time windows.

[0052] The present invention also provides a method for implementing a dynamic priority adjustment system for industrial data migration of a productivity platform using a computer. The computer is equipped with a storage medium that stores program instructions. When the program instructions are run on the computer, they execute the aforementioned method for dynamic priority adjustment of industrial data migration of a productivity platform.

[0053] The specific implementation methods of the above steps are described in detail below.

[0054] The specific implementation of step S01 is to establish a connection channel with the productivity middle platform database management system through the database metadata query interface, execute system view query statements to obtain the table structure information of all datasets to be migrated, calculate the number of rows or bytes of each dataset as the total data volume through statistical commands, read the business importance score pre-set by the business department from the task configuration table with a value range of 0 to 10, extract the migration deadline from the task scheduling system and convert it into a unified timestamp format, obtain the data access frequency by analyzing the database audit logs of the last 30 days to count the number of queries for each data table, query the device list of the storage management system to determine the physical storage medium type of each dataset, and format and store all the collected basic attribute information in the migration task metadata database. This step provides a complete data source for subsequent priority calculation and resource allocation.

[0055] The specific implementation of step S02 involves querying the database system tables to extract all foreign key constraint definitions, trigger configurations, and view dependency information. The data tables or views are then used as vertices of a graph and assigned unique identifiers. Dependencies are converted into directed edges to construct an adjacency list storage structure. The Taljan algorithm is executed to traverse the directed graph using depth-first search and a stack structure. The algorithm maintains a timestamp array and stack pointers to identify all strongly connected components in a single traversal. Nodes within strongly connected components form closed-loop dependency links. Foreign key constraints and triggers within these strongly connected components are extracted, marked as constraint disabling candidate sets, and indexed. This step automatically identifies circular dependencies using graph theory algorithms to avoid deadlock problems during migration execution.

[0056] The specific implementation of step S03 involves configuring different workload modes using a flexible input / output tester framework, setting parameters such as block size, queue depth, and test duration. For the 4KB random read mode, the queue depth is set to 32, and the number of input / output operations per second and the average latency are recorded continuously for 60 seconds. Independent tests are performed on the 4KB random write, 1MB sequential read, and 1MB sequential write modes respectively. Each test mode is repeated 5 times, and the median is taken to reduce measurement noise. The test is repeated under system load conditions of 20%, 40%, 60%, and 80% to establish a mapping relationship between load rate and performance indicators. The test results are stored in the performance feature library in the form of key-value pairs. The key is composed of the storage medium type identifier and the input / output mode, and the value is a composite data structure containing average throughput, operation latency, latency standard deviation, and measurement timestamp. This step provides a hardware performance parameter benchmark for migration time estimation by accurately measuring the characteristics of the storage device.

[0057] The specific implementation of step S04 involves layering the data tables according to the topological hierarchy of the directed graph of data dependencies to ensure that upstream tables migrate before downstream tables. Within the same layer, the data is sorted by business importance score to form a preliminary logical shard set. The data volume of each logical shard is controlled between 50GB and 200GB. Historical query logs are scanned to extract table association information and count the number of co-occurrences. A data access affinity matrix is ​​constructed, with its element values ​​being the number of co-occurrences of the two tables in the most recent 1000 queries, normalized to the interval between 0 and 1. The initial migration order optimization problem is modeled as a multi-stage decision process, with each stage corresponding to a time window. The state is defined as the set of currently migrated logical shards, and the state transition cost consists of the migration time cost and the affinity loss cost. The state transition matrix is ​​filled from bottom to top using a dynamic programming algorithm, and the optimal migration order is obtained by backtracking. This step decomposes the global optimization problem into computable sub-problems to obtain an approximate optimal solution under time constraints.

[0058] The specific implementation of step S05 involves constructing a 73-dimensional feature vector as model input. The features include the total amount of data in the logical slice, compression ratio, source and target storage media type encoding, network bandwidth and latency, number of concurrent migration tasks, historical average transmission rate, and storage performance parameters queried from the performance feature database. The model uses a fully connected neural network with three hidden layers containing 128, 64, and 32 neurons respectively. A modified linear unit is used as the activation function. The output layer contains two neurons that predict migration time and resource consumption respectively. A 10×10 self-organizing map layer is inserted after the second hidden layer to map the high-dimensional features to a two-dimensional grid space through competitive learning to maintain topological neighborhood relationships. The output of the self-organizing map layer is then input into a deep embedding network to optimize the cluster center distance and inter-class separation, automatically grouping tasks into typical migration scenarios. This step uses a prediction model trained with historical data to provide accurate time and resource estimates for each logical slice.

[0059] The specific implementation of step S06 involves normalizing the remaining data volume of each logical shard by dividing it by the initial total data volume, normalizing the remaining time until the migration deadline by dividing it by the total task time limit, and normalizing the percentage values ​​of the current network bandwidth utilization and storage device load by dividing them by 100. A multi-dimensional state space vector is constructed as the input to the bandwidth allocation strategy generator. The bandwidth allocation strategy generator uses a deep Q network architecture to output the bandwidth quota allocation ratio for each data migration stream. Through normalization processing, it is ensured that the sum of all quotas equals the current total available bandwidth. The attention mechanism weights are dynamically adjusted based on three parameters: business importance score, remaining time, and migration progress completion rate. The weight coefficients are set to 0.5, 0.3, and 0.2, respectively. In the traffic shaping parameters, the token bucket capacity is set to 1.5 times the number of bytes corresponding to the dynamic bandwidth quota, and the token generation rate is equal to the dynamic bandwidth quota. This step achieves dynamic allocation of bandwidth resources and controls burst traffic through deep reinforcement learning.

[0060] The specific implementation of step S07 is to establish a real-time monitoring thread after starting the data migration task, collect the amount of data transferred in each logical shard every 30 seconds and calculate the current migration rate, compare the actual migration rate with the estimated rate predicted in step S05 to calculate the percentage deviation, and trigger the priority reassessment process when the deviation exceeds 15%. The 15% threshold is set based on the statistical analysis of the rate fluctuation of historical migration tasks to ensure that abnormal situations can be responded to in a timely manner while avoiding system jitter caused by frequent adjustments. The monitoring thread also records the changing trends of network bandwidth utilization and storage device load rate. This step ensures that the system can quickly detect the migration progress deviation and start the adaptive adjustment process through the real-time monitoring mechanism.

[0061] The specific implementation of step S08 involves obtaining the current remaining data volume of all logical shards in the priority re-evaluation process, calculating the median of the remaining data volume of all shards as the standard data volume, obtaining the remaining time of all tasks before the migration deadline and calculating the median as the standard time, querying the average throughput of the corresponding storage medium type from the performance feature library as the standard throughput, dividing the remaining data volume of the logical shard by the standard data volume to obtain the normalized remaining amount, dividing the remaining time before the migration deadline by the standard time to obtain the normalized remaining time, dividing the current available throughput by the standard throughput to obtain the normalized throughput, and the urgency index is equal to the normalized remaining amount divided by the product of the normalized remaining time and the normalized throughput. When the urgency index is greater than 2, the priority of the logical shard is increased; when it is less than 0.5, the priority is decreased. The migration queue is rearranged in descending order according to the urgency index. This step achieves dynamic adjustment of priority by quantifying the urgency level to ensure that tasks nearing the deadline are completed first.

[0062] The specific implementation of step S09 involves using a bottleneck bandwidth and round-trip time congestion control algorithm in the transport layer protocol stack to replace the standard transport control protocol for cross-data center migration tasks. This algorithm periodically detects the network's bottleneck bandwidth and round-trip time and dynamically adjusts the transmission rate and congestion window size based on the measurement results. This avoids throughput reduction due to packet loss in high-bandwidth delay-product networks. Multipath transmission technology is enabled to establish multiple sub-streams at the transport layer. Each sub-stream transmits data through different network interfaces or routing paths. The receiving end is responsible for the reassembly and out-of-order processing of sub-streams. By statistically analyzing the temporal and spatial distribution characteristics of packet loss, packet loss patterns are identified. When continuous packet loss is determined to be congestion-related packet loss, the retransmission timeout parameter is set to 4 times the round-trip timeout. When discrete random packet loss is determined to be a link error, the retransmission timeout parameter is set to 2 times the round-trip timeout. This step improves network throughput and reliability for cross-data center migration by optimizing the transport protocol and path utilization.

[0063] The specific implementation of step S10 is to establish a task queue change monitoring mechanism. When a new high-priority migration task is detected or the business importance score of an existing task changes, the remaining data volume of all logical shards, the remaining time until the migration deadline, the current network bandwidth utilization, and the storage device load rate are collected again. The multi-dimensional state space vector is updated and input into the bandwidth allocation strategy generator to recalculate the dynamic bandwidth quota of each data migration stream. At the same time, the updated business importance score and data access affinity matrix are input into the data shard optimization migration algorithm based on dynamic programming. The state transition matrix filling process is re-executed to calculate the new optimal migration order. The recalculated migration order and bandwidth quota are applied to the ongoing migration task to achieve online adjustment. This step ensures that the system can dynamically respond to changes in business needs and the insertion of new tasks to maintain the optimality of global resource allocation.

[0064] The specific implementation of step S11 is to start a timed trigger to execute a global state refresh process every 5 minutes. It iterates through all currently executing logical shards, queries the amount of data that has been transmitted and calculates the amount of data remaining, obtains the current system time and calculates the remaining time until the migration deadline of each task, queries the current network bandwidth utilization and storage device load rate through the system monitoring interface, constructs the updated state information into a multi-dimensional state space vector and inputs it into the migration time prediction model to re-predict the expected migration time and resource consumption of each logical shard, recalculates the urgency index based on the new prediction results, compares the urgency index of each logical shard and adjusts its position in the migration queue in descending order. The 5-minute time interval balances the timeliness of state updates and system overhead. This step ensures that priority adjustment is always based on the latest runtime information through periodic global state refresh, thereby improving the on-time completion rate of migration tasks.

[0065] It should be noted that the key technical approaches include an automatic circular dependency detection and constraint disabling mechanism based on the Tarjan algorithm, a migration time prediction model integrating self-organizing map topology preservation and deep clustering, and a dynamic priority adjustment framework driven by a multi-dimensional state space. The circular dependency detection mechanism identifies strongly connected components and automatically generates constraint disabling policies in linear time using graph theory algorithms. Compared to traditional methods that require manual analysis of dependencies, this mechanism automates dependency processing, eliminates the risk of human error, and shortens the migration preparation cycle. The migration time prediction model maintains the topology of the high-dimensional feature space through self-organizing maps, avoiding the distortion of neighborhood relationships inherent in traditional dimensionality reduction methods. The deep clustering mechanism groups historical samples according to performance characteristics, enabling the model to use corresponding samples for precise predictions in different scenarios. Compared to a single regression model, it exhibits stronger generalization ability in heterogeneous storage environments and dynamic network conditions, providing reliable time estimates for dynamic priority adjustment and reducing the risk of delays caused by prediction bias. The dynamic priority adjustment framework continuously optimizes resource allocation through periodic global state updates and real-time deviation monitoring mechanisms. Compared to traditional static planning methods, it can adapt to runtime business changes and resource fluctuations, significantly improving on-time completion rates. The synergistic effect of these three elements is that circular dependency detection provides feasible topological sorting for data sharding, migration time estimation provides accurate time prediction for each shard, and dynamic adjustment framework integrates dependency constraints, time prediction, and real-time status to continuously optimize task priority. Compared with traditional methods, this synergistic mechanism significantly improves the on-time completion rate and resource utilization efficiency of migration tasks.

[0066] It should be noted that the present invention also solves the following technical problem: the existing technology has insufficient accuracy in predicting migration time when dealing with data migration in heterogeneous storage environments and dynamic network conditions, resulting in unreasonable resource allocation. This invention introduces a self-organizing map layer to maintain a two-dimensional neuron grid in the migration time prediction model. For each input sample, the Euclidean distance is calculated to find the best-matching winning neuron and update the weight vector of neurons in its neighborhood. The output of the self-organizing map layer is used as the input of a deep embedding network to optimize the cluster structure by minimizing the distance from the sample to the cluster center. The two networks are jointly trained through alternating optimization. The self-organizing map layer ensures that similar data migration tasks are spatially close on the two-dimensional grid. The deep embedding network aggregates tasks with similar migration characteristics into the same category. When a new data migration task arrives, the model first determines its cluster and then uses historical samples within the cluster for refined prediction. The topology preservation property avoids the local structure distortion caused by traditional dimensionality reduction methods, enabling the model to capture migration performance patterns under different storage media types and network conditions. For edge scenarios not fully covered in the training set, reasonable prediction results can be inferred through topological neighborhood relationships, thereby providing more accurate time estimation support and reducing the risk of task delays due to prediction errors.

[0067] Specifically, the principle of this invention is as follows: The invention solves the technical problem by establishing a closed-loop feedback mechanism from state perception to decision execution. A multi-dimensional state space unifies scattered migration progress information, time constraints, and resource occupancy status into a structured vector representation, providing complete contextual basis for priority decisions. The migration time prediction model maintains the coupling between topological projection and deep clustering through self-organizing mapping, automatically grouping historical migration tasks according to performance characteristics. When a new task arrives, it uses neighborhood samples based on its cluster for refined prediction, avoiding the insufficient generalization ability of traditional regression models in heterogeneous environments. The urgency index calculates the ratio after normalizing the remaining data volume, remaining time, and available throughput, quantifying the degree of task delay risk. The bandwidth allocation strategy generator outputs dynamic bandwidth quotas based on the urgency index. High-urgency tasks receive additional resource guarantees, while low-urgency tasks release bandwidth to avoid resource waste. Through periodic global state updates and priority reassessment, the system continuously tracks migration progress changes and adjusts resource allocation strategies, forming a dynamically adaptive closed-loop control.

[0068] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0069] The specific implementation methods of steps S01-S03, S07, and S09-S11 are the same as those described above, and will not be repeated in detail here.

[0070] The specific implementation of step S04 is as follows: First, the dataset to be migrated is logically partitioned according to the topological hierarchy of the data dependency directed graph and the business importance score, and the data access affinity matrix between each logical partition is calculated. The data access affinity matrix It is a symmetric matrix, and its elements are... Represents logical fragmentation logical partitioning The frequency of simultaneous access in business queries, dimensionless, where Number the first logical partition. The second logical shard number is obtained by analyzing table join patterns in historical query logs. A two-dimensional state transition matrix is ​​constructed based on a dynamic programming-based data sharding optimization migration algorithm to represent the migration cost of different logical shard combinations. The state function is defined as follows: , indicating within the time window Inward migration logic fragment set The minimum cost required is expressed by the state transition equation as follows:

[0071] ;

[0072] In the formula, For time window Inward migration logic fragment set The minimum cost, dimensionless; The time window is numbered, and is dimensionless. For logically partitioned sets; Add a new logical slice number for the current time window; For the previous time window Inward migration logic fragment set The minimum cost, dimensionless; Logical partitioning The migration time cost is dimensionless; The cost of affinity loss between fragments is dimensionless. This includes the migration time cost. The calculation formula is expressed as follows:

[0073] ;

[0074] In the formula, Logical partitioning The migration time cost is dimensionless; Logical partitioning Total data volume, in GB; To allocate to logical partitions The bandwidth is measured in Mbps. The maximum total data volume across all logical partitions, in GB; The minimum allocatable bandwidth for the system, in Mbps. Inter-fragment affinity penalty. The calculation formula is expressed as follows:

[0075] ;

[0076] In the formula, The cost of loss of affinity between fragments is dimensionless; The penalty coefficient is dimensionless and has an empirical value of 0.3. Number the migrated logical fragments; Logical partitioning logical partitioning The data access affinity matrix element values ​​are dimensionless. Let be the maximum element value in the data access affinity matrix, which is dimensionless. The initial migration order of each logical partition is obtained by solving dynamic programming.

[0077] The specific implementation of step S05 is to input a 73-dimensional feature vector into the migration time prediction model, which consists of the total amount of data in each logical shard, the business importance score, the migration deadline, the sequential read / write rate, the random read / write rate, the access latency, and the current network bandwidth status in the storage device performance feature library. After receiving the feature vector, the input layer of the migration time prediction model extracts features through a three-layer fully connected network. The number of neurons in each layer is 128, 64, and 32, respectively. The modified linear unit is used as the activation function. The output layer contains two neurons that output the estimated migration time and resource consumption, respectively.

[0078] The specific implementation of step S06 involves constructing a multi-dimensional state space based on the expected migration time and resource consumption. This multi-dimensional state space undergoes normalization processing, and the formula for calculating the normalized value of the remaining data is as follows:

[0079] ;

[0080] In the formula, The normalized value of the remaining data is dimensionless. The remaining data volume of the logical partition, in GB; The initial total data volume for the logical partitions, in GB. The formula for calculating the normalized value of the remaining time until the migration deadline is as follows:

[0081] ;

[0082] In the formula, This is the normalized value of the remaining duration, which is dimensionless. The remaining time until the migration deadline, in hours; This represents the total task duration, in hours. The formula for calculating the normalized value of current network bandwidth utilization is as follows:

[0083] ;

[0084] In the formula, This is a normalized value of network bandwidth utilization, dimensionless. This represents the current network bandwidth utilization percentage, dimensionless. The formula for calculating the normalized value of storage device load rate is as follows:

[0085] ;

[0086] In the formula, This is a dimensionless, normalized value of the storage device load rate. This represents the percentage of storage device load, a dimensionless value. It represents a multi-dimensional state space vector. The input bandwidth allocation policy generator uses a deep Q network architecture to output the bandwidth quota allocation ratio for each data migration flow. After normalization, the bandwidth quota allocation ratio ensures that the sum of the quotas for all data migration flows equals the currently available total bandwidth. Attention weights are also included. The calculation formula is expressed as follows:

[0087] ;

[0088] In the formula, Attention weights are dimensionless. The business importance of logical sharding is scored, dimensionless; Score the maximum business importance among all logical shards, dimensionless; This represents the current migration progress as a percentage of completion, dimensionless. Token bucket capacity. The calculation formula is expressed as follows:

[0089] ;

[0090] In the formula, This represents the token bucket capacity, in bytes. This represents the number of bytes corresponding to the dynamic bandwidth quota, in bytes. Token generation rate. Equal to dynamic bandwidth quota The unit is bytes per second.

[0091] The specific implementation of step S08 involves recalculating the urgency index of each logical fragment during the priority re-evaluation process. The formula for calculating the urgency index is as follows:

[0092] ;

[0093] In the formula, This is an indicator of urgency, and is dimensionless. The remaining data volume of the logical partition, in GB; This represents the median amount of remaining data across all logical shards, in GB. The remaining time until the migration deadline, in hours; This represents the median remaining time, in hours, for all tasks until the migration deadline. The current available throughput in the storage device performance characteristics library, in MBps; The average throughput, in MBps, is the corresponding storage medium type in the storage device performance characteristic library. The migration queue is adjusted in descending order of urgency. This indicates that logical sharding faces a high latency risk and needs to be prioritized. This indicates that the logical partition has sufficient time slack to lower its priority and make room for other tasks to release resources.

[0094] The specific implementation of the coupling between topology-preserving projection and deep clustering based on self-organizing maps in the migration time prediction model involves introducing a self-organizing map layer after the hidden layer of the migration time prediction model. This self-organizing map layer contains a 10x10 two-dimensional neuron grid, with each neuron maintaining a weight vector of the same dimension as the hidden layer output. For each input sample, the best-matching winning neuron is found by calculating the Euclidean distance. The formula for calculating the Euclidean distance is as follows:

[0095] ;

[0096] In the formula, For input samples and neurons The Euclidean distance between them is in the same units as the output dimension of the hidden layer; Number the neurons; For the first input sample Each dimension component; For neurons The weight vector of the first Each dimension component; For dimension indexing; Dimension is the number of dimensions in the hidden layer output; it is dimensionless and typically takes the value 32. The weight vectors of the winning neuron and its neighboring neurons are updated to move them closer to the input sample. The weight update formula is as follows:

[0097] ;

[0098] In the formula, For neurons In training rounds The weight vector of the first Each dimension component; For neurons In training rounds The weight vector of the first Each dimension component; Number the training rounds; For training rounds The learning rate is dimensionless, with an initial value of 0.5, and decreases as training progresses. For training rounds Winning neurons For neurons The neighborhood function value is dimensionless; Number the winning neuron; For the first input sample Each dimension component. Neighborhood function. The calculation formula is expressed as follows:

[0099] ;

[0100] In the formula, For training rounds Winning neurons For neurons The neighborhood function value is dimensionless; For the winning neuron Position coordinates in a two-dimensional grid, dimensionless; For neurons Position coordinates in a two-dimensional grid, dimensionless; For neurons With neurons The Euclidean distance between them is dimensionless. For training rounds The neighborhood radius parameter is dimensionless, initially set to 5, and decays to 1 as training progresses. The output of the self-organizing map layer serves as the input to the deep embedding network, which optimizes the clustering structure by minimizing the distance from samples to cluster centers and maximizing the separation between different clusters. The number of cluster centers is adaptively determined based on the data distribution.

[0101] To better understand and implement this invention, the following is a specific application scenario of this invention, Example 2:

[0102] A technical team was responsible for a data migration project for a productivity platform system. This platform comprises four core business modules: an order management system, an inventory management system, a customer relationship management system, and a financial settlement system. The project required migrating a total of 2.8TB of production data from an old local storage cluster to a newly built hybrid cloud storage environment. The migration task involved 127 data tables, containing complex foreign key constraints and trigger dependencies. The migration deadline was 72 hours, and the business systems needed to remain continuously running during the migration process.

[0103] The technical team first collected basic attribute information for the datasets to be migrated. The order management system dataset totaled 850GB, with a business importance score of 9.2, a migration deadline of 24 hours, a data access frequency of 1200 queries per second, and was stored on solid-state drives (SSDs). The inventory management system dataset totaled 620GB, with a business importance score of 8.7, a migration deadline of 36 hours, a data access frequency of 850 queries per second, and was stored on hard disk drives (HDDs). The customer relationship management system dataset totaled 730GB, with a business importance score of 8.1, a migration deadline of 48 hours, a data access frequency of 680 queries per second, and was stored on solid-state drives (SSDs). The financial settlement system dataset totaled 600GB, with a business importance score of 9.5, a migration deadline of 18 hours, a data access frequency of 950 queries per second, and was stored on NVMe SSDs.

[0104] The technical team performed a dependency scan on the dataset to be migrated, constructing a directed graph of data dependencies. For example... Figure 2 As shown, there are foreign key constraints between the order table and the customer table. The order details table depends on the order table and the product table. The inventory change table is linked to the order details table via triggers. The financial reconciliation view is linked to the order table and the payment record table. When using the Tarjan algorithm to detect strongly connected components, a circular dependency was found between the order table, the inventory change table, and the main inventory table. Therefore, the foreign key constraints of these three tables need to be marked as constraint disable candidates. During the migration process, the constraints are first disabled to complete the data transfer, and then re-enabled to verify data integrity.

[0105] The technical team used micro-benchmarking tools to collect performance characteristics of various storage media on both the source and target sides. The source SSD achieved 75,000 I / O operations per second (IOP / S) and an average latency of 0.13 milliseconds in 4KB random read mode, with a throughput of 520 MB / s in 1MB sequential read mode. The source HDD achieved 180 I / O operations per second (IOP / S) and an average latency of 5.6 milliseconds in 4KB random read mode, with a throughput of 150 MB / s in 1MB sequential read mode. The target NVMe SSD achieved 120,000 I / O operations per second (IOP / S) and an average latency of 0.08 milliseconds in 4KB random write mode, with a throughput of 680 MB / s in 1MB sequential write mode. Performance characteristic collection was repeatedly performed under different system load levels to establish a mapping curve between load rate and performance indicators. The storage device performance characteristic library recorded 48 sets of performance data for 12 storage media types and 4 I / O modes.

[0106] The technical team logically partitioned the dataset to be migrated according to the topological hierarchy of the directed graph of data dependencies and business importance scores, dividing the 127 data tables into 35 logical partitions. For example... Figure 3 As shown, logical shard 1 contains 280GB of data, including the order table and order details table; logical shard 2 contains 150GB of data, including the customer table and customer contact table; and logical shard 3 contains 420GB of data, including the inventory master table and inventory change table. The technical team analyzed table association patterns in historical query logs to obtain a data access affinity matrix. The affinity values ​​for logical shard 1 and logical shard 2 are 0.78, for logical shard 1 and logical shard 3 are 0.65, and for logical shard 2 and logical shard 3 are 0.42. A dynamic programming-based data sharding optimization migration algorithm was used, with the objective function of minimizing the total migration time and maximizing resource utilization. A variant of the knapsack problem was used to calculate the optimal sharding scheduling scheme within each time window. The initial migration order was determined to be: first migrate the high-priority logical shards of the financial settlement system, then migrate the logical shards of the order management system and inventory management system in parallel, and finally migrate the logical shards of the customer relationship management system.

[0107] The technical team input the total data volume of each logical shard, business importance score, migration deadline, sequential read / write rate, random read / write rate, access latency, and current network bandwidth status from the storage device performance feature library into the migration time prediction model. The input layer of the migration time prediction model receives a 73-dimensional feature vector, followed by a three-layer fully connected network with 128, 64, and 32 neurons in each layer, using a modified linear unit as the activation function. The model training dataset consists of 482 data migration task records completed within the past six months. Continuous values ​​in the feature vector are processed using min-max normalization, categorical features are one-hot encoded, and outliers are removed before the training and validation sets are divided in an 8:2 ratio. As shown in Table 1, the migration time prediction model outputs an estimated migration time of 3.2 hours for logical shard 1, with a resource consumption of 450MB bandwidth and 65% storage device load, and an estimated migration time of 1.8 hours for logical shard 2, with a resource consumption of 280MB bandwidth and 42% storage device load.

[0108] Table 1. Estimated migration time and resource consumption for typical logical partitions

[0109]

[0110] The technical team constructed a multi-dimensional state space, normalizing and mapping the remaining data volume of each logical shard, the remaining time until the migration deadline, the current network bandwidth utilization, and the storage device load rate to a range of 0 to 1. For example... Figure 4 As shown, the remaining data volume of logical shard 1 is 280GB, divided by the initial total data volume of 280GB, yielding a normalized value of 1.0. The remaining time until the migration deadline is 24 hours, divided by the total time limit of 24 hours, also yielding a normalized value of 1.0. The current network bandwidth utilization rate is 35%, divided by 100, yielding a normalized value of 0.35. The storage device load rate is 48%, divided by 100, yielding a normalized value of 0.48. A multi-dimensional state space input bandwidth allocation strategy generator is used, employing a deep Q network architecture. The network input is a multi-dimensional state space vector, and the output is the bandwidth quota allocation ratio for each data migration stream. The data migration stream of the financial settlement system receives a dynamic bandwidth quota of 550MB / s, the data migration stream of the order management system receives a dynamic bandwidth quota of 420MB / s, the data migration stream of the inventory management system receives a dynamic bandwidth quota of 380MB / s, and the data migration stream of the customer relationship management system receives a dynamic bandwidth quota of 250MB / s. In the traffic shaping parameters, the token bucket capacity is set to 1.5 times the number of bytes corresponding to the dynamic bandwidth quota. The token bucket capacity of the financial settlement system is 825MB, and the token generation rate is equal to the dynamic bandwidth quota of 550MB / s.

[0111] The technical team initiated the data migration task and monitored the migration progress of each logical shard in real time. Eight hours after the migration started, the actual migration rate of logical shard 1 in the order management system was detected to be 320MB / s, while the estimated migration rate corresponding to the expected migration time was 380MB / s. The deviation between the actual and estimated migration rates was 15.8%, exceeding the 15% threshold and triggering a priority reassessment process. In the priority reassessment process, the urgency index of each logical shard was recalculated. Logical shard 1 had 142GB of remaining data, 16 hours remaining until the migration deadline, and the current available throughput in the storage device performance characteristic database was 480MB / s. The standard data volume was taken as the median of the remaining data volume of all logical shards, which was 168GB; the standard duration was taken as the median of the remaining time until the migration deadline of all tasks, which was 22 hours; and the standard throughput was taken as the average throughput of the corresponding storage media type in the storage device performance characteristic database, which was 420MB / s. The calculated urgency index for logical shard 1 was 2.18, which is greater than the threshold of 2, indicating a high risk of latency and requiring an increase in priority. The technical team adjusted the migration queue in descending order of urgency, raising the priority of logical shard 1 from 3rd to 1st. The updated multi-dimensional state space was then input into the bandwidth allocation policy generator to redistribute the dynamic bandwidth quota, increasing the dynamic bandwidth quota of logical shard 1 from 420MB / s to 580MB / s.

[0112] The technical team adopted the BBR transport protocol instead of the standard TCP protocol for cross-datacenter migration tasks. The BBR transport protocol dynamically adjusts the sending rate and congestion window size by periodically probing the network's bottleneck bandwidth and round-trip latency. For example... Figure 5 As shown, during the 12th to 15th hour after the migration started, the network experienced intermittent packet loss, with the packet loss rate increasing from 0.08% to 0.32%. The technical team implemented multipath transmission technology, utilizing three physical links in parallel. Each sub-stream transmitted data through a different network interface, and the receiving end was responsible for reassembling and out-of-order processing of the sub-streams. Packet loss pattern recognition, through statistical analysis of the temporal and spatial distribution characteristics of packet loss, determined that it was discrete random packet loss due to link errors and did not trigger congestion control mechanisms. The retransmission timeout parameter was set to twice the round-trip delay, i.e., 180 milliseconds.

[0113] During the data migration task, the technical team performed a global status refresh every 5 minutes, updating the remaining data volume of all logical shards, the remaining time before the migration deadline, the current network bandwidth utilization, and the storage device load rate. The updated estimated migration time and resource consumption were then recalculated using the migration time estimation model. Twenty hours after the migration started, a new high-priority migration task was added to the system. This task involved migrating historical transaction data required for the urgent generation of financial statements, totaling 180GB, with a business importance score of 9.8 and a migration deadline of 4 hours. The technical team dynamically adjusted the position of each logical shard in the migration queue based on the urgency index, inserting the new task at the top of the queue. The initial migration order was recalculated using a data sharding optimization migration algorithm based on dynamic programming. A dynamic bandwidth quota of 650MB / s was allocated to the new task, while the bandwidth quotas for other low-urgency tasks were reduced to ensure that the total bandwidth did not exceed the limit.

[0114] The technological advancement of this invention compared to traditional static priority scheduling methods lies in the closed-loop feedback control formed by real-time status monitoring and dynamic priority adjustment mechanisms. Traditional methods pre-allocate fixed bandwidth quotas based on business importance before the migration task starts, failing to detect network fluctuations, storage device performance changes, and resource competition among concurrent tasks during the migration process. This results in high-priority tasks still occupying resources according to the initial allocation scheme even when facing deadline risks due to decreased transmission rates. This invention captures real-time changes in remaining data volume, remaining duration, network bandwidth utilization, and storage device load rate through a multi-dimensional state space. It accurately predicts migration time using a migration time prediction model combined with a coupling mechanism of topology-preserving projection of self-organizing mapping and deep clustering. When the deviation between the actual migration rate and the predicted rate exceeds a threshold, a priority reassessment process is triggered. The task delay risk is quantified based on urgency indicators, and the migration queue and bandwidth quota allocation are dynamically adjusted. This ensures that high-risk tasks receive additional resources in a timely manner to complete the migration on time, while low-risk tasks proactively release bandwidth to improve overall resource utilization efficiency. This solves the technical problem of delays in the migration of critical business data during industrial data migration due to the inability to dynamically adjust task priorities based on real-time status.

[0115] It should be noted that the variables involved in this invention are explained in detail in Tables 2 and 3.

[0116] Table 2. Variable Explanation Table (Part 1)

[0117]

[0118] Table 3. Variable Explanation Table (Part Two)

[0119]

[0120] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for dynamically adjusting the priority of industrial data migration in a productivity platform, characterized in that, This includes collecting basic attribute information of all datasets to be migrated in the productivity platform; performing dependency scanning on the datasets to be migrated to construct a directed graph of data dependencies and using the Tarjan algorithm to detect strongly connected components and identify cyclic dependencies; using micro-benchmarking tools to collect performance characteristics of various storage media on the source and target sides to establish a storage device performance characteristic library; and solving the initial migration order. The total data volume of each logical shard, business importance score, migration deadline, sequential read / write rate, random read / write rate, access latency, and current network bandwidth status in the storage device performance characteristic library are input into the migration time estimation model to output the estimated migration time and resource consumption. A multi-dimensional state space is constructed based on the estimated migration time and resource consumption. The multi-dimensional state space includes the remaining data volume of each logical shard, the remaining time until the migration deadline, the current network bandwidth utilization rate, and the storage device load rate. The multi-dimensional state space is input into the bandwidth allocation strategy generator. The bandwidth allocation strategy generator outputs the dynamic bandwidth quota for each data migration stream and controls burst traffic by combining traffic shaping parameters. The migration progress of each logical shard is monitored in real time. When the actual migration rate deviates from the estimated rate by more than a threshold, the priority reassessment process is triggered. The migration queue is adjusted according to the urgency index, and the position of each logical shard in the migration queue is dynamically adjusted through periodic global status refresh. The data dependency directed graph uses an adjacency list storage structure to represent the foreign key constraints, trigger dependencies, and view associations between data tables. Each node in the graph represents a data table or view, the directed edges represent the dependency direction, and the edge weights represent the dependency strength. The initial migration order is determined by first logically partitioning the dataset to be migrated according to the topological hierarchy of the data dependency directed graph and the business importance score, and then calculating the data access affinity matrix between each logical partition. The data access affinity matrix For a symmetric matrix, its elements Represents logical fragmentation logical partitioning The frequency of simultaneous access in business queries, dimensionless, where Number the first logical partition. Number the second logical partition; construct a two-dimensional state transition matrix based on the dynamic programming-based data partitioning optimization migration algorithm to represent the migration cost of different logical partition combinations, and define the state function as follows: , indicating within the time window Inward migration logic fragment set The minimum cost required is expressed by the state transition equation as follows: ； In the formula, For time window Inward migration logic fragment set The minimum cost, dimensionless; The time window is numbered, and is dimensionless. For logically partitioned sets; Add a new logical slice number for the current time window; For the previous time window Inward migration logic fragment set The minimum cost, dimensionless; Logical partitioning The migration time cost is dimensionless; The cost of inter-fragment affinity loss is dimensionless; among which, the migration time cost is... The calculation formula is expressed as follows: ； In the formula, Logical partitioning The migration time cost is dimensionless; Logical partitioning Total data volume, in GB; To allocate to logical partitions The bandwidth is measured in Mbps. The maximum total data volume across all logical partitions, in GB; This represents the minimum allocatable bandwidth of the system, measured in Mbps. Cost of inter-fragment affinity The calculation formula is expressed as follows: ； In the formula, The cost of loss of affinity between fragments is dimensionless; The penalty coefficient is dimensionless and has an empirical value of 0.

3. Number the migrated logical fragments; Logical partitioning logical partitioning The data access affinity matrix element values ​​are dimensionless. The maximum element value in the data access affinity matrix is ​​dimensionless; the initial migration order of each logical partition is obtained by solving dynamic programming. The formula for calculating the urgency index is as follows: ； In the formula, This is an indicator of urgency, and is dimensionless. The remaining data volume of the logical partition, in GB; The median amount of remaining data across all logical shards, in GB; The remaining time until the migration deadline, in hours; This represents the median remaining time, in hours, for all tasks until the migration deadline. The current available throughput in the storage device performance characteristics library, in MBps; This represents the average throughput of the corresponding storage medium type in the storage device performance characteristic library, expressed in MBps.

2. The method according to claim 1, characterized in that, The basic attribute information includes total data volume, business importance score, migration deadline, data access frequency, and storage medium type.

3. The method according to claim 2, characterized in that, The Tarjan algorithm identifies strongly connected components in a directed graph in linear time complexity using depth-first search and a stack structure. Nodes within strongly connected components that form circular dependencies need to be migrated in stages by temporarily disabling constraints. Specifically, the constraint disabling candidate set contains all foreign key constraints and triggers involved in the circular dependencies. During the data migration task, the constraints in the constraint disabling candidate set are disabled first to complete the data transmission, and then the constraints are re-enabled and the data integrity is verified.

4. The method according to claim 3, characterized in that, The micro-benchmark tool measures the number of input / output operations per second and average latency of the storage medium in 4KB random read, 4KB random write, 1MB sequential read, and 1MB sequential write modes by initiating input / output request sequences with different block sizes and queue depths.

5. The method according to claim 4, characterized in that, The storage device performance characteristic library is stored in a key-value pair structure. The key is a combination of storage medium type identifier and input / output mode, and the value is a quadruple containing sequential read / write rate, random read / write rate, access latency, and standard deviation.

6. The method according to claim 5, characterized in that, The data access affinity matrix is ​​a symmetric matrix, and the matrix elements represent the frequency at which two logical shards are accessed simultaneously in business queries. This frequency is obtained by analyzing the table association patterns in historical query logs.

7. The method according to claim 6, characterized in that, The input layer of the migration time prediction model receives a 73-dimensional feature vector, followed by three fully connected network layers with 128, 64, and 32 neurons in each layer, and uses a modified linear unit as the activation function.

8. The method according to claim 7, characterized in that, The migration time prediction model utilizes the coupling between topology-preserving projection based on self-organizing maps and deep clustering. It maintains the neighborhood relationships of high-dimensional data on low-dimensional grids through competitive learning, while using deep embedding networks to optimize the clustering objective.

Images