Method and system for smooth migration of scheduling tasks and data collaboration based on big data platform

Abstract

The present invention provides a method and system for smooth migration of scheduling tasks and data based on a big data platform, which relates to the field of big data technology, including building a digital twin environment of a target system, collecting task operation status data of a source system, and extracting task features using a deep neural network and performing clustering grouping. The task group is deployed to the digital twin environment, and the task association relationship and resource competition relationship are analyzed based on a graph neural network, and a task stability score is generated in combination with task operation indicators. A migration risk prediction model is constructed to quantitatively evaluate the migration risk, and the migration priority and execution sequence are determined accordingly. Corresponding resource configuration, data synchronization and fault recovery plans are formulated for each task group, and migration drills are performed in a digital twin environment, and the migration plan is optimized using a reinforcement learning algorithm. The present invention can effectively reduce migration risks, ensure the smoothness and stability of the migration process, and improve migration efficiency.

Description

Technical Field

[0001] The present invention relates to big data technology, and in particular to a method and system for scheduling tasks and collaborative smooth migration of data based on a big data platform. Background Art

[0002] The emergence of big data platforms enables enterprises to process and analyze massive amounts of data to gain valuable insights. As business grows and technology develops, enterprises need to continuously upgrade and migrate big data platforms to adapt to new needs and challenges. For example, an enterprise may need to migrate data and tasks from an old cluster to a new cluster, or from a local data center to a cloud platform. This migration process usually involves a large amount of scheduling tasks and data, which, if not handled properly, may lead to service interruptions, data loss, or performance degradation.

[0003] At present, the migration methods of big data platforms mainly rely on manual operations and experience judgment, lacking automated and intelligent tool support. Existing migration solutions usually do not fully consider the dependencies between tasks and the data flow path, resulting in problems such as data inconsistency or task execution failure during the migration process. In addition, existing migration solutions also lack an assessment and prediction mechanism for migration risks, making it difficult to effectively identify and avoid potential risks.

[0004] Lack of in-depth analysis of the complex relationship between tasks and data: Existing migration solutions usually treat tasks and data as independent entities, without fully considering the dependencies between them and the data flow path, which leads to problems such as data inconsistency or task execution failure during the migration process.

[0005] Incomplete migration risk assessment: Existing migration solutions lack a comprehensive assessment of migration risks, making it difficult to effectively identify and avoid potential risks, such as resource contention, network latency, and data corruption.

[0006] Lack of automated and intelligent migration solutions: The current migration process mainly relies on manual operations and experience judgment, which is inefficient and prone to errors. Lack of automated and intelligent tool support makes it difficult to cope with migration needs in large-scale and complex scenarios. Summary of the invention

[0007] The embodiments of the present invention provide a method and system for scheduling tasks and coordinating and smoothly migrating data based on a big data platform, which can solve the problems in the prior art.

[0008] According to a first aspect of the embodiments of the present invention,

[0009] Provides a method for smooth migration of tasks and data collaboration based on the big data platform, including:

[0010] A digital twin environment of the target system is constructed based on the physical topology and resource configuration parameters of the source system, and a task status collection probe is deployed in the digital twin environment; the running status data of the scheduled tasks in the source system are collected, and the running status data includes task execution time, resource occupancy rate, data flow path, task dependency, and fault occurrence frequency; a deep neural network is used to extract features from the running status data and a task feature vector matrix is ​​constructed; the scheduled tasks are clustered and grouped according to the task feature vector matrix to generate a task group list with a unique identifier;

[0011] Deploy the scheduled tasks in the task group list to the digital twin environment, and record the task operation indicators in real time through the task status collection probe; build a task association analysis model based on the graph neural network to identify the data flow relationship and resource competition relationship between tasks, and compare and analyze the task operation indicators with the operation status data in the source system to generate a task stability score; build a migration risk prediction model based on the task stability score and the analysis results of the task association analysis model, and quantitatively evaluate the migration risk of each task group based on the migration risk prediction model to generate a migration risk assessment report;

[0012] Based on the migration risk assessment report, a migration priority score is calculated for each task group, and the task groups are sorted according to the migration priority scores to generate a migration execution sequence; for each task group in the migration execution sequence, a corresponding resource allocation plan, data synchronization strategy and fault recovery plan are formulated in combination with its migration risk assessment result; migration drills are performed in the digital twin environment and abnormal events and handling measures are recorded, and a reinforcement learning algorithm is used to continuously optimize the migration risk prediction model and migration execution plan according to the abnormal events and handling measures, and finally a verified optimal migration plan is formed.

[0013] A task association analysis model is built based on a graph neural network to identify the data flow relationship and resource competition relationship between tasks. At the same time, the task operation indicators are compared and analyzed with the operation status data in the source system to generate a task stability score including:

[0014] Based on the running status data, a task association graph is constructed, wherein the nodes in the task association graph represent the scheduled tasks, and the edges in the task association graph represent the association relationship between tasks; based on the resource occupancy rate, the resource competition between tasks is analyzed to generate a resource competition matrix; according to the data flow path, the data flow frequency and data flow volume between tasks are calculated to generate a data flow matrix; the data flow matrix and the resource competition matrix are merged to form a task association weight matrix;

[0015] The task association weight matrix is ​​input into a graph neural network, wherein the graph neural network includes multiple graph attention layers, and each graph attention layer calculates attention weights based on node features and edge features; the graph neural network is used to extract association patterns between tasks, identify key task paths and task dependency chains, and generate task association features; a task association analysis model is constructed based on the task association features, and the task association analysis model is used to predict data flow bottlenecks and resource competition conflicts between tasks;

[0016] Deploy task status collection probes in the digital twin environment to collect task operation indicators in the target environment in real time; compare the task operation indicators with the operation status data in the source system, and calculate the task execution timing deviation, performance indicator difference and abnormal event probability; analyze the propagation impact of task operation anomalies on associated tasks based on the prediction results of the task association analysis model; comprehensively calculate the task stability score based on the task execution timing deviation, the performance indicator difference, the abnormal event probability and the abnormal propagation impact.

[0017] According to the task stability score and the analysis results of the task association analysis model, a migration risk prediction model is constructed, and the migration risk of each task group is quantitatively evaluated based on the migration risk prediction model. The generated migration risk assessment report includes:

[0018] Acquire task stability features, associated propagation features, and resource-sensitive features as an initial feature set, and build a migration risk prediction model based on the task stability score and the task association analysis model. The migration risk prediction model includes a feature fusion layer, a time series modeling layer, and a risk prediction layer. The feature fusion layer uses an attention mechanism to adaptively weight the features in the initial feature set to obtain a weighted feature vector.

[0019] The weighted feature vector is input into the time series modeling layer, and the time series modeling layer uses a long short-term memory network to extract risk evolution characteristics. The long short-term memory network selectively retains long-term influencing factors and short-term volatility factors through a gating mechanism; the risk prediction layer includes a gradient boosting tree model and a deep neural network, and the risk evolution characteristics are simultaneously input into the gradient boosting tree model and the deep neural network. The gradient boosting tree model processes the explicit risk characteristics to obtain a first prediction result, and the deep neural network extracts the implicit risk pattern to obtain a second prediction result; the first prediction result and the second prediction result are weightedly fused to obtain a comprehensive risk prediction value;

[0020] Based on the comprehensive risk prediction value, the risk status of a single task is evaluated, and a weighted combination of task criticality, execution complexity, and failure impact is calculated to obtain a task-level risk score; a task risk association network is constructed to analyze the risk propagation relationship between tasks, and the sensitivity of tasks to system resource fluctuations is evaluated; the task-level risk score, the risk propagation relationship, and the sensitivity of resource fluctuations are combined to generate a task-level risk assessment result;

[0021] Divide tasks with similar risk characteristics into groups, aggregate the task-level risk assessment results within the same group, and calculate the overall risk value of the group based on the correlation between tasks within the group; establish a dependency graph between groups to analyze the cross-group risk propagation path; perform weighted adjustment on the overall risk value of the group according to the business importance of the group to generate a group-level risk assessment result;

[0022] Construct a system-level risk propagation graph, use the group-level risk assessment results as the node initial state, and the risk propagation relationship between groups as the edge weight; perform risk diffusion simulation on the system-level risk propagation graph to calculate the cumulative effect of risk in the system; evaluate the overall risk tolerance of the system based on the risk diffusion results, and identify high-risk concentration areas; hierarchically integrate the task-level risk assessment results, the group-level risk assessment results, and the system risk tolerance assessment results to generate a migration risk assessment report.

[0023] Based on the comprehensive risk prediction value, the risk status of a single task is evaluated, and the weighted combination of task criticality, execution complexity, and failure impact is calculated to obtain a task-level risk score including:

[0024] Establishing a risk assessment benchmark based on the comprehensive risk prediction value, and using the risk assessment benchmark as a reference threshold for calculating the mission criticality, the execution complexity, and the impact of failure;

[0025] Constructing a task dependency graph, the task dependency graph includes a node set and an edge set, the node set represents a task set, and the edge set represents a dependency relationship between tasks; mapping the comprehensive risk prediction value to the task dependency graph as an initial weight of a node; calculating the criticality of the task using a PageRank algorithm based on the task dependency graph, the criticality of the task is calculated in an iterative manner, and the criticality value of the task in each iteration is obtained by combining the weighted sum of the ratio of the criticality value of the adjacent tasks to the out-degree adjusted by the damping coefficient with the initial weight of the node;

[0026] The resource utilization characteristics and performance fluctuation characteristics of the task are extracted based on the comprehensive risk prediction value, and the computational complexity of the task is calculated in combination with the resource utilization characteristics, and the computational complexity is calculated based on the time complexity and space complexity; the data complexity is calculated in combination with the performance fluctuation characteristics, and the data complexity is calculated based on the data scale, data type diversity and data quality requirements; the dependency complexity is calculated based on the task dependency graph, and the dependency complexity is calculated based on the in-degree, out-degree and closeness of the dependency relationship of the task; the computational complexity, the data complexity and the dependency complexity are weightedly combined to obtain a task execution complexity score;

[0027] Construct a fault tree with task failure as the top event, and use the comprehensive risk prediction value as the basic probability value; identify the key fault path through minimum cut set analysis; obtain the probability of occurrence of each minimum cut set based on the basic probability value and historical data statistics, and determine the corresponding severity based on the service interruption duration and impact scope; sum the product of the probability of occurrence and the severity over all minimum cut sets to obtain the task failure impact score;

[0028] The analytic hierarchy process is used to determine the weight coefficients of the task criticality, the execution complexity and the failure impact; the task criticality, the task execution complexity score and the task failure impact score are multiplied by the corresponding weight coefficients respectively and the sum is calculated to obtain the task-level risk score.

[0029] Calculating a migration priority score for each task group based on the migration risk assessment report, and sorting the task groups according to the migration priority scores, and generating a migration execution sequence includes:

[0030] Extract the risk level, risk propagation impact and risk tolerance of the task group from the migration risk assessment report to construct a risk feature vector; use the entropy weight method to calculate the risk level weight coefficient, risk propagation impact weight coefficient and risk tolerance weight coefficient according to the data distribution characteristics of the risk feature vector; multiply the risk level, the risk propagation impact and the risk tolerance by the corresponding weight coefficients respectively and sum them to obtain the first-stage migration priority score;

[0031] Extract business indicators based on the migration risk assessment report, calculate the business continuity requirement score according to the maximum allowable interruption time and the recovery point target, calculate the revenue contribution score according to the total revenue ratio, and calculate the user impact score according to the number of affected users and the importance of users; use the expert scoring method to determine the business continuity requirement weight coefficient, the revenue contribution weight coefficient, and the user impact weight coefficient; multiply the business continuity requirement score, the revenue contribution score, and the user impact score by the corresponding weight coefficients respectively and sum them to obtain the second-stage migration priority score;

[0032] A multi-objective optimization method is used to solve the optimal integration weights of the first-stage migration priority score and the second-stage migration priority score; the first-stage migration priority score and the second-stage migration priority score are respectively multiplied by the corresponding optimal integration weights and the sum is calculated to obtain the final migration priority score of the task group;

[0033] The task groups are preliminarily sorted according to the final migration priority scores of the task groups to obtain preliminary sorting results; an integer programming model is constructed, the preliminary sorting results are used as the benchmark sequence, and the technical dependency constraints, resource capacity constraints and risk threshold constraints are used as constraints; wherein the technical dependency constraints ensure that the dependent group migrates before the dependent group, the resource capacity constraints ensure that the resource occupancy at any point in time does not exceed the system capacity, and the risk threshold constraints control the number of risk groups that migrate simultaneously; the integer programming model is solved using a genetic algorithm to obtain a final migration execution sequence that meets the constraints.

[0034] For each task group in the migration execution sequence, a corresponding resource allocation plan, data synchronization strategy and fault recovery plan are formulated in combination with the migration risk assessment result, including:

[0035] Obtaining migration risk assessment results for each task group in the migration execution sequence; extracting CPU usage historical curves, business peak load data, data growth rate, data access mode, data synchronization scale, real-time requirements and key failure point information from the migration risk assessment results;

[0036] Based on the CPU usage history curve and the business peak load data, a time series analysis method is used to predict the peak demand for computing resources; based on the data growth rate and the data access mode, the storage capacity demand and the input and output operand demand are calculated; based on the data synchronization scale and the real-time requirement, the network bandwidth demand is calculated; the computing resource peak demand, the storage capacity demand, the input and output operand demand and the network bandwidth demand are integrated to form a resource allocation plan;

[0037] Based on the data synchronization scale, the data synchronization process is divided into a pre-synchronization phase, an incremental synchronization phase, and a switching verification phase, and the pre-synchronization phase, the incremental synchronization phase, and the switching verification phase are weightedly combined to obtain a synchronization mechanism; in the pre-synchronization phase, the full amount of data is divided into multiple data fragments for concurrent transmission; in the incremental synchronization phase, a write buffer pool and a log-based real-time data synchronization mechanism are constructed, and a two-phase commit protocol is adopted; in the switching verification phase, a combination of hash value comparison and sampling verification is used to perform data consistency verification; the synchronization mechanisms are integrated to form a data synchronization strategy;

[0038] A fault handling decision tree is constructed based on the key fault point information and a fault threshold is set. The fault handling decision tree includes a fault handling branch below a preset fault threshold and a fault handling branch above the preset fault threshold; a fault self-healing mechanism including an automatic retry and a load balancing mechanism is configured in the fault handling branch below the preset fault threshold; a disaster recovery protection mechanism including a system snapshot, an active-active switching channel and an emergency resource pool is configured in the fault handling branch above the preset fault threshold; and the fault handling decision tree is used as a fault recovery plan based on the fault self-healing mechanism and the disaster recovery protection mechanism.

[0039] Perform migration drills in the digital twin environment and record abnormal events and treatment measures. Use reinforcement learning algorithms to continuously optimize the migration risk prediction model and migration execution plan based on the abnormal events and treatment measures, and finally form a verified optimal migration plan including:

[0040] Using Poisson distribution to generate a user access request sequence, combining with Markov chain to simulate user behavior conversion to obtain load test data; performing migration drills in the digital twin environment based on the load test data, simulating hardware failures, network interruptions and performance degradation scenarios at key nodes of the system through fault injection configuration files; recording abnormal events and handling measures during the migration drill;

[0041] The sliding window method is used to analyze the changing trend of the task operation indicators, and the indicators that significantly deviate from the normal range are marked as abnormal events; a system component dependency graph is constructed, and a graph traversal algorithm is used to analyze the propagation path of the abnormal events; based on the decision tree model, multi-source abnormal indicators are correlated and analyzed to obtain the root cause of the fault; the processing process of each abnormal event is recorded, including alarm conditions, processing operations and processing results;

[0042] Encode migration strategy parameters into action space, wherein the migration strategy parameters include resource allocation ratio, synchronization strategy parameters and fault handling options; design a multi-dimensional reward function, wherein the multi-dimensional reward function includes performance improvement reward items, stability reward items and resource efficiency reward items; use a deep deterministic policy gradient algorithm to train a reinforcement learning strategy network, and improve sample utilization efficiency through an experience replay mechanism;

[0043] Construct a training sample set based on the abnormal event and the treatment measures, and use the training sample set to continuously optimize the migration risk prediction model; apply the optimized migration risk prediction model to real-time risk assessment, and dynamically adjust the migration strategy parameters according to the assessment results; use the reinforcement learning strategy network to output the control strategy, and optimize the migration execution plan based on the control strategy;

[0044] The optimized migration execution plan is verified by Monte Carlo simulation method, and the reliability of the plan is evaluated under different load conditions and fault scenarios. The probability of risk occurrence and the degree of impact are calculated, and a risk scoring model is established to quantitatively evaluate the residual risk. The final optimal migration plan is determined based on the evaluation results of the risk scoring model and the prediction results of the migration risk prediction model.

[0045] According to a second aspect of the embodiments of the present invention,

[0046] Provides a system for scheduling tasks and coordinating and smoothly migrating data based on a big data platform, including:

[0047] The first unit is used to build a digital twin environment of the target system based on the physical topology and resource configuration parameters of the source system, and deploy task status collection probes in the digital twin environment; collect the running status data of the scheduled tasks in the source system, and the running status data includes task execution time, resource occupancy rate, data flow path, task dependency and fault occurrence frequency; use a deep neural network to extract features of the running status data and build a task feature vector matrix; cluster and group the scheduled tasks according to the task feature vector matrix to generate a task group list with a unique identifier;

[0048] The second unit is used to deploy the scheduled tasks in the task group list to the digital twin environment, and record the task operation indicators in real time through the task status collection probe; build a task association analysis model based on the graph neural network to identify the data flow relationship and resource competition relationship between tasks, and compare and analyze the task operation indicators with the operation status data in the source system to generate a task stability score; build a migration risk prediction model based on the task stability score and the analysis results of the task association analysis model, and quantitatively evaluate the migration risk of each task group based on the migration risk prediction model to generate a migration risk assessment report;

[0049] The third unit is used to calculate the migration priority score for each task group based on the migration risk assessment report, and sort the task groups according to the migration priority scores to generate a migration execution sequence; for each task group in the migration execution sequence, formulate a corresponding resource allocation plan, data synchronization strategy and fault recovery plan in combination with its migration risk assessment result; perform migration drills in the digital twin environment and record abnormal events and handling measures, and use a reinforcement learning algorithm to continuously optimize the migration risk prediction model and migration execution plan according to the abnormal events and handling measures, and finally form a verified optimal migration plan.

[0050] According to a third aspect of the embodiments of the present invention,

[0051] An electronic device is provided, comprising:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0055] A fourth aspect of the embodiments of the present invention is:

[0056] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0057] The beneficial effects of this application are as follows:

[0058] 1. Reduce migration risks: By conducting migration drills and risk assessments in a digital twin environment, potential migration risks can be identified in advance, and corresponding resource allocation plans, data synchronization strategies, and fault recovery plans can be formulated, thereby effectively reducing risks in the actual migration process and ensuring the security and smoothness of the migration.

[0059] 2. Improve migration efficiency: Clustering, grouping, correlation analysis, and priority sorting of tasks based on technologies such as deep learning and graph neural networks can realize automated and intelligent migration plan formulation and execution, thereby improving migration efficiency and shortening the migration cycle.

[0060] 3. Optimize resource allocation: By analyzing the resource occupancy and competition relationship of tasks, a more reasonable resource allocation plan can be formulated to avoid resource waste and performance bottlenecks, thereby optimizing the resource utilization of the target system and improving the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flowchart of a method for smooth migration of tasks and data collaboration based on a big data platform scheduling according to an embodiment of the present invention;

[0062] Figure 2 A flowchart of a method for scheduling tasks based on a big data platform according to an embodiment of the present invention;

[0063] Figure 3 It is a structural diagram of a system for scheduling tasks and coordinating and smoothly migrating data based on a big data platform according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0066] Figure 1 FIG. 1 is a flow chart of a method for smoothly migrating tasks and data in a coordinated manner based on a big data platform according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S11. Build a digital twin environment of the target system based on the physical topology and resource configuration parameters of the source system, and deploy task status collection probes in the digital twin environment; collect the running status data of the scheduled tasks in the source system, the running status data including task execution time, resource occupancy rate, data flow path, task dependency, and fault occurrence frequency; use a deep neural network to extract features from the running status data and build a task feature vector matrix; cluster and group the scheduled tasks according to the task feature vector matrix to generate a task group list with a unique identifier;

[0068] S12. Deploy the scheduled tasks in the task group list to the digital twin environment, and record the task operation indicators in real time through the task status collection probe; build a task association analysis model based on the graph neural network to identify the data flow relationship and resource competition relationship between tasks, and compare and analyze the task operation indicators with the operation status data in the source system to generate a task stability score; build a migration risk prediction model based on the task stability score and the analysis results of the task association analysis model, and quantitatively evaluate the migration risk of each task group based on the migration risk prediction model to generate a migration risk assessment report;

[0069] S13. Calculate the migration priority score for each task group based on the migration risk assessment report, sort the task groups according to the migration priority scores, and generate a migration execution sequence; formulate corresponding resource allocation plans, data synchronization strategies, and fault recovery plans for each task group in the migration execution sequence in combination with its migration risk assessment results; perform migration drills in the digital twin environment and record abnormal events and handling measures, and use a reinforcement learning algorithm to continuously optimize the migration risk prediction model and migration execution plan based on the abnormal events and handling measures, and finally form a verified optimal migration plan.

[0070] In an optional implementation, a task association analysis model is constructed based on a graph neural network to identify the data flow relationship and resource competition relationship between tasks, and the task operation indicators are compared and analyzed with the operation status data in the source system to generate a task stability score including:

[0071] Based on the running status data, a task association graph is constructed, wherein the nodes in the task association graph represent the scheduled tasks, and the edges in the task association graph represent the association relationship between tasks; based on the resource occupancy rate, the resource competition between tasks is analyzed to generate a resource competition matrix; according to the data flow path, the data flow frequency and data flow volume between tasks are calculated to generate a data flow matrix; the data flow matrix and the resource competition matrix are merged to form a task association weight matrix;

[0072] The task association weight matrix is ​​input into a graph neural network, wherein the graph neural network includes multiple graph attention layers, and each graph attention layer calculates attention weights based on node features and edge features; the graph neural network is used to extract association patterns between tasks, identify key task paths and task dependency chains, and generate task association features; a task association analysis model is constructed based on the task association features, and the task association analysis model is used to predict data flow bottlenecks and resource competition conflicts between tasks;

[0073] Deploy task status collection probes in the digital twin environment to collect task operation indicators in the target environment in real time; compare the task operation indicators with the operation status data in the source system, and calculate the task execution timing deviation, performance indicator difference and abnormal event probability; analyze the propagation impact of task operation anomalies on associated tasks based on the prediction results of the task association analysis model; comprehensively calculate the task stability score based on the task execution timing deviation, the performance indicator difference, the abnormal event probability and the abnormal propagation impact.

[0074] The task stability scoring method includes the following steps:

[0075] First, build a task association graph. The monitoring system collects the running logs and system indicators of each scheduled task in real time, such as task start time, end time, CPU usage, memory usage, disk IO, etc. Task start and end time are used to determine the dependencies between tasks, and other system indicators are used for subsequent analysis of resource competition between tasks. For example, if task A is completed before task B, it is considered that there is an association between task A and task B, and a directed edge from task A to task B is added to the task association graph.

[0076] Next, analyze the resource competition between tasks. Analyze the resource competition between tasks based on the collected CPU usage, memory usage, disk IO and other resource occupancy data. If two tasks have a high occupancy rate for the same resource in the same time period, it is considered that there is a resource competition relationship between the two tasks. The resource competition relationship between all tasks is represented by a matrix, which is called the resource competition matrix. For example, if the CPU occupancy rate of task A and task B exceeds 80% in the same time period, then in the resource competition matrix, the element values ​​corresponding to task A and task B are set to 1, indicating that there is resource competition; otherwise, they are set to 0, indicating that there is no resource competition.

[0077] Then, calculate the data flow relationship between tasks. According to the input and output logs of the tasks, track the data flow path and flow times between different tasks. For example, if the output data of task A is used as the input data of task B, it is considered that the data flows from task A to task B. Count the frequency and amount of data flowing between different tasks and generate a data flow matrix. For example, if the data of task A flows to task B 100 times a day, and the data volume each time is 1GB, then the element value from A to B in the data flow matrix is ​​100GB / day.

[0078] Next, the data flow matrix and the resource competition matrix are integrated to form the task association weight matrix. The corresponding elements in the data flow matrix and the resource competition matrix are weighted and summed to obtain the task association weight matrix. For example, assuming that the data flow weight ratio is 0.6 and the resource competition weight ratio is 0.4, the association weight between tasks A and B is: 0.6*100+0.4*1=60.4.

[0079] Then, the task association weight matrix is ​​input into the graph neural network for training. The graph neural network adopts a multi-layer graph attention layer structure. Each graph attention layer calculates the attention weight based on the node features (such as task type, task execution time, etc.) and edge features (task association weight), thereby learning the association pattern between tasks. Through the training of the graph neural network, the association features between tasks are extracted, and the key task paths and task dependency chains are identified.

[0080] Next, we build a task association analysis model. We use the trained graph neural network to build a task association analysis model. This model can be used to predict data flow bottlenecks and resource competition conflicts between tasks. For example, if the model predicts that there is a data flow bottleneck between task A and task B, we can take measures in advance, such as increasing the data transmission bandwidth or optimizing the data transmission protocol, to avoid the bottleneck.

[0081] Next, deploy the task status collection probe in the digital twin environment. Collect the task running indicators in the target environment in real time, such as task execution time, CPU usage, memory usage, etc. Compare the collected task running indicators with the running status data in the source system to calculate the task execution timing deviation, performance indicator difference and abnormal event probability. For example, if the actual execution time of task A is 10% later than the expected execution time, the task execution timing deviation is 10%.

[0082] Then, based on the prediction results of the task association analysis model, the propagation effect of task operation anomalies on associated tasks is analyzed. For example, if task A has an anomaly, causing its output data to be delayed, and task B depends on the output data of task A, then the anomaly of task A will propagate to task B, causing task B to be delayed as well.

[0083] Finally, the task stability score is calculated comprehensively. The task stability score is calculated comprehensively based on the task execution timing deviation, performance indicator difference, abnormal event probability, and abnormal propagation impact. For example, the task stability score can be calculated by weighted summation, and the weight can be adjusted according to the importance of different indicators.

[0084] The solution of this application can:

[0085] Accurately identify task associations: Through graph neural networks and multi-source data fusion, it is possible to more accurately identify the associations between tasks, including data flow relationships and resource competition relationships, avoiding the limitations of relying solely on a single data source. Predict potential risks: By building a task association analysis model, it is possible to predict potential data flow bottlenecks and resource competition conflicts between tasks, so that preventive measures can be taken in advance to improve the stability and reliability of tasks. Improve system stability: By real-time monitoring of task operation status and combining the prediction results of the task association analysis model, task operation anomalies can be discovered and handled in a timely manner, and the impact of anomalies on related tasks can be evaluated, thereby effectively improving the stability of the entire system.

[0086] In an optional implementation, a migration risk prediction model is constructed according to the task stability score and the analysis result of the task association analysis model, and a migration risk prediction model is quantitatively evaluated for each task group based on the migration risk prediction model. The generation of a migration risk assessment report includes:

[0087] Acquire task stability features, associated propagation features, and resource-sensitive features as an initial feature set, and build a migration risk prediction model based on the task stability score and the task association analysis model. The migration risk prediction model includes a feature fusion layer, a time series modeling layer, and a risk prediction layer. The feature fusion layer uses an attention mechanism to adaptively weight the features in the initial feature set to obtain a weighted feature vector.

[0088] The weighted feature vector is input into the time series modeling layer, and the time series modeling layer uses a long short-term memory network to extract risk evolution characteristics. The long short-term memory network selectively retains long-term influencing factors and short-term volatility factors through a gating mechanism; the risk prediction layer includes a gradient boosting tree model and a deep neural network, and the risk evolution characteristics are simultaneously input into the gradient boosting tree model and the deep neural network. The gradient boosting tree model processes the explicit risk characteristics to obtain a first prediction result, and the deep neural network extracts the implicit risk pattern to obtain a second prediction result; the first prediction result and the second prediction result are weightedly fused to obtain a comprehensive risk prediction value;

[0089] Based on the comprehensive risk prediction value, the risk status of a single task is evaluated, and a weighted combination of task criticality, execution complexity, and failure impact is calculated to obtain a task-level risk score; a task risk association network is constructed to analyze the risk propagation relationship between tasks, and the sensitivity of tasks to system resource fluctuations is evaluated; the task-level risk score, the risk propagation relationship, and the sensitivity of resource fluctuations are combined to generate a task-level risk assessment result;

[0090] Divide tasks with similar risk characteristics into groups, aggregate the task-level risk assessment results within the same group, and calculate the overall risk value of the group based on the correlation between tasks within the group; establish a dependency graph between groups to analyze the cross-group risk propagation path; perform weighted adjustment on the overall risk value of the group according to the business importance of the group to generate a group-level risk assessment result;

[0091] Construct a system-level risk propagation graph, use the group-level risk assessment results as the node initial state, and the risk propagation relationship between groups as the edge weight; perform risk diffusion simulation on the system-level risk propagation graph to calculate the cumulative effect of risk in the system; evaluate the overall risk tolerance of the system based on the risk diffusion results, and identify high-risk concentration areas; hierarchically integrate the task-level risk assessment results, the group-level risk assessment results, and the system risk tolerance assessment results to generate a migration risk assessment report.

[0092] Obtain task stability features, associated propagation features, and resource-sensitive features as the initial feature set, and build a migration risk prediction model. First, extract task stability features from data sources such as task management systems, monitoring logs, and resource usage records, such as average task execution time, task failure rate, and number of task restarts. At the same time, analyze the calling relationship and data dependency between tasks, and extract associated propagation features, such as the calling frequency and data transmission volume between tasks. In addition, evaluate the degree of task dependence on resources such as CPU, memory, and network, and extract resource-sensitive features, such as the peak CPU usage rate and average memory usage of tasks. These features constitute the initial feature set.

[0093] Next, a migration risk prediction model is constructed. The model includes a feature fusion layer, a time series modeling layer, and a risk prediction layer. In the feature fusion layer, an attention mechanism is used to adaptively weight the features in the initial feature set. For example, for tasks with poor stability, the model will give higher weights to stability features. The weighted feature vector is input into the time series modeling layer. The time series modeling layer uses a long short-term memory network (LSTM) to extract risk evolution features. The LSTM network selectively retains long-term influencing factors and short-term fluctuation factors through a gating mechanism. For example, the impact of continuous high load on tasks will be retained for a long time, while occasional network fluctuations will be filtered out in the short term. The output of the LSTM network is the risk evolution feature. The risk prediction layer includes a gradient boosted tree (GBT) model and a deep neural network (DNN). The risk evolution feature is input into the GBT model and the DNN model at the same time. The GBT model focuses on processing explicit risk features, such as the historical migration failure rate, to obtain the first prediction result. The DNN model is used to extract implicit risk patterns, such as the nonlinear relationship between task features, to obtain the second prediction result. Finally, the first prediction result and the second prediction result are weighted and fused to obtain a comprehensive risk prediction value. For example, according to the training results of the model, the weight of the prediction result of the GBT model is 0.6, and the weight of the prediction result of the DNN model is 0.4. The comprehensive risk prediction value is the weighted average of the two prediction results.

[0094] Task-level risk assessment is performed based on the comprehensive risk prediction value. First, the risk status of a single task is assessed based on the comprehensive risk prediction value. For example, the higher the risk prediction value, the higher the risk of the task. Then, the task-level risk score is calculated by combining factors such as task criticality, execution complexity, and failure impact. For example, if the weight of task criticality is 0.3, the weight of execution complexity is 0.2, and the weight of failure impact is 0.5, then the task-level risk score is the weighted sum of these three factors. Next, a task risk association network is constructed to analyze the risk propagation relationship between tasks. For example, if task A depends on task B, the risk of task B may propagate to task A. In addition, the sensitivity of the task to system resource fluctuations is assessed. For example, if the task is very sensitive to CPU resources, fluctuations in CPU load may affect the execution of the task. Finally, the task-level risk score, risk propagation relationship, and sensitivity to resource fluctuations are combined to generate the task-level risk assessment results.

[0095] Perform group-level risk assessment and generate a migration risk assessment report. First, divide tasks with similar risk characteristics into groups. For example, tasks can be divided according to their functional modules or the systems they belong to. Then, aggregate the task-level risk assessment results within the same group and calculate the overall risk value of the group by combining the associations of the tasks within the group. For example, the risk scores of all tasks within the group can be averaged to obtain the overall risk value of the group. Next, establish a dependency graph between groups and analyze the cross-group risk propagation path. For example, if group A depends on group B, the risk of group B may propagate to group A. Then, weight the overall risk value of the group according to the business importance of the group to generate the group-level risk assessment results. For example, for groups with higher business importance, their risk values ​​will be assigned higher weights. Finally, construct a system-level risk propagation graph, using the group-level risk assessment results as the initial state of the node and the risk propagation relationship between groups as the edge weight. Perform risk diffusion simulation on the system-level risk propagation graph to calculate the cumulative effect of risk in the system. For example, you can simulate how risk propagates in the system after a group fails. Evaluate the overall risk tolerance of the system based on the risk diffusion results and identify high-risk clustering areas. For example, if the cumulative effect of risk in a certain area is high, the risk tolerance of that area is low. Finally, the task-level risk assessment results, group-level risk assessment results, and system risk tolerance assessment results are hierarchically integrated to generate a migration risk assessment report.

[0096] The solution of this application can:

[0097] Improve the comprehensiveness and accuracy of risk assessment: This method comprehensively considers multi-dimensional factors such as task stability, association propagation, resource sensitivity, etc., and combines the advantages of GBT model and DNN model, which can evaluate migration risks more comprehensively and accurately. For example, by analyzing the association relationship between tasks, potential risk propagation paths can be identified to avoid chain reactions caused by single point failures. By combining GBT and DNN models, explicit and implicit risk patterns can be captured to improve the accuracy of risk prediction. Refined risk management: This method conducts risk assessment at three levels: task level, group level, and system level, and constructs risk association network and risk propagation map to achieve refined risk management. For example, different migration strategies can be adopted for tasks with different risk levels to reduce migration risks. High-risk clustering areas can be identified and corresponding risk mitigation measures can be taken. Improve decision-making efficiency: This method provides quantitative risk assessment results and visual risk propagation maps, which can help decision makers quickly understand the overall risk status of the system and develop more effective migration plans. For example, the migration risk assessment report can provide decision makers with clear risk information, help them weigh the pros and cons of migration, and choose the best migration time and plan.

[0098] In an optional implementation, the risk status of a single task is evaluated based on the comprehensive risk prediction value, and a weighted combination of task criticality, execution complexity, and failure impact is calculated to obtain a task-level risk score, including:

[0099] Establishing a risk assessment benchmark based on the comprehensive risk prediction value, and using the risk assessment benchmark as a reference threshold for calculating the mission criticality, the execution complexity, and the impact of failure;

[0100] Constructing a task dependency graph, the task dependency graph includes a node set and an edge set, the node set represents a task set, and the edge set represents a dependency relationship between tasks; mapping the comprehensive risk prediction value to the task dependency graph as an initial weight of a node; calculating the criticality of the task using a PageRank algorithm based on the task dependency graph, the criticality of the task is calculated in an iterative manner, and the criticality value of the task in each iteration is obtained by combining the weighted sum of the ratio of the criticality value of the adjacent tasks to the out-degree adjusted by the damping coefficient with the initial weight of the node;

[0101] The resource utilization characteristics and performance fluctuation characteristics of the task are extracted based on the comprehensive risk prediction value, and the computational complexity of the task is calculated in combination with the resource utilization characteristics, and the computational complexity is calculated based on the time complexity and space complexity; the data complexity is calculated in combination with the performance fluctuation characteristics, and the data complexity is calculated based on the data scale, data type diversity and data quality requirements; the dependency complexity is calculated based on the task dependency graph, and the dependency complexity is calculated based on the in-degree, out-degree and closeness of the dependency relationship of the task; the computational complexity, the data complexity and the dependency complexity are weightedly combined to obtain a task execution complexity score;

[0102] Construct a fault tree with task failure as the top event, and use the comprehensive risk prediction value as the basic probability value; identify the key fault path through minimum cut set analysis; obtain the probability of occurrence of each minimum cut set based on the basic probability value and historical data statistics, and determine the corresponding severity based on the service interruption duration and impact scope; sum the product of the probability of occurrence and the severity over all minimum cut sets to obtain the task failure impact score;

[0103] The analytic hierarchy process is used to determine the weight coefficients of the task criticality, the execution complexity and the failure impact; the task criticality, the task execution complexity score and the task failure impact score are multiplied by the corresponding weight coefficients respectively and the sum is calculated to obtain the task-level risk score.

[0104] The task-level risk assessment method driven by the comprehensive risk prediction value can more accurately quantify the task risk and assist decision makers in risk management. This method is based on the comprehensive risk prediction value and finally obtains the task-level risk score by evaluating the mission criticality, execution complexity and failure impact.

[0105] First, establish a risk assessment benchmark. Analyze historical task data, such as the probability of risk occurrence and degree of loss of previous tasks, and combine expert experience to determine the risk level classification standard, such as the threshold range of high, medium and low risks. This benchmark will serve as a reference threshold for subsequent assessments of task criticality, execution complexity and the impact of failure. For example, based on historical data analysis, it is determined that a comprehensive risk prediction value greater than 0.8 is high risk, 0.5 to 0.8 is medium risk, and less than 0.5 is low risk.

[0106] Next, construct a task dependency graph. The task dependency graph describes the dependency relationship between tasks. The nodes in the graph represent tasks, and the edges between nodes represent the dependency relationship between tasks. The direction of the edge represents the dependency direction. For example, if task A depends on task B to be completed, there is an edge from B to A. The initial weight of each node is determined by the comprehensive risk prediction value. For example, the comprehensive risk prediction value of task A is 0.7, and the comprehensive risk prediction value of task B is 0.6. In the task dependency graph, the initial weights of task A and task B are 0.7 and 0.6, respectively. Task C depends on task A and task B, then there are two edges in the task dependency graph, one from A to C and one from B to C.

[0107] Then, the task criticality is calculated. The PageRank algorithm is used to iteratively calculate the task criticality. In each round of iteration, the criticality value of a task is determined by the criticality values ​​of the tasks it depends on, the out-degree of these tasks (that is, the number of tasks that depend on the task), and the damping coefficient, and is adjusted in combination with the initial weight of the node. The damping coefficient is used to balance the influence of the node's own weight and the weight of the dependent tasks, and is usually taken as 0.85. After multiple rounds of iterations, the criticality value of each task is finally obtained.

[0108] Subsequently, the task execution complexity is calculated. First, the resource utilization characteristics and performance fluctuation characteristics of the task are extracted according to the comprehensive risk prediction value. Resource utilization characteristics include CPU usage, memory occupancy, etc., and performance fluctuation characteristics include response time fluctuation, throughput fluctuation, etc. Then, the computational complexity of the task is calculated based on the resource utilization characteristics, taking into account the time complexity and space complexity. For example, if a task needs to process a large amount of data, its time complexity and space complexity are high. At the same time, the data complexity is calculated based on the performance fluctuation characteristics, taking into account the data scale, data type diversity, and data quality requirements. For example, if a task needs to process data of different types and qualities, its data complexity is high. In addition, the dependency complexity is calculated according to the task dependency graph, taking into account the in-degree, out-degree, and closeness of the dependency relationship of the task. For example, if a task depends on multiple other tasks, its dependency complexity is high. Finally, the computational complexity, data complexity, and dependency complexity are weighted and combined to obtain the task execution complexity score.

[0109] Next, evaluate the impact of task failure. Construct a fault tree with task failure as the top event. Fault tree is a graphical method for analyzing the causes and consequences of system or component failures. The comprehensive risk prediction value is used as the basic probability value to calculate the probability of occurrence of events in the fault tree. Identify the critical failure path through minimum cut set analysis. The minimum cut set refers to the smallest combination of events that causes the top event to occur. For example, if the failure of task A causes the entire system to crash, then the failure of task A is a minimum cut set. The probability of occurrence of each minimum cut set is obtained based on the basic probability value and historical data statistics. Determine the severity of each minimum cut set based on the duration of business interruption and the scope of impact. For example, if the occurrence of a minimum cut set causes system interruption for several hours and affects a large number of users, its severity is high. Add the product of the probability of occurrence of each minimum cut set and its severity to obtain the task failure impact score.

[0110] Finally, the task-level risk score is calculated. The analytic hierarchy process is used to determine the weight coefficients of task criticality, execution complexity, and failure impact. The analytic hierarchy process is a method used to determine the relative importance of multiple factors. For example, through expert scoring, the weight coefficients of task criticality, execution complexity, and failure impact are determined to be 0.5, 0.3, and 0.2, respectively. The task criticality, task execution complexity score, and task failure impact score are multiplied by the corresponding weight coefficients, and then added to obtain the task-level risk score.

[0111] The solution of this application can:

[0112] Improve the accuracy of risk assessment: By comprehensively considering the mission criticality, execution complexity and failure impact, as well as the relationship between them, the risk level of the mission can be assessed more comprehensively and accurately. Improve the efficiency of risk management: By quantifying mission risks, decision makers can more effectively identify high-risk missions and prioritize resource allocation for management, thereby improving the efficiency of risk management. Enhance risk response capabilities: By analyzing the impact of mission failure, companies can better understand potential risks and develop corresponding response measures, thereby enhancing risk response capabilities.

[0113] In an optional implementation, calculating a migration priority score for each task group based on the migration risk assessment report, and sorting the task groups according to the migration priority scores, and generating a migration execution sequence includes:

[0114] Extract the risk level, risk propagation impact and risk tolerance of the task group from the migration risk assessment report to construct a risk feature vector; use the entropy weight method to calculate the risk level weight coefficient, risk propagation impact weight coefficient and risk tolerance weight coefficient according to the data distribution characteristics of the risk feature vector; multiply the risk level, the risk propagation impact and the risk tolerance by the corresponding weight coefficients respectively and sum them to obtain the first-stage migration priority score;

[0115] Extract business indicators based on the migration risk assessment report, calculate the business continuity requirement score according to the maximum allowable interruption time and the recovery point target, calculate the revenue contribution score according to the total revenue ratio, and calculate the user impact score according to the number of affected users and the importance of users; use the expert scoring method to determine the business continuity requirement weight coefficient, the revenue contribution weight coefficient, and the user impact weight coefficient; multiply the business continuity requirement score, the revenue contribution score, and the user impact score by the corresponding weight coefficients respectively and sum them to obtain the second-stage migration priority score;

[0116] A multi-objective optimization method is used to solve the optimal integration weights of the first-stage migration priority score and the second-stage migration priority score; the first-stage migration priority score and the second-stage migration priority score are respectively multiplied by the corresponding optimal integration weights and the sum is calculated to obtain the final migration priority score of the task group;

[0117] The task groups are preliminarily sorted according to the final migration priority scores of the task groups to obtain preliminary sorting results; an integer programming model is constructed, the preliminary sorting results are used as the benchmark sequence, and the technical dependency constraints, resource capacity constraints and risk threshold constraints are used as constraints; wherein the technical dependency constraints ensure that the dependent group migrates before the dependent group, the resource capacity constraints ensure that the resource occupancy at any point in time does not exceed the system capacity, and the risk threshold constraints control the number of risk groups that migrate simultaneously; the integer programming model is solved using a genetic algorithm to obtain a final migration execution sequence that meets the constraints.

[0118] A method for determining the execution sequence of cloud migration can effectively manage migration risks and optimize the migration process. The core of this method is to build a two-stage migration priority assessment model and combine it with a constrained optimization method to generate the final migration execution sequence.

[0119] First, conduct a migration risk assessment and collect data on the risk level, risk communication impact, and risk tolerance for each task group. For example, the risk level of task group A is high, the risk communication impact is medium, and the risk tolerance is low; the risk level of task group B is medium, the risk communication impact is low, and the risk tolerance is high. The risk level can be divided into five levels, from low to high, 1 to 5. The risk communication impact and risk tolerance are also divided in a similar way.

[0120] Next, the entropy weight method is used to determine the weight coefficient of each risk factor. The entropy weight method determines the weight based on the discreteness of the data itself. For example, if the risk level of all task groups is the same, the weight coefficient of the risk level is 0 because it cannot distinguish the priorities of different task groups. Assume that the weight coefficients of risk level, risk transmission impact and risk tolerance calculated by the entropy weight method are 0.5, 0.3 and 0.2 respectively.

[0121] Then, calculate the first-stage migration priority score for each task group. This score is the sum of the product of the risk level, risk communication impact, and risk tolerance and their corresponding weight coefficients. For example, the first-stage migration priority score of task group A is: high risk level (assuming 5) * 0.5 + medium risk communication impact (assuming 3) * 0.3 + low risk tolerance (assuming 1) * 0.2 = 3.6. Similarly, the first-stage migration priority score of task group B can be calculated.

[0122] Next, collect the business indicators of each task group, including the maximum allowable interruption time, recovery point target, total revenue share, number of affected users, and user importance. For example, the maximum allowable interruption time of task group A is 2 hours, the recovery point target is 1 hour, the total revenue share is 50%, the number of affected users is 10,000, and the user importance is high; the maximum allowable interruption time of task group B is 4 hours, the recovery point target is 4 hours, the total revenue share is 20%, the number of affected users is 1,000, and the user importance is low.

[0123] Then, the business continuity requirement score, revenue contribution score, and user impact score are calculated based on the business indicators. For example, based on the maximum allowable outage time and recovery point objective, it can be calculated that the business continuity requirement score of task group A is higher, and the business continuity requirement score of task group B is lower. The revenue contribution score is calculated based on the proportion of total revenue, and the user impact score is calculated based on the number of affected users and the importance of users.

[0124] Next, the expert scoring method is used to determine the weight coefficients of business continuity requirements, revenue contribution and user impact. For example, after expert evaluation, the weight coefficient of business continuity requirements is 0.6, the weight coefficient of revenue contribution is 0.3, and the weight coefficient of user impact is 0.1.

[0125] Then, the second-stage migration priority score of each task group is calculated. This score is the sum of the business continuity requirement score, revenue contribution score, and user impact score multiplied by their corresponding weight coefficients.

[0126] Next, a multi-objective optimization method is used to determine the optimal integrated weights of the migration priority scores of the first and second phases. For example, through the optimization algorithm, the weight of the migration priority score of the first phase is determined to be 0.7, and the weight of the migration priority score of the second phase is determined to be 0.3.

[0127] Then, the final migration priority score of each task group is calculated, which is the sum of the products of the first-stage and second-stage migration priority scores and their corresponding integration weights.

[0128] Perform a preliminary ranking of task groups based on the final migration priority scores.

[0129] Finally, an integer programming model is constructed, taking the preliminary ranking results as a benchmark, and considering technical dependency constraints (e.g., the database must be migrated before the application), resource capacity constraints (e.g., the number of task groups migrated simultaneously cannot exceed the server capacity), and risk threshold constraints (e.g., the number of high-risk task groups migrated simultaneously cannot exceed a certain limit). The model is solved using an optimization algorithm such as a genetic algorithm to obtain the final migration execution sequence.

[0130] The solution of this application can:

[0131] Reduce migration risk: By considering risk level, risk propagation impact, and risk tolerance, this method can prioritize the migration of task groups with lower risks, thereby reducing overall migration risk. Minimize business interruption: By considering business continuity requirements, revenue contribution, and user impact, this method can prioritize the migration of task groups with less impact on the business, thereby minimizing business interruption. Optimize resource utilization: By considering technology dependency constraints and resource capacity constraints, this method can optimize resource utilization and improve migration efficiency.

[0132] In an optional implementation, for each task group in the migration execution sequence, formulating a corresponding resource allocation plan, data synchronization strategy and fault recovery plan in combination with the migration risk assessment result includes:

[0133] Obtaining migration risk assessment results for each task group in the migration execution sequence; extracting CPU usage historical curves, business peak load data, data growth rate, data access mode, data synchronization scale, real-time requirements and key failure point information from the migration risk assessment results;

[0134] Based on the CPU usage history curve and the business peak load data, a time series analysis method is used to predict the peak demand for computing resources; based on the data growth rate and the data access mode, the storage capacity demand and the input and output operand demand are calculated; based on the data synchronization scale and the real-time requirement, the network bandwidth demand is calculated; the computing resource peak demand, the storage capacity demand, the input and output operand demand and the network bandwidth demand are integrated to form a resource allocation plan;

[0135] Based on the data synchronization scale, the data synchronization process is divided into a pre-synchronization phase, an incremental synchronization phase, and a switching verification phase, and the pre-synchronization phase, the incremental synchronization phase, and the switching verification phase are weightedly combined to obtain a synchronization mechanism; in the pre-synchronization phase, the full amount of data is divided into multiple data fragments for concurrent transmission; in the incremental synchronization phase, a write buffer pool and a log-based real-time data synchronization mechanism are constructed, and a two-phase commit protocol is adopted; in the switching verification phase, a combination of hash value comparison and sampling verification is used to perform data consistency verification; the synchronization mechanisms are integrated to form a data synchronization strategy;

[0136] A fault handling decision tree is constructed based on the key fault point information and a fault threshold is set. The fault handling decision tree includes a fault handling branch below a preset fault threshold and a fault handling branch above the preset fault threshold; a fault self-healing mechanism including an automatic retry and a load balancing mechanism is configured in the fault handling branch below the preset fault threshold; a disaster recovery protection mechanism including a system snapshot, an active-active switching channel and an emergency resource pool is configured in the fault handling branch above the preset fault threshold; and the fault handling decision tree is used as a fault recovery plan based on the fault self-healing mechanism and the disaster recovery protection mechanism.

[0137] For each task group in the migration execution sequence, a method for formulating a corresponding resource allocation plan, data synchronization strategy and fault recovery plan is formulated in combination with the migration risk assessment results. The specific implementation methods are as follows:

[0138] First, obtain the migration risk assessment results for each task group in the migration execution sequence. The migration risk assessment results include the CPU usage history curve, business peak load data, data growth rate, data access mode, data synchronization scale, real-time requirements, and key failure point information. For example, the risk assessment results of task group A show that the CPU usage history curve shows periodic fluctuations, with peaks occurring at 10 and 14 o'clock every day; the business peak load data is 1000TPS; the data growth rate is 10% per month; the data access mode is mainly read; the data synchronization scale is 10TB; the real-time requirement is minute-level; and the key failure point is the database connection pool.

[0139] Next, formulate a resource allocation plan based on the risk assessment results. According to the historical curve of CPU usage and business peak load data, predict the peak demand for computing resources. For example, by analyzing the historical data of task group A, it is predicted that its peak CPU demand after migration is 8 cores. Calculate the storage capacity demand and input and output operation number demand according to the data growth rate and data access mode. For example, according to the data growth rate and read-dominated access mode of task group A, its storage capacity demand is estimated to be 12TB, and the input and output operation number demand is 10,000 times per second. Calculate the network bandwidth demand according to the data synchronization scale and real-time requirements. For example, according to the 10TB data scale and minute-level real-time requirements of task group A, the network bandwidth demand is calculated to be 1Gbps. Finally, the peak demand for computing resources, storage capacity demand, input and output operation number demand, and network bandwidth demand are integrated to form a resource allocation plan, for example, 8-core CPU, 12TB storage, and 1Gbps bandwidth are allocated to task group A.

[0140] Then, formulate a data synchronization strategy. According to the scale of data synchronization, the data synchronization process is divided into the pre-synchronization phase, the incremental synchronization phase, and the switching verification phase. In the pre-synchronization phase, the full amount of data is divided into multiple data shards for concurrent transmission. For example, the 10TB data of task group A is divided into 100 100GB data shards. In the incremental synchronization phase, a write buffer pool and a log-based real-time data synchronization mechanism are constructed, and a two-phase commit protocol is adopted to ensure data consistency. In the switching verification phase, a combination of hash value comparison and sampling verification is used to perform data consistency verification. For example, the hash values ​​of the data in the source database and the target database are compared, and 1% of the data is randomly selected for comparison. Finally, the three phases of pre-synchronization, incremental synchronization, and switching verification are integrated to form a data synchronization strategy.

[0141] Finally, formulate a fault recovery plan. Build a fault handling decision tree based on key fault point information and set fault thresholds. For example, for database connection pool failures, set the number of available connections in the connection pool to less than 10 as the fault threshold. Configure automatic retry and load balancing mechanisms for fault handling branches below the threshold. For example, when the number of available connections in the connection pool is less than 10, automatically retry the connection request and distribute the request to other available database connections. Configure system snapshots, active-active switching channels, and emergency resource pools for fault handling branches above the threshold. For example, when the number of available connections in the connection pool is less than 5, trigger a system snapshot, switch the business to the standby database, and enable the emergency resource pool to provide additional computing resources. Based on the fault self-healing mechanism and disaster recovery protection mechanism, use the fault handling decision tree as a fault recovery plan.

[0142] The solution of this application can:

[0143] Improved resource utilization: Through accurate resource demand prediction and allocation, we can avoid excess or insufficient resources, improve resource utilization, and reduce migration costs. For example, by analyzing historical data and predicting peak demand, we can avoid over-configuration of resources, thereby saving costs. Improved data synchronization efficiency and security: Through mechanisms such as phased synchronization, concurrent transmission, real-time synchronization, and data consistency verification, we can improve data synchronization efficiency and security, and reduce the risk of data loss and inconsistency. For example, the use of a pre-synchronization mechanism can reduce downtime during the switching phase and improve migration efficiency. Improved system stability and reliability: By building a fault handling decision tree and configuring a fault self-healing mechanism and a disaster recovery protection mechanism, we can improve system stability and reliability and reduce the impact of faults. For example, by configuring automatic retry and load balancing mechanisms, we can automatically handle some common faults and avoid system interruptions.

[0144] In an optional implementation, migration drills are performed in the digital twin environment and abnormal events and treatment measures are recorded. A reinforcement learning algorithm is used to continuously optimize the migration risk prediction model and the migration execution plan according to the abnormal events and treatment measures, and finally a verified optimal migration plan is formed, including:

[0145] Using Poisson distribution to generate a user access request sequence, combining with Markov chain to simulate user behavior conversion to obtain load test data; performing migration drills in the digital twin environment based on the load test data, simulating hardware failures, network interruptions and performance degradation scenarios at key nodes of the system through fault injection configuration files; recording abnormal events and handling measures during the migration drill;

[0146] The sliding window method is used to analyze the changing trend of the task operation indicators, and the indicators that significantly deviate from the normal range are marked as abnormal events; a system component dependency graph is constructed, and a graph traversal algorithm is used to analyze the propagation path of the abnormal events; based on the decision tree model, multi-source abnormal indicators are correlated and analyzed to obtain the root cause of the fault; the processing process of each abnormal event is recorded, including alarm conditions, processing operations and processing results;

[0147] Encode migration strategy parameters into action space, wherein the migration strategy parameters include resource allocation ratio, synchronization strategy parameters and fault handling options; design a multi-dimensional reward function, wherein the multi-dimensional reward function includes performance improvement reward items, stability reward items and resource efficiency reward items; use a deep deterministic policy gradient algorithm to train a reinforcement learning strategy network, and improve sample utilization efficiency through an experience replay mechanism;

[0148] Construct a training sample set based on the abnormal event and the treatment measures, and use the training sample set to continuously optimize the migration risk prediction model; apply the optimized migration risk prediction model to real-time risk assessment, and dynamically adjust the migration strategy parameters according to the assessment results; use the reinforcement learning strategy network to output the control strategy, and optimize the migration execution plan based on the control strategy;

[0149] The optimized migration execution plan is verified by Monte Carlo simulation method, and the reliability of the plan is evaluated under different load conditions and fault scenarios. The probability of risk occurrence and the degree of impact are calculated, and a risk scoring model is established to quantitatively evaluate the residual risk. The final optimal migration plan is determined based on the evaluation results of the risk scoring model and the prediction results of the migration risk prediction model.

[0150] A system migration optimization method based on digital twins and reinforcement learning aims to minimize migration risks and generate optimal migration solutions. This method conducts migration drills in a digital twin environment and uses reinforcement learning algorithms to continuously optimize migration strategies based on the drill results, ultimately forming a reliable and efficient migration solution.

[0151] First, build a digital twin environment that is highly consistent with the target system. The environment contains key elements such as system architecture, configuration information, and operating data to ensure the accuracy and effectiveness of the migration drill. In order to simulate real user access behavior, Poisson distribution is used to generate user access request sequences, and Markov chain is combined to simulate the transition of user states to generate load test data. For example, the average arrival rate of Poisson distribution is set to 100 requests per second, and the states of Markov chain include browsing products, adding to shopping carts, placing orders, etc., and the transition probabilities between different states are defined.

[0152] Then, the generated load test data is used to perform migration drills in the digital twin environment. In order to fully evaluate the robustness of the migration solution, various failure scenarios are simulated at key nodes of the system through fault injection profiles, such as hardware failure (e.g. simulated disk failure), network interruption (e.g. simulated network delay and packet loss), and performance degradation (e.g. simulated CPU overload).

[0153] During the migration drill, it is necessary to monitor and record various task operation indicators in real time, such as response time, error rate, throughput, etc. Use the sliding window method to analyze the change trend of these indicators, and mark the indicators that significantly deviate from the normal range as abnormal events. For example, set the sliding window size to 1 minute. If the average value of a certain indicator within 1 minute exceeds the preset threshold, it will be marked as an abnormal event.

[0154] When an abnormal event is detected, its root cause needs to be analyzed. First, a system component dependency graph is constructed, which describes the dependencies between different components. Then, a graph traversal algorithm is used to analyze the propagation path of the abnormal event to determine the scope of the affected components. For example, if the database server fails, the application server that depends on the database will also be affected. Finally, a correlation analysis of multi-source abnormal indicators is performed based on the decision tree model to determine the root cause of the failure. For example, by analyzing indicators such as CPU usage, memory usage, and disk IO, it can be determined that the failure is caused by the excessive load on the database server. At the same time, the processing process of each abnormal event is recorded in detail, including alarm conditions, processing operations, and processing results. For example, when the database server is overloaded, an alarm is triggered, and then a capacity expansion operation is performed to eventually reduce the load and restore normal service.

[0155] Next, a reinforcement learning model is built to optimize the migration strategy. Migration strategy parameters (such as resource allocation ratio, synchronization strategy parameters, and fault handling options) are encoded into an action space. A multi-dimensional reward function is designed, which contains performance improvement rewards (such as shortened response time), stability rewards (such as reduced error rate), and resource efficiency rewards (such as improved resource utilization). The reinforcement learning policy network is trained using a deep deterministic policy gradient algorithm, and the sample utilization efficiency is improved through the experience replay mechanism. For example, the resource allocation ratio, synchronization strategy parameters, and fault handling options are encoded as continuous values ​​between 0 and 1, and the reward value is calculated based on the migration drill results.

[0156] A training sample set is constructed using recorded abnormal events and treatment measures, and the sample set is used to continuously optimize the migration risk prediction model. The optimized migration risk prediction model is applied to real-time risk assessment, and the migration policy parameters are dynamically adjusted according to the assessment results. For example, if the risk of excessive database server load is predicted, the resource allocation ratio of the database server can be increased in advance. At the same time, the control policy is output using the reinforcement learning policy network, and the migration execution plan is optimized based on the policy.

[0157] Finally, the Monte Carlo simulation method is used to verify the optimized migration execution plan and evaluate the reliability of the plan under different load conditions and failure scenarios. The probability of risk occurrence and the degree of impact are calculated, and a risk scoring model is established to quantitatively evaluate the remaining risks. For example, 1,000 migration processes are simulated, the number of occurrences of various failures and the losses caused are counted, and the risk scores are calculated. Based on the evaluation results of the risk scoring model and the prediction results of the migration risk prediction model, the optimal migration plan is finally determined.

[0158] The solution of this application can:

[0159] Improve migration efficiency: By optimizing migration strategies through automation and reducing manual intervention, migration time can be shortened and migration efficiency can be improved. Reduce migration risks: By conducting simulation drills and risk assessments in a digital twin environment, potential risks can be identified in advance and corresponding preventive measures can be taken, thereby reducing the probability of failure during the migration process. Improve system performance: By optimizing resource allocation and fault handling strategies, the overall performance and stability of the system can be improved.

[0160] Figure 2 FIG. 1 is a flow chart of a method for scheduling tasks based on a big data platform according to an embodiment of the present invention. Figure 2 As shown, the method includes:

[0161] Big data platform migration task analysis: automatically or manually analyze the pre-migration platform tasks through the migration tool platform, compress them into a zip package, analyze and import them into the post-migration platform, and perform parameter mapping, function mapping, and task dependency mapping between the two platforms before and after the migration, and synchronize the data source information of the two platforms;

[0162] Data migration of the data lake base of the big data platform: Configure relevant synchronization tasks on the migration tool platform to automatically synchronize user accounts and passwords, table metadata, stock data, custom functions, and other data of the metadata lake base to the new data lake base;

[0163] Big data platform scheduling task execution: Through the migration comparison tool, the original big data platform scheduling information can be monitored, thereby driving the scheduling of the new big data platform, implementing a set of scheduling strategies for the two platforms, and achieving task running in parallel;

[0164] Migration comparison cluster deployment: deploy a Hadoop big data storage cluster and a Spark data engine, connect with the big data platform, realize real-time task monitoring, and use it for subsequent task data comparison. SSL encryption is used for data transmission to ensure the security of data transmission and storage.

[0165] Migration comparison strategy: The data task scripts of the new and old platforms are compared by MD5 encryption. The data results of the new and old platforms are extracted to a new Spark cluster at the same time. Through MD5 field-level encryption, the data verification process does not affect the normal use of production services and does not occupy service resources in the existing production environment. According to the actual results of the comparison in production, it can support the comparison of tens of thousands of task data results. Combined with the size of the data, parallel resources are reasonably allocated to ensure the stability of the comparison environment.

[0166] Seamless application migration and switching: After the parallel verification accuracy reaches 100% and is verified for 7 consecutive days, data applications seamlessly switch to use the new big data platform and new data lake base based on actual conditions.

[0167] Based on the analysis of corresponding big data tasks, with the data lake base as the foundation, big data tasks as the link, and big data migration comparison tools as the driver, intelligent big data migration tools are developed based on Internet software. Different big data migration scenarios can be matched according to actual business needs. The background management system is based on the microservice architecture, supports high concurrency, high availability and horizontal expansion capabilities, and the system big data storage is based on the hadoop cluster, which realizes the visualization, ease of use, accuracy and security of big data migration tasks.

[0168] The solution of this application can:

[0169] The present invention utilizes technologies such as big data and cloud computing to achieve dynamic and scientific management of the corresponding big data platform and data lake base migration. Managers can obtain the implementation progress of the big data platform and data lake base migration tasks in real time, which not only achieves efficient management, but also solves the problem that the actual production business cannot ensure seamless migration of production business continuity due to the lack of perfect migration comparison tools during the migration process.

[0170] Figure 3 FIG. 1 is a schematic diagram of a system for scheduling tasks and coordinating smooth migration of data based on a big data platform according to an embodiment of the present invention. Figure 3 As shown, the system comprises:

[0171] The first unit is used to build a digital twin environment of the target system based on the physical topology and resource configuration parameters of the source system, and deploy task status collection probes in the digital twin environment; collect the running status data of the scheduled tasks in the source system, and the running status data includes task execution time, resource occupancy rate, data flow path, task dependency and fault occurrence frequency; use a deep neural network to extract features of the running status data and build a task feature vector matrix; cluster and group the scheduled tasks according to the task feature vector matrix to generate a task group list with a unique identifier;

[0172] The second unit is used to deploy the scheduled tasks in the task group list to the digital twin environment, and record the task operation indicators in real time through the task status collection probe; build a task association analysis model based on the graph neural network to identify the data flow relationship and resource competition relationship between tasks, and compare and analyze the task operation indicators with the operation status data in the source system to generate a task stability score; build a migration risk prediction model based on the task stability score and the analysis results of the task association analysis model, and quantitatively evaluate the migration risk of each task group based on the migration risk prediction model to generate a migration risk assessment report;

[0173] The third unit is used to calculate the migration priority score for each task group based on the migration risk assessment report, and sort the task groups according to the migration priority scores to generate a migration execution sequence; for each task group in the migration execution sequence, formulate a corresponding resource allocation plan, data synchronization strategy and fault recovery plan in combination with its migration risk assessment result; perform migration drills in the digital twin environment and record abnormal events and handling measures, and use a reinforcement learning algorithm to continuously optimize the migration risk prediction model and migration execution plan according to the abnormal events and handling measures, and finally form a verified optimal migration plan.

[0174] According to a third aspect of the embodiments of the present invention,

[0175] An electronic device is provided, comprising:

[0176] processor;

[0177] a memory for storing processor-executable instructions;

[0178] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0179] A fourth aspect of the embodiments of the present invention is:

[0180] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0181] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for smooth migration of tasks and data coordination based on a big data platform, characterized in that: include: Building a digital twin environment of the target system based on the physical topology and resource configuration parameters of the source system, and deploying task status collection probes in the digital twin environment; Collecting the running status data of the scheduled tasks in the source system, the running status data includes the task execution time, resource occupancy rate, data flow path, task dependency and fault occurrence frequency; using a deep neural network to extract features from the running status data and construct a task feature vector matrix; clustering and grouping the scheduled tasks according to the task feature vector matrix and generating a task group list with a unique identifier; Deploy the scheduled tasks in the task group list to the digital twin environment, and record the task operation indicators in real time through the task status collection probe; build a task association analysis model based on the graph neural network to identify the data flow relationship and resource competition relationship between tasks, and compare and analyze the task operation indicators with the operation status data in the source system to generate a task stability score; According to the task stability score and the analysis results of the task association analysis model, a migration risk prediction model is constructed, and based on the migration risk prediction model, a migration risk of each task group is quantitatively evaluated to generate a migration risk assessment report; Calculating a migration priority score for each task group based on the migration risk assessment report, and sorting the task groups according to the migration priority scores to generate a migration execution sequence; For each task group in the migration execution sequence, a corresponding resource allocation plan, data synchronization strategy and fault recovery plan are formulated in combination with the migration risk assessment results; migration drills are performed in the digital twin environment and abnormal events and handling measures are recorded, and a reinforcement learning algorithm is used to continuously optimize the migration risk prediction model and migration execution plan based on the abnormal events and handling measures, ultimately forming a verified optimal migration plan.

2. The method according to claim 1, characterized in that A task association analysis model is built based on a graph neural network to identify the data flow relationship and resource competition relationship between tasks. At the same time, the task operation indicators are compared and analyzed with the operation status data in the source system to generate a task stability score including: Based on the running status data, a task association graph is constructed, wherein the nodes in the task association graph represent the scheduled tasks, and the edges in the task association graph represent the association relationship between tasks; based on the resource occupancy rate, the resource competition between tasks is analyzed to generate a resource competition matrix; according to the data flow path, the data flow frequency and data flow volume between tasks are calculated to generate a data flow matrix; the data flow matrix and the resource competition matrix are merged to form a task association weight matrix; The task association weight matrix is ​​input into a graph neural network, wherein the graph neural network includes multiple graph attention layers, and each graph attention layer calculates attention weights based on node features and edge features; the graph neural network is used to extract association patterns between tasks, identify key task paths and task dependency chains, and generate task association features; a task association analysis model is constructed based on the task association features, and the task association analysis model is used to predict data flow bottlenecks and resource competition conflicts between tasks; Deploy task status collection probes in the digital twin environment to collect task operation indicators in the target environment in real time; compare the task operation indicators with the operation status data in the source system, and calculate the task execution timing deviation, performance indicator difference and abnormal event probability; analyze the propagation impact of task operation anomalies on associated tasks based on the prediction results of the task association analysis model; comprehensively calculate the task stability score based on the task execution timing deviation, the performance indicator difference, the abnormal event probability and the abnormal propagation impact.

3. The method according to claim 1, characterized in that According to the task stability score and the analysis results of the task association analysis model, a migration risk prediction model is constructed, and the migration risk of each task group is quantitatively evaluated based on the migration risk prediction model. The generated migration risk assessment report includes: Acquire task stability features, associated propagation features, and resource-sensitive features as an initial feature set, and build a migration risk prediction model based on the task stability score and the task association analysis model. The migration risk prediction model includes a feature fusion layer, a time series modeling layer, and a risk prediction layer. The feature fusion layer uses an attention mechanism to adaptively weight the features in the initial feature set to obtain a weighted feature vector. The weighted feature vector is input into the time series modeling layer, and the time series modeling layer uses a long short-term memory network to extract risk evolution characteristics. The long short-term memory network selectively retains long-term influencing factors and short-term volatility factors through a gating mechanism; the risk prediction layer includes a gradient boosting tree model and a deep neural network, and the risk evolution characteristics are simultaneously input into the gradient boosting tree model and the deep neural network. The gradient boosting tree model processes the explicit risk characteristics to obtain a first prediction result, and the deep neural network extracts the implicit risk pattern to obtain a second prediction result; the first prediction result and the second prediction result are weightedly fused to obtain a comprehensive risk prediction value; Based on the comprehensive risk prediction value, the risk status of a single task is evaluated, and a weighted combination of task criticality, execution complexity, and failure impact is calculated to obtain a task-level risk score; a task risk association network is constructed to analyze the risk propagation relationship between tasks, and the sensitivity of tasks to system resource fluctuations is evaluated; the task-level risk score, the risk propagation relationship, and the sensitivity to resource fluctuations are combined to generate a task-level risk assessment result; Divide tasks with similar risk characteristics into groups, aggregate the task-level risk assessment results within the same group, and calculate the overall risk value of the group based on the correlation between tasks within the group; establish a dependency graph between groups to analyze the cross-group risk propagation path; perform weighted adjustment on the overall risk value of the group according to the business importance of the group to generate a group-level risk assessment result; Construct a system-level risk propagation graph, use the group-level risk assessment results as the node initial state, and the risk propagation relationship between groups as the edge weight; perform risk diffusion simulation on the system-level risk propagation graph to calculate the cumulative effect of risk in the system; evaluate the overall risk tolerance of the system based on the risk diffusion results, and identify high-risk concentration areas; hierarchically integrate the task-level risk assessment results, the group-level risk assessment results, and the system risk tolerance assessment results to generate a migration risk assessment report.

4. The method according to claim 3, characterized in that: Based on the comprehensive risk prediction value, the risk status of a single task is evaluated, and the weighted combination of task criticality, execution complexity and failure impact is calculated to obtain a task-level risk score including: Establishing a risk assessment benchmark based on the comprehensive risk prediction value, and using the risk assessment benchmark as a reference threshold for calculating the mission criticality, the execution complexity, and the impact of failure; Constructing a task dependency graph, the task dependency graph includes a node set and an edge set, the node set represents a task set, and the edge set represents a dependency relationship between tasks; mapping the comprehensive risk prediction value to the task dependency graph as an initial weight of a node; calculating the criticality of the task using a PageRank algorithm based on the task dependency graph, the criticality of the task is calculated in an iterative manner, and the criticality value of the task in each iteration is obtained by combining the weighted sum of the ratio of the criticality value of the adjacent tasks to the out-degree adjusted by the damping coefficient with the initial weight of the node; The resource utilization characteristics and performance fluctuation characteristics of the task are extracted based on the comprehensive risk prediction value, and the computational complexity of the task is calculated in combination with the resource utilization characteristics, and the computational complexity is calculated based on the time complexity and space complexity; the data complexity is calculated in combination with the performance fluctuation characteristics, and the data complexity is calculated based on the data scale, data type diversity and data quality requirements; the dependency complexity is calculated based on the task dependency graph, and the dependency complexity is calculated based on the in-degree, out-degree and closeness of the dependency relationship of the task; the computational complexity, the data complexity and the dependency complexity are weightedly combined to obtain a task execution complexity score; Construct a fault tree with task failure as the top event, and use the comprehensive risk prediction value as the basic probability value; identify the key fault path through minimum cut set analysis; obtain the probability of occurrence of each minimum cut set based on the basic probability value and historical data statistics, and determine the corresponding severity based on the service interruption duration and impact scope; sum the product of the probability of occurrence and the severity over all minimum cut sets to obtain the task failure impact score; The hierarchical analysis method is used to determine the weight coefficients of the task criticality, the execution complexity and the failure impact; the task criticality, the task execution complexity score and the task failure impact score are multiplied by the corresponding weight coefficients respectively and the sum is calculated to obtain the task-level risk score.

5. The method according to claim 1, characterized in that Calculating a migration priority score for each task group based on the migration risk assessment report, and sorting the task groups according to the migration priority scores, and generating a migration execution sequence includes: Extract the risk level, risk propagation impact and risk tolerance of the task group from the migration risk assessment report to construct a risk feature vector; use the entropy weight method to calculate the risk level weight coefficient, risk propagation impact weight coefficient and risk tolerance weight coefficient according to the data distribution characteristics of the risk feature vector; multiply the risk level, risk propagation impact and risk tolerance by the corresponding weight coefficients respectively and sum them to obtain the first-stage migration priority score; Extract business indicators based on the migration risk assessment report, calculate the business continuity requirement score according to the maximum allowable interruption time and the recovery point target, calculate the revenue contribution score according to the total revenue ratio, and calculate the user impact score according to the number of affected users and the importance of users; use the expert scoring method to determine the business continuity requirement weight coefficient, the revenue contribution weight coefficient and the user impact weight coefficient; multiply the business continuity requirement score, the revenue contribution score and the user impact score by the corresponding weight coefficients respectively and sum them to obtain the second-stage migration priority score; A multi-objective optimization method is used to solve the optimal integration weights of the first-stage migration priority score and the second-stage migration priority score; the first-stage migration priority score and the second-stage migration priority score are respectively multiplied by the corresponding optimal integration weights and the sum is calculated to obtain the final migration priority score of the task group; The task groups are preliminarily sorted according to the final migration priority scores of the task groups to obtain preliminary sorting results; an integer programming model is constructed, the preliminary sorting results are used as the benchmark sequence, and the technical dependency constraints, resource capacity constraints and risk threshold constraints are used as constraints; wherein the technical dependency constraints ensure that the dependent group migrates before the dependent group, the resource capacity constraints ensure that the resource occupancy at any point in time does not exceed the system capacity, and the risk threshold constraints control the number of risk groups that migrate simultaneously; the integer programming model is solved using a genetic algorithm to obtain a final migration execution sequence that meets the constraints.

6. The method according to claim 1, characterized in that For each task group in the migration execution sequence, a corresponding resource allocation plan, data synchronization strategy and fault recovery plan are formulated in combination with the migration risk assessment result, including: Obtaining migration risk assessment results for each task group in the migration execution sequence; extracting CPU usage historical curves, business peak load data, data growth rate, data access mode, data synchronization scale, real-time requirements and key failure point information from the migration risk assessment results; Based on the CPU usage history curve and the business peak load data, a time series analysis method is used to predict the peak demand for computing resources; based on the data growth rate and the data access mode, the storage capacity demand and the input and output operand demand are calculated; based on the data synchronization scale and the real-time requirement, the network bandwidth demand is calculated; the computing resource peak demand, the storage capacity demand, the input and output operand demand and the network bandwidth demand are integrated to form a resource allocation plan; Based on the data synchronization scale, the data synchronization process is divided into a pre-synchronization phase, an incremental synchronization phase, and a switching verification phase, and the pre-synchronization phase, the incremental synchronization phase, and the switching verification phase are weightedly combined to obtain a synchronization mechanism; in the pre-synchronization phase, the full amount of data is divided into multiple data fragments for concurrent transmission; in the incremental synchronization phase, a write buffer pool and a log-based real-time data synchronization mechanism are constructed, and a two-phase commit protocol is adopted; in the switching verification phase, a combination of hash value comparison and sampling verification is used to perform data consistency verification; the synchronization mechanisms are integrated to form a data synchronization strategy; A fault handling decision tree is constructed based on the key fault point information and a fault threshold is set. The fault handling decision tree includes a fault handling branch below a preset fault threshold and a fault handling branch above the preset fault threshold; a fault self-healing mechanism including an automatic retry and a load balancing mechanism is configured in the fault handling branch below the preset fault threshold; a disaster recovery protection mechanism including a system snapshot, an active-active switching channel and an emergency resource pool is configured in the fault handling branch above the preset fault threshold; and the fault handling decision tree is used as a fault recovery plan based on the fault self-healing mechanism and the disaster recovery protection mechanism.

7. The method according to claim 1, characterized in that Perform migration drills in the digital twin environment and record abnormal events and treatment measures. Use reinforcement learning algorithms to continuously optimize the migration risk prediction model and migration execution plan based on the abnormal events and treatment measures, and finally form a verified optimal migration plan including: Using Poisson distribution to generate a user access request sequence, combining with Markov chain to simulate user behavior conversion to obtain load test data; performing migration drills in the digital twin environment based on the load test data, simulating hardware failures, network interruptions and performance degradation scenarios at key nodes of the system through fault injection configuration files; recording abnormal events and handling measures during the migration drill; The sliding window method is used to analyze the changing trend of the task operation indicators, and the indicators that significantly deviate from the normal range are marked as abnormal events; a system component dependency graph is constructed, and a graph traversal algorithm is used to analyze the propagation path of the abnormal events; based on the decision tree model, multi-source abnormal indicators are correlated and analyzed to obtain the root cause of the fault; the processing process of each abnormal event is recorded, including alarm conditions, processing operations and processing results; Encode migration strategy parameters into action space, wherein the migration strategy parameters include resource allocation ratio, synchronization strategy parameters and fault handling options; design a multi-dimensional reward function, wherein the multi-dimensional reward function includes performance improvement reward items, stability reward items and resource efficiency reward items; use a deep deterministic policy gradient algorithm to train a reinforcement learning strategy network, and improve sample utilization efficiency through an experience replay mechanism; Construct a training sample set based on the abnormal event and the treatment measures, and use the training sample set to continuously optimize the migration risk prediction model; apply the optimized migration risk prediction model to real-time risk assessment, and dynamically adjust the migration strategy parameters according to the assessment results; use the reinforcement learning strategy network to output the control strategy, and optimize the migration execution plan based on the control strategy; The optimized migration execution plan is verified by Monte Carlo simulation method, and the reliability of the plan is evaluated under different load conditions and fault scenarios. The probability of risk occurrence and the degree of impact are calculated, and a risk scoring model is established to quantitatively evaluate the residual risk. The final optimal migration plan is determined based on the evaluation results of the risk scoring model and the prediction results of the migration risk prediction model.

8. A system for scheduling tasks and coordinating and smoothly migrating data based on a big data platform, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to build a digital twin environment of the target system based on the physical topology and resource configuration parameters of the source system, and deploy task status collection probes in the digital twin environment; collect the running status data of the scheduled tasks in the source system, and the running status data includes task execution time, resource occupancy rate, data flow path, task dependency and fault occurrence frequency; use a deep neural network to extract features of the running status data and build a task feature vector matrix; cluster and group the scheduled tasks according to the task feature vector matrix to generate a task group list with a unique identifier; The second unit is used to deploy the scheduled tasks in the task group list to the digital twin environment, and record the task operation indicators in real time through the task status acquisition probe; build a task association analysis model based on the graph neural network to identify the data flow relationship and resource competition relationship between tasks, and compare and analyze the task operation indicators with the operation status data in the source system to generate a task stability score; According to the task stability score and the analysis results of the task association analysis model, a migration risk prediction model is constructed, and based on the migration risk prediction model, a migration risk of each task group is quantitatively evaluated to generate a migration risk assessment report; A third unit is used to calculate a migration priority score for each task group based on the migration risk assessment report, and sort the task groups according to the migration priority scores to generate a migration execution sequence; For each task group in the migration execution sequence, a corresponding resource allocation plan, data synchronization strategy and fault recovery plan are formulated in combination with the migration risk assessment results; migration drills are performed in the digital twin environment and abnormal events and handling measures are recorded, and a reinforcement learning algorithm is used to continuously optimize the migration risk prediction model and migration execution plan based on the abnormal events and handling measures, ultimately forming a verified optimal migration plan.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images