A data migration method, device, computer equipment and storage medium

Abstract

This application relates to a data migration method, apparatus, computer equipment, and storage medium. By acquiring monitoring data from each node, extracting features of the target node from the monitoring data, and determining the feature vector of the target node, the node includes at least a primary node and a backup node. The feature vector is input into a machine prediction model to determine the failure probability of the target node and the corresponding reliability index. A comprehensive risk value is generated based on the failure probability and the reliability index, and the comprehensive risk value is compared with a preset set of failure thresholds to determine the migration level. A migration plan is generated based on the migration strategy corresponding to the migration level and the data differences between the primary node and the backup node. Data from the primary node is migrated to the backup node according to the migration plan, which can improve the timeliness of fault handling and reduce service interruptions.

Description

Technical Field

[0001] This application relates to the field of data migration technology, and in particular to a data migration method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the advent of the big data era, single user terminals can no longer meet the storage and computing needs of massive amounts of data, giving rise to the concept of cloud platforms. Cloud platforms can provide users with on-demand software services and development environments, thereby reducing application deployment and maintenance costs. With the commercialization of 5G, the widespread adoption of the Internet of Things, and the continuous increase in terminal types and application scenarios, higher demands are placed on the database reliability of cloud platforms. Common dual-machine hot standby solutions ensure business continuity by setting up a master node and a slave node. The master node handles write requests, while the slave node handles data synchronization. If the master node fails, the slave node can take over its work, thereby improving system availability.

[0003] Currently, cloud management platforms commonly employ a dual-machine hot standby solution with one primary and one backup node. This typically relies on heartbeat detection to determine if the primary node has failed, with the backup node taking over upon detection of an anomaly. However, this approach is a passive failover, often involving delays of several minutes from the occurrence of a failure to the completion of the switchover. This delay can cause service interruptions and negatively impact the system's Service Level Agreement (SLA). Existing dual-machine hot standby mechanisms still have shortcomings: firstly, relying on threshold-based alarms can lead to high false alarm rates, making it difficult to accurately determine node status; secondly, virtual machine-level switching cannot completely avoid hardware failures at the underlying physical machine level, such as disk bad sectors or memory leaks, which are progressive but lack effective prediction. Furthermore, due to insufficient resource warm-up, system performance can significantly degrade shortly after the switchover, impacting overall service quality and user experience. Summary of the Invention

[0004] Therefore, it is necessary to provide a data migration method, apparatus, computer equipment, and storage medium that can improve detection accuracy, enhance fault handling timeliness, and reduce service interruptions to address the aforementioned technical problems.

[0005] Firstly, a data migration method is provided, including: Acquire monitoring data from each node, extract the features of the target node from the monitoring data, and determine the feature vector of the target node. The nodes include at least: the master node and the backup node. The feature vector is input into the machine prediction model to determine the failure probability of the target node and the corresponding confidence index. A comprehensive risk value is generated based on the failure probability and credibility index, and the comprehensive risk value is compared with a set of preset failure thresholds to determine the migration level. A migration plan is generated based on the migration strategy corresponding to the migration level and the data differences between the primary and backup nodes; Migrate the data from the primary node to the backup node according to the migration plan; Upon completion of the migration, a consistency check is performed. If the consistency check fails, a rollback strategy is triggered to restore the state to that before the migration.

[0006] Secondly, a data migration apparatus is provided, the apparatus comprising: The data acquisition module is used to acquire monitoring data from each node, extract the features of the target node from the monitoring data, and determine the feature vector of the target node. The nodes include at least a master node and a backup node. The fault prediction module is used to input feature vectors into the machine prediction model to determine the fault probability of the target node and the corresponding confidence index. The migration level determination module is used to generate a comprehensive risk value based on the failure probability and credibility index, and compare the comprehensive risk value with a preset set of failure thresholds to determine the migration level. The migration plan determination module is used to generate a migration plan based on the migration strategy corresponding to the migration level and the data differences between the primary node and the backup node. The execution module is used to migrate data from the primary node to the backup node according to the migration plan.

[0007] Thirdly, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the data migration method described in the first aspect.

[0008] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the data migration method described in the first aspect is implemented.

[0009] By implementing the aforementioned data migration method, equipment, computer equipment, and storage medium, this method acquires monitoring data from each node and forms a target node feature vector. It then uses a machine prediction model to output the failure probability and its corresponding reliability index, enabling risk assessment to simultaneously provide both probability quantification and reliability basis. Furthermore, it integrates the failure probability and reliability index to obtain a comprehensive risk value, which is compared with a preset set of failure thresholds to determine the migration level. This triggers matching migration intensity and handling rhythm at different risk levels. Subsequently, it generates a migration plan based on the migration strategy corresponding to the migration level and the data differences between primary and backup nodes. This allows for dynamic adjustment of migration resource investment, migration order, and migration scope according to risk changes and data gaps, reducing resource waste and business disruption caused by accidental migration triggers, and minimizing the spread of failures due to delayed migration response when risks escalate. After migration is complete, a consistency check is performed to ensure the correctness of the migration results. If the check fails, a rollback strategy is triggered to restore the state to its pre-migration state, thus limiting the impact of migration failure to a controllable range. Therefore, this application achieves early prediction and proactive migration of node failures, shortening switching latency, reducing service interruption risk, improving database reliability, and ensuring high availability of the cloud platform. Overall, it improves the security, stability, interpretability, and business continuity assurance capabilities of data migration. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a data migration method in one embodiment; Figure 2 This is a structural block diagram of a data migration device in one embodiment; Figure 3 This is a schematic diagram of the data migration model prediction operation process in one embodiment; Figure 4 This is a schematic diagram of the data migration hierarchical migration operation process in one embodiment; Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0012] In one embodiment, such as Figure 1 As shown, a data migration method is provided, including: S100: Acquire monitoring data from each node, extract the features of the target node from the monitoring data, and determine the feature vector of the target node. The nodes include at least: the master node and the backup node. S200: Input the feature vector into the machine prediction model to determine the failure probability of the target node and the corresponding confidence index. S300: Generate a comprehensive risk value based on the failure probability and credibility index, and compare the comprehensive risk value with a set of preset failure thresholds to determine the migration level; S400: Generate a migration plan based on the migration strategy corresponding to the migration level and the data differences between the primary and backup nodes; S500: Migrate the data from the primary node to the backup node according to the migration plan; S600: In response to the completion of migration, a consistency check is performed. If the consistency check fails, a rollback strategy is triggered to restore the state to the state before migration.

[0013] In this context, a node refers to a basic computing unit in a distributed system or cloud platform that undertakes data processing and storage tasks. Monitoring data includes indicators that reflect the node's operating status, such as CPU utilization, memory usage, disk I / O, network latency, and logs. The machine learning model is an intelligent algorithm model trained on historical operating data. The preset fault threshold set refers to a set of thresholds that divide the comprehensive risk value into different risk ranges, used to map migration levels. The target node is the node selected for risk assessment and migration trigger judgment during a migration decision process. After obtaining the monitoring data of each node, a rapid health screening or pre-assessment can be performed on each node (e.g., forming a candidate set based on heartbeat status, error rate, resource level, alarm level, etc.). Then, the node with the highest comprehensive risk or exceeding the preset warning threshold is selected from the candidate set as the target node. If there are multiple candidate nodes, they can be determined sequentially according to business priority, impact scope, or risk value. After the target node is determined, the system extracts the data items corresponding to the target node from the monitoring data of each node and performs feature extraction to form a feature vector. Consistency verification refers to the process of verifying the consistency between primary and backup data and critical status after migration. A rollback strategy is a solution to restore the system to its stable state before migration when consistency checks fail, in order to avoid the risk of partial migration or inconsistency.

[0014] Specifically, after collecting and aggregating the operational monitoring data of each node, the system first grasps the status changes related to various resources, performance, and reliability from the overall perspective of the master and backup nodes. Then, it selects the target node to be evaluated and extracts key features characterizing its health and fault symptoms from the monitoring data of that target node. These features are further organized into a unified feature vector as model input. Through this vectorized expression, scattered and inconsistent monitoring signals are transformed into a calculable and reusable standardized data foundation, reducing judgment bias caused by inconsistent indicator dimensions. Subsequently, the system outputs the failure probability of the target node and its corresponding reliability index based on the feature vector. This ensures that the risk assessment simultaneously includes both the probability of occurrence and the reliability of the result, avoiding unnecessary switching caused by probability-triggered migration in the case of data noise, missing data, or distribution drift. Next, the failure probability and reliability are fused to obtain a comprehensive risk value, which is compared with a preset set of failure thresholds to determine the migration level. This allows the risk result to be mapped to an executable and tiered migration intensity, thereby suppressing excessive migration at low risk and promptly increasing migration intensity at high risk to compress the failure impact window. Then, based on the migration strategy corresponding to the migration level and combined with the data differences between the primary and backup nodes, a migration plan is generated to match the migration scope, sequence, synchronization depth, and resource investment with the actual differences, thereby improving migration efficiency and process controllability. After the data migration from the primary node to the backup node is completed according to the migration plan, the system performs a consistency check to confirm the reliability of the migration result; if the check fails, a rollback is triggered to restore the system to the state before the migration, avoiding the system remaining in a risky state of partial migration or inconsistency. This reduces the cost of accidental migration and business disruption, while improving the timeliness of migration decisions and the security of the migration process in high-risk scenarios, and enhancing the data consistency guarantee and business continuity capabilities of the primary and backup system.

[0015] In one embodiment, a comprehensive risk value is compared with a preset set of fault thresholds to determine a migration level. The migration levels include Level 1 migration, Level 2 migration, and Level 3 migration. The fault threshold set includes a first threshold, a second threshold, and a third threshold. If the overall risk value is greater than or equal to the first threshold and less than the second threshold, the migration level is Level 1 migration; If the overall risk value is greater than or equal to the second threshold and less than the third threshold, the migration level is Level 2 migration; If the overall risk value is greater than or equal to the third threshold, the migration level is three-level migration, where the first threshold is less than the second threshold, and the second threshold is less than the third threshold.

[0016] The first, second, and third thresholds correspond to the criteria for determining low, medium, and high risk, respectively. A failure probability greater than or equal to the first threshold is classified as a Level 1 migration; a failure probability greater than or equal to the second threshold but less than the third threshold is classified as a Level 2 migration; and a failure probability greater than or equal to the third threshold is classified as a Level 3 migration. The failure probability is a value obtained by the system through real-time monitoring and calculation of the link status, such as modeling and predicting based on indicators like packet loss rate, latency, jitter, or error rate, thereby quantifying the likelihood of a link failure.

[0017] Specifically, through a multi-tiered migration level design, node risks can be responded to in a graded manner, thus avoiding frequent migrations caused by overly conservative single threshold settings and preventing response lag issues caused by excessively high thresholds. The migration strategy can escalate progressively with the node's risk level. Lightweight measures can be taken in advance to reduce the probability of failure in the early stages of risk, and a rapid switch to more stringent data synchronization and business takeover operations can be initiated as the risk intensifies. Finally, a timely switchover can be completed under high-risk conditions to ensure uninterrupted business operations. This mechanism allows the system to balance performance and resource consumption while ensuring reliability, improving the accuracy and flexibility of fault handling and enhancing the stability and availability of the entire database or cloud platform. Matching different migration levels to different risk levels avoids resource waste caused by frequent large-scale switches triggered by minor fluctuations, and allows for rapid and robust fault-tolerant measures in truly high-risk scenarios, ensuring high availability of links and stability of data transmission. Simultaneously, this graded mechanism effectively reduces the false trigger rate, improves the system's adaptability and robustness to complex network environments, and gives the overall network stronger self-healing capabilities and continuous service capabilities.

[0018] In one embodiment, such as Figure 4 As shown, based on the migration strategy corresponding to the migration level and the data differences between the primary and backup nodes, a migration plan is generated, including: In response to a migration level of Level 1, the migration plan should include at least the following: enabling the preset computing resources of the standby node and establishing a shadow synchronization channel to synchronize the recorded incremental changes of the primary node to the standby node. In response to a migration level of 2, the migration plan should include at least the following: enabling all computing resources on the standby node, determining the hierarchical migration order according to data importance, and migrating high-priority data to the standby node first; In response to a Level 3 migration, the migration plan must include at least the following: performing a full migration from the primary node to the backup node and triggering session takeover to complete the business switchover; The importance of data is determined based on at least one or any combination of access popularity, write frequency, and business tags.

[0019] Among them, the shadow synchronization channel refers to the synchronization link used to continuously replicate incremental changes generated by the primary node to the backup node in a bypass or shadow manner without triggering service switching; the preset computing resources refer to a portion of computing and storage resource quotas pre-allocated to the backup node and enabled as needed, used to complete link establishment, synchronization warm-up and incremental alignment during the low-level migration phase, avoiding the one-time occupation of all resources; session takeover refers to the process of switching the business session or connection context running on the primary node to the backup node to continue providing services during service switching, in order to reduce interruptions and achieve smooth switching.

[0020] Specifically, the system selects the appropriate migration strategy based on the migration level and generates an executable migration plan based on the data differences between the primary and backup nodes: When the migration level is Level 1, the system first activates the backup node's preset computing resources, establishes a shadow synchronization channel, and continuously synchronizes incremental changes generated and recorded by the primary node to the backup node, thereby completing link warm-up and incremental alignment without business switching, reducing catch-up costs and switching risks in subsequent migration phases; When the migration level is upgraded to Level 2, the system further activates all computing resources of the backup node and determines the hierarchical migration order according to data importance, migrating high-priority data to the backup node first, so that critical data reaches a usable and consistent state earlier, thereby prioritizing the continuity of core business and reducing resource waste caused by ineffective migration when the migration window is limited or the risk increases; When the migration level is Level 3, the system performs a full migration from the primary node to the backup node and triggers session takeover to complete the business switch, thereby shortening the recovery path, accelerating the takeover speed, and achieving fault isolation in high-risk scenarios. Through the progressive planning generation mechanism described above, from "incremental shadow synchronization warm-up" to "layered priority migration" and then to "full migration and takeover", this solution can link migration intensity with risk level, data differences and data importance, which not only improves the feasibility and stability of migration decisions, but also achieves better resource utilization and lower business disruption at different risk stages, thereby improving the overall migration success rate and business continuity assurance capabilities.

[0021] In one embodiment, such as Figure 3 As shown, the machine learning models include: a long short-term memory neural network model, an extreme gradient boosting tree model, and an isolated forest anomaly detection model. Feature vectors are input into the machine prediction model to determine the failure probability of the target node and the corresponding confidence index, including: The first risk score is obtained by calculating the feature vector through a long short-term memory neural network model; The second risk score is obtained by calculating the feature vector using an extreme gradient boosting tree model. The third risk score is obtained by calculating the feature vector through the isolated forest anomaly detection model; The confidence levels of the first risk score, the second risk score, and the third risk score were assessed respectively to obtain the first confidence index, the second confidence index, and the third confidence index. The model weight set is determined based on the first confidence index, the second confidence index, and the third confidence index. The model weight set includes at least the first model weight, the second model weight, and the third model weight, and the model weight set is adaptively updated according to the recent prediction error or data drift of the target node. The first risk score, the second risk score, and the third risk score are weighted and fused based on the adaptively updated model weight set to obtain a fused score. The probability of failure is determined by probabilistic calibration of the fused score. The credibility index is determined based on the first confidence index, the second confidence index, and the third confidence index.

[0022] Among them, Long Short-Term Memory (LSTM) is an improved recurrent neural network that can capture long-term dependencies in time series data, making it suitable for dynamic trend modeling and risk prediction of node operating status data; Extreme Gradient Boosting Tree (XGBoost) is an efficient ensemble learning algorithm that can construct nonlinear relationships based on multi-dimensional features, thereby outputting stable risk scores under complex index conditions; Isolation Forest anomaly detection algorithm is an unsupervised learning algorithm based on data space partitioning, which can efficiently identify potential anomalous nodes by isolating outliers; Feature vector refers to the numerical representation of the target node's operating status data within a preset time window, formed by feature extraction and concatenation, which can be used as input to the model; Risk score refers to the intermediate quantity output by different machine learning models based on feature vectors, used to characterize the strength of fault risk; Probability calibration refers to the process of converting the fused score into a fault probability output that is more consistent with the frequency of actual fault occurrence, so that the output probability has interpretable statistical significance; Fusion score refers to the comprehensive score obtained by weighted fusion of risk scores from multiple models according to model weights; Credibility index refers to the index used to characterize the overall reliability of the output fault probability under the current data distribution and model state.

[0023] Specifically, the operational state of the target node is extracted as a feature vector, and then input into a Long Short-Term Memory Neural Network model, an Extreme Gradient Boosting Tree model, and an Isolated Forest Anomaly Detection model, respectively, to obtain three risk scores. These scores cover complementary capabilities such as temporal dependence, nonlinear relationships of structured features, and anomaly pattern detection. Subsequently, the confidence levels of the three risk scores are evaluated to form first, second, and third confidence indices. Based on these indices, a model weight set containing the weights of the first, second, and third models is generated. This weight set is adaptively updated according to the recent prediction error or data drift of the target node, thereby automatically reducing the impact of the corresponding model on the final result and improving performance stability when the model's recent performance deteriorates or the data distribution changes. The system contributes to the model's contribution. Based on this, the system uses adaptively updated model weights to weight and fuse the three risk scores to obtain a fusion score. The fusion score is then probabilistically calibrated to output the failure probability. Simultaneously, a credibility index is obtained by converging the three confidence indices to characterize the overall reliability of the failure probability in the current scenario. Through the above process, this solution not only achieves more robust failure probability estimation under multi-model complementarity, but also explicitly incorporates model quality and data stability into the decision through a closed-loop mechanism of "confidence-weight-fusion-calibration". This ensures that the stability and interpretability of the output can be maintained even when there is concept drift, noise disturbance, or single model inaccuracy, reducing the risk of migration false triggering or delayed triggering caused by false alarms and false negatives.

[0024] In one embodiment, before weighted fusion of the first risk score, the second risk score, and the third risk score to obtain the failure probability, the following steps are included: Determine whether the first risk score, second risk score, and third risk score are within the normalization range based on the normalization range; If any of the first risk score, second risk score, and third risk score is not within the normalization range, then the risk score is normalized. according to: , Determine the normalized risk score, where, This represents the original risk score. This represents the lower limit of the original range of the risk score. This represents the upper limit of the original range of the risk score. This represents the normalized risk score, mapped to the range [0,1].

[0025] The normalization range refers to the preset numerical mapping interval, which is [0,1] in this embodiment.

[0026] Specifically, for example, assuming the first risk score output by the Long Short-Term Memory Neural Network is 70, and the original range of the model's risk score is 0, 1000, 100, then the normalized result is: , The first risk score, after normalization, is 0.7. If the third risk score output by the Isolation Forest anomaly detection algorithm is -2, and the original range of the algorithm's risk score is -5, 5, then the normalized result is: , The third risk score, after normalization, is 0.3. After normalization, the risk scores output by the models are uniformly mapped to the [0,1] interval, enabling weighted fusion at the same scale. This avoids the problem of distorted fusion results caused by differences in the risk score ranges of different models. Since the output ranges of Long Short-Term Memory Neural Networks, Extreme Gradient Boosting Tree Models, and Isolated Forest Anomaly Detection Algorithms may differ, direct fusion would cause uneven weighting and computational bias. Normalization unifies the values ​​from different distributions into a standardized interval, ensuring the fairness of each model's contribution and the accuracy of the fusion calculation. Simultaneously, it enhances the system's scalability; when adding new machine learning models, only their outputs need to be normalized to seamlessly integrate into the existing framework, thereby improving the system's flexibility and robustness.

[0027] In one embodiment, migrating data from the primary node to the backup node according to the migration strategy corresponding to the migration level further includes: Initialize the dirty page bitmap and record the initial modification state of the memory pages in the master node's memory; Monitor memory write operations on the master node and mark the modified memory pages in the dirty page bitmap; Scan the dirty page bitmap, extract the marked memory pages, and generate a list of memory pages to be synchronized; The data in the list of memory pages to be synchronized is transferred to the standby node using the differential replication protocol; Calculate the hash value of the master node disk data block and generate the master node disk data identifier; Obtain the hash value of the standby node disk data block and generate the standby node disk data identifier; Compare the disk data identifiers of the primary node and the backup node to identify data blocks with inconsistent hash values ​​and generate a list of disk data blocks to be synchronized. The data in the list of disk data blocks to be synchronized is transferred to the backup node through a block-level incremental synchronization protocol. Parse the master node's network protocol stack, extract the sequence number, acknowledgment number, and window size of the current TCP session, and generate connection context data; The connection context data is transmitted to the backup node via the TCP session mirroring protocol.

[0028] The dirty page bitmap is a data structure used to record the modification status of memory pages. Each bit corresponds to a memory page, and when the page content is written to or modified, the corresponding bit is marked as "dirty," facilitating quick location of memory data that needs synchronization. Differential replication protocol is a synchronization protocol based on data differences, transmitting only the data content that has changed between the primary and backup nodes, thereby reducing redundant transmission and improving memory synchronization efficiency. Primary / backup disk data identifiers are unique identifiers generated by calculating hash values ​​for disk data blocks, used to determine whether the disk data on the primary and backup nodes is consistent. A hash value is a fixed-length digest information calculated by a hash function; different data content generates different hash values, while the same content generates the same hash value, suitable for quickly comparing whether data blocks are consistent. The list of disk data blocks to be synchronized refers to the set of data blocks whose content is found to be inconsistent after comparing the disk data identifiers of the primary and backup nodes. These data blocks are selected and transmitted to the backup node to maintain consistency. The block-level incremental synchronization protocol is a protocol for data differential synchronization on a per-disk-block basis, synchronizing only the primary and backup nodes. Differences exist between nodes, avoiding full copying and reducing disk I / O and network transmission pressure during migration; the network protocol stack refers to the layered protocol set that implements network communication in a computer system, responsible for data encapsulation, transmission, parsing, and reassembly. This embodiment mainly involves TCP protocol processing at the transport layer; connection context data: runtime state information of the TCP connection, including sequence number, acknowledgment number, and window size, used to describe the transmission progress and flow control status of the session; sequence number: a number used in the TCP session to identify the order of data bytes, ensuring that data can be assembled in the correct order; acknowledgment number: a number used in the TCP session to confirm that data has been successfully received, notifying the sender of the starting position of the next data to be transmitted; window size: a flow control parameter in the TCP session, used to represent the buffer capacity that the receiver can receive, thereby adjusting the sending rate and avoiding network congestion; TCP session mirroring protocol: a protocol that copies TCP connection state information from the primary node to the backup node, enabling the backup node to inherit the session state of the primary node, thereby achieving seamless takeover during failover and avoiding the application layer re-establishing the connection.

[0029] Specifically, by initializing the dirty page bitmap on the primary node and recording the modification status of memory pages in real time, the system can accurately identify changed memory pages, avoiding redundant transmission of unmodified data. Combined with the differential replication protocol, only dirty pages are transmitted to the standby node, significantly reducing bandwidth consumption and latency during memory migration. During disk data synchronization, a unique identifier is generated by calculating the hash value of disk data blocks on the primary and standby nodes, and inconsistent data blocks are identified by comparison. Block-level incremental synchronization is performed only on the differing parts. This approach ensures the consistency of disk data between the primary and standby nodes while avoiding the resource waste and long latency caused by full replication. At the same time, the system parses the network protocol stack of the primary node, extracts key information such as the TCP session sequence number, acknowledgment number, and window size, forms connection context data, and transmits it to the standby node via the TCP session mirroring protocol. This allows the standby node to seamlessly inherit the network session state of the primary node, eliminating the need for the application layer to re-establish connections during failover, thus ensuring business continuity and user experience stability. By combining memory differential copying, disk block-level incremental synchronization, and TCP session mirroring, the system significantly improves migration efficiency and reduces migration overhead while ensuring data consistency. Ultimately, it achieves the technical effect of flexibly adapting resource investment at different migration levels while ensuring high availability of business systems.

[0030] In one embodiment, a comprehensive risk value is generated based on the failure probability and a credibility index, including: according to: , Determine the overall risk value, whereby, This represents the overall risk value. Indicates credibility index, This represents the risk score output by the i-th model at time t. This represents the weight of the i-th model. , indicating the fusion score, Indicates the probability of failure. Indicates the rate of change of the probability of failure. This represents the dynamic adjustment coefficient, and n represents the number of models participating in the fusion.

[0031] Specifically, according to: , Determine the rate of change of the failure probability, where, This represents the failure probability at the current evaluation time t. This represents the probability of failure at the previous assessment time. The time interval between two adjacent assessments (e.g., 1 minute, 5 minutes, or a sampling period) can be used to approximate the difference between the two assessments. The rate of increase or decrease of the failure probability within a short time window can be obtained through the above difference approximation, which is used to reflect the dynamic trend of risk changes. according to: , Determine credibility indicators, among which Indicates credibility index, This indicates the first confidence level. This indicates the second confidence level. This indicates the third confidence level. This indicates the weight model corresponding to the first confidence level index. This indicates the weight model corresponding to the second confidence level. This indicates the weight model corresponding to the third confidence index. This represents the confidence convergence function, used to merge the confidence scores of multiple models into a single confidence index.

[0032] At time t, a target node is evaluated, and the previous time was... ,Pick (One sampling period). The fault probability has been obtained through probability calibration. , The rate of change of the failure probability can be approximated by the difference as follows: , Meanwhile, the confidence indices for the three models are as follows: , , The corresponding model weights are: , , The credibility aggregation adopts weighted aggregation: , Based on this, three risk scores are set as follows: , , Take the dynamic adjustment coefficient The fusion score is then: , Substituting the values ​​yields the overall risk value: ; The overall risk value of the target node at time t is determined to be approximately 0.434.

[0033] In one embodiment, in response to the completion of data migration, a consistency check is performed on the data, including: Calculate the checksum for the data before and after migration, respectively. The checksum is a data digest obtained by a preset hash function or checksum algorithm. Upon detecting that the data checksums of the primary and backup nodes are consistent, the migration is confirmed to be complete. If the data verification values ​​of the primary node and the backup node are found to be inconsistent, the difference compensation mechanism is triggered to resynchronize the inconsistent data to the backup node.

[0034] The checksum refers to a fixed-length data digest generated by applying a hash function or checksum algorithm to a data block. It is used to uniquely identify the consistency of the data content. For example, commonly used hash functions such as MD5 and SHA-256 can be used as checksum algorithms. A hash function is a function that maps an input of arbitrary length to a fixed-length output. The output result is usually called a hash value. Because hash functions are collision resistant, the same input will inevitably produce the same output, so they are suitable for data integrity verification. The data digest is a short identification information generated by a hash function or checksum algorithm, used to quickly determine whether a data block has been modified or whether an error has occurred during transmission. The difference compensation mechanism refers to the retransmission or repair process that is automatically triggered when inconsistencies are detected in the data verification between the primary and backup nodes. By resynchronizing the difference data, the final data consistency is ensured.

[0035] Specifically, a checksum is calculated for the data blocks before and after the migration. The checksum is a data digest obtained through a preset hash function or checksum algorithm. By comparing the checksums of the data blocks on the primary and backup nodes, when the checksums of the primary and backup data are found to be consistent, the migration is confirmed to be complete, ensuring that the data after the migration is completely consistent with the data before the migration, thereby ensuring the correctness of the business data when the backup node takes over. When the checksums of the primary and backup data are found to be inconsistent, a difference compensation mechanism is triggered to resynchronize the inconsistent data to the backup node. The eventual consistency is ensured through resynchronization, avoiding potential failures caused by network jitter, data loss, or transmission errors.

[0036] In one embodiment, confidence levels are assessed for the first risk score, the second risk score, and the third risk score, respectively, to obtain a first confidence index, a second confidence index, and a third confidence index, including: Acquire historical sample data of the target node within a preset time window and the corresponding actual fault results; The recent prediction errors of the Long Short-Term Memory Neural Network Model, the Extreme Gradient Boosting Tree Model, and the Isolated Forest Anomaly Detection Model within a preset time window are statistically analyzed. The recent prediction error is determined based on at least one or any combination of the following: the deviation between the prediction result and the actual fault result, the prediction calibration deviation, the false alarm rate, and the false negative rate. Calculate the feature distribution changes of the target node within a preset time window to obtain the degree of data drift. The degree of data drift is determined based on at least one or any combination of the following: feature mean change, variance change, quantile change, and distribution similarity change. The first confidence index, the second confidence index, and the third confidence index are generated based on the recent prediction error and the degree of data drift of each model, so that the smaller the recent prediction error and the lower the degree of data drift, the higher the corresponding confidence index. A set of model weights is generated based on the first confidence index, the second confidence index, and the third confidence index, so that the model with the higher the confidence index has the larger the model weight, and the set of model weights is normalized. A credibility index is generated based on the first confidence index, the second confidence index, and the third confidence index to characterize the overall reliability of the failure probability.

[0037] On the other hand, a data migration device is also provided, such as Figure 2 As shown, the device includes: a data acquisition module 210, a fault prediction module 220, a migration level determination module 230, a migration plan determination module 240, and an execution module 250, used for: The data acquisition module 210 is used to acquire monitoring data from each node, extract the features of the target node from the monitoring data, and determine the feature vector of the target node. The nodes include at least a master node and a backup node. The fault prediction module 220 is used to input the feature vector into the machine prediction model to determine the fault probability of the target node and the confidence index corresponding to the fault probability. The migration level determination module 230 is used to generate a comprehensive risk value based on the failure probability and credibility index, and compare the comprehensive risk value with a preset set of failure thresholds to determine the migration level. The migration plan determination module 240 is used to generate a migration plan based on the migration strategy corresponding to the migration level and the data differences between the primary node and the backup node. Execution module 250 is used to migrate data from the primary node to the backup node according to the migration plan; In one embodiment, the fault prediction module 220 is used for: The first risk score is obtained by calculating the feature vector through a long short-term memory neural network model; The second risk score is obtained by calculating the feature vector using an extreme gradient boosting tree model. The third risk score is obtained by calculating the feature vector through the isolated forest anomaly detection model; The confidence levels of the first risk score, the second risk score, and the third risk score were assessed respectively to obtain the first confidence index, the second confidence index, and the third confidence index. The model weight set is determined based on the first confidence index, the second confidence index, and the third confidence index. The model weight set includes at least the first model weight, the second model weight, and the third model weight, and the model weight set is adaptively updated according to the recent prediction error or data drift of the target node. The first risk score, the second risk score, and the third risk score are weighted and fused based on the adaptively updated model weight set to obtain a fused score. The probability of failure is determined by probabilistic calibration of the fused score. The confidence index is determined based on the first confidence index, the second confidence index, and the third confidence index. Specific limitations regarding an application to a data migration device can be found in the limitations of the data migration method described above, and will not be repeated here. The various modules in the aforementioned device for improving data write performance in a storage system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0038] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described data migration method embodiments at runtime.

[0039] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0040] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data migration method embodiments.

[0041] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both, such as Figure 5As shown, to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0042] The data migration method provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data migration method, characterized in that, include: Acquire monitoring data from each node, extract features of the target node from the monitoring data, and determine the feature vector of the target node. The nodes include at least a master node and a backup node. The feature vector is input into the machine prediction model to determine the failure probability of the target node and the confidence index corresponding to the failure probability. A comprehensive risk value is generated based on the failure probability and the credibility index, and the comprehensive risk value is compared with a preset set of failure thresholds to determine the migration level. A migration plan is generated based on the migration strategy corresponding to the migration level and the data differences between the primary node and the backup node; The data of the primary node is migrated to the backup node according to the migration plan.

2. The data migration method according to claim 1, characterized in that, The comprehensive risk value is compared with a preset set of fault thresholds to determine the migration level. The migration levels include: Level 1 migration, Level 2 migration, and Level 3 migration. The set of fault thresholds includes: a first threshold, a second threshold, and a third threshold. If the overall risk value is greater than or equal to the first threshold and less than the second threshold, then the migration level is the first-level migration. If the overall risk value is greater than or equal to the second threshold and less than the third threshold, then the migration level is the second-level migration. If the overall risk value is greater than or equal to the third threshold, the migration level is the third-level migration, wherein the first threshold is less than the second threshold, and the second threshold is less than the third threshold.

3. The data migration method according to claim 2, characterized in that, The step of generating a migration plan based on the migration strategy corresponding to the migration level and the data differences between the primary node and the backup node includes: In response to the migration level being Level 1 migration, the migration plan includes at least: enabling the preset computing resources of the backup node and establishing a shadow synchronization channel to synchronize the recorded incremental changes of the primary node to the backup node; In response to the migration level being a level 2 migration, the migration plan includes at least: enabling all computing resources of the backup node, determining the hierarchical migration order according to data importance, and migrating high-priority data to the backup node first; In response to the migration level being Level 3, the migration plan includes at least: performing a full migration from the primary node to the backup node, and triggering session takeover to complete the service switchover; The importance of the data is determined based on at least one or any combination of access popularity, write frequency, and business tags.

4. The data migration method according to claim 1, characterized in that, The machine learning model includes: a long short-term memory neural network model, an extreme gradient boosting tree model, and an isolated forest anomaly detection model. The step of inputting the feature vector into the machine prediction model to determine the failure probability of the target node and the corresponding confidence index includes: The feature vector is calculated using the long short-term memory neural network model to obtain the first risk score; The feature vector is calculated using the extreme gradient boosting tree model to obtain the second risk score; The feature vector is calculated using the isolated forest anomaly detection model to obtain the third risk score; The confidence levels of the first risk score, the second risk score, and the third risk score are assessed respectively to obtain a first confidence index, a second confidence index, and a third confidence index. The model weight set is determined based on the first confidence index, the second confidence index, and the third confidence index. The model weight set includes at least the first model weight, the second model weight, and the third model weight, and the model weight set is adaptively updated according to the recent prediction error or data drift degree of the target node. The first risk score, the second risk score, and the third risk score are weighted and fused based on the adaptively updated model weight set to obtain a fused score. The probability of failure is determined by probabilistic calibration of the fused score. The credibility index is determined based on the first confidence index, the second confidence index, and the third confidence index.

5. The data migration method according to claim 4, characterized in that, Before weightedly fusing the first risk score, the second risk score, and the third risk score to obtain the failure probability, the following steps are included: Determine whether the first risk score, the second risk score, and the third risk score are within the normalization range; If any one of the first risk score, the second risk score, and the third risk score is not within the normalization range, then the risk score is normalized. according to ， Determine the normalized risk score, where, This represents the original risk score. This represents the lower limit of the original range of the risk score. This represents the upper limit of the original range of the risk score. This represents the normalized risk score, mapped to the range [0,1].

6. The data migration method according to claim 1, characterized in that, The step of migrating the data from the primary node to the backup node according to the migration plan includes: Initialize the dirty page bitmap and record the initial modification state of the memory pages in the master node's memory; Monitor the memory write operations of the master node and mark the modified memory pages in the dirty page bitmap; Scan the dirty page bitmap, extract the marked memory pages, and generate a list of memory pages to be synchronized; The data in the list of memory pages to be synchronized is transferred to the backup node using a differential replication protocol. Calculate the hash value of the master node disk data block and generate the master node disk data identifier; Obtain the hash value of the standby node disk data block and generate the standby node disk data identifier; The master node disk data identifier and the backup node disk data identifier are compared to identify data blocks with inconsistent hash values ​​and generate a list of disk data blocks to be synchronized. The data in the list of disk data blocks to be synchronized is transmitted to the backup node through a block-level incremental synchronization protocol. Parse the master node's network protocol stack, extract the sequence number, acknowledgment number, and window size of the current TCP session, and generate connection context data; The connection context data is transmitted to the backup node via the TCP session mirroring protocol.

7. The data migration method according to claim 4, characterized in that, The generation of a comprehensive risk value based on the failure probability and the credibility index includes: according to: ， Determine the overall risk value, whereby, This represents the overall risk value. Indicates credibility index, This represents the risk score output by the i-th model at time t. This represents the weight of the i-th model. , indicating the fusion score, Indicates the probability of failure. Indicates the rate of change of the probability of failure. This represents the dynamic adjustment coefficient, and n represents the number of models participating in the fusion.

8. A data migration device, characterized in that, The device includes: The data acquisition module is used to acquire monitoring data from each node, extract the features of the target node from the monitoring data, and determine the feature vector of the target node. The nodes include at least a master node and a backup node. The fault prediction module is used to input feature vectors into the machine prediction model to determine the fault probability of the target node and the corresponding confidence index. The migration level determination module is used to generate a comprehensive risk value based on the failure probability and credibility index, and compare the comprehensive risk value with a preset set of failure thresholds to determine the migration level. The migration plan determination module is used to generate a migration plan based on the migration strategy corresponding to the migration level and the data differences between the primary node and the backup node. The execution module is used to migrate data from the primary node to the backup node according to the migration plan.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images