Snapshot checking and repairing method for service non-stop source end

Abstract

The invention discloses a snapshot checking and repairing method for a service non-stop source end, and relates to the technical field of data processing, the method comprises the following steps: according to a user instruction, activating a non-stop snapshot checking mode, pausing data storage of a target end, but not pausing data synchronization of the source end; obtaining a source end snapshot from a source end, and determining a source end submission transaction in the current snapshot; synchronizing data storage at the target end until the data is synchronized to the source end to submit the transaction, and generating a target end snapshot; and comparing the source end snapshot with the target end snapshot, if there is a difference, performing data restoration based on difference data, and after restoration is completed, recovering the target end data and storing the target end data in a database. According to the data synchronization and verification method and device, the technical problem that the continuity and availability of services are affected due to the fact that the source end services need to be paused for data verification in an existing data synchronization and verification method is solved, and the technical effects that efficient data verification and restoration are carried out under the condition that the source end services are not shut down, and the continuity and availability of the services are ensured are achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a snapshot verification and repair method for a source end without service downtime. Background Art

[0002] In the field of data synchronization, ensuring the consistency of source and target data is a critical task. However, existing data synchronization and verification methods often face many challenges when facing complex scenarios of real-time data synchronization. Traditional methods usually ensure data consistency by periodically suspending source business and performing full data verification, but this approach not only affects the continuity of source business, but also causes synchronization service interruptions, affecting business continuity and availability. Summary of the invention

[0003] The present application provides a snapshot verification and repair method for a source end without business downtime, which is used to solve the technical problem that the existing data synchronization and verification methods require suspending the source end business for data verification, affecting the continuity and availability of the business.

[0004] The present application provides a snapshot verification and repair method for a source end without business downtime, the method comprising: configuring a non-stop verification switch; receiving a user instruction, enabling the non-stop verification switch, activating a non-stop snapshot verification mode, suspending target end data storage according to the non-stop snapshot verification mode, but not suspending source end data synchronization; obtaining a source end snapshot from the source end, and determining the source end commit transaction in the current snapshot; synchronizing data storage at the target end until synchronization to the source end commit transaction, generating a target end snapshot; combining with an intelligent verification mechanism, reading data from the source end snapshot and the target end snapshot respectively for comparison, and generating a verification comparison result; if there is a difference in the verification comparison result, performing data repair based on the difference data, and resuming the target end data storage after the repair is completed.

[0005] One or more technical solutions provided in this application have at least the following technical effects or advantages: The snapshot verification and repair method for the source end without business interruption provided by the present application relates to the field of data processing technology. By configuring a non-stop verification switch and enabling a non-stop snapshot verification mode, snapshots of the source end and the target end are obtained without stopping the source end data synchronization, and data comparison is performed in combination with an intelligent verification mechanism. If there are differences, repairs are performed based on the difference data to ensure data consistency and then restore the target end data into the warehouse. The technical problem that the existing data synchronization and verification methods need to suspend the source end business for data verification, which affects the continuity and availability of the business, is solved. Efficient data verification and repair are achieved without stopping the source end business, thereby improving data consistency and repair efficiency and ensuring the technical effect of business continuity and availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0007] Figure 1 A flowchart of a snapshot verification and repair method for a source end without service downtime provided in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of activating a non-stop snapshot verification mode in a snapshot verification and repair method for a source end without service downtime provided in an embodiment of the present application; Figure 3 A schematic diagram of a flow chart of generating verification and comparison results in a snapshot verification and repair method for a source end without service downtime provided in an embodiment of the present application. DETAILED DESCRIPTION

[0008] The present application provides a snapshot verification and repair method for a source end without business downtime, which is used to solve the technical problem that the existing data synchronization and verification methods require suspending the source end business for data verification, affecting the continuity and availability of the business.

[0009] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0010] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices.

[0011] Examples, such as Figure 1 As shown, the present application provides a snapshot verification and repair method for a source end without service downtime, the method comprising: P10: Configure the non-stop verification switch.

[0012] Specifically, a switch is configured to control the non-stop verification function. This step is the basis for the implementation of the method, and aims to provide a flexible configuration mechanism so that users can decide whether to enable the non-stop snapshot verification mode according to actual needs. Through this configuration switch, users can flexibly choose the time to perform data verification without affecting the normal operation of the source database business, ensuring consistency and reliability during data synchronization.

[0013] This configuration switch is a logical control point that determines whether to start the non-stop snapshot verification process. When the user needs to perform data verification, simply set the switch to the on state, and the system will automatically enter the non-stop verification mode and start to execute subsequent data snapshot acquisition, comparison and repair steps. In order to meet the needs of different users and application scenarios, this configuration switch can be designed in a variety of configuration methods, such as through configuration files, environment variables, management interfaces, etc. Users can choose the configuration method that best suits them according to their operating habits and system architecture characteristics.

[0014] In order to ensure the effectiveness of the configuration switch, the system needs to monitor its status in real time and give users timely feedback when the status changes. For example, when the switch is successfully turned on, the system can inform the user through logging, message notification, etc. to ensure that the user understands the current data verification status. It is worth emphasizing that the process of configuring a non-stop verification switch is transparent to the source business, and there is no need to suspend or interrupt the business operations of the source database. This means that even during the data verification process, the source database can continue to process business requests normally, ensuring business continuity and stability.

[0015] P20: receiving a user instruction, enabling the non-stop verification switch, activating the non-stop snapshot verification mode, and suspending the storage of target-side data according to the non-stop snapshot verification mode, but not suspending the synchronization of source-side data.

[0016] Further, such as Figure 2 As shown, step P20 of the embodiment of the present application also includes: P21: Collect historical business processing data for clustering and generate multiple business scenarios; P22: Based on the multiple business scenarios, generate multiple corresponding initial snapshot verification strategies, which include but are not limited to full verification or fine-grained verification; P23: Based on the multiple business scenarios and multiple initial snapshot verification strategies, integrate and construct multiple non-stop snapshot verification modes; P24: Extract target business scenario features based on the user instructions, and according to the target business scenario features, traverse the multiple non-stop snapshot verification modes, match and obtain the target non-stop snapshot verification mode and activate it.

[0017] It should be understood that once the user issues a command to enable non-stop verification through the system interface or command line tool, the system will immediately respond and activate the preset non-stop snapshot verification mode. This mode is activated by suspending data entry to the target end, but at the same time ensuring that data synchronization on the source end is not interrupted. This means that when performing data verification, the business data on the source end is still generated and transmitted normally, thus avoiding interference with business operations.

[0018] In order to more accurately match the data verification requirements in different business scenarios, historical business processing data is first collected. These data cover various business operations, data traffic patterns, and system load conditions. Subsequently, clustering algorithms are used to process these historical data, and data with similar characteristics are grouped into the same category, thereby generating multiple representative business scenarios, so that customized verification strategies can be designed for different business needs in the future.

[0019] For each business scenario, the system will formulate a corresponding initial snapshot verification strategy based on its characteristics. These strategies include but are not limited to full verification and fine-grained verification. Full verification is suitable for scenarios that require high integrity and consistency, involving a comprehensive comparison of the entire data set. Fine-grained verification is suitable for certain specific scenarios, and only compares specific data blocks or fields, thereby reducing the burden of data processing and improving efficiency. Each strategy aims to minimize the impact on business operations while ensuring data consistency.

[0020] Furthermore, based on multiple business scenarios and initial verification strategies, multiple non-stop snapshot verification modes are constructed through fusion optimization through intelligent algorithms. These modes integrate the business characteristics of different scenarios to ensure that no matter what business environment, there is a corresponding verification mode to match, thereby achieving flexible and targeted verification and repair operations. Each non-stop snapshot verification mode is an optimized solution to meet the data consistency requirements in different scenarios.

[0021] When the user issues an instruction to enable non-stop verification, the system will first extract the characteristic information of the current business scenario, that is, the target business scenario characteristics. Then, based on this characteristic information, the system will traverse multiple non-stop snapshot verification modes that have been constructed. Through the intelligent matching algorithm, the non-stop snapshot verification mode that best matches the current business scenario is quickly found and activated. After activation, the system will perform subsequent data verification and repair operations according to this mode to ensure data consistency and integrity without affecting the operation of the source business. In this way, the system can dynamically adapt to different business scenarios and provide more targeted and efficient verification solutions, while avoiding business interruption problems in traditional methods.

[0022] Furthermore, the embodiment of the present application further includes step P22a, and step P22a further includes: P22-1a: Based on the multiple business scenarios, extract multiple historical business processing data sets respectively; P22-2a: According to the multiple historical business processing data sets, extract positive sample data and negative sample data respectively to predict abnormal areas, and obtain multiple abnormal area prediction results; P22-3a: Through the multiple abnormal area prediction results, extract multiple key verification data sets, and add the multiple key verification data sets to the corresponding multiple snapshot verification strategies.

[0023] Optionally, in order to further improve the accuracy and effectiveness of non-stop snapshot verification, based on the multiple business scenarios generated above, extract the historical business processing data sets corresponding to these business scenarios. Each data set contains the historical processing records under the business scenario, including data synchronization behavior, processing time, business load and other information. The core of this step is to lay the foundation for the subsequent anomaly prediction and verification strategy generation by collecting and extracting these historical data.

[0024] Next, these historical data sets are further processed. According to the historical business processing data sets of each business scenario, positive sample data and negative sample data are extracted respectively. Positive sample data usually represents normal data processing flow or synchronization operation, while negative sample data identifies abnormal situations or potential risks that may occur during data synchronization. By classifying positive and negative samples and combining them with machine learning for training, a prediction model for abnormal areas can be constructed to predict abnormal areas that may occur during business processing, thereby obtaining multiple abnormal area prediction results. The anomaly detection model involved in this step can use a variety of algorithms, such as density-based anomaly detection (DBSCAN), isolation forest (IsolationForest) or autoencoder (Autoencoder). These algorithms can effectively identify abnormal patterns from complex data and provide strong support for the formulation of verification strategies.

[0025] After obtaining the prediction results of abnormal areas, the system will further extract key verification data sets from these areas. These data sets contain the areas where data inconsistencies or errors are most likely to occur, and are the focus of attention during the verification process. These key verification data sets are combined with the corresponding business scenarios and initial snapshot verification strategies to generate more sophisticated and personalized snapshot verification strategies. These strategies not only focus on the consistency of the overall data, but also place special emphasis on in-depth verification of potential abnormal areas, thereby improving the accuracy and efficiency of data verification, especially in business scenarios without downtime, which can effectively discover and repair data anomalies and ensure high system availability and data consistency.

[0026] P30: Get a source snapshot from the source and determine the source-side committed transaction in the current snapshot.

[0027] It should be understood that obtaining a data snapshot from the source database and accurately identifying the last committed transaction in the snapshot, i.e., the source committed transaction, is the basis for subsequent data verification and repair work, and is crucial to ensuring data consistency and reducing business interruptions.

[0028] First, we generate a snapshot of the current source database by calling the snapshot function of the database. A snapshot is a data view of the database at a certain moment, which captures all the committed data at the current moment. The generation of a snapshot does not affect the normal business operations of the source database because it is only a static data view. This step ensures that we can obtain an accurate and instantaneous source data state.

[0029] First, the acquisition of the source snapshot is completed through the snapshot mechanism provided by the database management system (DBMS). A snapshot is a record of the data state of the database at a certain point in time. It contains all committed data changes but does not contain any uncommitted transactions. In the present invention, snapshot technology is used to create a data copy of the source database at a specific point in time, which will be used for subsequent data comparison and verification.

[0030] Next, in order to determine the source-side committed transaction in the current snapshot, the transaction XID (XID) is used as a unique identifier. Each database transaction is assigned a globally unique XID when it is committed. This XID not only identifies the order in which the transaction is committed, but is also used to determine whether the transaction has been committed or is active. By executing a specific SQL query, you can obtain the XID of the last committed transaction in the current snapshot. By obtaining this XID, all committed transactions in the current snapshot are locked, and subsequent comparison operations are ensured to be based on this consistent transaction state.

[0031] In addition, the snapshot and transaction XID acquisition process of the source side will be synchronized to the target side, ensuring that the target side can accurately track the transaction status of the source side, so that when the snapshot of the target side is generated, the same data consistency basis as the source side is achieved. It provides technical support for subsequent snapshot comparison and data verification, ensuring that the source and target sides compare and repair data at the same transaction time point.

[0032] P40: Synchronize data into the database on the target side until the transaction is committed to the source side and a snapshot of the target side is generated.

[0033] Specifically, data synchronization is performed on the target end until the target end is synchronized to the same commit transaction point as the source end, and then a target end snapshot is generated. The key to this step is to ensure that the data on the source and target ends are consistent at the same transaction point, providing a reliable basis for subsequent comparison and data verification.

[0034] First, the target side continuously receives the data synchronization stream from the source side and writes data in the order of source side transactions. A key operation in this process is data storage, which is to write the incremental data from the source side into the database of the target side in the logical order of transactions. To ensure data consistency, the system will continuously track the source side's committed transaction XID to ensure that the data written on the target side matches the transaction order of the source side.

[0035] To achieve this goal, the target-side data synchronization software monitors and analyzes the transaction identifiers (XIDs) in the synchronization data stream in real time. When it detects that the XID in the synchronization data stream matches the XID in the source-side snapshot, the target-side synchronization service immediately stops storing new data changes and enters the snapshot generation phase. At this point, the target generates a target-side snapshot at the current moment. The target-side snapshot is a static view of the target-side database at this moment, capturing all transaction data that has been written and committed. The transaction status in the snapshot should be consistent with the committed transaction status in the source-side snapshot, that is, they share the same XID.

[0036] In addition, during the process of generating a snapshot on the target side, although the data storage operation is suspended, the synchronous data flow from the source side to the target side will continue to be transmitted, and these new transaction changes will be temporarily cached (such as encapsulated as a kufl file). After the snapshot generation is completed, the storage operation will be resumed to ensure the integrity and real-time nature of the data.

[0037] P50: In combination with the intelligent verification mechanism, data is read from the source snapshot and the target snapshot respectively for comparison, and a verification comparison result is generated.

[0038] Further, such as Figure 3 As shown, step P50 of the embodiment of the present application also includes: P51: Based on the source-side snapshot and the target-side snapshot, respectively extract the source-side transaction XID and the target-side transaction XID, perform a transaction sequence consistency comparison, and obtain a first comparison result; P52: For the source-side snapshot and the target-side snapshot, perform a data content comparison of key fields to obtain a second comparison result; P53: Obtain incremental data from the source and target ends, perform database log analysis, and obtain a third comparison result; P54: Fusion the first comparison result, the second comparison result, and the third comparison result to generate the verification comparison result.

[0039] It should be understood that through the intelligent verification mechanism, data is extracted from the snapshots of the source and target ends for comparison to generate the final verification and comparison results. Through multi-dimensional data comparison, it is ensured that the data of the source and target ends are completely consistent.

[0040] First, based on the source snapshot and the target snapshot, the source transaction XID and the target transaction XID are extracted respectively, and the two transaction XID sequences are compared for transaction sequence consistency. Transaction XID is a unique identifier for each database transaction, which is used to record the order in which data is submitted. By comparing the transaction XID sequences of the source and target ends, the consistency of the transaction submission order of the two databases is identified, ensuring that there are no missed or disordered transaction submissions, and generating the first comparison result for determining consistency at the transaction level.

[0041] Next, the data content of the key fields in the source snapshot and the target snapshot is compared. In this stage, the core data fields in the database, such as primary keys, foreign keys, timestamps, etc., are compared to ensure the consistency of key business data. This step can detect whether there is data loss, damage or inconsistency during the data synchronization process through accurate data content comparison. The comparison result of this step is the second comparison result, which can reflect the consistency verification at the data content level.

[0042] Subsequently, the incremental data of the source and target ends are obtained, and the database logs of both ends are analyzed. Through log analysis, it is possible to identify which data has changed during the synchronization process, track the details of these changes, find the differences in data changes between the source and target ends, especially the unsynchronized data caused by network delays, transmission errors, etc., and generate a third comparison result for detecting the synchronization consistency of incremental data, especially the accuracy verification in real-time data flow.

[0043] Finally, the first comparison result (transaction sequence comparison result), the second comparison result (key field content comparison result) and the third comparison result (incremental data log analysis result) are integrated to generate the final verification comparison result. By integrating multi-dimensional comparison results, the system can fully verify the data consistency between the source and target ends, covering not only the transaction sequence and data content, but also the synchronization of real-time incremental data, providing a comprehensive basis for subsequent data repair work.

[0044] Furthermore, step P52 of the embodiment of the present application also includes: P52-1: Collect historical verification data, the historical verification data contains basic field characteristics, and the field basic characteristics include the field name, field type, position of the field in the transaction, historical verification difference frequency and risk level of each field; P52-2: Based on the historical verification data, use a machine learning algorithm to train a field weighting model; P52-3: For the source-side snapshot and the target-side snapshot, use the field weighting model to assign a weight value to each field; P52-4: Sort the fields according to the weight values ​​to obtain a field comparison priority sequence; P52-5: With reference to the field comparison priority sequence, perform data content comparison of key fields, including field value comparison, string field comparison, numeric field comparison and date / time field comparison, obtain a set of difference fields, and record it as the second comparison result.

[0045] Optionally, in order to further improve the accuracy and efficiency of data content comparison of key fields, historical verification data and machine learning algorithms can be combined to optimize the priority and weight distribution of field comparison to improve the intelligence and accuracy of verification.

[0046] First, collect historical verification data. This historical data contains the basic characteristics of each field, including field name, field type (such as string, value, date, etc.), the position of the field in the transaction, the frequency of differences in the field during historical verification, and the risk level of the field (that is, the degree of risk reflected by a field in data inconsistency). By collecting this historical data, the system can build comprehensive feature information of the field in past verifications as the basis for subsequent machine learning model training.

[0047] Next, based on the collected historical verification data, a machine learning algorithm is used to train a field weighting model. The purpose of this model is to assign a reasonable weight value to each field based on the basic characteristics of the field and the historical verification performance (such as the frequency of differences and risk level). Using machine learning algorithms, through learning from a large amount of historical data, it is possible to find which fields are more likely to have verification differences, and automatically adjust the importance of the fields according to their risk levels, ensuring that more important fields are given priority during the comparison process.

[0048] Furthermore, using the trained field weighting model, a weight value is assigned to each field for the snapshots of the source and target ends. These weight values ​​represent the importance of the field in the verification process. Fields with higher weights are usually fields with high frequency of differences and high risk levels in historical verifications. These fields need to be processed first during the comparison process to minimize the risk of data inconsistency.

[0049] According to the weight value of each field, all fields are sorted to generate a field comparison priority sequence. This sequence determines the processing order of each field during the verification process, and the fields with higher weights will be compared first. This sorting method optimizes the comparison process, ensuring that the fields that are most prone to errors or have the greatest impact are compared first, thereby improving the efficiency and accuracy of the overall verification.

[0050] Finally, according to the generated field comparison priority sequence, the key fields are compared in turn. During the comparison process, a variety of comparison strategies can be used, including field value comparison (directly comparing whether the stored values ​​of the fields are the same), string field comparison (considering factors such as string similarity, capitalization, etc.), numerical field comparison (considering factors such as the precision and range of the numerical value), and date / time field comparison (considering factors such as the accuracy of the timestamp and time zone differences). Through a refined comparison strategy, the difference field set is accurately identified and recorded as the second comparison result. This result can provide a clear goal and basis for subsequent data repair work.

[0051] P60: If there are differences in the verification and comparison results, data repair is performed based on the difference data, and after the repair is completed, the target end data is restored to the warehouse.

[0052] Furthermore, step P60 of the embodiment of the present application also includes: P61: If there is a difference in the verification and comparison results, analyze the current warehousing status of the target end, select the optimal pause point, and suspend the database warehousing operation of the target end according to the optimal pause point; P62: Record the current sequence number of the target end transaction queue as the starting node of the repair operation, and generate a snapshot of the target end transaction queue; P63: Pack the difference data into change.kufl files, and the change.kufl files include but are not limited to the difference fields, difference transaction XIDs, and data table names between the source and target ends; P64: Generate an undo.kufl file at the same time, and the undo.kufl file includes the transactions and data operations that need to be rolled back on the target end; P65: According to the undo.kufl file, roll back the erroneous transactions on the target end, and cancel all existing erroneous data operations; P66: Insert the change.kufl file of the difference data into the target end transaction queue node after the rollback, complete the repair of the target end transaction queue, and restore the target end data warehousing with reference to the starting node.

[0053] It should be understood that when a difference is detected between the data verification and comparison results of the source and target ends, a data repair operation is performed based on the difference data, and the data storage operation of the target end is resumed after the repair is completed. Ensure that the data on the target end is consistent with the source end, and after the data is repaired, the system can seamlessly continue the synchronous storage operation.

[0054] First, analyze the current storage status of the target end, and select an optimal pause point that has the least impact on the synchronization timeliness according to the synchronization progress and transaction processing. This optimal pause point is a key position in the transaction queue, which can ensure that the storage operation on the target end during the pause will not affect the data repair process. By pausing the storage operation at the appropriate time, data conflicts and synchronization interruptions can be avoided, ensuring smooth repair. At this time, the database storage operation on the target end will be suspended.

[0055] At the pause point, the serial number of the current target transaction queue is recorded as the starting node for data repair, and a snapshot of the target transaction queue is generated. The snapshot saves the current transaction status and data operations on the target side, and provides a reference point for subsequent repair operations. This step ensures the traceability and controllability of the repair operation, and facilitates rollback and repair.

[0056] Next, based on the verification and comparison results, the difference data is encapsulated into a change.kufl file. This file records the difference information between the source and target ends in detail, including difference fields, difference transaction IDs, data table names, etc. By generating a change.kufl file, the system can accurately capture the details of data inconsistencies and provide a clear data source for subsequent repair operations.

[0057] At the same time, before repairing the difference data, the system needs to roll back the transactions executed by the target side due to the erroneous data and generate an undo.kufl file, which contains the transactions and data operations that need to be rolled back on the target side. The main function of this file is to record the erroneous transactions and data change operations that need to be undone in the target database, and by rolling back these operations, ensure that the repair process will not be affected by the erroneous data.

[0058] Furthermore, according to the content of the undo.kufl file, a rollback operation is performed to cancel all erroneous transactions and data operations performed on the target side due to data inconsistency. This operation is based on the transaction management mechanism to ensure data integrity and accuracy during rollback and avoid misoperation or data loss. By canceling incorrect transactions, the target side can be restored to a state consistent with the source side data.

[0059] Finally, after completing the rollback of the erroneous transaction, the system inserts the change.kufl file of the difference data into the target transaction queue, and the insertion position is immediately after the node corresponding to the undo.kufl file. This ensures that the difference data can be executed in the correct order and avoids the problem of concurrent writing. Subsequently, referring to the starting node of the repair operation, the database entry operation on the target side is restored. At this point, the data repair process is completed, and the data in the target database is consistent with the source. Accurate verification and efficient repair of target data are achieved without stopping the source business.

[0060] Furthermore, the embodiment of the present application further includes step P70, and step P70 further includes: P71: Introduce a multi-level cache mechanism to extract the repair process of high-frequency difference data and cache it into the memory to generate a common repair process library; P72: Extract the remaining low-frequency difference repair data and store it in the external memory.

[0061] In a possible embodiment of the present application, in order to further improve the efficiency and response speed of the data repair process, a multi-level cache mechanism can be introduced, which effectively optimizes the repair process of high-frequency and low-frequency data by classifying the frequency of difference data.

[0062] First, by analyzing historical repair records, high-frequency difference data is extracted. High-frequency difference data refers to data that appears repeatedly and needs to be repaired frequently during data synchronization and verification. For these high-frequency difference data, the corresponding repair process is cached in memory to form an efficient memory cache structure. This structure stores commonly used repair processes and difference processing logic in memory in advance, so that when encountering similar differences, the system can quickly call the cached repair process in memory, thereby greatly shortening the response time.

[0063] The main advantage of memory caching is its fast access speed. Therefore, memory caching of high-frequency repair processes can significantly improve the system's real-time repair capabilities. To this end, a common repair process library can also be generated, which contains typical repair methods and operation steps for high-frequency difference data. This process library can be directly called by the system in subsequent repair operations to avoid repeated calculations and process design.

[0064] Next, for the infrequently occurring low-frequency difference repair data, store it in external memory. The repair requirements for low-frequency data are relatively rare, so there is no need to occupy precious memory resources. By analyzing the historical occurrence frequency and repair process of these low-frequency data, the repair process of these data is stored in an external storage device (such as a disk or solid-state drive), and the repair operation is performed by reading the external memory when needed.

[0065] This multi-level cache mechanism can improve the efficiency of high-frequency data repair, reduce the overall system load, and optimize the allocation and use of memory and external storage resources. It makes the entire data repair process more intelligent and efficient, and ensures the consistency and reliability of data synchronization.

[0066] Furthermore, the embodiment of the present application further includes step P80, and step P80 further includes: P81: Introduce fine-grained strategies, formulate multiple data levels and corresponding verification strategies; P82: Analyze and obtain the target data level based on the user's actual business needs; P83: Based on the target data level, match the target verification strategy, and process the target data in a targeted manner according to the target verification strategy.

[0067] Optionally, in order to further optimize the accuracy and adaptability of data verification, fine-grained strategies can be introduced to formulate corresponding verification strategies for different data levels. This strategy processes data in a hierarchical manner to ensure that the system can more flexibly handle business data of different types and importance when performing data verification, thereby improving the overall verification efficiency and accuracy.

[0068] First, by introducing fine-grained strategies, multiple data levels and their corresponding verification strategies are formulated. These data levels are divided according to different business needs and the importance of data, and may include high-priority data, medium-priority data, and low-priority data. For example, core business data and sensitive user data can be classified as high-priority data, while historical records or insensitive data can be classified as low-priority data. For each data level, the system designs different verification strategies to adapt to different levels of data requirements. For high-priority data, the verification strategy may be more stringent and comprehensive to ensure the consistency and integrity of the data; for low-priority data, the verification strategy may be more simplified to reduce resource consumption.

[0069] Analyze and obtain the target data level according to the user's actual business needs. This process can be completed by analyzing the needs of the user's current business scenario, such as evaluating which data has a higher priority in the business scenario and which data is secondary. Through this analysis, the target data level that needs to be checked in the business process is identified, thus providing a basis for the subsequent selection of verification strategies.

[0070] Next, based on the target data level determined in the previous step, the corresponding target verification strategy is matched. For data of different levels, the most suitable verification strategy is automatically selected for processing. For example, for high-priority data, a more rigorous verification strategy is used to perform full comparison, refined field verification, and transaction consistency verification; while for low-priority data, fine-grained verification or simplified verification strategies are used to verify only key fields. This process ensures the pertinence of data verification, which not only ensures the consistency of high-importance data, but also avoids wasting too much computing resources on unnecessary data.

[0071] By introducing fine-grained strategies, the system can flexibly respond to the needs of different data levels when processing complex business scenarios, thereby achieving more intelligent and accurate verification operations. This method can dynamically adjust the verification strategy according to the importance of the data, effectively improving the overall performance of the system.

[0072] In summary, the embodiments of the present application have at least the following technical effects: This application configures the non-stop verification switch and enables the non-stop snapshot verification mode. Without stopping the source data synchronization, the source and target snapshots are obtained, and the data is compared in combination with the intelligent verification mechanism. If there are differences, repairs are performed based on the difference data to ensure data consistency and restore the target data into the warehouse. The technical effect of performing efficient data verification and repair without stopping the source business is achieved, improving data consistency and repair efficiency, and ensuring business continuity and availability.

[0073] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0074] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0075] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A snapshot verification and repair method for a source end without service downtime, characterized in that: The method comprises: Configure the non-stop verification switch; Receive a user instruction, enable the non-stop verification switch, activate the non-stop snapshot verification mode, and suspend the storage of target-side data according to the non-stop snapshot verification mode, but do not suspend the synchronization of source-side data; Obtain a source snapshot from the source and determine the source-side committed transaction in the current snapshot; Synchronize data into the database on the target end until the transaction is committed to the source end, and generate a snapshot of the target end; Combined with the intelligent verification mechanism, data is read from the source snapshot and the target snapshot respectively for comparison, and a verification comparison result is generated; If there is a difference in the verification and comparison results, data repair is performed based on the difference data, and after the repair is completed, the target end data is restored to the warehouse.

2. The snapshot verification and repair method for a source end without service downtime according to claim 1, characterized in that: Receiving a user instruction, enabling the non-stop verification switch, and activating the non-stop snapshot verification mode, including: Collect historical business processing data for clustering and generate multiple business scenarios; Based on the multiple business scenarios, generate corresponding multiple initial snapshot verification strategies, wherein the multiple initial snapshot verification strategies include but are not limited to full verification or fine-grained verification; According to the multiple business scenarios and multiple initial snapshot verification strategies, multiple non-stop snapshot verification modes are integrated and constructed; The target business scenario feature is extracted based on the user instruction, and according to the target business scenario feature, the multiple non-stop snapshot verification modes are traversed, and the target non-stop snapshot verification mode is matched and obtained and activated.

3. The snapshot verification and repair method for the source end without service downtime according to claim 2, characterized in that: Based on the multiple business scenarios, multiple corresponding snapshot verification strategies are generated, including: Based on the multiple business scenarios, extract multiple historical business processing data sets respectively; According to the multiple historical business processing data sets, respectively extract positive sample data and negative sample data to perform abnormal area prediction, and obtain multiple abnormal area prediction results; A plurality of key verification data sets are extracted through the plurality of abnormal area prediction results, and the plurality of key verification data sets are added into the corresponding plurality of snapshot verification strategies.

4. The snapshot verification and repair method for a source end without service downtime according to claim 1, characterized in that: In combination with the intelligent verification mechanism, data is read from the source snapshot and the target snapshot respectively for comparison, and a verification comparison result is generated, including: Based on the source-end snapshot and the target-end snapshot, respectively extract the source-end transaction XID and the target-end transaction XID, perform transaction sequence consistency comparison, and obtain a first comparison result; Comparing the data contents of key fields of the source snapshot and the target snapshot to obtain a second comparison result; Obtain incremental data from the source and target ends, perform database log analysis, and obtain the third comparison result; The first comparison result, the second comparison result and the third comparison result are integrated to generate the verification comparison result.

5. The snapshot verification and repair method for the source end without service downtime according to claim 4, characterized in that: For the source snapshot and the target snapshot, data content comparison of key fields is performed, including: Collect historical verification data, the historical verification data includes field basic characteristics, the field basic characteristics include field name, field type, field position in the transaction, historical verification difference frequency and risk level of each field; Based on the historical verification data, a field weighting model is trained using a machine learning algorithm; For the source-side snapshot and the target-side snapshot, assign a weight value to each field using the field weighting model; Sort the fields according to the weight values ​​to obtain a field comparison priority sequence; Referring to the field comparison priority sequence, data content comparison of key fields is performed, including field value comparison, string field comparison, numeric field comparison and date / time field comparison, to obtain a set of difference fields, which is recorded as the second comparison result.

6. The snapshot verification and repair method for a source end without service downtime according to claim 1, characterized in that: If there is a difference in the verification and comparison results, data repair is performed based on the difference data, and after the repair is completed, the target end data is restored to the warehouse, including: If there is a difference in the verification and comparison results, the current storage status of the target end is analyzed, an optimal pause point is selected, and the database storage operation of the target end is paused according to the optimal pause point; Record the current sequence number of the target transaction queue as the starting node of the repair operation, and generate a snapshot of the target transaction queue; Pack the difference data into a change.kufl file, which includes but is not limited to the difference fields between the source and target ends, the difference transaction XID, and the data table name; At the same time, an undo.kufl file is generated, wherein the undo.kufl file includes the transaction and data operations that need to be rolled back on the target side; According to the undo.kufl file, roll back the erroneous transaction on the target end and cancel all existing erroneous data operations; Insert the change.kufl file of the difference data into the target end transaction queue node after the rollback, complete the repair of the target end transaction queue, and restore the target end data storage with reference to the starting node.

7. The snapshot verification and repair method for a source end without service downtime according to claim 1, characterized in that: The method further comprises: Introducing a multi-level cache mechanism, extracting the repair process of high-frequency difference data and caching it into memory, generating a library of commonly used repair processes; The remaining low-frequency difference repair data is extracted and stored in external memory.

8. The snapshot verification and repair method for a source end without service downtime according to claim 1, characterized in that: The method further comprises: Introduce fine-grained strategies and formulate multiple data levels and corresponding verification strategies; Obtain target data level based on actual business needs of users; Based on the target data level, a target verification strategy is matched, and the target data is processed in a targeted manner according to the target verification strategy.

Images