Snapshot Verification and Repair Method for the Source End with Business Without Downtime

Abstract

The present invention discloses a snapshot verification and repair method for a source end without business downtime, which relates to the field of data processing technology. The method comprises: according to user instructions, activating a non-stop snapshot verification mode, suspending target end data storage, but not suspending source end data synchronization; obtaining a source end snapshot from the source end, and determining the source end commit transaction in the current snapshot; synchronizing data storage at the target end until synchronization reaches the source end commit transaction, generating a target end snapshot; comparing the source end snapshot and the target end snapshot, and if there is a difference, performing data repair based on the difference data, and resuming target end data storage after the repair is completed. The present invention solves the technical problem that existing data synchronization and verification methods require suspending source end business for data verification, which affects the continuity and availability of the business, and achieves the technical effect of performing efficient data verification and repair without stopping the source end business, ensuring the continuity and availability of the business.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a snapshot verification and repair method for a source end without service downtime. Background Art

[0002] In the field of data synchronization, ensuring data consistency between the source and target is a crucial task. However, existing data synchronization and verification methods often face numerous challenges in the complex scenarios of real-time data synchronization. Traditional methods typically ensure data consistency by periodically pausing source-side services and performing full data verification. However, this practice not only affects the continuity of source-side services but also causes synchronization service interruptions, impacting business continuity and availability. Summary of the Invention

[0003] The present application provides a snapshot verification and repair method for the source end without business downtime, which is used to solve the technical problem that the existing data synchronization and verification methods require suspending the source end business for data verification, affecting the continuity and availability of the business.

[0004] The present application provides a snapshot verification and repair method for a source end without business downtime, the method comprising: configuring a non-stop verification switch; receiving a user instruction, enabling the non-stop verification switch, activating a non-stop snapshot verification mode, suspending target end data storage according to the non-stop snapshot verification mode, but not suspending source end data synchronization; obtaining a source end snapshot from the source end, and determining the source end commit transaction in the current snapshot; synchronizing data storage at the target end until synchronization reaches the source end commit transaction, generating a target end snapshot; combining an intelligent verification mechanism, reading data from the source end snapshot and the target end snapshot respectively for comparison, and generating a verification comparison result; if there is a difference in the verification comparison result, performing data repair based on the difference data, and resuming target end data storage after the repair is completed.

[0005] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0006] The snapshot verification and repair method for the source end without business interruption provided by the present application relates to the field of data processing technology. By configuring the non-stop verification switch and enabling the non-stop snapshot verification mode, snapshots of the source end and the target end are obtained without stopping the source end data synchronization, and the data is compared in combination with the intelligent verification mechanism. If there are differences, repairs are performed based on the difference data. After ensuring data consistency, the target end data is restored to the warehouse. This solves the technical problem that the existing data synchronization and verification methods need to suspend the source end business for data verification, which affects the continuity and availability of the business. It realizes efficient data verification and repair without stopping the source end business, improves data consistency and repair efficiency, and ensures the continuity and availability of the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 A flowchart of a snapshot verification and repair method for a source end without service downtime provided in an embodiment of the present application;

[0009] Figure 2 A schematic diagram of a process for activating a non-stop snapshot verification mode in a snapshot verification and repair method for a source end without service downtime provided in an embodiment of the present application;

[0010] Figure 3 A flowchart of generating verification and comparison results in a snapshot verification and repair method for a source end without service downtime provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The present application provides a snapshot verification and repair method for the source end without business downtime, which is used to solve the technical problem that the existing data synchronization and verification methods require suspending the source end business for data verification, affecting the continuity and availability of the business.

[0012] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0013] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0014] Examples, such as Figure 1As shown, the present application provides a snapshot verification and repair method for a source end without service downtime, the method comprising:

[0015] P10: Configure the non-stop verification switch.

[0016] Specifically, configure a switch specifically for controlling the non-stop snapshot verification function. This step is fundamental to implementing this method and aims to provide a flexible configuration mechanism, allowing users to decide whether to enable non-stop snapshot verification mode based on actual needs. This configuration switch allows users to flexibly select the timing of data verification without affecting the normal operation of the source database, ensuring consistency and reliability during data synchronization.

[0017] This configuration switch is a logical control point that determines whether to initiate the non-stop snapshot verification process. When the user requires data verification, simply set this switch to on, and the system will automatically enter non-stop verification mode, beginning the subsequent steps of acquiring, comparing, and repairing data snapshots. To accommodate the needs of different users and application scenarios, this configuration switch can be configured in a variety of ways, such as through configuration files, environment variables, and the management interface. Users can choose the configuration method that best suits their operating habits and system architecture.

[0018] To ensure the effectiveness of the configured switch, the system needs to monitor its status in real time and provide users with timely feedback when the status changes. For example, when the switch is successfully turned on, the system can notify the user through logging, message notifications, and other means to ensure that the user is aware of the current data verification status. It is worth emphasizing that the process of configuring the non-stop verification switch is transparent to the source-side business and does not require pausing or interrupting business operations on the source database. This means that even during the data verification process, the source database can continue to process business requests normally, ensuring business continuity and stability.

[0019] P20: Receive user instructions, enable the non-stop verification switch, activate the non-stop snapshot verification mode, and suspend the storage of target-side data according to the non-stop snapshot verification mode, but do not suspend the synchronization of source-side data.

[0020] Further, such as Figure 2 As shown, step P20 in this embodiment of the application also includes:

[0021] P21: Collect historical business processing data for clustering processing to generate multiple business scenarios; P22: Based on the multiple business scenarios, generate corresponding multiple initial snapshot verification strategies, and the multiple initial snapshot verification strategies include but are not limited to full verification or fine-grained verification; P23: Based on the multiple business scenarios and multiple initial snapshot verification strategies, integrate and construct multiple non-stop snapshot verification modes; P24: Extract target business scenario features based on the user instructions, and traverse the multiple non-stop snapshot verification modes according to the target business scenario features, match and obtain the target non-stop snapshot verification mode and activate it.

[0022] It should be understood that once the user issues a command to enable non-stop verification through the system interface or command-line tool, the system will immediately respond and activate the preset non-stop snapshot verification mode. This mode is activated by suspending data ingestion on the target side while ensuring that data synchronization on the source side is not interrupted. This means that while data verification is being performed, business data on the source side continues to be generated and transmitted normally, thus avoiding disruption to business operations.

[0023] To more accurately match the data verification requirements of different business scenarios, we first collect historical business processing data. This data covers various business operations, data traffic patterns, and system load. We then use a clustering algorithm to process this historical data, grouping data with similar characteristics into the same category. This generates multiple representative business scenarios, enabling us to design customized verification strategies for different business needs.

[0024] For each business scenario, the system develops a corresponding initial snapshot verification strategy based on its characteristics. These strategies include, but are not limited to, full verification and fine-grained verification. Full verification is suitable for scenarios requiring high integrity and consistency, involving a comprehensive comparison of the entire dataset. Fine-grained verification, on the other hand, is suitable for specific scenarios and only compares specific data blocks or fields, thereby reducing the data processing burden and improving efficiency. Each strategy aims to ensure data consistency while minimizing the impact on business operations.

[0025] Furthermore, based on multiple business scenarios and initial verification strategies, intelligent algorithms are used for integrated optimization to construct multiple non-stop snapshot verification modes. These modes integrate the business characteristics of different scenarios, ensuring that no matter the business environment, there is a corresponding verification mode to match, thus enabling flexible and targeted verification and repair operations. Each non-stop snapshot verification mode is an optimized solution to meet the data consistency requirements of different scenarios.

[0026] When a user issues a command to enable non-stop verification, the system first extracts the characteristics of the current business scenario—the target business scenario characteristics. Then, based on this characteristic information, it traverses multiple established non-stop snapshot verification patterns. Using an intelligent matching algorithm, it quickly finds the non-stop snapshot verification pattern that best matches the current business scenario and activates it. Once activated, the system performs subsequent data verification and repair operations according to this pattern, ensuring data consistency and integrity without impacting source-side business operations. This allows the system to dynamically adapt to different business scenarios, providing more targeted and efficient verification solutions while avoiding the business interruptions associated with traditional methods.

[0027] Furthermore, the embodiment of the present application further includes step P22a, which further includes:

[0028] P22-1a: Based on the multiple business scenarios, extract multiple historical business processing data sets respectively; P22-2a: Based on the multiple historical business processing data sets, extract positive sample data and negative sample data respectively to perform abnormal area prediction, and obtain multiple abnormal area prediction results; P22-3a: Through the multiple abnormal area prediction results, extract multiple key verification data sets, and add the multiple key verification data sets to the corresponding multiple snapshot verification strategies.

[0029] Optionally, to further improve the accuracy and effectiveness of non-stop snapshot verification, extract historical business processing datasets corresponding to the multiple business scenarios generated above. Each dataset contains historical processing records for that business scenario, including information such as data synchronization behavior, processing time, and business load. The key to this step is that collecting and extracting this historical data lays the foundation for subsequent anomaly prediction and verification strategy development.

[0030] Next, these historical data sets are further processed. Based on the historical business processing data sets for each business scenario, positive and negative sample data are extracted. Positive sample data typically represents normal data processing processes or synchronization operations, while negative sample data identifies anomalies or potential risks that may arise during data synchronization. By classifying positive and negative samples and combining them with machine learning training, a prediction model for abnormal areas can be constructed to predict abnormal areas that may arise during business processing, thereby obtaining multiple abnormal area prediction results. The anomaly detection model involved in this step can use a variety of algorithms, such as density-based anomaly detection (DBSCAN), isolation forest, or autoencoder. These algorithms can effectively identify abnormal patterns from complex data, providing strong support for the formulation of verification strategies.

[0031] After obtaining predictions for abnormal areas, the system further extracts key validation datasets from these areas. These datasets contain areas most likely to experience data inconsistencies or errors and serve as the primary focus of the validation process. These key validation datasets are combined with the corresponding business scenarios and initial snapshot validation strategies to generate more refined and personalized snapshot validation strategies. These strategies not only focus on overall data consistency but also place special emphasis on in-depth validation of potential abnormal areas, thereby improving the accuracy and efficiency of data validation. This is particularly true in non-downtime business scenarios, enabling the effective detection and correction of data anomalies, ensuring high system availability and data consistency.

[0032] P30: Obtain a source snapshot from the source and determine the source-side committed transactions in the current snapshot.

[0033] It should be understood that obtaining a data snapshot from the source database and accurately identifying the last committed transaction in that snapshot, i.e., the source-side committed transaction, is the foundation for subsequent data verification and repair work and is crucial for ensuring data consistency and minimizing business interruptions.

[0034] First, we generate a snapshot of the current source database by calling the database's snapshot function. A snapshot is a view of the database at a specific moment in time, capturing all committed data at that moment. Generating a snapshot does not affect normal business operations on the source database, as it is only a static view of the data. This step ensures that we can obtain an accurate, instantaneous state of the source data.

[0035] First, the source-side snapshot is obtained using the snapshot mechanism provided by the database management system (DBMS). A snapshot is a record of the database's data state at a specific point in time. It includes all committed data changes but excludes any uncommitted transactions. In this invention, snapshot technology is used to create a copy of the source database's data at a specific point in time. This copy is used for subsequent data comparison and verification.

[0036] Next, to identify the source-side committed transaction in the current snapshot, the transaction XID (XID) is used as a unique identifier. Each database transaction is assigned a globally unique XID upon submission. This XID not only identifies the transaction submission order but also determines whether the transaction has been submitted or is active. By executing a specific SQL query, you can obtain the XID of the last committed transaction in the current snapshot. By obtaining this XID, all committed transactions in the current snapshot are locked, ensuring that subsequent comparison operations are based on this consistent transaction state.

[0037] Furthermore, the source's snapshot and transaction XID acquisition process are synchronized to the target, ensuring the target can accurately track the source's transaction status. This ensures that when the target's snapshot is generated, it achieves the same data consistency as the source. This provides technical support for subsequent snapshot comparisons and data validation, ensuring that both the source and target perform data comparisons and repairs at the same transaction time point.

[0038] P40: Synchronize data into the database on the target side until the transaction is committed to the source side and a target side snapshot is generated.

[0039] Specifically, data synchronization is performed on the target side until it reaches the same commit transaction point as the source side, generating a target-side snapshot. This step is crucial to ensuring that the data on the source and target sides remains consistent at the same transaction point, providing a reliable foundation for subsequent comparisons and data verification.

[0040] First, the target side continuously receives data synchronization streams from the source side and writes data in the order of source side transactions. A key operation in this process is data storage, which involves writing incremental data from the source side to the target side's database in the logical order of transactions. To ensure data consistency, the system continuously tracks the source side's committed transaction XIDs to ensure that data written to the target side matches the source side's transaction order.

[0041] To achieve this, the target-side data synchronization software monitors and analyzes transaction identifiers (XIDs) in the synchronization data stream in real time. When a match is detected between an XID in the synchronization data stream and an XID in the source-side snapshot, the target-side synchronization service immediately stops storing new data changes and instead begins the snapshot generation phase. At this point, the target generates a snapshot of the current moment. This snapshot is a static view of the target database at that moment, capturing all written and committed transaction data. The transaction state in this snapshot should be consistent with the committed transaction state in the source-side snapshot, meaning that they share the same XID.

[0042] In addition, during the process of generating a target-side snapshot, although the data storage operation is suspended, the synchronous data flow from the source to the target will continue to be transmitted. These new transaction changes will be temporarily cached (such as encapsulated as a kufl file). After the snapshot generation is completed, the storage operation will be resumed to ensure the integrity and real-time nature of the data.

[0043] P50: In combination with an intelligent verification mechanism, data is read from the source snapshot and the target snapshot respectively for comparison, and a verification comparison result is generated.

[0044] Further, such as Figure 3 As shown, step P50 in the embodiment of the present application also includes:

[0045] P51: Based on the source-side snapshot and the target-side snapshot, extract the source-side transaction XID and the target-side transaction XID respectively, perform a transaction sequence consistency comparison, and obtain a first comparison result; P52: For the source-side snapshot and the target-side snapshot, perform a data content comparison of key fields to obtain a second comparison result; P53: Obtain incremental data from the source and target ends, perform database log analysis, and obtain a third comparison result; P54: Fusion the first comparison result, the second comparison result, and the third comparison result to generate the verification comparison result.

[0046] It should be understood that the intelligent verification mechanism extracts data from the snapshots of the source and target ends for comparison to generate the final verification and comparison results. Through multi-dimensional data comparison, it ensures that the data on the source and target ends are completely consistent.

[0047] First, based on the source and target snapshots, the source and target transaction XIDs are extracted, respectively. These two transaction XID sequences are then compared for transaction sequence consistency. A transaction XID is a unique identifier for each database transaction, recording the order in which data was submitted. By comparing the source and target transaction XID sequences, the consistency of transaction submission order between the two databases is identified, ensuring that no transactions were missed or submitted out of order. This results in a first comparison, used to determine transaction-level consistency.

[0048] Next, a data content comparison is performed on key fields in the source and target snapshots. This phase focuses on comparing core database data fields, such as primary keys, foreign keys, and timestamps, to ensure the consistency of critical business data. This precise data content comparison can detect any data loss, corruption, or inconsistencies during the synchronization process. The comparison results from this step are the second comparison results, reflecting consistency verification at the data content level.

[0049] Subsequently, incremental data from both the source and target ends is obtained, and database logs from both ends are analyzed. Log analysis identifies which data has changed during the synchronization process, tracks the details of these changes, and identifies discrepancies between source and target data changes, particularly unsynchronized data due to network delays, transmission errors, and other factors. This generates a third comparison result, which is used to verify the synchronization consistency of incremental data, particularly in real-time data flow.

[0050] Finally, the first comparison results (transaction sequence comparison), the second comparison results (key field content comparison), and the third comparison results (incremental data log analysis results) are combined to generate the final verification and comparison results. By integrating multi-dimensional comparison results, the system can fully verify the data consistency between the source and target ends, covering not only the transaction sequence and data content, but also the synchronization of real-time incremental data, providing a comprehensive basis for subsequent data repair work.

[0051] Furthermore, step P52 of the embodiment of the present application further includes:

[0052] P52-1: Collect historical verification data, the historical verification data contains basic field features, the field basic features include the field name, field type, position of the field in the transaction, historical verification difference frequency and risk level of each field; P52-2: Based on the historical verification data, use a machine learning algorithm to train a field weighting model; P52-3: For the source-side snapshot and the target-side snapshot, use the field weighting model to assign a weight value to each field; P52-4: Sort the fields according to the weight values to obtain a field comparison priority sequence; P52-5: With reference to the field comparison priority sequence, perform data content comparison of key fields, including field value comparison, string field comparison, numeric field comparison and date / time field comparison, obtain a set of difference fields, and record it as the second comparison result.

[0053] Optionally, to further improve the accuracy and efficiency of data content comparison of key fields, historical verification data and machine learning algorithms can be combined to optimize the priority and weight distribution of field comparison to improve the intelligence and accuracy of verification.

[0054] First, historical validation data is collected. This historical data contains the basic characteristics of each field, including the field name, field type (such as string, number, date), the field's position in the transaction, the frequency of discrepancies in the field during historical validation, and the field's risk level (i.e., the degree of risk represented by the field in data inconsistencies). By collecting this historical data, the system can build comprehensive information about the field's past validation characteristics, which serves as the foundation for subsequent machine learning model training.

[0055] Next, a machine learning algorithm is used to train a field weighting model based on the collected historical validation data. This model aims to assign appropriate weights to each field based on its underlying characteristics and historical validation performance (such as discrepancy frequency and risk level). By learning from a large amount of historical data, the machine learning algorithm identifies fields that are more likely to experience validation discrepancies and automatically adjusts the importance of fields based on their risk levels, ensuring that more critical fields are prioritized during the comparison process.

[0056] Furthermore, using the trained field weighting model, a weight value is assigned to each field in both the source and target snapshots. These weights represent the importance of the field in the verification process. Fields with higher weights typically have a high frequency of discrepancies and high risk levels in historical verifications. These fields should be prioritized during the comparison process to minimize the risk of data inconsistency.

[0057] All fields are sorted based on their weight, generating a field comparison priority sequence. This sequence determines the order in which fields are processed during the verification process, with fields with higher weights being compared first. This sorting optimizes the comparison process, ensuring that fields most prone to errors or with the greatest impact are compared first, thereby improving overall verification efficiency and accuracy.

[0058] Finally, key fields are compared sequentially according to the generated field comparison priority sequence. During this comparison process, various strategies can be employed, including field value comparison (directly comparing the stored values of fields for identity), string field comparison (considering factors such as string similarity and case), numeric field comparison (considering factors such as the precision and range of the numeric values), and date / time field comparison (considering factors such as timestamp accuracy and time zone differences). Through refined comparison strategies, the set of discrepant fields is accurately identified and recorded as the second comparison result. This result provides a clear target and basis for subsequent data repair work.

[0059] P60: If there are differences in the verification and comparison results, data repair is performed based on the difference data, and after the repair is completed, the target end data is restored and stored in the warehouse.

[0060] Furthermore, step P60 of the embodiment of the present application further includes:

[0061] P61: If there is a difference in the verification and comparison results, analyze the current warehousing status of the target end, select the optimal pause point, and suspend the database warehousing operation of the target end according to the optimal pause point; P62: Record the current serial number of the target end transaction queue as the starting node of the repair operation, and generate a snapshot of the target end transaction queue; P63: Pack the difference data into a change.kufl file, and the change.kufl file includes but is not limited to the difference fields, difference transaction XID, and data table name between the source and target ends; P64: Generate an undo.kufl file at the same time, and the undo.kufl file includes the transactions and data operations that need to be rolled back on the target end; P65: According to the undo.kufl file, roll back the erroneous transaction on the target end and cancel all existing erroneous data operations; P66: Insert the change.kufl file of the difference data into the target end transaction queue node after the rollback, complete the repair of the target end transaction queue, and restore the target end data warehousing with reference to the starting node.

[0062] It should be understood that when a discrepancy is detected between the data verification and comparison results on the source and target sides, a data repair operation is performed based on the discrepant data, and data storage operations on the target side are resumed after the repair is complete. This ensures that the data on the target side remains consistent with the source side, and after the data repair, the system can seamlessly continue the synchronized storage operation.

[0063] First, the target side's current data ingestion status is analyzed. Based on the synchronization progress and transaction processing, an optimal pause point is selected to minimize the impact on synchronization timeliness. This optimal pause point is a key position in the transaction queue, ensuring that data ingestion operations on the target side during the pause do not affect the data repair process. By pausing data ingestion operations at the appropriate time, data conflicts and synchronization interruptions can be avoided, ensuring smooth repair. At this point, data ingestion operations on the target side are paused.

[0064] At the pause point, the sequence number of the current target-side transaction queue is recorded as the starting point for data repair, and a snapshot of the target-side transaction queue is generated. This snapshot saves the current transaction status and data operations on the target side, providing a reference point for subsequent repair operations. This step ensures the traceability and controllability of the repair operation, facilitating rollback and repair.

[0065] Next, based on the verification and comparison results, the difference data is encapsulated into a change.kufl file. This file records the difference information between the source and target ends in detail, including the difference fields, transaction IDs, and table names. By generating the change.kufl file, the system can accurately capture the details of data inconsistencies and provide a clear data source for subsequent repair operations.

[0066] Before repairing discrepancies, the system rolls back transactions executed on the target side due to erroneous data. This generates an undo.kufl file, which contains the transactions and data operations that need to be rolled back on the target side. This file records erroneous transactions and data changes that need to be undone in the target database. By rolling back these operations, the repair process is not affected by erroneous data.

[0067] Furthermore, based on the contents of the undo.kufl file, a rollback operation is performed, undoing all erroneous transactions and data operations performed on the target side due to data inconsistencies. This operation, based on the transaction management mechanism, ensures data integrity and accuracy during the rollback, preventing misoperations or data loss. By undoing incorrect transactions, the target side can be restored to a state consistent with the source side's data.

[0068] Finally, after rolling back the erroneous transaction, the system inserts the change.kufl file containing the difference data into the target-side transaction queue, immediately following the node corresponding to the undo.kufl file. This ensures that the difference data is executed in the correct order and avoids concurrent writes. Subsequently, the target-side database data entry operation is resumed, referring to the start node of the repair operation. This completes the data repair process, and the target-side database remains consistent with the source. This allows for accurate verification and efficient repair of target-side data without disrupting source operations.

[0069] Furthermore, the embodiment of the present application further includes step P70, which further includes:

[0070] P71: Introduce a multi-level cache mechanism to extract the repair process of high-frequency difference data and cache it in memory to generate a common repair process library; P72: Extract the remaining low-frequency difference repair data and store it in external memory.

[0071] In a possible embodiment of the present application, in order to further improve the efficiency and response speed of the data repair process, a multi-level cache mechanism can be introduced, which effectively optimizes the repair process of high-frequency and low-frequency data by classifying the frequency of the difference data.

[0072] First, by analyzing historical repair records, we extract high-frequency discrepancy data. High-frequency discrepancy data refers to data that recurs during data synchronization and verification and requires frequent repair. For this high-frequency discrepancy data, the corresponding repair processes are cached in memory, forming an efficient in-memory cache structure. This structure pre-stores commonly used repair processes and discrepancy processing logic in memory. When similar discrepancies are encountered, the system can quickly invoke the cached repair process in memory, significantly reducing response time.

[0073] The primary advantage of memory caching is its fast access speed. Therefore, caching high-frequency repair processes in memory can significantly improve the system's real-time repair capabilities. To this end, a library of commonly used repair processes can be generated, containing typical repair methods and steps for high-frequency discrepancy data. This library can be directly called upon by the system during subsequent repair operations, avoiding repeated calculations and process design.

[0074] Next, the infrequently occurring, low-frequency difference repair data is stored in external memory. This low-frequency data requires relatively little repair, so it doesn't need to occupy valuable memory resources. By analyzing the historical frequency of this low-frequency data and the repair process, the repair process for this data is stored in an external storage device (such as a disk or solid-state drive), and repair operations are performed by reading from external memory when needed.

[0075] This multi-level caching mechanism improves the efficiency of high-frequency data repair, reduces the overall system load, and optimizes the allocation and use of internal and external storage resources. This makes the entire data repair process more intelligent and efficient, ensuring the consistency and reliability of data synchronization.

[0076] Furthermore, the embodiment of the present application further includes step P80, which further includes:

[0077] P81: Introduce fine-grained strategies, formulate multiple data levels and corresponding verification strategies; P82: Analyze and obtain the target data level based on the user's actual business needs; P83: Based on the target data level, match the target verification strategy, and process the target data in a targeted manner according to the target verification strategy.

[0078] Optionally, to further optimize the accuracy and adaptability of data validation, fine-grained policies can be introduced to develop corresponding validation strategies for different data levels. This policy, by classifying data, ensures that the system can more flexibly handle business data of different types and importance during data validation, thereby improving overall validation efficiency and accuracy.

[0079] First, by introducing fine-grained strategies, multiple data levels and their corresponding verification strategies are formulated. These data levels are divided according to different business needs and the importance of the data, and may include high-priority data, medium-priority data, and low-priority data. For example, core business data and sensitive user data can be classified as high-priority data, while historical records or non-sensitive data can be classified as low-priority data. For each data level, the system designs different verification strategies to adapt to different levels of data requirements. For high-priority data, the verification strategy may be more stringent and comprehensive to ensure data consistency and integrity; for low-priority data, the verification strategy may be more simplified to reduce resource consumption.

[0080] Analyze and obtain target data levels based on the user's actual business needs. This process can be accomplished by analyzing the needs of the user's current business scenario, such as assessing which data has higher priority in the business scenario and which data is less important. Through this analysis, the target data levels that require key verification in the business process are identified, providing a basis for subsequent verification strategy selection.

[0081] Next, based on the target data level determined in the previous step, the corresponding target verification strategy is matched. The most appropriate verification strategy is automatically selected for processing data of different levels. For example, for high-priority data, a more rigorous verification strategy is used, performing full comparison, refined field verification, and transaction consistency verification. For low-priority data, a fine-grained verification or simplified verification strategy is employed, verifying only key fields. This process ensures targeted data verification, ensuring the consistency of high-importance data while avoiding wasting excessive computing resources on unnecessary data.

[0082] By introducing fine-grained policies, the system can flexibly respond to the needs of different data levels when handling complex business scenarios, thereby achieving more intelligent and accurate verification operations. This approach can dynamically adjust verification policies based on data importance, effectively improving the overall performance of the system.

[0083] In summary, the embodiments of the present application have at least the following technical effects:

[0084] This application configures a non-stop verification switch and enables non-stop snapshot verification mode. Without stopping source data synchronization, it obtains snapshots of the source and target sides, and uses an intelligent verification mechanism to compare data. If discrepancies exist, repairs are performed based on the discrepant data, ensuring data consistency before restoring the target side data back into the database. This achieves the technical effect of performing efficient data verification and repair without stopping source business, improving data consistency and repair efficiency, and ensuring business continuity and availability.

[0085] It should be noted that the order in which the embodiments of the present application are presented is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0086] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

[0087] This specification and drawings are merely illustrative of the present application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Obviously, those skilled in the art may make various modifications and variations to this application without departing from the scope of this application. Thus, this application is intended to include such modifications and variations as fall within the scope of this application and its equivalents.

Claims

1. A snapshot verification and repair method for a source end without service downtime, characterized in that: The method comprises: Configure the non-stop verification switch; Receive a user instruction, enable the non-stop verification switch, activate the non-stop snapshot verification mode, and suspend the storage of target-side data according to the non-stop snapshot verification mode, but do not suspend the synchronization of source-side data; Obtain a source snapshot from the source and determine the source-side committed transactions in the current snapshot; Synchronize data into the database on the target side until the transaction is committed to the source side and a snapshot of the target side is generated; Combined with an intelligent verification mechanism, data is read from the source snapshot and the target snapshot respectively for comparison, and a verification comparison result is generated; If there is a difference in the verification and comparison results, data repair is performed based on the difference data, and after the repair is completed, the target end data is restored and stored in the warehouse; In combination with an intelligent verification mechanism, data is read from the source snapshot and the target snapshot respectively for comparison, and a verification comparison result is generated, including: Based on the source-side snapshot and the target-side snapshot, respectively extracting the source-side transaction XID and the target-side transaction XID, performing a transaction sequence consistency comparison, and obtaining a first comparison result; Comparing the data contents of key fields of the source snapshot and the target snapshot to obtain a second comparison result; Obtain incremental data from the source and target ends, perform database log analysis, and obtain the third comparison results; fusing the first comparison result, the second comparison result, and the third comparison result to generate the verification comparison result; Comparing the data content of key fields of the source snapshot and the target snapshot includes: Collect historical verification data, which includes basic field characteristics, including the field name, field type, position of the field in the transaction, historical verification difference frequency, and risk level of each field; Based on the historical verification data, a field weighting model is trained using a machine learning algorithm; For the source-side snapshot and the target-side snapshot, assign a weight value to each field using the field weighting model; Sort the fields according to the weight values to obtain a field comparison priority sequence; Referring to the field comparison priority sequence, data content comparison of key fields is performed, including field value comparison, string field comparison, numerical field comparison and date / time field comparison, to obtain a set of difference fields, which is recorded as the second comparison result.

2. The snapshot verification and repair method for a source terminal without service downtime according to claim 1, characterized in that: Receiving a user instruction, enabling the non-stop verification switch, and activating the non-stop snapshot verification mode, including: Collect historical business processing data for clustering and generate multiple business scenarios; Based on the multiple business scenarios, generating corresponding multiple initial snapshot verification strategies, the multiple initial snapshot verification strategies including but not limited to full verification or fine-grained verification; Based on the multiple business scenarios and multiple initial snapshot verification strategies, multiple non-stop snapshot verification modes are integrated and constructed; The target business scenario feature is extracted based on the user instruction, and according to the target business scenario feature, the multiple non-stop snapshot verification modes are traversed, and the target non-stop snapshot verification mode is matched and activated.

3. The snapshot verification and repair method for a source end without service downtime according to claim 2, characterized in that: Based on the multiple business scenarios, multiple corresponding snapshot verification strategies are generated, including: Based on the multiple business scenarios, extract multiple historical business processing data sets respectively; Extracting positive sample data and negative sample data from the plurality of historical business processing data sets to perform abnormal area prediction, thereby obtaining a plurality of abnormal area prediction results; A plurality of key verification data sets are extracted through the plurality of abnormal area prediction results, and the plurality of key verification data sets are added into a corresponding plurality of snapshot verification strategies.

4. The snapshot verification and repair method for a source end without service downtime according to claim 1, characterized in that: If there is a difference in the verification and comparison results, data repair is performed based on the difference data, and after the repair is completed, the target end data is restored to the database, including: If there is a difference in the verification and comparison results, the current storage status of the target end is analyzed, an optimal pause point is selected, and the database storage operation of the target end is paused according to the optimal pause point; Record the current sequence number of the target transaction queue as the starting point of the repair operation, and generate a snapshot of the target transaction queue; The difference data is packaged into a change.kufl file, which includes but is not limited to the difference fields between the source and target ends, the difference transaction XID, and the data table name; At the same time, an undo.kufl file is generated, which includes the transactions and data operations that need to be rolled back on the target side; According to the undo.kufl file, roll back the erroneous transaction on the target side and cancel all existing erroneous data operations; Insert the change.kufl file of the differential data into the target-side transaction queue node after the rollback, complete the repair of the target-side transaction queue, and restore the target-side data storage with reference to the starting node.

5. The snapshot verification and repair method for a source terminal without service downtime according to claim 1, characterized in that: The method further comprises: A multi-level cache mechanism is introduced to extract the repair process of high-frequency difference data and cache it in memory to generate a common repair process library; Extract the remaining low-frequency difference repair data and store it in external memory.

6. The snapshot verification and repair method for a source end without service downtime according to claim 1, characterized in that: The method further comprises: Introduce fine-grained strategies and formulate multiple data levels and corresponding verification strategies; Obtain target data levels based on actual business needs of users; Based on the target data level, a target verification strategy is matched, and the target data is processed in a targeted manner according to the target verification strategy.

Images