A method and related apparatus for handling abnormal data during data synchronization.

By identifying and handling dependency missing anomalies during heterogeneous database synchronization, and utilizing hierarchical fault judgment logic and proactive detection mechanisms, data writing is automatically restored. This solves the problem of synchronization task blocking caused by missing dependency objects in the target database, and improves the real-time performance and efficiency of data synchronization.

CN123115270APending Publication Date: 2026-11-17FAN RUAN SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611058733.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In heterogeneous database synchronization or data replication scenarios, data write failures caused by missing dependent objects in the target database cannot be automatically recovered, resulting in a backlog of synchronization tasks, link blockage, reduced synchronization real-time performance, and the consumption of a large amount of queue storage and system computing resources.

Method used

By acquiring write failure information of data change events, the hierarchical fault judgment logic is used to identify abnormal data types, especially dependency missing exception types. The data change events are suspended and added to the dependency waiting queue, and the status of the target object in the target database is actively probed. Once the target object meets the recovery conditions, the write is automatically retried.

Benefits of technology

It enables automatic resumption of data change event writing after the target object is completed, avoiding the backlog of synchronization tasks and link blockage, improving the real-time performance and throughput of data synchronization, and reducing the unnecessary occupation of queue resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115270A_ABST
    Figure CN123115270A_ABST
Patent Text Reader

Abstract

This application discloses a method and related apparatus for handling abnormal data during data synchronization, relating to the field of database and data synchronization technology. The method includes: collecting corresponding write failure information after a data change event fails to write to the target database; parsing the failure information based on hierarchical fault judgment logic to identify abnormal data types. For dependency missing exception types, the corresponding data change event is suspended and stored in a dependency waiting queue, and the status of the target object dependent on the data change event in the target database is actively probed; after the target object meets preset recovery conditions, the data change event is automatically retried to be written to the target database. This application eliminates the need to lock data change events for extended periods; synchronization of suspended data change events is automatically restored once the target object is complete, avoiding synchronization task backlog and link blockage, effectively improving data synchronization real-time performance, reducing unnecessary queue resource occupation, and optimizing the overall throughput of the synchronization link.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of database and data synchronization technology, and in particular to a method and related apparatus for handling abnormal data during data synchronization. Background Technology

[0002] In heterogeneous database synchronization or data replication scenarios, change logs generated by the source database need to be parsed and written to the target database. When the target database fails to write data due to factors such as missing dependencies, most existing solutions uniformly categorize the failed data change events into an exception queue for centralized shelving.

[0003] In the above processing method, even if the missing dependent objects have been created in the target database, the data change event cannot be automatically resumed to write, which can easily cause the accumulation of synchronization tasks, link blockage, reduced synchronization real-time performance, and the occupation of a large amount of queue storage and system computing resources. Summary of the Invention

[0004] In view of the above problems, this application provides a method and related apparatus for handling abnormal data during data synchronization, so as to automatically resume the operation of writing data change events to the target database after the missing dependent objects have been fully created in the target database. The specific solution is as follows:

[0005] The first aspect of this application provides a method for handling abnormal data during data synchronization, including:

[0006] In response to a failure to write a data change event to the target database, obtain the write failure information corresponding to the data change event;

[0007] Based on the preset hierarchical fault determination logic and the write failure information, the abnormal data type corresponding to the data change event is determined, and the abnormal data type includes at least the dependency missing exception type.

[0008] If the abnormal data type is the dependency missing exception type, the data change event is suspended and added to the dependency waiting queue. The dependency missing exception type indicates that the target object on which the data change event depends does not exist in the target database.

[0009] Detect the state of the target object in the target database;

[0010] If the state of the target object is detected to meet the preset recovery conditions, the data change event is written to the target database again.

[0011] In one possible implementation, the write failure information includes a target error attribute and an error message text; the target error attribute represents a structured failure reason, and the error message text represents an unstructured failure reason;

[0012] The method for determining the abnormal data type corresponding to the data change event based on the preset hierarchical fault determination logic and the write failure information includes:

[0013] From the preset mapping relationship between error attributes and abnormal data types, the abnormal data type corresponding to the target error attribute is searched to obtain the matching result;

[0014] If the matching result indicates that no abnormal data type was found, the abnormal data type is determined based on the semantic features of the error message text.

[0015] In one possible implementation, the abnormal data type is represented by an error code, and the step of searching for the abnormal data type corresponding to the target error attribute from a preset mapping relationship between error attributes and abnormal data types to obtain a matching result includes:

[0016] The error code corresponding to the target error attribute is searched from the preset mapping relationship between error attributes and error codes to obtain the matching result.

[0017] In one possible implementation, determining the exception data type based on the semantic features of the error message text includes:

[0018] The error message text is standardized to obtain a standardized error text.

[0019] The system detects whether the standardized error text matches target keywords in multiple preset keyword rule bases; different preset keyword rule bases correspond to different abnormal data types.

[0020] The abnormal data type corresponding to the preset keyword rule library to which the target keyword belongs is determined as the abnormal data type corresponding to the data change event.

[0021] One possible implementation also includes:

[0022] If the suspension time of the data change event in the dependency waiting queue exceeds the preset maximum waiting time, and the status of the target object still indicates that the target object does not exist in the target database, the abnormal data type of the data change event is updated to a permanent abnormal type.

[0023] Extract the core metadata of the data change event;

[0024] The core metadata is persistently stored in an exception repository to isolate the data change events from the main synchronization process.

[0025] One possible implementation also includes:

[0026] If the abnormal data type is a temporary abnormal type, the data change event is moved to the delayed retry queue;

[0027] If the number of consecutive failed retries for the data change event exceeds a preset threshold, the abnormal data type is updated to a permanent abnormal type.

[0028] If the abnormal data type is a permanent abnormal type, extract the core metadata of the data change event and persist the core metadata to the abnormal repository.

[0029] A second aspect of this application provides an apparatus for handling abnormal data during data synchronization, comprising:

[0030] The information acquisition module is used to acquire the write failure information corresponding to the data change event in response to the failure of writing to the target database in response to the data change event.

[0031] An anomaly classification module is used to determine the abnormal data type corresponding to the data change event based on a preset hierarchical fault judgment logic and the write failure information. The abnormal data type includes at least the dependency missing anomaly type.

[0032] The storage module is used to suspend the data change event and add it to the dependency waiting queue if the abnormal data type is the dependency missing exception type. The dependency missing exception type indicates that the target object on which the data change event depends does not exist in the target database.

[0033] A status detection module is used to detect the status of the target object in the target database;

[0034] The retry write module is used to write the data change event to the target database again if the state of the target object is detected to meet the preset recovery conditions.

[0035] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the abnormal data processing method in the data synchronization process of the first aspect or any implementation thereof.

[0036] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0037] The memory is used to store computer programs;

[0038] The processor is used to execute the computer program to enable the electronic device to implement the abnormal data processing method in the data synchronization process of the first aspect or any implementation thereof.

[0039] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to handle abnormal data during the data synchronization process of the first aspect or any implementation thereof.

[0040] By employing the above technical solution, this application provides a method for handling abnormal data during data synchronization. After a data change event fails to write to the target database, corresponding write failure information is collected. The failure information is analyzed using a hierarchical fault judgment logic to identify abnormal data types, overcoming the shortcomings of existing technologies that cannot distinguish the causes of abnormal write failures. For dependency missing exceptions, this application suspends the corresponding data change event and stores it in a dependency waiting queue, actively probing the status of the target objects that the data change event depends on in the target database. Once the target object meets the preset recovery conditions, the data change event is automatically retried and written to the target database. This application eliminates the need to lock data change events for extended periods; once the target object is complete, the suspended data change event is automatically restored, avoiding synchronization task backlog and link blockage. This effectively improves the real-time performance of data synchronization, reduces unnecessary queue resource consumption, and optimizes the overall throughput of the synchronization link. Attached Figure Description

[0041] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0042] Figure 1 This application provides an architectural diagram of a real-time synchronization system for heterogeneous databases.

[0043] Figure 2 A flowchart illustrating an abnormal data processing method during data synchronization provided in this application embodiment;

[0044] Figure 3 A schematic diagram of an abnormal data processing device during data synchronization provided in this application embodiment;

[0045] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0046] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0047] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0048] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0050] This application can be applied to the field of database and data synchronization technology, especially in scenarios such as real-time replication of heterogeneous databases, incremental extraction of data warehouses, and cross-cloud data migration. The following section will use real-time synchronization of heterogeneous databases as an example to introduce several application scenarios implemented in products.

[0051] During data synchronization from the source database to the target database, write failures may occur due to network fluctuations, constraint conflicts, or differences in synchronization timing. To address the cascading blocking problem caused by traditional pessimistic locking mechanisms, this embodiment provides a method for handling abnormal data during data synchronization.

[0052] This application may be applied, but is not limited to, to applications with database change log capture and real-time synchronization capabilities (such as real-time data synchronization middleware cluster 500 products, such as Canal, Debezium, etc.) or cloud services for data synchronization provided by cloud-side servers (such as cloud database data transmission service DTS, cross-cloud data migration service, etc.).

[0053] The abnormal data processing method and system provided in the data synchronization process of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0054] See Figure 1 , Figure 1 A schematic diagram of the architecture of a heterogeneous database real-time synchronization system is shown. The system may include a source database cluster 100, a target database cluster 200, a database management system 300, a CDC (Change Data Capture) log collection layer 400, a real-time data synchronization middleware cluster 500, and a persistent storage unit 600.

[0055] The source database cluster 100 includes multiple source databases, whose database types include, but are not limited to, MySQL, Oracle, PostgreSQL, and SQL Server (Structured Query Language Server). The source databases are used to host the data storage and modification operations of the business systems and are the data source for the entire real-time synchronization system.

[0056] For example, the source database includes a database management system kernel, a transaction manager, a log generation module, a log buffer, a disk, and a timestamp or sequence number generator.

[0057] The database management system kernel is used to receive data operation requests from business systems, i.e., transactions, such as INSERT requests, UPDATE requests, and DELETE requests.

[0058] The transaction manager executes data change events on a transaction-by-transaction basis and generates log entries based on the transactions. The log entries are used to record the content of the data changes, the time of the changes, and the execution order. The transaction manager is also responsible for committing or rolling back transactions.

[0059] The log generation module is used to automatically generate log entries during the execution of transactions.

[0060] The log buffer is used to temporarily store log entries that have been generated but not yet stored on disk.

[0061] The timestamp or sequence number generator is used to generate an SCN (System Change Number) and a timestamp for each log entry.

[0062] The disk is used to persistently store log entries from the log buffer, along with the corresponding SCN and timestamp.

[0063] The CDC log collection layer 400 is deployed on the source database or a standalone collection server, connecting to the source database and the real-time data synchronization middleware cluster 500. The CDC log collection layer monitors persistent log files on the disk in real time, capturing newly added log entries. It then parses the captured log entries to extract core metadata, which is then pushed to the real-time data synchronization middleware cluster 500 in real time.

[0064] A log file is a carrier of log entries; a log file can carry one or more log entries.

[0065] For example, the core metadata of a log entry includes, but is not limited to:

[0066] The instance identifier of the source database is a character or numeric identifier used to uniquely distinguish different source database instances.

[0067] The filename of the log file to which the log entry belongs is a unique identifier for the log file storing the log entry.

[0068] The transaction identifier of the transaction to which the log entry belongs.

[0069] The execution order of multiple data operation statements carried by a log entry refers to the order in which the multiple data operation statements (INSERT, UPDATE, DELETE, etc.) carried by the log entry are executed during the transaction process.

[0070] The table name that the log entry refers to refers to the table name of the database table to which the change event recorded by the log entry is located in the source database.

[0071] The primary key identifier in a database table refers to the field or combination of fields in the source database table used to uniquely identify each row of data records.

[0072] The pre-change image is a full-field snapshot of the target data row in the corresponding data table of the source database before the data change is executed. It includes the original values ​​of all valid business fields and non-system hidden fields, and is used to record the state before the data change.

[0073] The post-change mirror is a snapshot of the complete field values ​​of the data row corresponding to the log entry in the source database table after a data change event. It includes the changed values ​​of all valid business fields and non-system hidden fields in that row, with no fields missing. Its purpose is to record the final state of the data after the change, and in conjunction with the pre-change mirror, to clearly identify the specific content of the data change.

[0074] Log offset refers to the physical offset position of a log entry within its respective log file, measured in bytes, and is recorded by the log generation module of the source database. This field is used to locate the storage location of log entries, providing a physical location basis for log collection, point-based resume, missing data detection, breakpoint recovery, and log tracing, ensuring that log entries are accurately captured and processed in sequence.

[0075] The real-time data synchronization middleware cluster 500 is deployed on an independent application server, serving as the core hub of the entire heterogeneous database real-time synchronization system, receiving log entries pushed by the CDC log collection layer.

[0076] The real-time data synchronization middleware cluster 500 handles all core logic, including log entry parsing, transaction management, idempotency verification, data writing, and exception handling. Specifically, after receiving the core metadata pushed by the CDC log collection layer 400, the real-time data synchronization middleware cluster 500 generates a first identifier based on the source attribute and a second identifier based on the content attribute. It then executes a two-layer verification and three-branch processing logic and converts verified data change events into standardized data change commands, which are then sent to the target database cluster 200.

[0077] The persistent storage unit 600 is deployed on a separate storage node and is used to store the pre-defined correspondence between the first identifier and the second identifier, etc.

[0078] The persistent storage unit 600 can be implemented in the form of key-value database, relational database table or memory cache, providing efficient identification query and persistence services for the real-time data synchronization middleware cluster 500.

[0079] The target database cluster 200 includes multiple target databases.

[0080] The target database includes, but is not limited to: database management system kernel, transaction execution module, data storage module, log recording module, disk, constraint verification module, and time recording module.

[0081] The database management system kernel is used to receive standardized data change events issued by the real-time data synchronization middleware cluster 500. The data change events are generated by the real-time data synchronization middleware cluster 500 after parsing, verifying and processing the log entries in the source database, and correspond one-to-one with the log entries in the source database.

[0082] The transaction execution module is used to execute INSERT, UPDATE, and DELETE operations corresponding to data change events. After the synchronized data is temporarily stored by the data storage module, it is persistently stored to the local disk.

[0083] The logging module is used to automatically generate target database synchronization log entries during the execution of standardized data change commands. It records information such as the source of the data change (corresponding to the SCN of the log entry in the source database), operation type, execution result, and execution time, which facilitates synchronization traceability and anomaly investigation.

[0084] The constraint verification module is used to verify the legality of the data to be written corresponding to the data change event, including primary key uniqueness, foreign key dependency, data format standardization, etc. If the verification fails, an abnormal status is reported to the real-time data synchronization middleware cluster 500.

[0085] Through the above system architecture, this application realizes real-time data synchronization from the source database to the target database. By using the two-layer verification and three-branch processing mechanism of the real-time data synchronization middleware cluster 500, it effectively solves data consistency problems in scenarios such as retry writing caused by network timeout, overwrite errors caused by network jitter, and repeated reading of logs by the CDC component.

[0086] Reference Figure 2 , Figure 2 A flowchart illustrating an abnormal data handling method during data synchronization provided in this application embodiment is shown below. Figure 2 As shown in the embodiment of this application, an abnormal data processing method during data synchronization may include steps S201 to S205, which are described in detail below.

[0087] Step S201: In response to the failure of writing to the target database due to a data change event, obtain the write failure information corresponding to the data change event.

[0088] Data change events refer to data operation instructions generated by the source database that need to be synchronized to the target database. These instructions can take the form of INSERT, UPDATE, or DELETE statements in Structured Query Language (SQL), or standardized change objects parsed from the binary log.

[0089] Write failure information refers to the exception feedback data returned by the database driver or real-time data synchronization middleware cluster 500 when the target database fails to execute the aforementioned data change events. For example, in a Java database connection environment, write failure information is usually represented by an SQLException object, which contains an error code, an SQL status code, and a detailed error message.

[0090] For example, write failure information can be intercepted in real time through an exception handling mechanism. It should be understood that the method of obtaining write failure information is not limited to exception handling; it can also be obtained by querying the database audit logs or the error callback interface of the real-time data synchronization middleware cluster, as long as it accurately reflects the reason for the write failure.

[0091] Step S202: Based on the preset hierarchical fault determination logic and the write failure information, determine the abnormal data type corresponding to the data change event, wherein the abnormal data type includes at least the dependency missing exception type.

[0092] Layered fault determination logic consists of pre-defined rules for identifying anomaly types. It can be a progressive determination logic that combines precise matching and semantic analysis.

[0093] Exception data types are abstract classifications of reasons for write failures, including but not limited to: temporary failures, permanent failures, and dependency missing exception types.

[0094] Temporary failures refer to write failures caused by issues such as momentary database resource shortages, brief network fluctuations, or temporary interruptions in the target database connection. Understandably, if the abnormal data type is a temporary failure, it means that the data change event can be successfully written to the target database again after a certain number of retries.

[0095] Permanent failures refer to write failures caused by inherent errors in the data or database tables, such as mismatched field formats, duplicate primary keys, or inconsistent table structure definitions. It's understandable that if the abnormal data type is a permanent failure, it means the data change event cannot be repaired through retries and must be archived and isolated.

[0096] The Dependency Missing Exception type specifically refers to write failures caused by the target database lacking target objects referenced by data change events (such as parent table records, foreign key reference rows, etc.). For example, when synchronizing payment records in a child table, if the corresponding order records in the parent table have not yet arrived, the target database will throw a foreign key constraint violation exception, which is then classified as a Dependency Missing Exception type.

[0097] This application fundamentally distinguishes between timing issues at the data logic level and fault issues at the environmental resource level by introducing a dependency missing exception type, thus providing a premise for subsequent differentiated processing.

[0098] Step S203: If the abnormal data type is the dependency missing exception type, suspend the data change event and add it to the dependency waiting queue. The dependency missing exception type indicates that the target object on which the data change event depends does not exist in the target database.

[0099] Suspending refers to temporarily removing data change events from the main synchronization process and storing them in a memory queue or persistent storage area so that they no longer occupy main thread resources, but it is not considered a permanent failure.

[0100] A dependency wait queue is a buffer area specifically used to temporarily store data change events that are identified as dependency missing exceptions. It can be a concurrent safe queue in memory or a persistent message queue partition. Its core function is to isolate abnormal traffic and prevent blocking the main thread.

[0101] Step S204: Detect the state of the target object in the target database.

[0102] Probing refers to the system spontaneously, periodically, or event-triggeredly sending query requests to the target database to verify whether a missing target object already exists.

[0103] For example, for the data change event "Payment Record Synchronization Event" that fails to write due to the non-existence of the target object "Order ID", the system will periodically execute a query statement like "SELECT COUNT(*) FROM orders WHERE id = ?" to check whether the "Order ID" has been entered into the database.

[0104] The fundamental difference between proactive probing and passive retries is as follows: Passive retries only focus on time-dimensional delays, assuming the environment will automatically recover after a period of time. This is effective for temporary errors such as network timeouts, but ineffective for missing target objects, because if the parent table record has never been synchronized, it will fail no matter how many times it is retried. Proactive probing, on the other hand, focuses on data-dimensional state changes, directly verifying whether the preconditions that caused the failure have been eliminated. This mechanism fundamentally solves the cascading blocking problem caused by inconsistent timing of cross-table synchronization, avoiding the defect in traditional solutions where the entire primary key or transaction is permanently locked due to the missing target object.

[0105] For example, probing the state of a target object in a target database includes: sending a query request for the target object to the target database based on preset probing trigger conditions to obtain the existence status of the target object.

[0106] The detection trigger condition refers to the logic for determining when to initiate the status verification of the missing target object. Its design aims to balance the real-time performance of the detection with the consumption of system resources.

[0107] In this embodiment, the detection triggering condition can be implemented in various ways to adapt to different synchronization delay sensitivity requirements.

[0108] As an optional implementation, the detection trigger condition is periodic, meaning the system has a built-in timer that scans the data change events in the dependency waiting queue at preset time intervals (e.g., every 5 seconds or every 10 seconds) and initiates a query to the target database. This approach is simple to implement and suitable for general scenarios where real-time requirements are not high but eventual consistency must be guaranteed.

[0109] As another alternative implementation, the probe triggering condition can also be event-driven. This means the system subscribes to the target database's change log or receives a "parent table write successful" notification message from other components in the synchronization link. Once a write completion signal associated with the missing target object is captured, a targeted query for that target object is immediately triggered. This event-driven mechanism can reduce probe latency from seconds to milliseconds, significantly improving synchronization throughput efficiency in high-concurrency scenarios.

[0110] It should be understood that, regardless of the triggering method used, the essence of a query request is to verify the physical existence of the target object in the target database. Its specific form can be the "SELECT EXISTS" statement or the "SELECT COUNT" statement in the Structured Query Language (SQL), or a point lookup operation based on the primary key index, as long as it can return a boolean value or a count result.

[0111] The existence status refers to the result representation returned by the query request, which includes at least two discrete states: "exists" and "does not exist". In some extended scenarios, it may also include composite states such as "logical deletion" or "version mismatch".

[0112] Step S205: If the state of the target object is detected to meet the preset recovery conditions, write the data change event to the target database again.

[0113] Preset recovery conditions refer to the fact that the dependent objects are in a valid state that can be referenced in the target database, such as the target objects being created in the target database, having a compliant structure, and being accessible for reading and writing normally.

[0114] "Re-writing the data change event to the target database" means re-injecting the previously suspended data change event into the main synchronization process and attempting to write it to the target database again. For example, when the probe thread discovers that the record with order ID=100 has been successfully written to the target database, it immediately moves the suspended payment record synchronization event back to the main queue and performs the insertion operation according to the normal process.

[0115] It should be understood that "writing the data change event to the target database again" is not simply a retry; it is a scheduling based on deterministic state verification. Furthermore, in some complex scenarios, the preset recovery conditions can be extended to situations where the target object, although not existing, has been marked as logically deleted or merged, or other specific business states, as long as this state allows the current data change event to be legally executed. Through this closed-loop mechanism, the system achieves automated healing of abnormal data, repairing data inconsistencies caused by out-of-order timing without manual intervention.

[0116] This embodiment constructs a refined classification system for anomalous data types that include dependency missing anomalies, and implements an active detection and automatic recovery mechanism for anomalous data types, effectively eliminating unnecessary blocking caused by cross-table dependency timing differences during data synchronization. Compared to traditional pessimistic locking mechanisms, this application not only ensures data consistency but also significantly improves the throughput of the synchronization link and the system's self-healing capability, ensuring efficient real-time data flow in complex heterogeneous environments.

[0117] For example, step S205 includes: if the state of the target object indicates that the target object already exists in the target database, the data change event is moved from the dependency waiting queue back to the main synchronization process to continue execution.

[0118] The master synchronization process refers to the standard execution channel responsible for writing normal data change events into the target database, which typically includes transaction commit, position update, and confirmation feedback.

[0119] Understandably, when a data change event enters the dependency waiting queue, it is in a "suspended" state and does not participate in the sorting and writing of the main process; when the target object is detected to exist, the system marks the status of the data change event as "ready" and re-injects it into the appropriate position of the main synchronization process.

[0120] The moveback operation is not simply a matter of popping and pushing data from the queue; it also requires consideration of data consistency and order guarantees. For example, if the main synchronization process strictly maintains order, the moved-back data change event should be inserted after the current consumption point and at the earliest valid position that satisfies the dependency constraints. If the system supports out-of-order writes but eventual consistency, it can be directly appended to the tail of the main queue. Furthermore, before writing the moved-back data change event, the integrity of the target object can be selectively re-verified to prevent accidental deletion or modification of the target object within the extremely short window between probing and writing. Through this precise flow control, the system achieves a qualitative leap from "passively waiting for timeouts" to "actively sensing and recovering."

[0121] This embodiment constructs a complete automated recovery loop for scenarios where the target object is missing by refining the multi-dimensional implementation of detection trigger conditions and clarifying the atomic operations of flow between queues. Compared with the traditional approach that relies on fixed timeout retries or manual intervention, this mechanism can not only dynamically adjust the recovery timing based on the actual data arrival status, avoiding the waste of resources caused by invalid retries, but also instantly restore synchronization the moment the target object is satisfied, significantly reducing the end-to-end latency of cross-table synchronization. More importantly, by decoupling the suspended data change event from the main process and then integrating it on demand, the risk of cascading blockage of the entire synchronization link due to the missing target object of a single data change event is completely eliminated, ensuring the robustness and self-healing capability of the system in complex heterogeneous environments.

[0122] This application provides a method for handling abnormal data during data synchronization. After a data change event fails to write to the target database, corresponding write failure information is collected. The failure information is analyzed using a layered fault diagnosis logic to identify the abnormal data type, overcoming the shortcomings of existing technologies that cannot distinguish the causes of abnormal write failures. For dependency missing exceptions, this application suspends the corresponding data change event and stores it in a dependency waiting queue, actively probing the status of the target objects that the data change event depends on in the target database. Once the target object meets the preset recovery conditions, the data change event is automatically retried and written to the target database. This application eliminates the need to lock data change events for extended periods; synchronization of suspended data change events is automatically restored once the target object is complete. This avoids synchronization task backlog and link blockage, effectively improving the real-time performance of data synchronization, reducing unnecessary queue resource consumption, and optimizing the overall throughput of the synchronization link.

[0123] In an optional implementation, step S202 in this embodiment includes steps A1 to A2.

[0124] For example, the write failure information includes a target error attribute and an error message text. The target error attribute represents a structured reason for failure, while the error message text represents an unstructured reason for failure.

[0125] The target error attribute refers to a structured field in the write failure message that can be directly extracted and has a clear meaning. Typical examples include the database native error code, SQL status code, or a specific exception class name returned by the driver.

[0126] Error message text refers to a natural language description string returned by a database or driver. Although its format may vary depending on the environment, it usually contains key semantic information that reflects the nature of the fault.

[0127] Step A1: From the preset mapping relationship between error attributes and abnormal data types, find the abnormal data type corresponding to the target error attribute to obtain the matching result.

[0128] The mapping between error attributes and exception data types is usually pre-loaded in memory in the form of hash tables, configuration arrays, or enumeration mappings to create a fast index from machine-readable code to business exception types.

[0129] For example, MySQL error code 1452 is pre-configured to correspond to the dependency missing exception type, error code 1213 to correspond to the transient exception type, and error code 1062 to correspond to the permanent exception type.

[0130] Upon receiving a write failure message, the system first extracts the target error code and performs an O(1) lookup in the mapping relationship between error attributes and exception data types. If a corresponding entry is found, the exception data type associated with that entry is directly used as the judgment result, and the process immediately terminates without executing subsequent steps.

[0131] In an optional implementation, the abnormal data type is represented by an error code. Step A1 includes the following steps: searching for the error code corresponding to the target error attribute from a preset mapping relationship between error attributes and error codes to obtain the matching result.

[0132] For example, the mapping between error attributes and error codes is a structured lookup table pre-stored in memory. This mapping table is typically stored using a hash table (HashMap) or an enumeration map (EnumMap) data structure to ensure O(1) time complexity for query performance.

[0133] For example, in one optional embodiment of this application, the mapping relationship between error attributes and error codes includes: mapping MySQL database error code 1452 (Cannot add or update a child row) to a dependency missing exception type; mapping error codes 1213 (Deadlock found when trying to get lock) and 1205 (Lock waittimeout exceeded) to temporary exception types; and mapping error codes 1062 (Duplicate entry) and 1054 (Unknown column) to permanent exception types.

[0134] It should be understood that the above error codes are merely illustrative examples. In actual deployment, the mapping relationship between the corresponding error attributes and error codes can be dynamically loaded according to the type of the target database (such as Oracle, PostgreSQL, SQL Server, etc.) and its version documentation, thereby achieving compatibility support for multiple heterogeneous databases.

[0135] For example, if the error code corresponding to the target error attribute is empty, the error code corresponding to the target error attribute is not in the above mapping relationship, or the error code corresponding to the target error attribute is marked as a fuzzy type that needs further analysis in the above mapping relationship, then the matching result indicates that no abnormal data type was found.

[0136] Step A2: If the matching result indicates that no abnormal data type was found, determine the abnormal data type based on the semantic features of the error message text.

[0137] Semantic feature analysis is not a simple full-text comparison, but rather it is achieved by detecting whether the text contains keyword combinations, regular expressions, or syntactic structures that characterize specific abnormal patterns. For example, even if an unknown error code fails to match the mapping relationship in step A1, if its corresponding error message text contains semantic fragments such as "foreign key constraint fails," "parent row not found," or "referenced table," the system can still classify it as a dependency missing anomaly type based on these features. Similarly, if the text contains features such as "connection reset" or "timeout expired," it is classified as a transient anomaly type.

[0138] In real-world production environments, over 90% of write failures are known standard errors, and the data type of the exception can be determined through step A1. Only a small number of non-standard errors require further analysis based on step A2. This design avoids the performance overhead of performing complex text parsing on all exceptions, while also preventing omissions caused by relying solely on error codes. Through this two-tiered progressive mechanism, this application not only improves the accuracy and robustness of exception data types but also endows the system with good scalability. When a new exception type is added, the system can prioritize adding a mapping relationship between error attributes and exception data types. The semantic rule base only needs to be maintained when a stable error code cannot be obtained, thus achieving an optimal balance between operational costs and running efficiency.

[0139] Based on the two-layer progressive judgment logic framework provided in steps A1 to A2, this embodiment further specifies the specific implementation details to address the misjudgment problem caused by differences in database versions, different driver encapsulations, or minor adjustments to log formats. Specifically, step A2 in this embodiment includes steps A21 to A23.

[0140] Step A21: Standardize the error message text to obtain standardized error text.

[0141] Standardization processing refers to performing a series of cleaning and normalization operations on the original error message text, aiming to eliminate interference to subsequent keyword matching caused by differences in capitalization, punctuation, whitespace characters, or specific formatting placeholders.

[0142] In one optional implementation, the standardization process includes at least one or more combinations of the following operations: converting all alphanumeric characters in the error message text to lowercase; removing special characters (such as quotation marks, parentheses, colons, newlines, etc.) from the error message text; removing consecutive spaces or tabs from the error message text; and removing dynamically changing numeric parameters (such as specific ID values, timestamps, etc.) from the error message text and replacing them with generic placeholders.

[0143] For example, the original error message text is "Cannot ADD OR UPDATE a child row: aFOREIGN KEY constraint fails (db.order, CONSTRAINT fk_user_id)...", while the standardized error text after standardization might be "cannot add or update a child row a foreignkey constraint fails db order constraint fk user id".

[0144] Step A22: Detect whether the standardized error text matches the target keywords in multiple preset keyword rule bases; different preset keyword rule bases correspond to different abnormal data types.

[0145] The preset keyword rule base is a pre-built set of words or regular expressions grouped by anomalous data types.

[0146] In this embodiment, at least three independent rule bases are maintained: a temporary error keyword base, a permanent error keyword base, and a dependency missing error keyword base.

[0147] The temporary error keyword library contains words that characterize transient environmental failures or resource contention, such as "timeout", "connection reset", "deadlock", "resource busy", "socket closed", and "network unreachable".

[0148] The permanent error keyword library contains words that characterize data that is itself irregular or structurally incorrect, such as "data too long", "invalid value", "column not found", "syntax error", "type mismatch", "unique constraint", etc.

[0149] The dependency missing error keyword library contains words that indicate a failure in reference integrity, such as "foreign key", "parent row", "referenced by", "no such record", and "dependency missing".

[0150] The detection process in step A22 can use a simple substring inclusion judgment or a more flexible regular expression matching to adapt to complex statement structures.

[0151] It should be understood that the selection of keywords should not be limited to the above examples. Technical personnel can continuously add new keywords to each rule base based on the statistical analysis results of actual operation logs in order to achieve iterative evolution of classification capabilities.

[0152] Step A23: Determine the abnormal data type corresponding to the preset keyword rule library to which the target keyword belongs as the abnormal data type corresponding to the data change event.

[0153] For example, if the standardized error text contains the target keyword "foreign key", and this keyword belongs to the dependency missing error keyword library, the system will directly determine the abnormal data type of the data change event as the dependency missing exception type.

[0154] If multiple keywords from different rule bases are matched at the same time, a decision can be made based on a preset priority strategy. For example, it can be prioritized as a dependency missing exception type, because this type of exception often implies the possibility of recovery through waiting, thus avoiding prematurely classifying it as a permanent error and causing unnecessary isolation.

[0155] This embodiment concretizes the abstract classification logic into two executable actions: mapping relationship query and standardized text keyword matching. This not only ensures the accuracy of anomaly identification but also significantly enhances the system's robustness against format noise. In particular, the introduction of the standardization processing step effectively solves the pain point of missed detection caused by differences in capitalization or punctuation in traditional string matching schemes, making the semantic analysis layer a truly reliable guarantee and laying a solid foundation for the correct triggering of subsequent differentiated processing strategies.

[0156] While proactive detection and automatic recovery mechanisms can effectively resolve most cross-table synchronization timing issues, certain situations may still arise in actual production environments. These situations include the source database itself being missing, a partial interruption of the synchronization link, or business logic errors preventing the target object from ever reaching the target database. Without defense logic for such situations, data change events identified as dependency missing anomalies will be suspended indefinitely in the dependency waiting queue. Over time, the dependency waiting queue will accumulate until memory resources are exhausted, potentially even causing a deadlock in the entire synchronization link. To prevent dependency waiting queue accumulation or deadlocks and ensure the robustness and security of the system under abnormal boundary conditions, this embodiment provides a timeout degradation and fallback strategy. In an optional implementation, steps B1 to B3 are also included.

[0157] Step B1: If the suspension time of the data change event in the dependency waiting queue exceeds the preset maximum waiting time, and the status of the target object still indicates that the target object does not exist in the target database, update the abnormal data type of the data change event to a permanent abnormal type.

[0158] The preset maximum waiting time refers to the upper limit threshold for the time that the system allows data change events to remain suspended in the dependency waiting queue.

[0159] The preset maximum waiting time can be configured according to the business's sensitivity to data real-time performance. For example, it can be set to 30 seconds to 1 minute for financial transaction synchronization scenarios, and 10 minutes to 30 minutes for log analysis offline synchronization scenarios.

[0160] The starting point for calculating the suspension duration is usually the moment when a data change event enters the dependency waiting queue, or the moment when the state change of the target object was last successfully detected.

[0161] When the timer detects that the suspension duration exceeds the preset maximum waiting time, and the most recent active probe still indicates that the target object is missing, the system determines that the missing target object exceeds the normal range of network latency or timing out-of-order processing, and belongs to a logical fault that cannot be self-healed. At this time, the abnormal data type of the data change event is forcibly changed from the dependency missing exception type to the permanent exception type (i.e., downgraded processing). This means that the system no longer regards it as a temporary state that can be recovered by waiting, but recognizes it as a final state that requires manual intervention or special handling.

[0162] Step B2: Extract the core metadata of the data change event.

[0163] Core metadata refers to a set of key information sufficient to uniquely identify a data change event and support subsequent troubleshooting and data repair. For example, its content includes at least: the complete payload of the data change event, the complete error stack when the write fails, the identifier of the associated target object (such as a foreign key value), the timestamp of entering the dependency waiting queue, the timestamp and result of the last probe, and a snapshot of the system context during the degradation process.

[0164] An exception repository is a persistent storage area independent of the main synchronization link. It can be a dedicated audit table in a relational database, an archive file in an object storage service (OSS), or a dedicated fault ticketing system.

[0165] Step B3: Persistently store the core metadata in the exception repository to isolate the data change event from the main synchronization process.

[0166] After writing the core metadata to the exception repository, the system immediately removes the data change event from the dependency waiting queue and the main synchronization process, releasing the memory handle and monitoring resources it occupies.

[0167] It should be understood that the aforementioned isolation is not a simple discard, but a controlled transfer. By separating unrecoverable data change events from the high-speed main flow, it ensures that subsequent normal data change events and data change events of other recoverable target objects can continue to execute smoothly, avoiding overall blockage caused by a few stubborn anomalies. At the same time, the core metadata of persistent storage provides operations and maintenance personnel with complete fault scene reconstruction capabilities, enabling subsequent manual repair or automated compensation scripts to operate based on accurate information, rather than blindly retrying.

[0168] As an alternative implementation, before performing the degradation process, the system can trigger a high-priority immediate probe request as a final confirmation to rule out false positives caused by probe thread scheduling delays. If the target object is still not found during this immediate probe, the degradation and isolation process is then executed. Furthermore, the anomaly repository can be configured with an alarm notification module that automatically sends a notification message containing a summary of core metadata to the operations team when a new degradation record is received, shortening the fault detection and response time.

[0169] This embodiment introduces timeout degradation and isolation dumping mechanisms to build a final safety barrier for handling target object loss. This strategy effectively avoids the risk of resource exhaustion and deadlock due to permanent target object loss, ensuring the continuous operation of the main synchronization link under extreme failure conditions.

[0170] To achieve comprehensive and refined management of various abnormal data types and further improve the overall exception handling system, this embodiment provides a differentiated processing strategy for non-dependent missing exception types. In an optional implementation, steps C1 to C3 are also included.

[0171] Step C1: If the abnormal data type is a temporary abnormal type, move the data change event into the delayed retry queue.

[0172] The delayed retry queue is a buffer area independent of the main synchronization process, specifically used to temporarily store data change events that fail to be written due to transient environmental failures (such as network timeouts, lock waits, connection resets, etc.).

[0173] Unlike the dependency wait queue, the data change event in the delayed retry queue does not wait for the appearance of a specific target object, but waits for time to pass in the expectation that the external environment will recover automatically.

[0174] For example, for data change events stored in a delayed retry queue, a retry timestamp can be calculated based on an exponential backoff strategy to perform a retry operation.

[0175] The exponential backoff strategy is an algorithm that dynamically adjusts the retry interval. Its core idea is to increase the waiting time exponentially with the number of retries, thus avoiding exacerbating failures due to frequent retries during system overload or network congestion. For example, in one optional embodiment of this application, the delay time for the first retry is 1 second, the second is 2 seconds, the third is 4 seconds, the fourth is 8 seconds, and so on, until a preset maximum retry interval limit (e.g., 60 seconds) is reached. This mechanism can effectively smooth out retry traffic peaks, giving the target database or network link a sufficient self-healing window.

[0176] It should be understood that exponential backoff is only one possible strategy. In other embodiments, linear backoff, Fibonacci sequence backoff, or adaptive backoff strategies that are dynamically calculated based on the current system load can also be used, as long as the effect of reducing the retry frequency with the number of failures can be achieved.

[0177] Step C2: If the number of consecutive failed retries for the data change event exceeds a preset threshold, update the abnormal data type to a permanent abnormal type.

[0178] The preset threshold number refers to the maximum number of retries the system can make when it determines that a temporary fault has become an unhealable fault. For example, it can be configured to 5 or 10 times.

[0179] When a data change event fails to write successfully after a preset threshold of retries in the delayed retry queue, the system no longer treats it as a transient anomaly that can be mitigated by time. Instead, it recognizes a potentially underlying, unidentified problem (such as hidden data conflicts or configuration errors). In this case, the abnormal data type of the data change event is forcibly changed from a temporary anomaly to a permanent anomaly, triggering the subsequent isolation and dumping process. This degradation mechanism prevents the waste of resources due to infinite retries caused by misjudgment, ensuring the system can mitigate losses promptly.

[0180] Step C3: If the abnormal data type is a permanent abnormal type, extract the core metadata of the data change event and persist the core metadata to the abnormal repository.

[0181] After step C3 is executed, the data change event can be isolated from the main synchronization process. Isolation means that the data change event will be completely removed from the execution sequence of the main synchronization process and will no longer participate in subsequent sorting, writing, and position advancement, thus eliminating its blocking effect on normal data flow. In this way, the system removes unrecoverable dirty data from the high-speed channel, ensuring the continuous smooth operation and high throughput of the main link.

[0182] It is important to note that the temporary retry handling for transient anomalies in this embodiment differs fundamentally from the "probing" handling for dependency missing anomalies. Temporary retries are a probabilistic recovery mechanism based on the time dimension, assuming that "the fault is instantaneous and the environment will automatically improve over time." Therefore, it does not concern itself with the specific data state within the target database, only with the retry time. In contrast, dependency probing is a deterministic recovery mechanism based on the data state dimension, assuming that "the fault is caused by the absence of a specific target object, and the fault will only be eliminated when that target object appears." Therefore, it must actively query the existence of the target object in the target database, rather than blindly waiting. This fundamental difference in mechanism allows this application to separate the missing target object from traditional retry or dead-letter queues for separate processing, avoiding synchronization delays caused by invalid retries or manual intervention costs due to premature transfer to the dead-letter queue.

[0183] If a dependency missing exception is treated as a transient exception and retried exponentially, even if the maximum number of retries is reached, the result will inevitably be failure and eventually degrade to a permanent error if the parent table record is not reached, causing a large number of meaningless database interactions and wasted time. Conversely, if a transient exception is mistakenly treated as a dependency missing exception for probing, since there is no clear target object to query, the probing logic will not be able to execute or will always return false, causing the data change event to be permanently suspended. Therefore, this embodiment, together with the aforementioned embodiments, constitutes a logically rigorous and clearly defined exception classification and handling system.

[0184] This embodiment achieves comprehensive and refined management of various abnormal data change events during data synchronization by constructing a resilient retry mechanism for transient anomalies and a lossless isolation mechanism for permanent anomalies, complementing the proactive detection mechanism for dependent missing anomalies. This differentiated strategy not only enhances the system's resilience and self-healing efficiency in the face of network fluctuations or resource contention, but also avoids resource waste or congestion risks caused by a single processing mode through precise traffic routing. Thus, while ensuring eventual data consistency, it maximizes the overall throughput and stability of the data synchronization link.

[0185] The above describes an abnormal data processing method in the data synchronization process provided by the embodiments of this application. The following will describe the apparatus for performing the above-described abnormal data processing method in the data synchronization process.

[0186] Please see Figure 3 , Figure 3 This is a schematic diagram of an abnormal data processing device during data synchronization, provided as an embodiment of this application. Figure 3 As shown, the abnormal data processing device during the data synchronization process includes:

[0187] Information acquisition module 301 is used to acquire write failure information corresponding to the data change event in response to the operation of writing to the target database failing due to a data change event;

[0188] The anomaly classification module 302 is used to determine the abnormal data type corresponding to the data change event based on the preset hierarchical fault judgment logic and the write failure information. The abnormal data type includes at least the dependency missing anomaly type.

[0189] The storage module 303 is used to suspend the data change event and add it to the dependency waiting queue if the abnormal data type is the dependency missing exception type. The dependency missing exception type indicates that the target object on which the data change event depends does not exist in the target database.

[0190] The detection status module 304 is used to detect the status of the target object in the target database;

[0191] The retry write module 305 is used to write the data change event to the target database again if the state of the target object is detected to meet the preset recovery conditions.

[0192] In one optional implementation, the write failure information includes a target error attribute and an error message text; the target error attribute represents a structured failure reason, and the error message text represents an unstructured failure reason; the exception classification module includes:

[0193] The lookup type unit is used to search for the abnormal data type corresponding to the target error attribute from the preset mapping relationship between error attributes and abnormal data types, so as to obtain a matching result;

[0194] An anomaly determination unit is used to determine the abnormal data type based on the semantic features of the error message text if the matching result indicates that no abnormal data type was found.

[0195] In an optional implementation, the exception data type is represented by an error code, and the lookup type unit includes:

[0196] The lookup subunit is used to search for the error code corresponding to the target error attribute from the preset mapping relationship between error attributes and error codes, so as to obtain the matching result.

[0197] In one alternative implementation, the exception-determining unit includes:

[0198] The standardization subunit is used to standardize the error message text to obtain standardized error text.

[0199] The keyword detection subunit is used to detect whether the standardized error text matches target keywords in multiple preset keyword rule bases; different preset keyword rule bases correspond to different abnormal data types;

[0200] The type determination subunit is used to determine the abnormal data type corresponding to the preset keyword rule library to which the hit target keyword belongs as the abnormal data type corresponding to the data change event.

[0201] In one alternative implementation, it also includes:

[0202] The first update type module is used to update the abnormal data type of the data change event to a permanent abnormal type if the suspension time of the data change event in the dependency waiting queue exceeds the preset maximum waiting time and the status of the target object still indicates that the target object does not exist in the target database.

[0203] The metadata extraction module is used to extract the core metadata of the data change event;

[0204] The storage module is used to persistently store the core metadata in the exception repository.

[0205] In one alternative implementation, it also includes:

[0206] The migration queue module is used to move the data change event into the delayed retry queue if the abnormal data type is a temporary abnormal type.

[0207] The second update type module is used to update the abnormal data type to a permanent abnormal type if the number of consecutive retry failures of the data change event exceeds a preset threshold.

[0208] An extraction and storage module is used to extract the core metadata of the data change event if the abnormal data type is a permanent abnormal type, and to persistently store the core metadata in the abnormal repository.

[0209] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0210] like Figure 4As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0211] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0212] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the abnormal data processing methods provided in the data synchronization process of this application.

[0213] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the abnormal data processing methods provided in the data synchronization process of this application.

[0214] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0215] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0216] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0217] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for handling exceptional data in a data synchronization process, the method comprising: include: In response to a failure to write a data change event to the target database, obtain the write failure information corresponding to the data change event; Based on the preset hierarchical fault determination logic and the write failure information, the abnormal data type corresponding to the data change event is determined, and the abnormal data type includes at least the dependency missing exception type. If the abnormal data type is the dependency missing exception type, the data change event is suspended and added to the dependency waiting queue. The dependency missing exception type indicates that the target object on which the data change event depends does not exist in the target database. Detect the state of the target object in the target database; If the state of the target object is detected to meet the preset recovery conditions, the data change event is written to the target database again.

2. The method of claim 1, wherein, The write failure information includes the target error attribute and the error message text; The target error attribute represents a structured reason for failure, while the error message text represents an unstructured reason for failure. The method for determining the abnormal data type corresponding to the data change event based on the preset hierarchical fault determination logic and the write failure information includes: From the preset mapping relationship between error attributes and abnormal data types, the abnormal data type corresponding to the target error attribute is searched to obtain the matching result; If the matching result indicates that no abnormal data type was found, the abnormal data type is determined based on the semantic features of the error message text.

3. The method of claim 2, wherein, The abnormal data type is represented by an error code. The step of searching for the abnormal data type corresponding to the target error attribute from a preset mapping relationship between error attributes and abnormal data types to obtain a matching result includes: The error code corresponding to the target error attribute is searched from the preset mapping relationship between error attributes and error codes to obtain the matching result.

4. The method of claim 2 or 3, wherein, Determining the abnormal data type based on the semantic features of the error message text includes: The error message text is standardized to obtain a standardized error text. The system detects whether the standardized error text matches target keywords in multiple preset keyword rule bases; different preset keyword rule bases correspond to different abnormal data types. The abnormal data type corresponding to the preset keyword rule library to which the target keyword belongs is determined as the abnormal data type corresponding to the data change event.

5. The method of claim 1 to 3, wherein, Also includes: If the suspension time of the data change event in the dependency waiting queue exceeds the preset maximum waiting time, and the status of the target object still indicates that the target object does not exist in the target database, the abnormal data type of the data change event is updated to a permanent abnormal type. Extract the core metadata of the data change event; The core metadata is persistently stored in the exception repository.

6. The method of claim 1 to 3, wherein, Also includes: If the abnormal data type is a temporary abnormal type, the data change event is moved to the delayed retry queue; If the number of consecutive failed retries for the data change event exceeds a preset threshold, the abnormal data type is updated to a permanent abnormal type. If the abnormal data type is a permanent abnormal type, extract the core metadata of the data change event and persist the core metadata to the abnormal repository.

7. An abnormal data processing device during data synchronization, characterized in that, include: The information acquisition module is used to acquire the write failure information corresponding to the data change event in response to the failure of writing to the target database in response to the data change event. An anomaly classification module is used to determine the abnormal data type corresponding to the data change event based on a preset hierarchical fault judgment logic and the write failure information. The abnormal data type includes at least the dependency missing anomaly type. The storage module is used to suspend the data change event and add it to the dependency waiting queue if the abnormal data type is the dependency missing exception type. The dependency missing exception type indicates that the target object on which the data change event depends does not exist in the target database. A status detection module is used to detect the status of the target object in the target database; The retry write module is used to write the data change event to the target database again if the state of the target object is detected to meet the preset recovery conditions.

8. An electronic device, comprising: It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the abnormal data processing method in the data synchronization process as described in any one of claims 1 to 6.

9. A computer program product, characterised in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the abnormal data processing method in the data synchronization process as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that The computer storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the abnormal data processing method in the data synchronization process as described in any one of claims 1 to 6.