Data fault-tolerant processing method, device and equipment for heterogeneous database synchronization, and medium

Abstract

This invention relates to the field of database technology, and more particularly to a data fault tolerance processing method, apparatus, device, and medium for heterogeneous database synchronization. After capturing a preset operation event in the source database, current synchronization processing is performed. If an exception is triggered during the current synchronization process, the exception is classified. By classifying and processing synchronization exceptions, corresponding circuit breaker decisions and fault tolerance actions can be accurately matched. Circuit breaker processing is performed on the corresponding object that triggered the exception, without affecting the synchronization process of other normal data. This ensures effective exception handling while maximizing the preservation of data synchronization processing capabilities, thus improving overall synchronization efficiency. Simultaneously, by classifying exceptions and matching corresponding fault tolerance actions, targeted exception repair can be performed, improving the accuracy and timeliness of exception handling. This prevents the accumulation of exceptions from causing data deviations between the source and target ends, thereby ensuring the consistency of data synchronization between the source and target heterogeneous databases.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data fault-tolerant processing method, apparatus, device, and medium for heterogeneous database synchronization. Background Technology

[0002] With the deepening of enterprise digital transformation and the evolution of IT architecture, data synchronization technology has become a core infrastructure supporting data flow, business continuity, hybrid cloud deployment, and real-time data analysis. Its application scenarios are wide-ranging, including but not limited to: database migration and upgrades, business system decoupling and data distribution, building real-time data warehouses and data lakes, and implementing multi-active and disaster recovery architectures. Against this backdrop, the need for heterogeneous database logical synchronization is increasingly prominent. Enterprises often use multiple databases simultaneously to achieve different business objectives. However, heterogeneous environments bring significant compatibility challenges. Specifically, Data Definition Language (DDL) operations and Data Control Language (DCL) operations have numerous proprietary syntaxes, extended functions, and behavioral differences across different database vendors. These differences make synchronization interruptions highly likely during heterogeneous database logical synchronization, severely impacting data consistency. Therefore, improving data consistency during heterogeneous database synchronization has become an urgent problem to solve. Summary of the Invention

[0003] This invention provides a data fault tolerance processing method, apparatus, device, and medium for heterogeneous database synchronization, in order to solve the problem of poor data consistency during heterogeneous database synchronization.

[0004] Firstly, a data fault tolerance processing method for heterogeneous database synchronization is provided, the data fault tolerance processing method comprising: After capturing the preset operation event of the source database, the current synchronization process is performed. If an exception is triggered during the current synchronization process, the exception is classified to obtain the exception classification result. The current synchronization process is to synchronize the operation object of the operation event to the target heterogeneous database. Based on the anomaly classification result, determine the circuit breaker handling decision that matches the anomaly classification result, and based on the circuit breaker handling decision, determine the corresponding current circuit breaker object; Based on the anomaly classification results, determine the fault-tolerant action corresponding to the current circuit breaker object, and execute the fault-tolerant action.

[0005] Secondly, a data fault tolerance processing device for heterogeneous database synchronization is provided, the data fault tolerance processing device comprising: The classification module is used to perform current synchronization processing after capturing the preset operation events of the source database. If an exception is triggered during the current synchronization processing, the exception is classified to obtain the exception classification result. The current synchronization processing is to synchronize the operation object of the operation event to the target heterogeneous database. The matching module is used to determine the circuit breaker decision that matches the anomaly classification result, and to determine the corresponding current circuit breaker object based on the circuit breaker decision. The determination module is used to determine the fault-tolerant action corresponding to the current circuit breaker object based on the anomaly classification result, and execute the fault-tolerant action.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data fault tolerance processing method.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described data fault tolerance processing method.

[0008] The advantages of this application compared to the prior art are: This application, upon capturing a pre-defined operation event from the source database, performs current synchronization processing. If an exception is triggered during this process, it is categorized. By classifying and handling synchronization exceptions, corresponding circuit breaker decisions and fault-tolerance actions can be accurately matched. Circuit breaker processing is applied to the object triggering the exception without affecting the synchronization process of other normal data. This ensures effective exception handling while maximizing the preservation of data synchronization capabilities, thus improving overall synchronization efficiency. Furthermore, by matching exceptions with corresponding fault-tolerance actions, targeted exception repairs can be performed, improving the accuracy and timeliness of exception handling. This prevents the accumulation of exceptions from causing data discrepancies between the source and target databases, thereby ensuring the consistency of data synchronization between the source and target heterogeneous databases. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an application environment for a data fault tolerance processing method for heterogeneous database synchronization provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a data fault tolerance processing method for heterogeneous database synchronization provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a data fault-tolerant processing device for heterogeneous database synchronization provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0013] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0014] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0015] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0016] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0017] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0018] To illustrate the technical solution of the present invention, specific embodiments are described below. An embodiment of the present invention provides a data fault tolerance processing method for heterogeneous database synchronization, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. On the client side, when a user performs a change operation on the source database, the server, after capturing the preset operation event in the source database, performs current synchronization processing. If an exception is triggered during the current synchronization processing, the exception is classified to obtain an exception classification result. The current synchronization processing involves synchronizing the operation object of the operation event to the target heterogeneous database. Based on the exception classification result, a circuit breaker decision matching the exception classification result is determined. Based on the circuit breaker decision, the corresponding current circuit breaker object is determined. Based on the exception classification result, the corresponding fault tolerance action for the current circuit breaker object is determined and executed. The server is a database server, which can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0019] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data fault tolerance processing method for heterogeneous database synchronization provided in an embodiment of the present invention, which includes the following steps: S21: After capturing the preset operation event of the source data, perform current synchronization processing. If an exception is triggered during the current synchronization processing, classify the exception and obtain the exception classification result. The current synchronization processing is to synchronize the operation object of the operation event to the target heterogeneous database.

[0020] In step S21, the source database is the database on which the operation is performed, and the preset operation event is an event that modifies data records in the source database. The current synchronization process involves synchronizing the operation object of the operation event to the target heterogeneous database. The target heterogeneous database is a database system that differs from the source database in data model, storage structure, or query language. An exception is a situation where an error occurs during the synchronization process, and the exception classification result is a category identifier based on the exception characteristics.

[0021] In this embodiment, after capturing a preset operation event from the source database, current synchronization processing is performed. The preset operation event is a DDL (Data Definition Language) operation event or a DML (Data Manipulation Language) operation event. DDL operations may include creating, modifying, or deleting table structures, while DML operations may include inserting, updating, or deleting data records.

[0022] It should be noted that when capturing preset operation events from the source database using CDC (Change Data Capture) technology, after capturing these events, it is necessary to identify the data carrier type that matches the current operation event and determine the corresponding input. To identify the data carrier type and determine the corresponding input, the pre-configured acquisition mode identifier of the current synchronization task can be read. This identifier represents the data acquisition method enabled by the current synchronization link. There are two preset acquisition modes: SQL acquisition mode and system table monitoring mode. Only one acquisition mode is configured for each synchronization task. Based on the read acquisition mode identifier, the data carrier type is determined as follows: If the acquisition mode identifier is SQL acquisition mode, the data carrier matched by the current operation event is determined to be a raw SQL statement, and the corresponding input is SQL text; if the acquisition mode identifier is system table monitoring mode, the data carrier matched by the current operation event is determined to be a database system table change record, and the corresponding input is a database system table change record. The two collection modes operate independently and mutually exclusively. The SQL collection mode relies on the database binary log and audit log to capture execution statements, while the system table monitoring mode listens to the system metadata table of the repository, table, field, and permission information through timed polling or triggers. Both collection methods can independently capture complete DDL and DCL change events, with the only difference being the input data carrier.

[0023] After capturing the preset operation events in the source database, current synchronization processing is performed, which includes three stages: event parsing, semantic transformation, and target-side adaptation. Event parsing involves analyzing the input of the operation events to extract metadata information. This metadata information includes owner, object, type, and scope. The owner is the owner of the objects involved in the operation; the object is the name of the object; the type is the type of operation, such as CREATE TABLE or ALTER COLUMN; and the scope is the scope of the operation, such as table-level, column-level, or database-level. Semantic transformation maps the extracted metadata information to a semantic model compatible with the target heterogeneous database. Target-side adaptation further adapts the transformed semantic model to DDL or DCL statements that can be directly executed in the target heterogeneous database. Subsequently, by executing these adapted statements, changes to the table structure and access permissions are completed in the target heterogeneous database, ultimately achieving synchronized implementation of changes from the source database to the target database.

[0024] If an exception is triggered during the current synchronization process, the exception is classified to obtain an exception classification result. The classification process uses a preset classification model. The input parameters of the preset classification model may include an exception triggering stage identifier, heterogeneous database adaptation configuration parameters, and event processing context state. The exception triggering stage identifier marks the stage in the change synchronization process where the exception occurred, such as the input parsing stage, metadata extraction stage, heterogeneous statement conversion stage, or target database execution write stage. This parameter allows for quick location of the faulty link node. The heterogeneous database adaptation configuration parameters include preset synchronization configuration information such as source database type, target database type, field type mapping rules, syntax compatibility whitelist, and special object adaptation switch. The event processing context state represents the current event processing progress, cached data integrity, message queue transmission status, and log persistence status. The model output is an exception classification result that can be either predictable or unpredictable errors.

[0025] It should be noted that before using the preset classification model, the classification model is trained. During training, sample data and corresponding anomaly labels are obtained. The sample data includes the anomaly trigger identifier, heterogeneous database adaptation configuration parameters, and sample event processing context state of real anomalies generated in historical synchronization tasks. The anomaly labels are the specific categories to which the historical anomalies belong. A training dataset is constructed based on the sample data and their corresponding anomaly labels. The classification model is trained using a supervised learning algorithm until the model converges, resulting in the trained classification model, i.e., the preset classification model.

[0026] It should be noted that predictable errors can include SQL syntax parsing failures; system table change parsing failures; logical errors in cross-heterogeneous database syntax adaptation and conversion; write failures caused by generating illegal execution statements; concurrent write conflicts with the target database, network interruptions, database service failures, and insufficient storage resources. Unpredictable errors can include underlying program runtime failures, such as array out-of-bounds errors and memory overflows during parsing and conversion; internal failures in the synchronization client program, resulting in errors that cannot be predicted in advance; network link failures; abnormal errors in the target heterogeneous database service; and subsequent continuous data write failures caused by incompatibility issues with the underlying syntax, data types, and constraint mechanisms of the target heterogeneous database.

[0027] In this embodiment, after capturing a preset operation event in the source database, current synchronization processing is performed to synchronize the changed data in the source database to the target heterogeneous database. If an exception is triggered during the current synchronization processing, the exception is classified to determine its specific type, and a targeted recovery strategy is then adopted.

[0028] Optionally, the anomalies are classified to obtain anomaly classification results, including: Obtain the preset rule base, which is used to store the mapping relationship between the abnormal characteristics and the abnormal categories when an anomaly occurs; Extract the current anomaly features of the corresponding anomaly, determine the anomaly category that matches the current anomaly features based on the preset rule base, and obtain the corresponding anomaly classification result.

[0029] In this embodiment, a pre-configured rule base is embedded in the synchronization system, specifically adapted to DDL and DCL change synchronization scenarios between heterogeneous databases. The rule base stores a one-to-one mapping relationship between multiple anomaly features and anomaly categories. Anomaly categories are uniformly divided into predictable errors and unpredictable errors, each with several fixed anomaly features. Predictable errors are anomalies caused by deterministic problems during the change synchronization process, such as configuration mismatch, syntax incompatibility, and parameter errors. Corresponding anomaly features include, but are not limited to, cross-database syntax incompatibility, field type mapping failure, missing target database permissions, and incorrect synchronization configuration parameters. Unpredictable errors are sudden, random anomalies without a fixed pattern during the synchronization process. Corresponding anomaly features include, but are not limited to, database link disconnection, data read / write timeout, system memory anomalies, message transmission loss, and momentary service jitter.

[0030] During the current change synchronization process, if any stage—event parsing, semantic transformation, or target-end adaptation—triggers an exception, the system immediately locks the current exception event and extracts its characteristics from all aspects. These characteristics include at least the business process that triggered the exception, the system's returned error code, the exception text information, the input type of this change event, the metadata parsing status, the heterogeneous adaptation parameter verification results, and the data transmission status. This forms a structured set of current exception characteristics for subsequent rule matching and determination.

[0031] After extracting the current anomaly features, the system performs precise traversal and matching of the structured anomaly feature set with each mapping rule in the preset rule base, selecting the target mapping rule that highly matches the current anomaly features. Based on the anomaly category corresponding to the target mapping rule, the corresponding anomaly classification result is determined.

[0032] In this embodiment, the standardized matching and classification of abnormal features and abnormal categories is completed by using a preset rule base. This can unify the abnormal judgment criteria in heterogeneous data change synchronization scenarios, avoid judgment errors caused by manual classification, effectively improve the accuracy and matching efficiency of abnormal classification, provide a standardized classification basis for the rapid location and handling of subsequent abnormalities, reduce the complexity of abnormal investigation in heterogeneous data synchronization scenarios, and improve the stability and maintainability of the overall change synchronization process.

[0033] S22: Based on the anomaly classification results, determine the circuit breaker handling decision that matches the anomaly classification results, and based on the circuit breaker handling decision, determine the corresponding current circuit breaker object.

[0034] In step S22, the circuit breaker decision is a differentiated response strategy formulated for predictable and unpredictable errors. The current circuit breaker object is the specific data entity to which the circuit breaker operation is performed.

[0035] In this embodiment, based on the anomaly classification result, a circuit breaker decision matching the anomaly classification result is determined. If the anomaly classification result is a predictable error, the corresponding circuit breaker decision is determined to be a predictable error handling decision. If the anomaly classification result is an unpredictable error, the corresponding circuit breaker decision is determined to be an unpredictable error handling decision.

[0036] It should be noted that predictable error handling decisions are refined, localized circuit breaker decisions. Since predictable errors are deterministic, reproducible, and scenario-specific errors such as syntax incompatibility, type mapping failures, and missing permissions, the impact of the fault is limited to the operation object corresponding to this change. Based on the extracted complete metadata information, the system accurately identifies the database, table, target field, or permission object corresponding to this anomaly, and determines the business data entity corresponding to this single change operation as the current circuit breaker object. Circuit breaker isolation is only applied to the local database object associated with the anomaly, without affecting the synchronization link of other normal database objects, thus maximizing the stability of the overall synchronization task.

[0037] Unpredictable error handling decisions are broad-based overall circuit breaker decisions. Because unpredictable errors are sudden and uncertain anomalies such as link interruptions, transient system failures, and message loss, the causes and scope of impact cannot be precisely located, posing a risk of propagation. Based on currently extractable metadata or retained event context fragments, the system narrows down the scope of all database objects associated with the current change event. This allows the entire set of synchronization objects corresponding to the current synchronization task to be identified as the current circuit breaker object, enabling unified circuit breaker isolation of associated data entities. This prevents the continued propagation of unknown anomalies from causing synchronization link avalanche, data corruption, or batch synchronization failures.

[0038] In this embodiment, differentiated circuit breaker processing decisions are adopted to adapt the processing logic of local precise circuit breaker and global scope circuit breaker to two types of anomalies respectively. The circuit breaker object adapted to the current fault scenario is dynamically calculated. This can not only reduce the fault isolation scope and improve synchronization efficiency in deterministic anomaly scenarios, but also achieve risk fallback isolation in uncertain anomaly scenarios, thus taking into account the fault tolerance security and business continuity of heterogeneous database synchronization system.

[0039] Optionally, based on the circuit breaker handling decision, the corresponding current circuit breaker object is determined, including: If the circuit breaker decision is a predictable error handling decision, then obtain the change log when the preset operation event of the source database is captured, extract the metadata information based on the change log, and determine the corresponding current circuit breaker object based on the metadata information. If the circuit breaker decision is an unpredictable error handling decision, then the context information of the operation event is obtained, and the circuit breaker object is inferred based on the context information to obtain the current circuit breaker object.

[0040] In this embodiment, if the circuit breaker decision is a predictable error handling decision, the change log of the source database when a preset operation event is captured is obtained. The change log is a structured log data generated by the source database when the operation event is executed. When extracting metadata information based on the change log, the metadata information includes owner, object, type, and scope. The owner is the owner of the operation object involved in the operation; the object is the name of the operation object; the type is the type of operation, such as CREATETABLE or ALTER COLUMN; and the scope is the scope of the operation's impact, such as table-level, column-level, or database-level. Through these clearly verifiable identifiers, the specific data object corresponding to the occurrence of the anomaly can be directly identified. This specific operation-related object can be designated as the circuit breaker object that needs to be isolated, without additionally affecting other normally operating data synchronization processes in the current synchronization task. While controlling the scope of the fault's impact, the synchronization of other data is ensured to remain undisturbed, maintaining the business continuity of the overall synchronization task. For example, if an ALTER TABLE ... ADD COLUMN statement fails, it can be precisely determined that only that specific table needs to be circuit-broken without affecting other tables.

[0041] In this embodiment, a fine-grained object mapping relationship is constructed by extracting metadata information, achieving precise binding between abnormal objects and synchronization links, thus avoiding the resource waste caused by a one-size-fits-all global block. Optionally, determining the corresponding current circuit breaker object based on the circuit breaker handling decision also includes: If the circuit breaker decision is an unpredictable error handling decision, then obtain the context information of the operation event; Based on the context information, the circuit breaker object is inferred to obtain the current circuit breaker object.

[0042] In this embodiment, if the circuit breaker decision is an unpredictable error handling decision, the context information of the operation event is obtained. This context information includes related operations such as transactions and sessions associated with the preset operation event. For example, other related operations within the same database transaction or session as the operation event, i.e., other database change operations connected through the same database session or encapsulated within the same database transaction during the execution period of the DDL or DCL operation event that generated the exception. These operations are execution-related to the current operation, posing risks of mutual influence, synchronization linkage, and transaction rollback-related pollution. For example, when an operation event is captured as adding an index to a data table using SQL statements, the same transaction includes batch data update operations on that table. Current synchronization processing is performed. If an exception is triggered during the current synchronization processing, the corresponding exception classification result is an unpredictable error, and the corresponding handling decision is an unpredictable error handling decision. When calculating the corresponding current circuit breaker object, the context information of the operation event is obtained, which is the batch data update operation on that table.

[0043] When inferring circuit breaker objects based on context information, all database objects involved in the relevant operations in the context information are included in the scope of circuit breaker objects. That is, when the operation event involves other changed objects in the same transaction or other operations in the same session, these related objects will be identified together with the operation object at the time of the current exception as the objects that need to be circuit-broken.

[0044] It should be noted that when the circuit breaker decision is an unpredictable error handling decision, if some valid metadata information corresponding to the preset operation event can be extracted, the corresponding current circuit breaker object can be determined based on the valid metadata information. For example, if the corresponding stage is the event parsing stage when an exception occurs, and some table names or field names and other key metadata are parsed out, the affected database objects can be located directly based on this valid information, and these database objects can be determined as the current circuit breaker objects.

[0045] In this embodiment, differentiated circuit breaker strategies are adopted for different error types. If the circuit breaker decision is a predictable error handling decision, the corresponding current circuit breaker object is determined based on metadata information, and the specific data entity that caused the exception is precisely located. If the circuit breaker decision is an unpredictable error handling decision, the context information of the operation event is obtained, and the circuit breaker object is inferred based on the context information to obtain the current circuit breaker object. All related operations of the operation event are included in the circuit breaker handling scope to avoid secondary exceptions caused by unprocessed related operations.

[0046] S23: Based on the anomaly classification results, determine the fault-tolerant action corresponding to the current circuit breaker object and execute the fault-tolerant action.

[0047] In step S23, the fault-tolerance action is an isolation measure implemented for the current circuit breaker object. Different exception categories correspond to different fault-tolerance actions, and executing a fault-tolerance action means blocking subsequent access requests to that circuit breaker object.

[0048] In this embodiment, based on the anomaly classification results, the fault-tolerance action corresponding to the current circuit breaker object is determined. Fault-tolerance actions may include skipping, logging, issuing an alarm, or suspending for manual handling. Skipping indicates temporarily ignoring access requests to the circuit breaker object; logging is used to preserve the anomaly situation for subsequent traceability and analysis; issuing an alarm aims to notify operations and maintenance personnel to intervene and investigate in real time; and suspending for manual handling is suitable for severe anomaly scenarios that are logically complex and difficult to recover automatically. For example, fault-tolerance actions for predictable errors may include skipping and logging for subsequent auditing and investigation. Fault-tolerance actions for unpredictable errors may include issuing an alarm and suspending for manual handling.

[0049] It should be noted that when determining the fault-tolerance action corresponding to the current circuit breaker object based on the anomaly classification results, the current load status of the system and business priorities must also be comprehensively considered. When the system is under high load, to ensure the continuity of core business, anomalies of non-critical objects can be skipped first, thereby releasing system resources; while under low load or during critical business periods, stricter alarm and suspension policies are preferred to ensure data consistency and integrity.

[0050] In this embodiment, the fault-tolerant action corresponding to the current circuit breaker object is determined by the anomaly classification results. Based on the different types of anomalies, a more suitable fault-tolerant action can be matched, thus achieving a balance between the accuracy of fault-tolerant processing and the stability of system operation.

[0051] Optionally, after performing the fault-tolerance action, the following may also be included: Obtain the preset circuit breaker state library, update the circuit breaker state library based on the current circuit breaker object and the fault tolerance action corresponding to the current circuit breaker object, and obtain the updated circuit breaker state library. The circuit breaker state library is used to store each circuit breaker object and each circuit breaker object's fault tolerance action. When performing the next change synchronization process, the system determines whether the target operation object in the next change synchronization process is a circuit breaker object based on the updated circuit breaker status database. If it is a circuit breaker object, the system performs a fault tolerance action on the target operation object according to the corresponding fault tolerance action.

[0052] In this embodiment, a preset circuit breaker state database is obtained. This database stores each circuit breaker object and its corresponding fault-tolerance actions. The database is updated based on the current circuit breaker object and its corresponding fault-tolerance actions to obtain the updated database. The preset circuit breaker state database can be an in-memory hash table, a distributed cache (such as Redis), or a persistent database. Through persistence or caching mechanisms, it is ensured that circuit breaker objects can still be recovered after system restarts or node failures.

[0053] When performing the next change synchronization process, the updated circuit breaker status database is used to determine whether the target operation object in the next change synchronization process is a circuit breaker object; that is, whether the target operation object in the next change synchronization process matches the circuit breaker object recorded in the circuit breaker status database. If a match is found, the target operation object is considered a circuit breaker object; if not, it is treated as a normal object and processed according to the normal procedure. If it is a circuit breaker object, the corresponding fault-tolerant action is performed on the target operation object, such as skipping, logging, issuing an alarm, or suspending it for manual handling.

[0054] In this embodiment, the current circuit breaker object and its corresponding fault-tolerant action are updated in the circuit breaker state database to facilitate quick retrieval of the circuit breaker object and its fault-tolerant action in subsequent processing, thereby avoiding repeated anomaly detection overhead and significantly improving the system's response speed and throughput.

[0055] This application, upon capturing a pre-defined operation event from the source database, performs current synchronization processing. If an exception is triggered during this process, it is categorized. By classifying and handling synchronization exceptions, corresponding circuit breaker decisions and fault-tolerance actions can be accurately matched. Circuit breaker processing is applied to the object triggering the exception without affecting the synchronization process of other normal data. This ensures effective exception handling while maximizing the preservation of data synchronization capabilities, thus improving overall synchronization efficiency. Furthermore, by matching exceptions with corresponding fault-tolerance actions, targeted exception repairs can be performed, improving the accuracy and timeliness of exception handling. This prevents the accumulation of exceptions from causing data discrepancies between the source and target databases, thereby ensuring the consistency of data synchronization between the source and target heterogeneous databases.

[0056] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0057] In one embodiment, a data fault tolerance processing apparatus is provided, which corresponds one-to-one with the data fault tolerance processing methods described in the above embodiments. For example... Figure 3 As shown, the data fault-tolerant processing device 30 includes: a classification module 31, a matching module 32, and a determination module 33.

[0058] The classification module 31 is used to perform current synchronization processing after capturing the preset operation event of the source database. If an exception is triggered during the current synchronization processing, the exception is classified to obtain the exception classification result. The current synchronization processing is to synchronize the operation object of the operation event to the target heterogeneous database. The matching module 32 is used to determine the circuit breaker decision that matches the anomaly classification result based on the anomaly classification result, and to determine the corresponding current circuit breaker object based on the circuit breaker decision. The determination module 33 is used to determine the fault-tolerant action corresponding to the current circuit breaker object based on the anomaly classification result, and to execute the fault-tolerant action.

[0059] Optionally, the data fault-tolerant processing device 30 further includes: The acquisition module is used to acquire a preset circuit breaker state library, update the circuit breaker state library based on the current circuit breaker object and the fault tolerance action corresponding to the current circuit breaker object, and obtain the updated circuit breaker state library. The circuit breaker state library is used to store each circuit breaker object and each circuit breaker object's fault tolerance action. The judgment module is used to determine whether the target operation object in the next change synchronization process is a circuit breaker object based on the updated circuit breaker status database. If it is a circuit breaker object, the module performs a fault tolerance action on the target operation object according to the corresponding fault tolerance action.

[0060] Optionally, the classification module 31 includes: The first acquisition unit is used to acquire a preset rule base, which is used to store the mapping relationship between abnormal features and abnormal categories when an anomaly occurs. The extraction unit is used to extract the current abnormal features of the corresponding abnormality, determine the abnormality category that matches the current abnormal features according to the preset rule base, and obtain the corresponding abnormality classification result.

[0061] Optionally, the matching module 32 includes: The second acquisition unit is used to acquire the change log when a preset operation event in the source database is captured if the circuit breaker decision is a predictable error handling decision. The determination unit is used to extract metadata information from the change log and determine the corresponding current circuit breaker object based on the metadata information.

[0062] Optionally, the matching module 32 also includes: The third acquisition unit is used to acquire the context information of the operation event if the circuit breaker decision is an unpredictable error handling decision. The inference unit is used to infer the circuit breaker object based on the context information to obtain the current circuit breaker object.

[0063] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0064] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory, and a computer program stored in the memory and capable of running on at least one processor, wherein the processor executes the computer program to implement the steps in the above-described embodiments of the data fault tolerance processing method for synchronizing any of the heterogeneous databases.

[0065] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0066] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0067] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0068] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0069] The present invention can implement all or part of the processes in the above embodiments of the method, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0070] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0071] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0072] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be implemented through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application is authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data fault tolerance processing method for heterogeneous database synchronization, characterized in that, The data fault tolerance processing method includes: After capturing the preset operation event of the source database, the current synchronization process is performed. If an exception is triggered during the current synchronization process, the exception is classified to obtain the exception classification result. The current synchronization process is to synchronize the operation object of the operation event to the target heterogeneous database. Based on the anomaly classification result, determine the circuit breaker handling decision that matches the anomaly classification result, and based on the circuit breaker handling decision, determine the corresponding current circuit breaker object; Based on the anomaly classification results, determine the fault-tolerant action corresponding to the current circuit breaker object, and execute the fault-tolerant action.

2. The data fault tolerance processing method as described in claim 1, characterized in that, The classification of the anomalies to obtain anomaly classification results includes: Obtain a preset rule base, which is used to store the mapping relationship between abnormal features and abnormal categories when an anomaly occurs; Extract the current anomaly features of the corresponding anomaly, and determine the anomaly category that matches the current anomaly features according to the preset rule base to obtain the corresponding anomaly classification result.

3. The data fault tolerance processing method as described in claim 1, characterized in that, The anomaly classification results include predictable errors and unpredictable errors. The circuit breaker handling decision includes a predictable error handling decision that matches the predictable error and an unpredictable error handling decision that matches the unpredictable error.

4. The data fault tolerance processing method as described in claim 3, characterized in that, The step of determining the corresponding current circuit breaker object based on the circuit breaker processing decision includes: If the circuit breaker decision is a predictable error handling decision, then obtain the change log when the preset operation event of the source database is captured; Based on the change log, extract metadata information, and determine the corresponding current circuit breaker object based on the metadata information.

5. The data fault tolerance processing method as described in claim 3, characterized in that, The step of determining the corresponding current circuit breaker object based on the circuit breaker processing decision also includes: If the circuit breaker decision is an unpredictable error handling decision, then obtain the context information of the operation event; Based on the context information, the circuit breaker object is inferred to obtain the current circuit breaker object.

6. The data fault tolerance processing method as described in claim 1, characterized in that, After performing the fault-tolerant action, the method further includes: Obtain a preset circuit breaker state library, update the circuit breaker state library based on the current circuit breaker object and the fault-tolerant action corresponding to the current circuit breaker object, and obtain the updated circuit breaker state library. The circuit breaker state library is used to store each circuit breaker object and the fault-tolerant action of each circuit breaker object. When performing the next change synchronization process, the system determines whether the target operation object in the next change synchronization process is a circuit breaker object based on the updated circuit breaker status database. If it is a circuit breaker object, the system performs a fault tolerance action on the target operation object according to the corresponding fault tolerance action.

7. A data fault-tolerant processing device for heterogeneous database synchronization, characterized in that, The data fault-tolerant processing device includes: The classification module is used to perform current synchronization processing after capturing the preset operation events of the source database. If an exception is triggered during the current synchronization processing, the exception is classified to obtain the exception classification result. The current synchronization processing is to synchronize the operation object of the operation event to the target heterogeneous database. The matching module is used to determine the circuit breaker decision that matches the anomaly classification result, and to determine the corresponding current circuit breaker object based on the circuit breaker decision. The determination module is used to determine the fault-tolerant action corresponding to the current circuit breaker object based on the anomaly classification result, and execute the fault-tolerant action.

8. The data fault-tolerant processing apparatus as described in claim 7, characterized in that, The data fault-tolerant processing device further includes: The acquisition module is used to acquire a preset circuit breaker state library, update the circuit breaker state library based on the current circuit breaker object and the fault-tolerant action corresponding to the current circuit breaker object, and obtain the updated circuit breaker state library. The circuit breaker state library is used to store each circuit breaker object and each circuit breaker object's fault-tolerant action. The judgment module is used to determine whether the target operation object in the next change synchronization process is a circuit breaker object based on the updated circuit breaker status database. If it is a circuit breaker object, the module performs a fault tolerance action on the target operation object according to the corresponding fault tolerance action.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data fault tolerance processing method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data fault tolerance processing method as described in any one of claims 1 to 6.

Images