Data conflict processing method and device based on cdc synchronization and computer device

Abstract

This paper relates to the fields of big data and fintech, particularly to a method, apparatus, and computer equipment for handling data conflicts based on CDC synchronization. The method includes detecting the existence of synchronization conflicts in the CDC synchronization log; if a synchronization conflict is identified, extracting relevant data from the CDC synchronization log; analyzing the relevant data to determine the conflict type; determining a resolution component based on the conflict type, with each resolution component corresponding to the conflict type; and invoking the resolution component to handle the synchronization conflict. This paper uses keyword matching analysis of the CDC synchronization conflict log to filter out conflicts causing predetermined errors; resolves the corresponding conflict errors using several resolution components or methods, and automates the repair process using scripts; and restarts the predetermined replication process by recording the SCN number and using the SCN at the time of interruption, ensuring no data loss after interruption.

Description

Technical Field

[0001] This article covers the fields of big data and fintech, especially methods, devices, and computer equipment for handling data conflicts based on CDC synchronization. Background Technology

[0002] Change Data Capture (CDC) is an incremental data synchronization tool based on database logs, primarily used for data synchronization between two databases. The CDC tool architecture consists of a control server and two instance servers on the two databases. When using it, a data store needs to be established on the instance servers, and a reservation needs to be established on the control server. A reservation is a collection of tables from both the source and target databases, containing data synchronization mapping relationships.

[0003] During the operation of CDC tools, data synchronization conflicts can lead to pre-defined anomalies and direct interruptions in data replication. Current technology relies on manual analysis to determine the cause of these anomalies, which is inefficient and cannot guarantee the accuracy of the analysis. Summary of the Invention

[0004] To address the issues of low efficiency and inaccuracy in data synchronization interruption anomaly analysis in the prior art, this embodiment provides a data conflict handling method, apparatus, and computer equipment based on CDC synchronization.

[0005] This embodiment provides a method for handling data conflicts based on CDC synchronization. The method includes: detecting whether there is a synchronization conflict in the CDC synchronization log; if a synchronization conflict is determined to exist, extracting synchronization conflict-related data from the CDC synchronization log; analyzing the synchronization conflict-related data to determine the conflict type; determining a synchronization conflict resolution component based on the conflict type, wherein the resolution component corresponds one-to-one with the conflict type; and calling the resolution component to handle the synchronization conflict.

[0006] According to one aspect of the embodiments herein, detecting whether there is a synchronization conflict in the CDC synchronization log includes: performing regular expression matching on each line in the CDC synchronization log sequentially using a first keyword; determining whether an incorrect result is matched; if yes, determining that there is a synchronization conflict; if no, determining that there is no synchronization conflict, waiting for a new log to appear in the CDC synchronization log, and detecting whether there is a synchronization conflict in the new log.

[0007] According to one aspect of the embodiments herein, if a synchronization conflict is determined to exist, extracting synchronization conflict-related data from the CDC synchronization log includes: creating a synchronization conflict record table format, the synchronization conflict record table format including a timestamp field, an error code field, and an error description information field; recording time-related strings in the log where the synchronization conflict occurred into the timestamp field of the synchronization conflict record table; determining the string in the log where the synchronization conflict occurred that was matched using a first keyword regular expression; and recording the next string of the string into the error code field of the synchronization conflict record table.

[0008] According to one aspect of the embodiments herein, the extraction of synchronization conflict-related data from the CDC synchronization log further includes: if the error code is a first type of error code, recording the next string matched using the second and third keyword regular expressions from the next string position into the error description information field of the synchronization conflict record table; if the error code is a second type of error code, recording the complete SQL statement matched using the second and fourth keyword regular expressions from the next string position into the error description information field of the synchronization conflict record table.

[0009] According to one aspect of the embodiments herein, the method further includes: recording the three strings following the first string in the log of the synchronization conflict into the prename field, the database name field, and the table name field of the synchronization conflict record table, respectively.

[0010] According to one aspect of the embodiments herein, analyzing the synchronization conflict-related data to determine the conflict type of the synchronization conflict includes: if the error code in the synchronization conflict record table is a first type error code, determining the conflict type corresponding to the first type error code as a table structure conflict; if the error code in the synchronization conflict record table is a second type error code, determining the conflict type corresponding to the second type error code as a data conflict.

[0011] According to one aspect of the embodiments herein, determining the conflict type of a synchronization conflict includes: if the error code in the synchronization conflict record table is a first error code in a first type of error code, the conflict type of the synchronization conflict is determined to be a source table structure conflict; if the error code in the synchronization conflict record table is a second error code in a first type of error code, the conflict type of the synchronization conflict is determined to be a target table structure conflict.

[0012] According to one aspect of the embodiments herein, determining the synchronization conflict resolution component based on the conflict type of the synchronization conflict includes: if the conflict type is a source table structure conflict, determining the corresponding resolution component as a table definition update component; if the conflict type is a target table structure conflict, determining the corresponding resolution component as an original pre-delivery component, an original pre-deletion component, and a correct pre-update component; if the conflict type is a data conflict, determining the corresponding resolution component as a database statement modification component.

[0013] According to one aspect of the embodiments herein, handling synchronization conflicts using the resolution component includes: generating a table definition update script using the table definition update component, updating a pre-defined table definition by running the table definition update script, wherein the table definition update script is determined based on the pre-defined name and table name corresponding to the first error code in the synchronization conflict record; generating a script to import correct pre-defined information using the original pre-defined export component, the original pre-defined deletion component, and the correct pre-defined update component, importing correct pre-defined information by running the correct pre-defined information import script; using a database statement modification component, if the error code is the first error code in the second type of error codes, generating an insert statement based on the table name and error description information, and inserting the insert statement into the target database; using a database statement modification component, if the error code is the second error code in the second type of error codes, generating a delete statement based on the table name and error description information, running the delete statement, and deleting duplicate data in the target database.

[0014] According to one aspect of the embodiments herein, the method further includes: determining the conflict resolution progress based on the handling of the synchronization conflict by the resolution component, and recording it in the progress field of a synchronization conflict record table.

[0015] According to one aspect of the embodiments herein, the analysis of the synchronization conflict log further includes: determining the source database where the synchronization conflict occurred based on the database name field in the synchronization conflict record table; and obtaining the SCN number from the source database, wherein the SCN number is used to record the predetermined interruption time.

[0016] According to one aspect of the embodiments herein, when the progress is complete, the method includes: determining a predetermined startup script based on the SCN number; and executing the predetermined startup script to start the predetermined progress.

[0017] This embodiment provides a data conflict handling device based on CDC synchronization. The device includes: a conflict detection unit for detecting whether a synchronization conflict exists in the CDC synchronization log; a data extraction unit for extracting synchronization conflict-related data from the CDC synchronization log if a synchronization conflict is determined to exist; a conflict type determination unit for analyzing the synchronization conflict-related data to determine the conflict type; a resolution component determination unit for determining a resolution component for the synchronization conflict based on the conflict type, wherein the resolution component corresponds one-to-one with the conflict type; and a processing unit for calling the resolution component to handle the synchronization conflict.

[0018] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data conflict handling method based on CDC synchronization.

[0019] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data conflict handling method based on CDC synchronization.

[0020] This solution analyzes keywords in the CDC synchronization conflict logs to identify the conflicts that cause predetermined errors. It then resolves the corresponding conflicts using several components or methods, and automatically repairs them via scripts. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 The diagram shown is a flowchart of a data conflict handling method based on CDC synchronization, as described in this embodiment.

[0023] Figure 2 The diagram shown is a flowchart of a method for detecting synchronization conflicts in a CDC synchronization log according to an embodiment of this paper.

[0024] Figure 3 The diagram shown is a flowchart of a method for extracting synchronization conflict-related data according to an embodiment of this paper.

[0025] Figure 4 The diagram shown is a flowchart of another method for extracting synchronization conflict-related data in an embodiment of this paper.

[0026] Figure 5 The diagram shown is a flowchart of a method for determining the conflict type of a synchronization conflict according to an embodiment of this paper.

[0027] Figure 6 The diagram shown is a flowchart of another method for determining the conflict type of synchronization conflict in an embodiment of this paper;

[0028] Figure 7 The diagram shown is a flowchart of a method for determining a synchronization conflict resolution component according to an embodiment of this paper.

[0029] Figure 8 The diagram shown is a flowchart of a method for handling synchronization conflicts according to an embodiment of this paper.

[0030] Figure 9 The diagram shown is a flowchart of a method for determining the synchronization interruption time according to an embodiment of this paper;

[0031] Figure 10 The diagram shown is a flowchart of a method for initiating a pre-defined procedure according to an embodiment of this paper.

[0032] Figure 11 The diagram shown is a structural schematic of a data conflict processing device based on CDC synchronization according to an embodiment of this paper.

[0033] Figure 12 The diagram shown is a schematic representation of the specific structure of the data conflict handling device based on CDC synchronization in this embodiment.

[0034] Figure 13 The diagram shown is a structural schematic of a computer device according to an embodiment of this article.

[0035] Explanation of symbols in the attached drawings:

[0036] 1101. Conflict Detection Unit;

[0037] 11011, Regular expression matching module;

[0038] 1102. Data extraction unit;

[0039] 11021. Synchronization Conflict Record Table Creation Module;

[0040] 11022, Recording Module;

[0041] 1103. Conflict Type Determination Unit;

[0042] 11031. Table structure conflict determination module;

[0043] 11032. Data Conflict Determination Module;

[0044] 1104. Solve the component determination unit;

[0045] 1105. Processing Unit;

[0046] 1302. Computer equipment;

[0047] 1304, Processor;

[0048] 1306. Memory;

[0049] 1308. Drive mechanism;

[0050] 1310. Input / output module;

[0051] 1312. Input devices;

[0052] 1314. Output devices;

[0053] 1316. Presentation equipment;

[0054] 1318. Graphical User Interface;

[0055] 1320. Network interface;

[0056] 1322. Communication link;

[0057] 1324. Communication bus. Detailed Implementation

[0058] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments herein will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments herein, and not all of the embodiments. Based on the embodiments herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0059] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0060] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0061] It should be noted that the data conflict handling method based on CDC synchronization presented in this paper can be used in the fields of big data and fintech. This paper does not limit the application areas of the data conflict handling method and device based on CDC synchronization.

[0062] Figure 1 The diagram shown is a flowchart of a data conflict handling method based on CDC synchronization, as described in this embodiment. The method includes the following steps:

[0063] Step 101: Detect synchronization conflicts in the CDC synchronization log. In this application, the CDC tool, as an incremental data synchronization tool based on database logs, is used to achieve data synchronization between two databases. The architecture of the CDC tool includes: a control terminal on one server and instance terminals on two databases. The two databases are a source database and a target database. In the actual use of the CDC tool, a data storage device needs to be established on the instance terminals and a pre-defined device needs to be established on the control terminal. Data synchronization is the process of copying newly added data from some tables in the source database to the corresponding tables in the target database, or copying newly added data from the source database to the target database. A synchronization conflict is a situation where a replication anomaly occurs during the data synchronization process, causing an interruption.

[0064] The CDC synchronization log is used to record predetermined data synchronization statuses. A predetermined status is a set of tables in the source database and tables in the target database, containing data synchronization mapping relationships. Each table includes multiple fields. Data synchronization includes synchronization between the source and target databases, synchronization between tables in the source and target databases, and synchronization between fields in tables in the source and target databases.

[0065] Step 102: If a synchronization conflict is confirmed, extract the relevant data from the CDC synchronization log. In this step, if a conflict occurs during data synchronization, it will lead to a predetermined exception or direct interruption. Conflicts during data synchronization mainly manifest in two types: table structure conflicts and data conflicts. If no table structure conflict or data conflict occurs during data synchronization, no synchronization conflict will occur.

[0066] If a synchronization conflict is detected in the CDC synchronization log, the data related to the synchronization conflict is extracted. In some embodiments of this specification, the extracted synchronization conflict-related data can be recorded in a specific table in a fixed format, or it can be recorded in other forms, such as Word documents or TXT format. This application does not limit the method of recording the synchronization conflict-related data.

[0067] Step 103: Analyze the synchronization conflict-related data to determine the conflict type. In this step, the specific fields in the synchronization conflict-related data are analyzed to determine the conflict type to which the synchronization conflict data belongs.

[0068] Step 104: Determine the synchronization conflict resolution component based on the conflict type of the synchronization conflict. Each resolution component corresponds one-to-one with a specific conflict type. In this step, the resolution component corresponding to a conflict type can be determined based on a pre-defined mapping relationship between conflict types and resolution components. The resolution components in this application are pre-defined based on common causes of synchronization conflicts. Compared to manually analyzing the causes of conflicts and determining corresponding solutions after a synchronization conflict occurs, this approach enables automated error correction, replaces manual operation, and is more efficient in handling synchronization conflicts.

[0069] Step 105: Invoke the resolution component to handle the synchronization conflict. By invoking the corresponding resolution component, a solution method is formed, and the solution method is used to handle the synchronization conflict. In some embodiments of this specification, different resolution components may each correspond to a single synchronization conflict, and multiple resolution components may also be combined to jointly resolve a synchronization conflict.

[0070] Figure 2 The diagram shows a flowchart of a method for detecting synchronization conflicts in a CDC synchronization log according to an embodiment of this paper, which specifically includes the following steps:

[0071] Step 201: Perform regular expression matching on each line of the CDC synchronization log sequentially using the first keyword. In this step, the first keyword is used to perform regular expression matching on each line of log data in the CDC synchronization log. The first keyword for regular expression matching can be a keyword used for error matching, such as errCONFAUD.

[0072] Step 202: Determine whether an incorrect result was matched.

[0073] Step 203: If yes, confirm the existence of a synchronization conflict. If an error result is matched, it indicates that a conflict was detected in the CDC synchronization log. The format of a CDC synchronization log indicating a synchronization conflict can be as follows: 06 / 03 / 23 / 07:37.53.144GCCMCM1408 TS_GCCM PLG_HOUSE_TRADE_STORAGE[CDC][ORACLE JDBC Driver][errCONFAUD]ora-00904invalid identifier.Caused by: An SQL exception occurred.

[0074] Step 204: If not, confirm that there is no synchronization conflict, wait for new logs to appear in the CDC synchronization log, and check whether there is a synchronization conflict in the new logs. If no error result is found, it means that there is no synchronization conflict in the CDC synchronization log. Then continue to wait for new logs to appear in the CDC synchronization log, and use the first keyword regular expression matching method to check whether there is a synchronization conflict in the new logs.

[0075] Figure 3 The diagram shown is a flowchart of a method for extracting synchronization conflict-related data according to an embodiment of this paper, which specifically includes the following steps:

[0076] Step 301: Create a format for the synchronization conflict record table, which includes a timestamp field and an error code field. In this step, a synchronization conflict record table is created to record synchronization conflict-related data extracted from the CDC synchronization log. In addition to the timestamp and error code fields, the synchronization conflict record table may also include error description information fields.

[0077] Step 302: Record the time-related strings from the logs where synchronization conflicts occurred into the timestamp field of the synchronization conflict record table. This step involves storing the extracted synchronization conflict-related data into the synchronization conflict record table. From the logs where a synchronization conflict was detected, extract the time-related strings from the log entries and record these strings in the timestamp field of the synchronization conflict record table to indicate the time when the synchronization conflict occurred.

[0078] Step 303: Determine the string matched using the first keyword regular expression in the log where the synchronization conflict occurred. This is based on the string at the location of the error result obtained from the regular expression matching of the synchronization log using the first keyword in step 201. For example, this location might be the third string in the log where the synchronization conflict occurred.

[0079] Step 304: Record the next string of the given string into the error code field of the synchronization conflict record table. In some embodiments of this specification, the next string can be used to identify the error code of the synchronization conflict. If the position of the given string is the third string in the log where the synchronization conflict occurred, then the fourth string in the log is selected and recorded into the error code field of the synchronization conflict record table. This application extracts key fields from the logs where synchronization conflicts occurred, and organizes and records these key fields to facilitate subsequent matching analysis based on these key fields and further filtering of conflict errors.

[0080] In some other embodiments of this specification, the position of the string used to identify the error code of the synchronization conflict is not necessarily the next string after the string of the error result matched by the first keyword regular expression. In this case, the position of the error code indicating the synchronization conflict can be determined by other methods according to the format of the CDC synchronization log.

[0081] Figure 4 The diagram shows another method for extracting synchronization conflict-related data according to an embodiment of this paper, which specifically includes the following steps:

[0082] Step 401: If the error code is a first-type error code, starting from the next string position, the next string matched using the second and third keyword regular expressions is recorded in the error description information field of the synchronization conflict record table. In some embodiments of this specification, the first-type error code specifically includes four specific error codes. These specific error codes can be error codes directly displayed in the log of the synchronization conflict, or, for example, error codes transformed from those displayed in the log.

[0083] Taking the converted error codes as an example, the first type of error codes includes four types: a, b, c, and d. If the error code in the synchronization conflict record table is determined to be any one of a, b, c, or d, then starting from the next string after that error code, regular expression matching is performed using the second and third keywords. The second keyword is used in the regular expression keyword "{". If the regular expression reaches "{", then starting from that character position, regular expression matching is performed string by string using the second keyword "columns". If a regular expression matching result is obtained, the two strings following the result are recorded in the error description information field of the synchronization conflict record table.

[0084] Step 402: If the error code is a second type of error code, record the complete SQL statement matched by the second and fourth keywords in the error description information field of the synchronization conflict record table from the next string position.

[0085] Corresponding to step 401, taking the converted error code as an example, the second type of error code includes two types: e and f. If the error code in the synchronization conflict record table is determined to be either e or f, then starting from the next string after that error code, regular expression matching is performed using the second and fourth keywords. The second keyword is used in the regular expression keyword "{". If the regular expression reaches "{", then from that character position onwards, the fourth keyword "data" is used to match the SQL command. The matched complete SQL command is recorded in the error description information field of the synchronization conflict record table.

[0086] In some embodiments of this specification, the three strings following the first string in the log of a synchronization conflict are recorded sequentially into the predefined name field, database name field, and table name field of the synchronization conflict record table. Based on the CDC synchronization log format in this application, the three strings following the first string in the log represent the predefined name, database name, and table name, respectively. In other embodiments of this specification, the format of the log of a synchronization conflict is not limited, therefore the positions of the fields representing the predefined name, database name, and table name may differ. This application does not limit the positions of the predefined name field, database name field, and table name field, nor the method of determining these fields.

[0087] Figure 5 The diagram shown is a flowchart of a method for determining the conflict type of a synchronization conflict according to an embodiment of this paper, which specifically includes the following steps:

[0088] Step 501: If the error code in the synchronization conflict record table is a first-type error code, determine that the conflict type corresponding to the first-type error code is a table structure conflict. For example... Figure 4 As described above, if the error code of a certain synchronization conflict record is determined to be any one of a, b, c, d from the error code field of the synchronization conflict record table, the error code is determined to be a first type error code, and the conflict type corresponding to the first type error code can be determined to be a table structure conflict.

[0089] Step 502: If the error code in the synchronization conflict record table is a second type error code, determine that the conflict type corresponding to the second type error code is a data conflict. For example... Figure 4 As described above, if the error code of a certain synchronization conflict record is determined to be either e or f from the error code field of the synchronization conflict record table, the error code is determined to be a second type error code, and the conflict type corresponding to the first type error code can be determined to be a data conflict.

[0090] Figure 6 The diagram shows another method for determining the conflict type of synchronization conflict according to an embodiment of this paper, which specifically includes the following steps:

[0091] Step 601: The error code in the synchronization conflict record table is the first error code in the first type of error codes, and the conflict type of the synchronization conflict is determined to be a source table structure conflict. Further, if the error code of a synchronization conflict record is determined to be either a or b from the error code field of the synchronization conflict record table, this error code is determined to be the first error code in the first type of error codes, and the conflict type corresponding to the first error code can be determined to be a source table structure conflict. Even further, source table structure conflicts include two conflict types: the source end has more fields than the target end (error code a), and the source table fields are expanded (error code b).

[0092] The source end has more fields than the target end: During the scheduled operation, the CDC control end records that the fields registered on the source end are inherently more than those on the target end. If the CDC synchronization tool does not update the data for the extra fields, the scheduled synchronization process will not be interrupted; however, if the CDC synchronization tool updates the fields that are not present on the target end, the scheduled synchronization process will be interrupted.

[0093] Source field expansion: During scheduled operation, a DDL statement suddenly adds a new field to a table in the source database, causing the table structure recorded by the CDC control end to be inconsistent with the actual table structure, thus resulting in an interruption.

[0094] Step 602: The error code in the synchronization conflict record table is the second error code in the first type of error codes, determining the conflict type of the synchronization conflict as a target table structure conflict. Further, if the error code of a synchronization conflict record is determined to be either c or d from the error code field of the synchronization conflict record table, this error code is determined to be the second error code in the first type of error codes, and the conflict type corresponding to the second error code can be determined to be a target table structure conflict. Even further, the target table structure conflict includes two conflict types: the target end has more fields than the source end (error code c), and the target end's fields are expanded (error code d).

[0095] The phrase "the target has more fields than the source" means that during the scheduled operation, the target end has more fields registered in the CDC control terminal record than the source end (the reason for this scenario is currently unknown). Fields without mapping relationships will be written as null. If the target table does not restrict the table fields, no error will be reported. However, if the extra fields in the target table are restricted to not being null, the synchronization will be interrupted.

[0096] The target-side field expansion is as follows: During the scheduled operation, the target-side table suddenly underwent a DDL statement to add a field, causing the target table structure recorded by the CDC control end to be inconsistent with the actual table structure, resulting in an interruption.

[0097] Figure 7 The diagram shown is a flowchart of a method for determining a synchronization conflict resolution component according to an embodiment of this paper, which specifically includes the following steps:

[0098] Step 701: If the conflict type is a source table structure conflict, determine the corresponding resolution component as the table definition update component. In this step, the error code is obtained based on the value recorded in the error code field of the synchronization conflict record table. If the error code is determined to be the first error code of the first error type, the resolution component corresponding to the first error code can be determined as the table definition update component based on the pre-set mapping relationship between error codes and resolution components. The table definition update component is used to update the structure of the source table.

[0099] In this application, the mapping relationship record includes, but is not limited to: the mapping relationship between error codes and resolution components, and the mapping relationship between error codes and resolution methods. A resolution method consists of one or more resolution components. If the mapping relationship is between error codes and resolution methods, in some other embodiments of this specification, after determining that the error code is a first error code, the resolution method corresponding to the first error code can be determined as a first resolution method or resolution method A based on the error code and resolution method recorded in the mapping relationship. The first resolution method or resolution method A is composed of a resolution component table definition update component.

[0100] Step 702: If the conflict type is a target table structure conflict, determine the corresponding resolution components as the original scheduled export component, the original scheduled deletion component, and the correct scheduled update component. In this step, the error code is obtained based on the value recorded in the error code field of the synchronization conflict record table. If the error code is determined to be the second error code of the first error type, the resolution components corresponding to the second error code can be determined as the original scheduled export component, the original scheduled deletion component, and the correct scheduled update component based on the pre-set mapping relationship between error codes and resolution components. Specifically, the original scheduled export component is used to export the original erroneous scheduled mapping information, which is an error description of the original table structure; the original scheduled deletion component is used to delete the original erroneous scheduled reservations; and the correct scheduled update component is used to import the correct scheduled reservations.

[0101] In some other embodiments of this specification, after determining that the error code is the second error code, the solution corresponding to the second error code can be determined as the second solution or solution B according to the error code and solution method recorded in the mapping relationship. The second solution or solution B consists of the original predetermined export component, the original predetermined deletion component, and the correct predetermined update component.

[0102] Step 703: If the conflict type is a data conflict, determine the corresponding resolution component as a database statement modification component. Data conflicts include two types: target data non-existence and target primary key conflict.

[0103] The "target data does not exist" condition means that during the scheduled operation, the source end performed an update statement, but this data does not exist on the target end, which prevents modification of the current data and interrupts data synchronization, further leading to synchronization interruption.

[0104] A primary key conflict occurs when, during scheduled execution, an INSERT statement is executed on the source side, but the data to be inserted already exists in the target table, and the target table has a primary key constraint, causing the insertion to fail and thus leading to a synchronization interruption.

[0105] Corresponding to the two types of data conflicts mentioned above, database modification statements include insert statements and delete statements. This application pre-determines the corresponding resolution components based on the conflict type of the synchronization conflict. Compared to manually analyzing the cause of the conflict and determining the corresponding solution after a synchronization conflict occurs, this approach enables automated error repair, replaces manual operation, and is more efficient in handling synchronization conflicts.

[0106] Figure 8 The diagram shown is a flowchart of a method for handling synchronization conflicts according to an embodiment of this paper, which specifically includes the following steps:

[0107] Step 801: Generate a table definition update script using the table definition update component. Update the pre-defined table definitions by running the table definition update script. The table definition update script is determined based on the pre-defined name and table name corresponding to the first error code in the synchronization conflict record. This step corresponds to step 701. After determining that the component for resolving source table structure conflicts is the table definition update component, the corresponding pre-defined name and table name are obtained from the pre-defined name and table name fields in the synchronization conflict record table. Based on the pre-defined name and table name, a chcclp script usable by the CDC tool is constructed, and the pre-defined table definitions are updated by running the script.

[0108] Step 802: Using the original scheduled export component, the original scheduled deletion component, and the correct scheduled update component, generate a script to import the correct scheduled information. By running the script to import the correct scheduled information, the correct scheduled information is imported.

[0109] This step corresponds to step 702. After the component for resolving the target table structure conflict is determined, the original pre-defined export component obtains the pre-defined name from the synchronization conflict record table, concatenates the chcclp script usable by the CDC tool, and exports the CDC script file by running the script.

[0110] The original pre-defined deletion component first uses regular expressions to match the database name in the CDC synchronization log where the synchronization conflict occurred. From there, it continues matching to the table name, and then continues matching to the first string of the error description message. This component then constructs a chcclp script usable by the CDC tool. Running the script deletes the content of the line containing the first string of the error description message.

[0111] In some embodiments of this specification, the correct reservation update component includes: a reservation information modification component and a correct reservation import component. The reservation information modification component writes content reservation setting statements according to CDC script syntax rules and in conjunction with error description information, saves the changes, and exits.

[0112] The pre-order import component retrieves the pre-order name from the pre-order name field of the synchronization conflict record table, concatenates it to form a chcclp script usable by the CDC, and then runs the script to import the correct pre-order information.

[0113] Step 803: Using the database statement modification component, if the error code is the first error code in the second type of error code, an insert statement is generated based on the table name and error description information, and the insert statement is inserted into the target database. This step obtains the error code from the synchronization conflict record table. If the error code is determined to be the first error code of the second error type, it indicates that during the scheduled operation, the target table suddenly underwent a DDL statement to add a field, causing inconsistency between the target table structure recorded on the CDC control end and the actual table structure, resulting in an interruption. Therefore, this step connects to the database based on the database name, writes an insert statement based on the table name and error description information, generates the insert SQL statement, and runs the insert SQL statement to insert into the target database, ensuring consistency between the target table structure recorded on the CDC control end and the actual table structure.

[0114] In some other embodiments of this specification, after determining that the error code is the first error code in the second type of error codes, the solution corresponding to the first error code in the second type of error codes can be determined as the third solution or solution C according to the error code and solution method recorded in the mapping relationship. The third solution or solution C is: connecting to the database according to the database name and writing an insert statement according to the table name and error description information.

[0115] Step 804: Modify the component using database statements. If the error code is the second error code of the second type of error code, generate a deletion statement based on the table name and error description information, and run the deletion statement to delete duplicate data in the target database. This step obtains the error code in the synchronization conflict record table. If the error code is determined to be the second error code of the second error type, and the conflict type is determined to be a target primary key conflict, it indicates that an insert statement was performed on the source end during the scheduled operation. The inserted data already exists in the target table, and the target table has a primary key constraint, causing the insert to fail and synchronization to be interrupted. Therefore, this step writes a deletion statement based on the table name and error description information, generates the deletion SQL statement for this deletion statement, and deletes the corresponding data in the target database by running the deletion SQL statement.

[0116] In other embodiments of this specification, after determining that the error code is the second error code in the second type of error codes, the solution corresponding to the second error code in the second type of error codes can be determined as the fourth solution or solution D according to the error codes and solutions recorded in the mapping relationship. The fourth solution or solution D is: writing a deletion statement based on the table name and error description information, and generating the deletion SQL statement. In this application, the corresponding solution component is determined in advance according to the conflict type of the synchronization conflict. Compared with manually analyzing the cause of the conflict and determining the corresponding solution after the synchronization conflict occurs, this can achieve automated error repair, replace manual operation, and improve the efficiency of handling synchronization conflicts.

[0117] In some embodiments of this specification, the method further includes: determining the conflict processing progress based on the handling of the synchronization conflict by the resolution component, and recording it in the progress field of the synchronization conflict record table. In this application, each resolution component has the function of updating the processing progress. The processing progress can take multiple values, including 0, 1, and p. When the processing progress is 0, it indicates no processing has been performed; when the processing progress is 1, it indicates processing is complete; and when the processing progress is p, it indicates processing is in progress. Each resolution component updates its processing progress to p when it begins processing the corresponding conflict type; after processing the conflict type, it updates its processing progress to 1. In this application, based on the processing progress provided by each component, the current processing progress of the synchronization conflict can be clearly known, which is beneficial for the sequential processing between resolution components or for facilitating the synchronous execution of conflict processing between resolution components that do not interfere with each other. Furthermore, after all resolution components have completed processing, a prompt is given to the starting component to perform breakpoint recovery and restart, which helps improve the efficiency of synchronization conflict processing.

[0118] Figure 9 The diagram shown is a flowchart of a method for determining the synchronization interruption time according to an embodiment of this paper, which specifically includes the following steps:

[0119] Step 901: Determine the source database where the synchronization conflict occurred based on the database name field in the synchronization conflict record table. Specifically, determine which source database experienced the synchronization conflict based on the database name in the synchronization conflict record table.

[0120] Step 902: Obtain the SCN number from the source database. The SCN number is used to record the scheduled interruption time. In this step, timestamp information is obtained from the source database and converted into an SCN number. The SCN number is an identifier in the database used to represent data changes and is used to record the scheduled interruption time.

[0121] Figure 10 The diagram shown is a flowchart of a method for initiating a predetermined process according to an embodiment of this paper, which specifically includes the following steps:

[0122] Step 1001: Determine the pre-defined startup script based on the SCN number. In this step, after the processing progress of all the called resolution components in the previous steps is 1, breakpoint recovery is performed based on the SCN number, and the CDC-usable chcclp script is constructed based on the pre-defined name in the synchronization conflict record table.

[0123] Step 1002: Execute the predetermined startup script to start the scheduled task. By running the predetermined startup script, the scheduled task is restarted, ensuring that data is not lost after the interruption.

[0124] like Figure 11 The diagram shown is a structural schematic of a data conflict processing device based on CDC synchronization according to an embodiment of this paper. The basic structure of the device is illustrated in the diagram. The functional units and modules can be implemented in software, or using general-purpose chips or specific chips to implement data conflict processing based on CDC synchronization. The device specifically includes:

[0125] The conflict detection unit 1101 is used to detect whether there is a synchronization conflict in the CDC synchronization log;

[0126] Data extraction unit 1102 is used to extract synchronization conflict related data from the CDC synchronization log if a synchronization conflict is determined to exist;

[0127] The conflict type determination unit 1103 is used to analyze the synchronization conflict related data to determine the conflict type of the synchronization conflict;

[0128] The component determination unit 1104 is used to determine the component for resolving the synchronization conflict according to the conflict type of the synchronization conflict, wherein the component for resolving the synchronization conflict corresponds one-to-one with the conflict type of the synchronization conflict.

[0129] Processing unit 1105 is used to call the resolution component to handle synchronization conflicts.

[0130] This solution uses keyword matching analysis of CDC synchronization conflict logs to identify conflicts that cause pre-defined errors. Several resolution components or methods are then used to resolve these conflicts, and the repair process is automated via scripts. By recording the SCN number, the pre-defined replication is restarted using the SCN at the time of the interruption, ensuring that no data is lost after the interruption.

[0131] As one embodiment of this article, reference may also be made to, for example, Figure 12 The diagram shown is a schematic representation of the specific structure of the data conflict handling device based on CDC synchronization in this embodiment.

[0132] As one embodiment of this document, the collision detection unit 1101 further includes:

[0133] The regular expression matching module 11011 is used to perform regular expression matching on each line in the CDC synchronization log in turn using the first keyword;

[0134] As one embodiment of this document, the data extraction unit 1102 further includes:

[0135] Synchronization conflict record table creation module 11021 is used to create the format of the synchronization conflict record table;

[0136] The recording module 11022 is used to record data into the timestamp field and error code field of the synchronization conflict record table, respectively;

[0137] As one embodiment of this document, the conflict type determination unit 1103 further includes:

[0138] Table structure conflict determination module 11031 is used to determine that the conflict type corresponding to the first type of error code is a table structure conflict.

[0139] The data conflict determination module 11032 is used to determine that the conflict type corresponding to the second type of error code is a data conflict.

[0140] like Figure 13As shown in this embodiment, a computer device 1302 may include one or more processors 1304, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 1302 may also include any memory 1306 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 1306 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1302. In one case, when the processor 1304 executes associated instructions stored in any memory or combination of memories, the computer device 1302 can perform any operation of the associated instructions. The computer device 1302 also includes one or more drive mechanisms 1308 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0141] Computer device 1302 may also include an input / output module 1310 (I / O) for receiving various inputs (via input device 1312) and providing various outputs (via output device 1314). A specific output mechanism may include a presentation device 1316 and an associated graphical user interface (GUI) 1318. In other embodiments, the input / output module 1310 (I / O), input device 1312, and output device 1314 may be omitted, and the device may function solely as a computer device within a network. Computer device 1302 may also include one or more network interfaces 1320 for exchanging data with other devices via one or more communication links 1322. One or more communication buses 1324 couple the components described above together.

[0142] Communication link 1322 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 1322 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0143] Corresponding to Figures 1 to 10 In addition to the methods described above, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described methods.

[0144] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the following: Figures 1 to 10 The method shown.

[0145] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0146] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0147] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0148] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0149] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0151] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0153] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. A data conflict handling method based on CDC synchronization, characterized in that, The method includes: Check the CDC synchronization log for synchronization conflicts; If a synchronization conflict is confirmed, extract the relevant data from the CDC synchronization log; Analyze the synchronization conflict-related data to determine the conflict type. The synchronization conflict resolution component is determined according to the conflict type of the synchronization conflict, and the resolution component corresponds one-to-one with the conflict type of the synchronization conflict; The aforementioned resolution component is invoked to handle synchronization conflicts; Detecting synchronization conflicts in the CDC synchronization log includes: The first keyword is used to perform regular expression matching on each line in the CDC synchronization log; Determine if an incorrect result was found; If so, a synchronization conflict has been confirmed; If not, confirm that there is no synchronization conflict, wait for new logs to appear in the CDC synchronization log, and check whether there is a synchronization conflict in the new logs; If a synchronization conflict is confirmed, the synchronization conflict-related data extracted from the CDC synchronization log includes: If the error code is a Type 1 error code, the next string matched by the second and third keywords using regular expressions will be recorded in the error description information field of the synchronization conflict record table, starting from the next string position. If the error code is a second type of error code, the complete SQL statement matched by the second and fourth keywords using regular expressions from the next string position is recorded in the error description information field of the synchronization conflict record table.

2. The data conflict handling method based on CDC synchronization according to claim 1, characterized in that, If a synchronization conflict is confirmed, extracting synchronization conflict-related data from the CDC synchronization log also includes: The format for creating a synchronization conflict record table includes a timestamp field, an error code field, and an error description information field. Record time-related strings from the logs of synchronization conflicts into the timestamp field of the synchronization conflict record table; Identify the string matched by the first keyword regular expression in the log where the synchronization conflict occurred; Record the next string of the given string in the error code field of the synchronization conflict record table.

3. The data conflict handling method based on CDC synchronization according to claim 1, characterized in that, The method further includes: Record the three strings following the first string in the log of the synchronization conflict into the prename field, database name field, and table name field of the synchronization conflict record table, respectively.

4. The data conflict handling method based on CDC synchronization according to claim 3, characterized in that, Analyzing the synchronization conflict-related data, the conflict types are determined to include: If the error code in the synchronization conflict record table is a first type error code, the conflict type corresponding to the first type error code is determined to be a table structure conflict. If the error code in the synchronization conflict record table is a second type error code, the conflict type corresponding to the second type error code is determined to be a data conflict.

5. The data conflict handling method based on CDC synchronization according to claim 4, characterized in that, The types of collisions that can be identified as synchronization conflicts include: If the error code in the synchronization conflict record table is the first error code in the first type of error code, the conflict type of the synchronization conflict is determined to be a source table structure conflict; If the error code in the synchronization conflict record table is the second error code in the first type of error code, the conflict type of the synchronization conflict is determined to be a target table structure conflict.

6. The data conflict handling method based on CDC synchronization according to claim 5, characterized in that, The components for resolving synchronization conflicts, determined based on the conflict type, include: If the conflict type is a source table structure conflict, the corresponding resolution component is determined to be the table definition update component; If the conflict type is a target table structure conflict, the corresponding resolution components are determined to be the original pre-delivery component, the original pre-deletion component, and the correct pre-update component; If the conflict type is a data conflict, the corresponding resolution component is determined to be the database statement modification component.

7. The data conflict handling method based on CDC synchronization according to claim 6, characterized in that, Using the aforementioned resolution component, handling synchronization conflicts includes: The table definition update component is used to generate a table definition update script, and the table definition in the plan is updated by running the table definition update script. The table definition update script is determined according to the plan name and table name corresponding to the first error code in the synchronization conflict record. Using the existing pre-order export component, the existing pre-order deletion component, and the correct pre-order update component, a script for importing correct pre-order information is generated. By running the correct pre-order information import script, the correct pre-order information is imported. The component is modified using database statements. If the error code is the first error code in the second type of error code, an insert statement is generated based on the table name and error description information and inserted into the target database. The component is modified using database statements. If the error code is the second error code in the second type of error code, a deletion statement is generated based on the table name and error description information. The deletion statement is then executed to delete duplicate data in the target database.

8. The data conflict handling method based on CDC synchronization according to claim 7, characterized in that, The method further includes: Based on the handling of the synchronization conflict by the resolution component, the conflict resolution progress is determined and recorded in the progress field of the synchronization conflict record table.

9. The data conflict handling method based on CDC synchronization according to claim 8, characterized in that, Analysis of synchronization conflict logs also includes: Based on the database name field in the synchronization conflict record table, determine the source database where the synchronization conflict occurred; The SCN number is obtained from the source database. The SCN number is used to record the scheduled interruption time.

10. The data conflict handling method based on CDC synchronization according to claim 9, characterized in that, When the progress is completed, the method includes: The pre-defined startup script is determined based on the SCN number; The predetermined startup script is executed to start the predetermined process.

11. A data conflict handling device based on CDC synchronization, characterized in that, The device includes: The conflict detection unit is used to detect whether there are synchronization conflicts in the CDC synchronization log; The data extraction unit is used to extract synchronization conflict-related data from the CDC synchronization log if a synchronization conflict is determined to exist. A conflict type determination unit is used to analyze the synchronization conflict-related data to determine the conflict type of the synchronization conflict; The component determination unit is used to determine the component for resolving the synchronization conflict according to the conflict type of the synchronization conflict, wherein the component for resolving the synchronization conflict corresponds one-to-one with the conflict type of the synchronization conflict. The processing unit is used to invoke the resolution component to handle synchronization conflicts; The collision detection unit includes: The regular expression matching module is used to perform regular expression matching on each line in the CDC synchronization log in turn using the first keyword; The conflict detection unit is specifically used to determine whether an incorrect result is matched; if so, it determines that a synchronization conflict exists; if not, it determines that no synchronization conflict exists, waits for a new log to appear in the CDC synchronization log, and detects whether a synchronization conflict exists in the new log. The data extraction unit is specifically used to, if the error code is a first type of error code, record the next string matched by the second and third keywords using regular expressions from the next string position into the error description information field of the synchronization conflict record table; if the error code is a second type of error code, record the complete SQL statement matched by the second and fourth keywords using regular expressions from the next string position into the error description information field of the synchronization conflict record table.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images