Data synchronization conflict processing method and device, equipment, storage medium and program product

Abstract

The invention provides a data synchronization conflict processing method and device, equipment, a storage medium and a program product, and relates to the field of data processing. The method comprises the following steps: generating a conflict processing strategy by analyzing conflict processing rule configuration information; according to a comparison result of a current value correspondingly recorded in a target end database and a previous item value in the change data, determining a conflict type; based on the conflict processing strategy and the conflict type, data processing operation is executed, and the data processing operation comprises coverage updating, conflict ignoring or abnormal recording. The method is used for improving the accuracy of conflict processing.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus, device, storage medium, and program product for handling data synchronization conflicts. Background Technology

[0002] In distributed systems requiring real-time data synchronization, such as master-slave architectures in enterprise databases or cross-regional data replication, ensuring data consistency is crucial. Conflicts can lead to task interruptions and exceptions. Therefore, knowing how to handle conflicts during data synchronization is paramount.

[0003] Currently, existing methods for handling data synchronization conflicts mainly rely on passive detection and hard skipping mechanisms. This method only uses Structured Query Language (SQL) to identify conflicts by error. When a conflict is detected, the SQL statement that caused the error is skipped and logged, awaiting manual intervention.

[0004] However, existing data synchronization conflict handling methods suffer from poor accuracy due to passive detection and hard skipping. Summary of the Invention

[0005] This application provides a data synchronization conflict handling method, apparatus, device, storage medium, and program product to improve the accuracy of conflict handling.

[0006] Firstly, this application provides a method for handling data synchronization conflicts, including:

[0007] Conflict handling strategies are generated by parsing the conflict handling rule configuration information;

[0008] Based on the comparison between the current value of the corresponding record in the target database and the previous value in the changed data, the conflict type is determined.

[0009] Based on the conflict handling strategy and the conflict type, data processing operations are performed, including overwrite updates, ignoring conflicts, or recording anomalies.

[0010] In one possible implementation, generating a conflict handling strategy by parsing conflict handling rule configuration information includes:

[0011] By parsing the conflict handling rule configuration information, we can obtain the table name, operation type, comparison columns, and handling method;

[0012] A conflict resolution strategy is generated based on the table name, the operation type, the comparison column, and the processing method.

[0013] In one possible implementation, the step of performing data processing operations via a rules engine based on the conflict handling strategy and the conflict type includes:

[0014] Match the conflict handling strategy according to the conflict type to obtain the handling result;

[0015] Based on the processing results, perform operations such as overwrite update, ignore conflict, or log anomalies;

[0016] The conflict data, the conflict type, and the conflict handling strategy are recorded in the audit log file.

[0017] In one possible implementation, the step of performing an overwrite update, ignoring conflicts, or logging anomalies based on the processing result includes:

[0018] Based on the processing results, the conflict resolution task is assigned to multiple processing nodes;

[0019] The multiple processing nodes perform operations such as overlay updates, ignoring conflicts, or logging anomalies in parallel.

[0020] In one possible implementation, before parsing the conflict handling rule configuration information, the method further includes:

[0021] The priority of the conflict handling rules is dynamically adjusted based on business priorities.

[0022] In one possible implementation, the step of parsing the conflict handling rule configuration information includes at least one of the following:

[0023] The conflict handling rules are updated by parsing database-level rules, table-level rules, and field-level rules;

[0024] The historical conflict data is analyzed using a machine learning model, and the configuration information of the conflict resolution and handling rules is adjusted accordingly.

[0025] In one possible implementation, before obtaining the current value of the corresponding record in the target database, the previous value of the source-side changed data is prefetched and cached in local storage.

[0026] Secondly, this application provides a data synchronization conflict handling apparatus, comprising:

[0027] The generation module is used to generate conflict handling strategies by parsing the conflict handling rule configuration information;

[0028] The determination module is used to determine the conflict type based on the comparison between the current value of the corresponding record in the target database and the previous value in the changed data;

[0029] The execution module is used to perform data processing operations through the rule engine based on the conflict handling strategy and the conflict type. The data processing operations include overwrite updates, ignoring conflicts, or recording anomalies.

[0030] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0031] The memory stores computer-executed instructions;

[0032] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in any of the first aspects above.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects above.

[0034] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0035] This application provides a data synchronization conflict handling method, apparatus, device, storage medium, and program product. By parsing conflict handling rule configuration information, a conflict handling strategy can be generated, laying the foundation for subsequent data processing operations. Based on the comparison between the current value of the corresponding record in the target database and the previous value in the changed data, the conflict type can be determined, enabling full-type detection and flexible handling of data synchronization conflicts. Based on the conflict handling strategy and the conflict type, data processing operations are performed, including overwrite updates, ignoring conflicts, or recording anomalies, improving the accuracy of conflict handling. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] Figure 1 A flowchart illustrating a data synchronization conflict handling method provided in this application;

[0038] Figure 2 A flowchart illustrating another data synchronization conflict handling method provided in this application;

[0039] Figure 3 A schematic diagram of a data synchronization conflict handling device provided in this application;

[0040] Figure 4This is a schematic diagram of the structure of an electronic device provided in this application.

[0041] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0043] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0045] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0046] First, let's explain some of the terms used in this application:

[0047] A rules engine is a system that processes data according to the rules configured in a configuration file to generate expected results.

[0048] Previous value: refers to the data value before the data was changed.

[0049] In distributed systems requiring real-time data synchronization, such as master-slave architectures in enterprise databases or cross-regional data replication, ensuring data consistency is crucial. Conflicts can lead to task interruptions and exceptions. Therefore, knowing how to handle conflicts during data synchronization is paramount.

[0050] Currently, existing methods for handling data synchronization conflicts mainly rely on passive detection and hard skipping mechanisms. This method only uses Structured Query Language (SQL) to identify conflicts by error. When a conflict is detected, the SQL statement that caused the error is skipped and logged, awaiting manual intervention.

[0051] However, existing data synchronization conflict handling methods suffer from poor accuracy due to passive detection and hard skipping. Considering these problems, this application proposes a data synchronization conflict handling method that combines proactive detection of preceding values ​​with dynamic processing by a rule engine, thereby improving the accuracy of conflict handling.

[0052] The entity executing this data synchronization conflict handling method can be, for example, a processing system. Optionally, the processing system can be any existing electronic device with processing capabilities, such as a terminal or a server. In some embodiments, the processing system can also be deployed in a server cluster or cloud environment. This application does not limit the deployment environment of the processing system.

[0053] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0054] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0055] Figure 1 This is a flowchart illustrating a data synchronization conflict handling method provided in this application. Figure 1 As shown, the method includes:

[0056] S101 generates a conflict handling strategy by parsing the conflict handling rule configuration information.

[0057] In some embodiments, the processing system can parse the conflict handling rule configuration information to obtain the table name, operation type, comparison columns, and processing method. Then, based on the table name, operation type, comparison columns, and processing method, the system generates a conflict handling strategy.

[0058] For example, conflict handling rule configuration information can be a pre-defined set of structured information used to define data conflict handling logic.

[0059] Optionally, the processing system can use a structured parsing algorithm to perform syntax validation and field extraction on the conflict handling rule configuration information. It can locate core parameters such as table name, operation type, comparison columns, and processing method through keyword matching, while also validating the format of these parameters. For example, the processing system can verify whether the table name conforms to database naming conventions, whether the operation type belongs to a preset enumeration value, and whether the comparison columns exist in the target table structure.

[0060] Optionally, the processing system can generate a mapping conflict handling strategy for a single configuration information entry. For example, if the table name is A, the operation type is B, or the comparison column is C, the corresponding handling method is D. The processing system can also generate a set of conflict handling strategies by parsing multiple configuration information entries and creating an index between table names and operation types. Based on this set of conflict handling strategies, the processing system can create a memory-mapped table and store it using a key-value pair structure. Alternatively, the conflict handling strategies can be imported into the rule engine for storage.

[0061] The above method parses the conflict handling rule configuration information to obtain the table name, operation type, comparison columns, and processing method, providing data basis for generating a conflict handling strategy. Based on this table name, operation type, comparison columns, and processing method, a conflict handling strategy is generated to improve data consistency.

[0062] As one possible implementation, before parsing the conflict handling rule configuration information, the processing system can dynamically adjust the priority of the conflict handling rule based on business priorities.

[0063] For example, business priorities can be divided into priorities based on different business scenarios.

[0064] Optionally, the processing system can preset the priority of each service through a distributed configuration center, extract service identifiers from the service request header or the service tag field of the database table and associate them with a preset priority mapping table, and calculate service priorities based on metrics such as the number of queries per second and response latency.

[0065] Optionally, the processing system can uniformly convert business priorities from different sources into quantitative indicators, and assign a corresponding weight coefficient to each business priority. By calculating the weight associated with business priorities and business rules, the priority of conflict handling rules is obtained.

[0066] Optionally, when multiple services correspond to the same conflict handling rule, the processing system can take the highest service priority as the rule priority; when the same service corresponds to multiple rules, the processing system can allocate priorities according to the service contribution of the preset rule.

[0067] Optionally, the processing system can synchronize the updated conflict handling rule priority to the conflict handling rule configuration information storage module, and add a version identifier. For example, the version can be represented by a timestamp and a version number.

[0068] Optionally, the processing system can adjust the priority of the conflict handling rule at a fixed period, or adjust the priority of the conflict handling rule when the business priority is updated or the conflict handling scenario is switched.

[0069] By using the above method, before parsing the conflict handling rule configuration information, the priority of the conflict handling rule is dynamically adjusted according to the business priority, thereby improving the accuracy of conflict handling.

[0070] As one possible implementation, after parsing the conflict handling rule configuration information, the processing system can update the conflict handling rule by parsing database-level rules, table-level rules, and field-level rules. The processing system can also utilize machine learning models to analyze historical conflict data and adjust the parsed conflict handling rule configuration information.

[0071] For example, database-level rules can be conflict resolution rules that apply globally to the entire database.

[0072] For example, table-level rules can be conflict handling rules that apply to a specific database table.

[0073] For example, field-level rules can be conflict resolution rules that apply to specific fields in a particular table.

[0074] Optionally, the processing system can extract database-level rules, table-level rules, and field-level rules based on configuration files, distributed configuration centers, or database metadata, and obtain the table name, operation type, comparison columns, and processing method corresponding to the above rules through parsing tools.

[0075] Optionally, the processing system can preset the priorities of database-level rules, table-level rules, and field-level rules. For example, field-level rules have higher priority than table-level rules, and table-level rules have higher priority than database-level rules. Higher-priority rules can override lower-priority rules; for example, field-level rules can override table-level rules of the same table and operation type.

[0076] Optionally, the processing system can update only the level 3 rule conflicts or newly added rules based on the above rules, retain the unchanged rules, or regenerate a complete set of conflict handling rules and synchronize the updated rules to the memory mapping table or rule engine.

[0077] Optionally, the processing system can collect historical conflict data based on historical log files, input the table names, operation types, comparison columns, and conflict characteristics from the historical conflict data into the machine learning model, and output the optimal processing method. It can also periodically update the machine learning model with newly added historical conflict data. The processing system can verify the validity of the results output by the machine learning model, and automatically update the conflict processing rule configuration information upon successful verification.

[0078] By parsing the conflict handling rule configuration information using the above method, and then updating the conflict handling rules by parsing database-level rules, table-level rules, and field-level rules, the accuracy of conflict handling is improved. Analyzing historical conflict data using machine learning models allows for adjustments to the parsed conflict handling rule configuration information, enhancing the rules' adaptability and thus increasing the success rate of conflict handling.

[0079] S102, Based on the comparison result between the current value of the corresponding record in the target database and the previous value in the changed data, determine the conflict type.

[0080] As one possible implementation, before obtaining the current value of the corresponding record in the target database, the processing system can prefetch the previous value of the changed data from the source and cache it in local storage.

[0081] Optionally, the processing system can detect changes in the source database logs in real time. When a change log is detected in the source database, it immediately triggers the prefetching of the preceding values ​​of the changed data from the source. It can also determine conflicts and verify data consistency based on preset trigger conditions, and then execute the prefetching of the preceding values ​​of the changed data from the source.

[0082] Optionally, the processing system can parse the source change log and extract the preceding value before the change based on the change log. If the preceding value is not recorded in the change log, the original value before the change can be obtained by calling the source database query interface through the primary key or unique index of the changed data. The processing system can verify the non-nullability and field integrity of the preceding value. For example, it can check whether the number of fields is consistent with the changed data. If the verification fails, a retry extraction is triggered. If the verification fails after 3 extraction attempts, the processing system can issue a warning and generate an exception log.

[0083] Optionally, the processing system can synchronously write the preceding values ​​of source-side changed data in core business scenarios to a local memory cache for storage, and asynchronously write the preceding values ​​of source-side changed data in non-core business scenarios to a local disk for storage via a message queue. The processing system can also synchronously record the pre-fetch timestamp, expiration time, and change operation type when caching the aforementioned preceding values.

[0084] Optionally, the processing system can also set an expiration time for cached preceding values, automatically deleting them after expiration; if a preceding value has completed the target-side comparison process, the corresponding cache will be deleted immediately after the comparison is completed. If the same record on the source side undergoes multiple changes, the processing system can re-fetch the latest preceding value and overwrite the old value in the cache. When the memory cache reaches its capacity threshold, the processing system can evict low-frequency cached data based on the least recently used or least frequently used strategy, prioritizing the retention of preceding values ​​for high-frequency changed records.

[0085] By using the above method, before obtaining the current value of the corresponding record in the target database, the previous value of the changed data from the source can be prefetched and cached in local storage, thereby improving the efficiency of conflict handling.

[0086] S103, based on the conflict handling strategy and the conflict type, perform data processing operations, including overwrite updates, ignoring conflicts, or recording anomalies.

[0087] As one possible implementation, the processing system can match a conflict handling strategy based on the conflict type to obtain a processing result. Then, based on this result, it can perform an overwrite update, ignore the conflict, or log anomalies. Finally, the conflict data, the conflict type, and the conflict handling strategy are recorded in an audit log file.

[0088] For example, the conflict type can be a classification identifier defined based on the characteristics of the data operation conflict scenario. For instance, the conflict type could be a primary key duplicate conflict, a version number inconsistency conflict, or a field value conflict, etc.

[0089] Optionally, the processing system can identify conflict types by comparing column values ​​and analyzing operation scenarios. For example, if a setting operation on the source end triggers a duplicate primary key field on the target end, the processing system can determine that it is a primary key duplication conflict; if the download operations on the source and target ends modify the same field of the same record with different values, the processing system can determine that it is a field value conflict.

[0090] Optionally, the processing system can predefine rule-policy mapping tables for conflict types, table names, and operation types. Based on the conflict type, the system queries these rule-policy mapping tables through the rule engine to match conflict handling policies. If a match is successful, the processing system can output "processing successful" and a policy identifier; if a match fails, the system can output "processing failed" and the reason for the exception.

[0091] Optionally, the processing system can determine the conflict handling rules in the conflict handling strategy based on the strategy identifier in the output results. It can compare the timestamps or version numbers of the source and target ends and retain the latest data. It can also overwrite the corresponding field values ​​on the target end with the changed data from the source end, completing the overwrite update operation. After the update, the processing system can also verify whether the data on the target end and the data on the source end are consistent. If they are inconsistent, a retry update is triggered. If three retries fail, an exception is recorded.

[0092] Optionally, the processing system can avoid modifying the original data on the target end, only recording the conflict scenario and ignoring the conflict data. The processing system can record conflict data, conflict type, matching failure strategy conditions, exception timestamp, and business association information, write the exception information to a database table or message queue, and trigger an alarm notification.

[0093] Optionally, the processing system can obtain conflict data, conflict type, and conflict handling strategy information based on database tables or message queues, and asynchronously write the above information to the audit log file in a standardized format, while setting the information retention period.

[0094] By matching conflict handling strategies according to the conflict type using the above method, processing results can be obtained, improving conflict handling efficiency. Based on the processing results, overwrite updates, conflict ignoring, or abnormal operations can be performed, reducing manual intervention and improving the accuracy of conflict handling. Conflict data, the conflict type, and the conflict handling strategy are recorded in the audit log file, improving system maintainability.

[0095] As one possible implementation, the processing system can distribute the conflict resolution task to multiple processing nodes based on the processing result. Then, operations such as overwrite updates, ignoring conflicts, or logging anomalies can be performed in parallel on these multiple processing nodes.

[0096] Optionally, the processing system can distribute conflict resolution tasks to multiple nodes using hash sharding. The hash key is the unique identifier of the task and the target table name. The hash value is calculated and then modulo the result with the number of processing nodes to assign the conflict resolution task to the corresponding node. For example, if there are 3 processing nodes and the hash value is 4, the modulo result is 1, so the conflict resolution task is assigned to processing node 1.

[0097] Optionally, the processing system can also monitor the load status of each processing node in real time and assign new conflict processing tasks to the processing node with the lowest load. For example, load status can include processor utilization, memory usage, and the current number of tasks. For instance, if processing node 1 has a load of 60% and processing node 2 has a load of 30%, the conflict processing task will be assigned to processing node 2. The processing system can also assign high-priority conflict processing tasks to processing nodes with better performance, and low-priority conflict processing tasks to ordinary processing nodes.

[0098] Optionally, the processing system can allocate corresponding threads based on multiple processing nodes. By parsing the tasks of the processing nodes, it can perform operations such as overwrite updates, ignoring conflicts, or logging exceptions in parallel. The processing system can also set a timeout for each task on each processing node. If the task fails to complete within the timeout period, it is marked as a processing failure and reassigned to another processing node. If the task still fails to complete within the timeout period after one retrieval, the processing system can issue an alarm.

[0099] Optionally, the processing system can collect the execution results of each processing node, establish a task status table, and update the task execution status in real time. When all assigned conflict resolution tasks have reported their execution results, the processing system can determine that the parallel processing has been fully completed.

[0100] Using the method described above, the conflict resolution task is distributed to multiple processing nodes based on the processing results, reducing processing latency. These multiple processing nodes then perform overwrite updates, ignore conflicts, or log abnormal operations in parallel, improving conflict resolution efficiency.

[0101] In this embodiment, by parsing the conflict handling rule configuration information, a conflict handling strategy can be generated, laying the foundation for subsequent data processing operations. Based on the comparison between the current value of the corresponding record in the target database and the previous value in the changed data, the conflict type can be determined, enabling full-type detection and flexible handling of data synchronization conflicts. Based on this conflict handling strategy and the conflict type, data processing operations are performed, including overwrite updates, ignoring conflicts, or recording anomalies, improving the accuracy of conflict handling.

[0102] Figure 2 A flowchart illustrating another data synchronization conflict handling method provided in this application. Figure 2 As shown, the method includes:

[0103] (1) Data entry and conflict resolution stage

[0104] Initiate "Start Ingestion" and determine if conflict handling is enabled: If conflict handling is enabled, construct a statement based on the original key value and the previous value of the record in the changed data; if conflict handling is not enabled, construct the statement directly. After binding the values, execute Structured Query Language (SQL); after executing SQL, determine if a conflict has occurred (primary key anomaly or update result of 0): if a conflict has occurred, execute the data commit operation and end the process.

[0105] (2) Conflict resolution rule parsing and generation stage

[0106] Input the row change data (RowChangeData) that has not caused any conflicts, start "Start Processing", process the rule file, parse the rule engine file; obtain conflict handling rules based on the change data matching; obtain the value of the column to be compared in the current data from the target database, and generate conflict handling rules based on the comparison of the value of the comparison column with the change data.

[0107] The row change data is rewritten according to the processing rules, and the final processing behavior is generated, an exception is generated and thrown. The exception is caught to determine whether the processing strategy is overwrite. If the processing strategy is overwrite, the rewritten row change data is re-entered into the database, and the statement is constructed.

[0108] If the processing strategy is not OVERWRITE, determine if the processing strategy is IGNORE. If IGNORE, write the changed data to the Structured Query Language (SQL) record file, commit the data, and end the process. If the processing strategy is not IGNORE, determine if the processing strategy is DISCARD. If DISCARD, write the changed data to the Structured Query Language (SQL) record file, commit the data, and end the process. If the processing strategy is not DISCARD, end the process directly.

[0109] Optionally, the rule configuration file can include default processing rules, tables to be processed, types of operations to be processed, columns to be compared, and processing rules.

[0110] Optionally, the types of conflicts may include unique constraint conflicts during insertion, data non-existence conflicts during updates, data existence conflicts during updates but some columns being inconsistent with their predecessors, data non-existence conflicts during deletion, and data existence conflicts during deletion but some columns being inconsistent with their predecessors.

[0111] Optionally, conflict resolution methods may include the following:

[0112] (1) If the value listed in the intermediate file is inconsistent with the value in the target database, the value in the intermediate file shall prevail and the insertion shall be changed to an update, or the value in the intermediate file shall prevail and the update shall be changed.

[0113] (2) Delete and record the conflicting data;

[0114] (3) Rewrite based on the data in the intermediate file;

[0115] (4) The conflict between missing update rows and missing delete rows can be ignored since they cannot be rewritten.

[0116] Optionally, the data rewriting rule can be that when insert is changed to update, the operation type of the row change data (RowChangeData) is changed to update, and the primary key is used as the key value. Alternatively, when update is changed to insert, the operation type of the row change data (RowChangeData) is changed to insert, and the key value is deleted.

[0117] Optionally, the data synchronization conflict handling process based on the preceding value and the rules engine may include the following steps:

[0118] Step 1: The source end parses the data according to the previous value configuration file, which requires parsing the values ​​of the corresponding columns before the change;

[0119] Step 2: The target device receives the changed data and begins the data entry process.

[0120] Step 3: Determine whether conflict handling is enabled. If so, use the original key value and the previous value recorded in the changed data as conditions.

[0121] Step 4: Construct the statement, bind values, and execute Structured Query Language (SQL);

[0122] Step 5: Execute the SQL to determine if a conflict has occurred (reporting a primary key conflict error and the number of updated rows being 0); if a conflict occurs when executing the SQL, proceed to Step 6; otherwise, proceed to Step 8.

[0123] Step Six: Process the rules using the rule engine.

[0124] Step 7: After the rule engine processes the exception and rewritten row change data, if the processing type is overwrite, re-execute step 4 based on the rewritten data; if it is discard, record the row change data and execute step 8; if it is ignore, execute step 8 directly.

[0125] Step 8: After submitting the data, the changed data will be written to the SQL record file.

[0126] Optionally, the rule processing engine process may include the following:

[0127] (1) Obtain the change data that needs to be processed, obtain the rule configuration file, and parse the configuration;

[0128] (2) Obtain the conflict handling rules matched by the current data;

[0129] (3) Retrieve the value of the field to be compared from the database, and then compare it with the current field;

[0130] (4) Compare the values ​​of the columns to obtain the conflict handling rules;

[0131] (5) Handle conflicting single data changes and obtain the final handling type: Ignore, Discard, or Overwrite.

[0132] (6) Rewrite the row change data (RowChangeData) according to the rewriting rules and construct an exception to throw.

[0133] The above are the method embodiments provided in this application. The apparatus provided in this application will be described below.

[0134] Figure 3 This is a schematic diagram of the structure of a data synchronization conflict handling device provided in this application, as shown below. Figure 3 As shown, the data synchronization conflict handling device 300 provided in this embodiment includes: a generation module 301, a determination module 302, and an execution module 303. Wherein,

[0135] The generation module 301 is used to generate a conflict handling strategy by parsing the conflict handling rule configuration information;

[0136] The determination module 302 is used to determine the conflict type based on the comparison result between the current value of the corresponding record in the target database and the previous value in the changed data;

[0137] The execution module 303 is used to perform data processing operations based on the conflict handling strategy and the conflict type, including overwrite updates, ignoring conflicts, or recording anomalies.

[0138] Optionally, the generation module 301 is further configured to obtain the table name, operation type, comparison column, and processing method by parsing the conflict handling rule configuration information; and generate a conflict handling strategy based on the table name, operation type, comparison column, and processing method.

[0139] Optionally, the generation module 301 is further configured to, before parsing the conflict handling rule configuration information, dynamically adjust the priority of the conflict handling rule according to the business priority.

[0140] Optionally, the generation module 301 is also used to update the conflict handling rule by parsing database-level rules, table-level rules and field-level rules after parsing the conflict handling rule configuration information;

[0141] Analyze historical conflict data using machine learning models and adjust the configuration information for the conflict resolution rules.

[0142] Optionally, the determining module 302 is also used to prefetch the previous value of the source-side changed data and cache it in local storage before obtaining the current value of the corresponding record in the target-side database.

[0143] Optionally, the execution module 303 is further configured to match a conflict handling strategy according to the conflict type to obtain a processing result; perform an overwrite update, ignore the conflict, or record an anomaly operation based on the processing result; and record the conflict data, the conflict type, and the conflict handling strategy to an audit log file.

[0144] Optionally, the execution module 303 is further configured to allocate the conflict handling task to multiple processing nodes based on the processing result; and to perform operations such as overwrite update, ignoring conflict, or recording abnormality in parallel based on the multiple processing nodes.

[0145] The data synchronization conflict handling device provided in this embodiment can execute the methods provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0146] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 4 As shown, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 400 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0147] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0148] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0149] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0150] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0151] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0152] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0153] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0154] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0155] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0156] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0157] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0158] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0159] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0160] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0161] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0162] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for handling data synchronization conflicts, characterized in that, include: Conflict handling strategies are generated by parsing the conflict handling rule configuration information; Based on the comparison between the current value of the corresponding record in the target database and the previous value in the changed data, the conflict type is determined. Based on the conflict handling strategy and the conflict type, data processing operations are performed, including overwrite updates, ignoring conflicts, or recording anomalies.

2. The method according to claim 1, characterized in that, The step of generating a conflict handling strategy by parsing the conflict handling rule configuration information includes: By parsing the conflict handling rule configuration information, we can obtain the table name, operation type, comparison columns, and handling method; A conflict resolution strategy is generated based on the table name, the operation type, the comparison column, and the processing method.

3. The method according to claim 1, characterized in that, The data processing operation based on the conflict handling strategy and the conflict type includes: Match the conflict handling strategy according to the conflict type to obtain the handling result; Based on the processing results, perform operations such as overwrite update, ignore conflict, or log anomalies; The conflict data, the conflict type, and the conflict handling strategy are recorded in the audit log file.

4. The method according to claim 3, characterized in that, The operation of performing overwrite update, ignoring conflicts, or logging anomalies based on the processing result includes: Based on the processing results, the conflict resolution task is assigned to multiple processing nodes; The multiple processing nodes perform operations such as overlay updates, ignoring conflicts, or logging anomalies in parallel.

5. The method according to claim 1, characterized in that, Before parsing the conflict handling rule configuration information, the process also includes: The priority of the conflict handling rules is dynamically adjusted based on business priorities.

6. The method according to any one of claims 1-5, characterized in that, After parsing the conflict handling rule configuration information, it includes at least one of the following: The conflict handling rules are updated by parsing database-level rules, table-level rules, and field-level rules; The historical conflict data is analyzed using a machine learning model, and the configuration information of the conflict resolution and handling rules is adjusted accordingly.

7. The method according to claim 1, characterized in that, Before obtaining the current value of the corresponding record in the target database, the previous value of the source data change is prefetched and cached in local storage.

8. A data synchronization conflict handling apparatus, comprising: The generation module is used to generate conflict handling strategies by parsing the conflict handling rule configuration information; The determination module is used to determine the conflict type based on the comparison between the current value of the corresponding record in the target database and the previous value in the changed data. An execution module is used to perform data processing operations based on the conflict handling strategy and the conflict type, the data processing operations including overwrite update, ignore conflict or record anomaly.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images