Data synchronization method, electronic device, storage medium and program product

Abstract

The invention provides a data synchronization method, electronic equipment, a storage medium and a program product, and relates to the field of distributed technologies. The method comprises the following steps: in response to a data synchronization instruction, executing various data verifications for synchronous data; if the data verification result shows that the data exception exists, obtaining a first type of attribute, a second type of attribute, a third type of attribute and a fourth type of attribute corresponding to the synchronous data; determining an exception handling mechanism for the synchronous data according to the first type of attributes; determining a retry mechanism for the synchronous data according to the second type of attributes; determining a data compensation mechanism for the synchronous data according to the third type of attributes; determining a monitoring alarm mechanism for the synchronous data according to the fourth type of attributes; and executing data processing on the synchronous data according to an exception handling mechanism, a retry mechanism, a data compensation mechanism and a monitoring alarm mechanism. The real-time performance and the consistency of data synchronization are improved, and the accuracy and the reliability of data processing are improved.

Description

Technical Field

[0001] The present application relates to the field of distributed technology, and in particular to a data synchronization method, electronic device, storage medium, and program product. Background Art

[0002] In distributed systems and microservice architectures, as business scale expands, system complexity increases, and the demand for data collaboration across services and databases surges, real-time data synchronization and consistency assurance become core challenges.

[0003] In the existing technology, data synchronization methods include data synchronization based on database logs and data synchronization through the publish or subscribe model of message middleware. Data synchronization based on database logs captures incremental changes by parsing database transaction logs to achieve low-latency data synchronization, but it depends on the log format of a specific database, and the migration or heterogeneous database adaptation cost is high. In addition, log parsing of cross-database transactions or distributed transactions may cause data inconsistency due to network partitioning or delays. The publish or subscribe model based on message middleware publishes data changes as messages, which are consumed and processed by subscribers to achieve decoupling and asynchronous synchronization, but lacks a guarantee mechanism, which may lead to problems such as message disorder or duplication. In the asynchronous model, it is difficult to meet the real-time requirements of the business.

[0004] Based on this, in the existing technology, there are technical problems such as insufficient real-time performance and consistency of data synchronization. Summary of the Invention

[0005] The present application provides a data synchronization method, electronic device, storage medium and program product to solve the technical problems of insufficient real-time performance and consistency of data synchronization.

[0006] In a first aspect, the present application provides a data synchronization method, comprising:

[0007] In response to a data synchronization instruction, performing a plurality of data checks on the synchronization data;

[0008] If the data verification result shows that there is data anomaly, the first, second, third and fourth attributes corresponding to the synchronized data are obtained;

[0009] Based on the first type of attributes, determine the exception handling mechanism for synchronized data;

[0010] Based on the second type of attributes, determine the retry mechanism for synchronization data;

[0011] According to the third type of attributes, a data compensation mechanism for the synchronized data is determined;

[0012] Based on the fourth type of attributes, determine the monitoring and alarm mechanism for synchronized data;

[0013] Data processing of synchronized data is performed based on the exception handling mechanism, retry mechanism, data compensation mechanism, and monitoring alarm mechanism.

[0014] In a second aspect, the present application provides a data synchronization device, comprising:

[0015] A verification module, configured to perform multiple data verifications on the synchronization data in response to the data synchronization instruction;

[0016] An acquisition module, configured to acquire first, second, third, and fourth attributes corresponding to the synchronized data if the data verification result indicates that there is a data anomaly;

[0017] A first determining module, configured to determine an exception handling mechanism for synchronization data according to the first type of attributes;

[0018] A second determining module, configured to determine a retry mechanism for synchronization data according to the second type of attributes;

[0019] a third determining module, configured to determine a data compensation mechanism for the synchronized data according to the third type of attributes;

[0020] a fourth determining module, configured to determine a monitoring and alarm mechanism for the synchronization data according to the fourth type of attribute;

[0021] The execution module is used to perform data processing on the synchronization data according to the exception handling mechanism, retry mechanism, data compensation mechanism and monitoring alarm mechanism.

[0022] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0023] Memory stores computer-executable instructions;

[0024] The processor executes the computer-executable instructions stored in the memory to implement the first aspect or various possible implementations of the first aspect.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect or various possible implementations of the first aspect.

[0026] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the first aspect or various possible implementations of the first aspect.

[0027] The present application provides a data synchronization method, electronic device, storage medium and program product, which perform multiple data verifications on the synchronized data in response to different data synchronization instructions; according to the data verification results, obtain the first type of attributes, second type of attributes, third type of attributes and fourth type of attributes corresponding to the synchronized data respectively to determine the corresponding exception handling mechanism, retry mechanism, data compensation mechanism and monitoring alarm mechanism of the synchronized data; thereby performing corresponding data processing to improve the real-time and consistency of data synchronization, and improve the accuracy and reliability of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0029] Figure 1 A schematic diagram of a data synchronization system architecture provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 1 ;

[0031] Figure 3 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 2 ;

[0032] Figure 4 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 3 ;

[0033] Figure 5 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 4 ;

[0034] Figure 6 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 5 ;

[0035] Figure 7 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 6 ;

[0036] Figure 8 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application;

[0037] Figure 9 This is a schematic diagram of the structure of the electronic device provided in this application.

[0038] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0039] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0041] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0042] It should be noted that the data synchronization method, electronic device, storage medium and program product provided in this application can be used in the field of distributed technology, and can also be used in any field other than the field of distributed technology. The application field of the data synchronization method, electronic device, storage medium and program product in this application is not limited.

[0043] Canal (Change Analysis Notification And Logging, database log incremental parsing and data synchronization middleware) is an open source database middleware product that parses incremental logs of MySQL (My Structured Query Language, MySQL relational database management system) databases to provide incremental data subscription and consumption.

[0044] Existing Canal-based database log synchronization technology achieves incremental data capture and subscription by parsing MySQL's Binary log to reduce latency. However, existing technology is strongly coupled to the specific log format of the MySQL database, resulting in high costs for system migration or heterogeneous database adaptation. In addition, in the event of network partitions or delays, log parsing of cross-database transactions may cause data inconsistencies.

[0045] Another publish-subscribe model based on message middleware such as Kafka improves system scalability through asynchronous decoupling, but there are risks of message disorder and duplicate consumption, and the asynchronous mechanism is difficult to meet the needs of high-real-time business scenarios.

[0046] Furthermore, in the existing technology, the data transmission process lacks an effective verification mechanism, and there may be a risk of data loss in the event of network anomalies; and there is a lack of fault tolerance mechanism and synchronization delay monitoring during data synchronization. In high-concurrency scenarios, there is a lack of early warning and processing means, resulting in poor reliability, real-time and consistency of data synchronization.

[0047] In order to solve the above problems, the core concept of this application is: when performing data synchronization in response to a data synchronization instruction, multiple data verifications are performed; for the synchronized data with data anomalies in the data verification results, the corresponding first-category attributes, second-category attributes, third-category attributes and fourth-category attributes are obtained; and according to different categories of attributes, the exception handling mechanism, retry mechanism, data compensation mechanism and monitoring alarm mechanism for the synchronized data are determined to perform corresponding data processing, thereby improving the real-time, consistency and reliability of data synchronization.

[0048] Optionally, Figure 1 This is a schematic diagram of a data synchronization system architecture provided by an embodiment of the present application. Figure 1 As shown, the data synchronization system architecture includes at least one of a data acquisition device 101 , a processing device 102 and a display device 103 .

[0049] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the above architecture. In other feasible implementations of this application, the above architecture may include more or fewer components than shown, or combine or split certain components, or arrange the components differently. The specific configuration can be determined based on the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0050] In a specific implementation process, the data acquisition device 101 may include an input / output interface and may also include a communication interface. The data acquisition device 101 may be connected to the processing device via the input / output interface or the communication interface.

[0051] The processing device 102 can perform multiple data checks on the synchronized data in response to the data synchronization instruction; if the data check result is that there is a data abnormality, the first type of attributes, the second type of attributes, the third type of attributes and the fourth type of attributes corresponding to the synchronized data are obtained; based on the first type of attributes, the exception handling mechanism for the synchronized data is determined; based on the second type of attributes, the retry mechanism for the synchronized data is determined; based on the third type of attributes, the data compensation mechanism for the synchronized data is determined; based on the fourth type of attributes, the monitoring and alarm mechanism for the synchronized data is determined; according to the exception handling mechanism, the retry mechanism, the data compensation mechanism and the monitoring and alarm mechanism, data processing of the synchronized data is performed.

[0052] The display device 103 may also be a touch screen display or a screen of a terminal device, which is used to receive user instructions while displaying the above content to achieve interaction with the user.

[0053] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0054] Figure 2 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 1 ,like Figure 2 As shown, the method includes:

[0055] S201 . In response to a data synchronization instruction, perform multiple data checks on the synchronization data.

[0056] Optionally, data verification includes multiple types of data analysis verification, data transmission verification, data processing verification and data storage verification.

[0057] In this embodiment, by performing multiple checks on the synchronized data, including data parsing check, data transmission check, data processing check and data storage check, the accuracy, integrity, reliability and consistency of the data during the synchronization process are ensured, thereby improving the stability of the system.

[0058] S202: If the data verification result shows that there is data anomaly, obtain the first type of attributes, the second type of attributes, the third type of attributes, and the fourth type of attributes corresponding to the synchronization data.

[0059] In this embodiment, the first category of attributes includes the nature of the exception and the cause of the exception; the second category of attributes includes the retry condition judgment attribute and / or the retry policy determination attribute; the third category of attributes includes the source data and the target data; the fourth category of attributes includes the alarm condition judgment attribute and / or the alarm policy determination attribute.

[0060] S203: Determine an exception handling mechanism for synchronization data according to the first type of attributes.

[0061] In this embodiment, based on the first type of attributes, an exception handling mechanism for synchronized data is specifically determined, which reduces exception troubleshooting time, improves problem solving efficiency, reduces system interruptions, and ensures the accuracy and stability of data synchronization.

[0062] S204: Determine a retry mechanism for synchronization data according to the second type of attributes.

[0063] In this embodiment, a retry mechanism for synchronization data is determined based on the second type of attributes, which reduces the amount of synchronization data retransmission, optimizes data transmission reliability, and ensures the integrity of synchronization data and the success rate of data synchronization.

[0064] S205: Determine a data compensation mechanism for the synchronization data according to the third type of attributes.

[0065] In this embodiment, a data compensation mechanism for synchronized data is determined based on the third type of attributes, thereby ensuring consistency in data transmission and reducing business risks caused by data inconsistency.

[0066] S206: Determine a monitoring and alarm mechanism for the synchronization data based on the fourth attribute.

[0067] In this embodiment, based on the fourth type of attributes, a monitoring and alarm mechanism for synchronized data is determined to promptly warn of anomalies, ensure data synchronization security, improve the system's response speed and maintenance efficiency, and ensure the real-time nature of data synchronization.

[0068] S207: Execute data processing on the synchronization data according to the exception handling mechanism, the retry mechanism, the data compensation mechanism, and the monitoring alarm mechanism.

[0069] In this embodiment, data processing of synchronized data is performed based on the exception handling mechanism, retry mechanism, data compensation mechanism and monitoring alarm mechanism, which effectively solves the real-time and consistency problems in the data synchronization process, improves the accuracy and reliability of data processing, and adapts to data processing requirements of different scales and complexities.

[0070] A data synchronization method provided in this application solves the real-time and consistency problems of the existing technology in the data synchronization process through a data verification mechanism, an exception handling mechanism, a retry mechanism, a data compensation mechanism and a monitoring alarm mechanism, and improves the accuracy and reliability of data processing.

[0071] Figure 3 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 2 ,like Figure 3 As shown, in Figure 2 Based on the illustrated embodiment, data verification includes multiple types of data parsing verification, data transmission verification, data processing verification, and data storage verification; accordingly, in response to the data synchronization instruction in step S201 above, multiple data verifications are performed on the synchronized data in detail, including:

[0072] S301: Obtain database incremental logs of synchronized data.

[0073] In this embodiment, all fields of each record in the database incremental log of the synchronization data are obtained.

[0074] S302: Check whether there are any missing fields or field errors in the database incremental log.

[0075] In this embodiment, for the fields in the database incremental log, it is checked in sequence whether the fields are missing, whether the field data type is incorrect, and whether the data format is incorrect.

[0076] For example, if the synchronized data in Canal is the amount data of a transfer, and if the transfer record in the database incremental log of the synchronized data lacks the account number field, then the verification result of the data parsing verification is determined to be field missing; if the data type of the predefined field is a numeric field, and the field in the database incremental log includes letters, then the verification result of the data parsing verification is determined to be a field error; if the predefined data format is a date format, and the date format is "YYYY-MM-DD", and the date format of the field in the database incremental log is "YYYY year MM month DD day", then the verification result of the data parsing verification is determined to be a format error.

[0077] S303: When synchronization data is transmitted between the data synchronization tool and the target system, a first cyclic redundancy check code and a second cyclic redundancy check code corresponding to the synchronization data are obtained.

[0078] In this embodiment, for example, when synchronization data is transmitted between the Canal and the target system, a CRC (Cyclic Redundancy Check) check algorithm is used to calculate a first cyclic redundancy check code corresponding to the synchronization data, and the first cyclic redundancy check code is attached to the header or tail of a data packet corresponding to the synchronization data for transmission; when the target system receives the data packet of the synchronization data, a CRC check algorithm is used to calculate a second cyclic redundancy check code corresponding to the synchronization data.

[0079] S304: Determine whether there is a data transmission error in the synchronization data according to the first cyclic redundancy check code and the second cyclic redundancy check code.

[0080] In this embodiment, by comparing the first cyclic redundancy check code and the second cyclic redundancy check code, if the comparison result is that the first cyclic redundancy check code and the second cyclic redundancy check code are consistent, it is determined that there is no data transmission error in the synchronization data; if the comparison result is that the first cyclic redundancy check code and the second cyclic redundancy check code are inconsistent, it is determined that there is a data transmission error in the synchronization data.

[0081] S305: Before the target system processes the synchronized data, the synchronized data is checked for business logic and data duplication.

[0082] In this embodiment, data processing verification includes business logic verification and data duplication verification. Business logic verification refers to verifying whether the synchronized data complies with the business logic. For example, if the synchronized data is the user's age, it is verified whether the user's age field is a negative number. If the user's age field is a negative number, it indicates that the synchronized data has failed the business logic verification; if the user's age field is not a negative number, it indicates that the synchronized data has passed the business logic verification.

[0083] Data duplication check refers to checking whether the synchronized data is repeated to ensure the uniqueness of the synchronized data.

[0084] S306 : After writing the synchronization data to the storage medium, detecting whether the synchronization data is successfully written and the writing consistency.

[0085] In this embodiment, the writing consistency of the synchronization data is detected by reading the returned status code or using the CRC32 check algorithm, and comparing the synchronization data before writing and the synchronization data after writing to see if they are consistent.

[0086] The data synchronization method provided in the embodiment of the present application sequentially performs data analysis and verification on the synchronized data, thereby improving the data quality of the synchronized data; performs data transmission verification, thereby ensuring the integrity of the synchronized data during the transmission process; performs data processing verification, thereby improving the accuracy and processing efficiency of data processing; and performs data storage verification, thereby ensuring the accuracy and integrity of the synchronized data, so that the synchronized data remains consistent during the synchronization process.

[0087] Figure 4 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 3 ,like Figure 4 As shown, in Figure 2 Based on the embodiment shown, the first type of attributes includes the nature of the exception and the cause of the exception. Accordingly, the above step S203, in which the exception handling mechanism for the synchronization data is determined based on the first type of attributes, is described in detail, including:

[0088] S401. Determine a second abnormality type of the synchronization data according to the abnormality property and the abnormality cause; wherein the second abnormality type includes a network abnormality, a data format abnormality, and a business logic abnormality.

[0089] In this embodiment, the second abnormality type of the synchronous data is determined by performing data verification on the synchronous data and obtaining the abnormality property and abnormality cause in the data verification result.

[0090] For example, Canal synchronizes a transfer record to the target system. During data parsing and verification, if it is detected that the data format of the transfer record field is incorrect, the second anomaly type of the synchronized data is determined to be a data format anomaly; if during data transmission verification, some data packets of the synchronized data are lost due to network fluctuations, the second anomaly type of the synchronized data is determined to be a network anomaly; if during data verification, the transfer amount in the transfer record is a negative number, the second anomaly type of the synchronized data is determined to be a business logic anomaly.

[0091] S402: Obtain a preset processing strategy corresponding to the second exception type.

[0092] Optionally, the preset processing strategy includes whether to trigger a data compensation mechanism. For example, if it is determined that there is missing data in the synchronized data, the data compensation mechanism is triggered.

[0093] Optionally, the preset processing strategy includes whether to trigger a retry mechanism. For example, if it is determined that the second abnormality type of the synchronized data is a network abnormality, the retry mechanism is triggered.

[0094] Optionally, the preset processing strategy further includes whether to trigger a monitoring alarm mechanism. For example, if it is determined that the second abnormality type of the synchronized data is a data format abnormality, the monitoring alarm mechanism is triggered.

[0095] In this embodiment, the preset processing strategies corresponding to different second exception types are determined through a preset system configuration file to meet the needs of different business scenarios; for example, if synchronization data with an abnormal data format is parsed, the alarm mechanism is automatically triggered and the corresponding log information is recorded; if a network exception of network interruption is parsed, the retry mechanism is automatically triggered; if synchronization data with a business logic exception is parsed, for example, the transfer amount is -1, the predefined exception handling strategy is automatically matched according to the exception type, the erroneous transfer amount is modified to a default value of 0, and the log information is recorded.

[0096] S403: Determine an exception handling mechanism for synchronization data according to a preset processing strategy.

[0097] In this embodiment, the corresponding exception handling mechanism is executed according to the determined preset processing strategy.

[0098] For example, if the account number field is missing from the transfer record, the corresponding preset processing strategy is to trigger the data compensation mechanism and execute the data compensation mechanism;

[0099] Optionally, if some packets of synchronized data are lost due to network fluctuations, the corresponding preset processing strategy is to trigger a retry mechanism, and the retry mechanism is executed;

[0100] Optionally, if a data format error is detected in a transfer record field, the corresponding preset processing strategy is to trigger a monitoring alarm mechanism, and the monitoring alarm mechanism is executed.

[0101] The data synchronization method provided in this application identifies the exception type, obtains the preset processing strategy, and executes the corresponding exception handling mechanism; takes corresponding processing measures in a targeted manner, effectively improving the efficiency and accuracy of exception handling, reducing the impact of exceptions on the business, and improving the overall performance of the system.

[0102] Figure 5 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 4 ,like Figure 5 As shown, in Figure 2 Based on the illustrated embodiment, the second type of attributes includes a retry condition judgment attribute and / or a retry strategy determination attribute; accordingly, determining a retry mechanism for synchronization data based on the second type of attributes in step S204 is described in detail, including:

[0103] S501: Determine whether to perform a retry on the synchronization data according to the attribute judged by the retry condition.

[0104] In this embodiment, the retry condition determination attributes include at least one of the first exception type, the exception duration, and the current system state. The first exception type includes recoverable and unrecoverable exceptions, and the current system state includes system load and resource usage. A comprehensive consideration of the retry condition determination attributes determines whether to retry data synchronization. For example, if synchronization fails due to a network failure during data synchronization, since the network failure is a recoverable exception, a retry is determined for data synchronization.

[0105] S502: Determine attributes according to the retry strategy to determine a retry interval and a maximum number of retries for synchronization data.

[0106] In this embodiment, the retry policy determination attribute includes at least one of business importance and system performance requirement. For example, if the business importance includes real-time requirement, the retry interval is set to 5 seconds and the maximum number of retries is 3 times.

[0107] Optionally, a data synchronization method provided in the present application includes at least one of the above steps S501 or S502.

[0108] Furthermore, after determining to execute a retry for the synchronized data, the system executes the retry mechanism according to the preset retry interval and maximum number of retries for the synchronized data, and records the log information of the retry process, wherein the log information includes the exception type, timestamp, trigger reason and retry result of the retry; it provides a reference basis for subsequent problem troubleshooting and performance optimization, and completes the rapid location and processing of the problem.

[0109] The present application provides a data synchronization method that determines whether to perform a retry on the synchronized data by judging the attributes through retry conditions; so that the synchronized data can be restored and processed in a timely manner, thus avoiding resource waste, reducing the system burden, and improving system performance. In addition, the method determines the attributes according to different retry strategies, determines the retry interval and the maximum number of retries for the synchronized data, adapts to different business needs, and improves the flexibility and scalability of the system.

[0110] Figure 6 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 5 ,like Figure 6 As shown, in Figure 2 Based on the embodiment shown, the third type of attributes includes source data and target data; accordingly, determining the data compensation mechanism for the synchronized data according to the third type of attributes in step S205 is described in detail, including:

[0111] S601. Determine missing data based on source data and target data.

[0112] In this embodiment, two key-value pair mapping tables are constructed to store source data and target data respectively; the source data and target data are traversed and compared to check each record and each field in the data; if there is a missing record or missing field, the corresponding target data is marked as missing data; wherein, the source data refers to the synchronized data before synchronization, and the target data refers to the synchronized data after synchronization.

[0113] S602: Determine a data re-pull strategy based on the missing data.

[0114] In this embodiment, if missing data is detected, the data re-pull strategy is determined to be executed; based on the key-value pair mapping table where the missing data is located, the identifier of the missing data is determined, and the source data corresponding to the missing data is further determined, and the source data corresponding to the missing data is used as the new synchronization data.

[0115] S603. Determine a verification strategy for data compensation based on the missing data; wherein the verification strategy includes at least one of field integrity, data type correctness, data format standardization, and whether the data complies with business logic.

[0116] In this embodiment, after determining the data re-pull strategy, the data compensation verification strategy is initiated. For example, the integrity of the fields, and / or the correctness of the data type, and / or the standardization of the data format and / or whether the data complies with the business logic of the synchronized data is verified.

[0117] S604: Determine a data compensation mechanism based on the data re-pull strategy and the verification strategy.

[0118] In this embodiment, for example, if the synchronization data passes the verification of field integrity, and / or data type correctness, and / or data format standardization and / or whether the data complies with business logic, the synchronization data will be synchronized to the target system.

[0119] This application provides a data synchronization method that identifies missing data based on a comparison of source and target data; determines a data re-pull strategy; and verifies at least one of the following for the missing data to be synchronized: field integrity, data type correctness, data format compliance, and data compliance with business logic; and synchronizes the verified data to the target system. This ensures fast and accurate recovery of missing data, guarantees the accuracy and consistency of data synchronization, and prevents erroneous data from impacting the target system.

[0120] Figure 7 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 6 ,like Figure 7 As shown, in Figure 2Based on the illustrated embodiment, the fourth type of attributes includes alarm condition judgment attributes and / or alarm policy determination attributes. Accordingly, the determination of the monitoring alarm mechanism for the synchronized data based on the fourth type of attributes in step S206 is described in detail, including:

[0121] S701: Determine the attributes according to the alarm condition and whether to execute an alarm for the synchronization data.

[0122] In this embodiment, the alarm condition judgment attribute includes business requirements.

[0123] For example, a first threshold and a second threshold are set. If the synchronization speed of the synchronization data is lower than the first threshold, it is determined to execute an alarm for the synchronization data; if the queue length of the synchronization data exceeds the second threshold, it is determined to execute an alarm for the synchronization data; if the source data and target data of the synchronization data are inconsistent, it is determined to execute an alarm for the synchronization data, where the first threshold can be 1000 items / second or 500 items / second.

[0124] S702: Determine attributes according to the alarm policy, and determine the alarm information and alarm method for executing the synchronized data.

[0125] In this embodiment, the alarm policy determination attribute includes at least one of a parsing speed index, a processing speed index, and a synchronization speed index.

[0126] In this embodiment, the before and after execution time is obtained during verification processing to collect the parsing speed index, the before and after execution time is obtained when synchronizing the synchronization data to collect the processing speed index, and the before and after execution time is obtained during exception processing to collect the synchronization speed index; finally, the parsing speed index, processing speed index and synchronization speed index are saved in the database to complete the collection of attributes determined by the alarm strategy.

[0127] Optionally, if there is an anomaly in the synchronized data, the status indicators of the integrity, accuracy, and consistency of the data are determined according to the anomaly type to complete the collection of the status indicators.

[0128] Furthermore, when the collected alarm strategy determines that the attributes meet the alarm condition judgment attributes, the alarm mechanism is automatically triggered, and the alarm information is sent to the administrator or relevant users through the preset notification method, where the preset notification method includes email, SMS, system notification or visual monitoring interface, and the alarm information includes the Canal's operating status, data synchronization status, performance indicators and status indicators; among them, the visual monitoring interface includes charts, dashboards and real-time data, so as to provide administrators with data for rapid analysis and problem troubleshooting, and make corresponding optimization adjustments.

[0129] Optionally, a data synchronization method provided in the present application includes at least one of the above steps S701 or S702.

[0130] The data synchronization method provided in this application determines whether to issue an alarm for synchronized data by judging attributes based on alarm conditions; and determines the attributes based on alarm policies to determine the alarm information and method for executing the alarm for synchronized data. This method allows for rapid response and resolution to issues, shortening troubleshooting and repair time, minimizing the impact of anomalies on the system and services, ensuring the continuity and real-time nature of data synchronization, and improving system reliability and stability.

[0131] Figure 8 A structural diagram of a data synchronization device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the data synchronization device includes:

[0132] The verification module 801 is configured to perform various data verifications on the synchronization data in response to a data synchronization instruction.

[0133] Optionally, data verification includes multiple types of data analysis verification, data transmission verification, data processing verification and data storage verification.

[0134] The acquisition module 802 is configured to acquire the first, second, third, and fourth attributes corresponding to the synchronized data if the data verification result indicates that there is a data anomaly.

[0135] A first determining module 803 is configured to determine an exception handling mechanism for synchronization data according to the first type of attributes;

[0136] A second determining module 804 is configured to determine a retry mechanism for synchronization data according to the second type of attributes;

[0137] A third determining module 805 is configured to determine a data compensation mechanism for the synchronized data according to the third type of attribute;

[0138] A fourth determining module 806 is configured to determine a monitoring and alarm mechanism for the synchronization data according to the fourth type of attribute;

[0139] The execution module 807 is used to perform data processing on the synchronization data according to the exception handling mechanism, the retry mechanism, the data compensation mechanism and the monitoring alarm mechanism.

[0140] In a possible implementation, the verification module 801 may further be used to:

[0141] Get the database incremental log of synchronized data;

[0142] Check whether there are missing fields or field errors in the database incremental log.

[0143] In a possible implementation, the verification module 801 may further be used to:

[0144] When synchronization data is transmitted between the data synchronization tool and the target system, a first cyclic redundancy check code and a second cyclic redundancy check code corresponding to the synchronization data are obtained;

[0145] It is determined whether there is a data transmission error in the synchronization data according to the first cyclic redundancy check code and the second cyclic redundancy check code.

[0146] In a possible implementation, the verification module 801 may further be used to:

[0147] Before the target system processes the synchronized data, the synchronized data is verified for business logic and data duplication.

[0148] In a possible implementation, the verification module 801 may further be used to:

[0149] After the synchronization data is written to the storage medium, whether the synchronization data is successfully written and the writing consistency is detected.

[0150] In a possible implementation, the first determining module 803 may further be configured to:

[0151] Determine the second abnormality type of the synchronization data according to the abnormality nature and abnormality cause; wherein the second abnormality type includes network abnormality, data format abnormality and business logic abnormality;

[0152] Obtaining a preset processing strategy corresponding to the second exception type;

[0153] Determine the exception handling mechanism for synchronized data based on the preset processing strategy.

[0154] In a possible implementation, the second determining module 804 may further be configured to:

[0155] Determine whether to perform a retry on the synchronization data according to the retry condition judgment attribute; and\or,

[0156] Determine the properties according to the retry strategy to determine the retry interval and maximum number of retries for synchronization data;

[0157] The retry condition judgment attribute includes at least one of the first exception type, the exception duration, and the current state of the system; and the retry strategy determination attribute includes at least one of the business importance and the system performance requirement.

[0158] In a possible implementation, the third determining module 805 may further be configured to:

[0159] Determine missing data based on source and target data;

[0160] Determine the data re-pull strategy based on missing data;

[0161] Determine a data compensation verification strategy based on the missing data; wherein the verification strategy includes at least one of field integrity, data type correctness, data format standardization, and whether the data complies with business logic;

[0162] Determine the data compensation mechanism based on the data re-pull strategy and verification strategy.

[0163] In a possible implementation, the fourth determining module 806 may further be configured to:

[0164] Determine whether to execute an alarm for the synchronization data according to the alarm condition judgment attribute; and\or,

[0165] Determine the attributes based on the alarm policy, and determine the alarm information and alarm method for executing synchronized data;

[0166] The alarm condition judgment attribute includes business requirements; the alarm strategy determination attribute includes at least one of a parsing speed index, a processing speed index, and a synchronization speed index.

[0167] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present application may also be implemented in other ways. For example, the division of units / modules in the above-described embodiments is merely a logical functional division, and actual implementations may employ other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0168] In addition, unless otherwise specified, the functional units / modules in the various embodiments of the present application may be integrated into a single unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The aforementioned integrated units / modules may be implemented in the form of hardware or software program modules.

[0169] Figure 9 A schematic diagram of the structure of the electronic device provided in this application, such as Figure 9 As shown, the electronic device includes: at least one processor 901 and a memory 902. Optionally, the electronic device further includes a communication component 903. The processor 901, the memory 902 and the communication component 903 are connected via a bus 904.

[0170] During the specific implementation process, at least one processor 901 executes the computer-executable instructions stored in the memory 902, so that the at least one processor 901 performs the above method.

[0171] The specific implementation process of the processor 901 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0172] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0173] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0174] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0175] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0176] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0177] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0178] An exemplary readable storage medium is coupled to a processor, such that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit. Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0179] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0180] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0182] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as a mobile hard disk, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0183] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0184] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A data synchronization method, characterized in that: include: In response to a data synchronization instruction, performing a plurality of data checks on the synchronization data; If the data verification result shows that there is data anomaly, obtaining the first attribute, the second attribute, the third attribute, and the fourth attribute corresponding to the synchronization data; Determining an exception handling mechanism for the synchronization data according to the first type of attributes; determining a retry mechanism for the synchronization data according to the second type of attributes; determining a data compensation mechanism for the synchronized data according to the third type of attributes; Determining a monitoring and alarm mechanism for the synchronization data according to the fourth type of attributes; Data processing of the synchronization data is performed according to the exception handling mechanism, the retry mechanism, the data compensation mechanism and the monitoring alarm mechanism.

2. The method according to claim 1, characterized in that The first type of attributes includes the nature of the anomaly and the cause of the anomaly; Accordingly, determining an exception handling mechanism for the synchronization data according to the first type of attributes includes: Determining a second abnormality type of the synchronization data according to the abnormality nature and the abnormality cause; wherein the second abnormality type includes a network abnormality, a data format abnormality, and a business logic abnormality; Obtaining a preset processing strategy corresponding to the second exception type; According to the preset processing strategy, an exception handling mechanism for the synchronization data is determined.

3. The method according to claim 1, characterized in that The second type of attributes include retry condition judgment attributes and / or retry strategy determination attributes; Accordingly, determining a retry mechanism for the synchronization data according to the second type of attributes includes: Determine whether to perform a retry on the synchronization data according to the retry condition judgment attribute; and\or, Determine attributes according to the retry strategy to determine a retry interval and a maximum number of retries for executing the synchronization data; The retry condition judgment attribute includes at least one of the first exception type, the exception duration, and the current state of the system; and the retry strategy determination attribute includes at least one of the business importance and the system performance requirement.

4. The method according to claim 1, wherein The third type of attributes includes source data and target data; Accordingly, determining a data compensation mechanism for the synchronized data according to the third type of attributes includes: Determining missing data based on the source data and the target data; Determine a data re-pull strategy based on the missing data; Determine a data compensation verification strategy based on the missing data; wherein the verification strategy includes at least one of field integrity, data type correctness, data format standardization, and whether the data complies with business logic; The data compensation mechanism is determined according to the data re-pull strategy and the verification strategy.

5. The method according to claim 1, wherein The fourth type of attributes includes alarm condition judgment attributes and / or alarm strategy determination attributes; Accordingly, determining a monitoring and alarm mechanism for the synchronization data according to the fourth type of attributes includes: Determining whether to issue an alarm for the synchronization data according to the alarm condition judgment attribute; and\or, Determine attributes according to the alarm strategy, and determine the alarm information and alarm method for executing the synchronized data; The alarm condition judgment attribute includes business requirements; the alarm strategy determination attribute includes at least one of a parsing speed index, a processing speed index, and a synchronization speed index.

6. The method according to claim 1, characterized in that The data verification includes multiple types of data analysis verification, data transmission verification, data processing verification and data storage verification.

7. The method according to claim 6, characterized in that The data analysis and verification includes: Obtaining a database incremental log of the synchronized data; Check whether there are any missing fields or field errors in the fields in the incremental log of the database.

8. The method according to claim 6, characterized in that The data transmission verification includes: When the synchronization data is transmitted between the data synchronization tool and the target system, a first cyclic redundancy check code and a second cyclic redundancy check code corresponding to the synchronization data are obtained; Determine whether there is a data transmission error in the synchronization data according to the first cyclic redundancy check code and the second cyclic redundancy check code.

9. The method according to claim 6, characterized in that The data processing and verification includes: Before the target system processes the synchronization data, the synchronization data is subjected to business logic verification and data duplication verification.

10. The method according to claim 6, characterized in that The data storage verification includes: After the synchronization data is written into the storage medium, whether the synchronization data is successfully written and the writing consistency are detected.

11. A data synchronization device, characterized in that: include: A verification module, configured to perform multiple data verifications on the synchronization data in response to the data synchronization instruction; an acquisition module, configured to acquire the first attribute, the second attribute, the third attribute, and the fourth attribute corresponding to the synchronization data if the data verification result indicates that the data is abnormal; a first determining module, configured to determine an exception handling mechanism for the synchronization data according to the first type of attributes; a second determining module, configured to determine a retry mechanism for the synchronization data according to the second type of attributes; a third determining module, configured to determine a data compensation mechanism for the synchronized data according to the third type of attributes; a fourth determining module, configured to determine a monitoring and alarm mechanism for the synchronization data according to the fourth type of attributes; An execution module is used to perform data processing on the synchronization data according to the exception handling mechanism, the retry mechanism, the data compensation mechanism and the monitoring alarm mechanism.

12. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

14. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 10 when being executed by a processor.

Images