Data synchronization method and device, equipment and medium

Abstract

The invention belongs to the field of big data, and relates to a data synchronization method, comprising: acquiring a target timestamp recorded in a configuration table of a data management framework, the target timestamp being a maximum timestamp recorded in the configuration table when data synchronization processing is performed last time; the target timestamp is adopted as a screening condition, and incremental data with the timestamp larger than the target timestamp is inquired from the data source table; according to a field mapping relationship in the configuration table, determining a synchronization field which needs to be synchronized to the target table from the incremental data of the data source table; and on the basis of a preset unique field, matching the synchronization field of the data source table with the data of the target table, and on the basis of a matching result, synchronizing the synchronization field of the data source table to the target table. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the configuration table of the data management framework, the target timestamp and the synchronization field of the data source table can be stored in the block chain. The incremental data synchronization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data synchronization method, device, equipment and medium. Background Art

[0002] In today's era of information explosion and rapid data growth, data management has become an indispensable key link in all walks of life. The core of data management lies in how to efficiently and accurately process and synchronize data to ensure data integrity and consistency. Full and incremental modes are the two main strategies for data synchronization, each with distinct characteristics and applicable scenarios.

[0003] The full synchronization strategy has been widely used in initial data migration and scenarios with high data integrity requirements because it can ensure that the target system and the source system data are completely consistent. However, full synchronization also has obvious disadvantages, especially when the amount of data is huge. Transmitting a large amount of data will take up more network bandwidth and time, making frequent full synchronization impractical.

[0004] In contrast, the incremental synchronization strategy effectively saves transmission costs and time by only transmitting data that has changed since the last synchronization, and is particularly suitable for scenarios with frequent updates. However, incremental synchronization also faces the problems of complex initial synchronization and possible data synchronization omissions. In order to perform incremental synchronization, a full synchronization is required to establish the initial data state before switching to incremental synchronization mode. Incremental synchronization is prone to data omissions and repeated synchronization, which affects the accuracy and consistency of the data. Therefore, it is urgent to develop a method that can overcome the problems of data omissions and repeated synchronization in incremental synchronization. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a data synchronization method, device, equipment and medium to solve the problem that data omission and repeated synchronization are prone to occur in the existing incremental synchronization.

[0006] In order to solve the above technical problems, the present application provides a data synchronization method, which adopts the following technical solutions:

[0007] Get the target timestamp recorded in the configuration table of the data management framework. The target timestamp is the maximum timestamp recorded in the configuration table during the last data synchronization process. Use the target timestamp as the filtering condition to query the incremental data with a timestamp greater than the target timestamp from the data source table. According to the field mapping relationship in the configuration table, determine the synchronization fields that need to be synchronized to the target table from the incremental data of the data source table. Based on the preset uniqueness field, match the synchronization fields of the data source table with the data of the target table. Based on the matching result, synchronize the synchronization fields of the data source table to the target table.

[0008] Furthermore, before the step of obtaining the target timestamp recorded in the configuration table of the data management framework, the method further includes:

[0009] Get the preset key fields, and create a configuration table based on the key fields. The configuration table is used to record field mapping relationships, synchronize general scheduling job information, and record timestamps; get the preset job parameters, variables, and property files, and create a general scheduling job based on the job parameters, variables, and property files. The general scheduling job is used to cyclically capture the information that needs to be synchronized in the configuration table, and perform data synchronization operations in sequence; get the preset synchronization conversion template, which includes multiple key controls and the configuration file and path information of each key control; get the target attribute information; based on the target attribute information, update the configuration file and path information of each key control of the synchronization conversion template, and generate the target synchronization conversion file; build a data management framework based on the configuration table, general scheduling job, and target synchronization conversion file.

[0010] Further, the step of using the target timestamp as a screening condition to query the incremental data with a timestamp greater than the target timestamp from the data source table specifically includes:

[0011] Get the name of the data source table recorded in the configuration table, and build a query statement based on the name and target timestamp; based on the query statement, query the incremental data with a timestamp greater than the target timestamp from the data source table to obtain a query result set; if the query result set is not empty, use the data in the query result set as incremental data.

[0012] Furthermore, based on the preset unique field, the step of matching the synchronization field of the data source table with the data of the target table specifically includes:

[0013] The first primary key field value of the target table is obtained, and the second primary key field value of the synchronization field of the data source table is obtained; based on the first primary key field value, it is determined whether the second primary key field value exists in the target table.

[0014] Furthermore, based on the matching result, the step of synchronizing the synchronization fields of the data source table to the target table specifically includes:

[0015] If the second primary key field value does not exist in the target table, the synchronization field whose second primary key field value does not exist in the target table is inserted into the target table; if the second primary key field value already exists in the target table, the target table is updated based on the synchronization field whose second primary key field value already exists in the target table.

[0016] Furthermore, after the step of synchronizing the synchronization field to the target table based on the matching result, the method further includes:

[0017] Obtain target exception information of target exception data, where the target exception data refers to the synchronization fields that are not successfully synchronized during the synchronization of the synchronization fields of the data source table to the target table; obtain historical exception information of historical exception data within a historical time period, where the historical exception data refers to the historical synchronization fields that are not successfully synchronized during the synchronization of the historical synchronization fields of the data source table to the target table within the historical time period; group the target exception data and the historical exception data according to the primary key based on the target exception information and the historical exception information to obtain multiple exception groups; use an aggregate function to count the number of exception data in each exception group to obtain the number of exceptions corresponding to each primary key; determine whether the number of exceptions is greater than a preset threshold, and if so, trigger a warning.

[0018] Furthermore, after the step of synchronizing the synchronization fields of the data source table to the target table based on the matching result, the method further includes:

[0019] Get the maximum value of the timestamp field in the synchronization field of the current data source table; write the maximum value back to the timestamp field in the update configuration table, and use the maximum value as the starting timestamp for the next incremental data synchronization.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a data synchronization device, which adopts the following technical solutions:

[0021] An acquisition module is used to acquire a target timestamp recorded in a configuration table of a data management framework. The target timestamp is the maximum timestamp recorded in the configuration table during the last data synchronization process.

[0022] A query module is used to use the target timestamp as a screening condition to query the incremental data whose timestamp is greater than the target timestamp from the data source table;

[0023] A determination module is used to determine the synchronization fields that need to be synchronized to the target table from the incremental data of the data source table according to the field mapping relationship in the configuration table;

[0024] The matching module is used to match the synchronization fields of the data source table with the data of the target table based on a preset unique field, and synchronize the synchronization fields of the data source table to the target table based on the matching result.

[0025] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above data synchronization method are implemented.

[0026] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the above data synchronization method.

[0027] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: by efficiently utilizing the configuration table of the constructed data management framework (such as the ETL framework), the maximum timestamp of the last data synchronization process is accurately captured, and this is used as a screening condition to quickly locate and extract the incremental data updated by the timestamp from the data source table. This approach not only significantly improves the efficiency of data synchronization, but also effectively avoids unnecessary repeated data transmission, saving valuable network resources and processing time. At the same time, the embodiments of the present application make full use of the field mapping relationship in the configuration table to accurately identify the fields that need to be synchronized to the target table, ensuring the accuracy and pertinence of data synchronization. On this basis, data matching is performed through preset unique fields, which further improves the accuracy and reliability of data synchronization, effectively prevents data conflicts and omissions, and ensures the consistency and integrity of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0030] Figure 2 It is a flowchart of a data synchronization method provided by this application;

[0031] Figure 3 It is a structural diagram of a data synchronization device provided by the present application;

[0032] Figure 4 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0037] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0038] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0039] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0040] It should be noted that the data synchronization method provided in the embodiment of the present application is generally executed by a server, and accordingly, the data synchronization device is generally set in the server.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0042] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data synchronization method according to the present application. The data synchronization method comprises the following steps:

[0043] Step S201, obtaining a target timestamp recorded in a configuration table of a data management framework, where the target timestamp is the maximum timestamp recorded in the configuration table during the last data synchronization process.

[0044] In this embodiment, the electronic device (eg Figure 1 The server shown in the figure) can obtain the target timestamp recorded in the configuration table of the data management framework through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0045] The data management framework refers to a framework built based on configuration tables, general scheduling jobs, and target synchronization conversion files. The data management framework is an integrated management system architecture for data extraction, transformation, and loading, aiming to achieve automated data processing, integration, optimization, and data synchronization operations. For example, the data management framework can be an ETL framework (Extraction-Transformation-Loading Framework, ETL).

[0046] The configuration table refers to a table preset in the data management framework for recording and managing configuration information related to data synchronization tasks. The configuration table may include but is not limited to key parameters such as target timestamp and field mapping relationship, and is an important basis for guiding data synchronization operations.

[0047] The target timestamp refers to the maximum timestamp value in the data source table when the data was last synchronized in the configuration table. The target timestamp is used to identify the end time of the synchronized data and is the starting reference point for this data synchronization.

[0048] Step S202: Use the target timestamp as a screening condition to query the data source table for incremental data whose timestamp is greater than the target timestamp.

[0049] The filter condition refers to the condition set according to the target timestamp when querying data. It is used to filter out incremental data with a timestamp greater than the target timestamp from the data source table, which is a key step in achieving incremental data synchronization.

[0050] The data source table refers to the database table containing the original data and the data to be synchronized, and is the starting point of the data synchronization operation. It contains not only the original data but also the data records added since the last synchronization.

[0051] Among them, incremental data refers to data records in the data source table whose timestamp is greater than the target timestamp, that is, data that has changed since the last synchronization, and is the part that needs to be transmitted in this data synchronization.

[0052] Step S203: According to the field mapping relationship in the configuration table, the synchronization fields that need to be synchronized to the target table are determined from the incremental data in the data source table.

[0053] The field mapping relationship refers to the rules for the field correspondence between the data source table and the target table defined in the configuration table. It is used to guide how to map specific field values ​​in the data source table to the corresponding fields in the target table.

[0054] The target table refers to the destination of the data synchronization operation, that is, the database table where the data needs to be stored. The synchronization field values ​​in the incremental data can be written into it according to the field mapping relationship.

[0055] The synchronization fields refer to the fields that need to be transferred to the target table, which are determined from the incremental data of the data source table according to the field mapping relationship. They are the core content of the data synchronization operation.

[0056] Step S204, based on the preset unique field, the synchronization field of the data source table is matched with the data of the target table, and based on the matching result, the synchronization field of the data source table is synchronized to the target table.

[0057] Among them, the unique field refers to the field that exists in both the data source table and the target table and is used to uniquely identify each data record. It is the key basis for data matching and ensures the accuracy and consistency of data synchronization.

[0058] Matching refers to the process of comparing the synchronization fields of the data source table and the data of the target table based on unique fields. It is used to determine which data records already exist in the target table and which are new or updated records.

[0059] Synchronizing the synchronization fields to the target table refers to the process of writing the synchronization fields that meet the conditions in the data source table to the corresponding fields in the target table according to the matching results, which is to complete the incremental synchronization of data and ensure the real-time and accuracy of the target table data.

[0060] The embodiment of the present application can achieve accurate capture of the maximum timestamp during the last data synchronization process by efficiently utilizing the configuration table of the built data management framework (such as the ETL framework), and use this as a screening condition to quickly locate and extract the incremental data updated by the timestamp from the data source table. This approach not only significantly improves the efficiency of data synchronization, but also effectively avoids unnecessary repeated data transmission, saving valuable network resources and processing time. At the same time, the embodiment of the present application makes full use of the field mapping relationship in the configuration table to accurately identify the fields that need to be synchronized to the target table, ensuring the accuracy and pertinence of data synchronization. On this basis, data matching is performed through a preset unique field, which further improves the accuracy and reliability of data synchronization, effectively prevents data conflicts and omissions, and ensures the consistency and integrity of the data.

[0061] In some optional implementations of this embodiment, before obtaining the target timestamp recorded in the configuration table of the data management framework in step 201, the following steps are specifically included:

[0062] Get the preset key fields, and create a configuration table based on the key fields. The configuration table is used to record field mapping relationships, synchronize general scheduling job information, and record timestamps; get the preset job parameters, variables, and property files, and create a general scheduling job based on the job parameters, variables, and property files. The general scheduling job is used to cyclically capture the information that needs to be synchronized in the configuration table, and perform data synchronization operations in sequence; get the preset synchronization conversion template, which includes multiple key controls and the configuration file and path information of each key control; get the target attribute information; based on the target attribute information, update the configuration file and path information of each key control of the synchronization conversion template, and generate the target synchronization conversion file; build a data management framework based on the configuration table, general scheduling job, and target synchronization conversion file.

[0063] In one example, in a financial system, the key fields of the data to be synchronized can be first determined, such as the job name, job description, data source table, data target table, synchronization start time, synchronization end time, record increment timestamp, synchronization program file name, synchronization frequency, and whether the synchronization job is valid. Based on the above key fields, a configuration table is created in the database to record the field mapping relationship (such as the correspondence between the fields in the data source table and the fields in the target table), the information of the synchronization general scheduling job (such as the job name, execution frequency, etc.), and the record timestamp (used to identify the last execution time of data synchronization). Next, the preset job parameters (such as data source address, target data source address), variables (such as the number of cycles, error handling strategy) and property files (such as database connection information) can be obtained, and a general scheduling job is created based on this information. The job can cyclically capture the information that needs to be synchronized in the configuration table and perform data synchronization operations in sequence. Next, in the process of financial data synchronization, it is usually necessary to convert or clean the data format. To this end, this embodiment can provide a preset synchronization conversion template, which contains multiple key controls (such as data cleaning rules, format conversion rules) and configuration files and path information for each key control. By obtaining target attribute information (such as target data format, cleaning rules, etc.), the configuration file and path information of each key control in the synchronization conversion template are updated to generate the target synchronization conversion file. Finally, according to the configuration table, general scheduling job and target synchronization conversion file, a data management framework (such as ETL framework) is constructed. The data management framework can automatically perform data synchronization and conversion tasks to ensure the accuracy and timeliness of financial data.

[0064] The embodiment of the present application can achieve efficient management of field mapping relationships, synchronization job information and timestamps by obtaining preset key fields and creating a configuration table, providing a solid foundation for the synchronization operation of financial data. By creating a general scheduling job, the cyclic capture and synchronization of data is achieved, which significantly improves the efficiency of data processing. At the same time, the target synchronization conversion file is generated using the synchronization conversion template and target attribute information, which realizes the flexible conversion and cleaning of data and meets the strict requirements of the financial field for data format and quality. Finally, based on the configuration table, general scheduling job and target synchronization conversion file, a powerful data management framework is constructed, which provides strong support for the integration, management, analysis and data synchronization of financial data.

[0065] In some optional implementations of this embodiment, step S202 uses the target timestamp as a screening condition to query the data source table for incremental data whose timestamp is greater than the target timestamp, which specifically includes the following steps:

[0066] Get the name of the data source table recorded in the configuration table, and build a query statement based on the name and target timestamp; based on the query statement, query the incremental data with a timestamp greater than the target timestamp from the data source table to obtain a query result set; if the query result set is not empty, use the data in the query result set as incremental data.

[0067] In one example, the recorded data source table name and target timestamp can be obtained from the configuration table. The configuration table is a pre-set database table used to store information such as the name of the data source table, the timestamp field name, and the target timestamp. Based on the obtained data source table name and target timestamp, a SQL query statement is constructed. The query statement is used to filter out incremental data with a timestamp greater than the target timestamp from the data source table. The constructed query statement is submitted to the database for execution, and the incremental data that meets the conditions is queried from the data source table to obtain a query result set. If the query result set is empty, it is determined that no new data needs to be synchronized this time, and the synchronization process ends. If the query result set is not empty, it is determined that there is new data that needs to be synchronized this time, and the data in the query result set is used as incremental data for subsequent processing, such as data synchronization, data analysis, etc. For example, taking a transaction data synchronization project of a securities company as an example, the method of this embodiment is applied to the company's transaction data processing system. In this system, the configuration table records the name of each data source table, the timestamp field name, and the target timestamp of the last data synchronization. Whenever data synchronization is required, the system will first obtain the name of the trade data source table (such as "trade_data") and the target timestamp (such as "2023-04-01 00:00:00") from the configuration table. Next, the system will construct an SQL query statement based on this information, such as: "SELECT*FROM trade_dataWHERE timestamp>'2023-04-01 00:00:00'". Then, the system submits this query statement to the database for execution, and queries all incremental transaction data with a timestamp greater than "2023-04-01 00:00:00" from the trade data source table to obtain a query result set. Finally, the system will check the query result set. If the query result set is not empty, it indicates that there is incremental data that needs to be synchronized. At this point, the system will use the data in the query result set as incremental data for subsequent data synchronization and processing operations.

[0068] The embodiment of the present application can construct a targeted query statement by dynamically obtaining the data source table name and the target timestamp from the configuration table, thereby realizing the accurate capture of incremental data in the data source table. This method avoids the redundancy and inefficiency of full data synchronization under the traditional method, and significantly improves the data processing efficiency. At the same time, by checking whether the query result set is empty, it is possible to accurately determine whether there is incremental data, thereby ensuring the timeliness and accuracy of data synchronization. In addition, the technical solution also has good flexibility and scalability, and can easily adapt to the change requirements of different data source tables and timestamp fields, providing strong technical support for industries such as finance and e-commerce that need to efficiently process large-scale, high-frequency updated data. Through this solution, data resources can be managed more effectively, and business operation efficiency and decision-making accuracy can be improved.

[0069] In some optional implementations of this embodiment, step S204, matching the synchronization field of the data source table with the data of the target table based on a preset unique field, specifically includes the following steps:

[0070] The first primary key field value of the target table is obtained, and the second primary key field value of the synchronization field of the data source table is obtained; based on the first primary key field value, it is determined whether the second primary key field value exists in the target table.

[0071] In one example, the first primary key field value can be obtained from the target table, and the field value is used to uniquely identify the record in the target table. At the same time, the second primary key field value of the synchronization field is obtained from the data source table, and the field value is used to identify the record that needs to be synchronized in the data source table. Then, based on the obtained first primary key field value, it is determined whether the second primary key field value in the data source table exists in the target table. This step can be implemented by constructing a query statement, executing a database operation, etc. For example, taking the synchronization of customer transaction data of a securities company as an example, the method of this embodiment can be applied to the transaction data processing system of the company. In this system, the target table is the "customer transaction record table", which is used to store the transaction records of all customers, and its primary key field is the "transaction record ID". The data source table is the "transaction change record table", which is used to record the changes in customer transaction records, and its synchronization field is the "change transaction record ID", which corresponds to the "transaction record ID" field in the "customer transaction record table". Whenever there is a new change record in the "transaction change record table", the system will first perform the following steps: obtain all existing "transaction record ID" values ​​from the "customer transaction record table" as the first primary key field value. At the same time, the "Change Transaction Record ID" value that needs to be synchronized is obtained from the "Transaction Change Record Table" as the second primary key field value. The system will construct a query statement based on the obtained first primary key field value to query whether the same "Transaction Record ID" exists in the "Customer Transaction Record Table".

[0072] The embodiment of the present application can achieve efficient and accurate cross-table data synchronization by accurately obtaining the first primary key field value of the target table and the second primary key field value of the synchronization field of the data source table. Based on the judgment mechanism of the first primary key field value, it is possible to quickly identify which records in the data source table do not yet exist in the target table and which already exist but may have changed. This method not only avoids the tedious full-table scan and comparison in the traditional data synchronization process, significantly improves the efficiency of data synchronization, but also ensures the accuracy of data synchronization and reduces business risks caused by data inconsistency.

[0073] In some optional implementations of this embodiment, step S204, based on the matching result, synchronizing the synchronization fields of the data source table to the target table, specifically includes the following steps:

[0074] If the second primary key field value does not exist in the target table, the synchronization field whose second primary key field value does not exist in the target table is inserted into the target table; if the second primary key field value already exists in the target table, the target table is updated based on the synchronization field whose second primary key field value already exists in the target table.

[0075] In one example, if the second primary key field value does not exist in the target table, it indicates that the record in the data source table is newly added and needs to be synchronized to the target table. At this time, the synchronization field (that is, other related fields in the data source table) whose second primary key field value does not exist in the target table can be inserted into the target table to complete the synchronization of new data. If the second primary key field value already exists in the target table, it indicates that the record in the data source table already exists in the target table, but may have been changed. At this time, it is necessary to update the corresponding record in the target table based on the synchronization field (that is, the field in the data source table that has changed) whose second primary key field value already exists in the target table to complete the update synchronization of data.

[0076] The embodiment of the present application can achieve accurate synchronization and efficient management of data by intelligently judging the existence status of the second primary key field value in the target table. When the second primary key field value does not exist in the target table, the corresponding synchronization field data will be automatically inserted into the target table, ensuring the timely entry and integrity of new data. When the second primary key field value already exists in the target table, the target table is updated based on the data of the synchronization field, achieving real-time update and accuracy maintenance of the data. This intelligent synchronization strategy based on the primary key field not only avoids repeated entry and redundancy of data, but also significantly improves the efficiency and accuracy of data synchronization.

[0077] In some optional implementations of this embodiment, after synchronizing the synchronization field to the target table based on the matching result in step S204, the following steps are specifically included:

[0078] Obtain target exception information of target exception data, where the target exception data refers to the synchronization fields that are not successfully synchronized during the synchronization of the synchronization fields of the data source table to the target table; obtain historical exception information of historical exception data within a historical time period, where the historical exception data refers to the historical synchronization fields that are not successfully synchronized during the synchronization of the historical synchronization fields of the data source table to the target table within the historical time period; group the target exception data and the historical exception data according to the primary key based on the target exception information and the historical exception information to obtain multiple exception groups; use an aggregate function to count the number of exception data in each exception group to obtain the number of exceptions corresponding to each primary key; determine whether the number of exceptions is greater than a preset threshold, and if so, trigger a warning.

[0079] In one example, target exception information of target exception data can be obtained, which are synchronization fields of the data source table that were not successfully synchronized during synchronization to the target table. At the same time, historical exception information of historical exception data within a historical time period can be obtained, which are historical synchronization fields of the data source table that were not successfully synchronized during synchronization to the target table during synchronization to the target table.

[0080] Next, based on the acquired target exception information and historical exception information, the target exception data and historical exception data are grouped according to the primary key to obtain multiple exception groups. The primary key is a field used to uniquely identify each record in the data table. Grouping by primary key can ensure that the exception data under the same primary key are classified together. Then, use aggregate functions such as COUNT to count the number of exception data in each exception group and obtain the number of exceptions corresponding to each primary key. This step helps to understand which primary keys have more serious data synchronization problems and provide a basis for subsequent processing.

[0081] Finally, it is determined whether the number of abnormalities is greater than a preset threshold. If it is greater than the threshold, a warning is triggered to prompt relevant personnel to deal with the data synchronization problem in a timely manner. For example, taking the synchronization of customer transaction data of a bank as an example, the method of this embodiment can be applied to the transaction data processing system of the bank. In this system, the data source table is the "transaction change record table" and the target table is the "customer transaction record table". Whenever there is a new transaction record change in the "transaction change record table", these changes will be synchronized to the "customer transaction record table". However, in the actual synchronization process, due to network delays or data format mismatches, the synchronization of some transaction records will fail, forming abnormal data. In order to effectively monitor and process these abnormal data, the synchronization fields that have not been successfully synchronized will first be obtained from the "customer transaction record table" as the target abnormal data, and the historical synchronization fields that have not been successfully synchronized will be obtained from the "transaction change record table" and the "customer transaction record table" in the historical time period as the historical abnormal data. Then, the system will group these abnormal data according to the primary key (such as the transaction record ID) to obtain multiple abnormal groups. Next, the system uses the COUNT aggregation function to count the number of abnormal data in each abnormal group and obtain the number of abnormal times corresponding to each transaction record ID. Finally, the system determines whether the number of abnormal times is greater than the preset threshold (such as 5 times). If it is greater than the threshold, a warning is triggered, prompting the bank's data processing personnel to promptly handle the data synchronization problem under the transaction record ID.

[0082] The embodiments of the present application can achieve comprehensive monitoring of abnormal situations during data synchronization by accurately capturing target abnormal data and historical abnormal data. The grouping strategy based on the primary key effectively classifies and organizes abnormal data, providing a clear data structure for subsequent analysis. The use of aggregate functions to count the number of abnormalities corresponding to each primary key not only intuitively shows the frequency of data synchronization problems, but also provides a reliable basis for judgment for the early warning mechanism. When the number of abnormalities exceeds the preset threshold, a warning is automatically triggered to promptly remind relevant personnel to pay attention to and deal with potential problems in data synchronization, effectively reducing the risk of data inconsistency and improving the reliability and stability of data synchronization.

[0083] In some optional implementations of this embodiment, in step S204, based on the matching result, the synchronization fields of the data source table are synchronized to the target table, which specifically includes the following steps:

[0084] Get the maximum value of the timestamp field in the synchronization field of the current data source table; write the maximum value back to the timestamp field in the update configuration table, and use the maximum value as the starting timestamp for the next incremental data synchronization.

[0085] In one example, the timestamp field can be retrieved from the synchronization field of the current data source table, and the maximum value of the field can be obtained. This maximum value represents the timestamp of the latest data in the data source table and is the benchmark for the next incremental data synchronization. Then, the maximum value of the timestamp obtained is written back and updated to the timestamp field in the configuration table. By updating the timestamp field in the configuration table, the end point of the last synchronization is remembered, and an accurate starting timestamp is provided for the next synchronization. Finally, when the data synchronization operation is required next time, the new or changed data after the timestamp is filtered out from the data source table according to the timestamp field in the configuration table, and these data are synchronized to the target table. In this way, an efficient incremental data synchronization is completed. For example, taking the transaction data synchronization of a bank as an example, the method of this embodiment can be applied to the transaction data processing system of the bank. In this system, the data source table is the "real-time transaction record table", the target table is the "historical transaction record table", and the configuration table is the "synchronization configuration information table". Whenever a new transaction record is generated, the system will insert it into the "real-time transaction record table" and update the timestamp field of the record. In order to efficiently synchronize these newly added transaction records to the "Historical Transaction Record Table", the system will first retrieve the timestamp field from the synchronization field of the "Real-time Transaction Record Table" and obtain the maximum value of the field. Then, the system will write back this maximum value to update the timestamp field in the "Synchronization Configuration Information Table". In this way, the system knows the end point of the last synchronization, that is, the timestamp of the latest transaction record in the "Real-time Transaction Record Table" at the time of the last synchronization. Next, when the next data synchronization operation is required, the system will filter out the newly added transaction records after the timestamp from the "Real-time Transaction Record Table" according to the timestamp field in the "Synchronization Configuration Information Table". These newly added transaction records are the data that need to be synchronized to the "Historical Transaction Record Table" this time. Finally, the system inserts these newly added transaction records into the "Historical Transaction Record Table", completing an efficient data incremental synchronization.

[0086] The embodiment of the present application can achieve precise control of the starting point of data incremental synchronization by accurately capturing the maximum timestamp value of the synchronization field in the data source table. Writing back the captured maximum timestamp value to the configuration table not only ensures the continuity of the synchronization process, but also provides an accurate starting timestamp for the next incremental data synchronization. This strategy effectively avoids the problems of duplicate synchronization or data omission that may occur in traditional data synchronization methods, and significantly improves the accuracy and efficiency of data synchronization.

[0087] It should be emphasized that in order to further ensure the privacy and security of the configuration table, target timestamp and synchronization fields of the data source table of the above-mentioned data management framework, the configuration table, target timestamp and synchronization fields of the data source table of the above-mentioned data management framework can also be stored in a node of a blockchain.

[0088] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0089] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0090] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0091] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a data synchronization device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0092] like Figure 3 As shown, the data synchronization device 400 of this embodiment includes: an acquisition module 401, a query module 402, a determination module 403 and a matching module 404. Among them:

[0093] Acquisition module 401, used to acquire a target timestamp recorded in a configuration table of a data management framework, where the target timestamp is the maximum timestamp recorded in the configuration table during the last data synchronization process;

[0094] A query module 402 is used to use the target timestamp as a screening condition to query the incremental data with a timestamp greater than the target timestamp from the data source table;

[0095] Determination module 403, used to determine the synchronization fields that need to be synchronized to the target table from the incremental data of the data source table according to the field mapping relationship in the configuration table;

[0096] The matching module 404 is used to match the synchronization field of the data source table with the data of the target table based on a preset unique field, and synchronize the synchronization field of the data source table to the target table based on the matching result.

[0097] The embodiment of the present application can achieve accurate capture of the maximum timestamp during the last data synchronization process by efficiently utilizing the configuration table of the built data management framework (such as the ETL framework), and use this as a screening condition to quickly locate and extract the incremental data updated by the timestamp from the data source table. This approach not only significantly improves the efficiency of data synchronization, but also effectively avoids unnecessary repeated data transmission, saving valuable network resources and processing time. At the same time, the embodiment of the present application makes full use of the field mapping relationship in the configuration table to accurately identify the fields that need to be synchronized to the target table, ensuring the accuracy and pertinence of data synchronization. On this basis, data matching is performed through a preset unique field, which further improves the accuracy and reliability of data synchronization, effectively prevents data conflicts and omissions, and ensures the consistency and integrity of the data.

[0098] In one embodiment, the query module 402 includes:

[0099] The first acquisition submodule is used to obtain the name of the data source table recorded in the configuration table, and construct a query statement based on the name and the target timestamp;

[0100] The query submodule is used to query the incremental data whose timestamp is greater than the target timestamp from the data source table based on the query statement to obtain the query result set;

[0101] As a submodule, it is used to use the data in the query result set as incremental data if the query result set is not empty.

[0102] The embodiment of the present application can construct a targeted query statement by dynamically obtaining the data source table name and the target timestamp from the configuration table, thereby realizing the accurate capture of incremental data in the data source table. This method avoids the redundancy and inefficiency of full data synchronization under the traditional method, and significantly improves the data processing efficiency. At the same time, by checking whether the query result set is empty, it is possible to accurately determine whether there is incremental data, thereby ensuring the timeliness and accuracy of data synchronization. In addition, the technical solution also has good flexibility and scalability, and can easily adapt to the change requirements of different data source tables and timestamp fields, providing strong technical support for industries such as finance and e-commerce that need to efficiently process large-scale, high-frequency updated data. Through this solution, data resources can be managed more effectively, and business operation efficiency and decision-making accuracy can be improved.

[0103] In one embodiment, the matching module 404 includes:

[0104] The second acquisition submodule is used to acquire the first primary key field value of the target table and the second primary key field value of the synchronization field of the data source table;

[0105] The judgment submodule is used to judge whether the second primary key field value exists in the target table based on the first primary key field value.

[0106] The embodiment of the present application can achieve efficient and accurate cross-table data synchronization by accurately obtaining the first primary key field value of the target table and the second primary key field value of the synchronization field of the data source table. Based on the judgment mechanism of the first primary key field value, it is possible to quickly identify which records in the data source table do not yet exist in the target table and which already exist but may have changed. This method not only avoids the tedious full-table scan and comparison in the traditional data synchronization process, significantly improves the efficiency of data synchronization, but also ensures the accuracy of data synchronization and reduces business risks caused by data inconsistency.

[0107] In one embodiment, the matching module 404 includes:

[0108] The insert submodule is used for inserting the synchronization field whose second primary key field value does not exist in the target table into the target table if the second primary key field value does not exist in the target table;

[0109] The update submodule is used to update the target table based on the synchronization field whose value of the second primary key field already exists in the target table if the value of the second primary key field already exists in the target table.

[0110] The embodiment of the present application can achieve accurate synchronization and efficient management of data by intelligently judging the existence status of the second primary key field value in the target table. When the second primary key field value does not exist in the target table, the corresponding synchronization field data will be automatically inserted into the target table, ensuring the timely entry and integrity of new data. When the second primary key field value already exists in the target table, the target table is updated based on the data of the synchronization field, achieving real-time update and accuracy maintenance of the data. This intelligent synchronization strategy based on the primary key field not only avoids repeated entry and redundancy of data, but also significantly improves the efficiency and accuracy of data synchronization.

[0111] In one embodiment, the data synchronization device further includes:

[0112] The field acquisition module is used to obtain preset key fields and create a configuration table based on the key fields. The configuration table is used to record field mapping relationships, synchronize general scheduling job information, and record timestamps.

[0113] Create a module to obtain preset job parameters, variables and property files, and create a general scheduling job based on the job parameters, variables and property files. The general scheduling job is used to cyclically capture the information that needs to be synchronized in the configuration table and perform data synchronization operations in sequence;

[0114] The template module is used to obtain a preset synchronous conversion template, which includes multiple key controls and a configuration file and path information of each key control; obtain target attribute information; based on the target attribute information, update the configuration file and path information of each key control of the synchronous conversion template to generate a target synchronous conversion file;

[0115] A building module is used to build a data management framework based on configuration tables, common scheduling jobs, and target synchronization transformation files.

[0116] The embodiment of the present application can achieve efficient management of field mapping relationships, synchronization job information and timestamps by obtaining preset key fields and creating a configuration table, providing a solid foundation for the synchronization operation of financial data. By creating a general scheduling job, the cyclic capture and synchronization of data is achieved, which significantly improves the efficiency of data processing. At the same time, the target synchronization conversion file is generated using the synchronization conversion template and target attribute information, which realizes the flexible conversion and cleaning of data and meets the strict requirements of the financial field for data format and quality. Finally, based on the configuration table, general scheduling job and target synchronization conversion file, a powerful data management framework is constructed, which provides strong support for the integration, management, analysis and data synchronization of financial data.

[0117] In one embodiment, the data synchronization device further includes:

[0118] A first information acquisition module is used to acquire target abnormality information of target abnormal data, where the target abnormal data is the synchronization field that is not successfully synchronized during the synchronization of the synchronization field of the data source table to the target table;

[0119] The second information acquisition module is used to acquire historical abnormal information of historical abnormal data within a historical time period, where the historical abnormal data is the historical synchronization fields that were not successfully synchronized during the synchronization of the historical synchronization fields of the data source table to the target table within the historical time period;

[0120] A grouping module is used to group the target abnormal data and the historical abnormal data according to the primary key based on the target abnormal information and the historical abnormal information to obtain multiple abnormal groups;

[0121] The statistics module is used to use the aggregation function to count the number of abnormal data in each abnormal group and obtain the number of abnormal times corresponding to each primary key;

[0122] The judgment module is used to judge whether the number of abnormalities is greater than a preset threshold. If it is greater than the threshold, a warning is triggered.

[0123] The embodiments of the present application can achieve comprehensive monitoring of abnormal situations during data synchronization by accurately capturing target abnormal data and historical abnormal data. The grouping strategy based on the primary key effectively classifies and organizes abnormal data, providing a clear data structure for subsequent analysis. The use of aggregate functions to count the number of abnormalities corresponding to each primary key not only intuitively shows the frequency of data synchronization problems, but also provides a reliable basis for judgment for the early warning mechanism. When the number of abnormalities exceeds the preset threshold, a warning is automatically triggered to promptly remind relevant personnel to pay attention to and deal with potential problems in data synchronization, effectively reducing the risk of data inconsistency and improving the reliability and stability of data synchronization.

[0124] In one embodiment, the data synchronization device further includes:

[0125] The value acquisition module is used to obtain the maximum value of the timestamp field in the synchronization field of the current data source table;

[0126] The write-back module is used to write back the maximum value to the timestamp field in the update configuration table and use the maximum value as the starting timestamp for the next incremental data synchronization.

[0127] The embodiment of the present application can achieve precise control of the starting point of data incremental synchronization by accurately capturing the maximum timestamp value of the synchronization field in the data source table. Writing back the captured maximum timestamp value to the configuration table not only ensures the continuity of the synchronization process, but also provides an accurate starting timestamp for the next incremental data synchronization. This strategy effectively avoids the problems of duplicate synchronization or data omission that may occur in traditional data synchronization methods, and significantly improves the accuracy and efficiency of data synchronization.

[0128] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0129] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0130] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0131] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the data synchronization method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0132] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running a data synchronization method.

[0133] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0134] The embodiment of the present application can achieve accurate capture of the maximum timestamp during the last data synchronization process by efficiently utilizing the configuration table of the built data management framework (such as the ETL framework), and use this as a screening condition to quickly locate and extract the incremental data updated by the timestamp from the data source table. This approach not only significantly improves the efficiency of data synchronization, but also effectively avoids unnecessary repeated data transmission, saving valuable network resources and processing time. At the same time, the embodiment of the present application makes full use of the field mapping relationship in the configuration table to accurately identify the fields that need to be synchronized to the target table, ensuring the accuracy and pertinence of data synchronization. On this basis, data matching is performed through a preset unique field, which further improves the accuracy and reliability of data synchronization, effectively prevents data conflicts and omissions, and ensures the consistency and integrity of the data.

[0135] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data synchronization method as described above.

[0136] The embodiment of the present application can achieve accurate capture of the maximum timestamp during the last data synchronization process by efficiently utilizing the configuration table of the built data management framework (such as the ETL framework), and use this as a screening condition to quickly locate and extract the incremental data updated by the timestamp from the data source table. This approach not only significantly improves the efficiency of data synchronization, but also effectively avoids unnecessary repeated data transmission, saving valuable network resources and processing time. At the same time, the embodiment of the present application makes full use of the field mapping relationship in the configuration table to accurately identify the fields that need to be synchronized to the target table, ensuring the accuracy and pertinence of data synchronization. On this basis, data matching is performed through a preset unique field, which further improves the accuracy and reliability of data synchronization, effectively prevents data conflicts and omissions, and ensures the consistency and integrity of the data.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0138] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0139] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data synchronization method, characterized in that: The steps include: Obtain a target timestamp recorded in a configuration table of a data management framework, wherein the target timestamp is the maximum timestamp recorded in the configuration table during the last data synchronization process; Using the target timestamp as a screening condition, querying the data source table for incremental data whose timestamp is greater than the target timestamp; According to the field mapping relationship in the configuration table, determining the synchronization fields that need to be synchronized to the target table from the incremental data in the data source table; Based on a preset unique field, the synchronization field of the data source table is matched with the data of the target table, and based on the matching result, the synchronization field of the data source table is synchronized to the target table.

2. The method according to claim 1, characterized in that Before the step of obtaining the target timestamp recorded in the configuration table of the data management framework, the method further includes: Obtain preset key fields, and create a configuration table based on the key fields, wherein the configuration table is used to record field mapping relationships, synchronize general scheduling job information, and record timestamps; Obtaining preset job parameters, variables and property files, and creating the general scheduling job based on the job parameters, variables and property files, wherein the general scheduling job is used to cyclically capture the information to be synchronized in the configuration table, and perform data synchronization operations in sequence; Obtain a preset synchronous conversion template, the synchronous conversion template including multiple key controls and a configuration file and path information of each key control; obtain target attribute information; based on the target attribute information, update the configuration file and path information of each key control of the synchronous conversion template to generate a target synchronous conversion file; A data management framework is constructed according to the configuration table, the general scheduling job and the target synchronization conversion file.

3. The method according to claim 1, characterized in that The step of using the target timestamp as a screening condition to query the incremental data with a timestamp greater than the target timestamp from the data source table specifically includes: Obtain the name of the data source table recorded in the configuration table, and construct a query statement based on the name and the target timestamp; Based on the query statement, query the incremental data whose timestamp is greater than the target timestamp from the data source table to obtain a query result set; If the query result set is not empty, the data in the query result set is used as incremental data.

4. The method according to claim 1, characterized in that: The step of matching the synchronization field of the data source table with the data of the target table based on the preset unique field specifically includes: Acquire the first primary key field value of the target table, and acquire the second primary key field value of the synchronization field of the data source table; Based on the first primary key field value, determine whether the second primary key field value exists in the target table.

5. The method according to claim 4, characterized in that The step of synchronizing the synchronization fields of the data source table to the target table based on the matching result specifically includes: If the second primary key field value does not exist in the target table, inserting the synchronization field whose second primary key field value does not exist in the target table into the target table; If the second primary key field value already exists in the target table, the target table is updated based on the synchronization field in which the second primary key field value already exists in the target table.

6. The method according to claim 1, characterized in that After the step of synchronizing the synchronization field to the target table based on the matching result, the method further includes: Acquire target exception information of target exception data, where the target exception data is a synchronization field that is not successfully synchronized during the synchronization of the synchronization field of the data source table to the target table; Acquire historical abnormal information of historical abnormal data within a historical time period, where the historical abnormal data is historical synchronization fields that were not successfully synchronized during synchronization of historical synchronization fields of the data source table to the target table within the historical time period; Based on the target abnormality information and the historical abnormality information, the target abnormality data and the historical abnormality data are grouped according to the primary key to obtain a plurality of abnormality groups; Use the aggregation function to count the number of abnormal data in each abnormal group and obtain the number of abnormal times corresponding to each primary key; It is determined whether the number of abnormalities is greater than a preset threshold. If so, a warning is triggered.

7. The method according to claim 1, characterized in that After the step of synchronizing the synchronization fields of the data source table to the target table based on the matching result, the method further includes: Obtain the maximum value of the timestamp field in the synchronization field of the current data source table; The maximum value is written back to update the timestamp field in the configuration table, and the maximum value is used as the starting timestamp for the next data incremental synchronization.

8. A data synchronization device, characterized in that: include: An acquisition module, used to acquire a target timestamp recorded in a configuration table of a data management framework, wherein the target timestamp is a maximum timestamp recorded in the configuration table during the last data synchronization process; A query module, configured to use the target timestamp as a screening condition to query the incremental data whose timestamp is greater than the target timestamp from the data source table; A determination module, configured to determine the synchronization fields that need to be synchronized to the target table from the incremental data of the data source table according to the field mapping relationship in the configuration table; The matching module is used to match the synchronization field of the data source table with the data of the target table based on a preset unique field, and synchronize the synchronization field of the data source table to the target table based on the matching result.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data synchronization method as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 7 are implemented.

Images