Data synchronization method and device, electronic equipment and storage medium

Abstract

The invention discloses a data synchronization method and device, electronic equipment and a storage medium, and relates to the technical field of big data or other related fields, and the method comprises the steps: determining a to-be-synchronized target data table, and obtaining a table identifier of the target data table; calling a configuration table of the target data table based on the table identifier, and analyzing the configuration table to obtain data synchronization information of the target data table; a data synchronization task is constructed based on the data synchronization information, and the data synchronization task at least comprises data export, data comparison, data import and data verification; and executing the data synchronization task, and synchronizing the target data table from the source database to the target database. According to the method and the device, the technical problems of low development efficiency, high maintenance cost and poor expansibility caused by the fact that a data synchronization method between database tables is mainly realized through manual programming and a synchronization rule of each table needs to be independently coded in the related technology are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data or other related technical fields, in particular, to a data synchronization method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the financial industry, with the continuous growth of business transactions, the demand for rapid processing, backup and analysis of data is increasing. With the increase of business transactions, the amount of data in the database will increase sharply. In order to improve the response speed of the transaction system, the transaction database usually adopts the "hot data" strategy, that is, only the transaction data in the recent period is retained, and the historical data is migrated to the historical database. This requires efficient and reliable heterogeneous database table data synchronization technology to ensure that data is migrated from the transaction database to the historical database in a timely and accurate manner, so as to not occupy the transaction system resources and maintain its efficient operation. At the same time, data synchronization is the key to ensuring the consistency of data in heterogeneous databases (such as transaction databases and historical databases, databases between different business modules). In the financial field, the accuracy and consistency of data are crucial, and any inconsistency in data may cause transaction errors, regulatory problems or customer service problems. Through data synchronization, the data between the source database and the target database can be ensured to be consistent, especially when dealing with complex cross-business data.

[0003] In related technologies, the database table data synchronization method is mainly realized by manual programming, and the synchronization rules of each table need to be coded separately, which leads to code redundancy, low development efficiency, high maintenance cost, poor scalability and other problems. In addition, the existing data synchronization technology lacks effective exception monitoring and data consistency verification mechanism, which threatens the data security and real-time performance.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] The embodiments of the present application provide a data synchronization method and device, electronic equipment and storage medium to at least solve the technical problems in related technologies that the database table data synchronization method is mainly realized by manual programming, and the synchronization rules of each table need to be coded separately, which leads to low development efficiency, high maintenance cost and poor scalability.

[0006] According to one aspect of the present invention, a data synchronization method is provided, comprising: determining a target data table to be synchronized and obtaining a table identifier of the target data table; calling a configuration table of the target data table based on the table identifier, parsing the configuration table to obtain data synchronization information of the target data table; constructing a data synchronization task based on the data synchronization information, wherein the data synchronization task includes at least: data export, data comparison, data import, and data verification; and executing the data synchronization task to synchronize the target data table from a source database to a target database.

[0007] Furthermore, the configuration table is pre-built. The steps for building the configuration table include: obtaining all data tables in the database and configuring key fields for each data table. The key fields include at least: table identifier, business identifier of the business corresponding to the data table, import / export type, synchronization type, source database type, target database type, file naming rules, file sending path, field validation strategy, and anomaly detection strategy. The field validation strategy is used to verify whether data fields have been tampered with, and the anomaly detection strategy is used to detect abnormal changes in data volume. Based on the key fields, a configuration table corresponding to the data table is generated, and the field information corresponding to each key field in the configuration table is filled in based on the real-time monitoring information of the data table.

[0008] Further, the steps of executing the data synchronization task to synchronize the target data table from the source database to the target database include: filtering the data to be exported from the target data table based on the synchronization type in the configuration table, constructing a synchronization file based on the data to be exported, and naming the synchronization file according to the file naming rules in the configuration table; comparing the number of data rows in the synchronization file with the number of rows corresponding to the data to be exported in the data table to obtain a row count comparison result; if the row count comparison result indicates that the number of data rows in the synchronization file is the same as the number of rows corresponding to the data to be exported in the data table, sending the synchronization file to the target database according to the file sending path in the configuration table; importing the data in the synchronization file into the target database according to the synchronization type in the configuration table, and verifying the data table imported into the database.

[0009] Further, the step of filtering the data to be exported from the target data table based on the synchronization type in the configuration table, and constructing a synchronization file based on the data to be exported includes: when the synchronization type in the configuration table is full synchronization, using all the data in the target data table as the data to be exported, and constructing a full synchronization file based on the target data table; when the synchronization type in the configuration table is incremental synchronization, filtering incremental data from the target data table as the data to be exported based on the incremental conditions in the configuration table, and constructing an incremental synchronization file based on the incremental data.

[0010] Further, the steps of importing data from the synchronization file into the target database according to the synchronization type in the configuration table, and verifying the data table imported into the database, include: if the synchronization type in the configuration table is full synchronization, constructing a backup table in the target database with the same structure as the target data table; importing data from the full synchronization file into the backup table; performing row count verification on the backup table and the target data table; and performing field verification on the data in the backup table using the field verification strategy to obtain a verification result; if the verification result indicates that the backup table passes the verification, naming the backup table with the name of the target data table.

[0011] Further, the steps of importing data from the synchronization file into the target database according to the synchronization type in the configuration table, and verifying the data table imported into the database, include: if the synchronization type in the configuration table is incremental synchronization, cleaning the data in the target database according to the incremental conditions in the configuration table; importing the data from the incremental synchronization file into the cleaned target database; verifying the number of rows in the incremental data imported into the target database and the target data table; and using the field verification strategy to verify the fields of the incremental data imported into the target database.

[0012] Furthermore, after synchronizing the target data table from the source database to the target database, the method further includes: for database tables configured with anomaly detection strategies, collecting business announcement information and incremental value data within historical time periods related to the database tables; extracting business features from the business announcement information and constructing a business vector based on the business features, and constructing a trend vector based on the incremental value data; inputting the business vector and the trend vector into a prediction model to output the theoretical incremental value of the database table, wherein the prediction model is a pre-built machine learning model that integrates trend features, seasonal features, holiday features, and business variables to predict incremental values; obtaining the actual incremental value of the database table and calculating the difference between the theoretical incremental value and the actual incremental value to obtain an incremental deviation value; comparing the incremental deviation value with a deviation threshold, and triggering an anomaly warning for the database table if the incremental deviation value of the database table is greater than the deviation threshold.

[0013] According to another aspect of the present invention, a data synchronization apparatus is also provided, comprising: a determining unit, configured to determine a target data table to be synchronized and obtain a table identifier of the target data table; a parsing unit, configured to call a configuration table of the target data table based on the table identifier, parse the configuration table, and obtain data synchronization information of the target data table; a constructing unit, configured to construct a data synchronization task based on the data synchronization information, wherein the data synchronization task includes at least: data export, data comparison, data import, and data verification; and a synchronization unit, configured to execute the data synchronization task and synchronize the target data table from a source database to a target database.

[0014] Furthermore, the data synchronization device further includes: a first acquisition module, used to acquire all data tables in the database and configure key fields for each data table, wherein the key fields include at least: table identifier, business identifier of the business corresponding to the data table, import / export type, synchronization type, source database type, target database type, file naming rule, file sending path, field verification strategy, and anomaly detection strategy, wherein the field verification strategy is used to verify whether data fields have been tampered with, and the anomaly detection strategy is used to detect abnormal changes in data volume; and a first generation module, used to generate a configuration table corresponding to the data table based on the key fields, and fill in the field information corresponding to each key field in the configuration table based on the real-time monitoring information of the data table.

[0015] Further, the synchronization unit includes: a first filtering module, used to filter the data to be exported from the target data table based on the synchronization type in the configuration table, and to construct a synchronization file based on the data to be exported, and to name the synchronization file according to the file naming rules in the configuration table; a first comparison module, used to compare the number of data rows in the synchronization file with the number of rows corresponding to the data to be exported in the data table, and to obtain a row count comparison result; a first sending module, used to send the synchronization file to the target database according to the file sending path in the configuration table when the row count comparison result indicates that the number of data rows in the synchronization file is the same as the number of rows corresponding to the data to be exported in the data table; and a first import module, used to import the data in the synchronization file into the target database according to the synchronization type in the configuration table, and to verify the data table imported into the database.

[0016] Further, the first filtering module includes: a first construction submodule, used to, when the synchronization type in the configuration table is full synchronization, take all the data in the target data table as the data to be exported, and construct a full synchronization file based on the target data table; and a second construction submodule, used to, when the synchronization type in the configuration table is incremental synchronization, filter incremental data from the target data table as the data to be exported based on the incremental conditions in the configuration table, and construct an incremental synchronization file based on the incremental data.

[0017] Further, the first import module includes: a third construction submodule, used to construct a backup table in the target database with the same structure as the target data table when the synchronization type in the configuration table is full synchronization; a first import submodule, used to import the data in the full synchronization file into the backup table, perform row count verification on the backup table and the target data table, and perform field verification on the data in the backup table using the field verification strategy to obtain a verification result; and a first naming submodule, used to name the backup table with the name of the target data table when the verification result indicates that the backup table has passed the verification.

[0018] Furthermore, the first import module further includes: a first cleanup submodule, used to clean up the data in the target database according to the incremental conditions in the configuration table when the synchronization type in the configuration table is incremental synchronization; and a second import submodule, used to import the data in the incremental synchronization file into the cleaned target database, perform row count verification between the incremental data imported into the target database and the target data table, and perform field verification on the incremental data imported into the target database using the field verification strategy.

[0019] Furthermore, the data synchronization device further includes: a first acquisition module, used to acquire business announcement information and incremental value data within historical time periods of a database table configured with an anomaly detection strategy; a first extraction module, used to extract business features from the business announcement information, construct a business vector based on the business features, and construct a trend vector based on the incremental value data; a first prediction module, used to input the business vector and the trend vector into a prediction model, and output the theoretical incremental value of the database table, wherein the prediction model is a pre-built machine learning model that integrates trend features, seasonal features, holiday features, and business variables to predict incremental values; a first calculation module, used to obtain the actual incremental value of the database table, and calculate the difference between the theoretical incremental value and the actual incremental value to obtain an incremental deviation value; and a first comparison module, used to compare the incremental deviation value with a deviation threshold, and trigger an anomaly warning for the database table if the incremental deviation value of the database table is greater than the deviation threshold.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform any of the above-described data synchronization methods.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described data synchronization methods.

[0022] In this application, the following steps are performed: the target data table to be synchronized is determined and the table identifier of the target data table is obtained. Then, the configuration table of the target data table is called based on the table identifier, the configuration table is parsed to obtain the data synchronization information of the target data table, and a data synchronization task is constructed based on the data synchronization information. The data synchronization task includes at least: data export, data comparison, data import, and data verification. Finally, the data synchronization task is executed to synchronize the target data table from the source database to the target database.

[0023] In this application, a pre-built configuration table is used to centrally manage the synchronization rules of each data table. This configuration-driven approach significantly reduces the need for coding, improving development efficiency. For the target data table to be synchronized, the pre-built configuration table is invoked to automatically obtain the specific information required for data synchronization, thereby generating an execution plan for the data synchronization task. This reduces manual intervention in data synchronization and ensures the efficiency and accuracy of task execution. No core code modifications are required, saving significant maintenance time and costs. Furthermore, the system can quickly adapt to business expansion, enhancing the scalability of data synchronization. This solves the technical problems in related technologies where database table synchronization methods are mainly implemented through manual programming, requiring separate coding for each table's synchronization rules, resulting in low development efficiency, high maintenance costs, and poor scalability. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a mobile terminal performing a data synchronization method according to an embodiment of this application is shown;

[0026] Figure 2 This is a flowchart of an optional data synchronization method according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of an optional data synchronization process according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of an optional data synchronization device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that the data synchronization method and apparatus in this application can be used in the field of big data technology to synchronize data through a configuration table, and can also be used in any field other than the field of big data technology to synchronize data through a configuration table. This application does not limit the application field of the data synchronization method and apparatus.

[0032] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of such data all comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0033] It should be noted that in this application, when collecting and analyzing customer information, users are provided with corresponding operation entry points to choose whether to agree to or reject the automated decision-making results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0034] The following embodiments of the present invention can be applied to various data synchronization systems / applications / devices. This invention replaces hard-coding development with configuration tables; users only need to define the source and target table synchronization rules, and the system automatically performs data export, file renaming, data comparison, anomaly monitoring and early warning, file transfer, and data import. It supports three synchronization modes for table data: full, incremental, and update. Through the fusion of time series prediction models and structural theme models, it predicts daily data volume, monitors abnormal data, and issues early warnings. This reduces maintenance time and costs, improves the scalability of the data synchronization system, and ensures the accuracy and effectiveness of data synchronization.

[0035] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0036] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a data synchronization method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data synchronization method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] Example 1

[0039] According to an embodiment of the present invention, an embodiment of a data synchronization method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] Figure 2 This is a flowchart of an optional data synchronization method according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:

[0041] Step S201: Determine the target data table to be synchronized and obtain the table identifier of the target data table.

[0042] In step S201 above, the database tables that need to be synchronized are identified and locked. This process is usually completed automatically by the system when the data synchronization system starts, or in response to a specific user request. First, the system accesses the configuration management module, which stores information about all data tables that need to be synchronized, including table names, synchronization rules, and connection information between the source and target databases. The system traverses the table list in the configuration management module to determine which tables are marked as target data tables to be synchronized. Simultaneously, it obtains the table identifier for each table to be synchronized. The table identifier is used to uniquely identify and locate the data table; this step is fundamental to subsequent data synchronization operations, ensuring that the synchronization operation is performed on the correct database tables.

[0043] Step S202: Based on the table identifier, call the configuration table of the target data table, parse the configuration table, and obtain the data synchronization information of the target data table.

[0044] In step S202 above, based on the obtained table identifier, the system calls the configuration table, which stores information related to the target data table. The configuration table is a database table that centrally manages data synchronization rules, containing important parameters such as source database information, target database information, and synchronization type (e.g., full or incremental) for different database tables. The purpose of calling the configuration table is to read the synchronization rules and parameters related to the current target data table.

[0045] The data in the configuration table is stored in a structured manner, containing multiple fields, each potentially representing different synchronization parameters. The system uses a rule parsing engine to parse the configuration table, transforming these fields into computer-understandable instructions to form data synchronization information. The parsing process includes identifying the types of the source and target databases, determining the synchronization mode, reading and comparing data, and verifying fields. This information is used to construct the specific steps of the data synchronization task.

[0046] After parsing, the system can obtain complete data synchronization information for the target data table. This information includes, but is not limited to: connection information between the source and target databases, the name of the table to be synchronized, the synchronization type (full or incremental, with incremental involving data addition, deletion, and modification), whether field tampering verification is performed, whether abnormal data volume monitoring is enabled, file naming rules, and file transfer paths. This data synchronization information forms the basis for constructing specific synchronization tasks, ensuring that the synchronization process can be executed accurately according to preset rules.

[0047] By calling the configuration table and parsing its rules, the inefficient approach of programming each table separately is avoided, automating and standardizing the construction and execution process of data synchronization tasks, thus greatly improving development efficiency. Furthermore, the rules in the configuration table can be modified at any time. This means that when synchronization requirements change (such as adding a new synchronization table or changing the synchronization type), no modification to the core code is needed; only the configuration table needs to be updated, significantly enhancing the system's flexibility and responsiveness.

[0048] Furthermore, the configuration table is pre-built. The steps for building the configuration table include: obtaining all data tables in the database and configuring key fields for each data table. The key fields include at least: table identifier, business identifier of the business corresponding to the data table, import / export type, synchronization type, source database type, target database type, file naming rules, file sending path, field validation strategy, and anomaly detection strategy. The field validation strategy is used to verify whether the data fields have been tampered with, and the anomaly detection strategy is used to detect abnormal changes in data volume. Based on the key fields, the configuration table corresponding to the data table is generated, and the field information corresponding to each key field in the configuration table is filled in based on the real-time monitoring information of the data table.

[0049] Specifically, the configuration table is pre-built, and operations personnel can configure it through a visual interface, setting the fields that need to be configured. These fields include at least the following: Table identifier, a tag or ID used to uniquely identify the data table, such as "A_SYN_1"; Business identifier, the business line or product to which the associated data table belongs, such as "Product A"; Import / export type, defining whether to export, import, or both of the data, such as "1 Export + Import"; Synchronization type, selecting the type of full, incremental, or update synchronization; Source database type, the technology type of the source database; Target database type, the technology type of the target database; File naming rules, the naming pattern of the exported file, including information such as date and product ID; File sending path, the local path of the file before it is transferred to the target system; Field validation strategy, the validation method used to identify whether fields have been tampered with, such as using hash values ​​for validation; Anomaly detection strategy, the strategy for monitoring abnormal changes in data volume, which can use time series forecasting models and structural theme models for data volume prediction and anomaly identification. The system then first needs to access the source database, extracting information from all data tables through metadata queries or pre-defined database management tools. The purpose of this step is to ensure that the configuration table fully covers all database tables that need to be synchronized, avoiding any omissions. For each data table in the database, the system generates a configuration table based on predefined key fields. Then, based on real-time monitoring information of the data table, the system fills in the field information corresponding to each key field in the configuration table to obtain a complete configuration table. The configuration table is updated in real time according to the actual situation of the data table.

[0050] Step S203: Construct a data synchronization task based on the data synchronization information.

[0051] In step S203 above, a data synchronization task for the target data table is constructed based on the parsed data synchronization information. Each data table's data synchronization task includes at least the following stages: data export, data comparison, data import, and data verification. In addition, depending on the configuration information of each data table, tasks such as file renaming and data volume anomaly monitoring may also be included. Data export ensures that data from the source database is accurately exported to intermediate files, providing a data foundation for subsequent steps. Data comparison verifies the completeness and accuracy of the exported data, preventing data loss. Data import establishes or updates data tables in the target database, completing the data migration process from source to target. Data verification, after data import, re-verifies the accuracy and consistency of the data, preventing data loss and data tampering during data transmission.

[0052] Step S204: Execute the data synchronization task to synchronize the target data table from the source database to the target database.

[0053] In step S204 above, each subtask of the data synchronization task is executed step by step to synchronize data in the target data table from the source database to the target database. Specifically, this includes: connecting the source and target databases to ensure the system can access and operate both databases, providing a channel for data reading and import; data reading and exporting, exporting data from the source database according to preset rules for easy transmission and import operations; file transfer, securely transferring data files between the source and target systems, serving as a bridge in the data synchronization process; data import, importing data from the data files according to the table structure and rules of the target database to complete data migration and synchronization; and data verification and processing, ensuring data accuracy and consistency in the final stage of data synchronization, and promptly identifying and handling anomalies. By automatically executing data reading, exporting, transmitting, and importing, manual intervention is reduced, improving the automation level of data synchronization. The synchronization strategy based on the configuration table can flexibly adapt to different data synchronization needs, allowing for the expansion of new data synchronization tasks without modifying the core code.

[0054] Furthermore, the steps for executing the data synchronization task to synchronize the target data table from the source database to the target database include: filtering the data to be exported from the target data table based on the synchronization type in the configuration table, constructing a synchronization file based on the data to be exported, and naming the synchronization file according to the file naming rules in the configuration table; comparing the number of data rows in the synchronization file with the corresponding number of rows of the data to be exported in the data table to obtain the row count comparison result; if the row count comparison result indicates that the number of data rows in the synchronization file is the same as the corresponding number of rows of the data to be exported in the data table, sending the synchronization file to the target database according to the file sending path in the configuration table; importing the data in the synchronization file into the target database according to the synchronization type in the configuration table, and validating the data table imported into the database.

[0055] Specifically, firstly, based on the synchronization type (full or incremental) in the configuration table, the system filters the datasets that meet the requirements from the target data table in the source database. For full synchronization, all data in the entire data table is exported; for incremental synchronization, only the newly added data since the last synchronization is exported. The filtered data is then packaged into a synchronization file, which acts as the carrier for data transmission, ensuring that data can be safely and efficiently transferred from the source database to the target database. Following the file naming rules in the configuration table, a unique name is assigned to the constructed synchronization file. The filename can include key information such as the business identifier and synchronization date for easy subsequent management and tracking.

[0056] Next, the number of data rows in the synchronization file is compared with the number of rows of data to be exported from the data table recorded in the configuration table. This verifies the completeness and accuracy of the data export, ensuring that no omissions or extra data are included. If the number of rows matches, it indicates that there were no problems with the data export process; otherwise, it suggests that there may be errors or anomalies, requiring further inspection and processing. After confirming that the number of data rows indicated by the row count comparison result matches, the system will transfer the synchronization file to the server or storage location where the target database is located according to the file sending path defined in the configuration table. This ensures that the data file arrives at its destination accurately, preparing for subsequent data import.

[0057] Finally, the target database executes the corresponding data import logic based on the synchronization type defined in the configuration table. This includes importing full data and incremental data. After the data import is complete, the system verifies the data tables in the target database to check the correctness and completeness of the imported data. The verification process includes comparing the number of rows again and checking the consistency of field values.

[0058] Furthermore, the steps of filtering the data to be exported from the target data table based on the synchronization type in the configuration table, and constructing a synchronization file based on the data to be exported include: when the synchronization type in the configuration table is full synchronization, using all the data in the target data table as the data to be exported, and constructing a full synchronization file based on the target data table; when the synchronization type in the configuration table is incremental synchronization, filtering incremental data from the target data table using the incremental conditions in the configuration table as the data to be exported, and constructing an incremental synchronization file based on the incremental data.

[0059] Specifically, when the synchronization type in the configuration table is set to full synchronization, it means that all data in the target data table will be transferred to the target database. At this time, the system will perform a full data read process, selecting all data from the target data table in the source database, including all historical records and current records. This data will then be packaged into a synchronization file, i.e., a full synchronization file. The construction of the synchronization file includes not only the data itself but also all metadata and formatting information necessary to ensure the data can be correctly decoded and reconstructed in the target database. After the synchronization file is constructed, the system will assign it a specific name according to the file naming rules in the configuration table. This name typically includes the business identifier, synchronization date, and other necessary identification information for subsequent management and tracking.

[0060] When the synchronization type in the configuration table is incremental synchronization, the data filtering process becomes more refined. The system will refer to the incremental conditions specified in the configuration table and select only the data that has been added or changed since the last synchronization from the target data table as the data to be exported. The resulting incremental data will be used to construct the incremental synchronization file. Compared to the full synchronization file, the incremental synchronization file is usually smaller because it only contains the changed data, improving the efficiency of data transfer and import. The incremental synchronization file will also be named according to the file naming rules in the configuration table for easy identification and processing later.

[0061] Furthermore, the steps of importing data from the synchronization file into the target database according to the synchronization type in the configuration table, and verifying the imported data tables in the database include: if the synchronization type in the configuration table is full synchronization, constructing a backup table in the target database with the same structure as the target data table; importing data from the full synchronization file into the backup table, performing row count verification on both the backup table and the target data table, and using a field verification strategy to perform field verification on the data in the backup table to obtain the verification result; if the verification result indicates that the backup table passes the verification, naming the backup table after the target data table.

[0062] Specifically, for the import of full data synchronization, a backup table with the same structure as the target data table is first built in the target database. This includes copying all fields and data types of the target data table, but initially without populating it with any data. The purpose of building the backup table is to protect the existing data state before importing new data, ensuring that even if any problems occur during the import process, the original data remains intact and can be quickly recovered. Next, the data from the full synchronization file is imported into the backup table in the target database. The data import follows the data format and field mapping specified in the configuration table, ensuring that the data is accurately placed in the matching fields. After the data import is complete, the system compares the number of rows in the backup table and the target data table. This comparison step verifies the integrity of the full synchronization. If the number of rows matches, it indicates that the full data has been successfully imported into the backup table. Finally, the data in the backup table is thoroughly validated using the field validation strategy defined in the configuration table. This may include checking whether the values ​​of specific fields are consistent with the source database, or performing more complex validations, such as comparing field hash values, to ensure data integrity and consistency. Field validation is a crucial part of data validation; it can detect any changes or errors that occur during the data import process.

[0063] Once all validations pass—meaning both row count and field validation results indicate that the backup table data is consistent and error-free with the source data—the system will rename the backup table to the name of the target table. The backup table will replace the original target table, becoming the new target table, while the original target table will be renamed to the backup table or marked as a historical version and stored in the source database.

[0064] Furthermore, the steps of importing data from the synchronization file into the target database according to the synchronization type in the configuration table, and verifying the data table imported into the database include: when the synchronization type in the configuration table is incremental synchronization, cleaning the data in the target database according to the incremental conditions in the configuration table; importing the data from the incremental synchronization file into the cleaned target database; verifying the number of rows in the incremental data imported into the target database and the target data table; and using a field verification strategy to verify the fields of the incremental data imported into the target database.

[0065] Specifically, for incremental data synchronization, before performing incremental synchronization, the system cleans the target data table in the target database according to the incremental conditions in the configuration table. This removes unnecessary data records, making room for new data import and avoiding data redundancy. The cleanup operation strictly follows the incremental conditions defined in the configuration table, meaning only data related to incremental synchronization is affected, ensuring the accuracy and effectiveness of the cleanup. After data cleanup, the system begins importing the data from the incremental synchronization file into the target data table in the target database. Data import must ensure seamless integration of incremental data with existing data, maintaining the integrity and consistency of the data table. After import, the system performs row count verification, comparing the difference in row count before and after import in the target data table with the row count in the incremental synchronization file to confirm that the incremental data has been completely imported without any additional row increases or decreases. In addition to row count verification, the system also uses the field verification strategy set in the configuration table to perform detailed field value verification on the imported incremental data. This includes, but is not limited to, checking whether the values ​​of specific fields are consistent with the original data in the synchronization file, and the matching of field types. By performing field verification, the system can assess the quality of data import and promptly identify and handle potential data errors.

[0066] Furthermore, after synchronizing the target data table from the source database to the target database, the process also includes: for database tables configured with anomaly detection strategies, collecting business announcement information and incremental value data within historical time periods related to the database tables; extracting business features from the business announcement information and constructing business vectors based on these features, and constructing trend vectors based on the incremental value data; inputting the business vectors and trend vectors into the prediction model to output the theoretical incremental values ​​of the database tables, where the prediction model is a pre-built machine learning model that integrates trend features, seasonal features, holiday features, and business variables to predict incremental values; obtaining the actual incremental values ​​of the database tables and calculating the difference between the theoretical and actual incremental values ​​to obtain the incremental deviation value; comparing the incremental deviation value with a deviation threshold, and triggering an anomaly warning for the database tables if the incremental deviation value of the database tables exceeds the deviation threshold.

[0067] Specifically, for database tables with frequently changing data, anomaly detection strategies are configured through a configuration table. The system iterates through the configuration table to identify database tables with configured anomaly detection strategies and automatically collects announcement information related to the business of those database tables. This information may originate from the company's internal announcement system, news websites, or other public resources. The collected information includes factors that may affect business data, such as promotional activities, system maintenance, and strategy changes. Simultaneously, the system also collects incremental value data of the target data table over historical time periods (e.g., the past year), including changes in key business indicators such as daily, weekly, or monthly transaction volume and user activity. Business features are extracted from the business announcement information using natural language processing technology. These features include keywords mentioned in the announcement text, activity time, and scope of impact; these features are quantified and converted into business vectors. Based on historical incremental value data, the system uses statistical methods to analyze the long-term trends, seasonal variations, and holiday effects of the data. These analytical results are integrated into a trend vector.

[0068] The constructed business vector and trend vector are input into a pre-trained predictive model. This predictive model is a machine learning model that integrates trend features, seasonal features, holiday features, and business variables to predict the theoretical increment value of the database table. By learning patterns and relationships in historical data, the predictive model can take into account various influencing factors, including daily fluctuations, seasonal changes, and unexpected events, to more accurately predict the expected increment of the database table.

[0069] To obtain the actual incremental value of a database table, typically the day after synchronization is complete, the system calculates the difference between the latest synchronized data volume and the previous data volume. Next, the system calculates the difference between the theoretical and actual incremental values; this difference is called the incremental deviation value. The system compares the incremental deviation value with a preset deviation threshold. This threshold, set based on the normal incremental range of the database table and business stability, provides a standard for measuring anomalies. If the incremental deviation value of the database table exceeds the deviation threshold, the system will automatically trigger an anomaly warning, notifying relevant departments or personnel to check whether any abnormalities have occurred in the synchronization process or business operations.

[0070] Through the above steps, the target data table to be synchronized is determined and its table identifier is obtained. Then, based on the table identifier, the configuration table of the target data table is called, the configuration table is parsed, and the data synchronization information of the target data table is obtained. Based on the data synchronization information, a data synchronization task is constructed, which includes at least: data export, data comparison, data import, and data verification. Finally, the data synchronization task is executed to synchronize the target data table from the source database to the target database.

[0071] In this embodiment, a pre-built configuration table centrally manages the synchronization rules for each data table. This configuration-driven approach significantly reduces the need for coding, improving development efficiency. For the target data table to be synchronized, the pre-built configuration table is invoked to automatically obtain the specific information required for data synchronization, thereby generating an execution plan for the data synchronization task. This reduces manual intervention in data synchronization while ensuring the efficiency and accuracy of task execution. No core code modifications are required, saving substantial maintenance time and costs. Furthermore, the system can quickly adapt to business expansion, enhancing the scalability of data synchronization. This solves the technical problems in related technologies where database table synchronization methods are mainly implemented through manual programming, requiring separate coding for each table's synchronization rules, resulting in low development efficiency, high maintenance costs, and poor scalability.

[0072] The following describes in detail another optional implementation method.

[0073] Figure 3 This is a schematic diagram of an optional data synchronization process according to an embodiment of the present invention, such as... Figure 3 The image shows the data synchronization process for a database table synchronization task configured with a full-process workflow, which specifically includes:

[0074] Step 1, Begin;

[0075] Step 2: Configure visual synchronization rules;

[0076] First, through the visual data synchronization parameter configuration, the key information of the tables to be synchronized is configured in the synchronization table configuration table. This mainly includes financial product ID, table name to be synchronized, import / export type, synchronization type, source database type, target database type, whether to verify field tampering, verification fields, whether to monitor abnormal data volume, and whether to judge holidays, etc. The key fields and their meanings are shown in the table below:

[0077] Table 1 Synchronization Table Configuration

[0078]

[0079] Table 2 Synchronization Table Configuration Continued

[0080]

[0081] Taking a new table A_SYN_1 added to financial product A that needs to be synchronized from database A as an example, requiring a full synchronization to database B, field tampering verification and abnormal data volume monitoring are required. The main fields in the table are configured as follows: Business ID: A, Table Name to be Synchronized: A_SYN_1, Import / Export Type: 1 Export and Import, Synchronization Type: 1-Full, Source Database Type: Database A, Target Database Type: Database B, Target Table Name: A_SYN_1, File Name: 00-A_SYN_1-{date}, File Sending Path: / data / syn / , Field Tampering Verification: 0-Yes, Field Tampering Verification Fields: A1, A2, Anomaly Monitoring: 0-Yes.

[0082] Step 3: Rule parsing engine processing;

[0083] The rule parsing engine reads the synchronization table configuration data and generates corresponding execution plans based on different configurations. For example, for the A_SYN_1 table configured above, because the import / export type is set to 1 (export and import), the synchronization type is 1 (full), the source database type is database A, the target database type is database B, the field tampering check is 0 (yes), and the anomaly detection is 0 (yes), the A_SYN_1 table will undergo a full execution plan including data export, file renaming, exported data comparison, anomaly detection, file transfer, data import, and imported data comparison. As another example, for the table A_SYN_2 to be synchronized, with the import / export type set to 2 (export), the synchronization type to 1 (full), the field tampering check being 1 (no), and the anomaly detection being 1 (no), A_SYN_2 will undergo data export, file renaming, exported data comparison, and file transfer, but will not undergo anomaly detection or data import.

[0084] Step 4: Perform data export;

[0085] Step 5: Rename the file;

[0086] Step 6: Determine if the exported data comparison passes. If not, repeat steps 4 to 6. If yes, proceed to step 7.

[0087] Based on the synchronization type and incremental and update condition configurations, export the data file and rename it according to the file naming configuration. For update synchronization, during export, an incremental file will be exported based on the incremental condition, and an update file ending with "_update" will be exported based on the update condition; both files will be sent to the target system together. The renamed data file will be compared to the number of rows retrieved from the database based on the export conditions. If the number of exported rows matches, the export is considered successful; otherwise, it is considered a failure, and the data export process will be retried. If the field tamper verification configuration is set to 0-Yes, the hash value of the field configured in the field tamper verification field will be calculated, and an anti-tamper verification file will be generated, which will then be transferred along with the file.

[0088] Step 7: Determine whether the abnormal data volume monitoring has passed. If not, issue an early warning notification. If yes, proceed to step 8.

[0089] For data tables sensitive to daily data volume changes, anomaly monitoring can be configured. This anomaly monitoring function incorporates time series forecasting models and structured topic models, comparing the generated predicted values ​​with the actual values ​​for the day. It issues alerts for deviations from the confidence interval and notifies relevant personnel.

[0090] The historical business announcements are processed using a structural topic model. Key topic variables, such as "promotional activities" and "system maintenance," are extracted from the historical business announcement text. The strength of the topic is then quantified based on dimensions such as the public source of the historical business announcement, the publisher, and the frequency of the topic's appearance in the announcement, thus generating topic variables.

[0091] By combining time series forecasting algorithms with thematic variables generated by structural theme models, a forecasting model integrating trend, seasonality, holidays, and external covariates is built to identify abnormal fluctuations in table counts.

[0092] ;

[0093] in, The trend term represents the long-term trend of data (such as linear growth, saturation growth, etc.), and is fitted by a piecewise linear function or logistic regression function.

[0094] For seasonality, the periodic fluctuations of the data (such as daily, weekly, and annual cycles) are captured, and Fourier series is used to fit the periodic features.

[0095] For the holiday item, quantify the impact of special dates (such as statutory holidays and promotional days) on the data;

[0096] The error term represents random noise that is not captured by the model and follows a normal distribution.

[0097] Using the topic variables output by the fusion structure topic model as covariates, the above core formula is extended as follows:

[0098] ;

[0099] in, The influence coefficient of the k-th topic learned by the model (e.g., β for the topic "promotion") k >0 indicates a positive impact; the "System Maintenance" topic < 0 indicates a negative impact;

[0100] To quantify the combined impact of all themes on trading volume on a given day (e.g., promotional intensity) =0.8 and =500, then this topic contributes 400 in trading volume.

[0101] After training with historical data, the daily announcement information is synchronized to the prediction model. The model will predict the daily incremental data value, compare the actual data with the predicted data, and if the data deviation is within the confidence interval, the data is considered normal. If the deviation exceeds the threshold, an early warning is triggered.

[0102] Step 8: Cross-database transfer;

[0103] After successful comparison, the files are moved to the file sending directory configured in the configuration table and then transferred to the destination system via a file transfer platform. It offers multi-functional file sending, allowing the generated files to be moved to multiple file sending directories and sent to different target systems. A backup function is also provided, allowing the generated files to be backed up to a specified path for future reference.

[0104] Step 9, Data Import;

[0105] First, verify that the data file exists. If the file does not exist, wait for a retry and issue an alarm to notify relevant personnel. After obtaining the data file, perform the data import operation for database B according to the synchronization type, incremental conditions, and update conditions configuration.

[0106] For tables undergoing full synchronization, a backup table with the same structure is created in the database and named `target_table_old`. The target table retains the latest full data, while the `old` table retains the full data from the last import. During data import, the `old` table is first cleared, and the current data is imported into it. After import, based on the configured comparison type, row count comparison and key field tamper-proofing comparison are performed on the data in the `old` table. If the comparison fails, three retries are performed, and an alarm message is sent. If the comparison succeeds, it indicates that the data import is successful, and the `old` table is renamed to the target table, and the target table is renamed to `old` for retention.

[0107] For tables that are incrementally synchronized, the target database is cleaned before synchronization based on the incremental conditions to prevent errors or erroneous data from being imported repeatedly. After cleaning, the data is imported and compared according to the configuration. If the comparison fails, the process is restarted and an alarm is triggered.

[0108] For tables that are being updated and synchronized, data cleanup is first performed based on incremental conditions. Then, delete statements are generated based on the incremental file and the primary key of the source table to accurately clean up historical data that needs to be updated, preparing for subsequent insertion of updated data. After all data cleanup is complete, the data is imported, and data comparison is performed according to the configuration. If the comparison fails, a rerun and an alert are triggered.

[0109] Step 10: Determine if the imported data comparison passes. If not, repeat step 9. If yes, proceed to step 11.

[0110] Step 11, End.

[0111] This invention replaces hard-coding development with configuration tables. Users only need to define the synchronization rules for the source and target tables, and the system automatically performs data export, file renaming, data comparison, anomaly monitoring and early warning, file transfer, and data import. It supports three synchronization modes for table data: full, incremental, and update. By fusing time series forecasting models and structural theme models, it predicts daily data volume, monitors abnormal data, and issues early warnings. This reduces maintenance time and costs, improves the scalability of the data synchronization system, and ensures the accuracy and effectiveness of data synchronization.

[0112] The following is a detailed description with reference to another embodiment.

[0113] Example 2

[0114] The data synchronization device provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in the above embodiment one. The specific implementation method and beneficial effects can be referred to the foregoing method embodiment, and will not be repeated here.

[0115] Figure 4This is a schematic diagram of an optional data synchronization device according to an embodiment of the present invention, such as... Figure 4 As shown, the data synchronization device may include: a determining unit 41, a parsing unit 42, a constructing unit 43, and a synchronization unit 44, wherein,

[0116] The determining unit 41 is used to determine the target data table to be synchronized and obtain the table identifier of the target data table;

[0117] Parsing unit 42 is used to call the configuration table of the target data table based on the table identifier, parse the configuration table, and obtain the data synchronization information of the target data table;

[0118] Construction unit 43 is used to construct a data synchronization task based on data synchronization information, wherein the data synchronization task includes at least: data export, data comparison, data import, and data verification;

[0119] Synchronization unit 44 is used to perform data synchronization tasks, synchronizing the target data table from the source database to the target database.

[0120] The aforementioned data synchronization device determines the target data table to be synchronized through the determining unit 41 and obtains the table identifier of the target data table; the parsing unit 42 calls the configuration table of the target data table based on the table identifier, parses the configuration table, and obtains the data synchronization information of the target data table; the constructing unit 43 constructs a data synchronization task based on the data synchronization information, wherein the data synchronization task includes at least: data export, data comparison, data import, and data verification; and the synchronization unit 44 executes the data synchronization task to synchronize the target data table from the source database to the target database.

[0121] In this embodiment, a pre-built configuration table centrally manages the synchronization rules for each data table. This configuration-driven approach significantly reduces the need for coding, improving development efficiency. For the target data table to be synchronized, the pre-built configuration table is invoked to automatically obtain the specific information required for data synchronization, thereby generating an execution plan for the data synchronization task. This reduces manual intervention in data synchronization while ensuring the efficiency and accuracy of task execution. No core code modifications are required, saving substantial maintenance time and costs. Furthermore, the system can quickly adapt to business expansion, enhancing the scalability of data synchronization. This solves the technical problems in related technologies where database table synchronization methods are mainly implemented through manual programming, requiring separate coding for each table's synchronization rules, resulting in low development efficiency, high maintenance costs, and poor scalability.

[0122] Furthermore, the data synchronization device also includes: a first acquisition module, used to acquire all data tables in the database and configure key fields for each data table, wherein the key fields include at least: table identifier, business identifier of the business corresponding to the data table, import / export type, synchronization type, source database type, target database type, file naming rules, file sending path, field validation strategy, and anomaly detection strategy. The field validation strategy is used to verify whether the data fields have been tampered with, and the anomaly detection strategy is used to detect abnormal changes in data volume; and a first generation module, used to generate a configuration table corresponding to the data table based on the key fields, and fill in the field information corresponding to each key field in the configuration table based on the real-time monitoring information of the data table.

[0123] Furthermore, the synchronization unit includes: a first filtering module, used to filter the data to be exported from the target data table based on the synchronization type in the configuration table, and to construct a synchronization file based on the data to be exported, naming the synchronization file according to the file naming rules in the configuration table; a first comparison module, used to compare the number of data rows in the synchronization file with the corresponding number of rows of the data to be exported in the data table, and obtain a row count comparison result; a first sending module, used to send the synchronization file to the target database according to the file sending path in the configuration table if the row count comparison result indicates that the number of data rows in the synchronization file is the same as the corresponding number of rows of the data to be exported in the data table; and a first import module, used to import the data in the synchronization file into the target database according to the synchronization type in the configuration table, and to verify the data table imported into the database.

[0124] Furthermore, the first filtering module includes: a first construction submodule, used to take all data in the target data table as the data to be exported when the synchronization type in the configuration table is full synchronization, and construct a full synchronization file based on the target data table; and a second construction submodule, used to filter incremental data from the target data table as the data to be exported based on the incremental conditions in the configuration table when the synchronization type in the configuration table is incremental synchronization, and construct an incremental synchronization file based on the incremental data.

[0125] Furthermore, the first import module includes: a third construction submodule, used to construct a backup table in the target database with the same structure as the target data table when the synchronization type in the configuration table is full synchronization; a first import submodule, used to import the data from the full synchronization file into the backup table, perform row count verification on the backup table and the target data table, and perform field verification on the data in the backup table using a field verification strategy to obtain the verification result; and a first naming submodule, used to name the backup table with the name of the target data table when the verification result indicates that the backup table has passed the verification.

[0126] Furthermore, the first import module also includes: a first cleanup submodule, used to clean up the data in the target database according to the incremental conditions in the configuration table when the synchronization type in the configuration table is incremental synchronization; and a second import submodule, used to import the data in the incremental synchronization file into the cleaned target database, perform row count verification between the incremental data imported into the target database and the target data table, and perform field verification on the incremental data imported into the target database using a field verification strategy.

[0127] Furthermore, the data synchronization device also includes: a first acquisition module, used to acquire business announcement information and incremental value data within historical time periods for database tables configured with anomaly detection strategies; a first extraction module, used to extract business features from business announcement information, construct business vectors based on business features, and construct trend vectors based on incremental value data; a first prediction module, used to input business vectors and trend vectors into a prediction model, and output the theoretical incremental value of the database table, wherein the prediction model is a pre-built machine learning model that integrates trend features, seasonal features, holiday features, and business variables to predict incremental values; a first calculation module, used to obtain the actual incremental value of the database table, and calculate the difference between the theoretical incremental value and the actual incremental value to obtain the incremental deviation value; and a first comparison module, used to compare the incremental deviation value with a deviation threshold, and trigger an anomaly warning for the database table if the incremental deviation value of the database table is greater than the deviation threshold.

[0128] The aforementioned data synchronization device may also include a processor and a memory. The aforementioned determining unit 41, parsing unit 42, constructing unit 43, and synchronizing unit 44 are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0129] The processor described above contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and data synchronization is achieved by adjusting kernel parameters.

[0130] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0131] This invention provides a computer-readable storage medium including a stored program, wherein the program controls the device where the computer-readable storage medium is located to execute a data synchronization method during runtime.

[0132] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program executes a data synchronization method during runtime.

[0133] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: determining a target data table to be synchronized and obtaining the table identifier of the target data table; calling the configuration table of the target data table based on the table identifier, parsing the configuration table, and obtaining data synchronization information of the target data table; constructing a data synchronization task based on the data synchronization information, wherein the data synchronization task includes at least: data export, data comparison, data import, and data verification; and executing the data synchronization task to synchronize the target data table from the source database to the target database.

[0134] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0135] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps: determining a target data table to be synchronized and obtaining the table identifier of the target data table; calling the configuration table of the target data table based on the table identifier, parsing the configuration table, and obtaining the data synchronization information of the target data table; constructing a data synchronization task based on the data synchronization information, wherein the data synchronization task includes at least: data export, data comparison, data import, and data verification; and executing the data synchronization task to synchronize the target data table from the source database to the target database.

[0136] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0137] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0141] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0142] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0143] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0146] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data synchronization method, characterized in that, include: Identify the target data table to be synchronized and obtain the table identifier of the target data table; Based on the table identifier, the configuration table of the target data table is called, the configuration table is parsed, and the data synchronization information of the target data table is obtained; A data synchronization task is constructed based on the data synchronization information, wherein the data synchronization task includes at least: data export, data comparison, data import, and data verification; The data synchronization task is executed to synchronize the target data table from the source database to the target database.

2. The method according to claim 1, characterized in that, The configuration table is pre-built, and the steps for building the configuration table include: Obtain all data tables in the database and configure key fields for each data table. The key fields include at least: table identifier, business identifier of the business corresponding to the data table, import / export type, synchronization type, source database type, target database type, file naming rules, file sending path, field validation strategy, and anomaly detection strategy. The field validation strategy is used to verify whether the data fields have been tampered with, and the anomaly detection strategy is used to detect abnormal changes in data volume. A configuration table corresponding to the data table is generated based on the key fields, and the field information corresponding to each key field in the configuration table is filled in based on the real-time monitoring information of the data table.

3. The method according to claim 2, characterized in that, The steps of executing the data synchronization task to synchronize the target data table from the source database to the target database include: The data to be exported from the target data table is filtered based on the synchronization type in the configuration table, and a synchronization file is constructed based on the data to be exported. The synchronization file is named according to the file naming rules in the configuration table. The number of data rows in the synchronization file is compared with the number of rows corresponding to the data to be exported in the data table to obtain the row number comparison result; If the row count comparison result indicates that the number of data rows in the synchronization file is the same as the number of rows corresponding to the data to be exported in the data table, the synchronization file is sent to the target database according to the file sending path in the configuration table; According to the synchronization type in the configuration table, the data in the synchronization file is imported into the target database, and the data tables imported into the database are verified.

4. The method according to claim 3, characterized in that, The steps of filtering the data to be exported from the target data table based on the synchronization type in the configuration table, and constructing a synchronization file based on the data to be exported include: When the synchronization type in the configuration table is full synchronization, the full data in the target data table is used as the data to be exported, and a full synchronization file is constructed based on the target data table. When the synchronization type in the configuration table is incremental synchronization, incremental data is selected from the target data table as the data to be exported based on the incremental conditions in the configuration table, and an incremental synchronization file is constructed based on the incremental data.

5. The method according to claim 4, characterized in that, The steps of importing data from the synchronization file into the target database according to the synchronization type in the configuration table, and verifying the data tables imported into the database, include: If the synchronization type in the configuration table is full synchronization, a backup table with the same structure as the target data table is built in the target database. Import the data from the full synchronization file into the backup table, perform row count verification on the backup table and the target data table, and perform field verification on the data in the backup table using the field verification strategy to obtain the verification result; If the verification result indicates that the backup table passes the verification, the backup table shall be named after the target data table.

6. The method according to claim 4, characterized in that, The steps of importing data from the synchronization file into the target database according to the synchronization type in the configuration table, and verifying the data tables imported into the database, include: If the synchronization type in the configuration table is incremental synchronization, the data in the target database is cleaned according to the incremental conditions in the configuration table; The data in the incremental synchronization file is imported into the target database after data cleanup. The number of rows in the incremental data imported into the target database is verified against the number of rows in the target data table. The field verification strategy is then used to verify the fields in the incremental data imported into the target database.

7. The method according to claim 2, characterized in that, After synchronizing the target data table from the source database to the target database, the process also includes: For database tables configured with anomaly detection strategies, collect business announcement information and incremental value data within historical time periods related to the database tables; Extract business features from the business announcement information, construct a business vector based on the business features, and construct a trend vector based on the incremental value data; The business vector and the trend vector are input into the prediction model, and the theoretical incremental value of the database table is output. The prediction model is a pre-built machine learning model that integrates trend features, seasonal features, holiday features and business variables to predict incremental values. Obtain the actual increment value of the database table, and calculate the difference between the theoretical increment value and the actual increment value to obtain the increment deviation value; The incremental deviation value is compared with the deviation threshold. If the incremental deviation value of the database table is greater than the deviation threshold, an anomaly warning is triggered for the database table.

8. A data synchronization device, characterized in that, include: The determining unit is used to determine the target data table to be synchronized and to obtain the table identifier of the target data table; The parsing unit is used to call the configuration table of the target data table based on the table identifier, parse the configuration table, and obtain the data synchronization information of the target data table; A construction unit is used to construct a data synchronization task based on the data synchronization information, wherein the data synchronization task includes at least: data export, data comparison, data import, and data verification; The synchronization unit is used to execute the data synchronization task and synchronize the target data table from the source database to the target database.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data synchronization method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data synchronization method according to any one of claims 1 to 7.

Images