Data synchronization method, apparatus, device, and medium

Abstract

This invention discloses a data synchronization method, apparatus, device, and medium. The method includes: acquiring a data synchronization task and corresponding data synchronization rules, wherein the data synchronization rules are generated based on a selected preset rule template; when the data synchronization task and data synchronization rules meet synchronization pre-check conditions, acquiring data information to be synchronized from the source database corresponding to the data synchronization task; determining and storing the converted target data information according to a preset unified format and the data information to be synchronized; and synchronizing the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules. By converting the data information to be synchronized using a preset unified format and storing the unified format data, and synchronizing the target data information to the target database according to the data synchronization rules, the method achieves universality and efficiency in data synchronization between different types of databases, eliminates the need for manual synchronization configuration, and reduces the technical requirements for data synchronization configuration personnel.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to data synchronization methods, apparatus, devices and media. Background Technology

[0002] With the diversification of open-source software and information technology approaches, enterprises are increasingly using a wide variety of databases. This has led to frequent data migrations, requiring the same data to be copied and stored multiple times across different data scenarios. Furthermore, the quality of data is difficult to guarantee during these frequent copying and migration processes.

[0003] Existing technical solutions convert the data format in the source database to the corresponding format in the target database and synchronize it to the target database, while comparing the data during the synchronization process to ensure the accuracy of data synchronization.

[0004] However, existing data synchronization solutions have poor versatility. Synchronizing multiple homogeneous and heterogeneous data requires one-to-one development of a large amount of code; data quality is difficult to manage. Traditional data comparison solutions require full table scans of the entire data in both databases to compare data row by row. Data comparison is time-consuming and data differences are difficult to repair. In scenarios with large data volumes, the comparison time is long and it is difficult to meet users' requirements for high consistency data quality; the same data needs to be frequently migrated and transformed in different data usage scenarios, and each data synchronization requires a lot of time and resources. Summary of the Invention

[0005] This invention provides a data synchronization method, apparatus, device, and medium to achieve data synchronization between homogeneous and heterogeneous data sources.

[0006] According to a first aspect of the present invention, a data synchronization method is provided, comprising:

[0007] Obtain the data synchronization task and the corresponding data synchronization rules, wherein the data synchronization rules are generated based on the selected preset rule template;

[0008] When the data synchronization task and the data synchronization rule meet the synchronization pre-check conditions, the data information to be synchronized is obtained from the source database corresponding to the data synchronization task.

[0009] Based on the preset unified format and the data information to be synchronized, the target data information after conversion is determined and the data is stored.

[0010] According to the data synchronization rules, the target data information is synchronized to the target database corresponding to the data synchronization task.

[0011] According to a second aspect of the present invention, a data synchronization device is provided, comprising:

[0012] The rule acquisition module is used to acquire data synchronization tasks and corresponding data synchronization rules, wherein the data synchronization rules are generated based on the selected preset rule template;

[0013] The first determining module is used to obtain data information to be synchronized from the source database corresponding to the data synchronization task when the data synchronization task and the data synchronization rule meet the synchronization pre-detection conditions.

[0014] The second determining module is used to determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized.

[0015] The information synchronization module is used to synchronize the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data synchronization method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data synchronization method described in any embodiment of the present invention.

[0021] The technical solution of this invention involves acquiring a data synchronization task and its corresponding data synchronization rules, wherein the data synchronization rules are generated based on a selected preset rule template. When the data synchronization task and the data synchronization rules meet the synchronization pre-check conditions, the data information to be synchronized is obtained from the source database corresponding to the data synchronization task. Based on a preset unified format and the data information to be synchronized, the converted target data information is determined and stored. According to the data synchronization rules, the target data information is synchronized to the target database corresponding to the data synchronization task. By converting the data information to be synchronized using a preset unified format and storing the unified format data, the target data information is synchronized to the target database according to the data synchronization rules. This achieves universality and efficiency in data synchronization between different types of databases, eliminating the need for manual synchronization configuration and reducing the technical requirements for data synchronization configuration personnel.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a data synchronization method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 This is an example flowchart of a data synchronization method provided in Embodiment 1 of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of a data synchronization device according to Embodiment 2 of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of an electronic device that implements an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 The flowchart illustrates a data synchronization method provided in Embodiment 1 of the present invention. This embodiment is applicable to data synchronization between multiple homogeneous and heterogeneous databases. The method can be executed by a data synchronization device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110. Obtain the data synchronization task and the corresponding data synchronization rules, wherein the data synchronization rules are generated based on the selected preset rule template.

[0033] In this embodiment, a data synchronization task can be understood as a task created by the user through a graphical interface to synchronize data between a source database and a target database. Data synchronization tasks and rules can be created and edited through drag-and-drop, clicking, and other methods within the graphical interface. Users can create heterogeneous synchronization tasks through task scheduling, providing functions such as task start, task stop, task deletion, automated task start / stop, and task queuing.

[0034] In this embodiment, data synchronization rules can be understood as the rules adopted during heterogeneous data synchronization. These rules may include database table correspondence, field correspondence, field addition / removal rules, empty field filling rules, data anonymization rules, field type conversion rules, field value conversion rules, abnormal data handling rules, and response rules after synchronization anomalies. Users can select from provided preset rule templates or generate their own custom data synchronization rules. The preset rule templates include templates generated for common data synchronization needs. The processor can also manage and save user-defined data synchronization rules. Users can share their custom data synchronization rules to the template center.

[0035] Specifically, the processor can obtain the data synchronization tasks and corresponding data synchronization rules created by the user through the graphical interface.

[0036] S120. When the data synchronization task and data synchronization rules meet the synchronization pre-check conditions, obtain the data information to be synchronized from the source database corresponding to the data synchronization task.

[0037] In this embodiment, the synchronization pre-check conditions can be understood as conditions used to check the risks of task execution and the impact risks on the source and target ends before data synchronization. The source database can be understood as a relational database that needs data transformation. The data information to be synchronized can be understood as the data information in the source database that needs to be synchronized.

[0038] Specifically, after obtaining the data synchronization task and data synchronization rules, the processor can automatically check the compliance of the data synchronization rule configuration according to the set process. It determines whether it is compliant based on the dependency conditions of the source database and the target end, as well as the degree of impact on the source database and the target end. When the pre-check passes, the processor can add the data synchronization task to the task queue, add task monitoring, and load the data synchronization rules. When the data synchronization task and data synchronization rules meet the synchronization pre-check conditions, the processor can perform full and incremental extraction of the database structure, database objects and data of the source database, and provide breakpoint resume capability to obtain the data information to be synchronized.

[0039] S130. Based on the preset unified format and the data information to be synchronized, determine the target data information after conversion and store the data.

[0040] In this embodiment, the preset unified format can be understood as a preset standard data format used to convert data from different types of databases into data under the preset unified format. This includes database table structure information, primary key information, transaction operation type, transaction execution time, data acquisition time, the original SQL executed from the source, data information of rows affected before SQL execution, data information of rows affected after SQL execution, log information, etc. The target data information can be understood as the converted data under the preset unified format.

[0041] Specifically, the processor can convert the data information to be synchronized according to a preset unified format, and can also provide data processing such as data persistence, data encryption, and data compression to obtain the converted target data information and store it.

[0042] S140. According to the data synchronization rules, synchronize the target data information to the target database corresponding to the data synchronization task.

[0043] It is important to know that users need to provide access permissions to the source database, including connection and query permissions; they need to provide access permissions to the target database, including connection and query permissions; and they need to provide access to the transaction log of the source database so that the user can read the database transaction log and parse the transaction records therein.

[0044] In this embodiment, the target database can be understood as the database that needs to be written to. It can be a relational database, a non-relational database, a distributed database, a big data component, or a data storage service, etc.

[0045] Specifically, the processor can convert target data information into SQL statements for the target database according to data synchronization rules, and load them into the target database in batches or in real time according to the data synchronization rules.

[0046] The technical solution of this invention involves acquiring a data synchronization task and its corresponding data synchronization rules, wherein the data synchronization rules are generated based on a selected preset rule template. When the data synchronization task and the data synchronization rules meet the synchronization pre-check conditions, the data information to be synchronized is obtained from the source database corresponding to the data synchronization task. Based on a preset unified format and the data information to be synchronized, the converted target data information is determined and stored. According to the data synchronization rules, the target data information is synchronized to the target database corresponding to the data synchronization task. By converting the data information to be synchronized using a preset unified format and storing the unified format data, the target data information is synchronized to the target database according to the data synchronization rules. This achieves universality and efficiency in data synchronization between different types of databases, eliminating the need for manual synchronization configuration and reducing the technical requirements for data synchronization configuration personnel.

[0047] Furthermore, based on the above embodiments, the step of synchronizing target data information to the target database corresponding to the data synchronization task according to data synchronization rules can be optimized as follows:

[0048] a1. Based on the data synchronization rules, convert the target data information into Structured Query Language (SQL) statements.

[0049] In this embodiment, SQL statements can be understood as statements used to store data.

[0050] Specifically, the processor can convert the target data information into SQL statements according to the conversion method recorded in the data synchronization rules.

[0051] b1. Synchronize the SQL statements to the target database corresponding to the data synchronization task.

[0052] Specifically, the processor can synchronously transmit SQL statements to the corresponding target database.

[0053] The technical solution of this invention allows users to select the desired data synchronization rules through simple page editing. Based on these rules, complex heterogeneous data transformations are achieved, enabling universality for synchronous migration between homogeneous and heterogeneous data sources. It provides the necessary data transformation, processing, and loading capabilities during homogeneous or heterogeneous data synchronization, ensuring stable and efficient data synchronization. Real-time or streaming data acquisition can be achieved using simple SQL queries or low-code methods.

[0054] As a first optional embodiment of this embodiment, after writing the target data information into the target database corresponding to the data synchronization task according to the data synchronization rules, the method further includes:

[0055] When the data synchronization task meets the data management conditions, data fingerprint information is generated based on the target data information and stored in the data fingerprint database.

[0056] In this embodiment, data management conditions can be understood as conditions used for data quality verification. Data fingerprint information can be understood as a unique identifier representing data, used for data verification and quick querying. The data fingerprint database can be understood as a database used to store data fingerprint information.

[0057] Specifically, users can choose whether to perform data management in the provided graphical interface. When users choose to perform data management, the data synchronization task also includes data management instructions. The processor can parse the data synchronization task and determine that it includes data management instructions. When the data synchronization task meets the data management conditions, the processor can generate data fingerprint information based on the row data primary key, row core field, and full row data of the target data information record converted into a preset unified format.

[0058] In the first optional embodiment of this embodiment, when the data synchronization task meets the data management conditions, the target data is automatically converted into data fingerprint information and stored in the data fingerprint database, which provides a basis for subsequent data quality management. The method of forming data fingerprint information can also reduce the data scanning time and data occupation required for data comparison.

[0059] As a second optional embodiment of this first embodiment, based on the above embodiment, it further includes:

[0060] Obtain the database management request relative to the source database, and determine the target information after management based on the database management request.

[0061] In this embodiment, a database management request can be understood as a request for data acquisition. Target information can be understood as information determined relative to the database management request.

[0062] Specifically, the processor can obtain database management requests relative to the source database generated by the user through simple SQL, low-code, or SDK components, and call preset data services and DataAPI capabilities based on the database management requests to determine the target information after management.

[0063] Furthermore, based on the above embodiments, the step of determining the managed target information according to the database management request can be optimized as follows:

[0064] a2. Identify the data to be managed that matches the database management request from the stored data. The data to be managed is data converted according to a preset unified format.

[0065] In this embodiment, the data to be managed can be understood as data that has already been stored and converted to a preset unified format, which can be stored during the source database synchronization process.

[0066] Specifically, the processor can search for the data to be managed that matches the database management request in the stored data.

[0067] b2. Manage the data to be managed through preset data services or preset data application interfaces to obtain the target management data.

[0068] In this embodiment, the preset data service can be understood as a unified data streaming or batch processing capability provided in the form of an application. The preset data application interface can be understood as providing a unified data streaming or batch processing capability in the form of an application interface, providing a unified data query interface to the outside world, such as DataAPI. Target management data can be understood as data that satisfies database management requests.

[0069] Specifically, the processor can provide data processing functions that meet database management requests through preset data services or preset data application interfaces, such as data report analysis, to manage the data information to be managed and obtain the target management data.

[0070] In the second optional embodiment of this first example, upon receiving a database management request, data in a uniform format is managed by invoking a preset data service or a preset application programming interface. By storing uniformly formatted data and providing data services, a single set of data can be used multiple times. The same data only needs to be copied and migrated once to provide data services, reducing the time required to extract and convert data from the source database during data services.

[0071] As a third optional embodiment of this first embodiment, based on the above embodiments, it further includes:

[0072] When the data synchronization task meets the quality management conditions, quality management is performed on the source database corresponding to the data synchronization task to determine the quality management result of the source database.

[0073] In this embodiment, quality management conditions can be understood as conditions used to determine data quality, such as performing quality management periodically according to a set cycle. Quality management results can be understood as the outcome used to determine whether the data in the source database is consistent with the synchronized data.

[0074] Specifically, users can pre-select whether to perform quality management. The selection result can be added to the data synchronization task via instructions. When the data synchronization task includes quality management instructions, during the execution of the data synchronization task by the processor, when the execution time meets the quality management conditions, the processor can perform quality management on the data in the source database corresponding to the data synchronization task and the data that has been converted, and determine the quality management result of the source database.

[0075] Furthermore, the steps for performing quality management on the source database corresponding to the data synchronization task and determining the quality management results of the source database can be optimized as follows:

[0076] a3. Determine the quality data to be managed in the source database.

[0077] Specifically, the processor can query unmanaged quality data from the source database.

[0078] b3. Generate fingerprint information for each quality data to be managed.

[0079] Specifically, the processor can generate data fingerprints for each quality data to be managed in sequence according to a set method, thereby obtaining the fingerprint information of each quality data to be managed.

[0080] c3. Determine whether the data fingerprint database includes data fingerprint information that matches each data fingerprint information to be managed.

[0081] It is important to know that during the data synchronization process, the source database data is converted into a unified format to generate data fingerprints, and the generated data fingerprints are written into the fingerprint database so that data quality management can be carried out by comparing the data fingerprints later.

[0082] Specifically, the processor can compare the fingerprints in the fingerprint database to determine whether the fingerprints match the fingerprints of the data to be managed. This will determine whether the data to be managed has been synchronized to the target database or whether the synchronized fingerprints are correct.

[0083] d3. If so, then data consistency will be taken as the quality management result of the source database.

[0084] Specifically, if there is a data fingerprint in the fingerprint database that matches the fingerprint information of the data to be managed, that is, the data corresponding to the fingerprint information of the data to be managed has been synchronized to the target database, then the data consistency is taken as the quality management result of the source database.

[0085] e3. If not, the inconsistent fingerprint information of the data to be managed will be used as the quality management result.

[0086] Specifically, if there is no matching fingerprint information in the fingerprint database, meaning the data corresponding to the fingerprint information to be managed has not been synchronized to the target database or an error occurred during synchronization, then the inconsistent fingerprint information to be managed will be used as the quality management result of the source database.

[0087] In the third optional embodiment of this first embodiment, when the data synchronization task meets the quality management conditions, the data in the source database is converted and compared with the fingerprint information in the fingerprint database to determine the quality management result of the source database. Compared with the comparison method in the prior art, the comparison of fingerprint information reduces the data scanning time and resource consumption, quickly realizes the data comparison and data repair capabilities, and meets the user's requirements for high-quality data.

[0088] For example, to facilitate understanding of the invention, a specific example is provided. Figure 2 This is an example flowchart of a data synchronization method provided according to Embodiment 1 of the present invention. Figure 2 As shown, the steps may include:

[0089] S201. Obtain the data synchronization task and the corresponding data synchronization rules, wherein the data synchronization rules are generated according to the selected preset rule template;

[0090] S202. Perform a task pre-check based on the data synchronization task and data synchronization rules, and determine whether the pre-check passes. If yes, proceed to step S203; otherwise, proceed to step S201.

[0091] S203. Add the data synchronization task to the task queue, add task monitoring, and load the data synchronization rules.

[0092] S204. Obtain the data information to be synchronized from the source database;

[0093] S205. Convert the data information to be synchronized into target data information in a preset unified format;

[0094] S206. Write the target data information into the target database according to the data synchronization rules;

[0095] S207. Is the data quality management function enabled? If yes, proceed to step S208; otherwise, proceed to step S201.

[0096] S208. Generate data fingerprint information based on target data information and store it in the data fingerprint database; store target data information.

[0097] S209. Determine whether the data quality management conditions are met; if yes, proceed to step S210; otherwise, end.

[0098] S210. Read the data information to be managed from the source database and generate the fingerprint information of the data to be managed based on the data information to be managed;

[0099] S211. Query the fingerprint database based on the fingerprint information of the data to be managed;

[0100] S212. Determine if there is a matching data fingerprint; if yes, proceed to step S209; otherwise, proceed to step S213.

[0101] S213. Record inconsistent data and trigger the data repair function;

[0102] S214. Repair data based on the data synchronization rules corresponding to inconsistent data.

[0103] Example 2

[0104] Figure 3 This is a schematic diagram of a data synchronization device provided in Embodiment 2 of the present invention. Figure 3 As shown, the device includes: a rule acquisition module 31, a first determination module 32, a second determination module 33, and an information synchronization module 34. Among them,

[0105] The rule acquisition module 31 is used to acquire the data synchronization task and the corresponding data synchronization rules, wherein the data synchronization rules are generated according to the selected preset rule template;

[0106] The first determining module 32 is used to obtain data information to be synchronized from the source database corresponding to the data synchronization task when the data synchronization task and the data synchronization rule meet the synchronization pre-detection conditions.

[0107] The second determining module 33 is used to determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized;

[0108] The information synchronization module 34 is used to synchronize the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules.

[0109] The technical solution of this invention involves acquiring a data synchronization task and its corresponding data synchronization rules, wherein the data synchronization rules are generated based on a selected preset rule template. When the data synchronization task and the data synchronization rules meet the synchronization pre-check conditions, the data information to be synchronized is obtained from the source database corresponding to the data synchronization task. Based on a preset unified format and the data information to be synchronized, the converted target data information is determined and stored. According to the data synchronization rules, the target data information is synchronized to the target database corresponding to the data synchronization task. By converting the data information to be synchronized using a preset unified format and storing the unified format data, the target data information is synchronized to the target database according to the data synchronization rules. This achieves universality and efficiency in data synchronization between different types of databases, eliminating the need for manual synchronization configuration and reducing the technical requirements for data synchronization configuration personnel.

[0110] Furthermore, the information synchronization module 34 is specifically used for:

[0111] According to the data synchronization rules, the target data information is converted into a Structured Query Language (SQL) statement;

[0112] The SQL statement is synchronized to the target database corresponding to the data synchronization task.

[0113] Optionally, the device further includes a third determining module.

[0114] The third determining module is used to generate data fingerprint information based on the target data information and store it in the data fingerprint database when the data synchronization task meets the data management conditions after the target data information is written into the target database corresponding to the data synchronization task according to the data synchronization rules.

[0115] Optionally, the device further includes a fourth determining module.

[0116] The fourth determining module is used to obtain a database management request relative to the source database, and determine the target information after management based on the database management request.

[0117] Furthermore, the fourth determining module is specifically used for:

[0118] Identify the data information to be managed that matches the database management request from the stored data, wherein the data information to be managed is data converted according to a preset unified format;

[0119] The target management data is obtained by managing the data information to be managed through a preset data service or a preset data application interface.

[0120] Optionally, the device further includes a result determination module.

[0121] The result determination module is used to perform quality management on the source database corresponding to the data synchronization task when the data synchronization task meets the quality management conditions, and to determine the quality management result of the source database.

[0122] Furthermore, the result determination module is specifically used for:

[0123] Identify the quality data to be managed in the source database;

[0124] Generate fingerprint information for each of the quality data to be managed, corresponding to each of the aforementioned quality data to be managed.

[0125] Determine whether the data fingerprint database includes data fingerprint information that matches each of the aforementioned data fingerprint information to be managed;

[0126] If so, then data consistency will be taken as the quality management result of the source database;

[0127] If not, the inconsistent fingerprint information of the data to be managed will be taken as the quality management result.

[0128] The data synchronization device provided in the embodiments of the present invention can execute the data synchronization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0129] Example 3

[0130] Figure 4 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0131] like Figure 4As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded into the RAM 43 from storage unit 48. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0132] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as data synchronization methods.

[0134] In some embodiments, the data synchronization method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the data synchronization method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the data synchronization method by any other suitable means (e.g., by means of firmware).

[0135] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0139] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0140] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0141] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0142] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data synchronization method, characterized in that, include: Obtain the data synchronization task and the corresponding data synchronization rules, wherein the data synchronization rules are generated based on the selected preset rule template; When the data synchronization task and the data synchronization rule meet the synchronization pre-check conditions, the data information to be synchronized is obtained from the source database corresponding to the data synchronization task; wherein, the synchronization pre-check conditions are conditions used to check the risks of task execution and the risks of impact on the source and target ends before data synchronization; Based on the preset unified format and the data information to be synchronized, the target data information after conversion is determined and the data is stored. According to the data synchronization rules, the target data information is synchronized to the target database corresponding to the data synchronization task; After writing the target data information into the target database corresponding to the data synchronization task according to the data synchronization rules, the method further includes: When the data synchronization task meets the data management conditions, data fingerprint information is generated based on the target data information and stored in the data fingerprint database. The data synchronization method further includes: Obtain the database management request relative to the source database, and determine the target information after management based on the database management request; The step of determining the target management data based on the database management request includes: Identify the data information to be managed that matches the database management request from the stored data, wherein the data information to be managed is data converted according to a preset unified format; The target management data is obtained by managing the data information to be managed through a preset data service or a preset data application interface.

2. The method according to claim 1, characterized in that, The step of synchronizing the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules includes: According to the data synchronization rules, the target data information is converted into a Structured Query Language (SQL) statement; The SQL statement is synchronized to the target database corresponding to the data synchronization task.

3. The method according to claim 1, characterized in that, Also includes: When the data synchronization task meets the quality management conditions, quality management is performed on the source database corresponding to the data synchronization task, and the quality management result of the source database is determined.

4. The method according to claim 3, characterized in that, The quality management of the source database corresponding to the data synchronization task, and the determination of the quality management result of the source database, include: Identify the quality data to be managed in the source database; Generate fingerprint information for each of the quality data to be managed, corresponding to each of the aforementioned quality data to be managed. Determine whether the data fingerprint database includes data fingerprint information that matches each of the aforementioned data fingerprint information to be managed; If so, then data consistency will be taken as the quality management result of the source database; If not, the inconsistent fingerprint information of the data to be managed will be taken as the quality management result.

5. A data synchronization device, characterized in that, include: The rule acquisition module is used to acquire data synchronization tasks and corresponding data synchronization rules, wherein the data synchronization rules are generated based on the selected preset rule template; The first determining module is used to obtain the data information to be synchronized from the source database corresponding to the data synchronization task when the data synchronization task and the data synchronization rule meet the synchronization pre-check conditions; wherein, the synchronization pre-check conditions are conditions used to check the risks of task execution and the risks of impact on the source and target ends before data synchronization. The second determining module is used to determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized. The information synchronization module is used to synchronize the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules. The data synchronization device further includes: a third determining module and a fourth determining module; The third determining module is used to generate data fingerprint information based on the target data information and store it in the data fingerprint database when the data synchronization task meets the data management conditions after the target data information is written into the target database corresponding to the data synchronization task according to the data synchronization rules. The fourth determining module is used to obtain a database management request relative to the source database, and determine the target information after management based on the database management request; The fourth determining module is specifically used for: Identify the data information to be managed that matches the database management request from the stored data, wherein the data information to be managed is data converted according to a preset unified format; The target management data is obtained by managing the data information to be managed through a preset data service or a preset data application interface.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data synchronization method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data synchronization method of any one of claims 1-4.

Images