Database data synchronization methods and devices, storage media, and processors

Abstract

This application discloses a database data synchronization method, apparatus, storage medium, and processor. The method includes: acquiring data to be synchronized from a source database, the data to be synchronized including data generated by a target event; splitting the target event of the data to be synchronized into first data generated by a first event and second data generated by a second event; and performing data synchronization based on the first and second events, synchronizing the data to be synchronized to the corresponding target database. The target database and the source database are of different types, the target event executes different data structures in different types of databases, while the first and second events execute the same data structures in different types of databases. This solves the problem in related technologies involving distributed database data synchronization that requires merging, sorting, and aggregating logs, leading to poor operability and, in some cases, synchronization failure.

Description

Technical Field

[0001] This application relates to the field of data synchronization, and more specifically, to a data synchronization method and apparatus for a database, a storage medium, and a processor. Background Technology

[0002] The overall architecture for data synchronization in a distributed database consists of the following components: Source database (currently supporting distributed relational databases, distributed file systems, and unstructured databases); Target database (currently supporting distributed relational databases, distributed file systems, and unstructured databases); Management node cluster (used for data verification configuration, pushing the configuration to verification nodes, and receiving data synchronization status and progress information from the verification nodes); Synchronization node cluster (modules that execute the actual data verification process); and Coordinator cluster (modules that coordinate the data verification).

[0003] When a distributed database is used as a source to synchronize to a non-distributed database, due to differences in the synchronization progress of the underlying sharded databases, cross-shard updates on the source data require log aggregation and sorting to ensure eventual data consistency, which is extremely performance-intensive and has poor operability. When a non-distributed database is used as a source to synchronize to a distributed database, if a certain field is a shard key in the target distributed database, updating the value of this field on the source will prevent direct synchronization on the target because cross-shard operations are prohibited for clients. Furthermore, the data synchronization process cannot be replayed. If a record on the source has undergone consecutive primary key changes, resetting the synchronization point to a previous point in time and re-performing the entire process will result in primary key conflict errors.

[0004] For data synchronization involving distributed databases in related technologies, logs need to be merged, sorted, and aggregated, which leads to poor operability and synchronization failure in some cases. Currently, no effective solution has been proposed. Summary of the Invention

[0005] The main purpose of this application is to provide a data synchronization method and system for databases, in order to solve the problem that data synchronization of distributed databases in related technologies requires merging, sorting and aggregating logs, which leads to poor operability and failure to synchronize in some cases.

[0006] To achieve the above objectives, according to one aspect of this application, a database data synchronization method is provided, comprising: acquiring data to be synchronized from a source database to be synchronized, wherein the data to be synchronized includes data generated by a target event; splitting the target event of the data to be synchronized into first data and second data, wherein the first data is data generated by a first event, and the second data is data generated by a second event, and the superposition of the first event and the second event is the same as the target event; and performing data synchronization based on the first event and the second event to synchronize the data to be synchronized to a corresponding target database, wherein the target database is of a different type than the source database, the target event executes different data structures in different types of databases, and the first event and the second event execute the same data structures in different types of databases.

[0007] Optionally, data synchronization based on the first event and the second event, synchronizing the data to be synchronized to the corresponding target database, includes: determining whether the first data or the second data is modified based on the types of the source database and the target database; if the first data or the second data is modified, inputting the first event and the second event into the target database based on the modified first data or the unmodified second data or the first data, so as to synchronize the data to be synchronized to the target database; and / or, if the first data or the second data is not modified, inputting the first event and the second event into the target database based on the execution attributes of the first data or the second data, so as to synchronize the data to be synchronized to the target database.

[0008] Optionally, if the source database is a distributed database and the target database is a non-distributed database, the following steps are taken: determine that the first data needs to be modified; modify the first data according to a preset method; and input the first event and the second event into the target database based on the modified first data and the unmodified second data, so as to synchronize the data to be synchronized to the target database.

[0009] Optionally, if the source database is a non-distributed database and the target database is a distributed database, it is determined that the first data or the second data does not need to be modified; based on the first execution attribute of the first event and the second event, the first event and the second event are input into the target database to synchronize the data to be synchronized to the target database.

[0010] Optionally, the method further includes: setting attribute parameters of the second data to determine a second execution attribute of the second event; and inputting the first event and the second event into the target database according to the second execution attribute to synchronize the data to be synchronized to the target database.

[0011] Optionally, the target event is an update event, the first event is a delete event, and the second event is an insert event.

[0012] Optionally, modifying the first data according to a preset method includes: adding a shard key to the search conditions in the statement data of the deletion event, wherein the shard key is used to identify the storage location and modification version of the data, and in the case that the first event is a delete event, the first data is the statement data of the deletion event; and / or, the first execution attribute is the attribute of the delete event and the insert event being executed and inserted into the target database; and / or, the second execution attribute is the attribute of the insert event adopting the execution strategy of inserting if it does not exist and updating if it exists when it is inserted into the target database.

[0013] To achieve the above objectives, according to another aspect of this application, a database data synchronization apparatus is provided. The apparatus includes: an acquisition module for acquiring data to be synchronized from a source database, wherein the data to be synchronized includes data generated by a target event; a splitting module for splitting the target event of the data to be synchronized into first data and second data, wherein the first data is data generated by a first event, and the second data is data generated by a second event, and the superposition of the first event and the second event is the same as the target event; and a synchronization module for performing data synchronization based on the first event and the second event, synchronizing the data to be synchronized to a corresponding target database, wherein the target database is of a different type than the source database, the target event executes different data structures in different types of databases, and the first event and the second event execute the same data structures in different types of databases.

[0014] According to another aspect of this application, a computer-readable storage medium is also provided for storing a program, wherein the program executes the database data synchronization method described in any of the preceding claims.

[0015] According to another aspect of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the database data synchronization method described in any of the above.

[0016] This application obtains the data to be synchronized from the source database that needs to be synchronized, wherein the data to be synchronized includes data generated by a target event; the target event of the data to be synchronized is split into first data and second data, wherein the first data is the data generated by the first event and the second data is the data generated by the second event, and the superposition of the first event and the second event is the same as the target event; based on the first event and the second event, data synchronization is performed to synchronize the data to be synchronized to the corresponding target database, wherein the target database and the source database are of different types, the data structure executed by the target event is different in different types of databases, and the data structure executed by the first event and the second event is the same in different types of databases.

[0017] By acquiring the data to be synchronized generated by the target event, the target event is split into a first event and a second event that can be directly synchronized to the target database. This splits the data to be synchronized into first data and second data, achieving the goal of directly splitting and synchronizing the data to be synchronized for the target event, which relies on log merging and aggregation for data synchronization. This reduces the operational difficulty of synchronizing the data to be synchronized at the target time, and effectively and accurately achieves the technical effect of data synchronization without log merging and aggregation. In this way, it solves the problem in related technologies involving distributed database data synchronization, which requires log merging, sorting and aggregation, resulting in poor operability and synchronization failure in some cases. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart of a database data synchronization method provided according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating data synchronization of a distributed database based on the relevant technology provided in the embodiments of this application;

[0021] Figure 3 This is a flowchart illustrating data synchronization using a distributed database according to the embodiments of this application;

[0022] Figure 4 This is a schematic diagram illustrating an actual execution example of data synchronization using a distributed database according to the embodiments of this application;

[0023] Figure 5 This is a flowchart of data synchronization using a non-distributed database as the source, according to the embodiments of this application;

[0024] Figure 6 This is a flowchart illustrating the process of repeatedly inserting data into the database during data synchronization according to the implementation method of this application;

[0025] Figure 7 This is a schematic diagram of a database data synchronization device according to an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.

[0031] Example

[0032] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of a database data synchronization method provided according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0033] Step S101: Obtain the data to be synchronized from the source database that needs to be synchronized, wherein the data to be synchronized includes data generated by the target event;

[0034] Step S102: The target event of the data to be synchronized is split to obtain the first data and the second data. The first data is the data generated by the first event, and the second data is the data generated by the second event. The superposition of the first event and the second event is the same as the target event.

[0035] Step S103: Based on the first event and the second event, perform data synchronization and synchronize the data to be synchronized to the corresponding target database. The target database and the source database are of different types. The target event executes different data structures in different types of databases, while the first event and the second event execute the same data structures in different types of databases.

[0036] Through the above steps, by acquiring the data to be synchronized generated by the target event, the target event is split into a first event and a second event that can be directly synchronized to the target database. This splits the data to be synchronized into first data and second data, achieving the goal of directly splitting and synchronizing the data to be synchronized for the target event, which relies on log merging and aggregation for data synchronization, into the target database. This reduces the operational difficulty of synchronizing the data to be synchronized at the target time, and effectively and accurately achieves the technical effect of data synchronization without log merging and aggregation. In this way, it solves the problem in related technologies involving distributed database data synchronization, which requires merging, sorting and aggregating logs, resulting in poor operability and in some cases, the inability to synchronize.

[0037] The entity performing the above steps can be the aforementioned database, or a functional module for managing the database. This functional module can be located in the aforementioned source database or target database, or it can be located separately outside the aforementioned source database and target database, or it can be located on a third-party device outside the aforementioned source database and target database. By establishing a data link with the aforementioned source database and target database, it manages the data synchronization between the aforementioned source database and target database.

[0038] The data to be synchronized includes data that has changed in the database when the target event is an update event, allowing for incremental data synchronization by recording these changes. Distributed and non-distributed databases have different data structures. For update events, non-distributed databases cannot determine the order of data, while distributed databases can determine it based on the shard key. This creates a certain obstacle to synchronizing update events between distributed and non-distributed databases.

[0039] This embodiment splits the update event into a first event and a second event. The sum of the first and second events is identical to the target event, i.e., the update event. The first event can be a delete event, and the second event can be an insert event. In both distributed and non-distributed databases, delete and insert events can be directly synchronized. Furthermore, when the update event is split, the delete and insert events have a specific order. Therefore, by using the sequential delete and insert events, the data from the update event can be directly synchronized to the target database.

[0040] Optionally, data synchronization based on the first event and the second event, synchronizing the data to be synchronized to the corresponding target database, includes: determining whether the first data or the second data should be modified based on the types of the source database and the target database; if the first data or the second data is modified, inputting the first event and the second event into the target database based on the modified first data or the unmodified second data or the first data, so as to synchronize the data to be synchronized to the target database; and / or, if the first data or the second data is not modified, inputting the first event and the second event into the target database based on the execution attributes of the first data or the second data, so as to synchronize the data to be synchronized to the target database.

[0041] Specifically, in implementation, there are generally two scenarios: First, the source database is a distributed database, and the target database is a non-distributed database. Second, the source database is a non-distributed database, and the target database is a distributed database. In these two scenarios, the handling of delete and insert events differs.

[0042] Specifically, in the first scenario, considering the order of the delete and insert events, accidental deletion might occur when the delete event is executed because the data versions before and after the insert event are different. Therefore, it is necessary to add a data version identifier, i.e., the shard key, to the delete event to ensure that the data targeted by the delete event is accurate.

[0043] Optionally, if the source database is a distributed database and the target database is a non-distributed database, the first data needs to be modified; the first data is modified according to a preset method; based on the modified first data and the unmodified second data, the first event and the second event are input into the target database to synchronize the data to be synchronized to the target database.

[0044] In the second scenario, considering performance and security factors, distributed databases typically prohibit clients from performing cross-shard updates. This can lead to data synchronization issues when the source database is not distributed and related fields cannot be synchronized. Therefore, parsing the update event and constructing delete and insert events based on the information before and after the update event is equivalent to transforming statements that cannot be directly executed, thus inserting them into the target distributed database and achieving eventual consistency in data synchronization. In other words, it's not necessary to modify the delete and insert events; simply splitting the update event into delete and insert events is sufficient for data synchronization.

[0045] Optionally, if the source database is a non-distributed database and the target database is a distributed database, determine that the first data or the second data does not need to be modified; based on the first execution attribute of the first event and the second event, input the first event and the second event into the target database to synchronize the data to be synchronized to the target database.

[0046] It should be noted that, regardless of whether it's the first or second scenario, the `insert` event can employ a strategy of inserting if the data doesn't exist and updating if it does. This ensures data security before and after the `insert` event is executed.

[0047] Optionally, the method further includes: setting attribute parameters for the second data to determine the second execution attribute of the second event; and inputting the first event and the second event into the target database according to the second execution attribute to synchronize the data to be synchronized to the target database.

[0048] Optionally, modifying the first data according to a preset method includes: adding a shard key to the search conditions in the statement data of the delete event, wherein the shard key is used to identify the storage location and modification version of the data, and in the case of the first event being a delete event, the first data is the statement data of the delete event; and / or, the first execution attribute is the attribute for executing delete events and insert events to insert data into the target database; and / or, the second execution attribute is the attribute for the insert event to adopt the execution strategy of inserting if the data does not exist and updating if the data exists when inserting data into the target database.

[0049] Furthermore, during implementation, there are often continuous primary key changes to data over a period of time. The data synchronization component replays these changes through log parsing. Sometimes, due to data repair or other reasons, it's necessary to adjust the synchronization point to a historical time and synchronize again. In this case, without cleaning the target table, primary key conflicts may occur, preventing data from being inserted into the target database. The repeatable data synchronization method breaks down update events into delete and insert statements, and executes a strategy of inserting if the data doesn't exist and updating if it does, thus achieving replayability of the data synchronization process without affecting eventual data consistency.

[0050] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0051] It should be noted that this application also provides an optional implementation method, which will be described in detail below.

[0052] This implementation provides a replayable cross-shard synchronization method between non-distributed and distributed databases. It mainly includes methods for synchronizing data from a distributed database (as the source) to a non-distributed database when cross-shard updates occur; methods for inserting data from a non-distributed database (as the source) where a certain field is a shard key on the target database; and methods for repeatable data synchronization. This implementation cleverly avoids the log merging, sorting, and aggregation steps that are essential for synchronizing data from a distributed to a non-distributed database by using techniques such as equivalent data splitting, data insertion constraints, and reasonable invocation of native database primary key merging. This significantly improves the performance and operability of data synchronization; it overcomes the limitation that cross-shard updates cannot be performed when synchronizing data from a non-distributed database to a distributed database; and it achieves repeatable data insertion, eliminating the need to clean up existing data on the target database during data repair. Eventual data consistency is achieved when the synchronization points are aligned, making it highly valuable in practical operations and maintenance.

[0053] 1. Methods for synchronizing data from a distributed database (as the source) to a non-distributed database when cross-shard updates occur:

[0054] Figure 2 This is a flowchart illustrating data synchronization of a distributed database based on the relevant technology provided in the embodiments of this application, such as... Figure 2 As shown, the traditional processing flow is as follows: Figure 2 The log aggregation and sorting steps are designed to ensure eventual consistency even when there are discrepancies in the synchronization progress of the underlying sharded databases. Traditional methods are extremely performance-intensive and have poor operability.

[0055] Figure 3 This is a flowchart illustrating data synchronization using a distributed database according to the embodiments of this application, such as... Figure 3 As shown, this implementation method handles cross-shard update actions on the source end by splitting the delete and insert actions at the proxy layer. After log parsing, the logs of all underlying databases are read independently for synchronization. Additional processing is applied to the SQL statements during insertion: a constraint condition is added to the delete statement, specifically, a shard key is added to the original WHERE clause in addition to the primary key field; for inserts, a strategy of inserting if the statement does not exist and updating if it does exist is adopted. After these processing steps, regardless of whether the delete or insert statement is executed first on the target end, the eventual consistency of data synchronization will not be affected, and complex log sorting and merging processes are not required.

[0056] Figure 4 This is a schematic diagram illustrating an actual execution example of data synchronization using a distributed database according to the embodiments of this application, such as... Figure 4 As shown, with Figure 4Taking a specific execution process as an example, the method described in this implementation executes the `delete` and `insert...on duplicate key update` statements on different shards of the target distributed database during data synchronization. The order in which these statements are executed does not affect the eventual consistency of the data synchronization.

[0057] 1) The insert...on duplicate key update statement is executed first, followed by the delete statement. Since the where condition in the delete statement includes the shard key, the data inserted or updated by the insert statement executed first will not be broadcast deleted.

[0058] 2) Executing the delete statement first, followed by the insert...on duplicate key update statement, is consistent with the expected order, and data synchronization can achieve consistency.

[0059] 2. Method for inserting data into a non-distributed database when a certain field is a sharding key on the target database:

[0060] Figure 5 This is a flowchart illustrating data synchronization using a non-distributed database as the source, according to an embodiment of this application. Figure 5 As shown, distributed databases typically prohibit clients from performing cross-shard updates for performance and security reasons. This can lead to data synchronization problems when the source database is not distributed and related fields are involved. The method proposed in this implementation parses update events at the data synchronization component level, constructing delete and insert events based on the information before and after the update. This is equivalent to transforming statements that cannot be directly executed, thereby inserting them into the target distributed database and achieving eventual consistency in data synchronization.

[0061] 3. Data synchronization and repeatable data entry method:

[0062] In business scenarios, it's common for a single record to undergo continuous primary key changes over a period of time. The data synchronization component replays these changes through log parsing. Sometimes, due to data repair or other reasons, it's necessary to adjust the synchronization point to a historical timeframe and synchronize again. In this case, without cleaning the target table, primary key conflicts may occur, preventing data from being inserted into the target database. The repeatable data synchronization method addresses this by splitting update events into delete and insert statements and implementing a strategy of inserting if the data doesn't exist and updating if it does. This allows the data synchronization process to be replayable without affecting eventual data consistency.

[0063] Figure 6 This is a flowchart illustrating the process of repeatedly inserting data into the database during data synchronization according to the implementation method of this application, such as... Figure 6 As shown, Figure 6 The differences between traditional methods and the method of this implementation in handling repeated data synchronization were compared. Using this method, the data synchronization process can be replayed without cleaning the target table, even if the position is arbitrarily adjusted. It has strong operability and practicality.

[0064] Compared to existing technologies, this implementation cleverly uses inbound database constraints to bypass the log merging, sorting, and aggregation steps in the synchronization process from distributed to non-distributed databases, reducing the overall complexity of the synchronization system and significantly improving synchronization speed. It utilizes data equivalence partitioning technology to overcome the limitation that cross-shard updates cannot be performed during synchronization from a non-distributed database to a distributed database, while ensuring eventual consistency. By appropriately applying native database primary key conflict merging technology in conjunction with the data equivalence partitioning method, it achieves repeatable data synchronization, demonstrating strong operability and operational significance in practice.

[0065] For example, in a production system, a distributed database based on MySQL and an Oracle database are used simultaneously based on different responsibilities. Initially, a full data cutover is required to import all business data from the distributed database into the Oracle database. Then, approximately 110GB of business-related data generated in the distributed database is synchronized to the Oracle database every day, while relevant configuration data in the Oracle database is synchronized to the distributed database.

[0066] The specific implementation steps are as follows:

[0067] 1. The data synchronization system developed using the method of this embodiment can perform full synchronization and real-time incremental synchronization of table data.

[0068] 2. In this embodiment, data synchronization between distributed and non-distributed databases is required, so two separate synchronization links are established. One link is from the distributed database to the Oracle database, and the other is from the Oracle database to the distributed database. By filling in the corresponding source database information and target server information, the synchronization mapping relationship can be automatically generated with one click.

[0069] 3. Start a full data synchronization from the distributed database to the Oracle database and record the start time of the full synchronization;

[0070] 4. Adjust the incremental synchronization point to before the full time point, and enable incremental synchronization between the distributed database and the Oracle database. Cross-shard update operations are handled by the method described in this implementation, requiring no additional user intervention.

[0071] 5. Enable the synchronization link between the Oracle database and the distributed database, and perform incremental synchronization of the configured data. When a field is a shard key in the distributed database, due to the setting that prohibits cross-shard actions, the update statement cannot be executed directly in the distributed database. The synchronization system will use the method described in this implementation to perform equivalent data splitting and synchronize it to the target end.

[0072] 6. Occasionally, during routine synchronization, applications may accidentally delete some incremental data synchronized from the distributed database to the Oracle database. The synchronization point can be adjusted according to the time, and the data synchronization process can be redone using the method described in this implementation, thus achieving the effect of data repair.

[0073] Seamless and efficient synchronization between distributed and non-distributed databases; the data synchronization process can be redone; incremental synchronization points can be adjusted arbitrarily according to time points; and data can achieve eventual consistency once incremental synchronization catches up.

[0074] This application also provides a database data synchronization device. It should be noted that the database data synchronization device of this application can be used to execute the database data synchronization method provided in this application. The database data synchronization device provided in this application will be described below.

[0075] Figure 7 This is a schematic diagram of a database data synchronization device according to an embodiment of this application, such as... Figure 7 As shown, the device includes: an acquisition module 72, a splitting module 74, and a synchronization module 76. The device will be described in detail below.

[0076] The acquisition module 72 is used to acquire the data to be synchronized from the source database that needs to be synchronized, wherein the data to be synchronized includes data generated by the target event; the splitting module 74, connected to the acquisition module 72, is used to split the target event of the data to be synchronized into first data and second data, wherein the first data is the data generated by the first event and the second data is the data generated by the second event, and the superposition of the first event and the second event is the same as the target event; the synchronization module 76, connected to the splitting module 74, is used to perform data synchronization based on the first event and the second event, and synchronize the data to be synchronized to the corresponding target database, wherein the target database and the source database are of different types, the data structure executed by the target event is different in different types of databases, and the data structure executed by the first event and the second event is the same in different types of databases.

[0077] The data synchronization device for a database provided in this application acquires the data to be synchronized generated by a target event, and splits the target event into a first event and a second event that can be directly synchronized to the target database. This splits the data to be synchronized into first data and second data, achieving the goal of directly splitting and synchronizing the data to be synchronized for the target event, which relies on log merging and aggregation for data synchronization, into the target database. This reduces the operational difficulty of synchronizing the data to be synchronized at the target time, and effectively and accurately achieves data synchronization without the need for log merging and aggregation. It also solves the problem in related technologies involving distributed database data synchronization that requires merging, sorting, and aggregating logs, resulting in poor operability and in some cases, the inability to synchronize.

[0078] The database data synchronization device includes a processor and a memory. The aforementioned acquisition module 72, splitting module 74, synchronization module 76, etc., are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0079] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. Adjusting kernel parameters addresses data synchronization issues in distributed databases, which often require merging, sorting, and aggregating logs, leading to poor operability and, in some cases, synchronization failure.

[0080] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0081] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements a database data synchronization method.

[0082] This invention provides a processor for running a program, wherein the program executes a database data synchronization method during runtime.

[0083] Figure 8 This is a schematic diagram of an electronic device provided according to an embodiment of this application, such as... Figure 8 As shown, this application embodiment provides an electronic device 80, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the above methods.

[0084] The device in this application may be a server, PC, PAD, mobile phone, etc.

[0085] This application also provides a computer program product that, when executed on a database data synchronization device, is suitable for executing a program that initializes any of the above-described method steps.

[0086] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable database data synchronization device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable database data synchronization device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a data synchronization device of a computer or other programmable database to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable database data synchronization device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0091] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0092] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0093] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0094] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A database data synchronization method, characterized in that, include: Obtain the data to be synchronized from the source database that needs to be synchronized, wherein the data to be synchronized includes data generated by a target event, and the target event is a modification update event; The target event of the data to be synchronized is split into first data and second data. The first data is the data generated by the first event, and the second data is the data generated by the second event. The superposition of the first event and the second event is the same as the target event. The first event is a delete event, and the second event is an insert event. The attribute parameters of the second data are set to determine the second execution attribute of the second event. The second execution attribute is the attribute of the execution strategy of inserting the data into the target database if it does not exist, and updating it if it exists. According to the second execution attribute, the first event and the second event are synchronized to synchronize the data to be synchronized to the corresponding target database. The target database and the source database are of different types. The data structure executed by the target event is different in different types of databases. The data structure executed by the first event and the second event is the same in different types of databases. The execution order of the first event and the second event during the synchronization process is not limited. Specifically, data synchronization based on the first event and the second event, synchronizing the data to be synchronized to the corresponding target database includes: When the source database is a distributed database and the target database is a non-distributed database, it is determined that the first data needs to be modified; the first data is modified according to a preset method; based on the modified first data and the unmodified second data, the first event and the second event are input into the target database to synchronize the data to be synchronized to the target database. When the source database is a non-distributed database and the target database is a distributed database, it is determined that the first data or the second data does not need to be modified; based on the first execution attribute of the first event and the second event, the first event and the second event are input into the target database to synchronize the data to be synchronized to the target database. The first execution attribute is the attribute of the delete event and the insert event for executing and entering the data into the target database. Modifying the first data according to a preset method includes: A sharding key has been added to the search criteria in the statement data of the deletion event. The sharding key is used to identify the storage location and modification version of the data. In the case that the first event is a delete event, the first data is the statement data of the deletion event.

2. A data synchronization device for a database, characterized in that, include: The acquisition module is used to acquire the data to be synchronized from the source database that needs to be synchronized. The data to be synchronized includes data generated by a target event, wherein the target event is a modification update event. The splitting module is used to split the target event of the data to be synchronized into first data and second data. The first data is the data generated by the first event, and the second data is the data generated by the second event. The superposition of the first event and the second event is the same as the target event. The first event is a delete event, and the second event is an insert event. The setting module is used to set the attribute parameters of the second data and determine the second execution attribute of the second event. The second execution attribute is the attribute of the insert event when it is inserted into the target database using the execution strategy of inserting if it does not exist and updating if it exists. The synchronization module is used to synchronize the first event and the second event according to the second execution attribute, and synchronize the data to be synchronized to the corresponding target database. The target database is of a different type than the source database. The target event executes different data structures in different types of databases. The first event and the second event execute the same data structures in different types of databases. The execution order of the first event and the second event during the synchronization process is not limited. Specifically, the synchronization module is used for: When the source database is a distributed database and the target database is a non-distributed database, it is determined that the first data needs to be modified; the first data is modified according to a preset method; based on the modified first data and the unmodified second data, the first event and the second event are input into the target database to synchronize the data to be synchronized to the target database. When the source database is a non-distributed database and the target database is a distributed database, it is determined that the first data or the second data does not need to be modified; based on the first execution attribute of the first event and the second event, the first event and the second event are input into the target database to synchronize the data to be synchronized to the target database. The first execution attribute is the attribute of the delete event and the insert event for executing and entering the data into the target database. Modifying the first data according to a preset method includes: A sharding key has been added to the search criteria in the statement data of the deletion event. The sharding key is used to identify the storage location and modification version of the data. In the case that the first event is a delete event, the first data is the statement data of the deletion event.

3. A computer-readable storage medium, characterized in that, The storage medium is used to store a program, wherein the program executes the database data synchronization method of claim 1.

4. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the database data synchronization method of claim 1.

Images