Data synchronization method and device, electronic equipment and storage medium

Abstract

The invention provides a data synchronization method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a transaction identifier under the condition that a transaction is detected, adding the transaction identifier into a data change identifier corresponding to data change information belonging to the transaction, and sending the data change identifier to a downstream data pool, the downstream data pool obtains the data change identifiers based on the GTID and compares the data change identifiers with the number of the data change messages contained in the actual transaction, and if the data change identifiers are consistent with the number of the data change messages, the data consumption authority is started. A transaction identifier is added in a data change mark sent to a downstream data pool, whether the number of change events of the downstream data pool is consistent with the number of change messages of a transaction in MySQL is determined based on the transaction identifier, and under the condition that the number of change events of the downstream data pool is consistent with the number of change messages of the transaction, consumption permissions of the data are started. According to the method, synchronous processing and re-consumption of the change messages belonging to the same transaction are ensured, and a data intermediate state is avoided, so that the data consistency of the MySQL and the downstream data pool is improved, and the data reading accuracy is further improved.

Description

Technical Field

[0001] The present invention relates to the field of data synchronization technology, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art

[0002] With the widespread adoption and application of databases today, data migration between MySQL and other software has become increasingly important. For a variety of scenarios, such as real-time data stream processing and the decoupling of data production and consumption, MySQL data is sent to Kafka for processing. Related technologies typically achieve real-time data synchronization and processing by parsing MySQL's Binlog logs. Specifically, data synchronization software captures incremental data changes from MySQL's Binlog, parses them into structured events, and cleans, filters, and converts the data before pushing it to downstream systems (such as Kafka message queues, data warehouses, or stream processing platforms).

[0003] However, the data in MySQL has transaction attributes, which are used to identify the synchronization of data processing. Each piece of data in a transaction usually needs to be processed synchronously, but the data in the downstream Kafka does not have this attribute. As a result, when a transaction in MySQL contains multiple SQL statements, an intermediate state will appear in Kafka, which may cause users to read dirty data, that is, data that has not been processed yet, reducing the accuracy of data reading. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data synchronization method, device, electronic device, and storage medium to improve the accuracy of data reading.

[0005] According to one aspect of the present invention, a data synchronization method is provided, the method comprising:

[0006] When the received upstream MySQL data contains a transaction event, obtain the transaction identifier corresponding to the transaction event, generate a start identifier corresponding to the transaction event, and send it to the downstream data pool;

[0007] Performing format conversion on each data change message received after the transaction event to obtain a data change identifier, adding the transaction identifier to the data change identifier, and sending the data change identifier with the transaction identifier added to the downstream data pool, so that the downstream data pool performs data changes according to each data change message;

[0008] When the received upstream MySQL data includes an end event, an end identifier corresponding to the end event is generated, and the end identifier, the transaction identifier, and the number of data change information of the data change information generated between the transaction event and the end event are sent to the downstream data pool, so that after receiving the end identifier, the downstream data pool counts the number of received data change identifiers corresponding to the transaction identifier and determines whether the number of data change identifiers is consistent with the number of change information;

[0009] When the number of the data change identifiers is consistent with the number of the change information, the consumption rights of the data corresponding to each data change identifier are enabled.

[0010] In a possible embodiment, the data change identifier includes a data change type, and the data change type is insert, update, or delete; and counting the number of received data change identifiers corresponding to the transaction identifier and determining whether the number of data change identifiers is consistent with the number of change information includes:

[0011] The total number of changes of data change identifiers of the insert, update, and delete types with the same transaction identifier is counted, and it is determined whether the total number of changes is consistent with the number of data change information.

[0012] In a possible embodiment, the method further includes:

[0013] When the number of the data change identifiers is inconsistent with the number of the change information, wait for receiving the remaining data change identifiers included in the transaction event until the number of the data change identifiers is consistent with the number of the change information.

[0014] In a possible embodiment, before counting the number of received data change identifiers corresponding to the transaction identifier, the method further includes:

[0015] Performing a hash calculation based on each of the data change messages and the transaction identifier to obtain a unique identifier corresponding to each of the data change messages, and adding the unique identifier to the data change identifier;

[0016] The downstream data pool deduplicates each data change identifier based on the unique identifier to obtain deduplicated data change identifiers.

[0017] According to another aspect of the present invention, a data synchronization device is provided, the device comprising:

[0018] An acquisition module is used to obtain a transaction identifier corresponding to a transaction event when the received upstream MySQL data contains a transaction event, and to generate a start identifier corresponding to the transaction event and send it to the downstream data pool;

[0019] an adding module, configured to convert the format of each data change message received after the transaction event to obtain a data change identifier, add the transaction identifier to the data change identifier, and send the data change identifier with the transaction identifier added to the downstream data pool, so that the downstream data pool performs data changes according to each data change message;

[0020] A generation module is configured to, when the received upstream MySQL data contains an end event, generate an end identifier corresponding to the end event, and send the end identifier, the transaction identifier, and the number of data change information of the data change information generated between the transaction event and the end event to the downstream data pool, so that after receiving the end identifier, the downstream data pool counts the number of received data change identifiers corresponding to the transaction identifier and determines whether the number of data change identifiers is consistent with the number of change information; and when the number of data change identifiers is consistent with the number of change information, enable consumption rights for each data corresponding to each data change identifier.

[0021] In a possible embodiment, the data change identifier includes a data change type, and the data change type is insert, update, or delete; and counting the number of received data change identifiers corresponding to the transaction identifier and determining whether the number of data change identifiers is consistent with the number of change information includes:

[0022] The total number of changes of data change identifiers of the insert, update, and delete types with the same transaction identifier is counted, and it is determined whether the total number of changes is consistent with the number of data change information.

[0023] In a possible embodiment, the generation module is configured to, when the number of data change identifiers is inconsistent with the number of change information, wait to receive the remaining data change identifiers included in the transaction event until the number of data change identifiers is consistent with the number of change information.

[0024] In a possible embodiment, the apparatus further includes: a deduplication module, configured to perform a hash calculation based on each of the data change messages and the transaction identifier to obtain a unique identifier corresponding to each of the data change messages, and add the unique identifier to the data change identifier;

[0025] The downstream data pool deduplicates each data change identifier based on the unique identifier to obtain deduplicated data change identifiers.

[0026] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0027] processor; and

[0028] Memory for storing programs,

[0029] The program includes instructions, which, when executed by the processor, enable the processor to execute any of the above-mentioned data synchronization methods.

[0030] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned data synchronization methods.

[0031] One or more technical solutions provided in the embodiments of the present invention, when a transaction is detected, obtain the transaction identifier, and add the identifier to the data change identifier corresponding to the data change information belonging to the transaction, send the data change identifier to the downstream data pool, so that the downstream data pool performs synchronized data changes, and when a transaction end event is detected, generate a corresponding end identifier and send it to the downstream data pool, the downstream data pool obtains the data change identifier of the transaction based on the GTID, and compares the data change identifier with the number of data change messages contained in the actual transaction. If they are consistent, it is determined that the data synchronization is complete and the data consumption permission can be enabled. Applying the embodiments of the present invention, by adding a transaction identifier to the data change mark sent to the downstream data pool, and determining whether the number of change events in the downstream data pool is consistent with the number of change messages of the transaction in MySQL based on the transaction identifier, and enabling the consumption permission of these data in the case of consistency, it ensures that the change messages belonging to the same transaction are synchronously processed and then consumed, avoiding the occurrence of data intermediate states, thereby improving the data consistency between MySQL and the downstream data pool, and further improving the accuracy of data reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Further details, features and advantages of the present invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A schematic diagram of a flow chart of a data synchronization method provided by an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the interaction between the data synchronization software provided by an embodiment of the present invention and an upstream database;

[0035] Figure 3 A schematic diagram of another flow chart of a data synchronization method provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of the judgment logic of the data synchronization software provided by an embodiment of the present invention;

[0037] Figure 5 A schematic structural diagram of a data synchronization device provided by an embodiment of the present invention;

[0038] Figure 6 A block diagram of an exemplary electronic device capable of implementing the embodiments of the present invention is shown. DETAILED DESCRIPTION

[0039] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0040] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0041] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0042] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0043] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0044] Currently, data synchronization from MySQL to Kafka ignores MySQL transaction attributes. This results in downstream Kafka being unable to determine which SQL changes were implemented as a single transaction upstream after receiving messages corresponding to all SQL changes. This can lead to the reading of dirty intermediate data. For example, if three SQL statements A, B, and C are contained in a single transaction in the upstream database and are transmitted to the downstream database via data transmission software, only changes A and B are received downstream. The third change has not yet been sent downstream. At this point, the downstream consumer obtains all the data in Kafka for consumption. However, because the consumer does not receive the change corresponding to SQL statement C, the Kafka consumer receives dirty intermediate data.

[0045] Based on this, the present invention provides a data synchronization method, apparatus, electronic device, and storage medium. The data synchronization method provided in embodiments of the present invention can be applied to any electronic device with data synchronization capabilities, such as a server, computer, or mobile terminal. In one possible embodiment, the data synchronization method provided by the present invention can be applied to data synchronization software, which is used to synchronize data between an upstream database and a downstream data pool. The following describes the solution of the present invention with reference to the accompanying drawings:

[0046] Figure 1 A schematic flow chart of a data synchronization method provided in an embodiment of the present invention may include the following steps:

[0047] S101. When the received upstream MySQL data contains a transaction event, obtain a transaction identifier corresponding to the transaction event;

[0048] S102: Convert the format of each data change message received after the transaction event to obtain a data change identifier, add the transaction identifier to the data change identifier, and send the data change identifier with the transaction identifier added to the downstream data pool, so that the downstream data pool performs data changes according to each data change message;

[0049] S103. When the received upstream MySQL data includes an end event, generate an end identifier corresponding to the end event, and send the end identifier, the transaction identifier, and the number of data change information generated between the transaction event and the end event to the downstream data pool, so that after receiving the end identifier, the downstream data pool counts the number of received data change identifiers corresponding to the transaction identifier and determines whether the number of data change identifiers is consistent with the number of change information;

[0050] S104: When the number of the data change identifiers is consistent with the number of the change information, the consumption rights of the data corresponding to the data change identifiers are enabled.

[0051] In an embodiment of the present invention, when a transaction is detected, the transaction identifier is obtained, and the identifier is added to the data change identifier corresponding to the data change information belonging to the transaction, and the data change identifier is sent to the downstream data pool, so that the downstream data pool performs synchronized data changes. When a transaction end event is detected, a corresponding end identifier is generated and sent to the downstream data pool. The downstream data pool obtains the data change identifier of the transaction based on the GTID, and compares the data change identifier with the number of data change messages contained in the actual transaction. If they are consistent, it is determined that the data synchronization is complete and the data consumption permission can be enabled. Applying the embodiment of the present invention, by adding a transaction identifier to the data change mark sent to the downstream data pool, and determining whether the number of change events in the downstream data pool is consistent with the number of change messages of the transaction in MySQL based on the transaction identifier, and enabling the consumption permission of these data in the case of consistency, it is ensured that the change messages belonging to the same transaction are synchronously processed and then consumed, avoiding the occurrence of data intermediate states, thereby improving the data consistency between MySQL and the downstream data pool, and further improving the accuracy of data reading.

[0052] The following is an exemplary description of S101-S104:

[0053] The embodiments of the present invention can be applied to synchronization scenarios between upstream databases and downstream data pools. The types of the upstream database and downstream data pool can be set according to the actual application scenario. For example, the upstream database can be MySQL, and the downstream data pool can be a Kafka message queue, data warehouse, or stream processing platform. The following uses MySQL as the upstream database and Kafka as the downstream data pool to illustrate the data synchronization method provided by the present invention:

[0054] MySQL is a database that takes on the main data storage function. When data changes occur in actual business scenarios, such as when data is deleted, inserted or updated, the corresponding data changes will first be made in MySQL. MySQL usually uses Binlog (Binary Log) to record various data change operations that occur in the database. Binlog is a binary log file provided by the MySQL database, which is used to record all change operations performed on the database (such as INSERT, UPDATE, DELETE, etc.), but does not include query operations (such as SELECT). Binlog data can be used as a source of data synchronization, such as synchronizing MySQL data to a big data platform (such as Kafka, Hadoop) or a heterogeneous database. In a possible embodiment, the data synchronization software can obtain MySQL's binlog records in real time, determine whether data changes occur, and synchronize the data change events to Kafka if data changes occur. Specifically, the data synchronization software can connect to the upstream MySQL as a MySQL slave library, read the binlog in MySQL, read the changes in the binlog in sequence and pass them to the downstream.

[0055] In MySQL, data changes can be made by submitting transactions. In MySQL, a transaction is a logical execution unit consisting of a series of database operations (such as DML statements such as INSERT, UPDATE, and DELETE). These operations are either all executed successfully or all failed and rolled back to ensure the consistency and integrity of the database state.

[0056] In one possible embodiment, the data synchronization software can obtain information from the binlog according to a preset period. This period can be set according to the actual application scenario, such as 0.1 seconds, 10 minutes, etc. In other possible embodiments, the binlog synchronization period can also be set in MySQL, and MySQL can send the binlog to the data synchronization software according to this synchronization period.

[0057] In the binlog, different types of events are recorded using different identifiers. For example, a DML change SQL statement is a Row Event, a transaction start statement (Begin statement) corresponds to a Query Event, and a Commit statement corresponds to a Xid Event. MySQL executes transactions in the order of Begin statement first, DML change SQL statement second, and Commit statement last. Therefore, the order of a transaction in the binlog is Query Event -> Row Event -> Xid Event. Accordingly, data synchronization software can determine that a transaction event has occurred in MySQL by detecting Query Event data in the binlog.

[0058] After the data synchronization software detects a transaction event, it can obtain the transaction identifier Gtid (Global Transaction Identifier) ​​corresponding to the transaction event. GTID is a unique identifier used by MySQL to identify each transaction. The format is server_uuid:transaction_id, for example, 3E11FA47-71CA-11E1-9E33-C80AA9429562:1. Each transaction generates a unique GTID on the master database where it is executed and is passed to the slave database along with the Binlog during master-slave replication. Data change messages generated after the transaction event in the Binlog belong to that transaction. Therefore, when the data synchronization software synchronizes data change messages to the downstream data pool, it can add the GTID of the transaction event to the synchronization information to indicate that these data change messages belong to the same transaction event.

[0059] As one possible implementation, upon receiving a Row Event, the data synchronization software converts the SQL change event into a JSON message and transmits it to the downstream Kafka. This JSON message contains a status value that identifies the specific change type. Specifically, I represents an INSERT change, U represents an UPDATE change, and D represents a DELETE change. The data synchronization software can also add the GTID of the transaction event to the JSON message to identify the transaction to which the data change information belongs. For ease of explanation, the JSON message sent to the downstream data pool is referred to as the data change identifier.

[0060] After the downstream data pool receives the data change identifier, it can perform a specific change operation according to the data change type contained in the data change identifier and the target data that needs to be changed, until the downstream data pool receives the transaction end identifier. During the process of data change in the downstream data pool, the data synchronization software can count the received data change messages. For example, the data synchronization software can start a counter from the first data change identifier received after detecting the transaction event, and count up by 1. After that, the data synchronization software counts up by 1 each time a data change message is received.

[0061] As described above, the Xid Event is used to identify the end of the transaction. As a possible implementation, after the data synchronization software detects the end event Xid Event corresponding to the transaction event in the binlog, the data synchronization software can also convert the Xid Event into a JSON message, and the type is C, which represents COMMIT. At the same time, the GTID corresponding to the current transaction and the count of SQL in the transaction can also be put into the JSON message. The count of SQL in the transaction is specifically the number of data change information generated in Mysql between the transaction event and the end event. For ease of description, the JSON message corresponding to the end event is referred to as the end message hereinafter.

[0062] After the downstream data pool receives the end message, it can filter each data change identifier corresponding to the GTID contained in the end message, and determine the number of corresponding data change identifiers. As described above, the data change identifier contains a specific data change type, which can be insertion, deletion or modification. Therefore, as a possible implementation, the downstream data pool can count the total number of data change identifiers of the same transaction identifier, and determine whether the total number of changes is consistent with the number of data change information.

[0063] If the total number of data change identifiers is consistent with the number of data change information contained in the end message, it means that all changes in the transaction have been received, and the data change in the downstream data pool is consistent with the data change in MySQL. Therefore, data consumption can be performed, and therefore the consumption permission of these data can be opened, that is, these data can be queried and used.

[0064] If the total number of data change identifiers is not consistent with the number of data change information contained in the end message, it means that the change event contained in the transaction may not have been completely transmitted to Kafka. Therefore, the remaining data change identifiers contained in the transaction event can be continued to be received until the total number of data change identifiers is consistent with the number of data change messages in the end message before consumption.

[0065] During data transmission, the data transmission software may encounter a data transmission task restart, which may cause data synchronization to be repeated. Therefore, in a possible embodiment, the above method may also include: the data synchronization software performs a hash calculation based on each of the data change messages and the transaction identifier to obtain a unique identifier corresponding to each of the data change messages, and adds the unique identifier to the data change identifier; the downstream data pool deduplicates each data change identifier based on the unique identifier to obtain deduplicated data change identifiers.

[0066] In other words, data synchronization software can generate a unique identifier for each SQL statement and send this unique identifier to the downstream data pool. The downstream data pool can then use this unique identifier to deduplicate each data change identifier and count the number of data change identifiers after deduplication. For example, the data synchronization software can perform a hash calculation based on each SQL statement's GTID and identifying information such as the primary key and unique key to derive a unique numeric string for each change. The downstream data pool can then use this string to deduplicate data. This way, even if there are duplicates, all SQL statements in each transaction can be accurately retrieved.

[0067] like Figure 2 As shown, Figure 2 This diagram illustrates the interaction between the data synchronization software River and MySQL. Specifically, a transaction commit occurs in MySQL and is written to the binlog. MySQL periodically sends the binlog to the data synchronization software and returns a successful result. The data synchronization software parses the binlog data packets, formats the information within them, converts it into a JSON message (Core.Msg{}), and sends it to the downstream data pool.

[0068] like Figure 3 The data synchronization software River shown can include a data input module INPUT, a data filtering module EMITTER, a data scheduling module SCHEDULER and a data output module OUTPUT, wherein the data input module is used to receive binlog input from the upstream database, the data filtering module is used to filter data change messages, the data scheduling module is used to convert the binlog format to obtain JSON messages, and the data output module is used to send JSON messages to downstream services.

[0069] Figure 4A flowchart of a data synchronization method provided by an embodiment of the application is shown, specifically, a message enters a transmission service (data synchronization software), the data synchronization software can judge the message type, if it is a Row Event, the same GTID count is started, and subsequent data change messages are transmitted to a downstream, if it is an Xid Event, the GTID and the count result are transmitted to the downstream together. The downstream service judges the data type, and filters all changes according to the GTID and the count information, and compares the filtered data change identifiers consistent with the count information, to determine whether to start the consumption permission of the corresponding data.

[0070] In the synchronization from MySQL to Kafka, the transaction attribute is added according to the embodiment of the application, so that the consumers of Kafka can consume together after all the SQLs of the same transaction are obtained, to prevent the intermediate state of the data from being consumed, and to prevent dirty data from being read by the downstream.

[0071] Based on the same inventive concept, the application further provides a data synchronization device, as shown in the figure, the device 500 can include: Figure 5

[0072] The acquisition module 501 is configured to acquire a transaction identifier corresponding to a transaction event in the case that the received upstream MySQL data contains the transaction event, and generate a start identifier corresponding to the transaction event and send the start identifier to a downstream data pool;

[0073] The increase module 502 is configured to perform format conversion on each data change message received after the transaction event, to obtain a data change identifier, add the transaction identifier to the data change identifier, and send the data change identifier with the transaction identifier added to the downstream data pool, so that the downstream data pool performs data change according to each data change message;

[0074] The generation module 503 is configured to generate an end identifier corresponding to an end event in the case that the received upstream MySQL data contains the end event, and send the end identifier, the transaction identifier, and the data change information quantity of the data change information generated between the transaction event and the end event to the downstream data pool, so that the downstream data pool, after receiving the end identifier, counts the quantity of the data change identifiers corresponding to the transaction identifier received, and determines whether the quantity of the data change identifiers is consistent with the quantity of the change information; in the case that the quantity of the data change identifiers is consistent with the quantity of the change information, the consumption permission of each data corresponding to each data change identifier is started.

[0075] ​In a possible embodiment, the data change identifier includes a data change type, and the data change type is insert, update, or delete; and counting the number of received data change identifiers corresponding to the transaction identifier and determining whether the number of data change identifiers is consistent with the number of change information includes:

[0076] The total number of changes of data change identifiers of the insert, update, and delete types with the same transaction identifier is counted, and it is determined whether the total number of changes is consistent with the number of data change information.

[0077] In a possible embodiment, the generation module is configured to, when the number of data change identifiers is inconsistent with the number of change information, wait to receive the remaining data change identifiers included in the transaction event until the number of data change identifiers is consistent with the number of change information.

[0078] In a possible embodiment, the apparatus further includes: a deduplication module, configured to perform a hash calculation based on each of the data change messages and the transaction identifier to obtain a unique identifier corresponding to each of the data change messages, and add the unique identifier to the data change identifier;

[0079] The downstream data pool deduplicates each data change identifier based on the unique identifier to obtain deduplicated data change identifiers.

[0080] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0081] An exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of the present invention.

[0082] Exemplary embodiments of the present invention further provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0083] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is configured to cause the computer to perform a method according to an embodiment of the present invention.

[0084] refer to Figure 6, a block diagram of an electronic device 600 that can serve as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0085] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0086] Multiple components within electronic device 600 are connected to I / O interface 605, including an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. Input unit 606 can be any type of device capable of inputting information into electronic device 600. Input unit 606 can receive input numeric or character information and generate key input signals related to user settings and / or function control of the electronic device. Output unit 607 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 608 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 609 allows electronic device 600 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0087] The computing unit 601 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, any of the above-described data synchronization methods may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 can be configured to perform any of the above-described data synchronization methods in any other appropriate manner (e.g., by means of firmware).

[0088] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0089] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0090] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0092] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0093] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A data synchronization method, characterized in that: The method comprises: When the received upstream MySQL data contains a transaction event, obtain the transaction identifier corresponding to the transaction event; Performing format conversion on each data change message received after the transaction event to obtain a data change identifier, adding the transaction identifier to the data change identifier, and sending the data change identifier with the transaction identifier added to the downstream data pool, so that the downstream data pool performs data changes according to each data change message; When the received upstream MySQL data includes an end event, an end identifier corresponding to the end event is generated, and the end identifier, the transaction identifier, and the number of data change information of the data change information generated between the transaction event and the end event are sent to the downstream data pool, so that after receiving the end identifier, the downstream data pool counts the number of received data change identifiers corresponding to the transaction identifier and determines whether the number of data change identifiers is consistent with the number of change information; When the number of the data change identifiers is consistent with the number of the change information, the consumption rights of the data corresponding to each data change identifier are enabled.

2. The method according to claim 1, characterized in that The data change identifier includes a data change type, which is insert, update, or delete; and counting the number of received data change identifiers corresponding to the transaction identifier and determining whether the number of data change identifiers is consistent with the number of change information, including: The total number of changes of data change identifiers of the insert, update, and delete types with the same transaction identifier is counted, and it is determined whether the total number of changes is consistent with the number of data change information.

3. The method according to claim 1, characterized in that The method further comprises: When the number of the data change identifiers is inconsistent with the number of the change information, wait for receiving the remaining data change identifiers included in the transaction event until the number of the data change identifiers is consistent with the number of the change information.

4. The method according to claim 2, characterized in that Before counting the number of received data change identifiers corresponding to the transaction identifier, the method further includes: Performing a hash calculation based on each of the data change messages and the transaction identifier to obtain a unique identifier corresponding to each of the data change messages, and adding the unique identifier to the data change identifier; The downstream data pool deduplicates each data change identifier based on the unique identifier to obtain deduplicated data change identifiers.

5. A data synchronization device, characterized in that: The device comprises: An acquisition module is used to obtain a transaction identifier corresponding to a transaction event when the received upstream MySQL data contains a transaction event, and to generate a start identifier corresponding to the transaction event and send it to the downstream data pool; an adding module, configured to convert the format of each data change message received after the transaction event to obtain a data change identifier, add the transaction identifier to the data change identifier, and send the data change identifier with the added transaction identifier to the downstream data pool, so that the downstream data pool performs data changes according to each data change message; A generation module is configured to, when the received upstream MySQL data contains an end event, generate an end identifier corresponding to the end event, and send the end identifier, the transaction identifier, and the number of data change information of the data change information generated between the transaction event and the end event to the downstream data pool, so that after receiving the end identifier, the downstream data pool counts the number of received data change identifiers corresponding to the transaction identifier and determines whether the number of data change identifiers is consistent with the number of change information; and when the number of data change identifiers is consistent with the number of change information, enable consumption rights for each data corresponding to each data change identifier.

6. The device according to claim 5, characterized in that The data change identifier includes a data change type, which is insert, update, or delete; and counting the number of received data change identifiers corresponding to the transaction identifier and determining whether the number of data change identifiers is consistent with the number of change information, including: The total number of changes of data change identifiers of the insert, update, and delete types with the same transaction identifier is counted, and it is determined whether the total number of changes is consistent with the number of data change information.

7. The device according to claim 5, characterized in that The generating module is configured to, when the number of the data change identifiers is inconsistent with the number of the change information, wait to receive the remaining data change identifiers included in the transaction event until the number of the data change identifiers is consistent with the number of the change information.

8. The device according to claim 6, characterized in that The apparatus further includes: a deduplication module configured to perform a hash calculation based on each of the data change messages and the transaction identifier to obtain a unique identifier corresponding to each of the data change messages, and add the unique identifier to the data change identifier; The downstream data pool deduplicates each data change identifier based on the unique identifier to obtain deduplicated data change identifiers.

9. An electronic device comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 4.

Images