Bidirectional synchronization method and system for data system and electronic equipment

Abstract

The invention provides a data system bidirectional synchronization method and system and electronic equipment, and the method comprises the steps: analyzing a first log file to obtain a first analysis result; identifying a predefined identifier, wherein the predefined identifier is used for determining an event distribution rule; if the predefined identifier is not identified in the first analysis result, synchronizing the first log file to a second data system; analyzing the second log file to obtain a second analysis result; if the predefined identifier is not identified in the second analysis result, filtering out the first log file from the second log file to obtain the second log file; and sending the second log file to the second data system. The method does not additionally increase the data volume of the log file, does not depend on the version of the data system, can be expanded to other data supporting transactions due to the fact that the method does not depend on the characteristics of the data system, and can improve the flexibility during data two-way synchronization.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a data system two-way synchronization method, system and electronic device. Background Art

[0002] Bidirectional synchronization of database systems refers to the process of synchronizing data between two data systems. Bidirectional synchronization is used in distributed systems to achieve redundant data storage and failover. Bidirectional synchronization can identify and capture changes made to data in a data system, such as data insertion, update, and deletion, and then record the changes in the order in which they occur, and transmit them to downstream processes or systems in real time in the form of logs.

[0003] These synchronization technologies are difficult to meet the needs of bidirectional consistency of multiple data systems, especially in scenarios across data centers and across the Internet. Data synchronization can be performed through cross-Internet secure data synchronization, which can safely transmit and synchronize data between different network environments. However, during the data synchronization process of cross-Internet secure data synchronization, synchronization in both directions may cause data to jump back and forth between the two systems and fail to reach a stable state.

[0004] To solve the above problems, you can set an identifier on the data system when writing to other data systems, but you need to record the complete structured query language in the binary log; to reduce the generation of additional binary log data, you can also use the global transaction ID of the data system, but the data system needs to support the GTID (Global Transaction ID) function, that is, it only supports the use of a specific version of the data system, resulting in low flexibility when synchronizing data in both directions. Summary of the invention

[0005] The present application provides a data system bidirectional synchronization method, system and electronic device to solve the problem of low flexibility during bidirectional data synchronization.

[0006] In a first aspect, the present application provides a data system bidirectional synchronization method, comprising:

[0007] Parsing a first log file to obtain a first parsing result, where the first log file is a binary log file generated by a first data system;

[0008] Identifying a predefined identifier, wherein the predefined identifier is used to determine an event distribution rule;

[0009] If no predefined identifier is identified in the first parsing result, synchronizing the first log file to a second data system;

[0010] Parsing a second log file to obtain a second parsing result, where the second log file is a binary log file generated by the second data system;

[0011] If the predefined identifier is not identified in the second parsing result, filtering the first log file from the second log file to obtain a second log file;

[0012] The second log file is sent to a second data system.

[0013] In some feasible embodiments, parsing the first log file includes:

[0014] Obtaining the first log file, where the first log file includes operation events on the master node on the first data system, where the operation events include a transaction start instruction, an operation event, and a transaction end instruction;

[0015] In response to a request sent by a slave node to import a first log file, sending the first log file to the slave node, the request including a name and a location of the first log file;

[0016] The first log file is parsed to extract operational events.

[0017] In some feasible embodiments, parsing the first log file further includes:

[0018] Preset filtering rules, where the filtering rules are used to specify the type of operation events;

[0019] Operation events of non-specified types are filtered out according to the filtering rule to obtain a first log file.

[0020] In some feasible embodiments, in response to the request for importing the first log file sent by the slave node, sending the first log file to the slave node includes:

[0021] Responding to a connection request sent by a slave node to a master node to establish a communication connection between the master node and a slave node, the slave node being a simulated node;

[0022] In response to a request for importing a first log file sent by a slave node, the master node is controlled to read the first log file according to the position, and the read operation events are sent to the slave node, where the first log file includes multiple operation events.

[0023] In some feasible embodiments, before parsing the second log file, the process includes:

[0024] Establishing a first transaction, wherein the first transaction includes a record table;

[0025] In response to the second data system executing the operation event, an identification record is inserted into the record table, and the first transaction is saved to the second data system, the identification record includes a predefined identification, and the predefined identification is used to mark a synchronization operation.

[0026] In some feasible embodiments, parsing the second log file includes:

[0027] Acquire the second log file, where the second log file includes data operations on the master node on the second data system, where the data operations include transaction start instructions, operation events, and transaction end instructions;

[0028] In response to a request sent by a slave node to import a second log file, sending the second log file to the slave node;

[0029] The second log file is parsed to extract operational events.

[0030] In some feasible embodiments, the number of the second data systems is multiple;

[0031] Before parsing the second log file, the method includes:

[0032] Establishing selection criteria, the selection criteria including request parameters, user information, and time information;

[0033] A selection logic is established according to the selection criteria, and the selection logic is used to select the second data system.

[0034] In some feasible embodiments, the predefined identifier includes a synchronization ID, a timestamp, a synchronization status identifier, a first data system identifier, a second data system identifier, and an identifier validity period.

[0035] In a second aspect, the present application provides a data system bidirectional synchronization system, comprising:

[0036] A first parsing and distributing unit is configured to parse a first log file to obtain a first parsing result, wherein the first log file is a binary log file generated by a first data system; identify a predefined identifier, wherein the predefined identifier is used to determine an event distribution rule; and if the predefined identifier is not identified in the first parsing result, synchronize the first log file to a second data system;

[0037] a second parsing and distributing unit, configured to parse a second log file to obtain a second parsing result, wherein the second log file is a binary log file generated by the second data system; if the predefined identifier is not identified in the second parsing result, filtering out the first log file from the second log file to obtain a second log file;

[0038] A data synchronization unit is used to send the second log file to a second data system.

[0039] In a third aspect, the present application provides an electronic device, including:

[0040] Memory for storing computer programs;

[0041] A processor is used to execute the computer program, wherein the computer program is specifically used to implement the data system bidirectional synchronization method.

[0042] As can be seen from the above technical solutions, the present application provides a method, system and electronic device for bidirectional synchronization of a data system, the method comprising: parsing a first log file to obtain a first parsing result, the first log file being a binary log file generated by the first data system; then identifying a predefined identifier, the predefined identifier being used to determine an event distribution rule; if the predefined identifier is not identified in the first parsing result, synchronizing the first log file to a second data system; then parsing a second log file to obtain a second parsing result, the second log file being a binary log file generated by the second data system; if the predefined identifier is not identified in the second parsing result, filtering out the first log file from the second log file to obtain a second log file; and sending the second log file to the second data system. The method does not increase the amount of log file data, does not depend on the version of the data system, and because it does not depend on the characteristics of the data system itself, the method can be extended to other data that supports transactions, and can improve the flexibility of bidirectional data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram of a flow chart of a bidirectional synchronization method for a data system provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a synchronization method provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a bidirectional synchronization system of a data system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0048] Data synchronization is a method of maintaining data consistency between two or more database systems in database system management. Data synchronization can be applied to different scenarios, such as master-slave replication, read-write separation, cross-data center data replication, etc. In order to keep data consistent between two or more databases, especially when high availability and data redundancy are required, data synchronization is performed through database bidirectional synchronization.

[0049] Bidirectional synchronization is used in distributed systems to achieve redundant data storage and failover. These synchronization technologies are difficult to meet the needs of bidirectional consistency of multiple data systems, especially in cross-computer room and cross-Internet scenarios.

[0050] In some embodiments, two-way synchronization is performed by cross-Internet secure data synchronization. Cross-Internet secure data synchronization is to achieve real-time or scheduled data updates in a secure manner between different network environments, such as servers and cloud platforms in different geographical locations, to ensure data consistency and integrity. However, the cross-Internet secure data synchronization method may cause the problem of anti-looping of two-way synchronization.

[0051] When performing bidirectional synchronization, bidirectional synchronization loop refers to the bidirectional flow of data between two or more systems. If anti-loop measures are not taken, data duplication, data overwriting, and system performance degradation are prone to occur. Among them, data duplication means that the same data is repeatedly copied between two systems, resulting in data redundancy and storage space waste. Data overwriting means that the later synchronized data overwrites the earlier synchronized data, resulting in data loss or inconsistency. System performance degradation means that frequent data copying and comparison operations will increase the system burden and affect performance.

[0052] To solve the problem of two-way synchronization loops, in some embodiments, when synchronizing data, an additional field or metadata is added, and the additional field or metadata is used to mark the source of the data change. For example, when synchronizing data, an identifier is added to the structured query language (SQL statement) to identify which system the change of this record comes from. When receiving a change, the synchronization tool will check this identifier. If it finds that the change comes from itself, it will ignore the change, thereby avoiding the problem of circular synchronization.

[0053] In other embodiments, the GTID mode is enabled in the database system so that each transaction has a unique GTID. The GTID set of the current system is recorded before synchronization begins. When receiving changes from other systems, the recorded GTID set is compared. If the GTID contained in the change has been processed by the local system, the change is ignored.

[0054] In the method of using additional fields or metadata, the log output method needs to be configured, and a large number of SQL statements need to be recorded in the log file, resulting in log redundancy. In the method of using the GTID mode, no additional SQL statements are generated, but there are fixed requirements for the database version, resulting in low flexibility in bidirectional data synchronization.

[0055] To solve the problem of low flexibility in two-way data synchronization, such as Figure 1 As shown, some embodiments of the present application provide a data system bidirectional synchronization method, including:

[0056] S100: Parse a first log file to obtain a first parsing result.

[0057] The first log file is a binary log file (binlog) generated by the first data system (first database), and the first log file includes the change operations occurring on the first database. To parse the first log file, the first database is first determined, and then the binary log file is obtained from the first database. In some embodiments, the first log file is obtained, and in response to a request to import the first log file sent by the slave node, the first log file is sent to the slave node, and then the first log file is parsed to extract the operation event.

[0058] The slave node is a simulated slave node, and the master node is the source of data replication, which is used to receive write operations from the client and record these operations in the binary log file. The master node is used to record data changes, send change data, and maintain the Binlog file. Whenever there is a data change, the master node will record these changes in the binary log file. The master node will send the change data in the Binlog file to the slave node so that the slave node can apply these changes.

[0059] In a database system, the master-slave replication mechanism enables data to be replicated from a master node to one or more other slave nodes. The slave node receives the change data sent by the master node and applies these changes locally to maintain consistency with the master node data. The slave node is used to request Binlog data, parse Binlog data, apply change data, and monitor the status of the master node.

[0060] The slave node requests Binlog data from the master node and receives the change data sent by the master node. The slave node parses the received Binlog data to extract specific operation events. The slave node applies these changes in the local database based on the parsed operation events to ensure data consistency. The slave node also monitors the status of the master node to ensure that the change data can be obtained and applied in a timely manner.

[0061] In some embodiments, in response to a connection request sent by a slave node to a master node, a communication connection is established between the master node and the slave node, and the slave node is a simulation node; in response to a request to import a first log file sent by the slave node, the master node is controlled to read the first log file according to the location, and the read operation events are sent to the slave node, wherein the first log file includes multiple operation events.

[0062] In the replication environment of the database system, in this embodiment, the control parsing distribution component imitates the behavior of the slave node and requests binlog data from the master node. In some embodiments, the slave node will actively connect to the master node and request binlog data for data synchronization. In this embodiment, the parsing distribution component simulates the behavior of the slave node so that it can obtain binlog data from the master node.

[0063] Specifically, the parsing and distribution component initiates a connection with the Master node, which is similar to the connection between the Slave node and the Master node. It then sends a DUMP BINLOG (binary log storage) command to the Master node, requesting a specific binlog file and its position (POS). This command is similar to the command issued by the Slave node when requesting binlog data. After receiving the request, the Master node will send the corresponding binlog data to the requester (i.e., the parsing and distribution component) based on the binlog position information in the request. After receiving the binlog data, the parsing and distribution component parses it and decides whether to distribute it based on the parsing result.

[0064] It is understandable that the sending process is continuous, that is, the Master node will continue to read new BinlogEvents and send them to the parsing distribution component until the current Binlog file ends. During this process, the parsing distribution component sends a Dump Binlog request to the Master node. The Master node searches and reads the Binlog data according to the Binlog file name and location in the request, and continuously sends the read Binlog Events to the binlog parsing distribution component. This ensures that the parsing distribution component can receive the latest data changes and apply these changes to maintain data consistency.

[0065] Exemplarily, the Binlog recording function is enabled on the master node to record data change operations. The parsing and distribution component sends a Dump Binlog request to the master node, requesting a specific Binlog file and its position (POS). The master node sends the corresponding Binlog event to the slave node according to the request. The slave node parses the received Binlog event and extracts the specific operation event. The slave node applies the parsed operation event and performs the corresponding change operation in the local database.

[0066] The slave node sends a request to the master node to import the first log file. The request includes the name and location of the first log file, which is used to indicate the location from which to start reading the log file. After receiving the request from the slave node, the master node reads and sends the log file to the slave node according to the file name and location information in the request. After receiving the first log file, the slave node parses these files to extract the operation events therein. The result of the parsing is the first parsing result, which includes all valid operation events. The operation events include transaction start instructions, operation events, and transaction end instructions.

[0067] The binlog file generated by the first data system is read and parsed to obtain a first parsing result, and a filtering rule can be applied to filter out qualified operation events. In some embodiments, a filtering rule is preset, wherein the filtering rule is used to specify the type of operation event, and then the operation events of non-specified types are filtered out according to the filtering rule to obtain a first log file.

[0068] When parsing the first log file, you can add filtering rules for binlog events to synchronize only events of a specified type. For example, you can set filtering rules to synchronize only INSERT and UPDATE events and ignore DELETE events.

[0069] In some embodiments, files can also be filtered by filtering rules such as time window-based filtering and user role-based filtering. Among them, filtering based on time windows can determine whether to synchronize a certain operation event based on certain time periods. For example, it can be set to synchronize certain types of operations during working hours, and synchronize other types of operations during non-working hours. Important operations can be synchronized first during working hours to reduce the impact on system resources. User role-based filtering determines whether to synchronize a certain operation event based on the user's permissions or roles. For example, it can be set that only operations of administrator roles are allowed to be synchronized.

[0070] You can also combine time window-based filtering and user role-based filtering to filter operation events, ensuring that only user operations with corresponding permissions can be synchronized, thereby improving data security.

[0071] S200: Identify a predefined identifier.

[0072] In some embodiments, the predefined identifier includes a synchronization ID, a timestamp, a synchronization status identifier, a first data system identifier, a second data system identifier, and an identifier validity period.

[0073] The synchronization ID is used to uniquely identify a synchronization operation and can be an integer or a UUID. The timestamp is used to record the timestamp of the synchronization operation and is used to track the order of synchronization. The synchronization status identifier may include a synchronization identifier and an error identifier. The synchronization identifier, for example, "synchronized" or "unsynchronized", can be used to identify the processing status of the event. The error identifier is used to record errors in the synchronization process to facilitate tracking and debugging. The first data system identifier and the second data system identifier are used to mark which data system the synchronization operation comes from and to mark the target database of the synchronization operation. The identifier validity period is used to set the validity period of the predefined identifier, clean up expired events in a timely manner, and prevent unnecessary resources from being occupied.

[0074] Predefined identifiers are used to determine event distribution rules, prevent circular synchronization, and help track the progress and order of data synchronization. Specifically, how to distribute synchronization events can be determined based on the information contained in the predefined identifier. For example, it can be determined based on the synchronization ID or timestamp whether an event has been synchronized, or it can be determined based on the operation type whether an event needs to be synchronized. In some embodiments, other information fields can also be added to the predefined identifier, such as user name, operating environment, etc.

[0075] Predefined identifiers can also be specific SQL statements, specific table names or modes, custom event types, specific metadata, and specific event IDs. Specific SQL statements are certain comment lines or specific SQL keywords that can be used to identify a certain operation as special, such as "INSERT INTO special_table...". Specific table names or modes are certain tables or modes that are marked as objects for special processing. Custom event types are special events created through user-defined functions or stored procedures.

[0076] S300: If the predefined identifier is not identified in the first parsing result, synchronize the first log file to the second data system.

[0077] The first parsing result is an operation event. Check whether there is a predefined identifier in the first parsing result. If there is no predefined identifier, synchronize the corresponding log file to the second data system, which can ensure the correctness and order of synchronization and avoid circular synchronization. If the predefined identifier is identified in the first parsing result, the first parsing result is discarded. When the first parsing result is received again, the predefined identifier is re-identified until the predefined identifier cannot be identified in the first parsing result, and then the log file corresponding to the first parsing result is synchronized to the second data system.

[0078] In some cases, there may be certain operations that should not be replicated to slave nodes because they may have been processed by other mechanisms or are not applicable on slave nodes. By using special identifiers to filter these events, you can ensure that the data on the slave node is consistent with the master node while avoiding unnecessary operations. In this scenario, predefined identifiers can be used to filter events that do not need to be distributed, thereby ensuring the accuracy and effectiveness of data synchronization and avoiding unnecessary operations that affect the data consistency and performance of the slave node.

[0079] S400: Parse the second log file to obtain a second parsing result.

[0080] The second log file is a binary log file generated by the second data system (second database). Similar to the first log file, the second log file contains operation events extracted from the binlog. The second parsing result is the result obtained after parsing the second log file, and the second parsing result contains operation events extracted from the binlog.

[0081] The parsing and acquisition process of the second log file is the same as that of the first log file. In some embodiments, the second log file is acquired, and the second log file includes data operations on the master node on the second data system, and the data operations include transaction start instructions, operation events, and transaction end instructions; in response to a request sent by the slave node to import the second log file, the second log file is sent to the slave node, and the second log file is parsed to extract the operation events.

[0082] It is understandable that the process of obtaining the first log file is applicable to obtaining the second log file, and will not be described in detail here.

[0083] There are multiple second data systems, that is, multiple slave databases or synchronization targets. Each system may generate files containing data operation logs, which need to be parsed for data synchronization or analysis. Multiple second data systems may be distributed in different geographical locations, process different data types, or serve different business needs. Before parsing the second log file, you need to select one of them for synchronization according to the standard.

[0084] In some embodiments, a selection criterion is established, a selection logic is established according to the selection criterion, and the selection logic is used to select the second data system.

[0085] The selection criteria are used to screen and select the criteria or conditions for a specific second data system, and may be based on a variety of factors, such as request parameters (such as synchronization type, synchronization scope, etc.), user information (such as user permissions, user organization, etc.), and time information (such as data generation time, synchronization time window, etc.).

[0086] Exemplarily, first, based on business requirements, system architecture and synchronization strategies, establish standards for selecting a second data system. These standards can reflect the requirements and constraints of data synchronization, and then formulate selection logic based on the selection criteria. The selection logic guides the system to choose between multiple second data systems to ensure that only systems that meet the requirements are selected for parsing and synchronization of log files. In a scenario where the second log file needs to be parsed, select qualified systems from multiple second data systems according to the selection logic. This selection process may involve evaluation, sorting and screening of multiple systems. After determining the second data system, start parsing the log files on the system and extract the operation events therein for subsequent data synchronization or analysis. It is understandable that the selection criteria can be dynamically adjusted according to actual conditions and feedback.

[0087] To ensure the consistency and reliability of the synchronization operation, before parsing the second log file, in some embodiments, a first transaction is established, wherein the first transaction includes a record table; in response to the second data system executing an operation event, an identification record is inserted into the record table, and the first transaction is saved to the second data system, wherein the identification record includes a predefined identification, and the predefined identification is used to mark the synchronization operation.

[0088] A new transaction is created in the second data system (from the database) to record the relevant information of the synchronization operation. The record table is a table included in the transaction, which is used to record the identification information of the synchronization operation. The record table can be used to track which operations have been synchronized and the status of the synchronization. The record inserted in the record table contains a predefined identifier to mark the synchronization operation. This record can help the system identify which operations have been synchronized.

[0089] By recording synchronization operations in the second data system, it is possible to track which operations have been synchronized, thereby avoiding duplicate synchronization and circular synchronization. By managing synchronization operations through transactions, data consistency and integrity can be ensured. By persisting identification records in the record table, the reliability of the synchronization process can be improved, and the synchronization state can be restored even in the event of a system crash or restart.

[0090] During the data synchronization process, the predefined table is used to record metadata, which is used to store tag information related to data synchronization. It does not contain actual business data and is used to record information about synchronization operations. This table contains some fields, such as identifier, synchronization timestamp, synchronization flag, etc.

[0091] Saving the first transaction to the second data system is to apply all completed operations in the transaction to the database, making these operations permanent changes. That is, before saving the transaction, all changes in the transaction are temporary, and only after saving the transaction will these changes be officially saved to the database.

[0092] For example, before applying the binlog event, start a new transaction, apply the received operation event, and complete the data replication. For example, if the binlog event contains an insert operation, perform the corresponding insert operation in the target database. In the transaction, insert a record into the predefined table. This record contains a special identifier to mark this synchronization operation. After completing all operations, commit the transaction.

[0093] S500: If the predefined identifier is not identified in the second parsing result, the first log file is filtered out from the second log file to obtain a second log file.

[0094] During the bidirectional data synchronization process, the predefined identifiers in the first parsing result and the second parsing result are the same, used to identify the synchronization operation, and need to be consistent throughout the synchronization process. The same predefined identifiers can ensure that the data that has been synchronized will not be synchronized again. The same predefined identifiers help ensure the consistency of data during the bidirectional synchronization process. The same predefined identifiers facilitate tracking and management of synchronization events throughout the synchronization process.

[0095] To avoid repeated synchronization and cyclic synchronization, a search is performed in the second parsing result to determine whether there is a predefined identifier. If there is no predefined identifier, the first log file is filtered out from the second log file.

[0096] S600: Send the second log file to the second data system.

[0097] Prepare the generated second log file, which may include required meta information, such as version number, timestamp, etc. Send the prepared log data to the target second data system through a network interface, API, message queue, etc. For the sent data, you can wait for a confirmation response from the target system to ensure that the data has been successfully received and processed.

[0098] It is understandable that filtering rules may also be set during the process of S300 - S400 . By applying the filtering rules in each step, unnecessary data synchronization may be avoided, thereby improving the efficiency and accuracy of synchronization.

[0099] like Figure 2 As shown, for example, database A records the binlog file of all its change operations. The data synchronization component reads the binlog file of database A and sends it to the parsing and distribution component. The parsing and distribution component is responsible for parsing the binlog file and obtaining operation events from it. If the parsed operation event does not contain a predefined identifier, the operation event is distributed to the binlog file corresponding to database B, the binlog file of database B is read, the identifier is added, the operation event is copied, and a transaction is established to read the operation event. The data synchronization component uses these operation events to update database B and complete data synchronization.

[0100] By parsing the binlog file of database A, all change operations can be obtained and applied to database B to keep the two synchronized. At the same time, adding predefined identifiers in database B can avoid repeated synchronization of operations that have been processed in the subsequent synchronization process, thereby improving the efficiency and accuracy of data synchronization.

[0101] Based on the above data system two-way synchronization method, such as Figure 3 As shown, some embodiments of the present application also provide a data system bidirectional synchronization system, including:

[0102] a first parsing and distributing unit, configured to parse a first log file to obtain a first parsing result, wherein the first log file is a binary log file generated by a first data system; if a predefined identifier is identified in the first parsing result, synchronizing the first log file with a predefined mark and the predefined identifier to a second data system, wherein the predefined identifier is used to determine an event distribution rule;

[0103] a second parsing and distributing unit, configured to parse a second log file to obtain a second parsing result, wherein the second log file is a binary log file generated by the second data system; if a predefined identifier is identified in the second parsing result, the first log file and the predefined identifier are filtered out from the second log file according to the predefined identifier to obtain a second log file;

[0104] A data synchronization unit is used to send the second log file to a second data system.

[0105] The first parsing and distributing unit and the second parsing and distributing unit are parsing and distributing components, and are also used to simulate the behavior of the SLAVE node to obtain the data of the master node.

[0106] The present application provides an electronic device, including:

[0107] Memory for storing computer programs;

[0108] A processor is used to execute the computer program, wherein the computer program is specifically used to implement the data system bidirectional synchronization method.

[0109] The electronic device may be a server or terminal device that other electronic devices can log in and access. The electronic device may include at least one memory and at least one processor. The processor may execute multiple computer instructions stored in the memory to implement the steps of the information processing method proposed in the above-mentioned embodiment of the present application. The implementation process can refer to the description of the corresponding method embodiment above, and this embodiment will not be described in detail here.

[0110] In the embodiment of the present application, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device. The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, etc. The present application does not limit the structure and model of the above-mentioned memory and processor, and can be flexibly adjusted according to actual needs.

[0111] As can be seen from the above technical solutions, the present application provides a method, system and electronic device for bidirectional synchronization of a data system, the method comprising: parsing a first log file to obtain a first parsing result, the first log file being a binary log file generated by the first data system; then identifying a predefined identifier, the predefined identifier being used to determine an event distribution rule; if the predefined identifier is not identified in the first parsing result, synchronizing the first log file to a second data system; then parsing a second log file to obtain a second parsing result, the second log file being a binary log file generated by the second data system; if the predefined identifier is not identified in the second parsing result, filtering out the first log file from the second log file to obtain a second log file; and sending the second log file to the second data system. The method does not increase the amount of log file data, does not depend on the version of the data system, and because it does not depend on the characteristics of the data system itself, the method can be extended to other data that supports transactions, and can improve the flexibility of bidirectional data synchronization.

[0112] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A data system bidirectional synchronization method, characterized in that: include: Parsing a first log file to obtain a first parsing result, where the first log file is a binary log file generated by a first data system; Identifying a predefined identifier, wherein the predefined identifier is used to determine an event distribution rule; If no predefined identifier is identified in the first parsing result, synchronizing the first log file to a second data system; Parsing a second log file to obtain a second parsing result, where the second log file is a binary log file generated by the second data system; If the predefined identifier is not identified in the second parsing result, filtering the first log file from the second log file to obtain a second log file; The second log file is sent to a second data system.

2. The data system bidirectional synchronization method according to claim 1, characterized in that: The parsing of the first log file includes: Obtaining the first log file, where the first log file includes operation events on the master node on the first data system, where the operation events include a transaction start instruction, an operation event, and a transaction end instruction; In response to a request sent by a slave node to import a first log file, sending the first log file to the slave node, the request including a name and a location of the first log file; The first log file is parsed to extract operational events.

3. The data system bidirectional synchronization method according to claim 2, characterized in that: The parsing of the first log file further includes: Preset filtering rules, where the filtering rules are used to specify the type of operation events; Operation events of non-specified types are filtered out according to the filtering rule to obtain a first log file.

4. The data system bidirectional synchronization method according to claim 3, characterized in that: The step of sending the first log file to the slave node in response to the request for importing the first log file sent by the slave node comprises: Responding to a connection request sent by a slave node to a master node to establish a communication connection between the master node and a slave node, the slave node being a simulated node; In response to a request for importing a first log file sent by a slave node, the master node is controlled to read the first log file according to the position, and the read operation events are sent to the slave node, where the first log file includes multiple operation events.

5. The data system bidirectional synchronization method according to claim 4, characterized in that: Before parsing the second log file, the method includes: Establishing a first transaction, wherein the first transaction includes a record table; In response to the second data system executing the operation event, an identification record is inserted into the record table, and the first transaction is saved to the second data system, the identification record includes a predefined identification, and the predefined identification is used to mark a synchronization operation.

6. The data system bidirectional synchronization method according to claim 1, characterized in that: The parsing of the second log file comprises: Acquire the second log file, where the second log file includes data operations on the master node on the second data system, where the data operations include transaction start instructions, operation events, and transaction end instructions; In response to a request sent by a slave node to import a second log file, sending the second log file to the slave node; The second log file is parsed to extract operational events.

7. The data system bidirectional synchronization method according to claim 1, characterized in that: The number of the second data systems is multiple; Before parsing the second log file, the method further includes: Establishing selection criteria, the selection criteria including request parameters, user information, and time information; A selection logic is established according to the selection criteria, and the selection logic is used to select the second data system.

8. The data system bidirectional synchronization method according to claim 1, characterized in that: The predefined identifier includes a synchronization ID, a timestamp, a synchronization state identifier, a first data system identifier, a second data system identifier, and an identifier validity period.

9. A data system bidirectional synchronization system, characterized in that: include: A first parsing and distributing unit, configured to parse a first log file to obtain a first parsing result, wherein the first log file is a binary log file generated by a first data system; identifying a predefined identifier, the predefined identifier being used to determine an event distribution rule; if the predefined identifier is not identified in the first parsing result, synchronizing the first log file to a second data system; A second parsing and distributing unit, configured to parse a second log file to obtain a second parsing result, wherein the second log file is a binary log file generated by the second data system; If the predefined identifier is not identified in the second parsing result, filtering the first log file from the second log file to obtain a second log file; A data synchronization unit is used to send the second log file to a second data system.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, wherein the computer program is specifically used to implement the data system bidirectional synchronization method according to any one of claims 1 to 8.

Images