Real-time data synchronization processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a real-time data synchronization processing method and device, equipment and a storage medium. The method comprises the steps that in the data synchronization process, a transaction step table and a user operation table are updated according to log change information, transaction identifiers and operation information corresponding to the transaction identifiers are stored in the transaction step table, and the operation information comprises at least one operation step and an operation execution sequence; obtaining two operation steps belonging to the same transaction identifier in the transaction step table, and judging whether the operation execution sequence meets a preset sequence or not according to the transaction identifier and the operation execution sequence; if not, obtaining a target data updating instruction in the user operation table according to the transaction identifier; according to the data synchronization method and device, the first data compensation instruction is generated and sent to the distributed node, the data missing problem caused by the fact that the operation execution sequence of at least one operation step is wrong in the data synchronization process of the downstream system is compensated, and the data consistency of the downstream system is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a processing method, device, equipment and storage medium for real-time data synchronization. Background Art

[0002] With the rapid development of information technology, databases play a vital role in modern enterprises and society. Data synchronization, as an important part of database management, is of great significance for ensuring data consistency and integrity. However, traditional data synchronization methods are not suitable for real-time data update scenarios.

[0003] In order to overcome the limitations of traditional data synchronization methods, real-time data synchronization technology based on database logs has emerged. This technology monitors the binlog logs generated by the database, encapsulates data changes in real time and forwards them to the target end, thereby achieving efficient data synchronization.

[0004] However, in a distributed environment, a data change includes multiple operation steps, and binlog-based log synchronization can only reflect data changes, but cannot guarantee the order of multiple operations, making it impossible to guarantee downstream data consistency and easily causing data loss. Summary of the invention

[0005] The embodiments of the present application provide a real-time data synchronization processing method, device, equipment and storage medium to compensate for the data missing problem that may occur in the downstream system and ensure the data consistency of the downstream system.

[0006] In a first aspect, an embodiment of the present application provides a real-time data synchronization processing method, including:

[0007] During the data synchronization process, the transaction step table and the user operation table are updated according to the log change information, wherein the transaction step table stores the transaction identifier and the operation information corresponding to the transaction identifier, the operation information includes at least one operation step and an operation execution order, and the user operation table stores the transaction identifier and the data update instruction; the transaction identifier is used to indicate the transaction type of the current transaction, the transaction type is to delete data, insert data or update data, the operation execution order includes the execution order of at least one operation step, and the operation step is a deletion step or an insertion step.

[0008] Obtain two operation steps belonging to the same transaction identifier in the transaction step table, and determine whether the operation execution sequence satisfies a preset sequence based on the transaction identifier and the operation execution sequence; if the operation execution sequence does not satisfy the preset sequence, obtain the target data update instruction in the user operation table based on the transaction identifier.

[0009] A first data compensation instruction is generated according to the target data update instruction, and the first data compensation instruction is sent to the distributed node, wherein the first data compensation instruction is used to compensate for data loss caused by an error in the operation execution order of at least one of the operation steps in the data synchronization process.

[0010] In a possible implementation, if the transaction type is updating data, the transaction type corresponds to two steps of deletion and insertion; obtaining two operation steps belonging to the same transaction identifier in the transaction step table, and judging whether the operation execution order meets a preset order according to the transaction identifier and the operation execution order, includes: obtaining a target step group corresponding to the transaction identifier, the target step group including the correct execution order of the two operation steps; wherein the correct execution order is to delete first and then insert.

[0011] It is determined whether the operation execution order is consistent with the correct execution order. If consistent, it is determined that the operation execution order satisfies the preset order.

[0012] In a possible implementation, the data update instruction is a SQL statement, and generating the first data compensation instruction based on the target data update instruction includes: obtaining a keyword of the target data update instruction, the keyword indicating an update type of the target data update instruction; the update type is used to indicate updating data table attributes or updating data content in a data table.

[0013] A target code template is acquired according to the keyword, and the first data compensation instruction is generated according to the target data update instruction and the target code template, wherein different update types correspond to different target code templates.

[0014] In a possible implementation, updating the transaction step table and the user operation table according to the log change information includes: parsing the binlog change information in the log change information to obtain data in a JSON format.

[0015] The transaction identifier and the operation steps in the JSON format data are extracted according to the preset step information extraction rules, and the transaction identifier, the operation steps and the operation execution order of the operation steps are stored in the transaction step table.

[0016] For the operation data change information in the log change information, the data update instruction corresponding to the transaction identifier in the operation data change information is extracted according to the preset operation data extraction rule, and the transaction identifier and the data update instruction corresponding to the transaction identifier are stored in the user operation table.

[0017] In a possible implementation, before updating the transaction step table and the user operation table according to the log change information, the method further includes: monitoring the binlog log and obtaining the binlog change information, wherein the binlog change information is used to indicate an operation step recorded in the binlog log during the data synchronization process.

[0018] The operation data change information is obtained at a preset time interval, wherein the operation data change information is used to indicate an operation instruction for executing the data synchronization process; and the log change information is obtained according to the binlog change information and the operation data change information.

[0019] In a possible implementation, the transaction step table also includes a data primary key corresponding to the transaction identifier, and the method also includes: receiving a data anomaly deletion instruction, the anomaly deletion instruction carrying the data primary key; searching for the transaction identifier in the transaction step table according to the data primary key; generating a second data compensation instruction according to operation information corresponding to the transaction identifier, and sending the second data compensation instruction to the distributed node, the second data compensation instruction being used to compensate for data anomalies caused by failure to meet theoretical data.

[0020] In a possible implementation, the user operation table also includes a row number, and the row number is used to indicate the number of data items involved when the database executes the target data update instruction. The method also includes: for abnormal data update instructions whose row number exceeds a preset threshold, obtaining the target operation user of the abnormal data update instruction; storing the abnormal data update instruction and the corresponding target operation user in the abnormal data update table, and feeding back the abnormal data update table to the target user terminal.

[0021] In a second aspect, an embodiment of the present application provides a real-time data synchronization processing device, including:

[0022] An update module is used to update a transaction step table and a user operation table according to log change information during data synchronization, wherein the transaction step table stores a transaction identifier and operation information corresponding to the transaction identifier, the operation information includes at least one operation step and an operation execution order, and the user operation table stores the transaction identifier and a data update instruction; the transaction identifier is used to indicate the transaction type of the current transaction, the transaction type is deleting data, inserting data or updating data, and the operation execution order includes the execution order of at least one operation step, and the operation step is a deletion step or an insertion step.

[0023] The judgment module is used to obtain two operation steps belonging to the same transaction identifier in the transaction step table, and judge whether the operation execution sequence meets the preset sequence according to the transaction identifier and the operation execution sequence.

[0024] The judgment module is further configured to obtain a target data update instruction in the user operation table according to the transaction identifier if the operation execution sequence does not satisfy a preset sequence.

[0025] A compensation module is used to generate a first data compensation instruction according to the target data update instruction, and send the first data compensation instruction to the distributed node, wherein the first data compensation instruction is used to compensate for data loss caused by an error in the operation execution order of at least one of the operation steps in the data synchronization process.

[0026] In a possible implementation, if the transaction type is updating data, the transaction type corresponds to two steps of deletion and insertion, and the judgment module is specifically used to: obtain a target step group corresponding to the transaction identifier, the target step group including the correct execution order of the two operation steps; wherein the correct execution order is to delete first and then insert.

[0027] It is determined whether the operation execution order is consistent with the correct execution order. If consistent, it is determined that the operation execution order satisfies the preset order.

[0028] In a possible implementation, the data update instruction is a SQL statement, and the compensation module is specifically used to: obtain a keyword of the target data update instruction, wherein the keyword indicates an update type of the target data update instruction; and the update type is used to indicate an update of a data table attribute or data content in an update data table.

[0029] A target code template is acquired according to the keyword, and the first data compensation instruction is generated according to the target data update instruction and the target code template, wherein different update types correspond to different target code templates.

[0030] In a possible implementation, the update module is specifically used to: parse the binlog change information in the log change information to obtain data in a JSON format.

[0031] The transaction identifier and the operation steps in the JSON format data are extracted according to the preset step information extraction rules, and the transaction identifier, the operation steps and the operation execution order of the operation steps are stored in the transaction step table.

[0032] For the operation data change information in the log change information, the data update instruction corresponding to the transaction identifier in the operation data change information is extracted according to the preset operation data extraction rule, and the transaction identifier and the data update instruction corresponding to the transaction identifier are stored in the user operation table.

[0033] In a possible implementation, the update module is further used to: monitor a binlog log and obtain binlog change information, wherein the binlog change information is used to indicate an operation step recorded in the binlog log during the data synchronization process.

[0034] The operation data change information is obtained at a preset time interval, wherein the operation data change information is used to indicate an operation instruction for executing the data synchronization process; and the log change information is obtained according to the binlog change information and the operation data change information.

[0035] In a possible implementation, the transaction step table also includes a data primary key corresponding to the transaction identifier, and the compensation module is further used to: receive a data anomaly deletion instruction, the anomaly deletion instruction carries the data primary key; search for the transaction identifier in the transaction step table according to the data primary key; generate a second data compensation instruction according to the operation information corresponding to the transaction identifier, and send the second data compensation instruction to the distributed node, the second data compensation instruction being used to compensate for data anomalies caused by failure to meet theoretical data.

[0036] In a possible implementation, the user operation table also includes a row number, and the row number is used to indicate the number of data items involved when the database executes the target data update instruction. The compensation module is also used to: for abnormal data update instructions whose row number exceeds a preset threshold, obtain the target operation user of the abnormal data update instruction; store the abnormal data update instruction and the corresponding target operation user in the abnormal data update table, and feed back the abnormal data update table to the target user terminal.

[0037] In a third aspect, an embodiment of the present application provides a real-time data synchronization processing device, including: a memory, a processor;

[0038] The memory stores computer-executable instructions;

[0039] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0042] The processing method, apparatus, device and storage medium for real-time data synchronization provided by the embodiments of the present application update a transaction step table and a user operation table according to log change information during data synchronization, wherein the transaction step table stores a transaction identifier and operation information corresponding to the transaction identifier, the operation information includes at least one operation step and an operation execution sequence, and the user operation table stores a transaction identifier and a data update instruction; obtain two operation steps belonging to the same transaction identifier in the transaction step table, and determine whether the operation execution sequence meets a preset sequence according to the transaction identifier and the operation execution sequence; if not, obtain a target data update instruction in the user operation table according to the transaction identifier; generate a first data compensation instruction according to the target data update instruction, and send the first data compensation instruction to the distributed node, thereby compensating for the data missing problem caused by an error in the operation execution sequence of at least one operation step in the downstream system during data synchronization, thereby ensuring data consistency of the downstream system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] Figure 1 A schematic diagram of a scenario of a real-time data synchronization processing method provided in an embodiment of the present application;

[0045] Figure 2 Schematic diagram of the process of real-time data synchronization provided by this application Figure 1 ;

[0046] Figure 3 Schematic diagram of the process of real-time data synchronization provided by this application Figure 2 ;

[0047] Figure 4 Schematic diagram of the process of real-time data synchronization provided by this application Figure 3 ;

[0048] Figure 5 Schematic diagram of the process of real-time data synchronization provided by this application Figure 4 ;

[0049] Figure 6 A schematic diagram of the structure of a real-time data synchronization processing device provided by the present application;

[0050] Figure 7A schematic diagram of the structure of the real-time data synchronization processing device provided in this application.

[0051] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0053] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, products or devices.

[0054] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0055] First, the terms involved in this application are explained:

[0056] Structured query language (SQL): is a standard programming language used to manage relational database management systems. It allows users to create, query, update and manage data in the database.

[0057] Binary log file (binlog log): records all DDL (data definition language, such as CREATE, ALTER, etc.) and DML (data manipulation language, such as INSERT, UPDATE, DELETE, etc.) statements executed in the MySQL database that update the database, but does not include SELECT or SHOW statements that do not modify any data, and is saved on disk in binary form.

[0058] Primary key: One or more fields in a database table that uniquely identify each row in the table.

[0059] Figure 1 This is a scene diagram of a processing method for real-time data synchronization provided by an embodiment of the present application. The method provided by an embodiment of the present application is applied in a distributed environment. Figure 1 As shown, for example, the distributed environment is a distributed relational database, in which a DRDS node and multiple RDS nodes are included, wherein the DRDS node is the middle layer of the distributed relational database, used to distribute the user's SQL request to different RDS nodes, and the RDS node is the underlying storage node of the DRDS, used to store actual data. The DRDS node stores sqllog, which is used to record the data change instructions entered by the user, user identification, transaction identification, etc. In addition, each RDS node corresponds to a binlog, which is used to record all operations of the data change transaction.

[0060] When a user changes a piece of data in the distributed relational database (for example, insert, update, or delete), the user will initiate an SQL request to the database. After receiving the SQL request, the DRDS node will parse the corresponding SQL statement and determine the RDS node that needs to be operated, and record the SQL request in the sqllog. When the corresponding RDS node changes the corresponding data as required, it will record all operations of the change transaction in the local binlog log.

[0061] In a distributed environment, data synchronization is necessary to ensure data consistency on all RDS nodes. In the specific implementation process, the binlog log will be transmitted to the downstream system of the RDS node that needs to be synchronized. After the downstream system of the RDS node receives the binlog log, it will synchronize data based on the change information recorded in the binlog log.

[0062] However, if the user changes the database to a data update operation, in order to maintain the uniqueness of the data on the RDS node, all operations of the data update transaction are: first find the data to be updated in the RDS node and delete it, and then insert the updated data. However, in the actual data update process, since there may be multiple sub-tables on an RDS node, and each table may contain a large amount of data, it may take a long time to traverse the RDS node to find the data to be updated, or there may be network delays and other phenomena. In the actual update transaction, the operation of inserting the updated data first and then deleting the original data to be updated may occur. At this time, the steps recorded in the binlog are insert first and then delete. Although it has no effect on the data of the RDS node database itself, in the process of data synchronization based on the binlog log of the downstream system of the RDS node, if the updated data is inserted first and then deleted, both the original data and the newly inserted data will be deleted, resulting in data missing in the downstream system database of the RDS node.

[0063] In order to solve the above problems, an embodiment of the present application provides a real-time data synchronization processing method. If it is detected that all operations of the update transaction in the binlog log are to insert the updated data first and then delete it, a data compensation instruction is generated and sent to the downstream system. This can compensate for the data missing errors that occur during the data synchronization process of the downstream system according to the binlog log, thereby ensuring the data consistency of the downstream system of the RDS node.

[0064] It should be noted that the aforementioned downstream system may be other RDS nodes or other types of systems, and this application does not impose any specific limitation here.

[0065] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0066] Figure 2 Schematic diagram of the process of real-time data synchronization provided by this application Figure 1 ,like Figure 2 As shown, the method includes:

[0067] S201. During data synchronization, the transaction step table and the user operation table are updated according to log change information.

[0068] In some embodiments, the log change information includes the change information of the binlog log and the change information of the operation data log, wherein the binlog log records the detailed operation steps of each transaction, including insert, update and delete operations, and the operation data log stores the user's change operations on the database system, including the user's specific operation instructions on the database, the timestamp of the operation, and the data involved in the operation, etc. Further, during the data synchronization process, the transaction step table and the user operation table are updated according to the log change data.

[0069] Specifically, the transaction step table stores a transaction identifier and operation information corresponding to the transaction identifier, wherein the transaction identifier is used to indicate the transaction type of the current transaction. In some embodiments, the transaction type is to delete data, insert data, or update data.

[0070] In some embodiments, the transaction identifier is also used to uniquely identify a transaction in the database system.

[0071] In some embodiments, the operation information includes at least one operation step and an operation execution order. Optionally, the operation execution order is represented by "0" and "1", where "0" represents the first operation step executed in a transaction and "1" represents the second operation step executed in a transaction.

[0072] The user operation table is used to indicate the user's operation on the database system. In some embodiments, the user operation table stores transaction identifiers and data update instructions. When the user initiates a data update instruction to the database system, the database system automatically generates a transaction. As described in the above background technology, if the transaction type is to delete data, the transaction identifier only includes the operation step of deleting data; if the transaction type is to insert data, the transaction identifier only includes the operation step of inserting data; if the transaction type is to update data, the transaction includes two operation steps: deleting data and inserting data.

[0073] It is understandable that if the transaction type corresponding to the above transaction is updating data, the operation information corresponding to the transaction may be: "0": insert data, "1": delete data; or "0": delete data, "1": insert data.

[0074] S202, obtaining two operation steps belonging to the same transaction identifier in the transaction step table, and judging whether the operation execution sequence satisfies the preset sequence according to the transaction identifier and the operation execution sequence, if so, executing S203, if not, executing S204.

[0075] S203: Confirm that the operation execution sequence is correct and do not perform data compensation.

[0076] Specifically, it can be determined according to a preset time interval whether there is a transaction identifier corresponding to two operation steps in the transaction step table. If so, the two operation steps are obtained.

[0077] Further, a preset order is obtained according to the transaction type corresponding to the two operation steps. For example, if the transaction type is to update data, the preset order is 0: "delete data", 1: "insert data". For example, if the transaction corresponding to the two operation steps belonging to the same transaction identifier is a transaction type for updating data, the two corresponding operation information may be: "0": insert data, "1": delete data; or, "0": delete data, "1": insert data. If the two operation information are in the preset order, it is confirmed that the operation execution order is correct, and the downstream system will not lose data when performing data synchronization according to the binlog log, and no data compensation will be performed.

[0078] S204: Obtain a target data update instruction from the user operation table according to the transaction identifier.

[0079] If the operation execution order does not meet the preset order, the downstream system may lose data when performing data synchronization based on the binlog log, and data compensation is required. Specifically, the transaction identifiers corresponding to the two operation steps obtain the target data update instructions in the user operation table. The target data update instructions are the data update instructions corresponding to the two operation steps. That is, after the user initiates the target data update instruction to the database system, the order of completing the two operation steps of the data update process may be to complete data insertion first and then data deletion due to factors such as the large amount of data in the database or network delays.

[0080] S205. Generate a first data compensation instruction according to the target data update instruction, and send the first data compensation instruction to the distributed node, where the first data compensation instruction is used to compensate for data loss caused by an error in the operation execution order of at least one operation step in the data synchronization process.

[0081] Further, a corresponding first data compensation instruction is generated according to the target data update instruction, and the first data compensation instruction is sent to the distributed nodes.

[0082] It should be noted that the above-mentioned distributed nodes are downstream systems that rely on binlog logs for data synchronization.

[0083] In some embodiments, the first data compensation instruction may also be sent to other downstream systems, such as a data warehouse system, but this embodiment does not limit this.

[0084] In some embodiments, the first data compensation instruction is an insert instruction, through which the deleted data is reinserted into the corresponding database or data table to compensate for the data missing problem caused by the wrong operation execution order of at least one operation step in the data synchronization process.

[0085] The processing method for real-time data synchronization provided by the embodiment of the present application is implemented by updating a transaction step table and a user operation table according to log change information during the data synchronization process, wherein the transaction step table stores a transaction identifier and operation information corresponding to the transaction identifier, the operation information includes at least one operation step and an operation execution sequence, and the user operation table stores a transaction identifier and a data update instruction; obtaining two operation steps belonging to the same transaction identifier in the transaction step table, and judging whether the operation execution sequence satisfies a preset sequence according to the transaction identifier and the operation execution sequence; if not, obtaining a target data update instruction in the user operation table according to the transaction identifier; generating a first data compensation instruction according to the target data update instruction, and sending the first data compensation instruction to the distributed node, thereby compensating for the data missing problem caused by an error in the operation execution sequence of at least one operation step in the downstream system during the data synchronization process, and ensuring the data consistency of the downstream system.

[0086] Figure 3 Schematic diagram of the process of real-time data synchronization provided by this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, the process of generating the first data compensation instruction is described in detail. The method includes:

[0087] S301. Obtain a target step group corresponding to a transaction identifier, wherein the target step group includes a correct execution order of two operation steps; wherein the correct execution order is to delete first and then insert.

[0088] S302: Determine whether the operation execution sequence is consistent with the correct execution sequence. If it is consistent, execute S303; if it is inconsistent, execute S304.

[0089] In some embodiments, if the transaction type is to update data, the transaction type corresponds to two steps: deletion and insertion. In this embodiment, a target step group corresponding to the transaction identifier is obtained, and the target step group includes the correct execution order of the two operation steps. Exemplarily, the correct execution order of updating data is to first delete the original data and then insert the updated data.

[0090] It should be noted that the embodiment of the present application only provides an example in which the transaction type is updating data and includes two operation steps, but there may be other transaction types including two or more operation steps, and this embodiment does not make specific restrictions here.

[0091] S303: Determine that the operation execution sequence satisfies the preset sequence, and do not perform data compensation.

[0092] S304, obtaining a keyword of the target data update instruction, where the keyword indicates an update type of the target data update instruction; the update type is used to indicate an update data table attribute or data content in the update data table.

[0093] If the operation execution order is consistent with the correct execution order, it is determined that the operation execution order meets the preset order and no data compensation is performed; if it is inconsistent, a corresponding first data compensation instruction needs to be generated according to the corresponding target data update instruction. Specifically, first obtain the corresponding target data update instruction, and then obtain the keywords in the target data update instruction. It can be understood that the keywords indicate the update type of the target data update instruction. For example, the keyword is "UPDATE", which is used to update the data content in the data table, or the keyword is "CREATE", which is used to update the data table attributes.

[0094] For example, the target data update instruction is UPDATE TABLE SET NAME='TOM' WHERE ID='1001', which is used to update the data with 'ID' of '1001' in 'TABLE', and update the original NAME data to 'TOM'. The keyword in the instruction is UPDATE, which is used to indicate to update the data content in the data table.

[0095] In some embodiments, a keyword library may be preset, and a character string in the target data update instruction that is consistent with a keyword in the keyword library is obtained through a matching mechanism, which is the keyword of the target data update instruction.

[0096] S305: Obtain a target code template according to the keyword, and generate a first data compensation instruction according to the target data update instruction and the target code template, wherein different update types correspond to different target code templates.

[0097] In this embodiment, the target code template is used to generate the corresponding first data compensation instruction. Different keywords, that is, different update types, correspond to different target code templates. Further, the first data compensation instruction is generated according to the target data update instruction and the target code template.

[0098] In some embodiments, the target data update instruction also includes target data, that is, the data that needs to be compensated. Each target code template contains a corresponding placeholder for inserting specific target data when generating a first data compensation instruction. After obtaining the target code template, the target data in the target data update instruction is filled into the corresponding placeholder to obtain the first data compensation instruction.

[0099] Exemplarily, the target data update instruction is UPDATE TABLE SET NAME='TOM'WHERE ID='1001', and its corresponding two operation steps are 0: first insert NAME='TOM', ID='1001' in the data table, 1: then delete the data of ID='1001'. The operation steps executed in the wrong order will cause data missing. In order to compensate for the possible data missing problem, it is necessary to insert data, that is, the target code template of the first data compensation instruction is the code template for inserting data, for example, INSERT XXX WHERE XXX, where XXX is a placeholder for filling in keywords and target data, and finally the first data compensation instruction is: INSERT INTO TABLE(ID, NAME)→VALUES(1001, TOM).

[0100] The real-time data synchronization processing method provided in the embodiment of the present application obtains the target step group corresponding to the transaction identifier, and the target step group includes the correct execution order of two operation steps; determines whether the operation execution order is consistent with the correct execution order, if consistent, no data compensation is performed, if inconsistent, obtains the keyword of the target data update instruction, and obtains the target code template according to the keyword, and then generates a first data compensation instruction according to the target data update instruction and the target code template to compensate for the execution order of the operation steps of the erroneous transaction recorded in the binlog log, thereby ensuring the data consistency of the downstream system.

[0101] Figure 4 Schematic diagram of the process of real-time data synchronization provided by this application Figure 3 ,like Figure 4 As shown, in this embodiment Figure 3 Based on the embodiment, the process of updating the transaction step table and the user operation table is described in detail. The method includes:

[0102] S401. Monitor the binlog log and obtain binlog change information, where the binlog change information is used to indicate an operation step recorded in the binlog log during the data synchronization process.

[0103] Specifically, the binlog logs in the distributed database are monitored in real time and the change information therein is obtained. For example, binlog recording can be enabled by configuring the database server, and specific tools or APIs can be used to monitor the generation of binlogs. Whenever new transactions (such as data changes) are recorded in the binlog, these events will be captured and used as binlog change information.

[0104] S402: Acquire operation data change information at a preset time interval, wherein the operation data change information is used to indicate an operation instruction for executing a data synchronization process.

[0105] Specifically, a timing scanning device may be set to obtain operation data change information at a preset time interval. Figure 1 In the distributed relational database shown, the user makes data changes in the DRDS node, and the data update instructions input by the user are stored in the operation data log of the DRDS node. The operation data change information is obtained by periodically acquiring the change information in the operation data log.

[0106] S403. Obtain log change information according to the binlog change information and the operation data change information.

[0107] Further, log change information is obtained according to the binlog change information and the operation data change information. Exemplarily, complete log change information can be formed by integrating the binlog change information and the operation data change information.

[0108] S404: Parse the binlog change information in the log change information to obtain data in JSON format.

[0109] Exemplarily, for the binlog change information in the log change information, "jsonObject=JSONUtil.parseObj(jsonStr)" may be used to parse the binlog change information into data in JSON format, wherein jsonstr is the binlog change information.

[0110] S405: extract the transaction identifier and operation steps in the data in JSON format according to the preset step information extraction rule, and store the transaction identifier, operation steps and the operation execution order of the operation steps in the transaction step table.

[0111] For example, the parsed JSON data is as follows:

[0112]

[0113] Specifically, in the above two JSON format data, "trace_id" represents the transaction ID of the current operation. The transaction ID of the above transaction is "UPDATE_1894", which indicates the update transaction number 1894. "trace_num" is the execution order of operations within the transaction. "0" represents the first operation executed in the transaction, "1" represents the second operation executed in the transaction, "type" is the operation type of the operation step, and "pk" represents the data with the primary key value of "1001".

[0114] In some embodiments, the preset step information extraction rule is to obtain "trace_id", "trace_num", "type" and "pk" in the JSON format data through a preset algorithm, and then store the transaction identifier and the operation steps and the operation execution order of the operation steps into the transaction step table. Optionally, the transaction step table is shown in Table 1:

[0115] Table 1 Example of transaction steps table

[0116] id trace_id trace_num type pk 1 1894 1 delete 1001 2 1894 0 insert 1001

[0117] S406: For the operation data change information in the log change information, extract the data update instruction corresponding to the transaction identifier in the operation data change information according to the preset operation data extraction rule, and store the transaction identifier and the data update instruction corresponding to the transaction identifier in the user operation table.

[0118] In some embodiments, the operation data change information is as follows:

[0119]

[0120] Specifically, "user" is the user who performs the operation, "data" is the data update instruction input by the user, and "row" is the number of rows in the database affected by the data update instruction.

[0121] In some embodiments, the operation data extraction rule is to extract the data update instruction corresponding to the transaction identifier in the operation data change information through a preset algorithm, and store the transaction identifier and the data update instruction corresponding to the transaction identifier in the user operation table. Optionally, the user operation table is shown in Table 2:

[0122] Table 2 User operation table example

[0123] user data type trace_id row tom sql UPDATE 1894 80

[0124] In some embodiments, there is a data administrator who performs data statistics on the distributed database system. In the statistical process, unreasonable data change operations may occur. For example, the data B deleted by user A does not meet the preset deletion conditions. If the data administrator finds such data during the statistical process, the corresponding data that should not be deleted needs to be restored. The process is described below:

[0125] Figure 5 Schematic diagram of the process of real-time data synchronization provided by this application Figure 4 ,like Figure 5 As shown, the method provided in this embodiment also includes:

[0126] S501, receiving a data abnormality deletion instruction, wherein the abnormality deletion instruction carries a data primary key.

[0127] In some embodiments, the transaction step table also includes a data primary key corresponding to the transaction identifier, and the data primary key is used to uniquely identify the data in the database, such as "pk" in the above embodiment.

[0128] In the specific implementation process, a data abnormal deletion instruction initiated by a data administrator or other user is received, and the data abnormal deletion instruction carries the data primary key of the abnormally deleted data.

[0129] S502: Search for a transaction identifier in the transaction step table according to the data primary key.

[0130] Further, the transaction identifier corresponding to the data primary key of the abnormally deleted data is searched from the transaction step table according to the data primary key.

[0131] S503: Generate a second data compensation instruction according to the operation information corresponding to the transaction identifier.

[0132] Further, according to the operation information corresponding to the transaction identifier, a corresponding second data compensation instruction is generated. Exemplarily, the primary key carried in the abnormal data deletion instruction is "1002", that is, the data with the primary key "1002" in the database is abnormally deleted, and the operation information corresponding to the primary key obtained in the transaction step table is to delete the data corresponding to the field A with the primary key "1002", then the generated second data compensation instruction is to insert the data corresponding to the field A with the primary key "1002".

[0133] S504: Send a second data compensation instruction to the distributed node, where the second data compensation instruction is used to compensate for data anomalies caused by failure to meet theoretical data.

[0134] Further, the second data compensation instruction is sent to the distributed node. It can be understood that the second data compensation instruction is used to compensate for data anomalies caused by failure to meet theoretical data, that is, to compensate for the operation of deleting data that does not meet preset deletion conditions.

[0135] In some embodiments, after sending the second data compensation instruction to the distributed node, the user operation table may be searched according to the acquired transaction identifier to obtain the user who issued the abnormal instruction, and a notification of the data abnormality may be sent to the corresponding user.

[0136] The real-time data synchronization processing method provided in this embodiment can effectively handle data anomaly problems, ensure the integrity of the database system, and improve user experience by receiving data anomaly deletion instructions, searching for transaction identifiers, generating second data compensation instructions, sending second data compensation instructions to distributed nodes, and sending data anomaly notifications to users.

[0137] In some embodiments, as shown in Table 2, the user operation table also includes the number of rows, which is used to indicate the number of data items involved when the database executes the target data update instruction. It is understandable that when the number of data items involved when the database executes the target data update instruction is too large, the database load will be too large, and the table lock will also affect the import of data during the execution process, and even some important users will be affected. In response to this problem, the embodiment of the present application also provides: for abnormal data update instructions with a number of rows exceeding a preset threshold, the target operation user of the abnormal data update instruction is obtained; then the abnormal data update instruction and the corresponding target operation user are stored in the abnormal data update table, and the abnormal data update table is fed back to the target user terminal.

[0138] In some embodiments, the target user terminal may be a data administrator. After receiving the abnormal data update table, the data administrator limits the operation time for the target user in the abnormal data update table to isolate different users from performing database operations during peak hours and avoid concentrated use of the database.

[0139] The real-time data synchronization processing method provided in this embodiment can reduce database load, avoid table locking problems, ensure operations of important users, and improve user satisfaction by isolating database operations for different users during peak hours.

[0140] Figure 6 A schematic diagram of the structure of the real-time data synchronization processing device provided by the present application, such as Figure 6 As shown, the real-time data synchronization processing device 60 provided in this embodiment includes:

[0141] Update module 601 is used to update the transaction step table and the user operation table according to the log change information during the data synchronization process, wherein the transaction step table stores the transaction identifier and the operation information corresponding to the transaction identifier, the operation information includes at least one operation step and the operation execution order, and the user operation table stores the transaction identifier and the data update instruction; the transaction identifier is used to indicate the transaction type of the current transaction, the transaction type is to delete data, insert data or update data, and the operation execution order includes the execution order of at least one operation step, and the operation step is a deletion step or an insertion step.

[0142] The judgment module 602 is used to obtain two operation steps belonging to the same transaction identifier in the transaction step table, and judge whether the operation execution sequence meets the preset sequence according to the transaction identifier and the operation execution sequence.

[0143] The judgment module 602 is further configured to obtain a target data update instruction from the user operation table according to the transaction identifier if the operation execution sequence does not satisfy a preset sequence.

[0144] The compensation module 603 is used to generate a first data compensation instruction according to the target data update instruction and send the first data compensation instruction to the distributed node. The first data compensation instruction is used to compensate for data loss caused by an error in the operation execution order of at least one operation step in the data synchronization process.

[0145] In a possible implementation, if the transaction type is updating data, the transaction type corresponds to two steps of deletion and insertion, and the judgment module 602 is specifically used to: obtain a target step group corresponding to the transaction identifier, the target step group includes the correct execution order of the two operation steps; wherein the correct execution order is to delete first and then insert.

[0146] Determine whether the operation execution sequence is consistent with the correct execution sequence. If so, determine that the operation execution sequence satisfies the preset sequence.

[0147] In a possible implementation, the data update instruction is a SQL statement, and the compensation module 603 is specifically used to: obtain the keyword of the target data update instruction, the keyword indicates the update type of the target data update instruction; the update type is used to indicate the update data table attributes or the data content in the update data table.

[0148] A target code template is acquired according to the keyword, and a first data compensation instruction is generated according to the target data update instruction and the target code template, wherein different update types correspond to different target code templates.

[0149] In a possible implementation, the update module 601 is specifically used to: parse the binlog change information in the log change information to obtain data in a JSON format.

[0150] The transaction identifier and operation steps in the JSON format data are extracted according to the preset step information extraction rules, and the transaction identifier, operation steps and the operation execution order of the operation steps are stored in the transaction step table.

[0151] For the operation data change information in the log change information, the data update instruction corresponding to the transaction identifier in the operation data change information is extracted according to the preset operation data extraction rule, and the transaction identifier and the data update instruction corresponding to the transaction identifier are stored in the user operation table.

[0152] In a possible implementation, the update module 601 is further used to: monitor the binlog log and obtain binlog change information, wherein the binlog change information is used to indicate an operation step recorded in the binlog log during the data synchronization process.

[0153] The operation data change information is obtained at a preset time interval, wherein the operation data change information is used to indicate the operation instruction for executing the data synchronization process; and the log change information is obtained according to the binlog change information and the operation data change information.

[0154] In a possible implementation, the transaction step table also includes a data primary key corresponding to the transaction identifier, and the compensation module 603 is also used to: receive a data anomaly deletion instruction, which carries the data primary key; search for the transaction identifier in the transaction step table according to the data primary key; generate a second data compensation instruction according to the operation information corresponding to the transaction identifier, and send the second data compensation instruction to the distributed node, where the second data compensation instruction is used to compensate for data anomalies caused by failure to meet theoretical data.

[0155] In a possible implementation, the user operation table also includes the number of rows, which is used to indicate the number of data items involved when the database executes the target data update instruction. The compensation module 603 is also used to: for abnormal data update instructions whose number of rows exceeds a preset threshold, obtain the target operation user of the abnormal data update instruction; store the abnormal data update instruction and the corresponding target operation user in the abnormal data update table, and feed back the abnormal data update table to the target user terminal.

[0156] The real-time data synchronization processing device 60 provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0157] Figure 7 This is a schematic diagram of the structure of the real-time data synchronization processing device provided by this application. Figure 7As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0158] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0159] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0160] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0161] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0162] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0163] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0164] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0165] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0166] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0167] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0170] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0171] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0172] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A real-time data synchronization processing method, characterized in that: include: During the data synchronization process, the transaction step table and the user operation table are updated according to the log change information, wherein the transaction step table stores a transaction identifier and operation information corresponding to the transaction identifier, the operation information includes at least one operation step and an operation execution order, and the user operation table stores the transaction identifier and a data update instruction; the transaction identifier is used to indicate the transaction type of the current transaction, the transaction type is to delete data, insert data or update data, the operation execution order includes the execution order of at least one operation step, and the operation step is a deletion step or an insertion step; Obtain two operation steps belonging to the same transaction identifier in the transaction step table, and determine whether the operation execution sequence satisfies a preset sequence according to the transaction identifier and the operation execution sequence; If not, obtaining the target data update instruction in the user operation table according to the transaction identifier; A first data compensation instruction is generated according to the target data update instruction, and the first data compensation instruction is sent to the distributed node, wherein the first data compensation instruction is used to compensate for data loss caused by an error in the operation execution order of at least one of the operation steps in the data synchronization process.

2. The method according to claim 1, characterized in that If the transaction type is to update data, the transaction type corresponds to two steps of deletion and insertion; obtaining two operation steps belonging to the same transaction identifier in the transaction step table, and judging whether the operation execution order meets the preset order according to the transaction identifier and the operation execution order, includes: Obtaining a target step group corresponding to the transaction identifier, wherein the target step group includes a correct execution order of the two operation steps; wherein the correct execution order is to delete first and then insert; It is determined whether the operation execution order is consistent with the correct execution order. If consistent, it is determined that the operation execution order satisfies the preset order.

3. The method according to claim 2, characterized in that The data update instruction is a SQL statement, and the generating of the first data compensation instruction according to the target data update instruction includes: Acquire a keyword of the target data update instruction, wherein the keyword indicates an update type of the target data update instruction; the update type is used to indicate an update data table attribute or data content in an update data table; A target code template is acquired according to the keyword, and the first data compensation instruction is generated according to the target data update instruction and the target code template, wherein different update types correspond to different target code templates.

4. The method according to claim 1, characterized in that: The updating of the transaction step table and the user operation table according to the log change information includes: For the binlog change information in the log change information, parse the binlog change information to obtain data in JSON format; Extract the transaction identifier and the operation steps in the data in the JSON format according to a preset step information extraction rule, and store the transaction identifier, the operation steps, and the operation execution order of the operation steps in the transaction step table; For the operation data change information in the log change information, the data update instruction corresponding to the transaction identifier in the operation data change information is extracted according to the preset operation data extraction rule, and the transaction identifier and the data update instruction corresponding to the transaction identifier are stored in the user operation table.

5. The method according to claim 4, characterized in that Before updating the transaction step table and the user operation table according to the log change information, the method further includes: Monitor the binlog log and obtain binlog change information, wherein the binlog change information is used to indicate an operation step recorded in the binlog log during data synchronization; Acquiring operation data change information at a preset time interval, wherein the operation data change information is used to indicate an operation instruction for executing the data synchronization process; Obtain log change information according to the binlog change information and the operation data change information.

6. The method according to claim 1, characterized in that The transaction step table also includes a data primary key corresponding to the transaction identifier, and the method further includes: Receive a data abnormality deletion instruction, wherein the abnormality deletion instruction carries a data primary key; Searching for a transaction identifier in the transaction step table according to the data primary key; A second data compensation instruction is generated according to the operation information corresponding to the transaction identifier, and the second data compensation instruction is sent to the distributed node, where the second data compensation instruction is used to compensate for data anomalies caused by failure to meet theoretical data.

7. The method according to claim 1, characterized in that The user operation table also includes a row number, which is used to indicate the number of data items involved when the database executes the target data update instruction. The method also includes: For the abnormal data update instruction whose number of rows exceeds a preset threshold, obtaining a target operation user of the abnormal data update instruction; The abnormal data update instruction and the corresponding target operating user are stored in an abnormal data update table, and the abnormal data update table is fed back to the target user terminal.

8. A real-time data synchronization processing device, characterized in that: include: An update module is used to update a transaction step table and a user operation table according to log change information during data synchronization, wherein the transaction step table stores a transaction identifier and operation information corresponding to the transaction identifier, the operation information includes at least one operation step and an operation execution sequence, and the user operation table stores the transaction identifier and a data update instruction; the transaction identifier is used to indicate the transaction type of the current transaction, the transaction type is to delete data, insert data or update data, the operation execution sequence includes the execution sequence of at least one operation step, and the operation step is a deletion step or an insertion step; A judgment module, used for obtaining two operation steps belonging to the same transaction identifier in the transaction step table, and judging whether the operation execution sequence satisfies a preset sequence according to the transaction identifier and the operation execution sequence; The judgment module is further configured to obtain a target data update instruction in the user operation table according to the transaction identifier if the operation execution order does not satisfy a preset order; A compensation module is used to generate a first data compensation instruction according to the target data update instruction, and send the first data compensation instruction to the distributed node, wherein the first data compensation instruction is used to compensate for data loss caused by an error in the operation execution order of at least one of the operation steps in the data synchronization process.

9. A real-time data synchronization processing device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images