Data consistency synchronous transmission method, equipment and medium

Abstract

The invention discloses a data consistency synchronous transmission method, data consistency synchronous transmission equipment and a medium, belongs to the technical field of data transmission consistency processing, and aims to solve the problems that synchronous updating of data cannot be guaranteed under the condition that system performance is not reduced and data protection measures are lacked in a current transmission mode of synchronizing relational data into an ES (Extended Station). The timeliness and the consistency of the data are influenced. The method comprises the following steps: generating an index identification code for data of each data table in a current relational database; eS index generation information is configured in the project configuration file; wherein the ES index generation information at least comprises an index identification code associated with the data table; according to the ES index generation information, constructing an ES index; monitoring a binlog of the current relational database, and synchronizing change data in the binlog to a corresponding ES index; and performing consistency verification on the change data according to the index identification code stored in the ES index.

Description

[0001] This application claims priority based on the invention patent application filed with the Chinese Patent Office on December 29, 2023, with the application number 202311870376.3 and the invention title "A Method, Device, and Medium for Data Consistency Synchronous Transmission", and this application incorporates the full text of the above Chinese patent application by reference. Technical Field

[0002] The present invention relates to the technical field of data transmission consistency processing, and in particular, to a method, device, and medium for data consistency synchronous transmission. Background Art

[0003] During the project implementation process, due to the weak support ability of the relational database MySQL for full-text retrieval or fuzzy query, performance bottlenecks often occur when querying a large amount of data and frequent queries, resulting in a slow system response. It is necessary to send the data to a search engine to provide professional query services. ElasticSearch (abbreviation: ES) is an open-source distributed search engine based on RESTful web interfaces and built on Apache Lucene, supporting distributed deployment and query, which can just solve the query defects of relational databases and has a very large performance improvement compared to relational databases. Therefore, the system design often adopts the solution of storing data in the relational database MySQL and then synchronizing the data to ES for users to query data from ES.

[0004] The current data transmission methods include synchronous transmission and asynchronous transmission. Synchronous transmission is specifically synchronous dual writing, that is, when writing data to MySQL, the data is simultaneously written to ES. However, this method requires adding code to write to ES at all places where data is written to MySQL, which will reduce the system performance and increase the risk of dual-write failure. Asynchronous transmission mostly uses a timed task to scan the changes of the MySQL table and then synchronize the changed data to ES. However, the real-time update of this method is relatively low. The timed task scans at a fixed time interval, and the timer working cycle cannot be set at the second level. Therefore, ES has at least a 1-minute delay to update the data in MySQL. In some special industries, it will damage the timeliness of the data, and in severe cases, it may cause the data to lose its value. Therefore, neither of the two transmission methods can achieve the best transmission effect, and moreover, neither of these two transmission methods protects the data at all. The data may face security problems such as data leakage, identity impersonation, and data tampering during the transmission process, affecting the consistency of the data in MySQL and the data in ES. Summary of the Invention

[0005] An embodiment of the present invention provides a method, device, and medium for data consistency synchronous transmission, which are used to solve the following technical problems: The current transmission method for synchronizing relational data to ES cannot achieve synchronous update of data without reducing system performance, and lacks protection measures for data, affecting the timeliness and consistency of data.

[0006] The embodiment of the present invention adopts the following technical solutions:

[0007] On the one hand, an embodiment of the present invention provides a method for data consistency synchronous transmission. The method includes: generating index identification codes for the data of each data table in the current relational database; configuring ES index generation information in the project configuration file; where the ES index generation information at least includes the index identification codes of the associated data tables; constructing an ES index according to the ES index generation information; listening to the binlog log of the current relational database through the Canal component, and synchronizing the changed data in the binlog log to the corresponding ES index; and performing consistency verification on the changed data according to the index identification codes saved in the ES index.

[0008] To solve the data synchronization problem, the embodiment of the present invention uses the Canal component to monitor the binlog log file of MySQL in real time, and writes the changed data in MySQL into the ES cluster in real time, greatly reducing the data synchronization delay, making the business system respond faster and have higher performance. The newly written data in MySQL can be queried by users in a very short time. And the present invention also proposes an index identification code for the data table. By verifying the index identification code carried by the changed data, the changed data can be written into the ES cluster only after passing the verification, which can ensure the consistency of the data in the ES cluster and the data in MySQL, and prevent security problems such as data leakage, identity impersonation, and data tampering.

[0009] In a feasible implementation manner, generating index identification codes for the data of each data table in the current relational database specifically includes: obtaining the data flow directions of the data in each data table in the current relational database; where the data flow direction is used to indicate the ES index that the relational data needs to pass through when transmitted to the ES cluster; creating table-level index identification codes and field-level index identification codes for the data tables with different data flow directions respectively; and connecting the table-level index identification code and the field-level index identification code through a connector to obtain the final index identification code of the data table.

[0010] This embodiment analyzes the data flow directions of each data table, generates the same index identification codes for the data tables with the same data flow direction, and different index identification codes for the data tables with different data flow directions, which are used to verify the identity of the changed data after the ES index receives it, and ensure the consistency before and after data transmission.

[0011] In a feasible implementation manner, the ES index generation information further includes at least one of an ES index name, a server_id, MySQL connection information, ES connection information, the name of an associated database, and the table name of an associated data table.

[0012] In a feasible implementation manner, constructing an ES index according to the ES index generation information specifically includes: generating an ES index with a corresponding name according to the ES index name, and initializing the ES index; determining the associated database and the associated data table of each ES index according to the name of the associated database and the table name of the associated data table in the ES index generation information; binding each ES index to the index identification code of the associated data table to complete the construction of the ES index.

[0013] In this embodiment, according to the database name and the associated data table name associated with each ES index, the ES index is bound to the index identification code of the associated data table, which facilitates subsequent security verification of the received data.

[0014] In a feasible implementation manner, after generating an ES index with a corresponding name according to the ES index name and initializing the ES index, the method further includes: obtaining the data volume of all data in the current relational database; determining the number of shards of the ES index according to the data volume, and performing sharding processing on the ES index based on the number of shards to obtain ES index primary shards; performing several replications on each ES index primary shard to obtain several ES index secondary shards; storing the ES index primary shards in the primary server of the ES cluster, and storing the ES index secondary shards in other servers of the ES cluster.

[0015] In a network environment, data transmission failures may occur at any time. In the case where a certain shard / ES node is offline or suddenly disappears for any reason, a failover mechanism needs to be created. Therefore, in this embodiment, by sharding the ES index, each shard itself constitutes a fully functional and independent index, and each shard is replicated, and the replicated shards are not placed on the same node as the original shard. This not only improves the processing ability of the ES index for large amounts of data through sharding, expands the search volume / throughput, but also provides high availability through replicated shards in the case where a certain shard fails.

[0016] In a feasible implementation, monitor the binlog of the current relational database and synchronize the changed data in the binlog to the corresponding ES index. Specifically, it includes: obtaining the location of the MySQL-binlog file, screening out the binlog files to be monitored according to the project configuration file; using the Canal component to monitor the binlog files in real time; when new data is detected in the binlog files, parsing the binlog files through the Canal component to obtain the changed data; parsing the data table to which the changed data belongs, and obtaining the data flow direction and index identification code of the data table; converting the changed data into row data, splitting the index identification code into several parts, and inserting them into the preset positions of the row data; writing the processed row data into the ES index corresponding to the data flow direction.

[0017] In this embodiment, the Canal component is used to monitor the binlog files of MySQL in real time, and the changed data in MySQL is written into the ES cluster in real time, greatly reducing the data synchronization delay, making the business system respond faster and have higher performance. And when transmitting the changed data, multiple split index identification codes are inserted at the preset positions. If the data is tampered with midway, the number of bytes will definitely change, and then the positions of the index identification codes interspersed in it will also change, so it is impossible to parse the correct index identification code, that is, the situation where the data is tampered with can be identified.

[0018] In a feasible implementation, according to the index identification codes saved in the ES index, perform consistency verification on the changed data. Specifically, it includes: parsing the changed data received by the ES index, and extracting the index identification codes carried in the changed data according to the preset extraction algorithm; matching the index identification codes corresponding to the changed data among all the index identification codes saved in the ES index. If the match is successful, the consistency verification of the changed data is successful.

[0019] In this embodiment, by parsing the index identification codes in the changed data and comparing them with the index identification codes associated with the ES index, it is verified whether the currently received changed data is the original data sent by MySQL, which is beneficial to ensuring the consistency of the data on both sides.

[0020] In a feasible implementation, after performing consistency verification on the changed data according to the index identification codes saved in the ES index, the method further includes: after the consistency verification is successful, call the _bulk interface of the ES cluster where the ES index is located to write the changed data in the ES index into the ES cluster; according to the comparison situation between the changed data and the original data, perform modification operations or addition operations on the original data in the ES cluster respectively.

[0021] On the other hand, an embodiment of the present invention further provides a data consistency synchronous transmission device, which includes: a generation module for generating index identification codes for the data of each data table in the current relational database;

[0022] a configuration module for configuring ES index generation information in a project configuration file; wherein, the ES index generation information at least includes the index identification codes of associated data tables;

[0023] an index module for constructing an ES index according to the ES index generation information;

[0024] a synchronization module for listening to the binlog log of the current relational database and synchronizing the changed data in the binlog log to the corresponding ES index;

[0025] a verification module for performing consistency verification on the changed data according to the index identification codes stored in the ES index.

[0026] In a third aspect, an embodiment of the present invention further provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, so that the at least one processor can execute the data consistency synchronous transmission method.

[0027] Finally, an embodiment of the present invention further provides a non-volatile computer storage medium storing computer-executable instructions, and when the computer-executable instructions are executed by a computer, the data consistency synchronous transmission method can be implemented.

[0028] Compared with the prior art, a data consistency synchronous transmission method, device and medium provided by an embodiment of the present invention have the following beneficial effects:

[0029] The data consistency synchronous transmission solution provided by the present invention monitors the binlog log file of MySQL in real time through the Canal component, and writes the changed data in MySQL into the ES cluster in real time. While solving the data synchronization problem, it does not increase the code amount, does not increase the system processing burden, but reduces the data synchronization delay, making the business system respond faster and have higher performance. The newly written data in MySQL can be queried by users in a very short time. In addition, the present invention also proposes an index identification code for data tables. When transmitting changed data, multiple segments of split index identification codes are inserted at preset positions. If the data is tampered with midway, the number of bytes will definitely change, and then the positions of the index identification codes interspersed therein will also change accordingly, so that the correct index identification code cannot be parsed, and the situation where the data is tampered with can be identified. By verifying the index identification code carried by the changed data, the changed data can be written into the ES cluster only after passing the verification, which can ensure the consistency of the data in the ES cluster and the data in MySQL, and prevent security problems such as data leakage, identity impersonation, and data tampering. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0031] Figure 1 It is a flowchart of a data consistency synchronous transmission method provided by an embodiment of the present invention;

[0032] Figure 2 It is a schematic structural diagram of a data consistency synchronous transmission device provided by an embodiment of the present invention;

[0033] Figure 3 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0035] An embodiment of the present invention provides a data consistency synchronous transmission method, asFigure 1 As shown in Figure 1 , the data consistency synchronous transmission method specifically includes steps S101 - S105:

[0036] S101. Generate index identification codes for the data of each data table in the current relational database.

[0037] Specifically, in the present invention, the ES indexes through which each data table in the current relational database needs to be transmitted to the ES service node are solidified. It can be understood that the data table is bound to the ES index, and the ES index is also stored in different ES nodes in the ES cluster, so that the data flow direction of the data table can be determined according to the ES node where the ES index is located.

[0038] Furthermore, create table - level index identification codes and field - level index identification codes for data tables with different data flow directions respectively; connect the table - level index identification code and the field - level index identification code through a connector to obtain the final index identification code of the data table.

[0039] As a feasible implementation manner, table - level data is uniquely identified by sourceId, targetId, and tabId to obtain the table - level index identification code; field - level data is uniquely identified by colId to obtain the field - level index identification code. Finally, the two are connected through a connector to obtain the final index identification code of the data table: sourceId - targetId - tabId - colId.

[0040] In this embodiment, by analyzing the data flow directions of each data table, the data tables with the same data flow direction are generated with the same index identification code, and the data tables with different data flow directions are generated with different index identification codes, which are used to verify the identity of the changed data after the ES index receives it, ensuring the consistency before and after data transmission.

[0041] S102. Configure ES index generation information in the project configuration file; wherein, the ES index generation information includes at least one of the index identification code of the associated data table, the ES index name, server_id, MySQL connection information, ES connection information, the name of the associated database, and the name of the associated data table.

[0042] The above - mentioned ES index generation information is a part of the ES index generation rule. Before the initial data transmission between MySQL and ES, it is necessary to first construct the corresponding ES index according to the ES index generation rule and the ES index generation information for synchronizing the data in MySQL to the ES cluster. When incremental data update is required due to subsequent data changes, it is not necessary to reconstruct the ES index again.

[0043] S103. Construct the ES index according to the ES index generation information.

[0044] Specifically, the ES index generation rule is as follows: Generate an ES index with the corresponding name according to the ES index name, and initialize the ES index; Determine the associated database and the associated data table of each ES index according to the name of the associated database and the table name of the associated data table in the ES index generation information; Bind each ES index to the index identification code of the associated data table to complete the construction of the ES index.

[0045] As a feasible implementation manner, since all the data in MySQL needs to be transferred to the ES cluster during the first data migration, a large amount of data needs to be processed simultaneously, which is likely to cause a relatively large pressure on the ES cluster. Therefore, when creating an ES index in the present invention, first obtain the data volume of all the data in the current relational database MySQL. Then, determine the number of shards of the ES index according to the data volume and the data volume that each ES index can process in the normal state, and perform sharding processing on the ES index based on the number of shards to obtain the ES index primary shards. Make several copies of each ES index primary shard to obtain several ES index secondary shards. Finally, store the ES index primary shards in the primary server of the ES cluster, and store the ES index secondary shards in other servers of the ES cluster.

[0046] In a network environment, data transmission failures may occur at any time. In the case where a certain shard / ES node is offline or suddenly disappears for any reason, a failover mechanism needs to be created. Therefore, in this embodiment, by sharding the ES index, each shard itself constitutes a fully functional and independent index, and each shard is copied, and the copied shards are not placed on the same node as the original shards. Not only can the processing ability of the ES index for large amounts of data be improved through sharding, expanding the search volume / throughput, but also high availability can be provided through copying shards in the case where a certain shard fails.

[0047] S104. Monitor the binlog log of the current relational database, and synchronize the changed data in the binlog log to the corresponding ES index.

[0048] Specifically, obtain the location of the MySQL-binlog file, screen out the binlog log files that need to be monitored according to the project configuration file, and then use the Canal component to monitor these binlog log files in real time.

[0049] When new data is detected in the binlog log file, parse the binlog log file through the Canal component to obtain the changed data. Parse the data table to which the changed data belongs, and obtain the data flow direction and the index identification code of the data table.

[0050] Further, convert the changed data into row data, split the index identification code into several parts, and insert them into the preset positions of the row data. Write the processed row data into the ES index corresponding to the data flow.

[0051] In one embodiment, the index identification code is split into 3 parts and inserted into the 10th byte, 20th byte, and 30th byte of the row data in sequence.

[0052] In this embodiment, the Canal component is used to monitor the binlog log file of MySQL in real time, and the changed data in MySQL is written into the ES cluster in real time, which greatly reduces the data synchronization delay, makes the business system respond faster and have higher performance. And when transmitting the changed data, multiple split index identification codes are inserted at the preset positions. If the data is tampered with in the middle, the number of bytes will definitely change, and then the positions of the index identification codes interspersed in it will also change, so it is impossible to parse the correct index identification code, and the situation where the data is tampered with can be identified.

[0053] S105. Perform consistency verification on the changed data according to the index identification code stored in the ES index.

[0054] Specifically, parse the changed data received by the ES index, and extract the index identification code carried in the changed data according to the preset extraction algorithm. Match the index identification code corresponding to the changed data among all the index identification codes stored in the ES index. If the match is successful, the consistency verification of the changed data is successful.

[0055] As a feasible implementation method, according to the insertion position of the specified index identification code and the length of each part of the index identification code, set the extraction position of each part of the index identification code in the preset extraction algorithm. For example, if 3-byte local index identification codes are inserted after the 10th byte, 20th byte, and 30th byte of the changed data respectively, then the extraction position of the first part is the 11th byte, the extraction position of the second part is the 24th byte, and the extraction position of the third part is the 37th byte.

[0056] In this embodiment, by parsing the index identification code in the changed data and comparing it with the index identification code associated with the ES index, it is verified whether the currently received changed data is the original data sent by MySQL, which is beneficial to ensuring the consistency of both parties' data.

[0057] Further, after the consistency verification is successful, call the _bulk interface of the ES cluster where the ES index is located to write the changed data in the ES index into the ES cluster; according to the comparison situation between the changed data and the original data, perform modification operations or addition operations on the original data in the ES cluster respectively.

[0058] The data consistency synchronous transmission solution provided by the present invention monitors the binlog log file of MySQL in real time through the Canal component, and writes the changed data in MySQL into the ES cluster in real time. While solving the data synchronization problem, it does not increase the code volume and does not increase the system processing burden. Instead, it reduces the data synchronization delay, making the business system respond faster and have higher performance. The newly written data in MySQL can be queried by users in a very short time. In addition, the present invention also proposes an index identification code for data tables. When transmitting changed data, multiple segments of split index identification codes are inserted at preset positions. If the data is tampered with midway, the number of bytes will surely change, and then the positions of the index identification codes interspersed therein will also change, so that the correct index identification code cannot be parsed out, and the situation where the data is tampered with can be identified. By verifying the index identification code carried by the changed data, the changed data can be written into the ES cluster only after passing the verification, which can ensure the consistency of the data in the ES cluster and the data in MySQL, and prevent security problems such as data leakage, identity impersonation, and data tampering.

[0059] In addition, an embodiment of the present invention also provides a data consistency synchronous transmission device, as Figure 2 shown. The data consistency synchronous transmission device 200 specifically includes:

[0060] A generation module 210, configured to generate index identification codes for the data of each data table in the current relational database;

[0061] A configuration module 220, configured to configure ES index generation information in a project configuration file; wherein, the ES index generation information at least includes the index identification codes of associated data tables;

[0062] An index module 230, configured to construct an ES index according to the ES index generation information;

[0063] A synchronization module 240, configured to monitor the binlog log of the current relational database and synchronize the changed data in the binlog log to the corresponding ES index;

[0064] A verification module 250, configured to perform consistency verification on the changed data according to the index identification codes stored in the ES index.

[0065] In addition, an embodiment of the present invention also provides an electronic device, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the data consistency synchronous transmission method in any of the above embodiments. For example, in one embodiment, the above processor can execute:

[0066] Generate index identification codes for the data of each data table in the current relational database; configure ES index generation information in the project configuration file; where the ES index generation information at least includes the index identification codes of the associated data tables; construct an ES index according to the ES index generation information; monitor the binlog logs of the current relational database, and synchronize the changed data in the binlog logs to the corresponding ES index; perform consistency verification on the changed data according to the index identification codes saved in the ES index.

[0067] Finally, the embodiments of the present invention also provide a non-volatile computer storage medium storing computer-executable instructions, and when the computer-executable instructions are executed by a computer, they can implement the data consistency synchronization transmission method in any of the above embodiments. For example, in one embodiment, the above computer-executable instructions can implement:

[0068] Generate index identification codes for the data of each data table in the current relational database; configure ES index generation information in the project configuration file; where the ES index generation information at least includes the index identification codes of the associated data tables; construct an ES index according to the ES index generation information; monitor the binlog logs of the current relational database, and synchronize the changed data in the binlog logs to the corresponding ES index; perform consistency verification on the changed data according to the index identification codes saved in the ES index.

[0069] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and non-volatile computer storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0070] The above describes specific embodiments of this specification. Other embodiments are within the scope of the present invention. In some cases, the actions or steps recorded in this specification can be executed in a different order from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In certain embodiments, multi-tasking and parallel processing are also possible or may be advantageous.

[0071] The above description is only for one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, various changes and modifications can be made to one or more embodiments of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included within the protection scope of this specification.

Claims

1. A data consistency synchronous transmission method, characterized in that, The method includes: Generating index identification codes for the data in each data table of the current relational database; Configuring ES index generation information in the project configuration file; wherein, the ES index generation information at least includes the index identification codes of the associated data tables; Constructing an ES index according to the ES index generation information; Listening to the binlog log of the current relational database, and synchronizing the changed data in the binlog log to the corresponding ES index; Performing consistency verification on the changed data according to the index identification codes saved in the ES index.

2. The data consistency synchronous transmission method according to claim 1, characterized in that, Generating index identification codes for the data in each data table of the current relational database specifically includes: Obtaining the data flow directions in each data table of the current relational database; wherein, the data flow direction is used to indicate the ES index that the relational data needs to pass through when transmitted to the ES cluster; Creating table-level index identification codes and field-level index identification codes for the data tables with different flow directions respectively; Connecting the table-level index identification code and the field-level index identification code with a connector to obtain the final index identification code of the data table.

3. A data consistency synchronous transmission method according to claim 1, characterized in that, The ES index generation information further includes at least one of: ES index name, server_id, MySQL connection information, ES connection information, name of the associated database, and table name of the associated data table.

4. A data consistency synchronous transmission method according to claim 2, characterized in that Constructing an ES index according to the ES index generation information specifically includes: Generating an ES index with the corresponding name according to the ES index name, and initializing the ES index; Determining the associated database and the associated data table of each ES index according to the name of the associated database and the table name of the associated data table in the ES index generation information; Binding each ES index with the index identification code of the associated data table to complete the construction of the ES index.

5. A data consistency synchronous transmission method according to claim 4, characterized in that After generating an ES index with the corresponding name according to the ES index name and initializing the ES index, the method further includes: Obtaining the data volume of all the data in the current relational database; Determining the number of shards of the ES index according to the data volume, and performing sharding processing on the ES index based on the number of shards to obtain the ES index primary shards; Performing several replications on each ES index primary shard to obtain several ES index secondary shards; Storing the ES index primary shards in the primary server of the ES cluster, and storing the ES index secondary shards in other servers of the ES cluster.

6. A data consistency synchronous transmission method according to claim 1, characterized in that Listening to the binlog log of the current relational database, and synchronizing the changed data in the binlog log to the corresponding ES index specifically includes: Obtaining the MySQL-binlog file position, and screening out the binlog log files that need to be listened to according to the project configuration file; Real-time listening to the binlog log files through the Canal component; When new data is monitored in the binlog log files, parsing the binlog log files through the Canal component to obtain the changed data; Parse the data table to which the changed data belongs, and obtain the data flow direction and index identification code of the data table; Convert the changed data into row data, split the index identification code into several parts, and insert them into the preset positions of the row data; Write the processed row data into the ES index corresponding to the data flow direction.

7. A data consistency synchronous transmission method according to claim 1, characterized in that, Perform consistency verification on the changed data according to the index identification code stored in the ES index, specifically including: Parse the changed data received by the ES index, and extract the index identification code carried in the changed data according to the preset extraction algorithm; Match the index identification code corresponding to the changed data among all the index identification codes stored in the ES index. If the match is successful, the consistency verification of the changed data is successful.

8. A data consistency synchronous transmission method according to claim 1, characterized in that, After performing consistency verification on the changed data according to the index identification code stored in the ES index, the method further includes: After the consistency verification is successful, call the bulk interface of the ES cluster where the ES index is located, and write the changed data in the ES index into the ES cluster; According to the comparison between the changed data and the original data, perform modification operations or addition operations on the original data in the ES cluster respectively.

9. A data consistency synchronous transmission device, characterized in that, The device includes: A generation module for generating index identification codes for the data of each data table in the current relational database; A configuration module for configuring ES index generation information in the project configuration file; wherein, the ES index generation information at least includes the index identification codes of the associated data tables; An index module for constructing an ES index according to the ES index generation information; A synchronization module for listening to the binlog log of the current relational database and synchronizing the changed data in the binlog log to the corresponding ES index; A verification module for performing consistency verification on the changed data according to the index identification code stored in the ES index.

10. An electronic device, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the data consistency synchronization transmission method according to any one of claims 1-8.

11. A non-volatile computer storage medium, characterized in that, Stores computer-executable instructions that, when executed by a computer, can implement the data consistency synchronization transmission method according to any one of claims 1-8.

Images