Cross-environment data synchronization method, system, storage medium and computer

By analyzing and pushing database change data, determining the target environment and encrypting the write message queue, the coupling problem of cross-environment data synchronization in the existing technology is solved, and efficient and stable data synchronization is achieved, which is suitable for large-scale applications.

CN118708654BActive Publication Date: 2024-11-05ZIGUANG MODU EDUCATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411186990.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-05
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

In the existing cross-environment data synchronization methods, triggers are difficult to debug and maintain, and the business code is coupled with the data synchronization code, affecting performance and stability.

Method used

By obtaining the change data of the source environment database, parsing and pushing it to the message array, determining the target environment, encrypting data according to the synchronization table range, writing to the public network message queue, decrypting the target environment and executing data change statements, achieving cross-environment synchronization.

Benefits of technology

Real-time synchronization of cross-environment data is achieved, timeliness and accuracy is improved, suitable for large-scale promotion, and is non-invasive to business and has high stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118708654B_ABST
    Figure CN118708654B_ABST
Patent Text Reader

Abstract

The present invention provides a cross-environment data synchronization method, system, storage medium and computer. The cross-environment data synchronization method includes: obtaining and parsing the change data in the source environment database, and pushing the parsed change data to the source environment message array; after receiving the message in the source environment message array, determining the target environment where the change data needs to be synchronized; obtaining the synchronization table range configured in the target environment, determining whether the change data is valid according to the synchronization table range and the data table of the change data, encrypting the valid change data and writing it into the public network message queue; obtaining the message in the public network message queue and decrypting the data, converting the decrypted data into an execution statement, and executing it in the database in the target environment to synchronize the change data in the source environment to the target environment. The cross-environment data synchronization method provided by the present application has good stability and high timeliness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cross-environment data synchronization method, system, storage medium and computer. Background Art

[0002] With the development of random technology, cross-environment data synchronization is required between databases in multiple environments in many scenarios. For example, region A and school a under region A each build a software system. If a new user is added to school a in region A, the user data will be synchronized to the software environment built by school a in real time, and this user can be used to perform corresponding business operations in the environment of school a.

[0003] Currently, the commonly used cross-environment data synchronization methods are: first use triggers to monitor and obtain database changes, but triggers are executed inside the database, which is not easy to debug and maintain, and the triggers may affect the performance and stability of the database during execution; when the data changes, the data is directly sent to the message queue through the code, and then synchronized to other environments. This method couples the business code and the data synchronization code together, increasing the code complexity. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a cross-environment data synchronization method, system, storage medium and computer to solve the technical problems existing in the prior art.

[0005] The present invention provides a cross-environment data synchronization method, comprising:

[0006] Acquire the change data in the source environment database, parse the acquired change data, and push the parsed change data to the source environment message array;

[0007] After receiving the message in the source environment message array, the source environment data sending server determines the target environment where the changed data needs to be synchronized;

[0008] Obtain the synchronization table range of the target environment configuration, determine whether the change data is valid according to the synchronization table range and the data table of the change data, encrypt the valid change data and write it into the public network message queue;

[0009] Obtaining messages in a public network message queue and decrypting the data, converting the decrypted data into execution statements, and executing them in a database in the target environment;

[0010] If the execution is successful, the data table in the target environment is modified accordingly to synchronize the changed data in the source environment to the target environment, and the synchronization results are fed back to the source environment;

[0011] If the execution fails, analyze the reason for the failure and issue an alarm, and feed the alarm information back to the source environment.

[0012] Preferably, the step of parsing the acquired change data comprises:

[0013] Initialize the connection parameters to establish a connection with MySQL in the source environment, and request binlog data from the MySQL Master server through the COM_BINLOG_DUMP command;

[0014] After receiving the request, the MySQL Master server transmits the binlog data to the binlog parser according to the MySQL master-slave replication protocol based on the file name and location provided in the binlog data;

[0015] After receiving the binlog data, the binlog parsing program parses the data to decompose the database changes and data table changes in the change data.

[0016] Preferably, the binlog parsing program receives the binlog data and performs parsing to decompose the database changes and data table changes in the change data, including:

[0017] Obtaining a file structure of the binlog data, wherein the file structure of the binlog data includes at least a file identifier, an event identifier, and corresponding event data;

[0018] Determine the event type according to the event identifier, obtain the data change preceding event of the changed data based on the event type and the event data, and obtain the changed database and data table information in the changed data according to the data change preceding event, wherein the data table information includes at least a data table name and a data table structure;

[0019] The number of rows affected by the changed data and the specific changed data rows are determined according to the changed database and data table information, and the events in the changed data rows are converted into a standard message format to obtain the database changes and data table changes in the changed data.

[0020] Preferably, after parsing the acquired change data, the cross-environment data synchronization method further includes:

[0021] Get all data tables in the source environment and all data tables in the target environment respectively;

[0022] Determine associated data tables and unassociated data tables in the source environment and the target environment, wherein associated data tables push data in the same thread, and unassociated data tables push data in multiple parallel threads.

[0023] Preferably, after the source environment data sending server receives the message in the source environment message array, the step of determining the target environment to which the changed data needs to be synchronized includes:

[0024] Get all data tables of changed data, select some data tables for query in turn, and query the school ID field in all data tables through the associated query operation of the database;

[0025] Determine the school information to which the current change data belongs based on the queried school ID field, and judge whether the school has built a separate software environment;

[0026] If the school has built a software environment independently, the software environment of the school is the target environment that needs to be synchronized, and the region and platform to which the target environment belongs are determined based on the school information;

[0027] If the school does not build a separate software environment, the current change data will be ignored.

[0028] Preferably, the steps of obtaining the synchronization table range of the target environment configuration, determining whether the change data is valid according to the synchronization table range and the data table of the change data, and encrypting the valid change data and writing it into the public network message queue include:

[0029] Obtain the synchronization table range of the target environment from the configuration center, and determine whether there is a corresponding table in the synchronization table range for the data table of the changed data;

[0030] If a corresponding table exists, obtain the preceding associated table associated with the data table of the changed data;

[0031] Obtain the unique key of the target environment from the configuration center, encrypt the unique key of the target environment, the data table of the changed data, and the associated pre-association table, and write them into the public network message queue;

[0032] If the corresponding table does not exist, the current change data is ignored.

[0033] Preferably, the changed data includes one or more of newly added data, deleted data and updated data;

[0034] If the execution is successful, the steps of modifying the data table in the target environment accordingly to synchronize the changed data in the source environment to the target environment include:

[0035] If the changed data is new data or deleted data, a new statement or a delete statement is executed in the database of the target environment to correspondingly add or delete data rows in the target environment data table, so as to synchronize the changed data in the source environment to the target environment;

[0036] If the changed data is updated data, execute the update statement in the database of the target environment to determine the number of data rows affected after the update statement is executed in the target environment;

[0037] If the number of affected data rows is not zero, the data rows in the target environment data table are updated accordingly to synchronize the changed data in the source environment to the target environment;

[0038] If the number of affected data rows is zero, the update statement is converted into a new statement, and the converted new statement is executed in the target environment, so that a data row corresponding to the updated data is added to the data table of the target environment, so as to synchronize the changed data in the source environment to the target environment;

[0039] If the execution fails, the steps of analyzing the cause of the execution failure and issuing an alarm, and feeding back the alarm information to the source environment include:

[0040] If the execution fails due to a missing field error, find the missing field from the execution statement, remove it, and execute again;

[0041] If the execution fails due to a data primary key conflict, the current changed data is considered to have been synchronized to the target environment and the changed data is ignored;

[0042] If the execution fails due to content conflicts in some fields, clear the content of the conflicting fields in the script and execute again after clearing them;

[0043] If the reason for the execution failure is other than field missing error, data primary key conflict, and partial field content conflict, a separate message array is set to record the failure reason and retry the data execution. If the execution still fails after the preset number of retries, an alarm is issued and the alarm information is fed back to the source environment.

[0044] The present invention also provides a cross-environment data synchronization system, comprising:

[0045] A parsing module, used for acquiring change data in a source environment database, parsing the acquired change data, and pushing the parsed change data to a source environment message array;

[0046] A confirmation module, used for determining a target environment to which the changed data needs to be synchronized after the source environment data sending server receives a message in the source environment message array;

[0047] A writing module, used for obtaining the synchronization table range of the target environment configuration, determining whether the change data is valid according to the synchronization table range and the data table of the change data, and encrypting the valid change data and writing it into the public network message queue;

[0048] An execution module, used to obtain messages in a public network message queue and decrypt the data, convert the decrypted data into an execution statement, and execute it in a database in the target environment;

[0049] If the execution is successful, the data table in the target environment is modified accordingly to synchronize the changed data in the source environment to the target environment, and the synchronization results are fed back to the source environment;

[0050] If the execution fails, analyze the reason for the failure and issue an alarm, and feed the alarm information back to the source environment.

[0051] The present invention also provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned cross-environment data synchronization method is implemented.

[0052] The present invention also proposes a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned cross-environment data synchronization method when executing the computer program.

[0053] Compared with the prior art, the beneficial effects of the present invention are as follows: the cross-environment data synchronization method provided by the present application first obtains the change data in the source environment database, parses the obtained change data, and pushes the parsed change data to the source environment message array; after confirming the target environment that needs to be synchronized, obtains the data table of the change data of the synchronization table range configured in the target environment to determine whether the change data is valid, encrypts the valid change data and writes it into the public network message queue; the target environment decrypts and converts it into an execution statement for execution to synchronize data across environments. The cross-environment data synchronization method provided by the present application can configure the information that needs to be synchronized across environments in terms of libraries, tables, and fields; it is non-invasive to the current business and will not affect the functions and performance of the business itself; compared with scheduled synchronization, it has higher timeliness, can push the changed data to the target environment in real time, and supports the template engine for data association and filtering; it has high accuracy and is suitable for large-scale promotion.

[0054] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of a cross-environment data synchronization method in a first embodiment of the present invention;

[0056] Figure 2 It is a structural block diagram of a computer in the fifth embodiment of the present invention.

[0057] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0058] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0060] Embodiment 1

[0061] See also Figure 1 , which shows a cross-environment data synchronization method in a first embodiment of the present invention, and the cross-environment data synchronization method specifically includes steps S10 to S50:

[0062] S10, obtaining the change data in the source environment database, parsing the obtained change data, and pushing the parsed change data to the source environment message array;

[0063] For ease of explanation, in this embodiment, the source environment can be understood as the software environment built in area A, and the target environment can be understood as the software environment built in school a under area A.

[0064] In the specific implementation, firstly, the source environment is monitored to obtain the change data in the source environment database, and the change data is parsed and pushed to the source environment message array;

[0065] Optionally, the step of parsing the acquired change data includes:

[0066] Initialize the connection parameters to establish a connection with MySQL in the source environment, and request binlog data from the MySQL Master server through the COM_BINLOG_DUMP command;

[0067] After receiving the request, the MySQL Master server transmits the binlog data to the binlog parser according to the MySQL master-slave replication protocol based on the file name and location provided in the binlog data;

[0068] After receiving the binlog data, the binlog parsing program parses the data to decompose the database changes and data table changes in the change data.

[0069] Optionally, the binlog parsing program parses the binlog data after receiving it, and the step of decomposing the database changes and data table changes in the change data includes:

[0070] Obtaining a file structure of the binlog data, wherein the file structure of the binlog data includes at least a file identifier, an event identifier, and corresponding event data;

[0071] Determine the event type according to the event identifier, obtain the data change preceding event of the changed data based on the event type and the event data, and obtain the changed database and data table information in the changed data according to the data change preceding event, wherein the data table information includes at least a data table name and a data table structure;

[0072] The number of rows affected by the changed data and the specific changed data rows are determined according to the changed database and data table information, and the events in the changed data rows are converted into a standard message format to obtain the database changes and data table changes in the changed data.

[0073] In this embodiment, MySQL is a relational database management server, in which the relational database stores data in different tables. It is open source, fast, and flexible. It is a relational database management server widely used in software environments. MySQL can parse binlog to obtain change data. Binlog data is binary log data that stores change data in MySQL. MySQL includes a master server (MySQL Master) and a slave server (MySQL Slave). The connection parameters are initialized according to the configured target data, so that the software environment constructed by the source environment establishes a connection with MySQL, so that data can be obtained in MySQL.

[0074] Optionally, the slave server sends a COM_BINLOG_DUMP command to the master server, obtains binlog data from the master server according to the MySQL master-slave replication protocol, and transmits the binlog data to the binlog parser according to the file name and specific location of the change data in the binlog data; the parser parses the binlog data after receiving it, and decomposes the change details of the specific database change operation data table.

[0075] Specifically, the file structure of binlog data includes a file identifier, an event identifier and corresponding event data. The file identifier is used to identify that the file is a binlog file. The event identifier mainly includes the timestamp of the event generated in the change data, the event type (addition, update, deletion, etc.), the server ID of the event, the total length of the event, etc. The event data formats of different event types are different. In this embodiment, the following four events are mainly concerned: data change precursor event (Table MapEvent), new event (Write Rows Event), delete event (Write Rows Event), update event (UpdateRows Event); the data change precursor event occurs before the new event, delete event, and update event, and is used to record the database, table name, table structure and other information of the data change; the event data in the new event includes at least one or more rows of insert operations, the event data in the delete event includes at least one or more rows of delete operations, and the event data in the update event includes at least one or more rows of update operations.

[0076] Optionally, the changed database and data table information is obtained through the data change pre-event, the number of data rows affected by the changed data and the specific data rows that have changed are determined, and these events are converted into a standard message format to facilitate subsequent push to the source environment message array; the library and table are the basis of data changes, and all data changes are based on changes in the table data of the database; preferably, the standard message format can include information such as the database, data table, primary key list, event type, event occurrence time, data after the change, data before the change, etc.

[0077] Optionally, after parsing the acquired change data, in order to improve the accuracy and efficiency of data push, in this embodiment, the cross-environment data synchronization method further includes:

[0078] Get all data tables in the source environment and all data tables in the target environment respectively;

[0079] Determine associated data tables and unassociated data tables in the source environment and the target environment, wherein associated data tables push data in the same thread, and unassociated data tables push data in multiple parallel threads.

[0080] Optionally, the user is affiliated with the school and changes the data table in the database of the source environment. If there is a data table associated with the source environment data table in the target environment, for example, the data in the school table and the user table are changed at the same time, the changed data in the school table and the user table can be put into the same thread, and the data can be synchronized in the order of synchronizing school data first and then synchronizing user data, and the thread usage can be reduced through single-threaded processing; on the contrary, each school may have classroom information and course information. Although these two types of data are both affiliated with the school, there is no dependency between them. At this time, these two types of data can be pushed under different threads, and the data push rate can be accelerated through multi-threaded parallel processing. Each thread is an independent individual and does not interfere with each other.

[0081] S20, after receiving the message in the source environment message array, the source environment data sending server determines the target environment where the changed data needs to be synchronized;

[0082] Optionally, after the source environment data sending server receives the message in the source environment message array, the step of determining the target environment to which the changed data needs to be synchronized includes:

[0083] Get all data tables of changed data, select some data tables for query in turn, and query the school ID field in all data tables through the associated query operation of the database;

[0084] Determine the school information to which the current change data belongs based on the queried school ID field, and judge whether the school has built a separate software environment;

[0085] If the school has built a software environment independently, the software environment of the school is the target environment that needs to be synchronized, and the region and platform to which the target environment belongs are determined based on the school information;

[0086] If the school does not build a separate software environment, the current change data will be ignored.

[0087] Specifically, most data tables will have a school ID field, which can be used directly. If some data tables do not have a school ID field, you can indirectly obtain the school ID field through the database association query operation; obtain the school's ownership information based on the school ID field, and then determine whether the school has built its own software environment independently; if the school has built its own software environment independently, cross-environment data synchronization is required between the school and the source environment. Furthermore, the region and platform to which the school belongs can be determined based on the school's ownership information; if the school has not built its own software environment independently, that is, the school and the source environment are the same software environment, there is no need to make cross-environment changes, and the changed data can be ignored.

[0088] S30, obtaining the synchronization table range of the target environment configuration, determining whether the changed data is valid according to the synchronization table range and the data table of the changed data, encrypting the valid changed data and writing it into the public network message queue;

[0089] Optionally, the step of obtaining the synchronization table range of the target environment configuration, determining whether the changed data is valid according to the synchronization table range and the data table of the changed data, and encrypting the valid changed data and then writing it into the public network message queue includes:

[0090] Obtain the synchronization table range of the target environment from the configuration center, and determine whether there is a corresponding table in the synchronization table range for the data table of the changed data;

[0091] If a corresponding table exists, obtain the preceding associated table associated with the data table of the changed data;

[0092] Obtain the unique key of the target environment from the configuration center, encrypt the unique key of the target environment, the data table of the changed data, and the associated pre-association table, and write them into the public network message queue;

[0093] If the corresponding table does not exist, the current change data is ignored.

[0094] Specifically, suppose that there are data tables B and C in the target environment. If the data table changed in the source environment is data table B corresponding to data table B, i or data table C corresponding to data table C i ; then further obtain the predecessor association table associated with the data table; for example, each user has his own role, such as user A is an ordinary faculty member, user B is a student, and user C is a school administrator; the data table in the change data in the source environment is a user role association table, then further obtain the predecessor association table of the user role association table, the predecessor association table can be a user table and a role table, and send them together to the target environment; obtain the unique key of the target environment from the configuration center, encrypt the unique key of the target environment, the data table of the change data, and the predecessor association table that depends on it, and then encrypt them through AES (Advanced Encryption Standard, symmetric encryption algorithm) and write them into the public network message queue.

[0095] If the data table changed in the source environment is data table D i , which does not correspond to the data table in the target environment, the current changed data will be ignored.

[0096] S40, obtaining a message in a public network message queue and decrypting the data, converting the decrypted data into an execution statement, and executing the statement in a database in the target environment;

[0097] If the execution is successful, the data table in the target environment is modified accordingly to synchronize the changed data in the source environment to the target environment, and the synchronization results are fed back to the source environment;

[0098] If the execution fails, analyze the reason for the failure and issue an alarm, and feed the alarm information back to the source environment;

[0099] Optional, if the execution is successful, the specific steps include:

[0100] If the changed data is new data or deleted data, a new statement or a delete statement is executed in the database of the target environment to correspondingly add or delete data rows in the target environment data table, so as to synchronize the changed data in the source environment to the target environment;

[0101] If the changed data is updated data, execute the update statement in the database of the target environment to determine the number of data rows affected after the update statement is executed in the target environment;

[0102] If the number of affected data rows is not zero, the data rows in the target environment data table are updated accordingly to synchronize the changed data in the source environment to the target environment;

[0103] If the number of affected data rows is zero, the update statement is converted into a new statement, and the converted new statement is executed in the target environment, so that a data row corresponding to the updated data is added to the data table of the target environment, so as to synchronize the changed data in the source environment to the target environment;

[0104] Optionally, when the changed data in the source environment is new data and deleted data, after successful execution, the corresponding data rows are added or deleted in the target environment to complete the cross-environment data synchronization; when the changed data is updated data, only the data is modified. After successful execution, it is necessary to further determine the affected data rows of the data table in the target environment. If there are affected rows, it means that the changed data is synchronized to the target environment; if the affected data rows are zero, it means that the type of changed data does not exist in the target environment, then the update statement is converted into a new statement, and a new data row is added in the target environment; for example, if there is user Zhang San in the target environment, and Zhang San’s information is changed in the source environment, then the modified data row belonging to Zhang San’s information in the target environment will be affected as a change update; if there is no user Zhang San in the target environment, and Zhang San’s information is changed in the source environment, then the data row in the target environment will not change, then the update statement that modifies Zhang San is converted into a new statement and executed in the target environment; so as to add user Zhang San in the target environment.

[0105] Optionally, if the execution fails, the specific steps include:

[0106] If the execution fails due to a missing field error, find the missing field from the execution statement, remove it, and execute again;

[0107] If the execution fails due to a data primary key conflict, the current changed data is considered to have been synchronized to the target environment and the changed data is ignored;

[0108] If the execution fails due to content conflicts in some fields, clear the content of the conflicting fields in the script and execute again after clearing them;

[0109] If the reason for the execution failure is other than field missing error, data primary key conflict, and partial field content conflict, a separate message array is set to record the failure reason and retry the data execution. If the execution still fails after the preset number of retries, an alarm is issued and the alarm information is fed back to the source environment.

[0110] Optionally, other reasons may be environmental problems such as database connection exceptions and network exceptions. Such problems are not related to the program. After the environment is restored, the changed data can be re-executed through the retry mechanism. If the execution still fails after the preset number of retries, an alarm will be issued and the alarm information will be fed back to the source environment.

[0111] During the execution process, the database in the target environment can determine the source of the data modification based on specific data fields. If the data is modified in this environment, the data change is ignored to avoid data cyclic synchronization.

[0112] Specifically, the specific data field refers to the data source field, and each table that performs cross-environment data synchronization will have this field. The function of this field is to mark the source of data modification and avoid circular synchronization when there is two-way synchronization in some tables; two-way synchronization means: when area A changes the data of this table, it will be synchronized to the environment of school a, and when school a changes the data of this table, it will also be synchronized to area A; circular synchronization means: when the data is changed in area A, the database of area A will generate a change event, which will be pushed to school a at this time, and when school a executes this change operation, a change event will also be generated, which will be pushed to area A at this time. If there is no special processing, the database of area A will also execute the change event sent by school a, at this time area A will be updated, and the change event will be pushed to school a, and so on and so forth, and the cycle will never end; therefore, in this embodiment, the data source header field is used to determine in which environment the data is modified, and when the data is synchronized back to the data source, the change is ignored.

[0113] In summary, the cross-environment data synchronization method provided by the present application first obtains the change data in the source environment database, parses the obtained change data, and pushes the parsed change data to the source environment message array; after confirming the target environment that needs to be synchronized, obtain the data table of the change data in the synchronization table range configured by the target environment to determine whether the change data is valid, encrypt the valid change data and write it into the public network message queue; the target environment decrypts and converts it into an execution statement for execution to synchronize data across environments. The cross-environment data synchronization method provided by the present application can configure the information that needs to be synchronized across environments in terms of libraries, tables, and fields, which is relatively simple and orderly; it is non-invasive to the current business, will not affect the functions and performance of the business itself, and has high stability; compared with scheduled synchronization, it has higher timeliness, can push the changed data to the target environment in real time, and supports the template engine for data association and filtering; it has high accuracy and is suitable for large-scale promotion.

[0114] Embodiment 2

[0115] This embodiment provides a cross-environment data synchronization system, including:

[0116] A parsing module, used for acquiring change data in a source environment database, parsing the acquired change data, and pushing the parsed change data to a source environment message array;

[0117] A confirmation module, used for determining a target environment to which the changed data needs to be synchronized after the source environment data sending server receives a message in the source environment message array;

[0118] A writing module, used for obtaining the synchronization table range of the target environment configuration, determining whether the change data is valid according to the synchronization table range and the data table of the change data, and encrypting the valid change data and writing it into the public network message queue;

[0119] An execution module, used to obtain messages in a public network message queue and decrypt the data, convert the decrypted data into an execution statement, and execute it in a database in the target environment;

[0120] If the execution is successful, the data table in the target environment is modified accordingly to synchronize the changed data in the source environment to the target environment, and the synchronization results are fed back to the source environment;

[0121] If the execution fails, analyze the reason for the failure and issue an alarm, and feed the alarm information back to the source environment.

[0122] Optionally, the parsing module is specifically used for:

[0123] Initialize the connection parameters to establish a connection with MySQL in the source environment, and request binlog data from the MySQL Master server through the COM_BINLOG_DUMP command;

[0124] After receiving the request, the MySQL Master server transmits the binlog data to the binlog parser according to the MySQL master-slave replication protocol based on the file name and location provided in the binlog data;

[0125] The binlog parsing program receives the binlog data and parses it to decompose the database changes and data table changes in the change data;

[0126] The binlog parsing program receives the binlog data and parses it to decompose the database changes and data table changes in the change data, including:

[0127] Obtaining a file structure of the binlog data, wherein the file structure of the binlog data includes at least a file identifier, an event identifier, and corresponding event data;

[0128] Determine the event type according to the event identifier, obtain the data change preceding event of the changed data based on the event type and the event data, and obtain the changed database and data table information in the changed data according to the data change preceding event, wherein the data table information includes at least a data table name and a data table structure;

[0129] The number of rows affected by the changed data and the specific changed data rows are determined according to the changed database and data table information, and the events in the changed data rows are converted into a standard message format to obtain the database changes and data table changes in the changed data.

[0130] Optionally, the confirmation module is specifically used for:

[0131] Get all data tables of changed data, select some data tables for query in turn, and query the school ID field in all data tables through the associated query operation of the database;

[0132] Determine the school information to which the current change data belongs based on the queried school ID field, and judge whether the school has built a separate software environment;

[0133] If the school has built a software environment independently, the software environment of the school is the target environment that needs to be synchronized, and the region and platform to which the target environment belongs are determined based on the school information;

[0134] If the school does not build a separate software environment, the current change data will be ignored.

[0135] Optionally, the writing module is specifically used for:

[0136] Obtain the synchronization table range of the target environment from the configuration center, and determine whether there is a corresponding table in the synchronization table range for the data table of the changed data;

[0137] If a corresponding table exists, obtain the preceding associated table associated with the data table of the changed data;

[0138] Obtain the unique key of the target environment from the configuration center, encrypt the unique key of the target environment, the data table of the changed data, and the associated pre-association table, and write them into the public network message queue;

[0139] If the corresponding table does not exist, the current change data is ignored.

[0140] Optionally, the execution module is specifically used for:

[0141] If the execution is successful, the steps of modifying the data table in the target environment accordingly to synchronize the changed data in the source environment to the target environment include:

[0142] If the changed data is new data or deleted data, a new statement or a delete statement is executed in the database of the target environment to correspondingly add or delete data rows in the target environment data table, so as to synchronize the changed data in the source environment to the target environment;

[0143] If the changed data is updated data, execute the update statement in the database of the target environment to determine the number of data rows affected after the update statement is executed in the target environment;

[0144] If the number of affected data rows is not zero, the data rows in the target environment data table are updated accordingly to synchronize the changed data in the source environment to the target environment;

[0145] If the number of affected data rows is zero, the update statement is converted into a new statement, and the converted new statement is executed in the target environment, so that a data row corresponding to the updated data is added to the data table of the target environment, so as to synchronize the changed data in the source environment to the target environment;

[0146] If the execution fails, the steps of analyzing the cause of the execution failure and issuing an alarm, and feeding back the alarm information to the source environment include:

[0147] If the execution fails due to a missing field error, find the missing field from the execution statement, remove it, and execute again;

[0148] If the execution fails due to a data primary key conflict, the current changed data is considered to have been synchronized to the target environment and the changed data is ignored;

[0149] If the execution fails due to content conflicts in some fields, clear the content of the conflicting fields in the script and execute again after clearing them;

[0150] If the reason for the execution failure is other than field missing error, data primary key conflict, and partial field content conflict, a separate message array is set to record the failure reason and retry the data execution. If the execution still fails after the preset number of retries, an alarm is issued and the alarm information is fed back to the source environment.

[0151] Embodiment 3

[0152] This embodiment also provides a cross-environment data synchronization system. The difference between this embodiment and the cross-environment data synchronization system of the second embodiment is that: it also includes

[0153] The push module is used to obtain all data tables in the source environment and all data tables in the target environment respectively; determine the associated data tables and the unassociated data tables in the source environment and the target environment, wherein the associated data tables are pushed in the same thread, and the unassociated data tables are pushed in multiple parallel threads.

[0154] Embodiment 4

[0155] A fourth embodiment of the present invention provides a storage medium on which a computer program is stored. When the program is executed by a processor, the cross-environment data synchronization method as described above is implemented.

[0156] Embodiment 5

[0157] The present invention also provides a computer, see Figure 2 , shown is a computer in the fifth embodiment of the present invention, including a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20, and the processor 20 implements the above-mentioned cross-environment data synchronization method when executing the computer program 30.

[0158] The memory 10 includes at least one type of storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10 may be an internal storage unit of a computer, such as a hard disk of the computer. In other embodiments, the memory 10 may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Further, the memory 10 may also include both an internal storage unit of the computer and an external storage device. The memory 10 may be used not only to store application software and various types of data installed in the computer, but also to temporarily store data that has been output or is to be output.

[0159] Among them, in some embodiments, the processor 20 can be an electronic control unit (Electronic Control Unit, abbreviated as ECU, also known as a vehicle computer), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 10 or process data, such as executing access restriction programs, etc.

[0160] It should be pointed out that Figure 2 The structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0161] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0162] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0163] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0164] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A cross-environment data synchronization method, characterized in that: include; Acquire the change data in the source environment database, parse the acquired change data, and push the parsed change data to the source environment message array; After receiving the message in the source environment message array, the source environment data sending server determines the target environment where the changed data needs to be synchronized; Obtain the synchronization table range of the target environment configuration, determine whether the change data is valid according to the synchronization table range and the data table of the change data, encrypt the valid change data and write it into the public network message queue; Obtaining messages in a public network message queue and decrypting the data, converting the decrypted data into execution statements, and executing them in a database in the target environment; If the execution is successful, the data table in the target environment is modified accordingly to synchronize the changed data in the source environment to the target environment, and the synchronization results are fed back to the source environment; If the execution fails, analyze the reason for the failure and issue an alarm, and feed the alarm information back to the source environment.

2. The cross-environment data synchronization method according to claim 1, characterized in that: The step of parsing and obtaining the change data comprises: Initialize the connection parameters to establish a connection with MySQL in the source environment, and request binlog data from the MySQLMaster server through the COM_BINLOG_DUMP command; After receiving the request, the MySQL Master server transmits the binlog data to the binlog parser according to the MySQL master-slave replication protocol based on the file name and location provided in the binlog data; After receiving the binlog data, the binlog parsing program parses the data to decompose the database changes and data table changes in the change data.

3. The cross-environment data synchronization method according to claim 2, characterized in that: The binlog parsing program receives the binlog data and parses it to decompose the database changes and data table changes in the change data, including: Obtaining a file structure of the binlog data, wherein the file structure of the binlog data includes at least a file identifier, an event identifier, and corresponding event data; Determine the event type according to the event identifier, obtain the data change preceding event of the changed data based on the event type and the event data, and obtain the changed database and data table information in the changed data according to the data change preceding event, wherein the data table information includes at least a data table name and a data table structure; The number of rows affected by the changed data and the specific changed data rows are determined according to the changed database and data table information, and the events in the changed data rows are converted into a standard message format to obtain the database changes and data table changes in the changed data.

4. The cross-environment data synchronization method according to claim 3, characterized in that: After parsing the acquired change data, the cross-environment data synchronization method further includes: Get all data tables in the source environment and all data tables in the target environment respectively; Determine associated data tables and unassociated data tables in the source environment and the target environment, wherein associated data tables push data in the same thread, and unassociated data tables push data in multiple parallel threads.

5. The cross-environment data synchronization method according to claim 1, characterized in that: After the source environment data sending server receives the message in the source environment message array, the step of determining the target environment where the changed data needs to be synchronized includes: Get all data tables of changed data, select some data tables for query in turn, and query the school ID field in all data tables through the associated query operation of the database; Determine the school information to which the current change data belongs based on the queried school ID field, and judge whether the school has built a separate software environment; If the school has built a software environment independently, the software environment of the school is the target environment that needs to be synchronized, and the region and platform to which the target environment belongs are determined based on the school information; If the school does not build a separate software environment, the current change data will be ignored.

6. The cross-environment data synchronization method according to claim 1, characterized in that: The steps of obtaining the synchronization table range of the target environment configuration, determining whether the changed data is valid according to the synchronization table range and the data table of the changed data, and encrypting the valid changed data and writing it into the public network message queue include: Obtain the synchronization table range of the target environment from the configuration center, and determine whether there is a corresponding table in the synchronization table range for the data table of the changed data; If a corresponding table exists, obtain the preceding associated table associated with the data table of the changed data; Obtain the unique key of the target environment from the configuration center, encrypt the unique key of the target environment, the data table of the changed data, and the associated pre-association table, and write them into the public network message queue; If the corresponding table does not exist, the current change data is ignored.

7. The cross-environment data synchronization method according to claim 1, characterized in that: The changed data includes one or more of newly added data, deleted data and updated data; If the execution is successful, the steps of modifying the data table in the target environment accordingly to synchronize the changed data in the source environment to the target environment include: If the changed data is new data or deleted data, a new statement or a delete statement is executed in the database of the target environment to correspondingly add or delete data rows in the target environment data table, so as to synchronize the changed data in the source environment to the target environment; If the changed data is updated data, execute the update statement in the database of the target environment to determine the number of data rows affected after the update statement is executed in the target environment; If the number of affected data rows is not zero, the data rows in the target environment data table are updated accordingly to synchronize the changed data in the source environment to the target environment; If the number of affected data rows is zero, the update statement is converted into a new statement, and the converted new statement is executed in the target environment, so that a data row corresponding to the updated data is added to the data table of the target environment, so as to synchronize the changed data in the source environment to the target environment; If the execution fails, the steps of analyzing the cause of the execution failure and issuing an alarm, and feeding back the alarm information to the source environment include: If the execution fails due to a missing field error, find the missing field from the execution statement, remove it, and execute again; If the execution fails due to a data primary key conflict, the current changed data is considered to have been synchronized to the target environment and the changed data is ignored; If the execution fails due to content conflicts in some fields, clear the content of the conflicting fields in the script and execute again after clearing them; If the reason for the execution failure is other than field missing error, data primary key conflict, and partial field content conflict, a separate message array is set to record the failure reason and retry the data execution. If the execution still fails after the preset number of retries, an alarm is issued and the alarm information is fed back to the source environment.

8. A cross-environment data synchronization system, characterized in that: include: A parsing module, used for acquiring change data in a source environment database, parsing the acquired change data, and pushing the parsed change data to a source environment message array; A confirmation module, used for determining a target environment to which the changed data needs to be synchronized after the source environment data sending server receives a message in the source environment message array; A writing module, used for obtaining the synchronization table range of the target environment configuration, determining whether the change data is valid according to the synchronization table range and the data table of the change data, and encrypting the valid change data and writing it into the public network message queue; An execution module, used to obtain messages in a public network message queue and decrypt the data, convert the decrypted data into an execution statement, and execute it in a database in the target environment; If the execution is successful, the data table in the target environment is modified accordingly to synchronize the changed data in the source environment to the target environment, and the synchronization results are fed back to the source environment; If the execution fails, analyze the reason for the failure and issue an alarm, and feed the alarm information back to the source environment.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the cross-environment data synchronization method as described in any one of claims 1 to 7 is implemented.

10. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the cross-environment data synchronization method as described in any one of claims 1 to 7 is implemented.