Data synchronization method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data synchronization method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps that when it is detected that a first database executes a first data change operation, a first distributed lock is set for the first data change operation in a lock management database; when it is determined that the first distributed lock is set successfully, a second data change operation corresponding to a second database is determined according to the first data change operation; and setting a second distributed lock for the second data change operation in the lock management database, and controlling the second database to execute the second data change operation when determining that the second distributed lock is successfully set. According to the technical scheme provided by the embodiment of the invention, the anti-cyclic synchronization mechanism is realized through the distributed lock, the storage pressure and calculation burden of the database are reduced, the performance loss of the anti-cyclic synchronization mechanism is reduced, and meanwhile, the data synchronization process is simplified, so that the data synchronization efficiency and performance are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art

[0002] In the process of migrating data from the original area to the target area, first use the data integration tool to synchronize all the data of the original database to the target database to preliminarily align the data of the original database and the target database, where the original database is the database in the original area and the target database is the database in the target area; then divert user requests from the original area to the target area in proportion, and gradually increase the diversion ratio until 100% of user requests are diverted to the target area.

[0003] In the process of gradual diversion, incremental synchronization is enabled, that is, data synchronization log tables are created in the original database and the target database respectively, and a joint unique index is created for the two data synchronization log tables. Then, data change operations of the original database (or target database) are captured in real time, and Canal is used to synchronize the data change operations to the target database (or original database). At the same time, the data synchronization log table and the joint unique index are used to prevent circular synchronization, that is, to prevent essentially the same data change operations from being synchronized multiple times between the original database and the target database.

[0004] However, in the above technical solution, the anti-circular synchronization mechanism is implemented through the data synchronization log table and the joint unique index, which increases the burden on the original database and the target database and increases performance loss, resulting in low efficiency of data synchronization. Summary of the invention

[0005] The embodiments of the present application provide a data synchronization method, device, electronic device and storage medium, which implement a data synchronization function to solve the problem in the prior art that the anti-circular synchronization mechanism increases the burden on the database and causes performance loss.

[0006] In a first aspect, an embodiment of the present application provides a data synchronization method, the method comprising:

[0007] When it is detected that the first database executes a first data change operation, setting a first distributed lock for the first data change operation in a lock management database;

[0008] When it is determined that the first distributed lock is set successfully, determining a second data change operation corresponding to the second database according to the first data change operation;

[0009] A second distributed lock is set in the lock management database for the second data change operation, and when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation.

[0010] In an embodiment of the present application, when it is detected that the first database executes a first data change operation, a first distributed lock can be set for the first data change operation in the lock management database, and then when it is determined that the first distributed lock is set successfully, the second data change operation corresponding to the second database is determined according to the first data change operation, and then a second distributed lock is set for the second data change operation in the lock management database, and when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation, thereby realizing the data synchronization function. In the above technical scheme, when it is detected that the first database executes the first data change operation, the data synchronization process will be executed only when it is determined that the first distributed lock is set successfully, so that an anti-circular synchronization mechanism can be implemented, that is, essentially the same data change operation is synchronized only once between the first database and the second database, thereby effectively preventing circular synchronization; the prior art implements an anti-circular synchronization mechanism through a data synchronization log table and a joint unique index, which requires the creation and maintenance of additional data structures in the original database and the target database respectively, occupying the storage resources and computing resources of the database, while the above technical scheme implements an anti-circular synchronization mechanism through a distributed lock, which only requires the setting and management of locks in the lock management database, avoiding the creation and maintenance of complex data synchronization log tables and joint unique indexes in the first database and the second database, thereby reducing the storage pressure and computing burden of the first database and the second database, and reducing the performance loss of the anti-circular synchronization mechanism, while simplifying the data synchronization process, thereby improving the efficiency and performance of data synchronization.

[0011] In a second aspect, an embodiment of the present application provides a data synchronization device, the device comprising:

[0012] A first setting module is used to set a first distributed lock for the first data change operation in a lock management database when it is detected that the first database performs a first data change operation;

[0013] A determination module, configured to determine, when determining that the first distributed lock is successfully set, a second data change operation corresponding to the second database according to the first data change operation;

[0014] The second setting module is used to set a second distributed lock for the second data change operation in the lock management database, and when it is determined that the second distributed lock is set successfully, control the second database to execute the second data change operation.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0016] at least one processor; and a memory communicatively coupled to the at least one processor;

[0017] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the data synchronization method of any embodiment of the present application.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, a data synchronization method as in any embodiment of the present application is implemented.

[0019] The description of the second, third and fourth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects described in the second, third and fourth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0020] In this application, the name of the above-mentioned data synchronization device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

[0021] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0024] Figure 2 This is an example diagram of data synchronization between the original area and the target area;

[0025] Figure 3a is an example diagram of a data synchronization method provided in an embodiment of the present application;

[0026] Figure 3b is another example diagram of the data synchronization method provided in an embodiment of the present application;

[0027] Figure 4 is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0028] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", "target", and "original" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having", and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0031] Figure 1 It is a flow chart of the data synchronization method provided by an embodiment of the present application. This embodiment can be applied to the scenario of synchronizing the data in the original database and the target database in the process of migrating the data in the original area to the target area. A data synchronization method provided in this embodiment can be executed by the data synchronization device provided by an embodiment of the present application, and the device can be implemented by software and / or hardware. In a specific embodiment, the data synchronization device can be integrated in an electronic device, for example, the electronic device can be a computer or a server. The execution subject of this method can be an electronic device. The following is a detailed description taking the migration of data from the original area to the target area as an example, and the electronic device can provide data synchronization services by building a data integration platform (Apache SeaTunnel).

[0032] In the process of smoothly migrating data from the original area to the target area, you can use the data integration tool to synchronize all the data in the original database to the target database, and then divert user requests from the original area to the target area in proportion, and gradually increase the diversion ratio until 100% of user requests are diverted to the target area. At the same time, during the gradual diversion process, enable incremental synchronization. If the user request is diverted to the original area and the user request involves data change operations on the original database, such as insert, update, and delete, the data change operations of the original database are synchronized to the target database; if the user request is diverted to the target area and the user request involves data change operations on the target database, the data change operations of the target database are synchronized to the original database.

[0033] For example, Figure 2 The following is an example of data synchronization between the original area and the target area. Figure 2 The load balancing server in the system diverts 50% of the user requests to the application server 1 in the original area, and diverts the remaining 50% of the user requests to the application server 2 in the target area. When the user request 1 is diverted to the application server 1 in the original area, and the user request 1 is to insert data in the original database, the electronic device can synchronize the insert operation of the original database to the target database; when the user request 2 is diverted to the application server 2 in the target area, and the user request 2 is to update data in the target database, the electronic device can synchronize the update operation of the target database to the original database.

[0034] See also Figure 1 The data synchronization method of this embodiment includes but is not limited to the following steps:

[0035] S110: When it is detected that the first database executes a first data change operation, a first distributed lock is set in a lock management database for the first data change operation.

[0036] The first database is a database where data change operations occur; illustratively, the first database may be an original database in an original area, or a target database in a target area.

[0037] The first data change operation is an operation for changing the data in the first database, and may change the data in the first database; illustratively, the first data change operation may include inserting, updating, deleting, and the like.

[0038] The lock management database is a database specifically used to manage distributed locks, and is used to record relevant information of distributed locks, such as the key and value of distributed locks; the key of a distributed lock is the unique identifier of a distributed lock; the value of a distributed lock is used to store relevant information of the data change operation corresponding to the distributed lock. Exemplarily, the lock management database can be a Remote Dictionary Server (Redis) database.

[0039] The first distributed lock is a lock created for the first data change operation when it is detected that the first database executes the first data change operation. The lock is used to determine whether the first data change operation of the first database needs to be synchronized to the second database to prevent multiple synchronization of the first data change operation between the first database and the second database. If the first distributed lock is set successfully, it indicates that the first data change operation of the first database has not been synchronized to the second database; if the first distributed lock fails to be set, it indicates that the first data change operation of the first database has been synchronized to the second database.

[0040] The second database is a database that needs to maintain data consistency with the first database; illustratively, when the first database is the original database, the second database is the target database; when the first database is the target database, the second database is the original database. It should be noted that the database types of the first database and the second database can be the same or different.

[0041] Specifically, during the data synchronization process, the data in the first database can be synchronized to the second database in batches to align the data in the first database and the second database, and then the first database is detected in real time whether the first data change operation is performed, that is, in one implementation, a database trigger can be created for the first database, and the database trigger is used to detect in real time whether the first database performs the first data change operation. The database trigger is used to detect in real time whether the first database performs the data change operation.

[0042] In another implementation, the log file of the first database can be detected in real time to obtain the detection result, and whether the first database performs the first data change operation can be determined based on the detection result, that is, the log file of the first database can be detected in real time, and when a new log record is added to the log file, the log record is determined as the detection result, and then the detection result is parsed to obtain the operation corresponding to the detection result. If the operation corresponding to the detection result belongs to the preset operation type, it can be determined that the first database has performed the first data change operation; if the operation corresponding to the detection result does not belong to the preset operation type, it can be determined that the first database has not performed the first data change operation. By detecting the log file, the data change operation can be quickly detected, thereby improving the timeliness of data synchronization. Among them, the detection result is the basis for judging whether the first database performs the first data change operation; the preset operation type can include insert, update and delete, etc.

[0043] Afterwards, when it is detected that the first database executes the first data change operation, the first data change operation can be captured, for example, the newly added log record in the log file can be parsed to obtain the change operation record presented in the form of a structured query language (SQL) statement, so as to obtain the first data change operation; then a first distributed lock is set for the first data change operation in the lock management database, that is, the key of the first distributed lock and the value of the first distributed lock can be determined according to the first data change operation, and then the first distributed lock is further set in the lock management database for the first data change operation according to the key of the first distributed lock and the value of the first distributed lock.

[0044] Furthermore, a first distributed lock is set for the first data change operation in the lock management database, including Sa1-Sa3:

[0045] Sa1. Analyze the first data change operation to obtain change information corresponding to the first data change operation.

[0046] Among them, the change information may include a first change table, a first change type, a first change field and first change field data; the first change table is a database table involved in the first data change operation; the first change type is a specific type of the first data change operation, which may be insert, update or delete, etc.; the first change field is a specific field that is changed in the first change table when the first data change operation is performed in the first database; the first change field data is the new data (for insert and update operations) or old data (for delete operations) corresponding to the first change field.

[0047] Specifically, the first data change operation can be parsed to obtain change information. For example, the SQL statement corresponding to the first data change operation can be analyzed using a string processing function or a regular expression, and the table name part can be extracted to obtain the first change table. Then, the keywords in the beginning part are extracted from the SQL statement to obtain the change operation name, and the first change type is determined according to the change operation name. For example, if the change operation name is INSERT, the first change type is insert; if the change operation name is UPDATE, the first change type is update; if the change operation name is DELETE, the first change type is delete; then, the column name is extracted from the SQL statement to obtain the first change field, and the value is extracted from the SQL statement to obtain the first change field data. If there are multiple first change fields, the mapping relationship between the first change field and the first change field data can be determined.

[0048] Exemplarily, if the SQL statement corresponding to the first data change operation is INSERT INTO user_1(field1_1,field1_2)VALUES(value1,value2), then the first change table is user_1, the first change type is insert, the first change field 1 is field1_1, the first change field 2 is field1_2, the first change field data corresponding to the first change field 1 is value1, and the first change field data corresponding to the first change field 2 is value2.

[0049] Sa2. Determine the value of the first distributed lock and the key of the first distributed lock according to the change information.

[0050] Specifically, after obtaining the change information corresponding to the first data change operation, the change information can be analyzed, and then the value of the first distributed lock can be determined based on the first change field and the first change field data, and the key of the first distributed lock can be determined based on the first change table, the first change type and the value of the first distributed lock.

[0051] Further, the value of the first distributed lock is determined based on the first change field and the first change field data, including Sb1-Sb3:

[0052] Sb1. Sort the first changed field based on a preset sorting rule to obtain a sorting result.

[0053] The preset sorting rule is a preset sorting rule used to sort the first changed field. Exemplarily, the preset sorting rule may be sorting in ascending order of the dictionary.

[0054] The sorting result is a sequence obtained by sorting the first changed field according to a preset sorting rule.

[0055] Specifically, when there are multiple first change fields, the multiple first change fields can be sorted according to the preset sorting rule. For example, if the preset sorting rule is to sort in ascending dictionary order, the first change field 1 is username, the first change field 2 is email, and the first change field 3 is phone, then the sorting result is ["email", "phone", "username"].

[0056] Sb2. Based on the sorting result, encapsulate the first changed field and the corresponding first changed field data in a preset format to obtain changed data.

[0057] The preset format is a pre-set format for specifying how to combine the sorted first change field and the corresponding first change field data. Exemplarily, the preset format is a JS key-value pair data (JavaScript Object Notation, JSON) format.

[0058] The change data is a result obtained by encapsulating the sorted first change fields and the corresponding first change field data according to a preset format.

[0059] Exemplarily, if the preset format is JSON format, the first change field data corresponding to the first change field 1 (i.e., username) is value1, the first change field data corresponding to the first change field 2 (i.e., email) is value2, and the first change field data corresponding to the first change field 3 (i.e., phone) is value3, and the sorting result of the first change field is ["email", "phone", "username"], then the first change field and the corresponding first change field data are encapsulated according to the preset format, and the obtained change data is {"email":"value2","phone":"value3","username":"value1"}.

[0060] Sb3. Perform a hash operation on the changed data to obtain the value of the first distributed lock.

[0061] Specifically, after obtaining the changed data, a hash algorithm may be used to perform a hash operation on the changed data. For example, an SM3 algorithm may be used to perform a hash operation on the changed data to obtain a hash summary, and the hash summary is determined as the value of the first distributed lock.

[0062] In an embodiment of the present application, sorting the first change field based on a preset sorting rule can ensure that the same data change operation corresponds to the same sorting result, thereby avoiding the generation of different values ​​due to different field orders; the sorted first change field and the corresponding first change field data are encapsulated in accordance with a preset format, and the organization of the value is standardized, so that it can be determined that the same data change operation corresponds to the same change data; a hash operation is performed on the change data to obtain the value of the first distributed lock, which ensures the privacy and security of the data, while ensuring that different data change operations correspond to different values, and ensuring that the same data change operation corresponds to the same value, thereby providing a judgment basis for subsequent prevention of circular synchronization.

[0063] After obtaining the value of the first distributed lock, the first change table, the first change type and the value of the first distributed lock can be spliced ​​to obtain the key of the first distributed lock. At this time, the first change table can be represented by the table name or the table name with a database prefix. By generating the key of the first distributed lock by splicing, the uniqueness of the key can be ensured, so that the key of each distributed lock can accurately correspond to a specific data change operation, thereby providing a basis for determining whether the first distributed lock is set successfully, thereby providing a basis for preventing circular synchronization in the future.

[0064] Exemplarily, if the first change table is table1, the first change type is insert, and the value of the first distributed lock is sign, then the key of the first distributed lock is table1_INSERT_sign; or, if the first change table is db.table1, the first change type is insert, and the value of the first distributed lock is sign, then the key of the first distributed lock is db.table1_INSERT_sign.

[0065] Sa3. Based on the value of the first distributed lock and the key of the first distributed lock, set the first distributed lock in the lock management database.

[0066] Specifically, after obtaining the value of the first distributed lock and the key of the first distributed lock, the first distributed lock can be set in the lock management database, that is, the lock management database can be connected, and then the lock setting command provided by the lock management database, such as the SET command in Redis, is used to set the first distributed lock based on the value of the first distributed lock and the key of the first distributed lock, and determine whether the first distributed lock is set successfully. Specifically, when there is a lock with the same key as the first distributed lock in the lock management database, the first distributed lock fails to be set; when there is no lock with the same key as the first distributed lock in the lock management database, the first distributed lock is set successfully. It should be noted that other reasons such as network problems or failure of the server of the lock management database may also cause the failure of setting the first distributed lock.

[0067] If the first distributed lock is successfully set in the lock management database, it indicates that the same lock as the first distributed lock does not exist in the lock management database, that is, the first data change operation of the first database has not been synchronized to the second database. At this time, data synchronization is required, that is, S120 can be executed; if the first distributed lock fails to be set in the lock management database, the reason for the setting failure can be determined, and when the reason for the setting failure is that the same lock as the first distributed lock exists in the lock management database, it indicates that the first data change operation of the first database has been synchronized to the second database. At this time, there is no need to synchronize data again, and the first database can continue to be detected in real time whether the data change operation is executed; when the reason for the setting failure is other reasons, the first distributed lock for the first data change operation in the lock management database can be repeatedly set until the first distributed lock is successfully set.

[0068] In the embodiment of the present application, setting the first distributed lock according to the change information corresponding to the first data change operation can ensure the uniqueness of the first distributed lock, provide a basis for determining the subsequent prevention of circular synchronization, and thus effectively prevent circular synchronization.

[0069] S120. When it is determined that the first distributed lock is set successfully, determine a second data change operation corresponding to the second database according to the first data change operation.

[0070] Among them, the second data change operation is an operation determined according to the first data change operation and needs to be performed in the second database, which can change the data in the second database; and the change type of the second data change operation is the same as the change type of the first data change operation. For example, if the first data change operation is to insert a new order record in the first database, the second data change operation is to perform a corresponding insert operation in the second database to ensure that the order data in the two databases remain consistent.

[0071] Specifically, when it is determined that the first distributed lock is set successfully, it indicates that the second database has not yet synchronized the first data change operation. At this time, the second data change operation corresponding to the second database can be determined according to the first data change operation, that is, based on the data mapping relationship between the first database and the second database, and the change information corresponding to the first data change operation, the change information corresponding to the second data change operation can be determined, and the second data change operation can be generated according to the change information corresponding to the second data change operation.

[0072] Furthermore, a second data change operation corresponding to the second database is determined according to the first data change operation, including Sc1-Sc2:

[0073] Sc1. Based on the first change table and the first change field, a preset data mapping relationship is queried to obtain a second change table corresponding to the first change table and a second change field corresponding to the first change field.

[0074] The preset data mapping relationship is a pre-set mapping relationship used to describe the corresponding relationship between the data structure and the data content between the first database and the second database, and may include a table mapping relationship and a field mapping relationship.

[0075] The second change table is a table in the second database corresponding to the first change table. The second change field is a field in the second change table corresponding to the first change field in the first change table.

[0076] Specifically, the table mapping relationship in the preset data mapping relationship can be queried based on the first change table to obtain the second change table corresponding to the first change table, and then the field mapping relationship in the preset data mapping relationship can be queried based on the first change field to obtain the second change field corresponding to the first change field.

[0077] Sc2. Determine a second data change operation according to the first change type, the second change table, the second change field, and the first change field data.

[0078] Specifically, after obtaining the second change table and the second change field, the second change type, that is, the specific operation type of the second data change operation, can be determined according to the first change type. At this time, the second change type is the same as the first change type. Then, the first change field data is determined as the second change field data, and based on the mapping relationship between the first change field and the first change field data, and the mapping relationship between the first change field and the second change field data, the mapping relationship between the second change field and the second change field data is determined. Then, according to the second change type, the second change table, the second change field and the second change field data, the second data change operation is generated. At this time, the second data change operation can be expressed in the form of an SQL statement.

[0079] Exemplarily, if the SQL statement corresponding to the first data change operation is INSERT INTO user_1(field1_1,field1_2)VALUES(value1,value2), in the preset data mapping relationship, user_1 corresponds to user_2, field1_1 in user_1 corresponds to field2_1 in user_2, and field1_2 in user_1 corresponds to field2_2 in user_2, then the second change table is user_2, the second change type is insert, the second change field 1 is field2_1, the second change field 2 is field2_2, the second change field data corresponding to the second change field 1 is value1, and the second change field data corresponding to the second change field 2 is value2, and the SQL statement corresponding to the second data change operation is INSERT INTO user_2(field2_1,field2_2)VALUES(value1,value2).

[0080] In an embodiment of the present application, by presetting the data mapping relationship, errors in the data conversion process across databases can be effectively avoided, thereby improving the accuracy and efficiency of determining the second data change operation, and allowing the database type or data structure of the first database and the second database to be different, thereby being suitable for data synchronization between various homogeneous databases or heterogeneous databases, thereby improving the flexibility of data synchronization.

[0081] S130: Setting a second distributed lock for the second data change operation in the lock management database, and when it is determined that the second distributed lock is set successfully, controlling the second database to execute the second data change operation.

[0082] Among them, the second distributed lock is a lock created for the second data change operation, which is used to determine whether to control the second database to perform the second data change operation; at the same time, after the second database executes the second data change operation, the second distributed lock can also be used to determine whether it is necessary to synchronize the second data change operation of the second database to the first database, so as to prevent multiple synchronization of the second data change operation between the first database and the second database. At this time, the second data change operation is essentially the same as the first data change operation.

[0083] Specifically, after obtaining the second data change operation, a second distributed lock can be set for the second data change operation in the lock management database, that is, the value of the second distributed lock can be determined based on the second change field and the second change field data, and the key of the second distributed lock can be determined based on the second change table, the second change type and the value of the second distributed lock, and then based on the value of the second distributed lock and the key of the second distributed lock, the second distributed lock is set in the lock management database, and it is determined whether the second distributed lock is set successfully. The specific implementation details are the same as S110, and reference can be made to the description of S110, which will not be repeated here.

[0084] Since the second database did not execute the second data change operation when setting the second distributed lock, the reason for the failure of setting the second distributed lock is not that the same lock as the second distributed lock exists in the lock management database, but other reasons. Therefore, when it is determined that the setting of the second distributed lock has failed, the second distributed lock is repeatedly set in the lock management database for the second data change operation until the second distributed lock is successfully set. This provides a basis for determining whether to synchronize the second data change operation to the first database when it is detected that the second database executes the second data change operation, thereby effectively preventing circular synchronization.

[0085] If it is determined that the second distributed lock is set successfully, the second database can be controlled to execute the second data change operation, for example, the second database can be controlled to execute the SQL statement corresponding to the second data change operation to synchronize the data in the second database with the first database.

[0086] Optionally, the electronic device can detect the log file of the second database in real time, and when it detects that the second database executes the second data change operation, it can set a corresponding distributed lock for the second data change operation in the lock management database, and record it as a third distributed lock. However, at this time, the key of the third distributed lock is the same as the key of the second distributed lock, that is, the same lock of the third distributed lock already exists in the lock management database. Therefore, it can be determined that the setting of the third distributed lock has failed, that is, the second data change operation of the second database has been synchronized to the first database, and there is no need to synchronize the data again. At this time, it is possible to continue to detect in real time whether the second database executes the data change operation.

[0087] For example, Figure 3a FIG. 1 is an example diagram of a data synchronization method provided by an embodiment of the present application. Figure 3aThe first database in the data is a database in the original area, and the second database is a database in the target area. The data synchronization process can be expressed as follows: when the user request is diverted to the application server 1, the application server 1 controls the first database to execute the first data change operation corresponding to the user request. At this time, the electronic device can detect the first database in real time, and when it is detected that the first database executes the first data change operation, a first distributed lock is set for the first data change operation in the lock management database, and then it is determined whether the first distributed lock is set successfully. If it is determined that the first distributed lock is set successfully, the second data change operation corresponding to the second database is determined according to the first data change operation, and a second distributed lock is set for the second data change operation in the lock management database, and then when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation; if it is determined that the first distributed lock fails to be set, the data synchronization process is terminated to prevent circular synchronization.

[0088] For example, Figure 3b FIG. 2 is another example diagram of the data synchronization method provided by an embodiment of the present application. Figure 3b The first database in is the database in the target area, and the second database is the database in the original area. The data synchronization process can be expressed as follows: when the user request is diverted to the application server 2, the application server 2 controls the first database to execute the first data change operation corresponding to the user request. At this time, the electronic device can detect the first database in real time, and when it is detected that the first database executes the first data change operation, a first distributed lock is set for the first data change operation in the lock management database. If it is determined that the first distributed lock is set successfully, the second data change operation corresponding to the second database is determined according to the first data change operation, and a second distributed lock is set for the second data change operation in the lock management database. Then, when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation; if it is determined that the first distributed lock fails to be set, the data synchronization process is terminated to prevent circular synchronization.

[0089] The technical solution of the embodiment of the present application is that when it is detected that the first database executes a first data change operation, a first distributed lock can be set for the first data change operation in the lock management database, and then when it is determined that the first distributed lock is set successfully, the second data change operation corresponding to the second database is determined according to the first data change operation, and then a second distributed lock is set for the second data change operation in the lock management database, and when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation, thereby realizing the data synchronization function. In the above technical scheme, when it is detected that the first database executes the first data change operation, the data synchronization process will be executed only when it is determined that the first distributed lock is set successfully, so that an anti-circular synchronization mechanism can be implemented, that is, essentially the same data change operation is synchronized only once between the first database and the second database, thereby effectively preventing circular synchronization; the prior art implements an anti-circular synchronization mechanism through a data synchronization log table and a joint unique index, which requires the creation and maintenance of additional data structures in the original database and the target database respectively, occupying the storage resources and computing resources of the database, while the above technical scheme implements an anti-circular synchronization mechanism through a distributed lock, which only requires the setting and management of locks in the lock management database, avoiding the creation and maintenance of complex data synchronization log tables and joint unique indexes in the first database and the second database, thereby reducing the storage pressure and computing burden of the first database and the second database, and reducing the performance loss of the anti-circular synchronization mechanism, while simplifying the data synchronization process, thereby improving the efficiency and performance of data synchronization.

[0090] Figure 4 is a structural diagram of a data synchronization device provided in an embodiment of the present application, referring to Figure 4 , the data synchronization device may include:

[0091] A first setting module 410 is used to set a first distributed lock for the first data change operation in a lock management database when it is detected that the first database performs a first data change operation;

[0092] A determination module 420, configured to determine, when it is determined that the first distributed lock is set successfully, a second data change operation corresponding to the second database according to the first data change operation;

[0093] The second setting module 430 is used to set a second distributed lock for the second data change operation in the lock management database, and when it is determined that the second distributed lock is set successfully, control the second database to execute the second data change operation.

[0094] In one embodiment, the first setting module 410 is specifically used to:

[0095] Parsing the first data change operation to obtain change information corresponding to the first data change operation;

[0096] Determine a value of the first distributed lock and a key of the first distributed lock according to the change information;

[0097] The first distributed lock is set in a lock management database based on the value of the first distributed lock and the key of the first distributed lock.

[0098] In one embodiment, the change information includes a first change table, a first change type, a first change field, and first change field data. The first setting module 410 determines the value of the first distributed lock and the key of the first distributed lock according to the change information, including:

[0099] Determine a value of the first distributed lock based on the first change field and the first change field data;

[0100] A key of the first distributed lock is determined based on the first change table, the first change type, and the value of the first distributed lock.

[0101] In one embodiment, the first setting module 410 determines the value of the first distributed lock based on the first change field and the first change field data, including:

[0102] Sorting the first changed field based on a preset sorting rule to obtain a sorting result;

[0103] Based on the sorting result, encapsulating the first changed field and the corresponding first changed field data according to a preset format to obtain changed data;

[0104] Perform a hash operation on the changed data to obtain the value of the first distributed lock.

[0105] In one embodiment, the first setting module 410 determines the key of the first distributed lock based on the first change table, the first change type and the value of the first distributed lock, including:

[0106] The first change table, the first change type, and the value of the first distributed lock are concatenated to obtain the key of the first distributed lock.

[0107] In one embodiment, the determination module 420 is specifically configured to:

[0108] Based on the first change table and the first change field, a preset data mapping relationship is searched to obtain a second change table corresponding to the first change table and a second change field corresponding to the first change field;

[0109] A second data change operation is determined according to the first change type, the second change table, the second change field, and the first change field data.

[0110] In one embodiment, the data synchronization device further includes a detection module, and the detection module is specifically used to:

[0111] Before detecting that the first database executes the first data change operation, detecting the log file of the first database in real time to obtain a detection result;

[0112] Determine whether the first database executes the first data change operation according to the detection result.

[0113] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0114] The data synchronization device provided in this embodiment can be applied to the data synchronization method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0115] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. Figure 5 A block diagram of an exemplary electronic device 11 suitable for implementing the embodiments of the present application is shown. Figure 5 The electronic device 11 shown is only an example and should not bring any limitation to the function and scope of use of the present embodiment.

[0116] like Figure 5 As shown, the electronic device 11 is in the form of a general purpose computing electronic device. The components of the electronic device 11 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0117] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0118] The electronic device 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 11, including volatile and non-volatile media, removable and non-removable media.

[0119] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0120] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0121] The electronic device 11 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 11, and / or may communicate with any device that enables the electronic device 11 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 11 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20.

[0122] like Figure 5 As shown, the network adapter 20 communicates with other modules of the electronic device 11 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] The processing unit 16 executes various functional applications and page displays by running the programs stored in the system memory 28, for example, implementing a data synchronization method provided in this embodiment, the method comprising:

[0124] When it is detected that the first database executes a first data change operation, setting a first distributed lock for the first data change operation in a lock management database;

[0125] When it is determined that the first distributed lock is set successfully, determining a second data change operation corresponding to the second database according to the first data change operation;

[0126] A second distributed lock is set in the lock management database for the second data change operation, and when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation.

[0127] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data synchronization method provided in any embodiment of the present application.

[0128] The present application embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, a data synchronization method provided in the present application embodiment is implemented, for example, and the method includes:

[0129] When it is detected that the first database executes a first data change operation, setting a first distributed lock for the first data change operation in a lock management database;

[0130] When it is determined that the first distributed lock is set successfully, determining a second data change operation corresponding to the second database according to the first data change operation;

[0131] A second distributed lock is set in the lock management database for the second data change operation, and when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation.

[0132] The computer storage medium of this embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by instruction execution systems, devices or devices or used in combination with them.

[0133] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0134] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0135] Computer program code for performing the operation of the present application can be written in one or more programming languages ​​or combinations thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0136] It should be understood by those skilled in the art that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0137] In addition, the acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0138] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the inventive concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: When detecting that the first database executes a first data change operation, setting a first distributed lock for the first data change operation in a lock management database; When it is determined that the first distributed lock is set successfully, determining a second data change operation corresponding to the second database according to the first data change operation; A second distributed lock is set in the lock management database for the second data change operation, and when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation.

2. The data synchronization method according to claim 1, characterized in that: The step of setting a first distributed lock for the first data change operation in a lock management database includes: Parsing the first data change operation to obtain change information corresponding to the first data change operation; Determine a value of the first distributed lock and a key of the first distributed lock according to the change information; The first distributed lock is set in the lock management database based on the value of the first distributed lock and the key of the first distributed lock.

3. The data synchronization method according to claim 2, characterized in that: The change information includes a first change table, a first change type, a first change field, and first change field data, and determining the value of the first distributed lock and the key of the first distributed lock according to the change information includes: Determine a value of the first distributed lock based on the first change field and the first change field data; A key of the first distributed lock is determined based on the first change table, the first change type, and the value of the first distributed lock.

4. The data synchronization method according to claim 3, characterized in that: The determining the value of the first distributed lock based on the first change field and the first change field data includes: Sorting the first changed fields based on a preset sorting rule to obtain a sorting result; Based on the sorting result, encapsulating the first changed field and the corresponding first changed field data in a preset format to obtain changed data; A hash operation is performed on the changed data to obtain a value of the first distributed lock.

5. The data synchronization method according to claim 3, characterized in that: The determining the key of the first distributed lock based on the first change table, the first change type, and the value of the first distributed lock includes: The first change table, the first change type, and the value of the first distributed lock are concatenated to obtain a key of the first distributed lock.

6. The data synchronization method according to claim 3, characterized in that: The determining, according to the first data change operation, a second data change operation corresponding to the second database includes: Based on the first change table and the first change field, a preset data mapping relationship is searched to obtain a second change table corresponding to the first change table and a second change field corresponding to the first change field; The second data change operation is determined according to the first change type, the second change table, the second change field, and the first change field data.

7. The data synchronization method according to claim 1, characterized in that: Before detecting that the first database executes the first data change operation, the method further includes: Detecting the log files of the first database in real time to obtain detection results; Determine whether the first database executes a first data change operation according to the detection result.

8. A data synchronization device, characterized in that: The device comprises: A first setting module is used to set a first distributed lock for the first data change operation in a lock management database when detecting that the first database executes the first data change operation; a determination module, configured to determine, when determining that the first distributed lock is set successfully, a second data change operation corresponding to the second database according to the first data change operation; The second setting module is used to set a second distributed lock for the second data change operation in the lock management database, and when it is determined that the second distributed lock is set successfully, control the second database to execute the second data change operation.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data synchronization method as described in any one of claims 1 to 7 is implemented.

Images