A data synchronization method and device, electronic equipment and storage medium

Abstract

The embodiment of the application discloses a data synchronization method and device, electronic equipment and storage medium, and relates to the technical field of computers. The method comprises the following steps: when a first database executes a first data change operation, a first distributed lock is set for the first data change operation in a lock management database; when it is determined that the first distributed lock is successfully set, a second data change operation corresponding to a second database is determined according to the first data change operation; a second distributed lock is set for the second data change operation in the lock management database, and when it is determined that the second distributed lock is successfully set, the second database is controlled to execute the second data change operation. In the technical scheme provided by the embodiment of the application, the anti-circular synchronization mechanism is realized through the distributed lock, the storage pressure and the calculation burden of the database are reduced, the performance loss of the anti-circular synchronization mechanism is reduced, the data synchronization process is simplified, and therefore the efficiency and the performance of the data synchronization are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and particularly relate to a data synchronization method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the process of migrating data in the original region to the target region, first, the data integration tool is used to synchronize the data of the original database to the target database in full, so as to preliminarily align the data of the original database and the target database, wherein the original database is the database in the original region, and the target database is the database in the target region; then, the user requests are shunted from the original region to the target region in proportion, and the shunting proportion is gradually increased until the user requests are 100% shunted to the target region.

[0003] In the process of gradually shunting, incremental synchronization is started, that is, a data synchronization log table is created in the original database and the target database respectively, and a joint unique index is created for the two data synchronization log tables, then the data change operation of the original database (or the target database) is captured in real time, and the Canal is used to synchronize the data change operation to the target database (or the original database), and the data synchronization log table and the joint unique index are used to prevent circular synchronization, that is, to prevent the same data change operation from being synchronized between the original database and the target database multiple times.

[0004] However, in the above technical solution, the anti-circular synchronization mechanism is realized by using the data synchronization log table and the joint unique index, which increases the burden of the original database and the target database, and increases the performance loss, resulting in low efficiency of data synchronization. SUMMARY

[0005] Embodiments of the present application provide a data synchronization method and device, electronic equipment and storage medium, which realize the data synchronization function to solve the problem of increasing the burden and performance loss of the database in the prior art anti-circular synchronization mechanism.

[0006] In the first aspect, the embodiments of the present application provide a data synchronization method, which comprises:

[0007] When the first database executes the first data change operation, a first distributed lock is set for the first data change operation in the lock management database;

[0008] When the first distributed lock is set successfully, the second database corresponding to the second data change operation is determined according to the first data change operation;

[0009] A second distributed lock is set for the second data change operation in the lock management database, and when the second distributed lock is set successfully, the second database is controlled to execute the second data change operation.

[0010] In the embodiment of the present application, when it is detected that the first database executes the first data change operation, a first distributed lock is set for the first data change operation in the lock management database, and then when it is determined that the first distributed lock is set successfully, a second data change operation corresponding to the second database is determined according to the first data change operation, and then a second distributed lock is set for the second data change operation in the lock management database, and when it is determined that the second distributed lock is set successfully, the second database is controlled to execute the second data change operation, thereby realizing the data synchronization function. In the above technical solution, when it is detected that the first database executes the first data change operation, the data synchronization process is executed only when it is determined that the first distributed lock is set successfully, thereby realizing the anti-circular synchronization mechanism, that is, the same data change operation is synchronized only once between the first database and the second database, thereby effectively preventing circular synchronization. The prior art realizes the anti-circular synchronization mechanism through a data synchronization log table and a joint unique index, and needs to create and maintain an additional data structure in the original database and the target database, thereby occupying the storage resources and computing resources of the database. The above technical solution realizes the anti-circular synchronization mechanism through a distributed lock, and only needs to set and manage the lock in the lock management database, thereby avoiding the creation and maintenance of a complex data synchronization log table and a joint unique index in the first database and the second database, thereby reducing the storage pressure and computing burden of the first database and the second database, and reducing the performance loss of the anti-circular synchronization mechanism, and simplifying the data synchronization process, thereby improving the efficiency and performance of data synchronization.

[0011] In a second aspect, the embodiment of the present application provides a data synchronization device, which comprises:

[0012] A first setting module is configured to set a first distributed lock for the first data change operation in the lock management database when it is detected that the first database executes the first data change operation.

[0013] A determining module is configured to determine a second data change operation corresponding to the second database according to the first data change operation when it is determined that the first distributed lock is set successfully.

[0014] A second setting module is configured to set a second distributed lock for the second data change operation in the lock management database, and control the second database to execute the second data change operation when it is determined that the second distributed lock is set successfully.

[0015] In a third aspect, the embodiment of the present application provides an electronic device, which comprises:

[0016] at least one processor; and a memory connected with the at least one processor in communication;

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data synchronization method of any of the embodiments of the present application.

[0018] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data synchronization method of any of the embodiments of the present application.

[0019] The description of the second aspect, the third aspect and the fourth aspect of the present application can refer to the detailed description of the first aspect; and the beneficial effects of the second aspect, the third aspect and the fourth aspect can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0020] In the present application, the name of the above-mentioned data synchronization device does not constitute a limitation to the device or the function module itself, and in actual implementation, these devices or function modules can appear with other names. As long as the functions of each device or function module are similar to those of the present application, they belong to the scope of the claims of the present application and equivalent technologies.

[0021] These aspects or other aspects of the present application will be more apparent in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0023] Figure 1 is a flowchart of the data synchronization method provided by the embodiments of the present application;

[0024] Figure 2 is a data synchronization example diagram of the original area and the target area;

[0025] Figure 3a is an example diagram of the data synchronization method provided by the embodiments of the present application;

[0026] Figure 3b is another example diagram of the data synchronization method provided by the embodiments of the present application;

[0027] Figure 4 is a structural schematic diagram of the data synchronization device provided by the embodiments of the present application;

[0028] Figure 5 is a structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work should belong to the protection scope of the present application.

[0030] It should be noted that the terms “first”, “second”, “target” and “original” and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms “include”, “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Figure 1 is a flow diagram of a data synchronization method provided by the embodiments of the present application. The embodiments can be applied in the process of migrating data in an original region to a target region, and in the scene of synchronizing data in an original database and a target database. The data synchronization method provided by the embodiments can be executed by a data synchronization device provided by the embodiments of the present application, and the device can be realized by software and / or hardware. In a specific embodiment, the data synchronization device can be integrated in an electronic device, for example, the electronic device can be a computer or a server, etc. The execution subject of the present method can be an electronic device. The following will be described in detail taking the migration of data in an original region to a target region as an example, and the electronic device can provide data synchronization services through building a data integration platform (Apache SeaTunnel).

[0032] In the process of smoothly migrating the data of the original region to the target region, the data integration tool can be used to synchronize the data of the original database to the target database, then proportionally divert the user requests from the original region to the target region, and gradually increase the diversion ratio until the user requests are 100% diverted to the target region, while gradually diverting, the incremental synchronization is started, if the user request is diverted to the original region, and the user request involves data change operation of the original database, such as insertion, update and deletion, the data change operation of the original database is synchronized to the target database; if the user request is diverted to the target region, and the user request involves data change operation of the target database, the data change operation of the target database is synchronized to the original database.

[0033] As shown in Figure 2 is a data synchronization example diagram of the original region and the target region, Figure 2 The load balancing server in the application server 1 of the original region and the application server 2 of the target region divert 50% of the user requests to the original region, and the remaining 50% of the user requests to the target region, when the user request 1 is diverted to the application server 1 of the original region, and the user request 1 is inserted into the original database, the electronic device can synchronize the insertion operation of the original database to the target database; when the user request 2 is diverted to the application server 2 of the target region, and the user request 2 is updated in the target database, the electronic device can synchronize the update operation of the target database to the original database.

[0034] Referring to Figure 1 , the data synchronization method of the embodiment includes but is not limited to the following steps:

[0035] S110, when detecting that the first database executes the first data change operation, setting a first distributed lock for the first data change operation in the lock management database.

[0036] Wherein, the first database is a database that occurs data change operation; exemplary, the first database can be the original database of the original region, or the target database of the target region.

[0037] The first data change operation is an operation of changing the data in the first database, which can change the data in the first database; exemplary, the first data change operation can include insertion, update and deletion, etc.

[0038] The lock management database is a database specially used for managing the distributed lock, and is used for recording related information of the distributed lock, such as a key and a value of the distributed lock. The key of the distributed lock is a unique identifier of the distributed lock. The value of the distributed lock is used for storing related information of a data change operation corresponding to the distributed lock. For example, the lock management database can be a Redis (Remote Dictionary Server) database.

[0039] The first distributed lock is a lock created for the first data change operation when it is detected that the first database executes the first data change operation, and is used for determining whether the first data change operation of the first database needs to be synchronized to the second database, so as to prevent the first data change operation from being synchronized between the first database and the second database multiple times. If the first distributed lock is successfully set, it indicates that the first data change operation of the first database has not been synchronized to the second database. If the first distributed lock fails to be set, it indicates that the first data change operation of the first database has been synchronized to the second database.

[0040] The second database is a database that needs to keep data consistency with the first database. For example, when the first database is an original database, the second database is a target database. When the first database is a target database, the second database is an original database. It should be noted that the database types of the first database and the second database can be the same or different.

[0041] Specifically, in the process of data synchronization, the data in the first database can be batch-synchronized to the second database to align the data in the first database and the second database, and then it is detected in real time whether the first database executes the first data change operation. In an implementation manner, a database trigger can be created for the first database, and the database trigger is used to detect in real time whether the first database executes the first data change operation. The database trigger is used to detect in real time whether the first database executes the data change operation.

[0042] In another implementation, the log file of the first database can be detected in real time to obtain a detection result, and it is determined whether the first database executes the first data change operation according to the detection result, that is, the log file of the first database can be detected in real time, and when a log record is added to the log file, the log record is determined as the detection result, then the detection result is analyzed to obtain an operation corresponding to the detection result, if the operation corresponding to the detection result belongs to a preset operation type, it can be determined that the first database executes the first data change operation; if the operation corresponding to the detection result does not belong to the preset operation type, it can be determined that the first database does not execute the first data change operation. By detecting the log file, the data change operation can be quickly detected, and the timeliness of data synchronization is improved. The detection result is the basis for determining whether the first database executes the first data change operation; the preset operation type can include insertion, update, deletion, and the like.

[0043] After detecting that the first database executes the first data change operation, the first data change operation can be captured, for example, the newly added log record in the log file can be analyzed to obtain a change operation record in the form of a structured query language (SQL) statement, thereby obtaining the first data change operation; then a first distributed lock is set for the first data change operation in the lock management database, that is, the key of the first distributed lock and the value of the first distributed lock can be determined according to the first data change operation, and then the first distributed lock is further set for the first data change operation in the lock management database according to the key of the first distributed lock and the value of the first distributed lock.

[0044] Further, setting the first distributed lock for the first data change operation in the lock management database includes S a1-S a3:

[0045] S a1, the first data change operation is analyzed to obtain change information corresponding to the first data change operation.

[0046] The change information can include a first change table, a first change type, a first change field, and first change field data; the first change table is a database table involved in the first data change operation; the first change type is a specific type of the first data change operation, which can be insertion, update, or deletion; the first change field is a specific field in the first change table that is changed when the first data change operation is executed on the first database; and the first change field data is new data (for insertion and update operations) or old data (for deletion operations) corresponding to the first change field.

[0047] Specifically, the first data change operation can be parsed to obtain change information. For example, the SQL statement corresponding to the first data change operation can be analyzed by using a string processing function or a regular expression, and the table name part is extracted to obtain the first change table. Then, the keyword at the beginning of the SQL statement is extracted to obtain the change operation name, and the first change type is determined according to the change operation name. For example, if the change operation name is INSERT, the first change type is insertion; if the change operation name is UPDATE, the first change type is update; if the change operation name is DELETE, the first change type is deletion. Then, the column name is extracted from the SQL statement to obtain the first change field, and the value is extracted from the SQL statement to obtain the first change field data. If there are multiple first change fields, the mapping relationship between the first change field and the first change field data can be determined.

[0048] For example, if the SQL statement corresponding to the first data change operation is INSERT INTO user_1(field1_1, field1_2) VALUES(value1, value2), the first change table is user_1, the first change type is insertion, the first change field 1 is field1_1, the first change field 2 is field1_2, the first change field data corresponding to the first change field 1 is value1, and the first change field data corresponding to the first change field 2 is value2.

[0049] Sa2, determining the value of the first distributed lock and the key of the first distributed lock according to the change information.

[0050] Specifically, after obtaining the change information corresponding to the first data change operation, the change information can be analyzed, and then the value of the first distributed lock is determined based on the first change field and the first change field data, and the key of the first distributed lock is determined based on the first change table, the first change type, and the value of the first distributed lock.

[0051] Further, determining the value of the first distributed lock based on the first change field and the first change field data includes Sb1-Sb3:

[0052] Sb1, sorting the first change field based on a preset sorting rule to obtain a sorting result.

[0053] The preset sorting rule is a preset sorting rule for sorting the first change field. For example, the preset sorting rule can be a dictionary ascending order sorting.

[0054] The sorting result is a sequence obtained by sorting the first change field according to the preset sorting rule.

[0055] Specifically, when there are multiple first change fields, the multiple first change fields can be sorted according to a preset sorting rule. For example, if the preset sorting rule is ascending dictionary order, the first change field 1 is username, the first change field 2 is email, and the first change field 3 is phone, the sorting result is [“email”, “phone”, “username”].

[0056] Sb2, based on the sorting result, the first change field and the corresponding first change field data are encapsulated according to a preset format to obtain change data.

[0057] The preset format is a format set in advance, which is used to specify how to combine the sorted first change field and the corresponding first change field data. For example, the preset format is JS key-value pair data (JavaScript Object Notation, JSON) format.

[0058] The change data is the result obtained after the sorted first change field and the corresponding first change field data are encapsulated according to the preset format.

[0059] For example, if the preset format is JSON format, the first change field data corresponding to the first change field 1 (i.e., username) is value1, the first change field data corresponding to the first change field 2 (i.e., email) is value2, the first change field data corresponding to the first change field 3 (i.e., phone) is value3, and the sorting result of the first change field is [“email”, “phone”, “username”], the first change field and the corresponding first change field data are encapsulated according to the preset format, and the obtained change data is {“email”:“value2”, “phone”:“value3”, “username”:“value1”}.

[0060] Sb3, performing a hash operation on the change data to obtain the value of the first distributed lock.

[0061] Specifically, after obtaining the change data, a hash algorithm can be used to perform a hash operation on the change data. For example, an SM3 algorithm can be used to perform a hash operation on the change data to obtain a hash digest, and the hash digest is determined as the value of the first distributed lock.

[0062] In the embodiments of the present application, the first change field is sorted based on the preset sorting rule, which can ensure that the same data change operation corresponds to the same sorting result, thereby avoiding different values generated due to different field orders; the sorted first change field and the corresponding first change field data are encapsulated in the preset format, which standardizes the value organization form and can determine that the same data change operation corresponds to the same change data; the hash operation is performed on the change data to obtain the value of the first distributed lock, which ensures the privacy and security of the data, ensures that different data change operations correspond to different values, and ensures that the same data change operation corresponds to the same value, thereby providing a judgment basis for subsequent prevention of circular synchronization.

[0063] After obtaining the value of the first distributed lock, the first change table, the first change type and the value of the first distributed lock can be spliced to obtain the key of the first distributed lock, and the representation form of the first change table at this time can be a table name or a table name with a database prefix. The key of the first distributed lock is generated by splicing, which can ensure the uniqueness of the key, so that the key of each distributed lock can accurately correspond to a specific data change operation, thereby providing a judgment basis for subsequent determination of whether the first distributed lock is successfully set, thereby providing a judgment basis for subsequent prevention of circular synchronization.

[0064] For example, if the first change table is table1, the first change type is insertion, and the value of the first distributed lock is sign, the key of the first distributed lock is table1_INSERT_sign; or if the first change table is db.table1, the first change type is insertion, and the value of the first distributed lock is sign, the key of the first distributed lock is db.table1_INSERT_sign.

[0065] Sa3, based on the value of the first distributed lock and the key of the first distributed lock, setting the first distributed lock in the lock management database.

[0066] Specifically, after obtaining the value of the first distributed lock and the key of the first distributed lock, the first distributed lock can be set in the lock management database, that is, the lock management database can be connected, and then a lock setting command provided by the lock management database, such as a SET command in Redis, is used to set the first distributed lock based on the value of the first distributed lock and the key of the first distributed lock, and it is determined whether the first distributed lock is successfully set. Specifically, when the lock management database contains a lock with the same key as the first distributed lock, the first distributed lock fails to be set; when the lock management database does not contain a lock with the same key as the first distributed lock, the first distributed lock is successfully set. It should be noted that network problems or server failures of the lock management database and other reasons can also cause the first distributed lock to fail to be set.

[0067] If the first distributed lock is successfully set in the lock management database, it indicates that there is no same lock of the first distributed lock in the lock management database, i.e., the first data change operation of the first database is not synchronized to the second database, at this time, data synchronization needs to be performed, i.e., S120 can be executed; if the first distributed lock fails to be set in the lock management database, the reason for the setting failure can be determined, and when the reason for the setting failure is that there is a same lock of the first distributed lock in the lock management database, it indicates that the first data change operation of the first database has been synchronized to the second database, at this time, data synchronization does not need to be performed again, and real-time detection of whether the first database executes a data change operation can be continued; when the reason for the setting failure is other reasons, setting the first distributed lock for the first data change operation in the lock management database can be repeatedly executed until the first distributed lock is successfully set.

[0068] In the embodiment of the application, the first distributed lock is set according to the change information corresponding to the first data change operation, which can ensure the uniqueness of the first distributed lock and provide a judgment basis for subsequent prevention of circular synchronization, thereby effectively preventing circular synchronization.

[0069] S120, when it is determined that the first distributed lock is successfully set, a second data change operation corresponding to the second database is determined according to the first data change operation.

[0070] The second data change operation is determined according to the first data change operation and needs to be executed in the second database, and can change data in the second database; and the change type of the second data change operation is the same as that of the first data change operation, for example, if the first data change operation is to insert a new order record in the first database, the second data change operation is to execute a corresponding insertion operation in the second database to ensure that the order data in the two databases is consistent.

[0071] Specifically, when it is determined that the first distributed lock is successfully set, it indicates that the second database has not synchronized the first data change operation, at this time, the second data change operation corresponding to the second database can be determined according to the first data change operation, i.e., the change information corresponding to the second data change operation can be determined based on the data mapping relationship between the first database and the second database and the change information corresponding to the first data change operation, and the second data change operation is generated according to the change information corresponding to the second data change operation.

[0072] Further, the second data change operation corresponding to the second database is determined according to the first data change operation, including Sc1-Sc2:

[0073] Sc1, query the preset data mapping relationship based on the first change table and the first change field, to obtain a second change table corresponding to the first change table and a second change field corresponding to the first change field.

[0074] The preset data mapping relationship is a pre-set mapping relationship, and is used to describe the corresponding relationship between the data structure and the data content of the first database and the second database, and can include a table mapping relationship and a field mapping relationship.

[0075] The second change table is a table corresponding to the first change table in the second database. The second change field is a field corresponding to the first change field in the first change table in the second change table.

[0076] Specifically, the table mapping relationship in the preset data mapping relationship can be queried based on the first change table to obtain the second change table corresponding to the first change table, and then the field mapping relationship in the preset data mapping relationship can be queried based on the first change field to obtain the second change field corresponding to the first change field.

[0077] Sc2, determine the second data change operation according to the first change type, the second change table, the second change field and the first change field data.

[0078] Specifically, after obtaining the second change table and the second change field, the second change type, i.e. the specific operation type of the second data change operation, can be determined according to the first change type, and the second change type is the same as the first change type at this time. Then, the first change field data is determined as the second change field data, and the mapping relationship between the second change field and the second change field data is determined according to the mapping relationship between the first change field and the first change field data, and the mapping relationship between the first change field and the second change field. Then, the second data change operation is generated according to the second change type, the second change table, the second change field and the second change field data, and the form of the second data change operation at this time can be a SQL statement.

[0079] For example, if the first data change operation corresponds to the SQL statement INSERT INTO user_1 (field1_1, field1_2) VALUES (value1, value2), the preset data mapping relationship has user_1 corresponding to user_2, field1_1 in user_1 corresponding to field2_1 in user_2, and field1_2 in user_1 corresponding to field2_2 in user_2, the second change table is user_2, the second change type is insertion, the second change field 1 is field2_1, the second change field 2 is field2_2, the second change field 1 corresponds to the second change field data value1, the second change field 2 corresponds to the second change field data value2, and the second data change operation corresponds to the SQL statement INSERT INTO user_2 (field2_1, field2_2) VALUES (value1, value2).

[0080] In the embodiments of the present application, the preset data mapping relationship can effectively avoid errors in the data conversion process across databases, thereby improving the determination accuracy and determination efficiency of the second data change operation, and allowing the database types or data structures of the first database and the second database to be different, thereby being applicable to data synchronization between various homogeneous databases or heterogeneous databases, thereby improving the flexibility of data synchronization.

[0081] In the embodiments of the present application, the preset data mapping relationship can effectively avoid errors in the data conversion process across databases, thereby improving the determination accuracy and determination efficiency of the second data change operation, and allowing the database types or data structures of the first database and the second database to be different, thereby being applicable to data synchronization between various homogeneous databases or heterogeneous databases, thereby improving the flexibility of data synchronization.

[0082] The second distributed lock is a lock created for the second data change operation, and is used to determine whether to control the second database to execute the second data change operation. Meanwhile, after the second database executes the second data change operation, the second distributed lock can also be used to determine whether the second data change operation of the second database needs to be synchronized to the first database, to prevent the second data change operation from being synchronized between the first database and the second database multiple times, and the second data change operation at this time is essentially the same data change operation as the first data change operation.

[0083] Specifically, after obtaining the second data change operation, a second distributed lock can be set for the second data change operation in the lock management database, that is, the value of the second distributed lock can be determined based on the second change field and the second change field data, the key of the second distributed lock can be determined based on the second change table, the second change type and the value of the second distributed lock, and then the second distributed lock can be set in the lock management database based on the value of the second distributed lock and the key of the second distributed lock, and it is determined whether the second distributed lock is set successfully. The specific implementation details are the same as those of S110, and reference can be made to the description of S110, which will not be repeated here.

[0084] Since the second database does not execute the second data change operation when the second distributed lock is set, the reason why the second distributed lock fails to be set is not the existence of the same lock of the second distributed lock in the lock management database, but other reasons. Therefore, when it is determined that the second distributed lock fails to be set, the setting of the second distributed lock for the second data change operation in the lock management database is repeatedly executed until the second distributed lock is successfully set, thereby providing a basis for determining whether to synchronize the second data change operation to the first database when it is detected that the second database executes the second data change operation, and effectively preventing circular synchronization.

[0085] If it is determined that the second distributed lock is successfully set, the second database can be controlled to execute the second data change operation, for example, the SQL statement corresponding to the second data change operation is executed to synchronize the data in the second database and the first database.

[0086] Optionally, the electronic device can detect the log file of the second database in real time, and when it is detected that the second database executes the second data change operation, a third distributed lock corresponding to the second data change operation can be set in the lock management database, but the key of the third distributed lock is the same as that of the second distributed lock, that is, the same lock of the third distributed lock already exists in the lock management database, so it can be determined that the third distributed lock fails to be set, that is, the second data change operation of the second database has been synchronized to the first database, and there is no need to synchronize the data again. At this time, the second database can be continuously detected in real time whether to execute the data change operation.

[0087] As shown in FIG. 1, Figure 3a FIG. 2 shows an example diagram of a data synchronization method provided by an embodiment of the present application, Figure 3aThe first database in the original region is a database in the original region, the second database is a database in the target region, and the data synchronization process can be represented as: when the user request is distributed to the application server 1, the application server 1 controls the first database to perform a first data change operation corresponding to the user request. At this time, the electronic device can detect the first database in real time, and when detecting that the first database performs the first data change operation, set a first distributed lock for the first data change operation in the lock management database. Then, it is determined whether the first distributed lock is set successfully. If it is determined that the first distributed lock is set successfully, a second data change operation corresponding to the second database is determined according to the first data change operation, and a second distributed lock for the second data change operation is set in the lock management database. Then, when it is determined that the second distributed lock is set successfully, the second database is controlled to perform the second data change operation. If it is determined that the first distributed lock is not set successfully, the data synchronization process is ended, and circular synchronization is prevented.

[0088] As shown in FIG. 1, the first database in the original region is a database in the original region, the second database is a database in the target region, and the data synchronization process can be represented as: when the user request is distributed to the application server 1, the application server 1 controls the first database to perform a first data change operation corresponding to the user request. At this time, the electronic device can detect the first database in real time, and when detecting that the first database performs the first data change operation, set a first distributed lock for the first data change operation in the lock management database. Then, it is determined whether the first distributed lock is set successfully. If it is determined that the first distributed lock is set successfully, a second data change operation corresponding to the second database is determined according to the first data change operation, and a second distributed lock for the second data change operation is set in the lock management database. Then, when it is determined that the second distributed lock is set successfully, the second database is controlled to perform the second data change operation. If it is determined that the first distributed lock is not set successfully, the data synchronization process is ended, and circular synchronization is prevented. Figure 3b Figure 3b As shown in FIG. 1, the first database in the original region is a database in the original region, the second database is a database in the target region, and the data synchronization process can be represented as: when the user request is distributed to the application server 1, the application server 1 controls the first database to perform a first data change operation corresponding to the user request. At this time, the electronic device can detect the first database in real time, and when detecting that the first database performs the first data change operation, set a first distributed lock for the first data change operation in the lock management database. Then, it is determined whether the first distributed lock is set successfully. If it is determined that the first distributed lock is set successfully, a second data change operation corresponding to the second database is determined according to the first data change operation, and a second distributed lock for the second data change operation is set in the lock management database. Then, when it is determined that the second distributed lock is set successfully, the second database is controlled to perform the second data change operation. If it is determined that the first distributed lock is not set successfully, the data synchronization process is ended, and circular synchronization is prevented.

[0089] ​The technical scheme of the embodiment of the present application can set a first distributed lock for the first data change operation in the lock management database when detecting that the first database executes the first data change operation, then determine a second data change operation corresponding to the second database according to the first data change operation when determining that the first distributed lock is set successfully, then set a second distributed lock for the second data change operation in the lock management database, and control the second database to execute the second data change operation when determining that the second distributed lock is set successfully, thereby realizing the data synchronization function. In the above technical scheme, the data synchronization process is executed only when it is determined that the first distributed lock is set successfully in the case of detecting that the first database executes the first data change operation, which can realize the anti-circular synchronization mechanism, that is, the same data change operation is synchronized only once between the first database and the second database, thereby effectively preventing circular synchronization. The prior art realizes the anti-circular synchronization mechanism through a data synchronization log table and a joint unique index, and needs to create and maintain an additional data structure in the original database and the target database, thereby occupying the storage resources and computing resources of the database. The above technical scheme realizes the anti-circular synchronization mechanism through a distributed lock, which only needs to set and manage the lock in the lock management database, thereby avoiding the creation and maintenance of a complex data synchronization log table and a joint unique index in the first database and the second database, thereby reducing the storage pressure and computing burden of the first database and the second database, and reducing the performance loss of the anti-circular synchronization mechanism, and simplifying the data synchronization process, thereby improving the efficiency and performance of data synchronization.

[0090] Figure 4 is a structural schematic diagram of a data synchronization device provided by the embodiment of the present application, referring to Figure 4 The data synchronization device can include:

[0091] The first setting module 410 is configured to set a first distributed lock for the first data change operation in the lock management database when detecting that the first database executes the first data change operation.

[0092] The determination module 420 is configured to determine a second data change operation corresponding to the second database according to the first data change operation when determining that the first distributed lock is set successfully.

[0093] The second setting module 430 is configured to set a second distributed lock for the second data change operation in the lock management database, and control the second database to execute the second data change operation when determining that the second distributed lock is set successfully.

[0094] In an embodiment, the first setting module 410 is specifically configured to:

[0095] analyze the first data change operation to obtain change information corresponding to the first data change operation;

[0096] determine a value of the first distributed lock and a key of the first distributed lock according to the change information;

[0097] set the first distributed lock in the lock management database based on the value of the first distributed lock and the key of the first distributed lock.

[0098] In an embodiment, the change information includes a first change table, a first change type, a first change field and first change field data, and the first setting module 410 determines the value of the first distributed lock and the key of the first distributed lock according to the change information, including:

[0099] determine the value of the first distributed lock based on the first change field and the first change field data;

[0100] determine the key of the first distributed lock based on the first change table, the first change type and the value of the first distributed lock.

[0101] In an embodiment, the first setting module 410 determines the value of the first distributed lock based on the first change field and the first change field data, including:

[0102] sort the first change field based on a preset sorting rule to obtain a sorting result;

[0103] encapsulate the first change field and the corresponding first change field data in a preset format based on the sorting result to obtain change data;

[0104] perform a hash operation on the change data to obtain the value of the first distributed lock.

[0105] In an embodiment, the first setting module 410 determines the key of the first distributed lock based on the first change table, the first change type and the value of the first distributed lock, including:

[0106] splice the first change table, the first change type and the value of the first distributed lock to obtain the key of the first distributed lock.

[0107] In an embodiment, the determining module 420 is specifically configured to:

[0108] query a preset data mapping relationship based on the first change table and the first change field to obtain a second change table corresponding to the first change table and a second change field corresponding to the first change field;

[0109] determine a second data change operation according to the first change type, the second change table, the second change field and the first change field data.

[0110] In an embodiment, the data synchronization apparatus further includes a detecting module, which is specifically configured to:

[0111] Before detecting that the first database executes the first data change operation, a log file of the first database is detected in real time to obtain a detection result;

[0112] It is determined whether the first database executes the first data change operation according to the detection result.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the above described functional modules can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0114] The data synchronization device provided in the embodiment can be applied to the data synchronization method provided in any of the foregoing embodiments, and has corresponding functions and advantages.

[0115] Figure 5 is a structural schematic diagram of an electronic device provided in an embodiment of the present application. Figure 5 A block diagram of an exemplary electronic device 11 suitable for implementing the embodiments of the present application is shown. Figure 5 The electronic device 11 shown is merely an example and should not impose any limitation on the functions and scope of use of the embodiments.

[0116] As shown in Figure 5 The electronic device 11 is shown in the form of a general computing electronic device. The components of the electronic device 11 can include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects the various system components including the system memory 28 and the processing unit 16.

[0117] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures including, for example, an industry standard architecture (ISA) bus, an Intel® micro-channel architecture (MAC) bus, an enhanced ISA bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnect (PCI) bus.

[0118] The electronic device 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 11 and includes both volatile and nonvolatile media, removable and non-removable media.

[0119] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 As not shown, disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disc drives for reading and writing to removable non-volatile optical discs (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0120] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this application.

[0121] Electronic device 11 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 11, and / or with any device that enables electronic device 11 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 11 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20.

[0122] like Figure 5 As shown, network adapter 20 communicates with other modules of electronic device 11 via bus 18. It should be understood that, although... ​ As not shown, other hardware and / or software modules may be used in conjunction with electronic device 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] The processing unit 16 executes various function applications and page displays by running programs stored in the system memory 28, such as implementing a data synchronization method provided by the embodiment, which comprises:

[0124] When detecting that the first database executes the first data change operation, setting a first distributed lock for the first data change operation in the lock management database;

[0125] When determining that the first distributed lock is set successfully, determining a second data change operation corresponding to the second database according to the first data change operation;

[0126] Setting a second distributed lock for the second data change operation in the lock management database, and when determining that the second distributed lock is set successfully, controlling the second database to execute the second data change operation.

[0127] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the data synchronization method provided by any embodiment of the application.

[0128] The embodiment of the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement, for example, a data synchronization method provided by the embodiment of the application, which comprises:

[0129] When detecting that the first database executes the first data change operation, setting a first distributed lock for the first data change operation in the lock management database;

[0130] When determining that the first distributed lock is set successfully, determining a second data change operation corresponding to the second database according to the first data change operation;

[0131] Setting a second distributed lock for the second data change operation in the lock management database, and when determining that the second distributed lock is set successfully, controlling the second database to execute the second data change operation.

[0132] The computer storage medium of this embodiment can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any combination thereof. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device.

[0133] The computer-readable signal medium can include a computer-readable program code in a baseband or propagated as part of a carrier wave. Such a propagated computer-readable signal medium can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. Computer-readable signal media can be any computer-readable medium that is not a storage medium and that can be used to carry or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0134] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0135] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the "C" programming language, or the like. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, application specific circuitry, or field programmable gate array (FPGA) circuitry can execute the computer program code.

[0136] Those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by computer device executable program codes, so that they can be stored in storage devices and executed by computing devices, or they can be respectively made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to realize. Thus, the present application is not limited to any specific combination of hardware and software.

[0137] In addition, the acquisition, storage, use, processing, etc. of data in the technical solutions of the present application all comply with relevant provisions of national laws and regulations.

[0138] Note that the above are only preferred embodiments of the present application and the technical principles applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and those skilled in the art can make various obvious changes, re-adjustments and substitutions without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the inventive concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data synchronization method, characterized by, The method comprises: When detecting that a first database executes a first data change operation, parsing the first data change operation to obtain change information, the change information comprising a first change table, a first change type, a first change field and first change field data; Based on the first change field and the first change field data, determining the value of the first distributed lock, comprising: sorting the first change field based on a preset sorting rule to obtain a sorting result; based on the sorting result, encapsulating the first change field and the corresponding first change field data according to a preset format to obtain change data; performing a hash operation on the change data to obtain the value of the first distributed lock; Based on the first change table, the first change type and the value of the first distributed lock, determining the key of the first distributed lock; Based on the key and value of the first distributed lock, setting a first distributed lock for the first data change operation in an independent lock management database; When determining that the first distributed lock is set successfully, determining a second data change operation corresponding to a second database according to the first data change operation; Setting a second distributed lock for the second data change operation in the lock management database, and when determining that the second distributed lock is set successfully, controlling the second database to execute the second data change operation.

2. The data synchronization method of claim 1, wherein, The determination of the key of the first distributed lock based on the first change table, the first change type and the value of the first distributed lock comprises: Splicing the first change table, the first change type and the value of the first distributed lock to obtain the key of the first distributed lock.

3. The data synchronization method of claim 1, wherein, The determination of the second data change operation corresponding to the second database according to the first data change operation comprises: Querying a preset data mapping relationship based on the first change table and the first change field to obtain a second change table corresponding to the first change table and a second change field corresponding to the first change field; Determining the second data change operation according to the first change type, the second change table, the second change field and the first change field data.

4. The data synchronization method of claim 1, wherein, Before detecting that the first database executes the first data change operation, the method further comprises: Real-time detecting a log file of the first database to obtain a detection result; Determining whether the first database executes the first data change operation according to the detection result.

5. A data synchronization apparatus, characterized by comprising: The device comprises: A first setting module configured to set a first distributed lock for a first data change operation in an independent lock management database when detecting that a first database executes the first data change operation; A determination module configured to determine a second data change operation corresponding to a second database according to the first data change operation when determining that the first distributed lock is set successfully; A second setting module configured to set a second distributed lock for the second data change operation in the independent lock management database, and control the second database to execute the second data change operation when determining that the second distributed lock is set successfully; The first setting module is specifically configured to: The first data change operation is parsed to obtain change information, and the change information includes a first change table, a first change type, a first change field, and first change field data; Based on the first change field and the first change field data, the value of the first distributed lock is determined, including: based on a preset sorting rule, the first change field is sorted to obtain a sorting result; based on the sorting result, the first change field and the corresponding first change field data are packaged according to a preset format to obtain change data; the change data is subjected to a hash operation to obtain the value of the first distributed lock; Based on the first change table, the first change type, and the value of the first distributed lock, the key of the first distributed lock is determined; Based on the key and the value of the first distributed lock, the first distributed lock is set in a lock management database.

6. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data synchronization method of any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the data synchronization method of any one of claims 1 to 4.

Images