Database synchronization method, device and electronic equipment

Abstract

The present application discloses a database synchronization method, device and electronic device, the method comprising: obtaining a data processing instruction, wherein the data processing instruction at least includes a first identifier of data to be processed; deleting first data corresponding to the first identifier in a first database corresponding to the electronic device; processing the first data corresponding to the first identifier in a second database corresponding to the electronic device; the second database and the first database are different in database structure; wherein the processed first data in the second database is used to write into the first database, so that the first database and the second database are consistent with each other at least in terms of the first data.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a database synchronization method, device and electronic device. Background Art

[0002] Heterogeneous databases are increasingly being used. For example, in the same system, Redis database is used as an in-memory database to provide real-time services, while Oracle database is used as a storage database to ensure the stability and security of data storage. In order to ensure data consistency, data synchronization between heterogeneous databases is required during data processing. However, data inconsistency between heterogeneous databases may still occur.

[0003] Therefore, there is an urgent need for a technical solution that can ensure data consistency between heterogeneous databases. Summary of the invention

[0004] In view of this, the present application provides a database synchronization method, device and electronic device to solve the technical problem of data inconsistency between heterogeneous databases in the prior art.

[0005] The present application provides a database synchronization method, including:

[0006] Obtaining a data processing instruction, wherein the data processing instruction at least includes a first identifier of the data to be processed;

[0007] Deleting first data corresponding to the first identifier in a first database corresponding to the electronic device;

[0008] In a second database corresponding to the electronic device, processing first data corresponding to the first identifier; the second database is different from the first database in terms of database structure;

[0009] The processed first data in the second database is used to be written into the first database, so that the first database and the second database are consistent with each other at least with respect to the first data.

[0010] The above method, preferably, further comprises:

[0011] Obtaining a data reading instruction, wherein the data reading instruction at least includes a second identifier of the data to be read;

[0012] Searching the first database for second data corresponding to the second identifier;

[0013] If the second data corresponding to the second identifier is not found in the first database, searching the second database for the second data corresponding to the second identifier;

[0014] If the second data corresponding to the second identifier is found in the second database, the second data found in the second database is returned;

[0015] The second data found in the second database is written into the first database.

[0016] Preferably, the above method further comprises, before processing the first data corresponding to the first identifier:

[0017] Determining whether the first data in the first database is deleted successfully;

[0018] If the first data in the first database is deleted successfully, executing the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device;

[0019] If the first data in the first database is not deleted successfully, the first identifier corresponding to the first data is recorded in the to-be-deleted record table, and the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device;

[0020] The identifier in the to-be-deleted record table is used to delete the corresponding data.

[0021] In the above method, preferably, after recording the first identifier corresponding to the first data in the record table to be deleted, the method further includes:

[0022] Determine whether the first identifier is successfully recorded in the to-be-deleted record table;

[0023] If the first identifier is successfully recorded in the to-be-deleted record table, the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device;

[0024] If recording the first identifier in the to-be-deleted record table fails, the current processing flow of the data processing instruction is terminated.

[0025] The above method, preferably, further comprises:

[0026] When a deletion condition corresponding to the to-be-deleted record table is satisfied, reading at least one target identifier from the to-be-deleted record table;

[0027] Deleting the target data corresponding to the target identifier in the first database;

[0028] The deletion condition includes: the current duration recorded by the timer reaches a preset duration threshold; the timer restarts when the current duration recorded by the timer reaches the duration threshold.

[0029] In the above method, preferably, the target identifier is stored in a target list;

[0030] Wherein, deleting the target data corresponding to the target identifier in the first database includes:

[0031] Determine whether the target list is empty;

[0032] If the target list is empty, the current processing flow of the to-be-deleted record list is terminated;

[0033] If the target list is not empty, the target data corresponding to the target identifier is deleted from the first database according to the target list.

[0034] Preferably, the above method further comprises, before reading at least one target identifier from the to-be-deleted record table:

[0035] Determine whether the first database is in a target state, where the first database in the target state can read and write data;

[0036] If the first database is in the target state, executing the steps of: reading at least one target identifier from the to-be-deleted record table;

[0037] If the first database is not in the target state, the current processing flow of the to-be-deleted record table is terminated.

[0038] Preferably, in the above method, before searching the first database for the second data corresponding to the second identifier, the method further comprises:

[0039] Determine whether the first database is in a target state, where the first database in the target state can read and write data;

[0040] If the first database is in the target state, executing the steps of: searching the first database for second data corresponding to the second identifier;

[0041] If the first database is not in the target state, the current processing flow of the data reading instruction is terminated.

[0042] The present application also provides a database synchronization device, including:

[0043] An instruction obtaining unit, used for obtaining a data processing instruction, wherein the data processing instruction at least includes a first identifier of the data to be processed;

[0044] A data deleting unit, configured to delete first data corresponding to the first identifier in a first database corresponding to the electronic device;

[0045] A data processing unit, configured to process first data corresponding to the first identifier in a second database corresponding to the electronic device; the second database is different from the first database in terms of database structure;

[0046] The data writing unit is used to write the processed first data in the second database into the first database, so that the first database and the second database are consistent with each other at least with respect to the first data.

[0047] The present application also provides an electronic device, including:

[0048] A memory, used to store a computer program and data generated by the execution of the computer program;

[0049] A processor is used to execute the computer program to achieve: obtaining a data processing instruction, wherein the data processing instruction at least includes a first identifier of data to be processed; deleting first data corresponding to the first identifier in a first database corresponding to the electronic device; processing the first data corresponding to the first identifier in a second database corresponding to the electronic device; the second database is different from the first database in terms of database structure; wherein the processed first data in the second database is used to write into the first database, so that the first database and the second database are consistent with each other at least with respect to the first data.

[0050] It can be seen from the above technical scheme that in a database synchronization method, device and electronic device disclosed in the present application, when a data processing instruction is obtained, the first data corresponding to the first identifier of the data to be processed in the first database is first deleted, and then the first data corresponding to the first identifier is processed in the second database heterogeneous with the first database, and then the processed first data in the second database is written into the first database, and the first database and the second database can maintain consistency in the first data. It can be seen that in the data processing process in the present application, by first deleting the data in the first database, and then processing the data in the second database, and then writing the processed data in the second database into the first database, the data consistency between the first database and the first database can be guaranteed, and the situation of data inconsistency between heterogeneous databases during the data processing process can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A flowchart of a database synchronization method provided in Example 1 of the present application;

[0053] Figure 2 An architectural diagram of configuring the memory database and storage database for the application server to which this application applies;

[0054] Figure 3 A partial flow chart of a database synchronization method provided in Example 1 of the present application;

[0055] Figure 4 and Figure 5 They are respectively another flow chart of a database synchronization method provided in Embodiment 1 of the present application;

[0056] Figure 6-Figure 9 They are respectively another partial flow chart of a database synchronization method provided in Embodiment 1 of the present application;

[0057] Fig.10 A schematic diagram of the structure of a database synchronization device provided in Embodiment 2 of the present application;

[0058] Fig.11 Another structural diagram of a database synchronization device provided in Embodiment 2 of the present application;

[0059] Fig.12 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present application;

[0060] Fig.13 An example diagram of client access between a Redis database and an Oracle database configured for the application server to which this application applies;

[0061] Fig.14 A flowchart for reading customer data in an application server to which this application applies;

[0062] Fig.15 A flowchart of updating customer data in an application server to which this application is applicable;

[0063] Fig.16 The present invention is a flow chart of the compensation batch procedure in the application server to which the present application is applicable. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] refer to Figure 1 As shown, it is a flowchart of a database synchronization method provided in the first embodiment of the present application, which can be applied to electronic devices configured with heterogeneous databases and capable of data processing, such as servers, etc. The technical solution in this embodiment is mainly used to ensure data consistency between heterogeneous databases during data processing.

[0066] Specifically, the method in this embodiment may include the following steps:

[0067] Step 101: Obtain data processing instructions.

[0068] The data processing instruction at least includes the first identifier of the data to be processed. The data processing instruction can be a data write instruction, a data delete instruction or a data modification instruction, wherein the data write instruction is used to instruct the writing of the first data corresponding to the first identifier, the data delete instruction is used to instruct the deletion of the first data corresponding to the first identifier, the data modification instruction is used to instruct the modification of the first data corresponding to the first identifier, and so on.

[0069] Specifically, the first identifier may be an identifier that can uniquely represent the first data, such as a key value, etc. The data processing instruction may be generated when a user needs to process data and perform a processing operation on the electronic device, or the data processing instruction may be an instruction automatically generated during the operation of the electronic device to implement a corresponding function.

[0070] Step 102: Delete the first data corresponding to the first identifier in the first database corresponding to the electronic device.

[0071] Among them, the first database is the memory library data configured by the electronic device. In addition, the electronic device is also configured with a second database, which is a storage database. The first database and the second database are different in terms of database structure. For example, the first database can be a Redis database, which is a memory database with a master-slave mechanism. It can support high-concurrency queries through high performance and fast reading speed (up to 110,000 times / s) to ensure high availability. The second database can be an Oracle database, which is a relational database management system with good portability, powerful functions, high reliability and stable performance.

[0072] Specifically, in this embodiment, after obtaining a data write instruction, a data delete instruction, or a data modify instruction, the first data corresponding to the first identifier may be searched in the first database, and then the searched first data may be deleted.

[0073] Step 103: Processing the first data corresponding to the first identifier in the second database corresponding to the electronic device.

[0074] Specifically, in this embodiment, according to the data processing instruction, the first data corresponding to the first identifier can be written into the second database, or the first data corresponding to the first identifier in the second database can be deleted, or the first data corresponding to the first identifier in the second database can be modified, and so on.

[0075] Based on this, the processed first data in the second database is used to write into the first database, so that the first database is consistent with the second database at least with respect to the first data.

[0076] like Figure 2 As shown in , in this embodiment, when an electronic device such as an application server needs to process first data, it first deletes the first data in the memory database to avoid data inconsistency caused by unexpected interruptions during data processing. Thereafter, the first data is processed in the storage database to ensure the security of the first data by utilizing the stability of the storage database, and the processed first data can be written into the memory database. Thus, on the premise of realizing data processing, it is ensured that the memory database and the storage database are consistent with each other at least with respect to the first data, and then as data processing proceeds, data consistency can be ensured between the memory database and the storage database.

[0077] It can be seen from the above technical scheme that a database synchronization method provided in Example 1 of the present application, when obtaining a data processing instruction, first deletes the first data corresponding to the first identifier of the data to be processed in the first database, and then processes the first data corresponding to the first identifier in the second database heterogeneous with the first database, and then the processed first data in the second database is written into the first database, and the first database and the second database can maintain consistency in the first data. It can be seen that in the data processing process of the present application, by first deleting the data in the first database, and then processing the data in the second database, and then writing the processed data in the second database into the first database, the data consistency between the first database and the first database can be guaranteed, and the situation of data inconsistency between heterogeneous databases during the data processing process can be avoided.

[0078] In one implementation, the method in this embodiment may further include the following steps: Figure 3 As shown in:

[0079] Step 301: Obtain a data read instruction.

[0080] The data read instruction at least includes the second identifier to be read. The data read instruction is used to instruct to read the second data corresponding to the second identifier. The second identifier can be an identifier that can uniquely represent the second data, such as a key value.

[0081] Specifically, the data reading instruction may be generated when a user needs to read data and performs a reading operation on the electronic device, or the data reading instruction may be an instruction automatically generated during the operation of the electronic device to implement a corresponding function.

[0082] Step 302: Search the first database for second data corresponding to the second identifier. If the second data corresponding to the second identifier is found in the first database, execute step 303. If the second data corresponding to the second identifier is not found in the first database, execute step 304.

[0083] Specifically, in this embodiment, the data contained in the first database can be identified and compared according to the second identifier in the first database, thereby realizing the search for the second data. If the second data is found in the first database, execute step 303; if the second data is not found in the first database, execute step 304.

[0084] Step 303: Return the second data found in the first database.

[0085] Specifically, the second data is read from the first database and then returned to the electronic device. For example, the second data read from the memory database is returned to the application server used by the user.

[0086] Step 304: searching the second database for second data corresponding to the second identifier. If the second data corresponding to the second identifier is found in the second database, executing step 305.

[0087] Specifically, in this embodiment, the data contained in the second database can be identified and compared according to the second identifier in the second database, thereby realizing the search for the second data. If the second data is found in the second database, steps 305 and 306 are executed. If the second data is not found in the second database, it indicates that the second data has not been processed in any database, and the current process ends at this time.

[0088] Step 305: Return the second data found in the second database.

[0089] Specifically, the second data is read from the second database and returned to the electronic device. For example, if the second data is not found in the memory database, the second data read from the storage database is returned to the application server used by the user.

[0090] Step 306: Write the second data found in the second database into the first database.

[0091] Specifically, the second data read from the second database is written into the first database so that the first database and the second database are consistent with each other regarding the second data. For example, if the second data is not found in the memory database, the second data read from the storage database is returned to the application server used by the user, and the second data is written into the memory database, so that the memory database and the storage database are consistent with each other regarding the second data.

[0092] It should be noted that Figure 3 The second data in can be Figure 1 The first data in the first database is stored in the first database. Thus, after deleting the data in the first database in order to ensure data consistency during data processing, the read data can be written into the first database during data reading. Thus, the data can be cached in the first database to increase the number of data reads, thereby improving the read operation hit rate of the first database and ensuring high availability.

[0093] In one implementation, based on Figure 1 The embodiment shown in , before step 103, may further include the following steps, such as Figure 4 As shown in:

[0094] Step 104: Determine whether the first data in the first database is deleted successfully. If the first data in the first database is deleted successfully, execute step 103 and subsequent steps. If the first data in the first database is not deleted successfully, execute step 105.

[0095] Specifically, in this embodiment, the execution result of step 102 can be monitored to determine whether the first data in the first database is deleted successfully.

[0096] Step 105: Record the first identifier corresponding to the first data in the record table to be deleted, and execute step 103 and subsequent steps.

[0097] The identifier in the record table to be deleted is used to delete the corresponding data. Therefore, if the first data in the first database is not deleted successfully, the corresponding data can be deleted again through the identifier recorded in the record table to ensure the consistency of the first data between the first database and the second database.

[0098] Specifically, the content recorded in the to-be-deleted record table may be the first identifier itself, or may be content corresponding to the first identifier, such as a primary key and / or a table name, etc., to point to the first data, such as the user's stored data. For example, in the data processing process of a financial service, the customer number and the K-treasure binding table are recorded in the to-be-deleted record table to point to a piece of the user's financial stored data.

[0099] Furthermore, after step 105, the present embodiment may further include the following steps: Figure 5 As shown in:

[0100] Step 106: Determine whether the first identifier is recorded successfully in the record table to be deleted. If the first identifier is recorded successfully in the record table to be deleted, execute step 103 to process the first data corresponding to the first identifier in the second database corresponding to the electronic device; if the first identifier is recorded in the record table to be deleted, end the current processing flow of the data processing instruction.

[0101] That is to say, in this embodiment, if recording the first identifier to the record table to be deleted fails, it indicates that the first data in the first database cannot be successfully deleted through the record table to be deleted subsequently. Based on this, continuing the data processing process according to the data processing instruction will cause inconsistency between the first database and the second database regarding the first data. Therefore, if recording the first identifier in the record table to be deleted fails, then the current processing process is terminated and the response to the data processing instruction will no longer continue to avoid inconsistency between the first database and the second database regarding the first data.

[0102] Based on the above implementation, in this embodiment, it is possible to monitor whether the deletion condition corresponding to the record table to be deleted is satisfied, and then delete the corresponding data based on the record table to be deleted, specifically in the following steps: Figure 6 As shown in:

[0103] Step 601: Determine whether the deletion condition corresponding to the record table to be deleted is satisfied. If the deletion condition corresponding to the record table to be deleted is satisfied, execute step 602. If the deletion condition corresponding to the record table to be deleted is not satisfied, return to continue executing step 601 until the deletion condition is satisfied.

[0104] The deletion condition includes: the current duration recorded by the timer reaches a preset duration threshold; the timer restarts when the current duration recorded by the timer reaches the duration threshold. Specifically, the timer can start timing when the to-be-deleted record table is created, and restarts timing when the current duration of each record reaches the duration threshold. That is, in this embodiment, the corresponding deletion processing flow, i.e., step 602-step 603, can be executed once for the to-be-deleted record table every preset duration threshold.

[0105] It should be noted that step 601 is continuously executed, that is, if the deletion condition corresponding to the record table to be deleted is met, it will return to continue executing step 601, and the execution of step 601 is not restricted by whether steps 602 and 603 are executed.

[0106] Step 602: Read at least one target identifier from the record to be deleted.

[0107] Specifically, this embodiment can read one or more target identifiers in the to-be-deleted record table according to a first-in-first-out or first-in-last-out rule. Alternatively, this embodiment can read one or more target identifiers in the to-be-deleted record table according to a reading order corresponding to a preset reading rule.

[0108] Step 603: Delete the target data corresponding to the target identifier in the first database.

[0109] In this embodiment, the corresponding target data can be searched in the first database according to the target identifier, and then the found target data can be deleted.

[0110] Furthermore, after deleting the target data corresponding to the target identifier in the first database, the corresponding target identifier in the to-be-deleted record table is also deleted to avoid wasting operating resources due to repeated deletion operations.

[0111] Specifically, the target identifier read from the to-be-deleted record table can be stored in the target list to implement caching of the target list. Based on this, step 603 can be implemented in the following manner, such as Figure 7 As shown:

[0112] Step 701: determine whether the target list is empty. If the target list is empty, end the current processing flow of the record table to be deleted. If the target list is not empty, execute step 702.

[0113] Step 702: According to the target list, delete the target data corresponding to the target identifier in the first database.

[0114] Specifically, in this embodiment, the target identifiers in the target list may be traversed, and then each time a target identifier in the target list is traversed, the target data corresponding to the traversed target identifier is deleted from the first database.

[0115] Based on the above implementation, before step 602, this embodiment may further include the following steps: Figure 8 As shown in:

[0116] Step 604: determine whether the first database is in the target state. If the first database is in the target state, execute step 602; if the first database is not in the target state, end the current processing flow of the record table to be deleted.

[0117] The first database in the target state is capable of reading and writing data, and the target state can also be understood as the normal operating state of the first database.

[0118] That is to say, if the first database is not in normal operation, the subsequent data reading, writing and deletion may be unstable, resulting in unsuccessful deletion of data in the first database. Therefore, if the first database is not in normal operation, the current processing flow of the record table to be deleted is terminated until the first database is in normal operation, and then step 602 is executed to delete the corresponding data in the first database according to the record table to be deleted.

[0119] based on Figure 3 In the implementation shown in the figure, before step 302, the present embodiment may further include the following steps: Fig. 9 As shown in:

[0120] Step 307: determine whether the first database is in the target state; if the first database is in the target state, execute step 302; if the first database is not in the target state, end the current processing flow of the data read instruction.

[0121] The first database in the target state can read and write data. That is to say, if the first database is not in a normal operating state, the subsequent data search processing may be unstable, resulting in inaccurate results of data search in the first database. Therefore, if the first database is not in a normal operating state, the current processing flow of the data read instruction is terminated until the first database is in a normal operating state, and then step 302 is executed to search the first database for the second data corresponding to the second identifier.

[0122] refer to Fig.10 , is a schematic diagram of the structure of a database synchronization device provided in the second embodiment of the present application, which can be applied to electronic devices configured with heterogeneous databases and capable of data processing, such as servers, etc. The technical solution in this embodiment is mainly used to ensure data consistency between heterogeneous databases during data processing.

[0123] Specifically, the device in this embodiment may include the following units:

[0124] The instruction obtaining unit 1001 is used to obtain a data processing instruction, wherein the data processing instruction at least includes a first identifier of the data to be processed;

[0125] A data deleting unit 1002 is used to delete first data corresponding to the first identifier in a first database corresponding to the electronic device;

[0126] A data processing unit 1003 is used to process the first data corresponding to the first identifier in a second database corresponding to the electronic device; the second database is different from the first database in terms of database structure;

[0127] The data writing unit 1004 is used to write the processed first data in the second database into the first database, so that the first database and the second database are consistent with each other at least with respect to the first data.

[0128] It can be seen from the above technical scheme that a database synchronization device provided in the second embodiment of the present application, when obtaining a data processing instruction, first deletes the first data corresponding to the first identifier of the data to be processed in the first database, and then processes the first data corresponding to the first identifier in the second database heterogeneous with the first database, and then the processed first data in the second database is written into the first database, and the first database and the second database can maintain consistency in the first data. It can be seen that in the data processing process of the present application, by first deleting the data in the first database, and then processing the data in the second database, and then writing the processed data in the second database into the first database, the data consistency between the first database and the first database can be guaranteed, and the situation of data inconsistency between heterogeneous databases during the data processing process can be avoided.

[0129] In one implementation, the instruction obtaining unit 1001 is further used to obtain a data reading instruction, wherein the data reading instruction at least includes a second identifier of the data to be read. Based on this, the device in this embodiment may further include the following units, such as Fig.11 As shown in:

[0130] The data search unit 1005 is configured to search the first database for second data corresponding to the second identifier; if the second data corresponding to the second identifier is not found in the first database, search the second database for second data corresponding to the second identifier; if the second data corresponding to the second identifier is found in the second database, return the second data found in the second database;

[0131] Based on this, the data writing unit 1004 writes the second data found in the second database into the first database.

[0132] In one implementation, before the data processing unit 1003 processes the first data corresponding to the first identifier, the data processing unit 1003 is further used to: determine whether the first data in the first database is deleted successfully; if the first data in the first database is deleted successfully, process the first data corresponding to the first identifier in the second database corresponding to the electronic device; if the first data in the first database is not deleted successfully, record the first identifier corresponding to the first data in a to-be-deleted record table, and process the first data corresponding to the first identifier in the second database corresponding to the electronic device;

[0133] The identifier in the to-be-deleted record table is used to delete the corresponding data.

[0134] In one implementation, after the data processing unit 1003 records the first identifier corresponding to the first data in the record table to be deleted, it is also used to: determine whether the first identifier is successfully recorded in the record table to be deleted; if the first identifier is successfully recorded in the record table to be deleted, process the first data corresponding to the first identifier in the second database corresponding to the electronic device; if the first identifier fails to be recorded in the record table to be deleted, end the current processing flow of the data processing instruction.

[0135] In one implementation, the data processing unit 1003 is also used to: read at least one target identifier from the record table to be deleted when the deletion condition corresponding to the record table to be deleted is met; delete the target data corresponding to the target identifier in the first database; wherein the deletion condition includes: the current duration recorded by the timer reaches a preset duration threshold; the timer restarts the timing when the current duration recorded by the timer reaches the duration threshold.

[0136] Furthermore, the target identifier is stored in a target list; when the data processing unit 1003 deletes the target data corresponding to the target identifier in the first database, it is specifically used to: determine whether the target list is empty; if the target list is empty, end the current processing flow of the record table to be deleted; if the target list is not empty, delete the target data corresponding to the target identifier in the first database according to the target list.

[0137] In one implementation, before reading at least one target identifier from the record table to be deleted, the data processing unit 1003 is also used to: determine whether the first database is in a target state, in which the first database can read and write data; if the first database is in the target state, read at least one target identifier from the record table to be deleted; if the first database is not in the target state, end the current processing flow of the record table to be deleted.

[0138] In one implementation, before searching the first database for the second data corresponding to the second identifier, the data processing unit 1003 is also used to: determine whether the first database is in a target state, in which the first database can read and write data; if the first database is in the target state, search the first database for the second data corresponding to the second identifier; if the first database is not in the target state, end the current processing flow of the data reading instruction.

[0139] It should be noted that the specific implementation of each unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0140] refer to Fig.12 , is a schematic diagram of the structure of an electronic device provided in the third embodiment of the present application, and the electronic device may be an electronic device configured with a heterogeneous database and capable of data processing, such as a server, etc. The technical solution in this embodiment is mainly used to ensure data consistency between heterogeneous databases during data processing.

[0141] Specifically, the electronic device in this embodiment may include the following structure:

[0142] Memory 1201, used to store computer programs and data generated by the execution of the computer programs;

[0143] Processor 1202 is used to execute the computer program to achieve: obtaining a data processing instruction, wherein the data processing instruction includes at least a first identifier of the data to be processed; deleting the first data corresponding to the first identifier in a first database corresponding to the electronic device; processing the first data corresponding to the first identifier in a second database corresponding to the electronic device; the second database is different from the first database in terms of database structure; wherein the processed first data in the second database is used to write into the first database, so that the first database and the second database are consistent with each other at least with respect to the first data.

[0144] It can be seen from the above technical scheme that an electronic device provided in Example 3 of the present application, when obtaining a data processing instruction, first deletes the first data corresponding to the first identifier of the data to be processed in the first database, and then processes the first data corresponding to the first identifier in the second database heterogeneous with the first database, and then the processed first data in the second database is written into the first database, and the first database and the second database can be consistent in the first data. It can be seen that in the data processing process of the present application, by first deleting the data in the first database, and then processing the data in the second database, and then writing the processed data in the second database into the first database, the data consistency between the first database and the first database can be guaranteed, and the situation of data inconsistency between heterogeneous databases during the data processing process can be avoided.

[0145] Taking the application server configured with Redis database and Oracle database as an example, the technical solution of this application is described in detail below:

[0146] First of all, for multi-database systems, data consistency is divided into two types, strong consistency and weak consistency or eventual consistency. The corresponding processing strategy for strong consistency is that when the cache deletion fails, the transaction fails. Its disadvantage is that it intrudes on transactions, and the system is greatly affected by the stability of redis. The weak consistency or eventual consistency processing strategy is that when the cache deletion fails, the transaction is successful first, and then a compensation mechanism is used to delete the old data. Its disadvantage is that the old data will be read before the old data in redis is actually deleted, which may cause data inconsistency. In response to this, the present application proposes a solution, namely: in non-core derivative service scenarios that can tolerate a small amount of data loss, in the balance between high availability and data consistency, the purpose of high service availability is achieved by sacrificing data consistency for a short period of time with a small probability. Specifically, three core algorithms are involved, which are described in detail as follows:

[0147] by Fig.13 Take the client access between the application server and the Redis database and Oracle database as an example. Fig.14 As shown in the figure, it is a flowchart for reading customer data, which is as follows:

[0148] First, determine the working status of the Redis database. If it is abnormal, that is, it is judged as false, then read the data in the Oracle database; if the status is normal, that is, it is judged as true, then read the data in the Redis database, such as user information, etc. If the record exists, that is, it is judged as true, then return the obtained information and the program ends; if the user's information does not exist in the Redis database, that is, it is judged as false, then read the user information recorded in the Oracle database, and write the user information obtained from the Oracle database into the Redis database.

[0149] like Fig.15 As shown in the figure, it is a flowchart of customer data update, that is, the data processing flow in the previous text, which is as follows:

[0150] First, the business data is processed on the application server to prepare the fields that need to be written to the database, that is, the first data corresponding to the first identifier in the data processing instruction; then, the Redis database, that is, the Redis cache, is maintained, that is, the first data in the Redis database is deleted. If the maintenance fails, that is, it is judged as false, then it is recorded in the to-be-deleted record table, that is, the to-be-deleted table in the figure, and if the to-be-deleted table fails to be recorded, the transaction fails, that is, the data update fails, and this transaction business ends; if the to-be-deleted table is recorded successfully or the Redis database maintenance is successful, that is, it is judged as true, then the Oracle business table is processed, that is, the first data is processed in the Oracle database in the previous article.

[0151] Based on the above implementation, Fig.16 As shown in the figure, it is a flow chart of the compensation batch program, that is, the process of deleting data according to the record table to be deleted in the previous text, which is as follows:

[0152] First, determine the working status of the Redis cache database. If the status is abnormal, end the current process. If the status is normal, go to the record table to be deleted to get the first 100 (or other numbers) records for processing. The records obtained are stored in the list delList, which is the target list mentioned above. If the target list obtained is empty, that is, if deList is not empty or false, it means that there are no records in the record table to be deleted, and the program ends. If the list is not empty, that is, if deList is not empty or true, traverse the records in the list delList and delete the corresponding records from the Redis database. If the deletion is successful, delete this record in the record table to be deleted, so that the application server can get the latest value when it accesses this data next time. According to this process, the program ends until the table to be deleted is judged to be empty and true. Otherwise, it returns when the table to be deleted is judged to be empty or false, so as to get the first 100 records from the table to be deleted again and record them in deList, and so on, until the table to be deleted is empty.

[0153] It should be noted that Fig.16 The process shown in can be set to run once at a short interval (for example, 1 minute).

[0154] It can be seen that when the client reads data on the application server, if the Redis database is hit, it will return directly, avoiding access to the Oracle database. The more frequent the user, the greater the hit probability. If the Redis database is not hit, the Oracle database will be accessed, and the data obtained in Oracle will be written to the Redis database. The next time the user comes to access, the Redis database will be hit first.

[0155] There are two ways to maintain the Redis database, one is to modify and the other is to delete. In this embodiment, the deletion method is adopted. After the data in the Redis database is deleted, the user will read the latest value from the Oracle database and store it in the cache when he visits next time, and the user will also obtain the latest value.

[0156] When customer data is updated, the Redis database is maintained first, that is, the corresponding records in redis are directly deleted. If the record in the Redis database fails to be deleted, or the Redis database is unavailable at this time, the record will be added to the table of records to be deleted. This table exists in the Oracle database. If adding records to the table fails, the entire transaction will fail, otherwise data loss will occur. In other words, the success or failure of Redis database maintenance does not affect the system's external services, and the final consistency of data can be achieved. The program continues to execute and maintain the user information in the Oracle database. If this step of the program fails, it does not matter if the cache in redis was deleted or recorded in the table to be deleted in the previous step. Deleting more data (compared to the Oracle database) or deleting redis cache data multiple times will not affect the function of the system.

[0157] The compensation batch program in this embodiment is set to be executed once every short time. When executing, the data in the record table to be deleted is taken to try to delete the records in the Redis database again. It is used as compensation for the failure of Redis database maintenance when the user is in the data update program or when the Redis database service is unavailable. Generally, the amount of data in the record table to be deleted is not too large.

[0158] The above three algorithms work together to prioritize access to the Redis database when the Redis database service is available, and access the Oracle database as a backup when the service is unavailable or the Redis database is not hit. This fully utilizes the advantage of the Redis database's fast reading speed, avoids the disadvantage of the Redis database's unstable service, and ultimately achieves data consistency between the two databases.

[0159] It can be seen that in this embodiment, both heterogeneous databases can be accessed and used by applications at the same time, providing real-time services in most cases. Only when the Redis database fails to delete due to extreme circumstances such as hardware instability, that is, with a very low probability, will there be a short-term data inconsistency. The subsequent batch compensation mechanism is simple and easy to implement, and can achieve the final consistency of data in a very short time. This ensures the high availability of the system's external services.

[0160] Moreover, the typical CacheAside mode is improved in this embodiment. In the process of modifying customer data, the cache is processed before maintaining the Oracle database. If the maintenance fails, it is recorded in the record table to be deleted. If the record table fails again, the transaction failure strategy is set to avoid adding new transactions to the program, which can ensure the final consistency of the data. Imagine if the Oracle database is maintained first, if it succeeds, and then the maintenance of the Redis database fails, and then the record table to be deleted fails again, the entire process should be put into the same transaction here, and the maintenance operation of the Oracle database user data should be rolled back or revoked, otherwise the records that should be deleted will be lost. For the increasingly widespread data sub-library storage, it is difficult to do if the transaction crosses the database. It is relatively simple to execute transactions in a single-user, single-database environment, but it is more complicated to maintain the integrity of multiple databases in a distributed environment. The strategy proposed in this embodiment avoids the transaction consistency problem between recording in the record table to be deleted and processing the Oracle database user data. At the same time, it also avoids the unnecessary writing and rollback or revocation of Oracle database user data, as well as the overhead of the Oracle database and the problem of operation failure caused by these operations.

[0161] In summary, this application adopts a weak consistency strategy and corresponding compensation mechanism when reading and modifying customer data to ensure high availability of services. It also provides a customer data modification algorithm and a compensation batch program algorithm. In addition, the improvement of the typical CacheAside mode of the Redis database ensures that the Redis database operation itself does not affect the normal business process, and can trace and compensate for failed Redis database operations.

[0162] In addition, it should be noted that the technical solution in this embodiment can be implemented in the form of hardware or in the form of software functional units in the log monitoring server, message queue cluster, and data synchronization server.

[0163] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0164] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0165] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0166] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A database synchronization method, characterized in that: include: Obtaining a data processing instruction, wherein the data processing instruction at least includes a first identifier of the data to be processed; Deleting first data corresponding to the first identifier in a first database corresponding to the electronic device; Processing, in a second database corresponding to the electronic device, first data corresponding to the first identifier; The second database is different from the first database in terms of database structure; The processed first data in the second database is used to write into the first database, so that the first database and the second database are consistent with each other at least with respect to the first data; Before processing the first data corresponding to the first identifier, the method further includes: Determining whether the first data in the first database is deleted successfully; If the first data in the first database is deleted successfully, executing the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device; If the first data in the first database is not deleted successfully, the first identifier corresponding to the first data is recorded in a to-be-deleted record table, and the steps are performed: in a second database corresponding to the electronic device, the first data corresponding to the first identifier is processed; Wherein, the identifier in the record table to be deleted is used to delete the corresponding data; After recording the first identifier corresponding to the first data in the to-be-deleted record table, the method further includes: Determine whether the first identifier is successfully recorded in the to-be-deleted record table; If the first identifier is successfully recorded in the to-be-deleted record table, the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device; If recording the first identifier in the to-be-deleted record table fails, the current processing flow of the data processing instruction is terminated.

2. The method according to claim 1, characterized in that: The method further comprises: Obtaining a data reading instruction, wherein the data reading instruction at least includes a second identifier of the data to be read; Searching the first database for second data corresponding to the second identifier; If the second data corresponding to the second identifier is not found in the first database, searching the second database for the second data corresponding to the second identifier; If the second data corresponding to the second identifier is found in the second database, the second data found in the second database is returned; The second data found in the second database is written into the first database.

3. The method according to claim 1, characterized in that The method further comprises: When a deletion condition corresponding to the to-be-deleted record table is satisfied, reading at least one target identifier from the to-be-deleted record table; Deleting the target data corresponding to the target identifier in the first database; The deletion condition includes: the current duration recorded by the timer reaches a preset duration threshold; the timer restarts when the current duration recorded by the timer reaches the duration threshold.

4. The method according to claim 3, characterized in that The target identification is stored in a target list; Wherein, deleting the target data corresponding to the target identifier in the first database includes: Determine whether the target list is empty; If the target list is empty, the current processing flow of the to-be-deleted record list is terminated; If the target list is not empty, the target data corresponding to the target identifier is deleted from the first database according to the target list.

5. The method according to claim 3, characterized in that: Before reading at least one target identifier from the to-be-deleted record table, the method further includes: Determine whether the first database is in a target state, where the first database in the target state can read and write data; If the first database is in the target state, executing the steps of: reading at least one target identifier from the to-be-deleted record table; If the first database is not in the target state, the current processing flow of the to-be-deleted record table is terminated.

6. The method according to claim 2, characterized in that Before searching the first database for second data corresponding to the second identifier, the method further includes: Determine whether the first database is in a target state, where the first database in the target state can read and write data; If the first database is in the target state, executing the steps of: searching the first database for second data corresponding to the second identifier; If the first database is not in the target state, the current processing flow of the data reading instruction is terminated.

7. A database synchronization device, characterized in that: include: An instruction obtaining unit, used for obtaining a data processing instruction, wherein the data processing instruction at least includes a first identifier of the data to be processed; A data deleting unit, configured to delete first data corresponding to the first identifier in a first database corresponding to the electronic device; A data processing unit, configured to process first data corresponding to the first identifier in a second database corresponding to the electronic device; The second database is different from the first database in terms of database structure; a data writing unit, configured to write the processed first data in the second database into the first database, so that the first database and the second database are consistent with each other at least with respect to the first data; Before the data processing unit processes the first data corresponding to the first identifier, the method is further configured to: Determining whether the first data in the first database is deleted successfully; If the first data in the first database is deleted successfully, executing the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device; If the first data in the first database is not deleted successfully, the first identifier corresponding to the first data is recorded in the to-be-deleted record table, and the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device; Wherein, the identifier in the record table to be deleted is used to delete the corresponding data; After recording the first identifier corresponding to the first data in the to-be-deleted record table, the data processing unit is further configured to: Determine whether the first identifier is successfully recorded in the to-be-deleted record table; If the first identifier is successfully recorded in the to-be-deleted record table, the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device; If recording the first identifier in the to-be-deleted record table fails, the current processing flow of the data processing instruction is terminated.

8. An electronic device, characterized in that: include: A memory, used to store a computer program and data generated by the execution of the computer program; A processor, configured to execute the computer program to implement: obtaining a data processing instruction, wherein the data processing instruction at least includes a first identifier of data to be processed; deleting first data corresponding to the first identifier in a first database corresponding to the electronic device; processing the first data corresponding to the first identifier in a second database corresponding to the electronic device; the second database and the first database are different in database structure; wherein the processed first data in the second database are used to write into the first database, so that the first database and the second database are consistent with each other at least in terms of the first data; Before the processor processes the first data corresponding to the first identifier, the processor is further configured to: Determining whether the first data in the first database is deleted successfully; If the first data in the first database is deleted successfully, executing the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device; If the first data in the first database is not deleted successfully, the first identifier corresponding to the first data is recorded in the to-be-deleted record table, and the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device; Wherein, the identifier in the record table to be deleted is used to delete the corresponding data; After recording the first identifier corresponding to the first data in the to-be-deleted record table, the processor is further configured to: Determine whether the first identifier is successfully recorded in the to-be-deleted record table; If the first identifier is successfully recorded in the to-be-deleted record table, the steps of: processing the first data corresponding to the first identifier in the second database corresponding to the electronic device; If recording the first identifier in the to-be-deleted record table fails, the current processing flow of the data processing instruction is terminated.

Images