Method and System for Data Synchronization between Relational Database and Cache Database

Abstract

The present invention discloses a method for data synchronization between a relational database and a cache database. It receives a transaction start instruction issued by a client, parses the storage transaction included in the transaction start instruction to generate data storage information; separates the data storage information to generate first data storage information and second data storage information; sends the first data storage information to the relational database for temporary storage, and sends the second data storage information to a preset cache container; detects whether a data storage transaction commit instruction issued by the client is received. If the data storage transaction commit instruction is detected, all the second data storage information in the cache container is sent to the cache database to achieve synchronous writing of the relational database and the cache database. Thus, a real-time synchronization mechanism between the relational database and the cache database can be achieved, thereby improving the development and operation and maintenance efficiency of enterprises in the face of high-concurrency scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular, to a method and system for data synchronization between a relational database and a cache database. Background Art

[0002] A cache is a mechanism of computer memory and is the fastest access unit in a computer, which can provide a speed hundreds or thousands of times faster than a disk. Therefore, the cache has gradually become the core data capability to relieve the pressure on the database. In software development of computing, to meet the requirements of high-concurrency scenarios, data is usually written to the cache by code first and then written to the relational database. If the write to the relational database fails, the cache is cleared, which is equivalent to operating two databases simultaneously. To ensure the synchronization and consistency between the two, various complex processing mechanisms are used, but the data consistency management of the two databases still needs to be maintained, which will lead to complex operation and maintenance management, and the data consistency management is vulnerable to the quality of development code.

[0003] Currently, the client is mainly responsible for saving to the cache. When updating the cache, the relational database is updated first, and then the cache is set with an expiration time. Although this method can take effect in real time, it still cannot solve the problem of transaction consistency, and there are latency problems in database and client operations, and it cannot solve the ability to synchronize the cache for a large amount of data. In addition, there is also a method of using an external service to capture the changed data of the database and write it to the cache to achieve data consistency management. However, this method belongs to asynchronous replication and the real-time performance cannot be guaranteed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for data synchronization between a relational database and a cache database, which can realize a real-time synchronization mechanism between the relational database and the cache database, and further improve the development and operation and maintenance efficiency of enterprises in the face of high-concurrency scenarios.

[0005] To solve the above technical problem, in a first aspect of the present invention, a method for data synchronization between a relational database and a cache database is disclosed. The method includes: receiving a transaction start instruction sent by a client, parsing the storage transaction included in the transaction start instruction to generate data storage information; separating the data storage information to generate first data storage information and second data storage information; sending the first data storage information to the relational database for temporary storage, and sending the second data storage information to a preset cache container; detecting whether a data storage transaction commit instruction sent by the client is received. If the data storage transaction commit instruction is detected, all the second data storage information in the cache container is sent to the cache database to achieve synchronous writing of the relational database and the cache database.

[0006] In some embodiments, receiving the transaction start instruction sent by the client includes: configuring a virtual database connected to the client; and receiving the transaction start instruction sent by the client through the virtual database.

[0007] In some embodiments, separating the data storage information to generate first data storage information and second data storage information includes: copying the data storage information to generate two pieces of first data storage information; parsing one of the first data storage information to generate multiple pieces of second data storage information; wherein each piece of second data storage information has a matching cache container.

[0008] In some embodiments, sending all the second data storage information in the cache container to the cache database to achieve synchronous writing of the relational database and the cache database includes: after sending all the second data storage information in the cache container to the cache database, sending a data submission instruction to the relational database and the cache database; the relational database writing all the temporarily stored first data storage information according to the data submission instruction; and the cache database writing all the second data storage information according to the data submission instruction.

[0009] A second aspect of the present invention discloses a system for data synchronization between a relational database and a cache database. The system includes: a parsing module for receiving a transaction start instruction sent by a client and parsing the storage transaction included in the transaction start instruction to generate data storage information; a database operation separation device for separating the data storage information to generate first data storage information and second data storage information; a relational database for receiving the first data storage information and sending it to the relational database for temporary storage, and a cache container for receiving the second data storage information; and a synchronization module for detecting whether a data storage transaction submission instruction sent by the client is received. If the data storage transaction submission instruction is detected, all the second data storage information in the cache container is sent to the cache database to achieve synchronous writing of the relational database and the cache database.

[0010] In some embodiments, the system further includes: a virtual database connected to the client, and the virtual database is used for receiving the transaction start instruction sent by the client and transmitting it to the parsing module.

[0011] In some embodiments, the database operation separation device is implemented as: copying the data storage information to generate two pieces of first data storage information; parsing one of the first data storage information to generate multiple pieces of second data storage information; wherein each piece of second data storage information has a matching cache container.

[0012] In some embodiments, the synchronization module is implemented as follows: after sending all the second data storage information of the cache container to the cache database, a data submission instruction is sent to the relational database and the cache database; the relational database writes all the temporarily stored first data storage information according to the data submission instruction; the cache database writes all the second data storage information according to the data submission instruction.

[0013] A third aspect of the present invention discloses a device for synchronizing data between a relational database and a cache database, the device comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory and executes the method for synchronizing data between a relational database and a cache database as described above.

[0014] A fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which are used to execute the method for synchronizing data between a relational database and a cache database as described above when the computer instructions are called.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0016] Implementing the present invention can integrate the data synchronization technologies of heterogeneous databases, shield the problem of judging the operation consistency of two databases, namely the relational database and the cache database, in development, and achieve data consistency between the cache database and the entity relational database in the virtual database. Moreover, due to the transaction consistency guarantee mechanism of the virtual database, it can provide consistent data processing results for the client on the premise that both databases are successfully executed. Furthermore, the data operation separation device according to the present invention can automatically separate the data storage information and utilize the cache container to realize the memory management of operations. Thus, a real-time synchronization mechanism between the relational database and the cache database can be achieved, thereby improving the development and operation and maintenance efficiency of enterprises in the face of high-concurrency scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic flowchart of a method for synchronizing data between a relational database and a cache database disclosed in an embodiment of the present invention;

[0018] Figure 2 It is a schematic diagram of a system for synchronizing data between a relational database and a cache database disclosed in an embodiment of the present invention;

[0019] Figure 3 It is another schematic diagram of a system for synchronizing data between a relational database and a cache database disclosed in an embodiment of the present invention;

[0020] Figure 4Schematic structural diagram of a data synchronization device for a relational database and a cache database disclosed in an embodiment of the present invention. Detailed implementation manners

[0021] For better understanding and implementation, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0022] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or modules does not necessarily limit to those clearly listed steps or modules, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or devices.

[0023] The embodiments of the present invention disclose a method and system for data synchronization between a relational database and a cache database, which can integrate the data synchronization technologies of heterogeneous databases, shield the problem of judging the operation consistency of two databases, namely the relational database and the cache database, in development, and achieve the data consistency between the cache database and the entity relational database in the virtual database. Moreover, due to the transaction consistency guarantee mechanism of the virtual database, it can provide a consistent data processing result for the client on the premise that both databases are successfully executed. Moreover, the data operation separation device according to the present invention can automatically separate the data storage information and use the cache container to implement the memory management of operations. Therefore, a real-time synchronization mechanism between the relational database and the cache database can be achieved, thereby improving the development and operation and maintenance efficiency of enterprises in the face of high-concurrency scenarios.

[0024] Embodiment 1

[0025] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a method for data synchronization between a relational database and a cache database disclosed in an embodiment of the present invention. Among them, the method for data synchronization between the relational database and the cache database can be applied to a data storage system in which the relational database and the cache database coexist, and can also include a synchronization system of other types of databases. The embodiments of the present invention do not limit the application of this method. As Figure 1 shown, the method for data synchronization between the relational database and the cache database may include the following operations:

[0026] 101. Receive the transaction start instruction sent by the client, and parse the storage transaction included in the transaction start instruction to generate data storage information.

[0027] In the interaction with the database, the international SQL standard stipulates that the interaction between the client and the database is mainly divided into three major parts: starting a transaction, repeatedly executing various database operations (in SQL form), and committing / rolling back the transaction and ending. After starting a transaction, the client can continuously request database operations through syntax. These operations are subject to transaction consistency constraints, that is, other clients cannot know the content of the data operations. For example, the database operations written at this time have actually been committed to the database storage, but are marked as uncommitted through tags. After the client initiates transaction submission, a series of database operations will be submitted to the database change tags as an "atomic" method. If the write is successful, the tag will show that all writes are successful. If the write fails, whether it is a single operation failure or all operation failures, it will be counted as all failures. This process is also known as the strong consistency of database transactions.

[0028] However, the inventor found that there is no such strong consistency transaction mode in the cache database as in the relational database. Therefore, in order to ensure the consistency of data storage operations in the relational database and the cache database, it is necessary to perform a one-time cache "atomic" operation before transaction submission. Compared with the time consumption and failure rate of database operations, the abnormal probability of database transaction submission is the lowest. Given that the efficiency of the cache is much higher than that of the database, putting the most efficient and safest operations at the end is the best process to ensure front-back consistency. This is the source of the main technical concept of the present invention by the inventor.

[0029] First, receive the transaction start instruction sent by the client. This transaction start instruction includes the storage transaction to be written, and then parse the SQL syntax of the storage transaction according to the SQL parsing rules for subsequent unified operations.

[0030] As a preferred implementation, a virtual database connected to the client is also configured. The transaction start instruction sent by the client is received through this virtual database. The advantage of this is that it can prevent the client from directly contacting the relational database and the cache database, eliminating the need for the client development to switch back and forth between these two databases and manage exceptions, and only by accessing this virtual database, thus simplifying the complexity of development and operation and maintenance. Moreover, the virtual database has a transaction consistency guarantee mechanism, which can provide consistent data processing results for the client on the premise that both databases are successfully executed.

[0031] 102. Separate the data storage information to generate the first data storage information and the second data storage information.

[0032] In order to process data storage information separately, during the processing stage of data storage information, the operations of the database and the cache are separated. The specific implementation is as follows: First, the data storage information is copied to generate two identical first data storage information. The first data storage information refers to the data storage information itself. Then, one of the first data storage information is parsed to generate multiple second data storage information. The parsing method is divided according to the type of data storage information. Exemplarily, if the content of a certain data storage information is of the "apple" category, then when parsing this "apple", multiple pieces of data such as "apple core", "apple peel", and "apple flesh" will be generated. Among them, "apple core", "apple peel", and "apple flesh" can all be used as a kind of second data storage information. An appropriate cache container is configured for each second data storage information, which is convenient for subsequent unified data management. However, in other embodiments, in order to reduce the internal cache space, all the second data storage information can also be stored in the same cache container. Through this data separation method, it can be ensured that the operations of the cache and the database are completed within the same transaction cycle.

[0033] 103. Send the first data storage information to the relational database for temporary storage, and send the second data storage information to the preset cache container.

[0034] After that, sending the first data storage information to the relational database for temporary storage does not actually commit to achieve the purpose of saving the original data storage information. At the same time, the second data storage information is sent to the preset cache container to handle the change operations of the client during this transaction processing cycle. Among them, for the implementation method of the cache container, the lifecycle methods init() and beforeDestory() can be used for definition. Exemplarily, at the very beginning when the container starts, the init() method is executed. In the init() method, loadCacheFromDisk() reloads the cache saved before normal shutdown into the memory. Finally, a background thread is started. The main work of the background thread is to periodically clean up expired cache and take snapshots of the data at an appropriate time.

[0035] Through the above separation operation and storage in the cache container, the process of data synchronization between the cache and the database can be made imperceptible to the client, and data consistency will not be inconsistent due to the client's self-switching.

[0036] 104. Detect whether a data storage transaction commit instruction is received from the client. If the data storage transaction commit instruction is detected, send all the second data storage information in the cache container to the cache database to achieve synchronous writing of the relational database and the cache database.

[0037] As the last link in implementing the entire synchronization process, within a transaction cycle, the instructions sent by the client can be polled until the data storage transaction commit instruction sent by the client is detected, indicating that the transaction in this stage has been completely completed and no longer involves changes. After the data transaction can be committed, the second data storage information in the cache container is uniformly written into the cache database at one time, and the relational database is requested to commit the transaction. Specifically, after all the second data storage information in the cache container is sent to the cache database, a data commit instruction is sent to the relational database and the cache database. The relational database writes all the temporarily stored first data storage information according to the data commit instruction, and the cache database writes all the second data storage information according to the data commit instruction.

[0038] The method disclosed in this embodiment can integrate the data synchronization technologies of heterogeneous databases, shield the problem of judging the operation consistency of two databases, namely the relational database and the cache database, in development, and achieve the data consistency between the cache database and the physical relational database in the virtual database. Moreover, due to the transaction consistency guarantee mechanism of the virtual database, it can provide a consistent data processing result for the client on the premise that both databases are successfully executed. Moreover, the data operation separation device according to the present invention can automatically separate the data storage information and use the cache container to implement the memory management of the operation. Thus, a real-time synchronization mechanism between the relational database and the cache database can be achieved, thereby improving the development and operation and maintenance efficiency of enterprises in the face of high-concurrency scenarios.

[0039] Embodiment 2

[0040] Please refer to Figure 2 , Figure 2 which is a schematic diagram of a data synchronization system for a relational database and a cache database disclosed in an embodiment of the present invention. As Figure 2 shown, the system for data synchronization between a relational database and a cache database may include:

[0041] Parsing module 1, configured to receive a transaction start instruction sent by a client, and parse the storage transaction included in the transaction start instruction to generate data storage information. The specific implementation is: receiving the transaction start instruction sent by the client, where the transaction start instruction includes the storage transaction to be written, and then parsing the SQL syntax of the storage transaction according to the SQL parsing rules for subsequent unified operations.

[0042] The database operation separation device 2 is used to separate the data storage information to generate the first data storage information and the second data storage information. The specific implementation is as follows: First, the data storage information is copied to generate two identical first data storage information. The first data storage information refers to the data storage information itself. Then, one of the first data storage information is parsed to generate multiple second data storage information. The parsing method is divided according to the type of data storage information. Exemplarily, if the content of a certain data storage information is of the "apple" category, then when parsing this "apple", multiple pieces of data such as "apple core", "apple peel", and "apple flesh" will be generated. Among them, "apple core", "apple peel", and "apple flesh" can all be used as a kind of second data storage information. An appropriate cache container is configured for each second data storage information, which facilitates subsequent unified data management. However, in other implementation manners, in order to reduce the internal cache space, all the second data storage information can also be stored in the same cache container. Through this data separation method, it can be ensured that the cache operation and the database operation are completed within the same transaction cycle.

[0043] The relational database 3 is used to receive the first data storage information and send it to the relational database for temporary storage. The cache container 4 is used to receive the second data storage information. Sending the first data storage information to the relational database 3 for temporary storage does not actually commit to achieve the purpose of saving the original data storage information. At the same time, the second data storage information is sent to the preset cache container to handle the change operations of the client during this transaction processing cycle. Among them, for the implementation manner of the cache container, the lifecycle methods init() and beforeDestory() can be used for definition. Exemplarily, at the beginning of the container startup, the init() method is executed. In the init() method, loadCacheFromDisk() reloads the cache saved before normal shutdown into the memory. Finally, a background thread is started. The main work of the background thread is to regularly clean up the expired cache and take snapshots of the data at an appropriate time. Through the above separation operation and storage in the cache container, the process of data synchronization between the cache and the database can be made imperceptible to the client, and data consistency will not be inconsistent due to the client's self-switching.

[0044] The synchronization module 5 is used to detect whether a data storage transaction commit instruction issued by the client is received. If a data storage transaction commit instruction is detected, all the second data storage information of the cache container is sent to the cache database 6 to realize synchronous writing of the relational database and the cache database. In a transaction cycle, the instructions sent by the client can be polled until the data storage transaction commit instruction issued by the client is detected, which indicates that the transaction at this stage has been completely completed and no changes are involved. After the data transaction can be committed, the second data storage information in the cache container is uniformly written to the cache database at one time, and the relational database is requested to commit the transaction. Specifically, after all the second data storage information of the cache container is sent to the cache database, a data commit instruction can be sent to the relational database and the cache database. The relational database writes all the temporarily stored first data storage information according to the data commit instruction, and the cache database writes all the second data storage information according to the data commit instruction.

[0045] In other preferred embodiments, Figure 3 As shown, the system also includes a virtual database 6 connected to the client. The virtual database 6 is used to receive transaction start instructions issued by the client and transmit them to the parsing module. This allows the client to avoid direct access to the relational database and cache database, eliminating the need for client development to switch back and forth between the two databases and manage exceptions. Client development can simply access the virtual database, thereby simplifying the complexity of development and operation and maintenance. Moreover, the virtual database has a transaction consistency guarantee mechanism, which can provide the client with consistent data processing results while ensuring successful execution in both databases.

[0046] Therefore, the system disclosed in this embodiment can integrate the data synchronization technology of heterogeneous databases, shield the problem of judging the operational consistency of two databases, namely the relational database and the cached data, during development, and realize the data consistency of the cache database and the entity's relational database in the virtual database. Moreover, since the virtual database has a transaction consistency guarantee mechanism, it can provide the client with consistent data processing results under the premise of ensuring that both databases are executed successfully. Moreover, the data operation separation device according to the present invention can automatically separate the data storage information and use the cache container to realize the memory management of the operation, thereby realizing the real-time synchronization mechanism of the relational database and the cache database, thereby improving the development and operation efficiency of the enterprise in the face of high concurrency scenarios.

[0047] Example 3

[0048] See also Figure 4 , Figure 4 This is a schematic diagram of a data synchronization device for a relational database and a cache database disclosed in an embodiment of the present invention. Figure 4The described data synchronization device for relational databases and cache databases can be applied to systems with two or more databases. The application system of this data synchronization device for relational databases and cache databases is not limited in the embodiments of the present invention. For example, Figure 4 As shown, the device may include:

[0049] A memory 601 storing executable program code;

[0050] A processor 602 coupled to the memory 601;

[0051] The processor 602 invokes the executable program code stored in the memory 601 to execute the method for data synchronization between relational databases and cache databases described in Embodiment 1.

[0052] Embodiment 4

[0053] The embodiments of the present invention disclose a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program causes a computer to execute the method for realizing real-time data synchronization between relational databases and cache databases described in Embodiment 1.

[0054] Embodiment 8

[0055] The embodiments of the present invention disclose a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the method for data synchronization between relational databases and cache databases described in Embodiment 1.

[0056] The above-described embodiments are merely illustrative. The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0057] Through the specific descriptions of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, which includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disc memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0058] Finally, it should be noted that: the method and system for synchronizing data between a relational database and a cache database disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for data synchronization between a relational database and a cache database, characterized in that, The method includes: Receiving a transaction start instruction issued by a client, and parsing the storage transaction included in the transaction start instruction to generate data storage information; Separating the data storage information to generate first data storage information and second data storage information; Separating the data storage information to generate first data storage information and second data storage information includes: Copying the data storage information to generate two pieces of first data storage information; Parsing one of the first data storage information to generate multiple pieces of second data storage information; Wherein, each piece of second data storage information has an adapted cache container; Sending the first data storage information to a relational database for temporary storage, and sending the second data storage information to a preset cache container; Detecting whether a data storage transaction commit instruction issued by the client is received. If the data storage transaction commit instruction is detected, sending all the second data storage information in the cache container to a cache database to achieve synchronous writing of the relational database and the cache database.

2. The method for synchronizing data between a relational database and a cache database according to claim 1, wherein The receiving the transaction start instruction issued by the client includes: Configuring a virtual database connected to the client; Receiving the transaction start instruction issued by the client through the virtual database.

3. The method for data synchronization between a relational database and a cache database according to claim 1, wherein Sending all the second data storage information in the cache container to a cache database to achieve synchronous writing of the relational database and the cache database includes: After sending all the second data storage information in the cache container to the cache database, sending a data commit instruction to the relational database and the cache database; The relational database writing all the temporarily stored first data storage information according to the data commit instruction; The cache database writing all the second data storage information according to the data commit instruction.

4. A system for data synchronization between a relational database and a cache database, characterized in that, The system includes: A parsing module, configured to receive a transaction start instruction issued by a client, and parse the storage transaction included in the transaction start instruction to generate data storage information; A database operation separation device, configured to separate the data storage information to generate first data storage information and second data storage information; The database operation separation device is implemented as: Copying the data storage information to generate two pieces of first data storage information; Parsing one of the first data storage information to generate multiple pieces of second data storage information; Wherein, each piece of second data storage information has an adapted cache container; A relational database, configured to receive the first data storage information and send it to the relational database for temporary storage; A cache container, configured to receive the second data storage information; A synchronization module, configured to detect whether a data storage transaction commit instruction issued by the client is received. If the data storage transaction commit instruction is detected, sending all the second data storage information in the cache container to a cache database to achieve synchronous writing of the relational database and the cache database.

5. The system for data synchronization between a relational database and a cache database according to claim 4, wherein The system further includes: A virtual database connected to the client, where the virtual database is configured to receive the transaction start instruction issued by the client and transmit it to the parsing module.

6. The system for data synchronization between a relational database and a cache database according to claim 4, wherein The synchronization module is implemented as: After sending all the second data storage information of the cache container to the cache database, a data submission instruction is sent to the relational database and the cache database; The relational database writes all the temporarily stored first data storage information according to the data submission instruction; The cache database writes all the second data storage information according to the data submission instruction.

7. A data synchronization device for a relational database and a cache database, characterized in that, The device includes: A memory storing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory and executes the method for data synchronization between the relational database and the cache database according to any one of claims 1-3.

8. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which are used to execute the method for data synchronization between the relational database and the cache database according to any one of claims 1-3 when the computer instructions are called.

Images