Database middleware and method for realizing high-consistency database

Abstract

The invention belongs to the technical field of databases, and particularly discloses database middleware and a method for realizing a high-consistency database. According to the database middleware and the implementation method, automatic monitoring of database consistency is achieved, when it is detected that the database state is abnormal, the recovery assembly can be automatically called to execute the data synchronous recovery operation, and therefore the consistency of the database in the running process is ensured, the requirement for human intervention is reduced, and the safety of the database is improved. And the reliability and the stability of the system are improved. Besides, according to the method provided by the invention, the data synchronization process is simplified, the time required for data synchronization recovery is shortened, the accurate synchronization of the data in the data writing process is ensured, and the consistency of the database in the data writing process is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and in particular relates to a database middleware and method for realizing a strong consistency database. Background Art

[0002] As corporate business expands geographically, data needs to be stored and accessed in different regions to meet the needs of local customers. Storing data closer to users or applications can not only reduce network transmission time and increase data access speed, but also ensure real-time sharing and access of data between different regions, realize cross-regional teamwork and knowledge sharing, and improve work efficiency and innovation capabilities.

[0003] At present, common technologies for ensuring database consistency do include distributed transaction management, consistency protocols and mechanisms, and distributed caching mechanisms. However, no matter which technical solution is used, it is difficult to achieve automatic monitoring of database consistency and data synchronization recovery. Due to the complexity of distributed systems and the frequent interaction of data between multiple nodes, manual monitoring is not only inefficient but also prone to errors. Secondly, since each database usually runs independently and data copies are generally stored in their own databases, it is difficult to achieve coordination and consistency between the various databases when performing data synchronization or recovery operations. This may result in the data of some databases being updated during the recovery process, while the data of other databases is still in the old state, thereby causing new inconsistency problems. In addition, in terms of the preservation and synchronous update of data copies, existing technologies mostly rely on the storage method of full copies. Although this method can ensure the integrity and consistency of data, it will not only cause a large waste of storage space and transmission efficiency, but may also cause new data synchronization recovery problems due to excessive cache size. Summary of the invention

[0004] In order to solve at least one of the problems mentioned in the above background technology, the present invention proposes a database middleware and method for implementing a strong consistency database.

[0005] A database middleware for realizing a strongly consistent database comprises a concurrent writing component, a permission synchronization component, a state detection module, a disaster recovery component and a synchronous recovery component. Specifically, the concurrent writing component is used for synchronously writing data to each database; the permission synchronization component is used for managing the read and write permissions of each database; the state monitoring module is used for monitoring whether each database is in an abnormal state; the disaster recovery component is used for storing and managing data copies; and the synchronous recovery component is used for synchronizing data coverage to each database.

[0006] A method for implementing a strong consistency database, specifically comprising the steps of:

[0007] Step S1, the database middleware accesses the third-party service and receives the proxy request of the third-party service;

[0008] After the third-party service is connected to the database middleware, the database middleware will proxy the transactions submitted by the third-party service and treat complex distributed transaction processing as ordinary transactions, thereby simplifying the management and use burden of multiple databases for the third-party service.

[0009] Step S2, connecting the database middleware to each database;

[0010] Step S3, the database middleware performs data operations on each database according to the received proxy request;

[0011] Specifically, before performing data operations on each database, it is necessary to determine whether each database can write data. If each database can write data, first call the permission synchronization component to enable the write permission of each database, and then call the concurrent write component to write data to each database synchronously; if at least one of the databases cannot write data, call the permission synchronization component to close the write permission of each database.

[0012] Specifically, before synchronously writing data to each database, a data copy needs to be created.

[0013] The specific method of establishing a data copy is: taking the data that needs to be written to each database in the concurrent writing component as incremental data, calling the disaster recovery backup component, establishing an incremental data copy of the incremental data, and using the timestamp as an identifier to assign a unique identifier to the incremental data;

[0014] In order to solve the problem that the existing technologies mostly rely on full copy storage methods, and usually store data copies in respective databases, resulting in wasted storage space, low data transmission efficiency, and difficulty in achieving coordination and consistency between various databases when performing data synchronization or recovery operations. The present invention proposes to establish incremental data copies only for the incremental data that has changed in the database middleware, thereby ensuring that all databases can receive data synchronization and recovery instructions, thereby enabling all databases to obtain the latest incremental data in a timely and accurate manner, avoiding data inconsistency problems caused by asynchrony. The present invention proposes database middleware and an implementation method, which not only simplifies the data synchronization process and shortens the time required for data synchronization and recovery, but also helps to achieve more flexible and efficient data management and disaster recovery strategies, further enhancing the reliability and availability of the system.

[0015] Specifically, to determine whether data can be written to each database, a write request is sent to each database through the database middleware, and a judgment is made based on the returned request type. If the database returns a successful request, it is determined that data can be written; if the database returns a failed request, it is determined that data cannot be written.

[0016] Specifically, whether the database returns a request success or a request failure is determined by checking various parameters of the database. Only when all parameters are met will the request be returned as successful; if at least one parameter is not met, the request is returned as failed.

[0017] The present invention proposes that before writing data to all databases, a write request needs to be sent to all databases. Only when the database returns a successful request, the write permissions of all databases are enabled and the data is written. Otherwise, the write permissions of all databases are closed. Specifically, before executing a data write operation on any database, the mechanism will first send a unified write request to all predefined databases. This step is crucial because it sets a prerequisite for subsequent write operations, that is, only when all databases return confirmation information of successful requests, the system will unlock the write permissions of all databases and then perform actual data writing. If any database fails to successfully respond to the write request or returns a request failure during this process, the system will immediately take safety measures to close the write permissions of all databases, thereby ensuring accurate synchronization of data during the data writing process and achieving consistency of data in each database during the data writing process.

[0018] Step S4: The database middleware monitors each database and decides whether to perform data synchronization recovery according to the monitoring result.

[0019] Among them, the monitoring of each database is carried out by the state detection module in the database middleware, which obtains the characteristic information of each database and determines whether the database has an abnormal state based on the obtained characteristic information;

[0020] The characteristic information is an indicator parameter in the database that has a linear correlation and a nonlinear correlation with the abnormal state.

[0021] The specific monitoring method is: each feature information is used as the input of the state detection module, and the output of the state detection module is used to determine whether the database is in an abnormal state. If the output is category 1, it is determined to be a normal state; if the output is category 2, it is determined to be an abnormal state.

[0022] The state detection module is specifically a trained binary classifier, and the specific calculation formula of the multi-classifier is:

[0023]

[0024]

[0025] In the formula, Ann S For artificial neural networks with multiple classification outputs, Info i is the characteristic information, is the feature extracted by Ann, Sigmoid is the binary classification function, Y cls It is the classification output of the artificial neural network, including category 1 and category 2, which represent the normal state and abnormal state respectively. Specifically, the abnormal state caused by the feature information is regarded as category 2, and the other non-abnormal states are regarded as category 1.

[0026] The training of the binary classifier specifically involves learning the nonlinear mapping relationship between input feature information and output state information.

[0027] The data synchronization recovery is specifically performed by calling the recovery component in the database middleware, and specifically includes the following steps:

[0028] Step A1, calling a difference checking tool to identify whether the incremental data in each database in the data set is consistent, and obtaining identifiers of inconsistent incremental data.

[0029] Step A2, indexing the incremental data copy represented by the identifier in the disaster recovery backup component according to the identifier;

[0030] Step A3, calling the synchronization recovery component, synchronizing the indexed incremental data copies to each database, and overwriting the original incremental data in each database.

[0031] In order to solve the problem that the current technical solution is difficult to realize the automatic monitoring of database consistency and data synchronization recovery, which is inefficient and prone to errors. The present invention proposes to use the index parameters with linear correlation and nonlinear correlation with abnormal states in the database as feature information, and the abnormal states caused by the feature information, such as data loss, data tampering, network packet loss, etc. and other normal states, as categories 1 and 2 respectively, and use an artificial neural network as a binary classifier, and use the feature information as the input of the binary classifier, and the categories 1 and 2 as the output of the binary classifier, and learn the nonlinear mapping relationship between input and output by training the neural network, thereby realizing the automatic monitoring of database consistency, and when the database state is detected to be abnormal, the data synchronization recovery of all databases is performed by calling the recovery component, and the automatic recovery of the database operation process when there is an abnormal state. The present invention proposes a database middleware and an implementation method, which not only realizes the automatic monitoring of database consistency, but also can automatically call the recovery component to perform the necessary data synchronization recovery operation when the database state is detected to be abnormal, thereby ensuring the consistency of the database during operation, reducing the need for human intervention, and improving the reliability and stability of the system.

[0032] The present invention proposes a database middleware and method for realizing a strong consistency database, which has the following beneficial effects compared with the existing technology:

[0033] The present invention proposes a database middleware and an implementation method, which not only realizes the automatic monitoring of database consistency, but also can automatically call the recovery component to perform necessary data synchronization recovery operations when an abnormal database status is detected, thereby ensuring the consistency of the database during operation, reducing the need for human intervention and improving the stability of the system.

[0034] The present invention proposes database middleware and implementation methods, which not only simplify the data synchronization process and shorten the time required for data synchronization recovery, but also help to achieve more flexible and efficient data management and disaster recovery strategies, and further enhance the reliability of the system.

[0035] The present invention provides database middleware and an implementation method, which ensures accurate synchronization of data during the data writing process and realizes consistency of the database during the data writing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the process of the present invention;

[0037] Figure 2 It is a schematic diagram of the architecture of the database middleware of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose and features of the present invention more obvious and easy to understand, the present technical solution is described in detail below through embodiments and in conjunction with the accompanying drawings.

[0039] like Figure 2 As shown, a database middleware for implementing a strongly consistent database includes a concurrent write component, a permission synchronization component, a status detection module, a disaster recovery component and a synchronous recovery component. Specifically, the concurrent write component is used to synchronously write data to each database; the permission synchronization component is used to manage the read and write permissions of each database; the status monitoring module is used to monitor whether each database is in an abnormal state; the disaster recovery component is used to store and manage data copies; and the synchronous recovery component is used to synchronize data coverage to each database.

[0040] like Figure 1 As shown, a method for implementing a strong consistency database specifically includes the following steps:

[0041] Step S1, the database middleware accesses the third-party service and receives the proxy request of the third-party service;

[0042] After the third-party service is connected to the database middleware, the database middleware will proxy the transactions submitted by the third-party service and treat complex distributed transaction processing as ordinary transactions, thereby simplifying the management and use burden of multiple databases for the third-party service.

[0043] Step S2, connecting the database middleware to each database.

[0044] Step S3: The database middleware performs data operations on each database according to the received proxy request.

[0045] Specifically, before performing data operations on each database, it is necessary to determine whether each database can write data. If each database can write data, first call the permission synchronization component to enable the write permission of each database, and then call the concurrent write component to write data to each database synchronously; if at least one of the databases cannot write data, call the permission synchronization component to close the write permission of each database.

[0046] During the specific implementation process, it is also necessary to set whether to send alarm information to a third-party service based on the actual application scenario.

[0047] Specifically, before synchronously writing data to each database, a data copy needs to be created.

[0048] The specific method of establishing a data copy is: the data that needs to be written to each database in the concurrent write component is used as incremental data, and the disaster recovery backup component is called to establish an incremental data copy of the incremental data, and a timestamp is used as an identifier to assign a unique identifier to the incremental data.

[0049] In order to solve the problems that the existing technologies mostly rely on full copy storage and usually store data copies in their respective databases, resulting in wasted storage space, low data transmission efficiency, and difficulty in achieving coordination and consistency between databases when performing data synchronization or recovery operations, the present invention proposes to establish incremental data copies only for the incremental data that has changed in the database middleware, thereby ensuring that all databases can receive data synchronization and recovery instructions, thereby enabling all databases to obtain the latest incremental data in a timely and accurate manner, avoiding the data inconsistency problem caused by asynchrony.

[0050] Specifically, to determine whether data can be written to each database, a write request is sent to each database through the database middleware, and a judgment is made based on the returned request type. If the database returns a successful request, it is determined that data can be written; if the database returns a failed request, it is determined that data cannot be written.

[0051] Specifically, whether the database returns a request success or a request failure is determined by checking various parameters of the database. Only when all parameters are met will the request be returned as successful; if at least one parameter is not met, the request is returned as failed.

[0052] Optionally, the parameters include the network connection status of the database, storage capacity, whether the data table is locked, whether the data type matches, etc.

[0053] The present invention proposes that before writing data to all databases, a write request needs to be sent to all databases. Only when the database returns a successful request, the write permissions of all databases are enabled and the data is written. Otherwise, the write permissions of all databases are closed. Specifically, before executing a data write operation on any database, the mechanism will first send a unified write request to all predefined databases. This step is crucial because it sets a prerequisite for subsequent write operations, that is, only when all databases return confirmation information of successful requests, the system will unlock the write permissions of all databases and then perform actual data writing. If any database fails to successfully respond to the write request or returns a request failure during this process, the system will immediately take safety measures to close the write permissions of all databases, thereby ensuring accurate synchronization of data during the data writing process and achieving consistency of data in each database during the data writing process.

[0054] Step S4: The database middleware monitors each database and decides whether to perform data synchronization recovery according to the monitoring result.

[0055] Among them, the monitoring of each database is carried out by the state detection module in the database middleware, which obtains the characteristic information of each database and determines whether the database has an abnormal state based on the obtained characteristic information;

[0056] The characteristic information is an indicator parameter in the database that has a linear correlation and a nonlinear correlation with the abnormal state.

[0057] In the specific implementation process, these indicator parameters can be selected based on experience, or they can be obtained by calculating and analyzing each indicator parameter using a principal component analysis algorithm.

[0058] Optionally, the indicator parameters may include network delay, network jitter, network bandwidth, data synchronization delay rate, system load rate, and storage utilization rate.

[0059] The specific monitoring method is: each feature information is used as the input of the state detection module, and the output of the state detection module is used to determine whether the database is in an abnormal state. If the output is category 1, it is determined to be a normal state; if the output is category 2, it is determined to be an abnormal state.

[0060] The state detection module is specifically a trained binary classifier, and the specific calculation formula of the multi-classifier is:

[0061]

[0062]

[0063] In the formula, Ann S For artificial neural networks with multiple classification outputs, Info i is the characteristic information, is the feature extracted by Ann, Sigmoid is the binary classification function, Y cls It is the classification output of the artificial neural network, including category 1 and category 2, which represent the normal state and abnormal state respectively. Specifically, the abnormal state caused by the feature information is regarded as category 2, and the other non-abnormal states are regarded as category 1.

[0064] The training of the binary classifier specifically involves learning the nonlinear mapping relationship between input feature information and output state information.

[0065] In the specific implementation process, the abnormal conditions caused by the characteristic information may include data loss, data tampering, network packet loss, etc.

[0066] The data synchronization recovery is specifically performed by calling the recovery component in the database middleware, and specifically includes the following steps:

[0067] Step A1, calling a difference checking tool to identify whether the incremental data in each database in the data set is consistent, and obtaining identifiers of inconsistent incremental data.

[0068] In the specific implementation process, the difference checking tool can be a difference detection algorithm implemented by Python, or it can be a mature difference checking software, such as KDiff3, CSV Diff, etc.

[0069] Step A2, indexing the incremental data copy represented by the identifier in the disaster recovery backup component according to the identifier;

[0070] Step A3, calling the synchronization recovery component, synchronizing the indexed incremental data copies to each database, and overwriting the original incremental data in each database.

[0071] In order to solve the problem that the current technical solution is difficult to realize the automatic monitoring of database consistency and data synchronization recovery, which is inefficient and prone to errors. The present invention proposes to use the index parameters in the database that have linear correlation and nonlinear correlation with abnormal states as feature information, and the abnormal states caused by the feature information, such as data loss, data tampering, network packet loss, etc. and other normal states, as category 1 and category 2 respectively, and use an artificial neural network as a binary classifier, and use the feature information as the input of the binary classifier, and the category 1 and category 2 as the output of the binary classifier, and learn the nonlinear mapping relationship between input and output by training the neural network, thereby realizing the automatic monitoring of database consistency, and when the database state is detected to be abnormal, the data synchronization recovery of all databases is performed by calling the recovery component, so as to realize the automatic recovery when the database operation process has an abnormal state.

[0072] So far, according to the method disclosed in the present invention, the working process of the present invention has been implemented once.

[0073] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0075] The above are only preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for implementing a strong consistency database, characterized in that: Includes steps: Step S1, the database middleware accesses the third-party service and receives the proxy request of the third-party service; Step S2, connecting the database middleware to each database; Step S3, the database middleware performs data operations on each database according to the received proxy request; Step S4: The database middleware monitors each database and decides whether to perform data synchronization recovery according to the monitoring result.

2. The method for implementing a strong consistency database according to claim 1, characterized in that: The data operations performed on each database in step S3, specifically, before performing data operations on each database, it is necessary to determine whether each database can write data. If each database can write data, first call the permission synchronization component to enable the write permission of each database, and then call the concurrent write component to synchronously write data to each database; if at least one of the databases cannot write data, call the permission synchronization component to close the write permission of each database.

3. The method for realizing a strong consistency database according to claim 2, characterized in that: The calling concurrent writing component synchronously writes data to each database. Specifically, before synchronously writing data to each database, a data copy needs to be established. The specific method of establishing a data copy is: the data that needs to be written to each database in the concurrent write component is used as incremental data, and the disaster recovery backup component is called to establish an incremental data copy of the incremental data, and a timestamp is used as an identifier to assign a unique identifier to the incremental data.

4. The method for implementing a strong consistency database according to claim 2, characterized in that: The determination of whether data can be written to each database is specifically to send a write request to each database through the database middleware, and to make a determination based on the returned request type. If the database returns a successful request, it is determined that data can be written; If the database returns a request failure, it is determined that the data cannot be written.

5. The method for realizing a strong consistency database according to claim 4, characterized in that: The database returns request success or request failure by checking various parameters of the database. Specifically, the request is returned as successful only if all parameters are met; if at least one parameter is not met, the request is returned as failed.

6. The method for implementing a strong consistency database according to claim 1, characterized in that: The monitoring of each database described in step S4 is performed by the state detection module in the database middleware, which obtains the characteristic information of each database and determines whether the database has an abnormal state based on the obtained characteristic information. The specific monitoring method is: each characteristic information is used as the input of the state detection module, and the database is judged to be in an abnormal state based on the output of the state detection module. If the output is category 1, it is judged to be a normal state; if the output is category 2, it is judged to be an abnormal state.

7. The method for implementing a strong consistency database according to claim 6, characterized in that: The characteristic information is an indicator parameter in the database that has linear correlation and nonlinear correlation with the abnormal state; the state detection module is specifically a trained binary classifier, and the specific calculation formula of the multi-classifier is: In the formula, Ann S For artificial neural networks with multiple classification outputs, Info i is the characteristic information, is the feature extracted by Ann, Sigmoid is the binary classification function, Y cls is the classification output of the artificial neural network, including category 1 and category 2, which represent the normal state and abnormal state respectively. Specifically, the abnormal state caused by the feature information is regarded as category 2, and the other non-abnormal states are regarded as category 1; The training of the binary classifier specifically involves learning the nonlinear mapping relationship between input feature information and output state information.

8. The method for implementing a strong consistency database according to claim 1, characterized in that: The data synchronization recovery described in step S4 is specifically performed by calling the recovery component in the database middleware to perform data synchronization recovery, which specifically includes the steps of: Step A1, calling a difference checking tool to identify whether the incremental data in each database in the data set is consistent, and obtaining identifiers of inconsistent incremental data; Step A2, indexing the incremental data copy represented by the identifier in the disaster recovery backup component according to the identifier; Step A3, calling the synchronization recovery component, synchronizing the indexed incremental data copies to each database, and overwriting the original incremental data in each database.

9. A database middleware for implementing a strong consistency database, characterized in that: The operational implementation of the database middleware relies on the methods described in claims 1 to 8. Specifically, the database middleware includes a concurrent write component, a permission synchronization component, a status detection module, a disaster recovery backup component, and a synchronous recovery component, wherein the concurrent write component is used to synchronously write data to each database; the permission synchronization component is used to manage the read and write permissions of each database; the status monitoring module is used to monitor whether each database is in an abnormal state; the disaster recovery backup component is used to store and manage data copies; and the synchronous recovery component is used to synchronize data coverage to each database.

Images