Database middleware and method for implementing strong consistency database

Abstract

The present invention belongs to the field of database technology, and specifically discloses a database middleware and method for realizing a strongly consistent database. The present invention proposes a database middleware and implementation method, which not only realizes the automatic monitoring of database consistency, but also can automatically call the recovery component to perform data synchronization recovery operations when an abnormal database status is detected, thereby ensuring the consistency of the database during operation, reducing the need for human intervention, and improving the reliability and stability of the system. In addition, the method proposed by the present invention also simplifies the data synchronization process, shortens the time required for data synchronization recovery, and ensures the accurate synchronization of data during the data writing process, thereby realizing the consistency of the database during the data writing process.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and in particular relates to database middleware and a method for implementing a strongly consistent database. Background Art

[0002] As corporate businesses expand geographically, data storage and access are required in different regions to meet the needs of local customers. Storing data closer to users or applications not only reduces network transmission time and increases data access speed, but also ensures real-time sharing and access of data across regions, enabling cross-regional teamwork and knowledge sharing, and improving work efficiency and innovation capabilities.

[0003] Currently, common technologies for ensuring database consistency do include distributed transaction management, consistency protocols and mechanisms, and distributed caching mechanisms. However, no matter which technical solution is used, it is difficult to achieve automated monitoring of database consistency and data synchronization and recovery. Due to the complexity of distributed systems and the frequent interaction of data between multiple nodes, manual monitoring is not only inefficient but also prone to errors. Secondly, since each database usually runs independently and data copies are generally stored in their own databases, it is difficult to achieve coordination and consistency between the various databases when performing data synchronization or recovery operations. This may result in the data in some databases being updated during the recovery process, while the data in other databases is still in the old state, thereby causing new inconsistency problems. In addition, in terms of the preservation and synchronous update of data copies, existing technologies mostly rely on the storage method of full copies. Although this method can ensure the integrity and consistency of data, it not only causes a large waste of storage space and transmission efficiency, but may also cause new data synchronization and recovery problems due to the excessive size of the cache. Summary of the Invention

[0004] In order to solve at least one of the problems mentioned in the above background technology, the present invention proposes a database middleware and method for implementing a strong consistency database.

[0005] A database middleware for implementing a strongly consistent database includes a concurrent write component, a permission synchronization component, a status detection module, a disaster recovery component, and a synchronous recovery component. Specifically, the concurrent write component is used to synchronously write data to each database; the permission synchronization component is used to manage the read and write permissions of each database; the status monitoring module is used to monitor whether each database is in an abnormal state; the disaster recovery component is used to store and manage data copies; and the synchronous recovery component is used to synchronize data overwrites to each database.

[0006] A method for implementing a strongly consistent database, specifically comprising the steps of:

[0007] Step S1: The database middleware accesses the third-party service and receives a proxy request from the third-party service;

[0008] After the third-party service is connected to the database middleware, the database middleware will proxy the transactions submitted by the third-party service and treat complex distributed transaction processing as ordinary transactions, thereby simplifying the management and use burden of multiple databases for the third-party service.

[0009] Step S2, connecting the database middleware to each database;

[0010] Step S3: The database middleware performs data operations on each database according to the received proxy request;

[0011] Specifically, before performing data operations on each database, it is necessary to determine whether each database can write data. If each database can write data, first call the permission synchronization component to enable the write permission of each database, and then call the concurrent write component to write data to each database synchronously; if at least one database cannot write data, call the permission synchronization component to close the write permission of each database.

[0012] Specifically, before synchronously writing data to each database, a data copy needs to be created.

[0013] The specific method of establishing a data copy is as follows: the data that needs to be written to each database in the concurrent writing component is used as incremental data, and the disaster recovery backup component is called to establish an incremental data copy of the incremental data, and a timestamp is used as an identifier to assign a unique identifier to the incremental data;

[0014] In order to solve the problem that the existing technologies mostly rely on full copy storage methods, and usually store data copies in their respective databases, resulting in wasted storage space, low data transmission efficiency, and difficulty in achieving coordination and consistency between various databases when performing data synchronization or recovery operations. The present invention proposes to establish incremental data copies only for the incremental data that has changed in the database middleware, so as to ensure that all databases can receive data synchronization and recovery instructions, thereby enabling all databases to obtain the latest incremental data in a timely and accurate manner, avoiding data inconsistency problems caused by asynchrony. The present invention proposes database middleware and an implementation method, which not only simplifies the data synchronization process and shortens the time required for data synchronization and recovery, but also helps to achieve more flexible and efficient data management and disaster recovery strategies, further enhancing the reliability and availability of the system.

[0015] Specifically, to determine whether each database can write data, a write request is sent to each database through the database middleware, and a judgment is made based on the returned request type. If the database returns a successful request, it is determined that data can be written; if the database returns a failed request, it is determined that data cannot be written.

[0016] Specifically, the database returns request success or request failure by checking various parameters of the database. Only when all parameters are met will the request be returned as successful; if at least one parameter is not met, the request is returned as failed.

[0017] The present invention proposes that before writing data to all databases, a write request needs to be sent to all databases. Only when the database returns a successful request, the write permissions of all databases are enabled and the data is written. Otherwise, the write permissions of all databases are closed. Specifically, before executing a data write operation on any database, the mechanism will first send a unified write request to all predefined databases. This step is crucial because it sets a prerequisite for subsequent write operations, that is, only when all databases return confirmation information that the request is successful, the system will unlock the write permissions of all databases and then perform the actual data writing. If, during this process, any database fails to successfully respond to the write request, or returns a request failure, the system will immediately take safety measures and close the write permissions of all databases, thereby ensuring the accurate synchronization of data during the data writing process and achieving the consistency of data in each database during the data writing process.

[0018] In step S4, the database middleware monitors each database and decides whether to perform data synchronization recovery based on the monitoring results.

[0019] Among them, the monitoring of each database is carried out by the status detection module in the database middleware, which obtains the characteristic information of each database and determines whether the database has any abnormal status based on the obtained characteristic information;

[0020] The characteristic information is an indicator parameter in the database that has linear correlation and nonlinear correlation with the abnormal state.

[0021] The specific monitoring method is: each feature information is used as the input of the state detection module, and the output of the state detection module is used to determine whether the database is in an abnormal state. If the output is category 1, it is determined to be a normal state; if the output is category 2, it is determined to be an abnormal state.

[0022] The state detection module is specifically a trained binary classifier, and the specific calculation formula of the multi-classifier is:

[0023]

[0024]

[0025] In the formula, Ann S For artificial neural networks with multiple classification outputs, Info i is the feature information, is the feature extracted by Ann, Sigmoid is the binary classification function, Y cls It is the classification output of the artificial neural network, including category 1 and category 2, which represent the normal state and abnormal state respectively. Specifically, the abnormal state caused by the feature information is regarded as category 2, and the other non-abnormal states are regarded as category 1.

[0026] The training of the binary classifier specifically involves learning the nonlinear mapping relationship between input feature information and output state information.

[0027] The data synchronization recovery is specifically performed by calling the recovery component in the database middleware, and specifically includes the following steps:

[0028] Step A1: Call a difference checking tool to identify whether the incremental data in each database in the data set is consistent, and obtain identifiers of inconsistent incremental data.

[0029] Step A2: indexing the incremental data copy represented by the identifier in the disaster recovery backup component according to the identifier;

[0030] In step A3, the synchronization recovery component is called to synchronize the indexed incremental data copies to each database and overwrite the original incremental data in each database.

[0031] In order to solve the problem that the current technical solutions are difficult to achieve automatic monitoring of database consistency and data synchronization recovery, which is inefficient and prone to errors. The present invention proposes to use indicator parameters with linear and nonlinear correlations with abnormal states in the database as feature information, and to classify abnormal states caused by the feature information, such as data loss, data tampering, network packet loss, and other normal states, as categories 1 and 2 respectively. An artificial neural network is used as a binary classifier, and the feature information is used as the input of the binary classifier, and the categories 1 and 2 are used as the output of the binary classifier. The neural network is trained to learn the nonlinear mapping relationship between input and output, thereby achieving automatic monitoring of database consistency. When an abnormal database state is detected, the recovery component is called to perform data synchronization recovery of all databases, thereby achieving automatic recovery when an abnormal state occurs during the database operation process. The present invention proposes a database middleware and implementation method, which not only achieves automatic monitoring of database consistency, but also can automatically call the recovery component to perform necessary data synchronization recovery operations when an abnormal database state is detected, thereby ensuring the consistency of the database during operation, reducing the need for human intervention, and improving the reliability and stability of the system.

[0032] The present invention proposes a database middleware and method for implementing a strongly consistent database, which has the following beneficial effects compared with existing technologies:

[0033] The present invention proposes database middleware and implementation method, which not only realizes the automatic monitoring of database consistency, but also can automatically call the recovery component to perform necessary data synchronization recovery operations when an abnormal database status is detected, thereby ensuring the consistency of the database during operation, reducing the need for human intervention, and improving the stability of the system.

[0034] The present invention proposes database middleware and implementation methods, which not only simplifies the data synchronization process and shortens the time required for data synchronization recovery, but also helps to achieve more flexible and efficient data management and disaster recovery strategies, further enhancing the reliability of the system.

[0035] The present invention provides database middleware and an implementation method, which ensures accurate synchronization of data during the data writing process and achieves consistency of the database during the data writing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic flow diagram of the present invention;

[0037] Figure 2 It is a schematic diagram of the architecture of the database middleware of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose and features of the present invention more obvious and easy to understand, the present technical solution is described in detail below through embodiments and in conjunction with the accompanying drawings.

[0039] like Figure 2 As shown, a database middleware for implementing a strongly consistent database includes a concurrent write component, a permission synchronization component, a status detection module, a disaster recovery backup component, and a synchronous recovery component. Specifically, the concurrent write component is used to synchronously write data to each database; the permission synchronization component is used to manage the read and write permissions of each database; the status monitoring module is used to monitor whether each database is in an abnormal state; the disaster recovery backup component is used to store and manage data copies; and the synchronous recovery component is used to synchronize data coverage to each database.

[0040] like Figure 1 As shown, a method for implementing a strong consistency database specifically includes the following steps:

[0041] Step S1: The database middleware accesses the third-party service and receives a proxy request from the third-party service;

[0042] After the third-party service is connected to the database middleware, the database middleware will proxy the transactions submitted by the third-party service and treat complex distributed transaction processing as ordinary transactions, thereby simplifying the management and use burden of multiple databases for the third-party service.

[0043] Step S2: Connect the database middleware to each database.

[0044] In step S3, the database middleware performs data operations on each database according to the received proxy request.

[0045] Specifically, before performing data operations on each database, it is necessary to determine whether each database can write data. If each database can write data, first call the permission synchronization component to enable the write permission of each database, and then call the concurrent write component to write data to each database synchronously; if at least one database cannot write data, call the permission synchronization component to close the write permission of each database.

[0046] During the specific implementation process, it is also necessary to set whether to send alarm information to third-party services based on the actual application scenario.

[0047] Specifically, before synchronously writing data to each database, a data copy needs to be created.

[0048] The specific method of establishing a data copy is: the data that needs to be written to each database in the concurrent write component is used as incremental data, and the disaster recovery backup component is called to establish an incremental data copy of the incremental data, and a timestamp is used as an identifier to assign a unique identifier to the incremental data.

[0049] To address the problems of existing technologies that rely heavily on full copy storage and typically store data copies in separate databases, resulting in wasted storage space, inefficient data transmission, and difficulty achieving consistency across databases during data synchronization or recovery operations, the present invention proposes establishing incremental data copies in the database middleware only for the incremental data that has changed, thereby ensuring that all databases receive data synchronization and recovery instructions. This allows all databases to obtain the latest incremental data in a timely and accurate manner, avoiding data inconsistencies caused by asynchrony.

[0050] Specifically, to determine whether each database can write data, a write request is sent to each database through the database middleware, and a judgment is made based on the returned request type. If the database returns a successful request, it is determined that data can be written; if the database returns a failed request, it is determined that data cannot be written.

[0051] Specifically, the database returns request success or request failure by checking various parameters of the database. Only when all parameters are met will the request be returned as successful; if at least one parameter is not met, the request is returned as failed.

[0052] Optionally, the parameters include the network connection status of the database, storage capacity, whether the data table is locked, whether the data type matches, etc.

[0053] The present invention proposes that before writing data to all databases, a write request needs to be sent to all databases. Only when the database returns a successful request, the write permissions of all databases are enabled and the data is written. Otherwise, the write permissions of all databases are closed. Specifically, before executing a data write operation on any database, the mechanism will first send a unified write request to all predefined databases. This step is crucial because it sets a prerequisite for subsequent write operations, that is, only when all databases return confirmation information that the request is successful, the system will unlock the write permissions of all databases and then perform the actual data writing. If, during this process, any database fails to successfully respond to the write request, or returns a request failure, the system will immediately take safety measures and close the write permissions of all databases, thereby ensuring the accurate synchronization of data during the data writing process and achieving the consistency of data in each database during the data writing process.

[0054] In step S4, the database middleware monitors each database and decides whether to perform data synchronization recovery based on the monitoring results.

[0055] Among them, the monitoring of each database is carried out by the status detection module in the database middleware, which obtains the characteristic information of each database and determines whether the database has any abnormal status based on the obtained characteristic information;

[0056] The characteristic information is an indicator parameter in the database that has linear correlation and nonlinear correlation with the abnormal state.

[0057] In the specific implementation process, these indicator parameters can be selected based on experience, or they can be obtained by calculating and analyzing each indicator parameter using a principal component analysis algorithm.

[0058] Optionally, the indicator parameters may include network delay, network jitter, network bandwidth, data synchronization delay rate, system load rate, and storage usage rate.

[0059] The specific monitoring method is: each feature information is used as the input of the state detection module, and the output of the state detection module is used to determine whether the database is in an abnormal state. If the output is category 1, it is determined to be a normal state; if the output is category 2, it is determined to be an abnormal state.

[0060] The state detection module is specifically a trained binary classifier, and the specific calculation formula of the multi-classifier is:

[0061]

[0062]

[0063] In the formula, Ann S For artificial neural networks with multiple classification outputs, Info i is the feature information, is the feature extracted by Ann, Sigmoid is the binary classification function, Y cls It is the classification output of the artificial neural network, including category 1 and category 2, which represent the normal state and abnormal state respectively. Specifically, the abnormal state caused by the feature information is regarded as category 2, and the other non-abnormal states are regarded as category 1.

[0064] The training of the binary classifier specifically involves learning the nonlinear mapping relationship between input feature information and output state information.

[0065] During the specific implementation process, abnormal conditions caused by characteristic information may include data loss, data tampering, network packet loss, etc.

[0066] The data synchronization recovery is specifically performed by calling the recovery component in the database middleware, and specifically includes the following steps:

[0067] Step A1: Call a difference checking tool to identify whether the incremental data in each database in the data set is consistent, and obtain identifiers of inconsistent incremental data.

[0068] In the specific implementation process, the difference checking tool can be a difference detection algorithm implemented by Python, or it can be a mature difference checking software, such as KDiff3, CSV Diff, etc.

[0069] Step A2: indexing the incremental data copy represented by the identifier in the disaster recovery backup component according to the identifier;

[0070] In step A3, the synchronization recovery component is called to synchronize the indexed incremental data copies to each database and overwrite the original incremental data in each database.

[0071] In order to solve the problem that the current technical solution is difficult to realize the automatic monitoring of database consistency and data synchronization recovery, which is inefficient and prone to errors. The present invention proposes to use the indicator parameters with linear correlation and nonlinear correlation with abnormal states in the database as feature information, and the abnormal states caused by the feature information, such as data loss, data tampering, network packet loss, etc. and other normal states as category 1 and category 2 respectively. An artificial neural network is used as a binary classifier, and the feature information is used as the input of the binary classifier, and the category 1 and category 2 are used as the output of the binary classifier. The nonlinear mapping relationship between input and output is learned by training the neural network, thereby realizing the automatic monitoring of database consistency. Moreover, when an abnormal database state is detected, the data synchronization recovery of all databases is performed by calling the recovery component, thereby realizing automatic recovery when an abnormal state exists in the database operation process.

[0072] So far, according to the method disclosed in the present invention, the working process of the present invention has been implemented once.

[0073] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are for descriptive purposes only and do not represent the advantages or disadvantages of the embodiments. In addition, the terms "including", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method comprising the element.

[0074] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0075] The above are only preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for implementing a strong consistency database, characterized in that: Including steps: Step S1: The database middleware accesses the third-party service and receives a proxy request from the third-party service; Step S2, connecting the database middleware to each database; Step S3: The database middleware performs data operations on each database according to the received proxy request; Step S4: The database middleware monitors each database and decides whether to perform data synchronization recovery based on the monitoring results; The data synchronization recovery described in step S4 is specifically performed by calling the recovery component in the database middleware to perform data synchronization recovery, which specifically includes the following steps: Step A1: Call a difference checking tool to identify whether the incremental data in each database in the data set is consistent, and obtain identifiers of inconsistent incremental data; Step A2: indexing the incremental data copy represented by the identifier in the disaster recovery backup component according to the identifier; In step A3, the synchronization recovery component is called to synchronize the indexed incremental data copies to each database and overwrite the original incremental data in each database.

2. The method for implementing a strong consistency database according to claim 1, characterized in that: The data operations performed on each database as described in step S3, specifically, before performing data operations on each database, it is necessary to determine whether each database can write data. If each database can write data, first call the permission synchronization component to enable the write permission of each database, and then call the concurrent write component to synchronously write data to each database; if at least one database cannot write data, call the permission synchronization component to close the write permission of each database.

3. The method for implementing a strong consistency database according to claim 2, characterized in that: The calling concurrent writing component synchronously writes data to each database. Specifically, before synchronously writing data to each database, a data copy needs to be created; The specific method of establishing a data copy is: the data that needs to be written to each database in the concurrent write component is used as incremental data, and the disaster recovery backup component is called to establish an incremental data copy of the incremental data, and a timestamp is used as an identifier to assign a unique identifier to the incremental data.

4. The method for implementing a strong consistency database according to claim 2, wherein: The determination of whether data can be written to each database is specifically to send a write request to each database through the database middleware and make a determination based on the type of request returned. If the database returns a successful request, it is determined that data can be written; If the database returns a request failure, it is determined that the data cannot be written.

5. The method for implementing a strong consistency database according to claim 4, characterized in that: The database returns request success or request failure by checking various parameters of the database. Specifically, the request is returned as successful only if all parameters are met; if at least one parameter is not met, the request is returned as failed.

6. The method for implementing a strong consistency database according to claim 1, characterized in that: The monitoring of each database described in step S4 is performed by the status detection module in the database middleware, which obtains the characteristic information of each database and determines whether the database has an abnormal state based on the obtained characteristic information. The specific monitoring method is: each characteristic information is used as the input of the status detection module, and the output of the status detection module is used to determine whether the database is in an abnormal state. If the output is category 1, it is determined to be a normal state; if the output is category 2, it is determined to be an abnormal state.

7. The method for implementing a strong consistency database according to claim 6, characterized in that: The characteristic information is an indicator parameter in the database that has linear correlation and nonlinear correlation with the abnormal state; the state detection module is specifically a trained binary classifier, and the specific calculation formula of the binary classifier is: In the formula, Ann S For artificial neural networks, Info i is the feature information, is the feature extracted by Ann, Sigmoid is the binary classification function, Y cls is the classification output of the artificial neural network, including category 1 and category 2, which represent the normal state and abnormal state respectively. Specifically, the abnormal state caused by the feature information is regarded as category 2, and the other non-abnormal states are regarded as category 1; The training of the binary classifier specifically involves learning the nonlinear mapping relationship between input feature information and output state information.

8. A database middleware for implementing a strong consistency database, characterized in that: The operational implementation of the database middleware relies on the method described in any one of claims 1 to 7. Specifically, the database middleware includes a concurrent write component, a permission synchronization component, a status detection module, a disaster recovery backup component, and a synchronous recovery component, wherein the concurrent write component is used to synchronously write data to each database; the permission synchronization component is used to manage the read and write permissions of each database; the status monitoring module is used to monitor whether each database is in an abnormal state; the disaster recovery backup component is used to store and manage data copies; and the synchronous recovery component is used to synchronize data coverage to each database.

Images