Data automatic synchronization method and system for active-active architecture real-time database

Abstract

The invention belongs to the technical field of electric signal data processing, and discloses a data automatic synchronization method and system for an active-active architecture real-time database, and the method comprises the steps: after a real-time database service is started, one end of a main database is responsible for processing a time sequence data write-in request of a client, and after the main database receives data, the main database is responsible for processing a time sequence data write-in request of the client; writing the data into a synchronous data queue, and writing the data into storage; when data exists in the synchronous data queue, writing the data into a synchronous cache file, and preparing to send the data to a slave library for synchronization; comparing the metadata information of the master library with the metadata information of the slave library, querying whether the metadata versions are consistent or not, and when the metadata are not synchronized, pausing data synchronization and performing metadata synchronization; under the metadata synchronization condition, the data synchronization process continues; and after receiving the synchronous data, the slave library writes the synchronous data. According to the method, the data synchronization rate of the two ends of the real-time database of the active-active architecture is improved, and the complete consistency and high availability of the data are ensured.

Description

Technical Field

[0001] This invention relates to the field of electrical signal data processing technology, specifically to a method and system for automatic data synchronization of a real-time database with a dual-active architecture. Background Technology

[0002] With the rapid development of digitalization and intelligentization in various fields, massive amounts of data have gradually become an important part of enterprise assets. Especially in the manufacturing sector, in order to vigorously develop applications such as big data, the Internet of Things, and big models, the requirements for the integrity and reliability of production data are becoming increasingly higher. Compared with stand-alone or offline data backup real-time databases, highly available real-time databases can guarantee availability in the event of a single point of failure.

[0003] To further achieve uninterrupted service and no data loss, a dual-active architecture is needed to replace the traditional master-slave hot standby mechanism. Dual-active architecture allows both nodes to provide complete data services, making it the most complex high-availability architecture. The two nodes not only need to replicate data bidirectionally but also guarantee extremely low data latency, extremely fast fault recovery speed, and complete and consistent data quality. Relational data synchronization mechanisms are generally not suitable for time-series data synchronization. Some data synchronization methods designed for the physical characteristics of time-series data typically perform well in master-slave hot standby architectures. However, in a dual-active architecture, both database services need to undertake complete read and write functions to ensure data integrity and business continuity. A synchronization strategy that guarantees high availability is required. Existing synchronization strategies struggle to balance synchronization success rate and data integrity consistency. Furthermore, the inconsistency between the two nodes due to metadata changes and index file reconstruction must be considered to prevent synchronization failures. Summary of the Invention

[0004] To address the problem that the efficiency and success rate of real-time database data synchronization in the existing technologies are insufficient to support the high availability of a dual-active architecture, the present invention aims to propose an automatic data synchronization method and system for a dual-active architecture real-time database. By improving the data synchronization method and adding an automatic metadata comparison and priority synchronization mechanism, data access conflicts and synchronization failures are avoided. Furthermore, automatic processing ensures complete and consistent data synchronization, guaranteeing the efficient and normal operation of a dual-active high-availability real-time database.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of this invention is to provide a method for automatic data synchronization of a real-time database with a dual-active architecture, comprising the following steps: After the real-time database service starts, the master database is responsible for handling the client's time-series data write requests. After receiving the data, the master database writes the data to the synchronization data queue and then writes the data to the storage. When there is data in the synchronization data queue, it is written to the synchronization cache file and prepared to be sent to the slave for synchronization. Compare the metadata information of the master database and the slave database, and check whether the metadata versions are consistent. If the metadata is out of sync, pause data synchronization and start metadata synchronization; if the metadata is synchronized, the data synchronization process continues. After receiving the synchronization data from the database, write the synchronization data.

[0006] As a further improvement of the present invention, the master database is determined by the data arrival priority. After the real-time database service starts, the data arrival priority of the two active ends is calculated according to the high availability IP group configuration. The API regards the node with the higher data arrival priority as the master database and the other node as the slave database.

[0007] As a further improvement of the present invention, the data synchronization is bidirectional and real-time, specifically including: By default, real-time data write requests are processed first by the master database, which then synchronizes the data with the slave database. Historical data write requests are handled by load balancing between two nodes, with the master and slave databases synchronizing data. When the current master database is overloaded, the overloaded master database will be switched to a new slave database, and the remaining data will continue to be synchronized to the peer database. The slave database will be regarded as the new master database, receiving data and synchronizing it to the peer database.

[0008] As a further improvement of the present invention, the metadata information includes database tables, tag points, tag point recycle bins, custom types, and metadata versions. When all metadata information of the master database and the slave database is consistent, the metadata versions are the same, and they are in a metadata synchronization state.

[0009] As a further improvement of the present invention, the metadata asynchronization includes adding a dual-active database node, metadata initialization, and metadata modification; wherein, the metadata modification operation specifically includes adding, updating, and deleting metadata, the API performs metadata modification operations on the master database, and writes the modification results into the cache and metadata write-ahead log, periodically stores them on the disk, the metadata version number is incremented after modification, and the master database synchronizes the metadata with the slave database.

[0010] As a further improvement of the present invention, the metadata synchronization includes two synchronization modes: full synchronization and incremental synchronization, specifically: The full synchronization mode is used to synchronize the entire metadata information from the master database to the slave database, and is used for adding dual-active database nodes and initializing metadata. Incremental synchronization mode is used for real-time updates and synchronization of metadata changes from the master database to the slave database, and is used for real-time synchronization during normal operation of both ends.

[0011] As a further improvement of the present invention, the specific steps of the metadata synchronization are as follows: S31: Compare the maximum version of the master database's metadata write-ahead log with the slave database's metadata version. If the maximum version of the metadata write-ahead log is the same as the slave database's metadata version, adopt the incremental synchronization mode and synchronize directly for each modification operation on the master database. S32: If the maximum version of the metadata write-ahead log is greater than the version of the slave database metadata, the incremental synchronization mode is adopted. First, the metadata write-ahead log is synchronized, and then the modification operations during the master database are put into the queue for synchronization. S33: If the difference in metadata versions exceeds the threshold, use the full synchronization mode to initialize the metadata; S34: If the slave is a newly added dual-active database node, the full synchronization mode is adopted. If the metadata is modified during the synchronization, the synchronization is interrupted and restarted. S35: After synchronization is complete, the metadata versions on both ends are the same, and subsequent metadata synchronization will use incremental synchronization mode.

[0012] As a further improvement of the present invention, when a node of the real-time database experiences a network outage or malfunction causing processing failure, automatic failover will be performed during data synchronization, specifically including: When the master database fails to process the data, the API will automatically switch to the slave database and increase the data arrival priority of the slave database, treating it as the new master database. At the same time, the API will send the metadata write-ahead log, the data in the memory cache, the data that failed to be written this time, and the subsequent write data to the new master database, which will continue the data synchronization. When both databases fail to process, the data written in this batch is cached in a cache file. Once a data node recovers, the API treats it as the new master database and fills in the metadata write-ahead log and data in the cache file for the new master database. Once the other data node recovers, the new master database continues data synchronization.

[0013] Secondly, the present invention provides an automatic data synchronization system for a dual-active architecture real-time database, comprising: The data write API is used to connect the active and passive ends of a real-time database. It identifies the master and slave databases based on the data arrival priority, distributes the time-series data and metadata written from the client to the master and slave databases according to the data synchronization strategy, and writes them to the cache. The main data synchronization thread is used to repeatedly process time-series data write requests from clients and synchronous data write requests from peers after the database starts. When the master database receives data, it writes the data to the synchronous data queue and then writes the data to storage; when the slave database receives synchronous data, it writes the data directly to storage. The data synchronization cache thread is used to repeatedly check the data in the synchronization data queue and write it to the synchronization cache file in batches. Data synchronization connection thread: used to repeatedly check the data in the synchronization cache file and send it in batches to the slave database for synchronization; Metadata synchronization thread: Used to repeatedly compare the metadata information of the active and passive ends of the real-time database, and query whether the metadata versions are consistent. When the metadata of the two ends is out of sync, the data synchronization connection thread is notified to pause data synchronization and perform metadata synchronization.

[0014] Thirdly, this invention provides a method for automatic data synchronization of a real-time database with a dual-active architecture, comprising the following steps: The system connects to both ends of a real-time database in an active-active manner. It identifies the master and slave databases based on the data arrival priority. It distributes time-series data and metadata written from the client to the master and slave databases according to the data synchronization strategy and writes them to the cache. When the master database receives data, it writes the data to the synchronization data queue and then writes the data to the storage. When the slave database receives synchronization data, it directly writes the data to the storage. The data in the synchronization data queue is repeatedly checked and written to the synchronization cache file in batches; the master database repeatedly checks the data in the synchronization cache file and sends it to the slave database in batches for synchronization. Repeatedly compare the metadata information of the two active-active ends of the real-time database, and check whether the metadata versions are consistent. When the metadata of the two ends is out of sync, notify the data synchronization connection thread to pause data synchronization and perform metadata synchronization.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention incorporates a metadata priority synchronization mechanism into its data synchronization method, ensuring consistency between the master and slave databases' metadata and avoiding conflicts caused by inconsistencies. Simultaneously, a bidirectional real-time data synchronization mechanism effectively improves the data synchronization rate between the two ends, laying the foundation for providing peer-to-peer data services in a dual-active architecture. In the event of a node failure, automatic node switching ensures data integrity and continuous data service provision. Based on the bidirectional real-time synchronization mechanism, the slave and master databases synchronize most data changes, enabling seamless switching and achieving complete data consistency and high availability. This invention's real-time automatic database data synchronization method for dual-active architecture achieves efficient, reliable, and real-time data synchronization through master-slave architecture, data queues, metadata synchronization, and data snapshots, providing strong support for dual-active architecture database systems. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This invention provides a flowchart of an automatic data synchronization method for a real-time database with a dual-active architecture. Figure 2 This is a flowchart illustrating the bidirectional real-time data synchronization process in an embodiment of the present invention. Figure 3 This is a data synchronization flowchart according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the second method for accessing data with missing conditions according to an embodiment of the present invention; Figure 5 The data writing process flowchart provided for this invention; Figure 6 The present invention also provides an automatic data synchronization system for a real-time database with a dual-active architecture. Detailed Implementation

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. This invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of this invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0020] Terminology Explanation: An API (Application Programming Interface) is a set of definitions, procedures, and protocols that enable the building and development of software applications, allowing these applications to interact with other software applications or operating systems. Simply put, an API is a communication bridge between software programs, allowing data exchange and function calls between them. The functions of an API are as follows: Data sharing: APIs allow different applications to share data. For example, an e-commerce website can use an API to provide its product information to a price comparison website.

[0021] Function reuse: Developers can use existing APIs to implement specific functions without having to write code from scratch, which can save development time and costs.

[0022] System integration: Enterprises can integrate their own systems with other systems (such as payment systems, logistics systems, etc.) through APIs to automate and seamlessly connect business processes.

[0023] Extensibility: The API provides a flexible way to extend the functionality of an application without modifying its core code.

[0024] This method is based on the premise of data synchronization between two nodes in a real-time database active-active architecture. If there is a single node offline hot standby or two or more database nodes, it is not within the scope of this method.

[0025] The purpose of this invention is to address the problem in the prior art that it is difficult to balance data synchronization success rate and data integrity consistency. It proposes an automatic data synchronization method for a dual-active architecture real-time database. By improving the data synchronization method and adding an automatic metadata comparison and priority synchronization mechanism, data access conflicts and synchronization failures are avoided. Furthermore, automatic processing is used to achieve complete data synchronization consistency, ensuring the efficient and normal operation of a dual-active high-availability real-time database.

[0026] like Figure 1 As shown, the first objective of this invention is to provide a method for automatic data synchronization of a real-time database with a dual-active architecture, comprising the following steps: S11, after the real-time database service starts, the master database is responsible for handling the client's time-series data write requests. After receiving the data, the master database writes the data to the synchronization data queue and then writes the data to the storage. S12, when there is data in the synchronization data queue, write it to the synchronization cache file and prepare to send it to the slave for synchronization; S13: Compare the metadata information of the master database and the slave database, and check whether the metadata versions are consistent. If the metadata is out of sync, pause data synchronization and start metadata synchronization; if the metadata is synchronized, the data synchronization process continues. S14: After receiving the synchronization data from the database, write the synchronization data.

[0027] The principle of the dual-active architecture real-time database automatic data synchronization method of the present invention is as follows: Master-Slave Architecture and Data Writing: The system consists of two database instances, a master and a slave, forming a dual-active architecture. The master database handles time-series data write requests from clients and serves as the entry point for the database system. Upon receiving a data write request from a client, the master database first writes the data to a synchronization data queue. This approach temporarily caches the data, avoiding performance bottlenecks or data loss risks that might occur if the data is written directly to storage.

[0028] At the same time, the master database will also write the data directly to its own storage to ensure data persistence.

[0029] Data synchronization preparation: When there is data to be synchronized in the synchronization data queue, the system retrieves this data from the queue and writes it to a synchronization cache file. This step prepares the data for transmission from the master database to the slave database for synchronization. The advantage of using a synchronization cache file instead of directly transmitting data over the network is that it reduces the real-time requirements of network transmission and allows for batch processing of synchronization data, improving synchronization efficiency.

[0030] Metadata synchronization: Before data synchronization, the system compares the metadata (such as database table structure, tags, indexes, etc.) of the master and slave databases to check if the metadata on both ends is consistent. Metadata is an important component of a database system, describing the structure, attributes, and constraints of the data. If the metadata of the master and slave databases is inconsistent, even if the data itself is synchronized, the data may not be usable or queried correctly due to mismatches in structure and tags.

[0031] Therefore, if metadata inconsistency is detected, the system will pause data synchronization and prioritize metadata synchronization to ensure structural consistency between the two databases.

[0032] Data synchronization execution: Once the metadata of the master and slave databases is confirmed to be synchronized, the system will continue the data synchronization process. At this time, the master database will send the data in the synchronization cache file to the slave database. After receiving this synchronization data, the slave database will write the synchronization data to ensure data consistency and recoverability.

[0033] The greatest advantage of this active-active architecture lies in its high availability and fault tolerance. Since both the master and slave databases can handle data write requests and synchronize data when necessary, even if one database instance fails, the other can continue to provide services, ensuring business continuity. The active-active architecture also improves system scalability and load balancing capabilities. By rationally allocating the workload between the master and slave databases, system performance can be optimized, and data processing efficiency can be improved.

[0034] See Figure 1This invention provides a method for automatic data synchronization of a real-time database with a dual-active architecture, comprising the following steps: Step S11: After the real-time database service starts, the master database is responsible for handling the client's time-series data write requests. After receiving the data, the master database writes the data to the synchronization data queue and then writes the data to the storage. Specifically, the primary database is determined by the data arrival priority. After the real-time database service starts, it calculates the data arrival priority of the two active-active ends based on the high availability IP group configuration. The end with the first IP has a data arrival priority of 1, and the other end has a priority of 0. The API regards the node with the higher data arrival priority as the primary database and the other node as the secondary database.

[0035] In this embodiment of the invention, the two ends refer to the master database and the slave database.

[0036] For example, one node of the real-time database is configured with IP address 126.168.3.1, and the other node is configured with IP address 126.168.3.2. After the dual-active service is started, the data arrival priority of the node with IP address 126.168.3.1 is set to 1, and the data arrival priority of the node with IP address 126.168.3.2 is set to 0. After the API connects to both ends, the node with IP address 126.168.3.1 is regarded as the master database, and the other node is regarded as the slave database.

[0037] Step S12: When there is data in the synchronization data queue, write it to the synchronization cache file, ready to send it to the slave for synchronization; Specifically, data synchronization in steps S11 and S12 is performed bidirectionally in real time, such as... Figure 2 As shown, any metadata changes and data updates on one end will be automatically synchronized to the other end. When a client sends a real-time data write request, the master database will handle it first by default, and the master database will synchronize the data with the slave database. When a historical data write request is sent, the two nodes will perform load balancing, and the two ends will synchronize the data. If the current master database is overloaded, the overloaded master database will be switched to a new slave database, and the remaining data will continue to be synchronized to the other end. The slave database will be regarded as the new master database, receiving data and synchronizing it with the other end.

[0038] In this embodiment of the invention, "peer" refers to the master database and the slave database.

[0039] See Figure 3For example, when a client sends a real-time data write request, the master database receives it. The master database's data synchronization process involves a main synchronization thread, a synchronization cache thread, and a synchronization connection thread. The main synchronization thread, upon receiving the client's data write request, writes data to the synchronization data queue of the synchronization cache object and writes the real-time data to storage. The synchronization cache thread repeatedly checks the data queue for data, retrieves it in batches, and writes it to the synchronization cache file. The synchronization connection thread repeatedly checks the synchronization cache file for data, retrieves it in batches, and prepares to send it to the slave database. The slave database's data synchronization process involves the main synchronization thread, which, upon receiving synchronization data from the master database, writes the real-time data to storage. For example, when a client sends a historical data write request, the load balancing algorithm determines the receiving end. The receiving end repeats the process of the master database for real-time data write requests and synchronizes with the other end.

[0040] Step S13: Compare the metadata information of the master database and the slave database, and check whether the metadata versions are consistent. If the metadata is out of sync, pause data synchronization and start metadata synchronization; if the metadata is synchronized, the data synchronization process continues. Specifically, the metadata information includes database tables, tag points, tag point recycle bins, custom types, and metadata versions. When all metadata information is consistent between the two ends, the metadata versions are the same, and the two ends are in a metadata synchronization state.

[0041] Reasons for metadata asynchrony include adding new dual-active database nodes, metadata initialization, and metadata changes. Specifically, metadata change operations include adding, updating, and deleting metadata. The API performs metadata change operations on the master database, writes the modification results to the cache and metadata write-ahead log, periodically saves them to disk, increments the metadata version number after modification, and the master database synchronizes the metadata with the slave database.

[0042] For example, during data synchronization between the two ends, assuming that the metadata on both ends is completely consistent and the metadata version is 2, if a user adds 10 tags and writes data to them, the API will add 10 tags to the master database, increment the master database's metadata version by 1, and write the data to the master database. At this point, the metadata versions on both ends are inconsistent, so metadata synchronization needs to be performed first; otherwise, the data from the 10 newly added tags cannot be synchronized. At this time, the master database writes all data from the beginning of the metadata change to the cache and metadata write-ahead log, waiting for the metadata synchronization to complete. Through metadata synchronization, the slave database synchronizes 10 tags, and the metadata version is incremented by 1, making it 3 in both databases. Data synchronization then continues.

[0043] Specifically, the metadata synchronization includes two synchronization modes: full synchronization and incremental synchronization. The full synchronization mode is used to synchronize the entire metadata information from the master database to the slave database, which is suitable for adding new dual-active database nodes and initializing metadata. The incremental synchronization mode is used to update and synchronize metadata changes from the master database to the slave database in real time, which is suitable for real-time synchronization during normal operation of both ends.

[0044] Specifically, the steps for metadata synchronization in step S13 are as follows: S31: Compare the maximum version of the master database's metadata write-ahead log with the slave database's metadata version. If the maximum version of the metadata write-ahead log is the same as the slave database's metadata version, adopt the incremental synchronization mode and synchronize directly for each modification operation on the master database. In the example above, before adding the tag point, the metadata versions at both ends are consistent. At this time, the maximum version of the metadata write-ahead log is also consistent with the metadata version, both being 2. Therefore, after adding the tag point, the incremental synchronization mode is adopted to directly synchronize the newly added tag point.

[0045] S32: If the maximum version of the metadata write-ahead log is greater than the version of the slave database metadata, the incremental synchronization mode is adopted. First, the metadata write-ahead log is synchronized, and then the modification operations during the master database are put into the queue for synchronization. In the example above, suppose a new metadata change operation is performed before the metadata synchronization, such as changing the name of a database table. At this time, when comparing the metadata versions, the version of the master database metadata write-ahead log has changed to 3, which is greater than the version of the slave database metadata. The metadata write-ahead log is synchronized first, and then the master database table name change operation is put into the queue for synchronization.

[0046] S33: If the metadata versions differ too much, use the full synchronization mode to initialize the metadata; In the example above, assuming a series of metadata change operations were performed before metadata synchronization, when comparing the metadata versions, the version of the master database's metadata write-ahead log has changed to 10, which is much larger than the slave database's metadata version. The difference between the two databases' metadata is too large, so the full synchronization mode is directly adopted to initialize the slave database's metadata and fully synchronize it with the master database.

[0047] S34: If the slave is a newly added dual-active database node, the full synchronization mode is adopted. If the metadata is modified during the synchronization, the synchronization is interrupted and restarted. For example, suppose the database metadata version is 30 at this time. If the dual-active high availability mode is enabled and a new database node is added, and the initialization is performed using full synchronization mode, the metadata version of the master database will have changed to 31 before the process is completed. In this case, the previous synchronization will be interrupted and restarted.

[0048] S35: After synchronization is complete, the metadata versions on both ends are the same, and subsequent metadata synchronization will use incremental synchronization mode.

[0049] Step S14: After receiving the synchronization data from the database, write the data to the storage.

[0050] Specifically, if a real-time database node experiences a network outage or malfunction causing processing failure, automatic failover will occur during data synchronization. See also... Figure 4 When the primary database fails to process the data, the API will automatically switch to the secondary database and prioritize data arrival on the secondary database, treating it as the new primary database. Simultaneously, it will send the data in its memory cache, the data from the failed write operation, and subsequent write data to the new primary database, which will then continue data synchronization. (See also...) Figure 4 When the slave database also fails to process, the client caches the data written in this batch to a cache file. Once a data node recovers, the API treats it as the new master database and fills the cache file with the data of the new master database. Once another data node recovers, the new master database continues data synchronization.

[0051] See Figure 5 For example, suppose that during the data writing process, the API writes to the memory cache while forwarding write requests to the master database. If the network of the server where the master database is located is interrupted and the connection fails, the database node is immediately switched to the slave database. At the same time, the data in the memory cache is written to the slave database, and the slave database is promoted to the new master database. After the interrupted node is reconnected, data synchronization is performed with it.

[0052] Furthermore, assuming that the slave database also fails and the connection fails when switching slave databases, the data in the cache will continue to be written to the file cache. After any node reconnects, the data in the file cache will be read and written again to ensure data integrity.

[0053] Furthermore, if there are metadata modification operations during the above example process, the metadata modification operations that have not been written or synchronized will be saved in the metadata write-ahead log, and will be written or synchronized after the node reconnects.

[0054] This invention is applied to a self-developed dual-active high-availability real-time database. After deploying and enabling dual-active database nodes, data synchronization can be performed through the above embodiments. Currently, it is mainly used in large-scale thermal power plant-level information monitoring systems. Applying this invention can effectively improve the efficiency and effectiveness of real-time database data synchronization, ensure high data availability, and achieve the application effect of no data loss and no business interruption.

[0055] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0056] like Figure 6 As shown, the present invention also provides an automatic data synchronization system for a dual-active architecture real-time database, comprising: Data write API: Used to connect the active and passive ends of a real-time database, identify the master and slave databases based on the data arrival priority, distribute the time-series data and metadata written from the client to the master and slave databases according to the data synchronization strategy, and write them to the cache; The main data synchronization thread is used to repeatedly process time-series data write requests from clients and synchronous data write requests from peers after the database starts. When the master database receives data, it writes the data to the synchronous data queue and then writes the data to storage; when the slave database receives synchronous data, it writes the data directly to storage. Data synchronization cache thread: used to repeatedly check the data in the synchronization data queue and write it to the synchronization cache file in batches; Data synchronization connection thread: used to repeatedly check the data in the synchronization cache file and send it in batches to the slave database for synchronization; Metadata synchronization thread: Used to repeatedly compare the metadata information of the active and passive ends of the real-time database, and query whether the metadata versions are consistent. When the metadata of the two ends is out of sync, the data synchronization connection thread is notified to pause data synchronization and perform metadata synchronization.

[0057] For example, when data is written into the API, the primary database is determined by the data arrival priority. After the real-time database service starts, the data arrival priority of the two active-active ends is calculated according to the high availability IP group configuration. The data arrival priority of the end with the first IP is 1, and the data arrival priority of the other end is 0. The API regards the node with the higher data arrival priority as the primary database and the other node as the secondary database.

[0058] The data synchronization strategy specifically includes: real-time data write requests are prioritized by the master database, which synchronizes the data with the slave database; historical data write requests are handled by load balancing between the two nodes, and the two ends synchronize the data; when the current master database is overloaded, the overloaded master database is switched to a new slave database, and the remaining data is synchronized to the other end. The slave database is then regarded as a new master database, receiving data and synchronizing it with the other end.

[0059] The cache includes: memory cache, file cache, and metadata write-ahead log.

[0060] For example, in the metadata synchronization thread, metadata synchronization includes two synchronization modes: full synchronization and incremental synchronization. Specifically, the full synchronization mode is used to synchronize the entire metadata information from the master database to the slave database, which is suitable for adding a dual-active database node and initializing metadata; the incremental synchronization mode is used to update and synchronize metadata changes from the master database to the slave database in real time, which is suitable for real-time synchronization during normal operation of both ends.

[0061] As an optional embodiment, the specific steps of metadata synchronization in the metadata synchronization thread are as follows: S31: Compare the maximum version of the master database's metadata write-ahead log with the slave database's metadata version. If the maximum version of the metadata write-ahead log is the same as the slave database's metadata version, adopt the incremental synchronization mode and synchronize directly for each modification operation on the master database. S32: If the maximum version of the metadata write-ahead log is greater than the version of the slave database metadata, the incremental synchronization mode is adopted. First, the metadata write-ahead log is synchronized, and then the modification operations during the master database are put into the queue for synchronization. S33: If the metadata versions differ too much, use the full synchronization mode to initialize the metadata; S34: If the slave is a newly added dual-active database node, the full synchronization mode is adopted. If the metadata is modified during the synchronization, the synchronization is interrupted and restarted. S35: After synchronization is complete, the metadata versions on both ends are the same, and subsequent metadata synchronization will use incremental synchronization mode.

[0062] The third objective of this invention is to provide an automatic data synchronization method for a real-time database with a dual-active architecture, applied to a client, comprising the following steps: The system connects to both ends of a real-time database in an active-active manner. It identifies the master and slave databases based on the data arrival priority. It distributes time-series data and metadata written from the client to the master and slave databases according to the data synchronization strategy and writes them to the cache. When the master database receives data, it writes the data to the synchronization data queue and then writes the data to the storage. When the slave database receives synchronization data, it directly writes the data to the storage. The data in the synchronization data queue is repeatedly checked and written to the synchronization cache file in batches; the master database repeatedly checks the data in the synchronization cache file and sends it to the slave database in batches for synchronization. Repeatedly compare the metadata information of the two active-active ends of the real-time database, and check whether the metadata versions are consistent. When the metadata of the two ends is out of sync, notify the data synchronization connection thread to pause data synchronization and perform metadata synchronization.

[0063] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0066] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

Claims

1. A method for automatic synchronization of data for a dual-active architecture real-time database, comprising the following steps: After the real-time database service is started, one end of the master database is responsible for processing the time sequence data write request of the client, the master database receives the data, writes the data into the synchronization data queue, and writes the data into the storage; When there is data in the synchronization data queue, write to the synchronization cache file, and prepare to send to the slave database for synchronization; Compare the metadata information of the master database and the slave database, query whether the metadata version is consistent, and if the metadata is not synchronized, suspend data synchronization and synchronize the metadata; When the metadata is synchronized, the data synchronization process continues; After the slave database receives the synchronization data, write the synchronization data. 2.The method for automatically synchronizing data of a dual-active real-time database according to claim 1, wherein: The master database is determined by the data arrival priority, after the real-time database service is started, the data arrival priority of the dual-active two ends is calculated according to the high-availability IP group configuration, the API regards the node with high data arrival priority as the master database, and the other node as the slave database. 3.The method for automatically synchronizing data of a dual-active real-time database according to claim 1, wherein: The data synchronization is carried out in real time in both directions, specifically including: The write request of real-time data is processed by the master database by default, and the master database synchronizes data to the slave database; The write request of historical data is processed by load balancing of two nodes, and the master database and the slave database synchronize data; When the current master database has a large load, the master database with a large load is switched to a new slave database, the remaining data is continuously synchronized to the opposite end, and the slave database is regarded as a new master database to receive data and synchronize to the opposite end.

4. The method for automatic data synchronization of a dual-active real-time database according to claim 1, characterized in that: The metadata information includes database tables, label points, label point recycle bins, custom types, and metadata versions, and when the metadata information of the master database and the slave database is consistent, the metadata versions are the same and in a metadata synchronization state.

5. The method for automatic data synchronization of a dual-active real-time database according to claim 1, characterized in that: The metadata synchronization includes two synchronization modes: full synchronization and incremental synchronization, specifically including:

6. The method for automatic data synchronization of a dual-active real-time database according to claim 1, characterized in that: The full synchronization mode is used for the master database to synchronize the entire metadata information to the slave database, and is used for adding dual-active database nodes and metadata initialization; The incremental synchronization mode is used for the master database to update and synchronize metadata changes in real time, and is used for real-time synchronization during normal operation of the two ends. The specific steps of the metadata synchronization are:

7. The method for automatic data synchronization of a dual-active real-time database according to claim 6, characterized in that: Compare the metadata prewrite log maximum version of the master database with the metadata version of the slave database, if the metadata prewrite log maximum version and the metadata version of the slave database are consistent, use the incremental synchronization mode, and synchronize the modification operation of the master database directly each time; If the metadata prewrite log maximum version is greater than the metadata version of the slave database, use the incremental synchronization mode, synchronize the metadata prewrite log first, and then put the modification operation of the master database during the period into the queue for synchronization; If the metadata version difference exceeds the threshold, use the full synchronization mode to initialize the metadata; If the slave database is a newly added dual-active database node, use the full synchronization mode, and if there is a modification operation during synchronization, interrupt the synchronization and start again; ​ After synchronization is completed, the metadata versions of both ends are the same, and subsequent metadata synchronization uses the incremental synchronization mode. 8.The method for automatically synchronizing data of a dual-active real-time database according to claim 1, wherein: When the nodes of the real-time database are disconnected or fail to process, the system automatically switches to the fault mode, including: When the master fails to process, the API automatically switches to the slave, and the data of the slave is given priority, and is considered as a new master. The data in the metadata pre-write log, the memory cache, the failed data and the subsequent write data are sent to the new master, and the new master continues data synchronization. When both the master and the slave fail to process, the data is cached in the cache file. When the data node is restored, the API considers it as a new master, and the data in the metadata pre-write log and the cache file is recorded to the new master. When the other data node is restored, the new master continues data synchronization.

9. A data automatic synchronization system for a dual-active architecture real-time database, characterized in that, The API for writing data is used to connect the real-time database dual-active two ends, identify the master and slave according to the data priority, distribute the time-series data and metadata from the client to the master and slave according to the data synchronization strategy, and write the data to the cache. The data synchronization main thread is used to repeatedly process the time-series data write request from the client and the synchronization data write request from the other end after the database is started. The master receives the data, writes the data to the synchronization data queue, and writes the data to the storage. The slave receives the synchronization data and writes the data to the storage. The data synchronization cache thread is used to repeatedly check the data in the synchronization data queue and write the data to the synchronization cache file in batches. The data synchronization connection thread is used to repeatedly check the data in the synchronization cache file and send the data to the slave for synchronization in batches. The metadata synchronization thread is used to repeatedly compare the metadata information of the real-time database dual-active two ends, query whether the metadata versions are consistent, and notify the data synchronization connection thread to suspend data synchronization when the metadata of both ends are not synchronized.

10. An automatic data synchronization method for a real-time database with a dual-active architecture, comprising the following steps: Connecting the real-time database dual-active two ends, identifying the master and slave according to the data priority, distributing the time-series data and metadata from the client to the master and slave according to the data synchronization strategy, and writing the data to the cache; the master receives the data, writes the data to the synchronization data queue, and writes the data to the storage; the slave receives the synchronization data and writes the data to the storage; Repetitively checking the data in the synchronization data queue and writing the data to the synchronization cache file in batches; The master repeatedly checks the data in the synchronization cache file and sends the data to the slave for synchronization in batches. Repetitively comparing the metadata information of the real-time database dual-active two ends, querying whether the metadata versions are consistent, and notifying the data synchronization connection thread to suspend data synchronization when the metadata of both ends are not synchronized. ​

Images