Database synchronization method and distributed database system

Abstract

The application provides a database synchronization method and a distributed database system. The database synchronization method comprises the steps of data acquisition, data verification, result judgment and screening synchronization. The database synchronization method adopts a 1-to-1 comparison and verification data synchronization mode of database to database, improves the utilization rate of each database in the distributed database system, reduces the synchronization difficulty between databases in the multi-database system, and further reduces the development difficulty of the distributed database system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a database synchronization method and a distributed database system. Background Technology

[0002] For distributed systems, data synchronization across various databases is fundamental to improving system efficiency and stability. Industrial and shipbuilding systems are complex and require persistent storage of system data using real-time or time-series databases. Traditional data synchronization, supported or provided by the database itself, relies on internal proprietary protocols, hindering database software updates and iterations, and impeding the development of distributed systems and microservices. Furthermore, different databases offer varying synchronization methods, necessitating the development of specific interface adaptations, which presents challenges for software application and deployment. Moreover, synchronization operations are scattered across various system components, hindering efficient database utilization and maintenance, and increasing project development difficulty and complexity. Summary of the Invention

[0003] The purpose of this invention is to provide a database synchronization method and a distributed database system to solve the technical problems that existing database synchronization methods are not conducive to the development of distributed systems, cannot improve database utilization, and affect the operation and maintenance of databases.

[0004] To achieve the above objectives, the present invention provides a database synchronization method, which includes the following steps:

[0005] Data Acquisition: The first database receives and stores data transmitted from external data sources;

[0006] Data verification: The synchronous sending module and the synchronous receiving module respectively acquire historical data information of the first database and the second database within the same time period and perform verification;

[0007] Result judgment: The synchronization sending module determines whether synchronization is needed based on the verification result; wherein, when the verification results are consistent, the first database and the second database do not need to be synchronized; when the verification results are inconsistent, the filtering synchronization step is initiated.

[0008] Filtering and Synchronization: Filter out the data that needs to be synchronized in the first database and the second database within the same time period, and send the filtered data to the second database.

[0009] Further, the data checking step comprises: the synchronization sending module acquires first historical data information in the first database in the time period and sends to the synchronization receiving module; the synchronization receiving module acquires second historical data information in the second database in the time period and checks the second historical data information with the first historical data information; the synchronization receiving module sends the checking result to the synchronization sending module.

[0010] Further, the screening and synchronizing step comprises: the synchronization sending module and the synchronization receiving module respectively divide the time period into multiple first time slices of the same size and take each first time slice as a first storage point; the synchronization sending module and the synchronization receiving module respectively acquire data amounts of the first database and the second database in each first storage point; the synchronization sending module sends the data amount of each first storage point in the first database to the synchronization receiving module, the synchronization receiving module compares the data amount of each first storage point in the second database with the data amount of the corresponding first storage point in the first database, and sends the comparison result of each first storage point to the synchronization sending module; the synchronization sending module receives the comparison result of the first storage point and finds out the first time slice with inconsistent data amount according to the comparison result, and the first storage point corresponding to the first time slice contains data needing synchronization.

[0011] Further, the screening and synchronizing step further comprises: the synchronization sending module and the synchronization receiving module respectively divide the first time slice into multiple second time slices of the same size and take each second time slice as a second storage point; the synchronization sending module and the synchronization receiving module respectively acquire data amounts of the first database and the second database in each second storage point; the synchronization sending module sends the data amount of each second storage point in the first database to the synchronization receiving module, the synchronization receiving module compares the data amount of each second storage point in the second database with the data amount of the corresponding second storage point in the first database, and sends the comparison result of each second storage point to the synchronization sending module; the synchronization sending module receives the comparison result of the second storage point and finds out the second time slice with inconsistent data amount according to the comparison result, and the corresponding data in the second time slice is the data needing synchronization; the synchronization sending module sends the corresponding data in the time slice to the synchronization receiving module.

[0012] Further, before the data checking step, the method further comprises a time period obtaining step, which comprises: the synchronization sending module obtains the data insertion time closest to the running time in the first database, denoted as a first time point; the first time point is offset by 1 minute to 5 minutes forward, denoted as a second time point; the second time point is offset by 5 to 60 minutes forward, denoted as a third time point; and the time interval before the second time point and the third time point is taken as the time period.

[0013] Further, the database synchronization method further comprises the following steps: changing the time interval in which the time period is located, and performing the data checking step again. In the Nth data checking step, the maximum value in the time period interval is equal to the minimum value in the time period interval in the N+1th data checking step, and N is a positive integer greater than or equal to 1.

[0014] Further, the database synchronization method further comprises the following steps: when the minimum value in the time period is separated from the first time point by 24 hours or more, the interval range of a single time period is expanded, and the interval range is modified to 24 hours or more.

[0015] Further, the synchronization sending module and the synchronization receiving module communicate through a network long connection, and the information transmitted between the synchronization sending module and the synchronization receiving module is compressed by an lz4 compression algorithm before being sent.

[0016] The present application provides a distributed database system, which uses the database synchronization method as described above to synchronize data.

[0017] The distributed database system comprises at least one first database, at least one second database, at least one synchronization sending module, and at least one synchronization receiving module. The synchronization sending module is in communication connection with the first database and the synchronization receiving module respectively. The synchronization receiving module is in communication connection with the second database and the synchronization sending module respectively. The first database and the second database realize data filtering and synchronization through the synchronization sending module and the synchronization receiving module.

[0018] Further, the first database and the second database are each provided with a data input module and a data output module. The data input module is used for inputting data into the first database or the second database, and the data output module is used for calling out data in the first database or the second database. The distributed database system further comprises at least one data query module, which is in communication connection with the data output module of the first database or the data output module of the second database.

[0019] Further, the distributed database system further comprises a log module, which records the running information of the distributed database system by adding a flag bit.

[0020] The advantage of the present application is that the database synchronization method in the present application adopts a 1-to-1 comparison and verification data synchronization method of database to database, and develops the distributed database system based on this, which encapsulates data operation interfaces, obtains data that needs to be inserted in real time, and forwards the data from the edge end to the center end through the network, at the same time, according to time slicing, checks the data volume of historical data at regular intervals, further divides the time slices for which the data volume is inconsistent, and synchronizes the time slice data after reaching the minimum time slice, so as to improve the utilization rate of each database in the distributed database system, reduce the synchronization difficulty between databases in the multi-database system, and further reduce the development difficulty of the distributed database system. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 The synchronization principle diagram of the data synchronization method in the embodiment of the present application is shown in the figure.

[0023] Figure 2 The synchronization architecture diagram of the data synchronization method in the embodiment of the present application is shown in the figure.

[0024] Figure 3 The architecture diagram of the distributed database system in the embodiment of the present application is shown in the figure.

[0025] Figure 4 The architecture diagram of the multi-to-1 type database system in other embodiments of the present application is shown in the figure.

[0026] Figure 5 The architecture diagram of the 1-to-multi type database system in other embodiments of the present application is shown in the figure.

[0027] Figure 6 The architecture diagram of the bidirectional type database system in other embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present application will be described herein below with reference to the accompanying drawings, of which the preferred embodiments of the present application are illustrated to demonstrate that the present application can be implemented, and the preferred embodiments of the present application can be used to completely introduce the present application to those skilled in the art, so that the technical contents of the present application can be more clearly and easily understood. The present application can be embodied in many different forms of the present application, and the scope of protection of the present application is not limited to the embodiments described herein.

[0029] In the drawings, components of the same structure are denoted by the same reference numerals, and components having similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application is not limited to the size and thickness of each component. In order to make the drawings clearer, the thickness of some components is appropriately exaggerated in some places in the drawings.

[0030] In addition, the following description of the embodiments of the present application is made with reference to the accompanying drawings, which illustrate specific embodiments of the present application that can be used to implement the present application. The directional terms mentioned in the present application, such as "upper", "lower", "front", "rear", "left", "right", "inner", "outer", etc., are only with reference to the direction of the accompanying drawings, and therefore, the directional terms used are for better, clearer and easier understanding of the present application, and are not intended to indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third", etc. are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0031] When a component is described as "on" another component, the component can be directly placed on the other component; or there can be an intermediate component, and the component is placed on the intermediate component, and the intermediate component is placed on the other component. When a component is described as "mounted to" or "connected to" another component, it can be understood as being directly "mounted" or "connected", or the component is indirectly "mounted to" or "connected to" the other component through an intermediate component.

[0032] A database synchronization method is provided in the embodiments of the present application to synchronize data between multiple databases. Specifically, the synchronization principle of the data synchronization method is as shown in Figure 1 The synchronization architecture of the data synchronization method is as shown in Figure 2 The database synchronization method includes steps S10-S80.

[0033] Step S10) Data acquisition:

[0034] The input end of the first database is in communication connection with the output end of a data input module, and meanwhile, the output end of the data input module is in communication connection with an external data source, the external data source sends external data to the data input module, and the data input module inserts the received external data into the first database. Meanwhile, the data input module can also directly transmit the received data to a synchronous sending module.

[0035] Step S20) obtaining a time period:

[0036] The first database is in communication connection with a synchronous sending module, the second database is in communication connection with a synchronous receiving module, the synchronous sending module is in communication connection with the synchronous receiving module, and the first database and the second database are in communication connection through the synchronous sending module and the synchronous receiving module.

[0037] The synchronous sending module obtains the data insertion time closest to the running time in the first database, which is recorded as a first time point. Since the external data source will insert new data into the first database in real time, in order to prevent the newly inserted data from affecting the verification of the old data, the first time point is shifted forward by 1 minute to 5 minutes, preferably, by 2 minutes, which is recorded as a second time point. The second time point is shifted forward by 5 to 60 minutes, which is recorded as a third time point, and the time interval between the second time point and the third time point is taken as a time period; that is, when the second time point and the third time point are different by 30 minutes, the time period is 30 minutes. Preferably, the time period is 10 minutes.

[0038] Step S30) data verification:

[0039] The synchronous sending module obtains the first historical data information of the first database in the time period and sends it to the synchronous receiving module. The synchronous receiving module obtains the second historical data information of the second database in the same time period, verifies the second historical data information with the first historical data information, and sends the verification result to the synchronous sending module.

[0040] The first historical data information and the second historical data information respectively include single station information of the corresponding database in the time period, a timestamp of a time region, and a data size of the corresponding database in the time period.

[0041] Step S40) result judgment:

[0042] The synchronous sending module judges whether the data of the first database and the second database in the time period needs to be synchronized according to the received verification result.

[0043] Specifically, when the check result is consistent, it means that the data in the first database is the same as the data in the second database in the time period, and synchronization is not needed, and the step S60 is jumped to obtain a new time period and perform data checking according to the new time period; when the check result is inconsistent, it means that the data in the first database is different from the data in the second database in the time period, and the synchronization step is entered.

[0044] Step S50) screening synchronization:

[0045] The synchronization sending module and the synchronization receiving module respectively divide the time period into a plurality of first time slices of the same size, and take each first time slice as a first storage point. For example, a 10-minute time period can be divided into 20 first time slices, each first time slice has a time interval of 30 seconds, and each first time slice does not overlap in the time period.

[0046] The synchronization sending module and the synchronization receiving module respectively obtain the data size of each first storage point in the first database and the second database.

[0047] The synchronization sending module sends the data size of each first storage point in the first database to the synchronization receiving module, the synchronization receiving module compares the data size of each first storage point in the second database with the data size of the corresponding first storage point in the first database, and sends the comparison result of each first storage point to the synchronization sending module. The synchronization sending module receives the comparison result sent by the synchronization receiving module, and finds the first time slice with inconsistent data size according to the comparison result.

[0048] The synchronization sending module and the synchronization receiving module respectively divide the first time slice into a plurality of second time slices of the same size, and take each second time slice as a second storage point. For example, a 30-second first time slice can be divided into 20 second time slices, each second time slice has a time interval of 1 second, and each second time slice does not overlap in the first time slice. The corresponding data in the second time slice is the data that needs to be synchronized.

[0049] The synchronization sending module sends the corresponding data in the second time slice with inconsistent comparison result to the synchronization receiving module, and the synchronization receiving module inserts the received data into the second database, thereby completing the synchronization between the data in the first database and the data in the second database in the same time period.

[0050] Step S60) obtaining a new time period:

[0051] When the data in the first database and the second database in the current time period is consistent or when the data in the first database and the second database in the current time period is synchronized, the interval of the time period is changed, a new time period is obtained, and the data checking step, the result judging step and the screening synchronization step are run based on the newly obtained time period.

[0052] In the Nth data checking step, the maximum value in the interval of the time period is equal to the minimum value in the interval of the time period in the N+1th data checking step, and N is a positive integer greater than or equal to 1. For example, the maximum value of the interval of the time period in the first data checking step is 3 o'clock, and the minimum value of the time period in the second data checking step is 3 o'clock.

[0053] Step S70) expanding the interval range of the time period in single checking:

[0054] When the minimum value in the newly obtained time period in step S60 is separated from the first time point by 24 hours or more, the interval range of the time period in single checking can be expanded, and the interval range can be modified to 24 hours or more. For example, the interval range of the time period in single checking is expanded to 24 hours, that is, the data amount in 24 hours is checked each time, and comparison is performed. If the comparison result is inconsistent, the time period is further divided, and the checking, comparison and synchronization are refined according to steps S30-S50.

[0055] Step S80) twice expanding the interval range of the time period in single checking:

[0056] When the minimum value in the newly obtained time period is separated from the first time point by 72 hours or more, the interval range of the time period in single checking can be further expanded, and the interval range can be modified to 30 days or more. For example, the interval range of the time period in single checking is expanded to 30 days, that is, the data amount in 30 days is checked each time, and comparison is performed. If the comparison result is inconsistent, the time period is further divided, and the checking, comparison and synchronization are refined according to steps S30-S50.

[0057] In the embodiment of the application, a distributed database system is also provided, as shown in Figure 3As shown, the distributed data system has a plurality of database service ends, each of which is provided with a database, a data input module and a data output module. The input end of the database is in communication connection with the output end of the data input module, and the output end of the database is in communication connection with the data output module. The data input module is used for receiving external data and inserting the received data into the database. The data output module is used for receiving external data query requests and obtaining corresponding data from the database. Preferably, the database is a time series database.

[0058] Further, the output end of the data output module in each database service end can be respectively in communication connection with a data query module. The data query module can send a query command for corresponding data to the data output module, and the data output module can return the corresponding data to the data query module according to the query command.

[0059] Specifically, the database service end includes a first edge end, a second edge end, a first center end, a second center end and a third center end.

[0060] The first edge end is further provided with a management module, and the management module is used for parameter configuration interface of the edge database.

[0061] The first edge end and the second edge end are each provided with a synchronous sending module, the first center end, the second center end and the third center end are each provided with a synchronous receiving module, and the second center end is further provided with a synchronous sending module. The synchronous sending module and the synchronous receiving module are respectively in communication connection with the corresponding database and are used for realizing the interaction of database information.

[0062] Among them, the synchronous sending module of the first edge end is further in communication connection with the synchronous receiving modules of the first center end and the second center end. The synchronous sending module of the second edge end is in communication connection with the synchronous receiving module of the second center end. The synchronous sending module of the second center end is in communication connection with the synchronous receiving module of the third center end.

[0063] Specifically, the data in the database in the first edge end and the data in the databases in the first and second center ends can be respectively synchronized through the above database synchronization method. At this time, between the first edge end and the first center end, the database in the first edge end can be regarded as a first database in the database synchronization method, and the database in the first center end can be regarded as a second database in the database synchronization method, so as to realize data synchronization between the first edge end and the first center end. Between the first edge end and the second center end, the database in the first edge end can be regarded as a first database in the database synchronization method, and the database in the second center end can be regarded as a second database in the database synchronization method, so as to realize data synchronization between the first edge end and the second center end.

[0064] The data in the database in the second edge end and the data in the database in the second center end can also be respectively synchronized through the above database synchronization method. At this time, between the second edge end and the second center end, the database in the second edge end can be regarded as a first database in the database synchronization method, and the database in the second center end can be regarded as a second database in the database synchronization method, so as to realize data synchronization between the second edge end and the second center end.

[0065] The data in the database in the second center end and the data in the database in the third center end can also be respectively synchronized through the above database synchronization method. At this time, between the second center end and the third center end, the database in the second center end can be regarded as a first database in the database synchronization method, and the database in the third center end can be regarded as a second database in the database synchronization method, so as to realize data synchronization between the second center end and the third center end.

[0066] Further, in order to improve the data communication efficiency, the communication between the synchronization sending module and the synchronization receiving module is established through a network long connection, and the information transmitted between the synchronization sending module and the synchronization receiving module is pre-compressed through an lz4 compression algorithm before being sent.

[0067] Further, the distributed database system also has a log module, which records the running information of the distributed database system by adding an internal flag bit as the running state basis of the distributed database system. The flag bit includes a start flag bit, a stop flag bit and a running flag bit.

[0068] When the database system is normally started, the first valid data received by any database in the database system is taken as the start time of the current system operation and is inserted into the database as a start flag time.

[0069] When the database system is normally stopped, the last valid data received by any database before the system is stopped is taken as the stop time of the current system operation and is inserted into the database as a stop flag time. When the database system is normally started, the last valid data is taken as the end time of the previous system operation and is inserted into the database as a stop flag time.

[0070] When the database system is normally operated for a long time, the running flag time can be inserted into the database at regular intervals to record and cut time slices.

[0071] When an operator wants to check the operation of the distributed database system, the start flag time, the stop flag time and the running flag time can be called to analyze the running time and running progress of the distributed system.

[0072] In the embodiment of the application, the database synchronization method adopts a 1-to-1 comparison and verification data synchronization mode of database-to-database, and the distributed database system is developed based on the mode. The distributed database system encapsulates a data operation interface, acquires data to be inserted in real time, and forwards the data from an edge end to a center end through a network. Meanwhile, the distributed database system checks the data volume of historical data at regular intervals according to time slicing, further divides time slices for which the data volume is inconsistent, synchronizes the data of the time slices after the time slices are divided to the minimum, so as to improve the utilization of each database in the distributed database system and reduce the synchronization difficulty between databases in the multi-database system, thereby reducing the development difficulty of the distributed database system.

[0073] In other embodiments of the application, a multi-to-1 type distributed database system as shown in Figure 4 a 1-to-multi type distributed database system as shown in Figure 5 or a bidirectional type distributed database system as shown in Figure 6 The communication mode and synchronization mode between databases of the distributed database systems are the same as those of the distributed database system provided in the embodiment of the application, and thus will not be described in detail. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application.

[0074] While the application has been described with reference to particular embodiments thereof, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present application. It will be apparent to those skilled in the art that numerous modifications can be made within the scope of the present application as defined by the appended claims. It is intended that all such modification fall within the spirit and scope of the present application. It will be understood that the features described in connection with one embodiment can be used in connection with another embodiment.

Claims

1. A database synchronization method, characterized by, The method comprises the following steps: Data acquisition: a first database receives and stores data transmitted by an external data source; Data verification: a synchronization sending module and a synchronization receiving module respectively acquire historical data information of the first database and a second database in a same time period, and perform verification; Result judgment: the synchronization sending module judges whether synchronization is needed according to the verification result; when the verification result is consistent, the first database and the second database do not need to be synchronized; when the verification result is inconsistent, a screening synchronization step is entered; Screening synchronization: data needing to be synchronized in the first database and the second database in the same time period is screened out, and the screened data is sent to the second database; wherein the synchronization sending module and the synchronization receiving module respectively divide the time period into a plurality of first time slices of the same size, cut the first time slices in which the data quantity of the first database and the second database is inconsistent in the time period, and synchronize the data corresponding to the minimum time slice when the minimum time slice is reached.

2. The database synchronization method of claim 1, wherein, The data verification step comprises: The synchronization sending module acquires first historical data information in the first database in the time period, and sends the first historical data information to the synchronization receiving module; The synchronization receiving module acquires second historical data information in the second database in the time period, and verifies the second historical data information with the first historical data information; The synchronization receiving module sends the verification result to the synchronization sending module.

3. The database synchronization method of claim 1, wherein, The screening synchronization step comprises: Each first time slice is taken as a first storage point; The synchronization sending module and the synchronization receiving module respectively acquire the data quantity of the first database and the second database in each first storage point; The synchronization sending module sends the data quantity of each first storage point in the first database to the synchronization receiving module, the synchronization receiving module compares the data quantity of each first storage point in the second database with the data quantity of the corresponding first storage point in the first database, and sends the comparison result of each first storage point to the synchronization sending module; The synchronization sending module receives the comparison result of the first storage point, and finds out the first time slice with inconsistent data quantity according to the comparison result, wherein the first storage point corresponding to the first time slice contains data needing to be synchronized.

4. The database synchronization method of claim 3, wherein, The screening synchronization step further comprises: The synchronization sending module and the synchronization receiving module respectively divide the first time slice with inconsistent data quantity into a plurality of second time slices of the same size, and take each second time slice as a second storage point; The synchronization sending module and the synchronization receiving module respectively acquire the data quantity of the first database and the second database in each second storage point; The synchronization sending module sends the data volume of each second storage point in the first database to the synchronization receiving module, the synchronization receiving module compares the data volume of each second storage point in the second database with the data volume of the corresponding second storage point in the first database, and sends the comparison result of each second storage point to the synchronization sending module; The synchronization sending module receives the comparison result of the second storage point, and finds out the second time slice with inconsistent data volume according to the comparison result, and the corresponding data in the second time slice is the data that needs to be synchronized; The synchronization sending module sends the corresponding data in the time slice to the synchronization receiving module.

5. The database synchronization method of claim 1, wherein, Before the data checking step, there is also a time period obtaining step, which includes: The synchronization sending module obtains the data insertion time closest to the running time in the first database, which is recorded as the first time point; The first time point is offset by 1 minute to 5 minutes forward, which is recorded as the second time point; The second time point is offset by 5 to 60 minutes forward, which is recorded as the third time point; The time interval between the second time point and the third time point is taken as the time period.

6. The database synchronization method of claim 1, wherein, It also includes the following steps: Change the time interval of the time period, and execute the data checking step again; Wherein, the maximum value in the time period interval in the Nth data checking step is equal to the minimum value in the time period interval in the N+1th data checking step, N is a positive integer greater than or equal to 1.

7. The database synchronization method of claim 6, wherein, It also includes the following steps: When the minimum value in the time period interval is more than 24 hours apart from the first time point, the interval range of the single time period is expanded, and the interval range is modified to more than 24 hours.

8. The database synchronization method of claim 1, wherein, The communication between the synchronization sending module and the synchronization receiving module is established through network long connection, and the information transmitted between the synchronization sending module and the synchronization receiving module is pre-compressed by lz4 compression algorithm before being sent out.

9. A distributed database system, characterized by The database synchronization method according to any one of claims 1-8 is used for data synchronization; The distributed database system includes: At least one first database, at least one second database, at least one synchronization sending module and at least one synchronization receiving module; The synchronization sending module is respectively connected with the first database and the synchronization receiving module in communication; The synchronization receiving module is respectively connected with the second database and the synchronization sending module in communication; The first database and the second database realize data screening and synchronization through the synchronization sending module and the synchronization receiving module.

10. The distributed database system of claim 9, wherein, The first database and the second database are respectively provided with a data input module and a data output module, the data input module is used for inputting data into the first database or the second database, and the data output module is used for calling out the data in the first database or the second database; The distributed database system also includes: At least one data query module, the data query module is connected with the data output module of the first database or the data output module of the second database in communication; Log module, the running information of the distributed database system is recorded by adding a flag bit.

Images