Heterogeneous time sequence database synchronization system

Abstract

The invention relates to the technical field of databases, in particular to a heterogeneous time sequence database synchronization system, in the system, a file is obtained from a source node through an interaction module, and a data file is obtained according to a corresponding strategy and an obtaining strategy in the file obtaining process; processing the received file through a conversion module, and segmenting file data according to a data file segmentation strategy; sending the file processed by the conversion module to a destination node through a distribution module, and distributing the file according to a data file distribution strategy; according to the invention, the high efficiency and accuracy of the data synchronization process can be ensured, and the overall performance and stability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a heterogeneous time series database synchronization system. Background Art

[0002] In the modern industrial and information age, various systems and devices generate a large amount of time series data, which records different status information at different time points in chronological order, such as sensor data, log data, etc. The management and analysis of time series data are crucial in scenarios such as equipment monitoring, fault diagnosis, and performance optimization.

[0003] Currently, many enterprises and institutions use time series databases to store and manage these time series data. Time series databases are database systems designed specifically for efficient storage, query, and processing of time series data, and usually have the characteristics of high write performance, time series compression, and query optimization. However, due to the distributed nature of systems and devices, time series data is usually scattered across multiple nodes or databases, which brings up the problem of data synchronization.

[0004] Data synchronization refers to the real-time or periodic updating and matching of data in two or more different locations or systems to maintain data consistency. Database synchronization is a specific form of data synchronization, focusing on copying and updating data between different database systems to maintain data consistency between two or more databases. Database synchronization can be performed between databases of the same type (such as two MySQL databases) or between databases of different types (such as between MySQL and Oracle).

[0005] In the process of database synchronization, the source database and the target database are two core concepts. The source database is the original source of data in the data synchronization process, and contains all the data that needs to be synchronized to another database. In most cases, the source database is an active database that continuously receives new data insertion, update, and deletion operations. The source database is usually the core database of the business system, reflecting the latest status of business data in real time. The target database is the recipient of data in the data synchronization process, receiving data from the source database and updating itself to reflect the latest data status of the source database. The purpose of the target database is to maintain data consistency with the source database, and is usually used in scenarios such as backup, analysis, report generation, or data redundancy.

[0006] In terms of time series database synchronization, the current time series database synchronization methods mainly include methods based on stream processing frameworks, methods based on change data capture, and methods based on event-driven methods. These methods have some shortcomings and defects:

[0007] 1. The high frequency generation and large data volume characteristics of time series data require that synchronization technology continuously improves data transmission and processing efficiency. Therefore, the synchronization efficiency of existing time series database synchronization methods still has room for improvement.

[0008] 2. Existing time series database synchronization methods mainly synchronize the data in the database, but not the structure of the database. When the database structures of the source database and the target database are inconsistent, the database structure of the target database needs to be modified to be consistent with the source database before synchronizing the data. In addition, if the database structure of the source database changes during data synchronization, problems will occur in the data synchronization process.

[0009] 3. Due to the requirements of time series databases for high storage capacity, high write rate and high availability, most time series databases use a distributed architecture. However, the existing time series database synchronization methods cannot dynamically adjust the synchronization strategy and synchronization rate according to the load conditions of the source database and the target database during synchronization. Summary of the invention

[0010] In order to synchronize data and database structures between two heterogeneous time series databases, and to predict the load of each distributed node of the source database and the target database at the next moment according to the real-time and historical load of each distributed node of the source database and the target database, and dynamically adjust the data file acquisition strategy, data file segmentation strategy and data file distribution strategy, so as to improve the synchronization rate, the present invention proposes a heterogeneous time series database synchronization system for synchronizing data between the source database and the target database, including a configuration module, a monitoring module, a cache module, a prediction module, an interaction module, a filtering module, a conversion module and a distribution module, wherein:

[0011] The interaction module is used to obtain load information and data files from the source database, and send the load information to the monitoring module and send the data files to the filtering module;

[0012] The monitoring module is used to pre-process the acquired load information and store the pre-processed load information in the cache module;

[0013] The prediction module is used to predict the load of the node at the next moment based on the historical load of each node in the database;

[0014] The filtering module is used to filter the acquired data files and delete the file data that does not need to be synchronized;

[0015] Configuration module, used to set the configuration parameters of the source database and the target database;

[0016] A conversion module is used to divide the data to be synchronized into multiple data files and convert the data files into the file format of the target database;

[0017] The distribution module is used to send file data or operation statements to the corresponding nodes in the target database according to the load conditions and configuration information of the nodes;

[0018] The interaction module acquires the data file according to the corresponding strategy and the acquisition strategy during the process of acquiring the file, the conversion module segments the file data according to the data file segmentation strategy, and the distribution module distributes the file according to the data file distribution strategy.

[0019] Furthermore, the correspondence strategy between data files and source database distributed nodes is as follows: if the number of data files is less than or equal to the number of source database distributed nodes, the data files are sorted according to file size, and correspond one-to-one with the load size sorting of the source database distributed nodes, and the data file with the largest file size is obtained from the distributed node with the smallest load; if the number of data files is greater than the number of source database distributed nodes, then after the first correspondence, the remaining data files are re-correlated one-to-one with the source database distributed nodes.

[0020] Furthermore, the source database agent's acquisition strategy for obtaining the source database data file is as follows: the source database agent will search for the source database's data file according to the data file path in the configuration information. If the data file can be obtained, the data file will be directly sent to the interaction module of the synchronization tool. If the data file cannot be obtained, the source database agent needs to query the data through the database API interface in a polling manner, and generate the data file according to the data file format of the configuration information, and then send the generated data file to the interaction module of the synchronization tool.

[0021] Furthermore, the conversion module divides the data files of the source database into several small data files. The division process includes:

[0022] Obtain the load size LB (n) of each distributed node of the target database calculated by the prediction module j ), determine a transmission size T according to the load size of all distributed nodes of the target database, and extract a data file set F with a total size of about T from the data file set F that needs to be synchronized this time T ;

[0023] According to the load size of the node, N is divided into high-load nodes N H , medium load node N M and low-load node N L , whose proportions are p H 、p M and p L , and p H +p M +p L =1;

[0024] The data file set F with a total size of about T extracted T is divided into small data file sets F with a size of S S , medium data file sets F with a size of M M and large data file sets F with a size of H H . Adjust the sizes of each set according to the proportion to ensure that the total size is close to T, that is, the total number of files is closest to T but does not exceed T.

[0025] Furthermore, in the process of splitting the data file set, a time-based splitting strategy or a database structure-based splitting strategy is adopted for partitioning, where:

[0026] The time-based splitting strategy is to partition according to the time range to which the time stamps of the time series data belong, reducing the size of the files transmitted each time;

[0027] The database structure-based splitting strategy is to partition according to the time series in which the time series data is located, that is, the time series data belonging to the same path is stored in a data partition in an appended manner.

[0028] Furthermore, the filtering module includes a path filtering unit, an operation filtering unit, and a time filtering unit, where:

[0029] The path filtering unit is responsible for filtering out the paths that do not need to be synchronized in the data files;

[0030] The operation filtering unit is responsible for comparing the data files synchronized historically, inferring data modification operations, data deletion operations, and the possible changed database structure, and generating corresponding data modification operation statements, data deletion operation statements, and DDL operation statements available for the target database;

[0031] The time filtering unit is responsible for filtering out the data that has been synchronized in the data files.

[0032] Furthermore, the distribution module is responsible for executing the data file distribution strategy, and this distribution strategy includes the following operations:

[0033] Sort the data file set F to be distributed in descending order of file size, sort the distributed node set N of the target database in ascending order of load, let X be the number of data files that have not been corresponded currently, and let Y be the number of distributed nodes of the target database. Repeat the following steps until F is empty:

[0034] If X < Y, select the X nodes with the smallest load: N′ = {n 1 , n 2 , …, n X}, put F and N' in one-to-one correspondence, the largest data file corresponds to the node with the smallest load;

[0035] If X=Y, directly match F and N one by one, and the largest data file corresponds to the node with the smallest load;

[0036] If X>Y, select the Y data files with the largest file size: F′={f 1 , f 2 , …, f Y}, match F′ with N one by one, the largest data file corresponds to the node with the smallest load, remove F′ from F, and update X=XY.

[0037] The agent in the present invention is independently installed on the source database end and the target database end nodes, can connect to the synchronization tool and can interact with the synchronization tool. The agent installed on the source database end is the source database agent, which is responsible for collecting the load information of the source database node and transmitting it to the synchronization tool; in addition, the agent is also responsible for capturing the data files of the source database and transmitting these files to the synchronization tool for further processing and synchronization. The agent installed on the target database end is the target database agent, which is mainly responsible for receiving the source database node load information from the synchronization tool and forwarding it to the synchronization tool; at the same time, the agent is also responsible for receiving the data files or operation statements sent by the synchronization tool, and applying these data or operation statements to the target database to realize the synchronous update of data. Through the design of this agent module, the present invention can ensure the efficiency and accuracy of the data synchronization process, while improving the overall performance and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a system schematic diagram of an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a database tree structure according to an embodiment of the present invention;

[0040] Figure 3 It is a schematic diagram of the database table structure of an embodiment of the present invention;

[0041] Figure 4 is a flow chart of obtaining source database data files according to an embodiment of the invention;

[0042] Figure 5 is a flow chart of splitting data files according to an embodiment of the present invention;

[0043] Figure 6 is a flow chart of distributing data files according to an embodiment of the present invention;

[0044] Figure 7 This is a flow chart of a source database agent acquiring a data file according to an embodiment of the present invention;

[0045] Figure 8 The present invention is a flowchart of a target database agent importing data files according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The present invention proposes a heterogeneous time series database synchronization system for synchronizing data between a source database and a target database, including a configuration module, a monitoring module, a cache module, a prediction module, an interaction module, a filtering module, a conversion module, and a distribution module, wherein:

[0048] The interaction module is used to obtain load information and data files from the source database, and send the load information to the monitoring module and send the data files to the filtering module;

[0049] The monitoring module is used to pre-process the acquired load information and store the pre-processed load information in the cache module;

[0050] The prediction module is used to predict the load of the node at the next moment based on the historical load of each node in the database;

[0051] The filtering module is used to filter the acquired data files and delete the file data that does not need to be synchronized;

[0052] Configuration module, used to set the configuration parameters of the source database and the target database;

[0053] A conversion module is used to divide the data to be synchronized into multiple data files and convert the data files into the file format of the target database;

[0054] The distribution module is used to send file data or operation statements to the corresponding nodes in the target database according to the load conditions and configuration information of the nodes;

[0055] The interaction module acquires the data file according to the corresponding strategy and the acquisition strategy during the process of acquiring the file, the conversion module segments the file data according to the data file segmentation strategy, and the distribution module distributes the file according to the data file distribution strategy.

[0056] The present invention is a method and tool for synchronizing heterogeneous time series databases, and its system diagram is shown in FIG. Figure 1As shown. This synchronization tool is independent of the database and contains eight parts: configuration module, monitoring module, cache module, prediction module, interaction module, filtering module, conversion module, and distribution module. In addition, a source database agent is installed on each distributed node of the source database, which is the medium for the synchronization tool to interact with the source database. Similarly, a target database agent is installed on each distributed node of the target database, which is the medium for the synchronization tool to interact with the target database.

[0057] Apache IoTDB is used as the source database and target database in the database synchronization process. Both the source database and the target database are deployed in a distributed cluster manner, and the data synchronization between the two is used as an implementation example of data synchronization between heterogeneous time series databases.

[0058] In this embodiment, the user configures the information required by the synchronization tool through the configuration module. The configuration module sends the corresponding configuration information to the source database node through the interaction module, sends the corresponding configuration information to the target database node through the distribution module, and stores each configuration information in the cache module. The information related to the source database will be handed over to the interaction module, sent to the source database agent through the interaction module, and used for the connection between the source database agent and the source database. Similarly, the information related to the target database will be handed over to the distribution module, sent to the target database agent through the distribution module, and used for the connection between the target database agent and the target database. In addition, information related to data synchronization also needs to be configured to support the implementation of the data synchronization process. All of the above information will also be sent to the cache module for storage.

[0059] This embodiment provides a specific implementation of a configuration module. The configuration module includes three parts: a source database configuration unit, a target database configuration unit, and a synchronization configuration unit.

[0060] The source database configuration unit is responsible for configuring the source database name, source database type, source database user name, source database password, source database data file path, source database load monitoring interval, file extension, port and data compression format and other information related to the source database. Once the above parameters are configured, the source database configuration unit encapsulates this information and sends it to the cache module to achieve temporary storage of data, so that the interaction module can retrieve and obtain the stored configuration information from the configuration module for subsequent data synchronization and processing operations;

[0061] The target database configuration unit is responsible for configuring the target database name, target database type, target database user name, target database password, target database data file path, target database load monitoring interval, file extension, port and data compression format and other information related to the target database. When the parameters are configured successfully, the target database configuration unit encapsulates this information and sends it to the cache module to achieve temporary storage of data. The distribution module can also obtain this configuration information from the configuration module;

[0062] The synchronization configuration unit is responsible for configuring the data file format of the source database and the target database, the path for synchronizing the source database and the target database, the synchronization time interval, the filtering conditions, the number of threads and other information related to data synchronization. The data file format of the source database and the target database is used for the conversion module, the path for synchronizing the source database and the target database is used for local synchronization, the synchronization time interval indicates how long the synchronization tool takes to initiate a synchronization, if it is empty, it means only one synchronization, the filtering conditions are used to specify which strategy to use to implement data filtering, such as time filtering, column value filtering, type filtering, etc., and the number of threads controls how many threads the interaction module and the distribution module use to complete the synchronization task.

[0063] The path correspondence between the source database and the target database synchronization can be expressed by the following formula:

[0064] S(s 1 ,s 2 ,…,s n )→T(t 1 , t 2 ,…,t n )

[0065] Among them, S represents the source database, s 1 ,s 2 ,…,s n Each represents a path of the source database, T represents the target database, and t 1 , t 2 ,…,t n Both represent a path of the target database, and the path of the source database is s 1 The path to the target database that will be synchronized 1 , the path of the source database 2 The path to the target database that will be synchronized 2 And so on.

[0066] If the data structures of the source database and the target database are similar to the tree structure of Apache IoTDB, they can be directly matched through the tree structure. Figure 2As shown in the figure, the last layer is the sensor and the second to last layer is the device. Then "root.1n.Wf01.wt01.temperature→root.ln.mb01.mb01.temperature" means synchronizing the data in the time series root.ln.wf01.wt01.temperature of the source database to root.ln.mb01.mb01.temperature of the target database.

[0067] If the data structures of the source database and the target database are similar to the table structure of Timescale DB, the table structure can be directly matched. Figure 3 As shown in the figure, the table contains columns including timestamps, which are used to store time series data. Then "root.name1.temperature→root.name2.temperature" means synchronizing the data in the temperature column of the name1 table of the source database root to the temperature column of the name2 table of the target database root.

[0068] If the data structure of the source database and the target database is a tree structure and a table structure respectively, the time series of the tree structure corresponds to the columns of the table structure, and the devices of the tree structure correspond to the table names. "root.ln.wf01.wt01.temperature→root.name2.temperature" means synchronizing the data in the time series root.ln.wf01.wt01.temperature of the source database to the temperature column in the name2 table of the root of the target database. "root.ln.wf01.wt01→root.name2" means synchronizing the data in the device root.ln.wf01.wt01 of the source database to the name2 table of the root of the target database.

[0069] The strategy for verifying the configuration in this embodiment is as follows: when loading the configuration information, first check whether the corresponding data structures exist in the source database and the target database. For each data structure mapping in the configuration, such as "root.ln.wf01.wt01.temperature" to "root.ln.mb01.mb01.temperature", if "root.ln.mb01.mb01.temperature" does not exist in the target database, a new data structure is created in the target database based on the data structure of "root.ln.wf01.wt01.temperature" in the source database; if "root.ln.mb01.mb01.temperature" already exists in the target database, continue to execute the synchronization task; if "root.ln.wf01.wt01.temperature" does not exist in the source database, stop the synchronization task and report the non-existent time series.

[0070] In this embodiment, the monitoring module is responsible for monitoring the load information of each distributed node of the source database and the target database and preprocessing it, and finally handing it over to the cache module for storage for use by the prediction module. The monitoring module obtains the load information of each source database node through the interaction module and obtains the load information of each target database node through the distribution module. The load information of each distributed node of the source database will be monitored by the source database agent on each distributed node of the source database, and sent to the interaction module, and then sent to the monitoring module by the interaction module. Similarly, the load information of each distributed node of the target database will be monitored by the target database agent on each distributed node of the target database, and sent to the distribution module, and then sent to the monitoring module by the distribution module.

[0071] This embodiment provides a specific implementation of a monitoring module. The detection module includes two parts: a data sorting unit and a data normalization processing unit, wherein:

[0072] The data sorting unit is responsible for receiving the load information of each distributed node of the source database and the target database, including CPU utilization, memory utilization, disk remaining space, disk I / O rate, network utilization, etc., and sorting the cache according to the distributed node classification, and then handing these load data to the data normalization processing unit in batches for normalization processing;

[0073] The data normalization processing unit is responsible for normalizing the load data using the following formula:

[0074]

[0075] Among them, x irepresents the detection value of a certain load characteristic value of node i; min and max represent the minimum and maximum detection values ​​of a certain characteristic value in all nodes respectively; x′ i Represents a feature value that is consistent with [0, 1]. These normalized load data and the monitoring time of the data will be sent to the cache module, which will store these load data by different node classifications and time order for subsequent prediction modules to train models and make predictions.

[0076] In this embodiment, the prediction module is responsible for predicting the load conditions of each distributed node of the source database and the target database at the next moment, the segmentation strategy of the data file of the conversion module and the distribution strategy of the distribution module. The prediction module predicts the load conditions at the next moment based on the load information monitored up to the current moment. Specifically: first, the prediction module obtains the historical load data of each distributed node of the source database and the target database from the cache module, trains the LSTM long short-term memory neural network model, and gives the best model to the cache module for storage; then, the prediction module predicts the load conditions of each distributed node of the source database and the target database at the next moment through the real-time load data of each distributed node of the source database and the target database and the trained model.

[0077] This embodiment provides a specific implementation of a prediction module, which includes two parts: a model training unit and a load prediction unit.

[0078] The model training unit is responsible for regularly retraining the model using the historical load data of each distributed node of the source database and the target database. Specifically: First, the model training unit divides the historical load data of each distributed node into training set, validation set and test set, and the ratio is set to 6:2:2. The training set is used to train the LSTM long short-term memory neural network model, the validation set is used to adjust hyperparameters and model selection, and the test set is used to evaluate the final performance of the model; then, training is performed to obtain the latest model; finally, the latest model is saved to the cache module.

[0079] The load prediction unit is responsible for predicting the load conditions of each distributed node of the source database and the target database at the next moment through the real-time load data of each distributed node of the source database and the target database and the trained model. Specifically: first, obtain the latest LSTM long short-term memory neural network model and the latest load data of each distributed node of the source database and the target database from the cache module; then, predict the load data of each distributed node of the source database and the target database at the next moment; finally, bring the predicted load data into the formula to obtain the numerical value for evaluating the load size of each node, and sort them according to the load size of each distributed node of the source database and the target database, providing data support for the conversion module and the distribution module.

[0080] The load formula for evaluating the distributed node load size is as follows:

[0081] LB i =f 1 *cpu i +f 2 *memory i +f 3 *disk i +f 4 *speed i +f 5 *process i +f 6 *thread i +f 7 *network i +f 8 *connection i +f 9 *CHR i +f 10 *ART i +f 11 *ER i +f 12 *TQL i +f 13 *TPS i +f 14 *RPS i +f 15 *CT i

[0082] Among them, the weight f 1 、f 2 、f 3 、f 4 、f 5 、f 6 、f 7 、f 8 、f 9 、f 10 、f 11 、f 12 、f 13 、f 14 、f 15 The sum is 1; cpu i Represents the predicted normalized CPU utilization of the i-th node; memory i Represents the predicted normalized memory utilization of the i-th node; disk i Represents the latest normalized free disk space of the i-th node; speed irepresents the predicted normalized disk I / O rate of the ith node; process i Indicates the normalized number of processes of the i-th node; thread i Indicates the normalized number of threads of the i-th node; network i Indicates the predicted normalized network utilization of the ith node; connection i represents the normalized number of network connections of the ith node; CHR i represents the normalized cache hit rate of the i-th node; ART i represents the normalized average response time of the ith node; ER i represents the normalized error rate of the i-th node; TQL i Represents the normalized task queue length of the i-th node; TPS i Represents the normalized number of transactions per second of the i-th node; RPS i represents the normalized number of requests per second of the i-th node; CT i Represents the normalized connection timeout number of the i-th node.

[0083] In addition, those skilled in the art may appropriately increase or decrease indicators in the node load calculation formula given in this embodiment according to actual conditions, or may select other indicators as indicators for calculating node load.

[0084] In this implementation, the cache module is responsible for storing the files and information required by the synchronization tool to ensure that the database synchronization process proceeds smoothly. This embodiment provides a specific implementation of the cache module, which includes five parts: a configuration storage unit, a load data storage unit, a model storage unit, a data file storage unit, and a database structure storage unit, among which:

[0085] Configuration storage unit, responsible for storing information related to the source database, information related to the target database and information related to data synchronization;

[0086] A load data storage unit is responsible for storing the load data of each distributed node of the source database and the target database detected by the source database agent and the target database agent according to node classification and time sorting;

[0087] Model storage unit, responsible for storing the LSTM long short-term memory neural network model trained by the prediction module;

[0088] The data file storage unit is responsible for storing all synchronized data files, including real-time synchronized data files and historical synchronized data files;

[0089] The database structure storage unit is responsible for storing the source database structure inferred from the data file and is used to generate DDL operations to synchronize the source database structure.

[0090] The interaction module is responsible for interacting with the source database agent, including sending information to the source database agent and receiving files and information sent by the source database agent. As an optional method, the database source node sends the data file to the source database agent, and the source database agent sends the data file to the interaction module.

[0091] This embodiment provides a specific implementation of an interaction module, which includes four parts: a receiving unit, a sending unit, a retransmission unit, and a control unit.

[0092] The receiving unit is responsible for receiving data files or information sent by the source database agent. If a data file is received, it is handed over to the cache module for storage. If information is received, it is handed over to the control unit of the interaction module for processing;

[0093] The sending unit is responsible for sending information to the source database agent, telling the source database agent which files or information should be sent to the synchronization tool;

[0094] The resending unit is responsible for resending the information after the sending unit fails to send it, so as to ensure the successful sending of the information;

[0095] The control unit is responsible for controlling the receiving unit, sending unit, and resending unit, and obtaining the data files that need to be synchronized in the source database according to the corresponding strategy and acquisition strategy;

[0096] The process of the synchronization tool obtaining the source database file is as follows Figure 4 As shown, the synchronization tool selects a distributed node with the smallest load among the distributed nodes of the source database to send a message. If it is the first synchronization, it returns the number and size of all current data files and records the time of this synchronization. If it is not the first synchronization, it returns the number and size of data files that have changed since the last synchronization. Then, the synchronization tool will receive this information and obtain the data files according to the corresponding strategy and acquisition strategy.

[0097] The corresponding strategy between data files and source database distributed nodes is as follows: if the number of data files is less than or equal to the number of source database distributed nodes, the data files are sorted according to file size and correspond one-to-one with the load size sorting of the source database distributed nodes, and the data file with the largest file size is obtained from the distributed node with the smallest load, and so on; if the number of data files is greater than the number of source database distributed nodes, the remaining data files are re-correlated one-to-one with the source database distributed nodes after the first correspondence.

[0098] The source database agent's acquisition strategy for obtaining the source database data file is as follows: the source database agent will search for the source database data file according to the data file path in the configuration information. If the data file can be obtained, the data file will be directly sent to the interactive module of the synchronization tool. If the data file cannot be obtained, the source database agent needs to query the data through the database API interface in a polling manner, and generate the data file according to the data file format of the configuration information, and then send the generated data file to the interactive module of the synchronization tool.

[0099] In this embodiment, the first task of the filtering module is to simplify the data file, filter out the parts of the data file that do not need to be synchronized, and only leave the parts that need to be synchronized; the second task is to send the synchronized data file to the cache module for storage, and compare the data file of the source database with the historically synchronized data file, infer the data modification operation and data deletion operation and the possible change of the database structure, and generate the corresponding data modification operation statement, data deletion operation statement and DDL operation statement available for the target database. The second task of the filtering module will be performed after the first task realizes the local synchronization process and before the first task realizes the incremental synchronization process.

[0100] This embodiment provides a specific implementation scheme of a filtering module, which includes three parts: a path filtering unit, an operation filtering unit, and a time filtering unit.

[0101] 1) Path filtering unit, responsible for filtering out paths in the data file that do not need to be synchronized. The path that the synchronization tool needs to synchronize is "root.ln.wf01.wt01.temperature", that is, only the data under the root.ln.wf01.wt01.temperature time series needs to be synchronized, but the acquired data file may contain data under other time series. Therefore, the filtering module will delete the data under other time series and only leave the data under the root.ln.wf01.wt01.temperature time series to achieve local synchronization.

[0102] As an optional implementation, in order to efficiently synchronize time series data, multiple filtering strategies can be used: first, time filtering allows the configuration module to set a specific time range and only synchronize data within that period; second, column value filtering filters based on data columns, for example, only synchronize temperature values ​​or speed measurement values; type filtering filters based on data types, such as only synchronizing specific types of sensor data, which can be done by specifying column types or using regular expressions to match data type identifiers. After setting the filtering strategy and number of threads, the data file is divided into multiple small blocks and then evenly distributed to each thread to complete the filtering, thereby reducing the overall processing time;

[0103] 2) Operation filtering unit, responsible for comparing historical synchronized data files, inferring data modification operations and data deletion operations as well as possible changes in database structure, and generating data modification operation statements, data deletion operation statements and DDL operation statements available for the corresponding target database.

[0104] Inferring data modification and data deletion operation process: First, parse the data files of this synchronization and the data files of historical synchronization. For each data in the data files of historical synchronization, search for data with the same timestamp and database structure in the data files of this synchronization. If the data is the same, no processing is done. If the data is different, it is inferred that there is a data modification operation, and a data modification operation statement available for the target database will be generated based on the configuration information of the target database. If not, it is inferred that there is a data deletion operation, and a data deletion operation statement available for the target database will be generated based on the configuration information of the target database. Since the first synchronization is a full synchronization and there is no historical synchronization data file, the inferring data modification and data deletion operation process will not be performed in the first synchronization.

[0105] Inferring DDL operation process: The data file usually contains relevant information about the database structure. For example, the Apache IoTDB data file TsFile will store whether there is data under "root.ln.wf01.wt01.temperature", which means that "root.ln.wf01.wt01.temperature" is part of the database structure of the source database. Therefore, the filtering module will infer the database structure of the part that needs to be synchronized from the data file and send it to the cache module for storage. If it is the first synchronization, the DDL operation available to the target database of the newly created database structure is generated based on the inferred database structure. If it is not the first synchronization, the inferred database structure is compared with the last inferred database structure. If there is a change, the corresponding DDL operation statement available to the target database is generated.

[0106] 3) Time filtering unit, responsible for filtering out the data that has been synchronized in the data file. If it is the first synchronization, no additional operation is performed, and only the time of this synchronization is recorded; if it is not the first synchronization, the data with a timestamp before the time of the last synchronization is additionally deleted, that is, the data that has been synchronized before is deleted to achieve incremental synchronization.

[0107] In this embodiment, the first task of the conversion module is to divide the data file into several small data files, so that the distribution module can use the distribution strategy to send these small data files to the agents on multiple target database distributed nodes; the second task is to convert the data files of the source database into the data files of the target database, so that the data files synchronized to the target database agent can be correctly read and processed by the target database.

[0108] The present invention proposes a specific implementation of a conversion module, which includes two parts: a data file segmentation unit and a data file conversion unit, wherein:

[0109] The data file splitting unit is responsible for splitting the data files of the source database into several small data files;

[0110] The data file segmentation strategy is as follows Figure 5 As shown, there is a distributed node set N of the target database = {n 1 , n 2 ,…,n k}, where LB(n j ) represents the predicted node n j The load size, let T be the total size of the data files that need to be synchronized, F = {f 1 , f 2 , …, f m} is the set of data files that need to be synchronized this time. The segmentation strategy includes the following steps:

[0111] Obtain the load size LB (n) of each distributed node of the target database calculated by the prediction module j ), determine a transmission size T according to the load size of all distributed nodes of the target database, and extract a data file set F with a total size of about T from the data file set F that needs to be synchronized this time T ;

[0112] According to the load size of the node, N is divided into high-load nodes N H , medium load node N M and low-load node N L , whose proportions are p H 、p M and p L , and p H +p M +p L =1, calculate the ratio of high-load nodes, medium-load nodes and low-load nodes:

[0113]

[0114] The total size of the extracted data file set F is about T TDivide into a set F of small data files of size S S , a set of medium data files F of size M M and a large data file set F of size H H , adjust the size of each set according to the ratio to ensure that the total size is close to T:

[0115] |F S |≈p H *|F T |,|F M |≈p M *|F T |,|F H |≈p L *|F T |

[0116] The filtering module supports two segmentation strategies, which can operate simultaneously to meet different processing requirements. The time-based segmentation strategy is to divide the time series data according to the time range to which the timestamp belongs, reducing the size of each transmission file, thereby improving the file transfer efficiency; while the database structure-based segmentation strategy is to divide the time series data according to the time series in which the time series data is located. In the Apache IoTDB time series database, in order to facilitate query and storage, the time series data belonging to the same path is stored in an appended manner in a data partition. Therefore, segmenting the data file based on the path will reduce the distributed distribution operations on the receiving end of the database, thereby improving the synchronization efficiency of the database.

[0117] In this embodiment, the data file conversion unit is responsible for converting the data file TsFile of the source database into the format of the data file of the target database, so that the data file synchronized to the target database agent can be correctly read and processed by the target database.

[0118] In this embodiment, the distribution module is responsible for sending data files, data modification operations, data deletion operations and DDL operations to agents of multiple distributed nodes of the target database according to a certain distribution strategy. The agent will apply the data files, data modification operations, data deletion operations and DDL operations to the target database to complete data synchronization. As an optional method, the distribution module sends the data files to the target database agent corresponding to the target database, and the target database agent sends the data files to the corresponding target database node.

[0119] The priority of the target database's proxy to execute DDL operations is higher than the priority of running data files, and the priority of executing data files is higher than the priority of executing data modification operation statements and data deletion operation statements. After receiving the DDL operation, the proxy of the target database quickly applies it to the target database through the API interface to make the structure of the target database consistent with the source database, and ensure that it is completed before processing the data file. Then process the data file. If the target database supports importing data files, the synchronized data files are directly imported into the target database. If the target database, such as TimescaleDB, does not support importing data files, the synchronized data files are parsed, the corresponding data addition operation statements are generated, and these statements are input into the target database through the API interface of the target database. After processing the data file, if there are synchronized data modification operation statements and data deletion operation statements, the data modification operation statements and data deletion operation statements are then input into the target database through the API interface of the target database.

[0120] This embodiment provides a specific implementation of a distribution module, which includes four parts: a receiving unit, a distribution unit, a retransmission unit, and a control unit.

[0121] The receiving unit is responsible for receiving the load data and other related information of the target database node sent by the target database agent;

[0122] The distribution unit is responsible for sending data files and operation statements to the agents of the target database nodes according to the load size and distribution weight of each distributed node of the target database. The distribution unit first sends the DDL operation to the agent of the target database node with the smallest load, and then sends the split data files to the agents of the target database nodes according to the distribution strategy. Finally, the data modification operation statements and data deletion operation statements are sent to the agents of the target database nodes with the smallest load.

[0123] The distribution strategy of data files is as follows Figure 6 As shown, let F = {f 1 , f 2 , …, f m} is the set of data files to be distributed, where |f i | indicates file f i The size of |f 1 |≥|f 2 |≥…≥|f m |, let N = {n 1 , n 2 ,…,n k} is the distributed node set of the target database, where LB(n j ) represents the predicted node n j The load size, and LB(n1 )≤LB(n 2 )≤…≤LB(n k ), the distribution strategies of data files include:

[0124] First, sort F from large to small according to file size, and sort N from small to large according to load. Let X be the number of currently unmatched data files, and let Y be the number of distributed nodes of the target database.

[0125] Repeat the following steps until F is empty:

[0126] If X<Y, select X nodes with the smallest load: N′={n 1 , n 2 ,…,n X}, F and N′ are matched one by one, the largest data file corresponds to the node with the smallest load, and so on: (f i , n i ), i = 1, 2, ..., X, and finally clear F;

[0127] If X=Y, directly associate F with N one by one. The largest data file corresponds to the node with the smallest load, and so on. i , n i ), i = 1, 2, ..., X, and finally clear F;

[0128] If X>Y, select the Y data files with the largest file size: F′={f 1 , f 2 , …, f Y}, F′ and N are matched one by one, the largest data file corresponds to the node with the smallest load, and so on: (f i , n i ), i = 1, 2, …, Y, and finally remove F′ from F and update X = XY.

[0129] In this embodiment, the resending unit is responsible for resending the data file and the operation statement after the distribution unit fails to send the data file, so as to ensure the successful sending of the data file and the operation statement.

[0130] The synchronization process of data files in the synchronization tool includes:

[0131] First, the filtering module processes the data files to be synchronized and filters out the parts that do not need to be synchronized.

[0132] The data file segmentation and distribution process is carried out in multiple steps. First, the optimal size T of the data file to be synchronized is determined based on the load size of each distributed node of the target database. Then, one or more data files with a total size closest to T are obtained from all the data files that need to be synchronized.

[0133] The conversion module then divides these data files into three categories: large data files, medium data files, and small data files according to the segmentation strategy, and converts them into the data file format of the target database. The distribution module matches the three categories of data files, large data files, medium data files, and small data files, with low-load nodes, medium-load nodes, and high-load nodes according to the distribution strategy, and completes the sending of this data file. After the segmentation and distribution of this data file is completed, it is determined whether there are any unsynchronized data files. If there are any unsynchronized data files, the next data file segmentation and distribution will be performed;

[0134] The work of the filtering module and the splitting module is carried out continuously. After the source data file is parsed, it is first filtered, then split, and finally the format of the data file is converted. This can reduce the number of times the data file is parsed and improve synchronization efficiency.

[0135] In this embodiment, the source database agent is responsible for monitoring the load data of the source database node and obtaining the data files that need to be synchronized from the source database node.

[0136] This embodiment provides an implementation method of a source database proxy, which includes a receiving unit, a sending unit, a resending unit, a monitoring unit, an acquiring unit, and a connecting unit, wherein:

[0137] The receiving unit is responsible for receiving and processing the information sent by the interaction module;

[0138] The sending unit is responsible for sending data files, data file information and load information to the interaction module;

[0139] The resending unit is responsible for resending after the sending unit fails to send, ensuring the successful sending of data files and information;

[0140] The monitoring unit is responsible for monitoring the load data of the source database node, including CPU utilization, memory utilization, disk remaining space, disk I / O rate, and network utilization;

[0141] The acquisition unit is responsible for acquiring the data files that need to be synchronized from the path where the data files are stored in the source database;

[0142] The connection unit is responsible for connecting to the source database and querying data through the API of the source database to generate data files.

[0143] The source database agent obtains data files as follows: Figure 7As shown, the source database agent will search for the data file of the source database according to the data file path in the configuration information. If the data file can be obtained, the data file will be directly sent to the interactive module of the synchronization tool. If the data file cannot be obtained, the source database agent needs to query the data through the database API interface in a polling manner, and generate the data file according to the data file format of the configuration information, and then send the generated data file to the interactive module of the synchronization tool.

[0144] In this embodiment, the target database agent is responsible for monitoring the load data of the target database node, as well as the data files or operation statements input into the target database node.

[0145] This embodiment provides a specific implementation of a target database proxy. Similar to the source database proxy, the target database proxy internally includes a receiving unit, a sending unit, a resending unit, a monitoring unit, a connecting unit, and a parsing unit, wherein:

[0146] The receiving unit is responsible for receiving and processing the information sent by the distribution module;

[0147] The sending unit is responsible for sending load information to the distribution module;

[0148] The resending unit is responsible for resending the information after the sending unit fails to send it, so as to ensure the successful sending of the information;

[0149] The monitoring unit is responsible for monitoring the load data of the target database node, including CPU utilization, memory utilization, disk remaining space, disk I / O rate, and network utilization;

[0150] The connection unit is responsible for connecting to the target database and inputting operation statements to the target database through the API of the target database;

[0151] The parsing unit is responsible for placing the data file into the directory where the target database stores data files. If there are files with the same name, the files with the same name are merged.

[0152] The target database agent places the data file process as follows Figure 8 As shown, including:

[0153] The target database agent receives the data file sent by the distribution module and then connects to the target database;

[0154] If the target database supports importing data files, the synchronized data files will be directly imported into the target database;

[0155] If the target database, such as TimescaleDB, does not support importing data files, the synchronized data files are parsed, the corresponding data addition operation statements are generated, and these statements are input into the target database.

[0156] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous time series database synchronization system, used to synchronize data between a source database and a target database, including a configuration module, a monitoring module, a cache module, a prediction module, an interaction module, a filtering module, a conversion module, and a distribution module, wherein: The interaction module is used to obtain load information and data files from the source database, and send the load information to the monitoring module and send the data files to the filtering module; The monitoring module is used to pre-process the acquired load information and store the pre-processed load information in the cache module; The prediction module is used to predict the load of the node at the next moment based on the historical load of each node in the database; The filtering module is used to filter the acquired data files and delete the file data that does not need to be synchronized; Configuration module, used to set the configuration parameters of the source database and the target database; A conversion module is used to divide the data to be synchronized into multiple data files and convert the data files into the file format of the target database; The distribution module is used to send file data or operation statements to the corresponding nodes in the target database according to the load conditions and configuration information of the nodes; It is characterized in that the interaction module obtains data files according to the corresponding strategy and the acquisition strategy during the process of obtaining files, the conversion module divides the file data according to the data file segmentation strategy, and the distribution module distributes files according to the data file distribution strategy.

2. According to claim 1, a heterogeneous time series database synchronization system is characterized in that: The corresponding strategy between data files and source database distributed nodes is as follows: if the number of data files is less than or equal to the number of source database distributed nodes, the data files are sorted according to file size, and correspond one-to-one with the load size sorting of the source database distributed nodes, and the data file with the largest file size is obtained from the distributed node with the smallest load; if the number of data files is greater than the number of source database distributed nodes, the remaining data files are re-correlated one-to-one with the source database distributed nodes after the first correspondence.

3. The heterogeneous time series database synchronization system according to claim 1, characterized in that: The source database agent's acquisition strategy for obtaining the source database data file is as follows: the source database agent will search for the source database data file according to the data file path in the configuration information. If the data file can be obtained, the data file will be directly sent to the interactive module of the synchronization tool. If the data file cannot be obtained, the source database agent needs to query the data through the database API interface in a polling manner, and generate the data file according to the data file format of the configuration information, and then send the generated data file to the interactive module of the synchronization tool.

4. The heterogeneous time series database synchronization system according to claim 1, characterized in that: The conversion module splits the data files of the source database into several small data files. The splitting process includes: Obtain the load size LB (n) of each distributed node of the target database calculated by the prediction module j ), determine a transmission size T according to the load size of all distributed nodes of the target database, and extract a data file set F with a total size of about T from the data file set F that needs to be synchronized this time T ; According to the load size of the node, N is divided into high-load nodes N H , medium load node N M and low-load node N L , whose proportions are p H 、p M and p L , and p H +p M +p L =1; The total size of the extracted data file set F is about T T Divide into a set F of small data files of size S S , a set of medium data files F of size M M and a large data file set F of size H H , adjust the size of each set according to the ratio to ensure that the total size is close to T.

5. The heterogeneous time series database synchronization system according to claim 4, characterized in that: In the process of segmenting the data file collection, a time-based segmentation strategy or a database structure-based segmentation strategy is used for segmentation, where: The time-based segmentation strategy is to divide the time series data according to the time range to which the timestamp belongs, reducing the size of each transferred file; The database structure-based partitioning strategy is to divide the time series data according to the time series in which it is located, that is, the time series data belonging to the same path is stored in one data partition in an appended manner.

6. A heterogeneous time series database synchronization system according to claim 4, characterized in that: The filtering module includes a path filtering unit, an operation filtering unit and a time filtering unit, wherein: The path filtering unit is responsible for filtering out the paths in the data file that do not need to be synchronized; The operation filtering unit is responsible for comparing historically synchronized data files, inferring data modification operations and data deletion operations as well as possible database structures that may change, and generating corresponding data modification operation statements, data deletion operation statements and DDL operation statements available for the target database; The time filtering unit is responsible for filtering out the data that has been synchronized in the data file.

7. The heterogeneous time series database synchronization system according to claim 1, characterized in that: The distribution module is responsible for executing the data file distribution strategy, which includes the following operations: Sort the data file set F to be distributed from large to small according to file size, and sort the distributed node set N of the target database from small to large according to load. Let X be the number of currently unmatched data files, and let Y be the number of distributed nodes of the target database. Repeat the following steps until F is empty: If X<Y, select X nodes with the smallest load: N′={n1,n2,…,n X }, put F and N' in one-to-one correspondence, the largest data file corresponds to the node with the smallest load; If X=Y, directly match F and N one by one, and the largest data file corresponds to the node with the smallest load; If X>Y, select the Y data files with the largest file size: F′={f1,f2,…,f Y }, match F′ with N one by one, the largest data file corresponds to the node with the smallest load, remove F′ from F, and update X=XY.

Images