A method for synchronizing cloud-edge data in a time series database

Abstract

The present invention discloses a method for synchronizing cloud-edge data in a time-series database, and relates to the field of data communication technology. Both terminal-edge data synchronization and edge-to-edge data synchronization include quasi-real-time data synchronization and real-time data synchronization. Edge-cloud data synchronization includes Block data synchronization, and data synchronization is performed between cloud nodes using the Raft protocol.

Description

Technical Field

[0001] The present invention discloses a method, which relates to the field of data communication technology, and specifically a method for synchronizing cloud-edge data in a time series database. Background Art

[0002] Based on the deployment requirements of the time series database, the time series database Agent service is deployed on the terminal, edge, and cloud at the same time. The terminal collects data in real time and synchronizes the data to the edge, and the edge synchronizes the data to the cloud. At the same time, the edge and cloud also provide external functions such as data aggregation query. Each process involves data synchronization, but the data synchronization process between the terminal, edge, and cloud is still imperfect. Summary of the Invention

[0003] In response to the problems of the existing technology, the present invention provides a method for cloud-edge-terminal data synchronization of a time series database, which synchronizes time series data between the cloud, edge, and terminal. Different synchronization schemes are adopted according to the different external services provided by different terminals, ensuring the high availability of the entire system and the data integrity of each terminal.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a method for synchronizing cloud-edge data of a time series database, including data synchronization between terminals and edges, data synchronization between edges, data synchronization between edges and the cloud, and data synchronization between cloud nodes.

[0006] Data synchronization between terminals and edges, and data synchronization between edges, both include quasi-real-time data synchronization and real-time data synchronization. When terminals and edges, or edges and edges perform quasi-real-time synchronization, the party acting as the source writes data into the synchronization data queue of the Block in the Table, and transmits the data in the synchronization data queue to the party acting as the target through the synchronization service process. When terminals and edges, or edges and edges perform real-time data synchronization, the party acting as the source determines whether there is a synchronization offset based on the synchronization situation. If there is a synchronization offset, the data is synchronized to make up for it.

[0007] Data synchronization between the edge and the cloud includes Block data synchronization. The source side obtains the latest Block status of the target side, compares it with its own current Block status, and determines whether the Block data is synchronized. If not, it continues to send the corresponding Block data for Block data synchronization until the Block data on the target side is consistent with the source side.

[0008] Data synchronization between cloud nodes is performed through the Raft protocol.

[0009] Furthermore, when the terminal and edge, edge and edge, and edge and cloud synchronize data in the method described above, communication is established through Socket. The source party uses the synchronization service process through the source agent to send data to the target agent as the target party. The target agent uses the application listener to parse the data and write the parsed data to the target end.

[0010] Furthermore, when performing quasi-real-time synchronization in the method described above, the Block's synchronization data queue includes a sequential data queue and an out-of-order data queue. Data is written into the corresponding synchronization data queue and synchronously transmitted to the target party through the synchronization service process of the source end.

[0011] Furthermore, when the terminal and the edge or the edge and the edge perform real-time data synchronization in the method described above, there is a synchronization offset. The synchronization service process on the source end scans the synchronization data queue of the Block in the Table to obtain the offset data amount, and sends a synchronization request to the target end to send the data to be synchronized corresponding to the offset data amount in the synchronization data queue to the target end. The target end receives the synchronization request, parses the data to be synchronized in the synchronization request, writes the data to be synchronized, and returns a synchronization success response.

[0012] Furthermore, in the method described, the edge and the cloud synchronize block data, and the source side periodically sends a request to the target side to obtain the latest block status of the target side. The source side compares the latest block status of the target side with its own current block status to determine whether the block data is synchronized. If the source side has an unsynchronized block with a lagging timestamp, it is considered to be out of sync and will continue to send synchronization requests to the target side to synchronize block data. The synchronization request includes the timestamp range of the unsynchronized block to identify the block data range. The target side receives the synchronization request, uses the application listener to parse the data of the block to be synchronized in the synchronization request, writes the data of the block to be synchronized, and returns a synchronization success response.

[0013] Furthermore, in the method described above, the edge completes data synchronization with a node in the cloud, and the node automatically synchronizes the data to other nodes using the Raft protocol.

[0014] The present invention also provides a device for synchronizing cloud-edge data of a time series database, comprising at least one memory and at least one processor;

[0015] The at least one memory is configured to store a machine-readable program;

[0016] The at least one processor is used to call the machine-readable program to execute the method for synchronizing cloud-edge data of a time series database.

[0017] The present invention also provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method for synchronizing cloud-edge data of a time series database.

[0018] The benefits of the present invention are:

[0019] The present invention provides a method for synchronizing cloud-edge data of a time-series database, which is applied between terminals, edge terminals and cloud terminals to synchronize data to each terminal. The terminal and edge terminals, as well as the edge terminals and the edge terminals, can perform quasi-real-time synchronization. The quasi-real-time synchronization difference interval is very small, which is close to real-time synchronization, so that terminals without persistent devices can release memory as soon as possible. When the terminal and the edge terminals, as well as the edge terminals and the edge terminals, perform real-time synchronization, the synchronization offset is also padded to maintain the integrity of the data. The edge terminals and the cloud terminal perform block data synchronization, which facilitates the synchronous transmission of large amounts of data. The method of the present invention achieves overall high availability among the three terminals and maintains the data integrity of each terminal. The application can directly obtain data from the local terminal, which is more convenient. At the same time, it can conveniently use the powerful computing power of the cloud terminal to perform other pre-calculation processing on the data, which can ensure the persistence of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a schematic diagram of the general synchronization process between the source end and the target end in the method of the present invention.

[0022] Figure 2 It is a schematic diagram of the quasi-real-time synchronization process of the method of the present invention.

[0023] Figure 3 It is a schematic diagram of the Block data synchronization process of the method of the present invention.

[0024] Figure 4 It is a schematic diagram of the data synchronization process between the edge and the cloud and between cloud nodes in the method of the present invention. DETAILED DESCRIPTION

[0025] Nouns involved in the present invention:

[0026] Original time series data, that is, the time series data collected on the terminal and recorded according to timestamps, is the row-stored time series data of each table that was most recently written to the memory and is the basic unit for real-time synchronization.

[0027] Block data: Each table stores block data in columns after column encoding and compression. Result data may be persisted on disk or in an immutable buffer. Immutable buffers are strings of data held in memory. Synchronization and modification of this result data must be handled block by block. A block is the smallest unit of data stored on disk in the database.

[0028] Table is a data table, and the corresponding hard disk in Table is divided into multiple data partitions (ZObject).

[0029] A socket is an endpoint for bidirectional communication between application processes on different hosts on a network. A socket is one end point for process communication on a network, providing a mechanism for application-layer processes to exchange data using network protocols. In terms of its position, a socket connects the application process above and the network protocol stack below. It is the interface through which applications communicate using network protocols and the interface between applications and the network protocol stack.

[0030] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0031] The present invention provides a method for synchronizing cloud-edge data of a time series database, including data synchronization between terminals and edges, data synchronization between edges, data synchronization between edges and the cloud, and data synchronization between cloud nodes.

[0032] Data synchronization between terminals and edges, and data synchronization between edges, both include quasi-real-time data synchronization and real-time data synchronization. When terminals and edges, or edges and edges perform quasi-real-time synchronization, the party acting as the source writes data into the synchronization data queue of the Block in the Table, and transmits the data in the synchronization data queue to the party acting as the target through the synchronization service process. When terminals and edges, or edges and edges perform real-time data synchronization, the party acting as the source determines whether there is a synchronization offset based on the synchronization situation. If there is a synchronization offset, the data is synchronized to make up for it.

[0033] Data synchronization between the edge and the cloud includes Block data synchronization. The source side obtains the latest Block status of the target side, compares it with its own current Block status, and determines whether the Block data is synchronized. If not, it continues to send the corresponding Block data for Block data synchronization until the Block data on the target side is consistent with the source side.

[0034] Data synchronization between cloud nodes is performed through the Raft protocol.

[0035] The method of the present invention involves terminals, edge ends and cloud ends. The Agent service of the time series database can be used to establish a one-way synchronization process from the source end to the target end, and communication can be established through Socket, and different synchronization schemes can be implemented according to different scenarios.

[0036] In some embodiments of the present invention, the Inspur Yunxi database ZNBase and the time series database ZDP are taken as examples. The terminal is referred to as T-end, the edge is referred to as E-end, and the cloud is referred to as C-end. The ZDP Agent service is deployed on the T-end, E-end, and C-end. Figure 1 The two ends that need data synchronization establish communication through Socket. The source end can start the synchronization service process (SyncService) to send the data to be synchronized to the target agent. The target agent uses the application listener (ApplyListener) registered at startup to write the synchronization data to the target end system.

[0037] The cluster synchronization information required to start the synchronization service is obtained through the metadata stored in the time series database ZDP.

[0038] SyncService is a synchronization service process established on the source side, responsible for acquiring and sending the data to be synchronized. ApplyListener is an application listener established on the target side, responsible for parsing the received synchronization messages and writing the data to be synchronized.

[0039] The method of the present invention involves data synchronization between the T-end and the E-end, and between the primary E-end and the backup E-end, in near real-time. Near-real-time synchronization refers to a solution that synchronizes data by sending time-series data one by one. Because the data difference interval between the target and source ends is very small, it is close to real-time synchronization, hence the name near-real-time synchronization.

[0040] When performing quasi-real-time synchronization, you can trigger data synchronization by periodic scanning, and confirm the scanning interval based on the latency requirement. If you have high performance requirements, you can use multiple quasi-real-time synchronization service processes and use hashing algorithms to allocate tables to each quasi-real-time synchronization service process. Each table's block contains a sequential data queue and a disordered data queue. Every time data is written, the data enters the corresponding queue and is synchronized from the source to the target through the data synchronization queue to achieve data consistency. This ensures that data is written only once, thereby ensuring read and write performance. For reference, Figure 2 .

[0041] When data is synchronized in real time from the T end to the E end and from the primary E end to the backup E end, the source end determines whether there is a synchronization offset based on the synchronization status. If there is a synchronization offset, the data synchronization is completed. For example, when the quasi-real-time synchronization service process (TsSyncService) is started / reconnected, there is a synchronization offset and synchronization completion is required. The specific process is as follows:

[0042] The source-side TsSyncService scans the Block data synchronization queue of the corresponding Table. If the recorded queue offset is different from the queue length, it indicates that there is a synchronization offset. Then, a synchronization request is sent to the target side to send the data to be synchronized in the queue to the target side. The source-side synchronization service process scans the synchronization data queue of the Block in the Table. The Block range to be scanned starts from the latest Block and ends when the first Block with a synchronization offset equal to the maximum offset is scanned. The offset data amount is obtained. In order to prevent the amount of data in each synchronization request from being small, the unsynchronized Block data of multiple Tables can be synchronized in one synchronization request.

[0043] Send a synchronization request to the target end to send the data to be synchronized corresponding to the offset data in the synchronization data queue to the target end.

[0044] The target receives the synchronization request, parses the data to be synchronized and the block identification information in the synchronization request, processes the sequential and out-of-order data separately according to the normal memory data writing process, updates the index, and returns a synchronization success response after processing is complete. After the synchronization request returns a successful response, the source updates the synchronization offset of the corresponding block.

[0045] If the source side crashes during the synchronization process, the target side may have completed synchronization but the source side may not have updated the synchronization offset information. In this case, the source side will synchronize this part of data after restarting. The processing of this part of redundant data can be handled by the target side's data writing process.

[0046] The method of the present invention involves synchronization from the E-end to the C-end and data synchronization between cloud nodes. The synchronization of block data from the E-end to the C-end performs a read operation on the source end, but since the entire block is read through the index, there is no encoding or decoding operation, which is equivalent to copying the file block and has little impact on the performance of the source system. Figure 3 .

[0047] During block data synchronization, when the source system starts or resumes synchronization service, it periodically sends GetBlockState requests to the target system to obtain the target's latest BlockState. This request compares the source system's BlockState with the target's current BlockState. If the source system has unsynchronized blocks with lagging timestamps, it sends a batch of unsynchronized Block data to the target system via a SyncBlockData synchronization request. The synchronization request includes the timestamp range of the blocks in question, identifying the range of the Block data. The BlockState also includes properties such as the corresponding maximum timestamp (maxTS).

[0048] When synchronizing the Immutable Buffer in memory, it is necessary to lock it to prevent it from being released by flushing to disk midway.

[0049] Each time the target side receives a SyncBlockData request, it parses the request through the data disk application listener (BlockApplyListener), mounts the Block data directly to the corresponding ZObject, writes it to the disk according to the normal writing process, updates the index, and updates the maxTS of the corresponding ZObject, and returns a synchronization success response to the source side.

[0050] After receiving the response from SyncBlockData, the source will continue to send unsynchronized data to the target until the BlockState on the target is consistent with that on the source.

[0051] When synchronizing data between cloud nodes involved in the method of the present invention, if there are more than three nodes, distributed consistency protocols such as Raft can be used to ensure data consistency between nodes. After the data is synchronized to a node on the C-end, the node on the C-end will automatically distribute the data to multiple nodes through the Raft protocol, and provide external operations such as query, deletion, and aggregation of this part of the data.

[0052] Time series data is stored on the deployed T end. Since there is no persistent device on the T end, all time series data will be stored in memory. In order to release the occupied memory on the T end in time, the method of the present invention uses a quasi-real-time synchronization method to synchronize the data to the E end with a persistent device as soon as possible. In view of the situation where the interface for providing external query services is provided by the Agent service on the corresponding server deployed on the C end, the method of the present invention synchronizes from the E end to the C end using a Block synchronization method. For a Table, it has multiple Block blocks in memory. These Block blocks can store a certain amount of time series data according to the set size. When the time series data is full, part of the data in the Block block is converted to column storage and compressed and stored on disk. For time series databases, users usually perform query operations on data within a time range, so the system has a higher tolerance for data delays between the E end and the C end, and there is no need to ensure the same short data delay from the T end to the E end.

[0053] The present invention also provides a device for synchronizing cloud-edge data of a time series database, comprising at least one memory and at least one processor;

[0054] The at least one memory is configured to store a machine-readable program;

[0055] The at least one processor is used to call the machine-readable program to execute the method for synchronizing cloud-edge data of a time series database.

[0056] Since the information interaction, execution process and other contents of the processor in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0057] Similarly, the device of the present invention synchronizes data to each end, and the terminal and the edge end, as well as the edge end and the edge end, can perform quasi-real-time synchronization. The quasi-real-time synchronization difference interval is very small, which is close to real-time synchronization, so that terminals without persistent devices can release memory as soon as possible. When the terminal and the edge end, as well as the edge end and the edge end perform real-time synchronization, the synchronization offset is also padded to maintain data integrity. The edge end and the cloud end perform block data synchronization, which facilitates the synchronous transmission of large amounts of data. The device of the present invention achieves high data availability and maintains data integrity at each end. Applications can obtain data directly from this end, which is more convenient. At the same time, the powerful computing power of the cloud can be used to perform other pre-calculation processing on the data to ensure data persistence.

[0058] The present invention also provides a computer-readable medium having computer instructions stored thereon. When executed by a processor, the computer instructions cause the processor to execute the method for synchronizing cloud-edge data in a time series database. Specifically, a system or device equipped with a storage medium can be provided. The storage medium stores software program code that implements the functions of any of the above-described embodiments, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0059] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0060] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0061] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0062] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0063] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A method for synchronizing cloud-edge data in a time series database, characterized by: Including data synchronization between terminals and edges, data synchronization between edges, data synchronization between edges and the cloud, and data synchronization between cloud nodes. Data synchronization between terminals and edges, and between edges and edges, includes quasi-real-time data synchronization and real-time data synchronization. When performing quasi-real-time synchronization between terminals and edges, or between edges and edges, the source writes data to the synchronization data queue of the block in the table. The synchronization service process then synchronously transmits the data in the synchronization data queue to the target. When the terminal and the edge or the edge and the edge perform real-time data synchronization, the source side determines whether there is a synchronization offset based on the synchronization situation. If there is a synchronization offset, the data synchronization is completed. Data synchronization between the edge and the cloud includes Block data synchronization. The source side obtains the latest Block status of the target side, compares it with its own current Block status, and determines whether the Block data is synchronized. If not, it continues to send the corresponding Block data for Block data synchronization until the Block data on the target side is consistent with the source side. Data synchronization between cloud nodes is performed through the Raft protocol.

2. The method according to claim 1, wherein When data is synchronized between the terminal and the edge, between the edge and the edge, or between the edge and the cloud, communication is established through Socket. The source party uses the synchronization service process through the source agent to send data to the target agent, which is the target party. The target agent uses the application listener to parse the data and write the parsed data to the target end.

3. The method according to claim 1 or 2, wherein When performing quasi-real-time synchronization, Block's synchronization data queue includes a sequential data queue and an out-of-order data queue. Data is written into the corresponding synchronization data queue and synchronously transmitted to the target party through the synchronization service process on the source side.

4. The method according to claim 1, wherein When the terminal and the edge or the edge and the edge perform real-time data synchronization, there is a synchronization offset. The synchronization service process on the source end scans the synchronization data queue of the Block in the Table to obtain the offset data amount, sends a synchronization request to the target end, and sends the data to be synchronized corresponding to the offset data amount in the synchronization data queue to the target end. The target end receives the synchronization request, parses the data to be synchronized in the synchronization request, writes the data to be synchronized, and returns a synchronization success response.

5. The method according to claim 1 or 4, wherein The edge and the cloud synchronize block data. The source side regularly sends requests to the target side to obtain the latest block status of the target side. The source side compares the latest block status of the target side with its own current block status to determine whether the block data is synchronized. If the source side has an unsynchronized block with a lagging timestamp, it is considered out of sync and will continue to send synchronization requests to the target side to synchronize block data. The synchronization request contains the timestamp range of the unsynchronized block to identify the block data range. The target side receives the synchronization request, uses the application listener to parse the data of the block to be synchronized in the synchronization request, writes the data of the block to be synchronized, and returns a synchronization success response.

6. The method according to claim 1, wherein The edge completes data synchronization with a node in the cloud, and the node automatically uses the Raft protocol to synchronize data to other nodes.

7. A device for synchronizing cloud-edge data in a time series database, characterized by comprising at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is used to call the machine-readable program to execute a method for synchronizing cloud-edge data of a time series database according to any one of claims 1 to 6.

8. Computer readable medium, characterized in that The computer-readable medium stores computer instructions, which, when executed by a processor, enable the processor to execute a method for synchronizing cloud-edge data of a time series database as described in any one of claims 1 to 6.

Images