Method and system for managing data time sequence in time sequence database

Abstract

The invention provides a management method and system for a data time sequence in a time sequence database, and the method comprises the steps: taking a time window mechanism as a core, and distributing a corresponding time period for each window and initializing an oldest time window pointer and a latest time window pointer through creating a plurality of time windows arranged according to a time sequence; when the data is inserted, the timestamps of the to-be-inserted data are accurately distributed to the corresponding time windows, the data in the windows are sequenced according to the timestamps, the time sequence consistency of the data in the single window is ensured, meanwhile, the data in the oldest time window is stored regularly in a persistent mode, and pointers of the new and old time windows are updated in a rolling mode. And dynamic propulsion of the time window is realized. According to the scheme, the problem of low efficiency of total data sorting is avoided through sub-window management, the time sequence data sorting and writing efficiency is effectively improved, and the data time sequence accuracy is guaranteed based on a timestamp distribution and sorting mechanism.

Description

Technical Field

[0001] This invention relates to the field of data storage management technology, and in particular to a method for managing data time sequence in a time-series database, a management system for data time sequence in a time-series database, an electronic device, and a computer-readable medium. Background Technology

[0002] I. What is time series data? Time-series data, literally understood, is data sorted by time. Baidu Baike defines it as: "Time-series data refers to data in a time series. Time-series data is a data series that records the same statistical indicator in chronological order." To illustrate, consider common household data, such as the smart meters used in every home. Each meter records household electricity consumption at different times, and the constantly flashing indicator lights represent these data fluctuations. The data generated by tens of thousands of smart meters in every building, every neighborhood, and even every city at the same time intervals can be recorded using a time-series database.

[0003] Characteristics of time series data: 1. All collected data is sorted by time. 2. The data source for a single collection point is unique. 3. Data is rarely updated or deleted. 4. Huge amount of data II. What is time disorder? Out-of-order time refers to the phenomenon where the timestamp of data when it is written is inconsistent with the natural / expected time order of the data (for example, the timestamp of data written later is earlier than that of stored data). Since the core value of time-series data depends on the orderliness of the time dimension, out-of-order data can seriously affect query accuracy, aggregation calculation efficiency, compression efficiency, and stream processing results.

[0004] Time-disorder issues are generally caused by network latency, network failures, and data re-entry. The latency varies depending on the scenario.

[0005] In existing technologies, the common method for resolving out-of-order issues in time-series databases is to use an out-of-order buffer window, with the basic approach as follows: When writing data, if its timestamp is within the range of "current system time - maximum tolerable latency", even if it is slightly earlier than the buffered data, it will be stored in the buffer window first, rather than being written directly to disk; The buffer window will periodically (or when a threshold is reached) sort the data by timestamp and write the ordered data in batches to the underlying storage, which is equivalent to exchanging "short delay" for "data ordering".

[0006] Typical applications include InfluxDB's out-of-order-time-window configuration and Prometheus's --storage.tsdb.out-of-order.time-window parameter (default 1 hour).

[0007] Then, a sorting algorithm is used to sort the window within the buffer window to ensure local ordering of the data.

[0008] The main problems with this approach are: 1. There is only one time window. As time goes on, the data within the window becomes larger and larger, and the sorting time becomes longer and longer.

[0009] 2. In order to improve sorting efficiency, some time-series databases use data structures such as skip lists and red-black trees, which also increases the complexity of implementation. Summary of the Invention

[0010] In view of the above problems, the present invention is proposed to provide a method for managing data timing in a time-series database that overcomes or at least partially solves the above problems, a corresponding management system for data timing in a time-series database, an electronic device, and a computer-readable medium.

[0011] This invention discloses a method for managing data time series in a time series database, the method comprising: Create multiple time windows arranged in chronological order; each time window corresponds to a time period, and initialize the pointers for the oldest and newest time windows; Receive the data to be inserted, and allocate the data to be inserted into the corresponding time window according to the timestamp of the data to be inserted; Based on the timestamps of the data, perform internal sorting on the data within each time window in chronological order; Periodically persist the data in the oldest time window and update the pointers for the oldest and newest time windows to advance the time windows in a rolling manner.

[0012] Optionally, before the step of creating multiple time windows arranged in chronological order, the following steps are included: The number of time windows and the length of the time period corresponding to each time window are determined based on the configuration information.

[0013] Optionally, receiving data to be inserted and allocating the data to be inserted into corresponding time windows according to the timestamp of the data to be inserted includes: Receive the data to be inserted; Calculate the time window index based on the difference between the timestamp of the data to be inserted and the start time of the latest time window, as well as the length of the time period; The target time window is determined based on the data window index, and the data to be inserted is inserted into the target time window.

[0014] Optionally, the method further includes: If the timestamp of the data to be inserted exceeds the range of all time windows, the data to be inserted will be stored in a separate buffer and periodically persisted.

[0015] Optionally, based on the timestamps of the data, perform internal sorting on the data within each time window in chronological order, including: Based on the timestamps of the data, insertion sort is used to perform internal sorting of the data within each time window in chronological order.

[0016] Optionally, the method further includes: Perform internal data sorting for multiple time windows in parallel.

[0017] Optionally, update the oldest and latest time window pointers, including: The oldest and newest time windows are identified by cyclically moving the pointer.

[0018] This invention also discloses a data time series management system in a time series database, the system comprising: The time window creation module is used to create multiple time windows arranged in chronological order; each time window corresponds to a time period and initializes the pointers for the oldest and newest time windows. The time window matching module is used to receive the data to be inserted and allocate the data to be inserted into the corresponding time window according to the timestamp of the data to be inserted; The time window internal sorting module is used to perform internal sorting of data within each time window according to the timestamp of the data and in chronological order; The periodic disk persistence and new / old window update module is used to periodically persist the data in the oldest time window and update the pointers of the oldest and newest time windows to advance the time windows in a rolling manner.

[0019] Optionally, the system further includes: The module for determining the number of time windows and the length of the time period is used to determine the number of time windows and the length of the time period corresponding to each time window based on the configuration information.

[0020] Optionally, the time window matching module includes: The data receiving unit is used to receive the data to be inserted. The time window index calculation unit is used to calculate the time window index based on the difference between the timestamp of the data to be inserted and the start time of the latest time window, as well as the length of the time period. The target time window determination unit is used to determine the target time window based on the data window index and insert the data to be inserted into the target time window.

[0021] Optionally, the system further includes: A separate cache module for time-limited data is used to store the data to be inserted into a separate buffer and periodically persist it if the timestamp of the data to be inserted exceeds the range of all time windows.

[0022] Optionally, the sorting module within the time window includes: The internal sorting unit within a time window is used to sort the data within each time window in chronological order using insertion sort based on the data's timestamp.

[0023] Optionally, the system further includes: The parallel sorting module is used to perform internal data sorting in parallel across multiple time windows.

[0024] Optionally, the periodic disk write and old / new window update modules include: The dynamic update unit for new and old time windows is used to identify the current oldest and newest time windows by cyclically moving the pointer.

[0025] The present invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the data timing management method in the timing database as described in this invention.

[0026] The present invention also discloses a computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform a data timing management method in a time-series database as described in the present invention.

[0027] This invention has the following advantages: The data time-series management method in this invention is based on a time window mechanism. It creates multiple time windows arranged chronologically, assigns a corresponding time period to each window, and initializes pointers for the oldest and newest time windows. During data insertion, the data is precisely allocated to the corresponding time window based on its timestamp. Then, the data within each window is sorted sequentially according to its timestamp, ensuring the time-series consistency of data within a single window. Simultaneously, the oldest time window data is periodically persisted, and the pointers for new and old time windows are updated dynamically, enabling the time windows to advance. This scheme avoids the inefficiency of full-data-volume sorting through window-based management, effectively improving the efficiency of time-series data sorting and writing. The timestamp-based allocation and sorting mechanism ensures the accuracy of data time-series timing. The rolling persistent storage releases memory resources while maintaining both real-time data insertion efficiency and historical data security, effectively improving the read / write performance, management flexibility, and long-term storage stability of time-series data. It meets the core requirements of high-concurrency insertion and time-ordered querying in time-series databases. Attached Figure Description

[0028] Figure 1 This is a flowchart of the steps of the method for managing the time sequence of data in a time-series database provided in an embodiment of the present invention; Figure 2 This is a structural block diagram of the data time-series management system in the time-series database provided in the embodiments of the present invention; Figure 3 This is a block diagram of an electronic device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Reference Figure 1 The flowchart illustrates the steps of a method for managing data time series in a time series database provided in an embodiment of the present invention, which may specifically include the following steps: Step 101: Create multiple time windows arranged in chronological order; each time window corresponds to a time period, and initialize the pointers for the oldest and newest time windows; Step 102: Receive the data to be inserted, and allocate the data to be inserted into the corresponding time window according to the timestamp of the data to be inserted; Step 103: Based on the timestamps of the data, perform internal sorting on the data within each time window in chronological order; Step 104: Periodically persist the data in the oldest time window and update the pointers of the oldest and newest time windows to advance the time windows in a rolling manner.

[0031] In an optional embodiment of the invention, prior to the step of creating multiple time windows arranged in chronological order, the method includes: The number of time windows and the length of the time period corresponding to each time window are determined based on the configuration information.

[0032] In an optional embodiment of the present invention, receiving data to be inserted and allocating the data to be inserted into a corresponding time window according to the timestamp of the data to be inserted includes: Receive the data to be inserted; Calculate the time window index based on the difference between the timestamp of the data to be inserted and the start time of the latest time window, as well as the length of the time period; The target time window is determined based on the data window index, and the data to be inserted is inserted into the target time window.

[0033] In an optional embodiment of the present invention, the method further includes: If the timestamp of the data to be inserted exceeds the range of all time windows, the data to be inserted will be stored in a separate buffer and periodically persisted.

[0034] In an optional embodiment of the present invention, internal sorting is performed on the data within each time window according to the timestamp of the data, in chronological order, including: Based on the timestamps of the data, insertion sort is used to perform internal sorting of the data within each time window in chronological order.

[0035] In an optional embodiment of the present invention, the method further includes: Perform internal data sorting for multiple time windows in parallel.

[0036] In an optional embodiment of the present invention, updating the oldest time window pointer and the latest time window pointer includes: The oldest and newest time windows are identified by cyclically moving the pointer.

[0037] This invention utilizes the increasing characteristics of time series and the inherent time key of time series, and adopts a design with multiple rolling time windows to reduce the sorting time and implementation complexity of buffer data, thereby improving the performance of data writing.

[0038] (1) Initialization During initialization, M + 1 windows [M+1] are created according to the configuration. `oldest` is initialized to 0, and `newest` to `M`. Windows [oldest] to `newest` sequentially store data from oldest to newest, with each window storing N minutes of data. `newest` stores data after the time the window was last created, or data after the first data insertion point when the system started. Each window corresponds to a time period, and newly inserted data falls within that time period, effectively partitioning the data by N minutes. Insertion sort is used within each window to ensure chronological ordering within that window. This guarantees the data ordering across the M + 1 windows, meaning the system can guarantee the ordering of older data within N*M minutes. M and N are configurable to accommodate delayed data sorting in different environments.

[0039] (2) Data Insertion The first time data is inserted, the time T is recorded. Data after T is inserted into window[newest], and data before T is inserted into the corresponding window. After N minutes, the data in window[oldest] is written to disk, and oldest = (oldest + 1)%(M + 1) and newest = (newest + 1)%(M + 1) are updated. Accordingly, the start time of each window can be calculated, and newly inserted data is placed in the corresponding window.

[0040] (3) Parallel data sorting Data from multiple time windows is independent, allowing for parallel internal sorting. This fully utilizes multi-core CPU resources, improves sorting efficiency, and prevents large data volumes in one window from causing time-consuming insertion sorting that could impact subsequent data insertion. Each window is sorted independently without interference, ensuring overall throughput increases linearly with the number of windows while mitigating the cumulative latency issues caused by sorting individual windows. Since windows are sequentially ordered over time, merge sort is unnecessary; windows are simply pushed to disk as needed to maintain overall data order.

[0041] (4) Calculation of the time window corresponding to the inserted data The start time of Windows[newest] is the time T when the new window is created, and so on. The start time of windows[newest-1] is T-N. The calculation method for the time window index is: index = M - (T - t) / N; where M is configurable and represents the number of windows to store old data, N is the time length of each window, T represents the start time of window[newest], and t represents the time when the new data is inserted.

[0042] (5) Data processing outside the time window If index < 0, it means the data is very old and exceeds the data range of the time window buffer. This data is saved to a separate buffer and periodically written to disk.

[0043] The technical advantages of this invention are as follows: (1) This invention utilizes the increasing characteristics of time series and the inherent time key of time series, and employs multiple rolling time windows to cache data. The rolling time windows partition the data according to time periods, and the time windows are naturally ordered. The data order is ensured within each window through insertion sort.

[0044] (2) The present invention uses multiple time windows for sorting, which can significantly improve the sorting speed. Newly generated time series data first enters the corresponding time window, and then is sorted within the selected window. Since the amount of data in a single window is relatively small, a simple sorting algorithm can achieve high processing performance. This advantage is more obvious in scenarios where time series data is generated frequently, and in resource-constrained scenarios (such as embedded scenarios).

[0045] (3) The problem solved by this invention is the common problem of time disorder in time series databases. Compared with other methods, it is simpler to implement, more efficient, and avoids complex data structures. At the same time, it utilizes the characteristics of time series itself to greatly improve sorting efficiency, thereby improving data writing efficiency.

[0046] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0047] Reference Figure 2 This diagram illustrates the structural block diagram of a data time series management system provided in an embodiment of the present invention, which may specifically include the following modules: The time window creation module 201 is used to create multiple time windows arranged in chronological order; each time window corresponds to a time period, and the pointers for the oldest and newest time windows are initialized. The time window matching module 202 is used to receive the data to be inserted and allocate the data to be inserted into the corresponding time window according to the timestamp of the data to be inserted. The time window internal sorting module 203 is used to perform internal sorting of the data within each time window according to the timestamp of the data and in chronological order; The periodic disk persistence and new / old window update module 204 is used to periodically persist the data in the oldest time window and update the pointers of the oldest and latest time windows to advance the time windows in a rolling manner.

[0048] In an optional embodiment of the present invention, the system further includes: The module for determining the number of time windows and the length of the time period is used to determine the number of time windows and the length of the time period corresponding to each time window based on the configuration information.

[0049] In an optional embodiment of the present invention, the time window matching module includes: The data receiving unit is used to receive the data to be inserted. The time window index calculation unit is used to calculate the time window index based on the difference between the timestamp of the data to be inserted and the start time of the latest time window, as well as the length of the time period. The target time window determination unit is used to determine the target time window based on the data window index and insert the data to be inserted into the target time window.

[0050] In an optional embodiment of the present invention, the system further includes: A separate cache module for time-limited data is used to store the data to be inserted into a separate buffer and periodically persist it if the timestamp of the data to be inserted exceeds the range of all time windows.

[0051] In an optional embodiment of the present invention, the sorting module within the time window includes: The internal sorting unit within a time window is used to sort the data within each time window in chronological order using insertion sort based on the data's timestamp.

[0052] In an optional embodiment of the present invention, the system further includes: The parallel sorting module is used to perform internal data sorting in parallel across multiple time windows.

[0053] In an optional embodiment of the present invention, the periodic disk write-to-disk and old / new window update module includes: The dynamic update unit for new and old time windows is used to identify the current oldest and newest time windows by cyclically moving the pointer.

[0054] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0055] In addition, embodiments of the present invention also provide an electronic device, such as... Figure 3 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. Memory 303 is used to store computer programs; When the processor 301 executes the program stored in the memory 303, it implements the data timing management method in the timing database as described in the above embodiments.

[0056] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0057] The communication interface is used for communication between the aforementioned terminal and other devices.

[0058] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0059] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0060] like Figure 4As shown, in another embodiment of the present invention, a computer-readable storage medium 401 is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the data timing management method in the time-series database described in the above embodiment.

[0061] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the data timing management method in the time-series database described in the above embodiments.

[0062] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0063] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for managing the time series of data in a time series database, characterized in that, The method includes: Create multiple time windows arranged in chronological order; each time window corresponds to a time period, and initialize the pointers for the oldest and newest time windows; Receive the data to be inserted, and allocate the data to be inserted into the corresponding time window according to the timestamp of the data to be inserted; Based on the timestamps of the data, perform internal sorting on the data within each time window in chronological order; Periodically persist the data in the oldest time window and update the pointers for the oldest and newest time windows to advance the time windows in a rolling manner.

2. The method according to claim 1, characterized in that, Before creating multiple time windows arranged in chronological order, the following steps are included: The number of time windows and the length of the time period corresponding to each time window are determined based on the configuration information.

3. The method according to claim 1, characterized in that, Receiving data to be inserted and allocating the data to be inserted into corresponding time windows according to the timestamp of the data to be inserted, including: Receive the data to be inserted; Calculate the time window index based on the difference between the timestamp of the data to be inserted and the start time of the latest time window, as well as the length of the time period; The target time window is determined based on the data window index, and the data to be inserted is inserted into the target time window.

4. The method according to claim 3, characterized in that, The method further includes: If the timestamp of the data to be inserted exceeds the range of all time windows, the data to be inserted will be stored in a separate buffer and periodically persisted.

5. The method according to claim 1, characterized in that, Based on the timestamps of the data, perform internal sorting on the data within each time window in chronological order, including: Based on the timestamps of the data, insertion sort is used to perform internal sorting of the data within each time window in chronological order.

6. The method according to claim 1, characterized in that, The method further includes: Perform internal data sorting for multiple time windows in parallel.

7. The method according to claim 1, characterized in that, Update the oldest and latest time window pointers, including: The oldest and newest time windows are identified by cyclically moving the pointer.

8. A data time-series management system in a time-series database, characterized in that, The system includes: The time window creation module is used to create multiple time windows arranged in chronological order; each time window corresponds to a time period and initializes the pointers for the oldest and newest time windows. The time window matching module is used to receive the data to be inserted and allocate the data to be inserted into the corresponding time window according to the timestamp of the data to be inserted; The time window internal sorting module is used to perform internal sorting of data within each time window according to the timestamp of the data and in chronological order; The periodic disk persistence and new / old window update module is used to periodically persist the data in the oldest time window and update the pointers of the oldest and newest time windows to advance the time windows in a rolling manner.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the data timing management method in the timing database as described in any one of claims 1-7.

10. A computer-readable medium, characterized in that, It stores instructions that, when executed by one or more processors, cause the processors to perform the data timing management method in the timing database as described in any one of claims 1-7.

Images