An industrial time series data warehousing and indexing method satisfying high concurrency and low latency

Abstract

The application discloses an industrial time series data warehousing and indexing method satisfying high concurrency and low delay, aggregates time series data into micro batches according to a preset time window, reduces network interaction times, preallocates a lock-free ring buffer for each measurement point ID, isolates the writing operation of each measurement point in an independent buffer and does not need to be locked, eliminates lock waiting and context switching overhead caused by index structure contention during concurrent writing of multiple measurement points, reads time series data from the lock-free ring buffer in batches according to a preset period and flushes into a disk storage file, reduces disk I / O times, plays the advantage of sequential writing bandwidth, simultaneously updates a flat index table to decouple index maintenance and data disk landing, avoids the lengthening of the writing path caused by the synchronous maintenance of the index during the writing of each piece of data, directly reads data from the disk storage file after obtaining a physical offset according to the target measurement point ID during the query, and realizes O(1) addressing by using the key value mapping of the flat index table, thereby omitting the multi-level index lookup of the B+ tree or the LSM tree.

Description

Technical Field

[0001] This invention belongs to the field of database technology and relates to a method for storing and indexing industrial time-series data that meets the requirements of high concurrency and low latency. Background Technology

[0002] With the rapid development of industrial internet and IoT technologies, the number of measurement points (sensors, monitoring equipment, etc.) deployed in industrial sites is growing exponentially. These measurement points continuously generate massive amounts of time-series data at millisecond or microsecond frequencies. Taking typical industrial scenarios such as smart grids, petrochemicals, and smart manufacturing as examples, tens of thousands or even hundreds of thousands of measurement points are often involved. Each measurement point generates data at a high frequency (e.g., tens to hundreds of times per second), and the overall data write throughput can reach tens of thousands to hundreds of thousands of points per second.

[0003] The simultaneous writing of massive amounts of high-frequency real-time data from various measurement points places enormous write pressure on traditional database systems. General-purpose time-series databases typically face the following technical bottlenecks when dealing with such scenarios: First, the write path is too long: General time-series databases often need to go through multiple stages such as connection establishment, SQL parsing, transaction initiation, index maintenance, log writing, and data persistence to disk. Each write involves multiple disk I / O operations and lock contention, and it is difficult to control the single-point write latency to the millisecond level.

[0004] Second, severe lock contention: In multi-point concurrent write scenarios, general databases usually use index structures such as B+ trees or LSM trees. During the write process, locking is required to maintain the index. Under high concurrency, lock conflicts are significant, resulting in a sharp drop in write throughput and severe latency jitter.

[0005] Third, lack of measurement point-level isolation: General-purpose time series databases usually mix all measurement point data into the same storage structure, and the writes of different measurement points interfere with each other. A large amount of data written to one measurement point may block the write requests of other measurement points.

[0006] Fourth, high interaction overhead: Traditional writing methods send requests independently for each piece of data. Due to network round-trip latency, the latency of writing a single piece of data is amplified by network latency, making it difficult to meet the strict real-time requirements in industrial scenarios.

[0007] It is evident that without in-depth customization and optimization, general-purpose time-series databases cannot stably achieve an ingestion latency of less than 200 milliseconds for 50,000 data points, making it difficult to meet the stringent real-time requirements of industrial real-time monitoring, fault early warning, and other scenarios. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing general-purpose time-series databases, such as high latency in data entry and difficulty in achieving stable data entry latency of less than 200 milliseconds for 50,000 data points under high-frequency concurrent writing scenarios. This invention provides a method and system for industrial time-series data entry and indexing that meets the requirements of high concurrency and low latency.

[0009] To achieve the above objectives, the present invention employs the following technical solution: A method for storing and indexing industrial time-series data that meets the requirements of high concurrency and low latency includes the following steps: Acquire time-series data from industrial measurement points and aggregate the time-series data into micro-batches according to a preset time window; Pre-allocate a corresponding lock-free circular buffer for each measurement point ID in the database engine, and allocate a corresponding lock-free circular buffer for time series data based on the lock-free circular buffer and micro-batch. According to a preset period, time-series data is read in batches from the unlocked circular buffer, the read time-series data is flushed to the disk storage file, and the flat index table is updated. Obtain the query request, retrieve the target measurement point ID based on the query request, query the flat index table based on the target measurement point ID to obtain the physical offset, and read the time series data from the disk storage file based on the physical offset.

[0010] A further improvement of the present invention is that: The step of aggregating time-series data into micro-batches according to a preset time window includes: Using a fixed time window as the batch division unit, multiple time-series data falling within the same time window are encapsulated into a micro-batch, with the duration of the fixed time window ranging from 10 milliseconds to 100 milliseconds.

[0011] The step of aggregating time-series data into micro-batches according to a preset time window also includes: When the number of time-series data entries accumulated within the time window reaches the preset threshold, the currently accumulated data is pre-packaged into a micro-batch and sent. The preset threshold number of entries is between 100 and 1000.

[0012] The pre-allocation of a lock-free circular buffer for each measurement point ID in the database engine includes: Locate the corresponding unlocked circular buffer based on the measurement point ID to which the time series data belongs; The write pointer of the current lock-free circular buffer is updated using a CAS atomic operation. Write the timing data to the buffer location that the write pointer was pointing to before updating; Atomically update the data count in the current lock-free circular buffer.

[0013] The flat index table is stored in memory, and is loaded from disk into memory when the database engine starts. The flat index table adopts a hash table structure, with the measurement point ID as the key and the physical offset of the data corresponding to the current measurement point ID in the disk storage file as the value.

[0014] The step of reading time-series data in batches from an unlocked circular buffer according to a preset period and flushing the read time-series data to a disk storage file includes: The background thread iterates through all lock-free circular buffers at a period of 10 to 100 milliseconds. Read all currently accumulated unflushed data in all lock-free circular buffers; After merging all the read data, it is sequentially written to the end of the disk storage file.

[0015] A system for storing and indexing industrial time-series data that meets the requirements of high concurrency and low latency includes: The data gateway aggregation module is used to acquire time-series data from industrial measurement points and aggregate the time-series data into micro-batches according to a preset time window. The lock-free buffer write module is used to pre-allocate a corresponding lock-free circular buffer for each measurement point ID in the database engine, and allocate a corresponding lock-free circular buffer for time series data based on the lock-free circular buffer and micro-batch. The batch disk flushing and index update module is used to read time-series data in batches from the lock-free circular buffer according to a preset period, flush the read time-series data to the disk storage file, and update the flat index table. The query and index addressing module is used to obtain query requests, retrieve target measurement point IDs based on query requests, query the flat index table based on target measurement point IDs to obtain physical offsets, and read time series data from disk storage files based on physical offsets.

[0016] A computer program product includes a computer program that, when executed by a processor, implements any one of the methods described.

[0017] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of any of the methods described above.

[0018] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described herein.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention discloses a method for storing and indexing industrial time-series data while maintaining high concurrency and low latency. It acquires time-series data from industrial measurement points and aggregates it into micro-batches according to a preset time window, reducing the number of network interactions between the data gateway and the database engine. This reduces the cumulative contribution of network round-trip latency to storage latency at the source. By pre-allocating a corresponding lock-free circular buffer for each measurement point ID in the database engine and allocating a corresponding lock-free circular buffer for the time-series data based on the lock-free circular buffer and micro-batches, the write operation for each measurement point is isolated in its own independent buffer, and the write process does not require locking. This eliminates the lock wait and context switching overhead caused by contention for the index structure during concurrent writes from multiple measurement points. The data is stored in micro-batches according to a preset period. The timing data is read in batches from the ring buffer and flushed to the disk storage file. Random small block memory writes are transformed into periodic large block sequential disk writes, which reduces the number of disk I / O operations and takes advantage of the bandwidth of sequential disk writes. At the same time, the flat index table is updated to decouple index maintenance from data writing to disk, avoiding the extension of the write path caused by synchronous index maintenance when writing each data. After obtaining the physical offset by querying the flat index table according to the target measurement point ID, the timing data is read directly from the disk storage file. The key-value mapping of the flat index table realizes O(1) addressing, which saves the multi-level index lookup process of general database B+ tree or LSM tree, so that the delay of the query is independent of the total number of measurement points. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 Figure for this invention; Detailed Implementation To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0024] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0026] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0027] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 This invention discloses a method for storing and indexing industrial time-series data that satisfies high concurrency and low latency, comprising the following steps: Step 1: The data gateway receives time-series data from industrial measurement points, aggregates the time-series data into micro-batches according to a preset time window or data volume threshold, and then sends them to the database engine.

[0028] In this embodiment, N measurement points are deployed in the industrial site, where N ≥ 10000. Each measurement point generates time-series data at a frequency of 100Hz, and each data entry contains three fields: measurement point ID, timestamp, and measurement point value. The data gateway aggregates data from all measurement points falling within the same 50-millisecond window into a micro-batch, with each micro-batch containing approximately 50,000 data entries. The data gateway sends the micro-batch to the database engine via a persistent TCP connection.

[0029] In addition, data gateways can also employ data volume threshold strategies: When the accumulated data reaches 1000 records, the currently accumulated data is immediately packaged into a micro-batch and sent without waiting for the time window to expire. These two strategies can be flexibly configured or combined according to the actual scenario.

[0030] Step 2: The database engine pre-allocates a lock-free circular buffer for each measurement point ID. After receiving the micro-batch, it writes the time series data into the lock-free circular buffer corresponding to the measurement point ID.

[0031] At startup, the database engine initializes a lock-free circular buffer for each measurement point ID. Each lock-free circular buffer contains a fixed-size byte array, a write pointer, and a read pointer. Both the write pointer and the read pointer are of type AtomicLong, supporting CAS atomic operations.

[0032] After the database engine receives the micro-batch sent by the data gateway, it parses out the measurement point ID of each data item and then performs the following write operation: Locate the corresponding unlocked circular buffer zone based on the measurement point ID; The write pointer is atomically moved forward by the data length using a CAS operation; Write the timing data to the buffer position pointed to before the write pointer moves; Atomically increment the data count in the buffer.

[0033] Furthermore, since the write operation involves only one CAS atomic operation and one memory copy, and the buffers at different measurement points are independent of each other, the write latency is extremely low and there is no lock contention.

[0034] Step 3: The background thread reads time-series data in batches from each lock-free circular buffer at a preset period and flushes it to the disk storage file, while updating the flat index table.

[0035] The database engine's background disk flushing thread executes periodically, with a 10-millisecond interval. Within each cycle, the background thread iterates through all lock-free circular buffers. For each buffer: Atomically read the read pointer and the current data count of the buffer; Read all unread data in the buffer at once; Atomically update the read pointer to the current write pointer position and clear the buffer.

[0036] The background thread merges all read data and writes it sequentially to the disk storage file in append mode. For each piece of data written, it records its measurement point ID, timestamp, measurement point value, and physical offset in the disk file. The physical offset is equal to the starting offset address plus the data length.

[0037] Furthermore, in this embodiment, the flat index table adopts a hash table structure, using the measurement point ID as the key and the physical offset of the latest data corresponding to that measurement point in the disk file as the value. After each batch disk flush is completed, the background thread updates the physical offset of the corresponding measurement point ID in the flat index table in batches.

[0038] Furthermore, in this embodiment, the flat index table resides in memory and is persisted to disk every minute. If the database engine restarts abnormally, the latest flat index table is loaded from disk to ensure that index data is not lost.

[0039] Step 4: In response to the query request, query the flat index table based on the target measurement point ID to obtain the physical offset, and read the time series data from the disk storage file based on the physical offset.

[0040] When a query request is received, an O(1) lookup is first performed in the flat index table based on the target measurement point ID to obtain the physical offset of the measurement point data in the disk file. Then, based on this physical offset, the data is directly read from the corresponding position in the disk file, and the query result is returned.

[0041] Furthermore, in this embodiment, the query request is: to query the historical data of a certain measuring point within a specified time period.

[0042] Because the flat index table uses a hash structure, the search time complexity is O(1), which is independent of the total number of test points. Even in scenarios with millions of test points, the index lookup time for a single query can still be controlled in the microsecond range.

[0043] This embodiment discloses an industrial time-series data storage and indexing system that meets the requirements of high concurrency and low latency, including: The data gateway aggregation module is used to acquire time-series data from industrial measurement points and aggregate the time-series data into micro-batches according to a preset time window. The lock-free buffer write module is used to pre-allocate a corresponding lock-free circular buffer for each measurement point ID in the database engine, and allocate a corresponding lock-free circular buffer for time series data based on the lock-free circular buffer and micro-batch. The batch disk flushing and index update module is used to read time-series data in batches from the lock-free circular buffer according to a preset period, flush the read time-series data to the disk storage file, and update the flat index table. The query and index addressing module is used to obtain query requests, retrieve target measurement point IDs based on query requests, query the flat index table based on target measurement point IDs to obtain physical offsets, and read time series data from disk storage files based on physical offsets.

[0044] A schematic diagram of a terminal device according to an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0045] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0046] The terminal device can be a desktop computer, laptop computer, cloud server, or other device with strong computing power. The terminal device may include, but is not limited to, a processor and memory.

[0047] The optimal choice for the processor is a multi-core high-speed central processing unit (CPU).

[0048] The memory can be used to store the computer program and / or module. The processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0049] If the modules / units integrated into the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0050] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for storing and indexing industrial time-series data while maintaining high concurrency and low latency, characterized in that, Includes the following steps: Acquire time-series data from industrial measurement points and aggregate the time-series data into micro-batches according to a preset time window; Pre-allocate a corresponding lock-free circular buffer for each measurement point ID in the database engine, and allocate a corresponding lock-free circular buffer for time series data based on the lock-free circular buffer and micro-batch. According to a preset period, time-series data is read in batches from the unlocked circular buffer, the read time-series data is flushed to the disk storage file, and the flat index table is updated. Obtain the query request, retrieve the target measurement point ID based on the query request, query the flat index table based on the target measurement point ID to obtain the physical offset, and read the time series data from the disk storage file based on the physical offset.

2. The method for storing and indexing industrial time-series data with high concurrency and low latency as described in claim 1, characterized in that, The step of aggregating time-series data into micro-batches according to a preset time window includes: Using a fixed time window as the batch division unit, multiple time-series data falling within the same time window are encapsulated into a micro-batch, with the duration of the fixed time window ranging from 10 milliseconds to 100 milliseconds.

3. The method for storing and indexing industrial time-series data with high concurrency and low latency as described in claim 1, characterized in that, The step of aggregating time-series data into micro-batches according to a preset time window also includes: When the number of time-series data entries accumulated within the time window reaches the preset threshold, the currently accumulated data is pre-packaged into a micro-batch and sent. The preset threshold number of entries is between 100 and 1000.

4. The method for storing and indexing industrial time-series data with high concurrency and low latency as described in claim 1, characterized in that, The pre-allocation of a lock-free circular buffer for each measurement point ID in the database engine includes: Locate the corresponding unlocked circular buffer based on the measurement point ID to which the time series data belongs; The write pointer of the current lock-free circular buffer is updated using a CAS atomic operation. Write the timing data to the buffer location that the write pointer was pointing to before updating; Atomically update the data count in the current lock-free circular buffer.

5. The method for storing and indexing industrial time-series data with high concurrency and low latency as described in claim 1, characterized in that, The flat index table is stored in memory, and is loaded from disk into memory when the database engine starts. The flat index table adopts a hash table structure, with the measurement point ID as the key and the physical offset of the data corresponding to the current measurement point ID in the disk storage file as the value.

6. The method for storing and indexing industrial time-series data with high concurrency and low latency as described in claim 5, characterized in that, The step of reading time-series data in batches from an unlocked circular buffer according to a preset period and flushing the read time-series data to a disk storage file includes: The background thread iterates through all lock-free circular buffers at a period of 10 to 100 milliseconds. Read all currently accumulated unflushed data in all lock-free circular buffers; After merging all the read data, it is sequentially written to the end of the disk storage file.

7. A system for storing and indexing industrial time-series data that meets the requirements of high concurrency and low latency, characterized in that, include: The data gateway aggregation module is used to acquire time-series data from industrial measurement points and aggregate the time-series data into micro-batches according to a preset time window. The lock-free buffer write module is used to pre-allocate a corresponding lock-free circular buffer for each measurement point ID in the database engine, and allocate a corresponding lock-free circular buffer for time series data based on the lock-free circular buffer and micro-batch. The batch disk flushing and index update module is used to read time-series data in batches from the lock-free circular buffer according to a preset period, flush the read time-series data to the disk storage file, and update the flat index table. The query and index addressing module is used to obtain query requests, retrieve target measurement point IDs based on query requests, query the flat index table based on target measurement point IDs to obtain physical offsets, and read time series data from disk storage files based on physical offsets.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images