A time series data storage method and related apparatus

Abstract

The application provides a time series data storage method and related device, and relates to the technical field of software, which comprises the following steps: obtaining a numerical sequence to be written, the numerical sequence being composed of time series data points, the time series data points comprising metadata, a timestamp and a high-precision floating-point numerical value; encoding and block processing the high-precision floating-point numerical value to obtain a physical data block; mapping the metadata to a time sequence identifier of the physical data block, and writing the physical data block, and an inverted mapping relationship between the metadata and the time sequence identifier into a physical time slice to which the timestamp belongs; determining an active time window and an inactive time window through index rotation, and storing a first physical time slice in the active time window in the physical time slice to a memory and a second physical time slice in the inactive time window to a disk. The application effectively solves the problem of unlimited expansion of the memory caused by high-base sequence shock, and reduces the disk space occupation and input / output overhead.

Description

Technical Field

[0001] This application relates to the field of software technology, and in particular to a time-series data storage method and related apparatus. Background Technology

[0002] With the rapid development of cloud-native architecture, microservices, and IoT technologies, the monitoring metrics data generated by IT infrastructure and business systems are experiencing exponential growth. This data is typically in time-series format, characterized by continuous high-concurrency writes, rich multi-dimensional key-value pair labels, and the need for long-term storage for historical trend analysis. To cope with this massive amount of time-series data, dedicated time-series databases have emerged and become the infrastructure of the monitoring field.

[0003] When storing time-series data, a global static inverted index architecture is mainly adopted. The system maps the parsed metadata to an internally unique time series identifier (TSID), and constructs a huge global index structure by inverting the mapping relationship between all metadata and time series identifiers throughout the entire database lifecycle. This global index structure is required to reside in the application layer memory for a long time in order to facilitate fast retrieval.

[0004] However, in the face of modern cloud-native massive high-concurrency monitoring scenarios, the global static inverted index architecture is highly dependent on memory residency. In the face of the high cardinality sequence oscillation caused by the frequent start-up and shutdown of containers and dynamic label changes in the cloud-native environment, the memory usage is very likely to continuously collide due to the infinite accumulation of historical time series data, which can lead to memory overflow and crash. Summary of the Invention

[0005] In view of the above problems, this application provides a time-series data storage method and related apparatus to solve the problem of infinite memory expansion caused by high-cardinality sequence oscillations. The specific solution is as follows:

[0006] A first aspect of this application provides a time-series data storage method, the time-series data storage method comprising:

[0007] Obtain the numerical sequence to be written, the numerical sequence being composed of time-series data points, the time-series data points including metadata, timestamps, and high-precision floating-point values;

[0008] The high-precision floating-point values ​​are encoded and segmented to obtain physical data blocks;

[0009] The metadata is mapped to the time-series identifier of the physical data block, and the physical data block, as well as the inverted mapping relationship between the metadata and the time-series identifier, are written into the physical time slice to which the timestamp belongs;

[0010] Active and inactive time windows are determined by index rotation, and the first physical time slice located in the active time window is stored in memory, while the second physical time slice located in the inactive time window is stored in disk.

[0011] In one possible implementation, before performing the encoding and block processing of the high-precision floating-point value to obtain physical data blocks, the time-series data storage method further includes:

[0012] The high-precision floating-point value is subjected to precision control and truncation based on the metadata to remove random mantissas.

[0013] In one possible implementation, the process of encoding and segmenting the high-precision floating-point value to obtain physical data blocks includes:

[0014] The high-precision floating-point value is sequentially subjected to integerization, second-order difference operation, positive number mapping, and byte compression to obtain the target value;

[0015] The target value is divided into its corresponding physical data blocks according to the preset capacity threshold.

[0016] In one possible implementation, determining the active and inactive time windows through index rotation includes:

[0017] The time window rotation event is triggered by maintaining a state machine in memory;

[0018] In response to the time window rotation event, a time window is instantiated in the memory to rotate into a new current time window, the current time window before rotation is downgraded to a new historical time window, and the historical time window before rotation is marked as read-only.

[0019] The new current time window and the new historical time window are designated as the active time window, and the historical time window marked as read-only is designated as the inactive time window.

[0020] In one possible implementation, the time-series data storage method further includes:

[0021] In response to the query command, a distributed parallel retrieval and restoration aggregation is performed on the first physical time slice and the second physical time slice.

[0022] In one possible implementation, the distributed parallel retrieval and restoration aggregation of the first physical time slice and the second physical time slice includes:

[0023] The target timestamp, target metadata, and target aggregation method in the query instruction are obtained through parsing.

[0024] Parallel retrieval of the memory and the disk is performed based on the target timestamp to determine the target time series identifier corresponding to the target metadata from the first physical time slice and the second physical time slice using an inverted mapping relationship, and to determine the target physical data block based on the target time series identifier;

[0025] The target physical data block is decoded to restore the target high-precision floating-point value, and the target high-precision floating-point value is processed according to the target aggregation method.

[0026] A second aspect of this application provides a time-series data storage device, the time-series data storage device comprising:

[0027] The time-series data acquisition module is used to acquire the numerical sequence to be written. The numerical sequence consists of time-series data points, which include metadata, timestamps, and high-precision floating-point values.

[0028] The time-series data storage module is used to encode and segment the high-precision floating-point values ​​to obtain physical data blocks; map the metadata to the time-series identifiers of the physical data blocks, and write the physical data blocks, as well as the inverted mapping relationship between the metadata and the time-series identifiers, into the physical time slice to which the timestamp belongs; determine the active time window and the inactive time window through index rotation, and store the first physical time slice located in the active time window into memory and the second physical time slice located in the inactive time window into disk.

[0029] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the timing data storage method of the first aspect or any implementation thereof.

[0030] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0031] The memory is used to store computer programs;

[0032] The processor is used to execute the computer program so that the electronic device can implement the timing data storage method of the first aspect or any implementation thereof.

[0033] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the time-series data storage method described in the first aspect or any implementation thereof.

[0034] Using the above technical solution, this application provides a time-series data storage method and related apparatus, comprising: acquiring a numerical sequence to be written, the numerical sequence being composed of time-series data points, the time-series data points including metadata, timestamps, and high-precision floating-point values; encoding and segmenting the high-precision floating-point values ​​to obtain physical data blocks; mapping the metadata to time-series identifiers of the physical data blocks, and writing the physical data blocks, as well as the inverted mapping relationship between the metadata and the time-series identifiers, into the physical time slice to which the timestamp belongs; determining active and inactive time windows through index rotation, and storing the first physical time slice located within the active time window into memory and the second physical time slice located within the inactive time window into disk. This application combines the encoding and block processing of high-precision floating-point values ​​with the writing of timestamp-based physical time slices. This transforms the original high-entropy data into compact physical data blocks, while the metadata index is strictly confined to specific physical time slices. A mechanism based on index rotation to determine active and inactive time windows allows only active physical time slices to be retained in memory, while expired physical time slices are persisted to disk, thus decoupling memory usage from the total amount of historical data. This process effectively solves the problem of infinite memory expansion caused by high-cardinality sequence oscillations, avoiding the risk of system memory overflow and crash. Simultaneously, by encoding and block processing high-precision floating-point values, the information entropy of data storage is reduced, disk space usage and input / output overhead are decreased, and the stability and reliability of long-term storage of massive time-series data are improved, ensuring that the system can operate efficiently and at low cost with limited hardware resources. Attached Figure Description

[0035] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0036] Figure 1 A flowchart illustrating a time-series data storage method provided in an embodiment of this application;

[0037] Figure 2 This is a partial flowchart illustrating a time-series data storage method provided in an embodiment of this application;

[0038] Figure 3 This is another schematic flowchart of a time-series data storage method provided in an embodiment of this application;

[0039] Figure 4 This is a partial flowchart illustrating a time-series data storage method provided in an embodiment of this application.

[0040] Figure 5 This is a schematic diagram of the structure of a time-series data storage device provided in an embodiment of this application;

[0041] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0043] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0044] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0045] To facilitate understanding of this application, the relevant concepts involved in this application will be explained below:

[0046] Numerical sequence: In this application, it specifically refers to a set of time-series data points with timestamps that are continuously generated by monitoring systems or sensors and other devices. A complete time-series data point typically includes a metric name, a set of key-value pairs (Labels), a timestamp, and a high-precision floating-point value (Value).

[0047] High cardinality sequence oscillation: A unique problem in the field of time series databases. It refers to the phenomenon that, over time, due to frequent changes in system restarts, container redeployment, or dynamic tags (such as UUIDs, temporary IPs, etc.), the total number of globally unique time series identifiers (i.e., combinations of indicator names and key-value pair labels) explodes, and old sequences constantly become invalid while new sequences are constantly generated. This phenomenon leads to the unlimited expansion of the memory usage of traditional global indexes.

[0048] Time Series Identifier (TSID): In this application, this is an internal numeric ID used to uniquely identify a time series data point. The system uses a consistent hashing algorithm to map the originally long metadata (i.e., indicator name and key-value pair label) into a globally unique integer TSID. The underlying disk data blocks are all associated and stored based on TSIDs to significantly reduce storage redundancy.

[0049] Inverted Index: In this application, it specifically refers to a data structure that maps metadata to corresponding time-series identifiers. When a user queries a specific tag, the system quickly finds the set of all matching time-series identifiers using the inverted index.

[0050] Effective bit truncation (dynamic precision control): A lossy preprocessing technique described in this application. It refers to forcibly removing minute fluctuations after the P decimal places of the original high-precision floating-point number (IEEE 754 standard) through mathematical scaling (multiplication and rounding) according to configuration rules. Its purpose is to remove meaningless random noise (information entropy) caused by sensor jitter or computation, making the sequence smoother and creating mathematical conditions for subsequent compression.

[0051] Second-order difference algorithm (Delta-of-Delta): A coding algorithm for continuous numerical sequences. First-order difference calculates the difference between two adjacent values, while second-order difference calculates the difference between two adjacent first-order differences. After pre-truncation and smoothing, the second-order difference result of the time-series data will mostly approach 0, thus transforming a large number of absolute values ​​into a very small integer sequence.

[0052] ZigZag encoding: a coding method that maps signed integers (positive and negative) to unsigned positive integers, so that negative numbers with small absolute values ​​(such as -1, -2) are converted into smaller positive numbers, avoiding the use of too many leading 1s when negative numbers are represented in two's complement in the computer.

[0053] LEB128 encoding (Little-Endian Base 128): A variable-length integer compression encoding technique that can represent small integers close to 0 with very few bytes (usually 1-2 bytes). The combination of the two is the golden combination for time-series differential data compression.

[0054] Memory-mapped file (mmap): An operating system-level memory management mechanism. In this application, it is used to handle historical data queries. The system does not directly load the historical index file into application-level memory; instead, it maps the disk file to the system's virtual memory address space. The required index pages are loaded locally on demand via the operating system's page fault, and automatically reclaimed by the system kernel's LRU algorithm when memory is scarce, thus ensuring extremely low application-level memory overhead.

[0055] Traditional time-series databases face two intertwined and fatal bottlenecks when dealing with large-scale monitoring scenarios:

[0056] Firstly, at the metadata indexing level, existing systems generally adopt a global static inverted index architecture. Over time, and with the frequent changes of dynamic tags (such as Pod IP, UUID, temporary container identifiers, etc.) in cloud-native environments (i.e., high cardinality sequence oscillations), a large amount of historically invalid time-series metadata remains in the application layer memory for a long time and cannot be released. This causes the index size to expand infinitely over time, which will eventually lead to system memory overflow and crash under limited hardware resources.

[0057] Secondly, at the physical data storage level, the raw time-series data collected by monitoring equipment or sensors is mostly high-precision floating-point numbers (such as IEEE 754 standard floating-point numbers) carrying a large amount of meaningless random fluctuation noise. This high information entropy tail noise causes a sharp drop in the efficiency of lossless compression algorithms (such as the Gorilla algorithm) widely used in traditional time-series databases, which in turn leads to rapid consumption of disk storage space and extreme congestion of underlying I / O bandwidth when massive amounts of historical data are written to disk.

[0058] Therefore, how to completely decouple system memory usage from the total amount of historical data under a unified storage architecture, and break through the performance bottleneck of traditional lossless compression algorithms when dealing with floating-point noise, so as to achieve long-term, stable, low-cost storage and efficient retrieval of massive high-cardinality time-series data under limited hardware resources, has become an urgent problem to be solved at this stage.

[0059] To address the aforementioned issues, this application provides a time-series data storage method that strictly divides the index structure on logical memory and physical disk into multiple independent physical time slices according to a fixed physical time span. Furthermore, a lifecycle management mechanism is introduced at runtime, ensuring that only the physical time slices of the currently active time window being written reside in memory, while expired physical time slices are completely serialized and written to disk and thoroughly unloaded from the application layer memory. This decouples memory usage from the total amount of historical data, ensuring that the system's memory consumption is only related to the number of active sequences within the current extremely short time window (i.e., the instantaneous cardinality). Regardless of how long the system runs or how much historical data has accumulated, its memory level remains stable and controllable, significantly mitigating the memory crash crisis in high cardinality scenarios. Furthermore, before the data is written to disk, this application uses a dynamic routing mechanism to mathematically scale and truncate the original floating-point numbers, forcibly eliminating minor fluctuations beyond the set precision. This eliminates the randomness of the data in principle, making the originally chaotic numerical stream highly smooth and predictable. Subsequently, these smoothed values ​​are converted into integers and a second-order difference (Delta-of-Delta) algorithm is introduced for calculation. At this time, variable-length coding (ZigZag and LEB128) and a general block compression algorithm (ZSTD) after being divided into fixed data points (such as 8192) are used to form cascaded compression. Through mathematical transformation, the information entropy of the time-series data is greatly reduced, making it perfectly fit the theoretical model of the underlying compression algorithm. Thus, a very small precision concession is made in exchange for an order of magnitude reduction in disk space occupation and I / O overhead.

[0060] In other words, this application, through the collaborative working mechanism of constructing a physical time-sharded index architecture and a cascaded data compression pipeline, forms a complete technical closed loop from data writing, encoding and compression, sharded storage to rotation management. This not only logically and completely severs the dependence of historical accumulated data on real-time memory resources, but also breaks through the performance bottleneck of traditional lossless compression algorithms in processing high-frequency noise data at the physical storage level. It achieves visualized and controllable system resource consumption and an order-of-magnitude improvement in storage efficiency, providing a systematic solution for the long-term stable storage of massive high-cardinality time-series data in cloud-native environments. The time-series data storage method of this application's embodiments will be described in detail below with reference to the accompanying drawings.

[0061] See Figure 1 , Figure 1 This is a flowchart illustrating a time-series data storage method provided in an embodiment of this application. Figure 1 As shown in the figure, the time-series data storage method provided in this application embodiment may include steps S101 to S104, which are described in detail below.

[0062] S101, obtain the numerical sequence to be written. The numerical sequence consists of time-series data points, which include metadata, timestamps, and high-precision floating-point values.

[0063] In this embodiment, a numerical sequence can be collected in real time from monitoring systems or sensors. This numerical sequence consists of continuous time-series data points, which include metadata, timestamps, and high-precision floating-point values. The metadata specifically includes an indicator name and a set of key-value pair labels. Specifically, the key-value pair labels contain a set of labels in key-value format, used to uniquely represent the business attributes of the data; the timestamp refers to the time when the data was generated or collected, typically represented in Unix timestamp format; the high-precision floating-point values ​​can be floating-point numbers conforming to the IEEE 754 standard, containing the actual collected value and any accompanying minor jitter noise. By acquiring time-series data points containing metadata, timestamps, and high-precision floating-point values, a standardized input basis can be provided for subsequent encoding compression and indexing sharding, ensuring that the system can identify the business and time dimensions of the data.

[0064] In one possible implementation, to eliminate redundant information entropy at the tail of floating-point numbers caused by sensor jitter or computational inaccuracy within a controllable error range, this application embodiment performs noise reduction and entropy reduction preprocessing through dynamic precision matching. This smooths the originally complex and variable floating-point number sequence, providing a high-quality data source that conforms to the mathematical assumptions of subsequent efficient compression algorithms. To this end, this application embodiment provides a time-series data storage method, which, before executing step S102, further includes the following steps:

[0065] Precision control and truncation of high-precision floating-point values ​​are performed based on metadata to remove random mantissas.

[0066] In this embodiment, a precision configuration routing table based on regular expressions is pre-maintained. A matching operation is then performed on this precision configuration routing table based on the metric name and / or a set of key-value pair tags in the metadata. If a match is successful, the precision parameters specified in the precision configuration routing table are used; if a match fails, the globally default precision parameters are used. This configuration routing table supports hot reloading, allowing the system to dynamically adjust the precision parameters of different data sources without restarting.

[0067] Furthermore, based on the determined precision parameters, the high-precision floating-point values ​​are processed using a mathematical scaling truncation method, specifically calculated according to the following formula (1):

[0068] (1);

[0069] in, This represents the truncated value; This represents the rounding function; Represents high-precision floating-point values; Indicates the precision parameter.

[0070] After processing using mathematical scaling and truncation, the decimal part can be forcibly removed. This process actively removes meaningless random mantissas introduced by sensor physical jitter, signal transmission noise, or floating-point calculation errors, transforming the original floating-point number with high information entropy into a smooth numerical sequence with low information entropy.

[0071] The truncated values ​​generated in this embodiment can be used as direct input for subsequent encoding processing, ensuring that subsequent second-order difference operations can produce a large number of extremely small integers close to zero, thus laying the necessary mathematical foundation for improving overall compression efficiency. Therefore, by controlling the precision of high-precision floating-point values ​​and truncating them based on metadata, lossy preprocessing and entropy reduction optimization before data writing are achieved, solving the problem of traditional lossless compression algorithms failing due to random fluctuations in the tail of floating-point numbers.

[0072] S102 encodes and divides high-precision floating-point values ​​into blocks to obtain physical data blocks.

[0073] In this embodiment, high-precision floating-point values ​​are mathematically transformed to eliminate random noise and converted into an integer sequence. Then, a difference algorithm is used to calculate the difference between adjacent values, converting absolute values ​​into extremely small incremental values. Furthermore, variable-length encoding technology is used to map these small integers into fewer bytes. Finally, the encoded data stream is divided into fixed-size physical data blocks according to a preset capacity threshold. It should be noted that the physical data blocks are binary data blocks.

[0074] For example, the fixed size of the physical data block can be set to 8192 data points. That is, a block compression algorithm is executed once every 8192 high-precision floating-point values ​​to generate an independent physical data block. Through this encoding and block processing, the high-precision floating-point numbers that originally occupied 8 bytes can be compressed to an average of 1-2 bytes, significantly reducing the amount of data written to disk and providing granular support for subsequent parallel retrieval based on data blocks.

[0075] In one possible implementation, through the coordinated use of integerization, second-order differencing, positive number mapping, and byte compression, the originally messy high-precision floating-point values ​​can be reduced to highly sparse and compact target values, significantly lowering the theoretical lower bound of data storage. See also Figure 2 , Figure 2 This is a partial flowchart illustrating a time-series data storage method provided in an embodiment of this application. Figure 2As shown in the embodiment of this application, a time-series data storage method is provided, wherein step S102, "encoding and segmenting high-precision floating-point values ​​to obtain physical data blocks", may include steps S201 to S202, which are described in detail below.

[0076] S201 sequentially performs integerization, second-order difference operation, positive number mapping, and byte compression on the high-precision floating-point value to obtain the target value.

[0077] In this embodiment, a fixed scaling factor is used to convert high-precision floating-point values ​​into pure integers, thereby eliminating ambiguity in the underlying representation of floating-point numbers in the computer and completing the underlying data type conversion losslessly and efficiently.

[0078] Furthermore, a second-order difference algorithm is introduced to calculate the integerized result of the high-precision floating-point value. By calculating the difference between two adjacent first-order difference values, the trend characteristics of the numerical change are extracted. Since the pre-truncation has eliminated random noise, after the smoothed sequence undergoes second-order difference, most results will approach 0, thus transforming the huge absolute values ​​into a very small integer sequence.

[0079] Furthermore, the ZigZag encoding method is used to map signed second-order difference values ​​to unsigned positive integers, avoiding the use of too many leading 1s in the high-order bits when negative numbers are represented in two's complement in the computer, and ensuring that negative numbers with small absolute values ​​can also be mapped to small positive numbers.

[0080] Furthermore, by utilizing LEB128 variable-length encoding technology, the mapped unsigned small integers are compressed to 1 to 2 bytes.

[0081] In this embodiment of the application, during the data flow process of high-precision floating-point values ​​through integerization, second-order difference, positive number mapping, and byte compression, the data bit width significantly narrows as the processing stage progresses, which intuitively reflects the process of reducing information entropy.

[0082] S202, divide the target value into the corresponding physical data blocks according to the preset capacity threshold.

[0083] In this embodiment, the preset capacity threshold is the upper limit of the number of data points contained in a single physical data block. The capacity threshold is typically set to balance memory management efficiency and the optimal working range of the compression algorithm. For example, setting the capacity threshold to 8192 data points means that every 8192 target values ​​processed by the above encoding method form an independent logical unit.

[0084] Physical data blocks are fixed-length storage units that carry the target values, and they are the direct targets of general block compression algorithms such as ZSTD. When the number of target values ​​reaches the capacity threshold, a second deep compression is immediately performed on the physical data block and it is persistently written to disk. Dividing physical data blocks into fixed-length segments according to the capacity threshold not only makes the size of each physical data block predictable, facilitating memory management and concurrent read / write operations, but also allows subsequent block compression algorithms to maximize the exploitation of data redundancy within a fixed context window, thereby achieving a compression ratio superior to streaming compression. Therefore, this multi-level, interconnected processing mechanism can fundamentally solve the problem of low compression efficiency of traditional lossless compression algorithms when dealing with high-precision floating-point numbers with random noise, achieving an order-of-magnitude reduction in disk space usage and I / O overhead.

[0085] S103, map metadata to time-series identifiers of physical data blocks, and write the physical data blocks, as well as the inverted mapping relationship between metadata and time-series identifiers, into the physical time slice to which the timestamp belongs.

[0086] In this embodiment, a consistent hashing algorithm can be used to map the indicator name and a set of key-value pair labels in the metadata to the time series identifiers of the physical data blocks, thereby constructing an inverted index mapping relationship between the metadata and the time series identifiers to support tag-based fast retrieval.

[0087] By parsing the timestamps of time-series data points within a physical data block at the current time, the time interval to which it belongs, such as the corresponding calendar day, is determined. The physical data block and its inverted index are then written to the physical time slice within that time interval, rather than to the global index. By strongly binding the metadata index to physical time, the traditional architecture of an infinitely accumulating global index over time is broken. This ensures that each physical time slice only carries active sequence information within a specific time period, laying the data structure foundation for subsequent memory rotation mechanisms.

[0088] S104: The active time window and the inactive time window are determined by index rotation, and the first physical time slice located in the active time window is stored in memory, and the second physical time slice located in the inactive time window is stored in disk.

[0089] In this embodiment, index rotation is a memory lifecycle management mechanism implemented in conjunction with physical time sharding. It triggers time window switching events by maintaining a state machine in memory. Index rotation designates the period currently receiving writes or recently completed writes as the active time window, while historical periods that have crossed time boundaries and no longer receive new data writes are designated as inactive time windows. Furthermore, the first physical time shard within the active time window is stored in memory, while the second physical time shard within the inactive time window is stored on disk. For example, if the active window is set to the most recent 48 hours, only the physical time shards for today and the previous day reside in memory, while physical time shards from the day before yesterday and earlier are unloaded to disk. When querying physical time shards on disk, the operating system's memory-mapped file mechanism can be used to load them partially as needed, rather than loading them entirely into application-level memory. Through this index rotation and hierarchical storage strategy, the system's memory consumption is strictly limited to the number of active sequences within the current very short time window. No matter how many TB of historical data accumulates, the memory level remains stable and controllable, thus completely solving the memory overflow problem caused by high cardinality sequence oscillations.

[0090] In one possible implementation, a state machine is maintained in memory, keeping only the index of the current time window being written and its adjacent previous time window in a memory-resident state. For example, using a calendar day, at most only the data tags of the currently being written 24 hours and the adjacent past 24 hours are retained in memory. When the system time crosses the boundary of a calendar day, a rotation event is automatically triggered. See also Figure 3 , Figure 3 This is another schematic flowchart illustrating a time-series data storage method provided in an embodiment of this application. For example... Figure 3 As shown in the embodiment of this application, a time-series data storage method is provided, wherein step S104, "determining the active time window and the inactive time window by index rotation", may include steps S301 to S303, which are described in detail below.

[0091] S301 triggers time window rotation events by maintaining a state machine in memory.

[0092] In this embodiment, a state machine runs in memory within the application layer of the time-series database to monitor the system clock and preset time-slice boundaries in real time. Specifically, the state machine continuously compares the current system time with the start time of the next physical time slice. If the current system time reaches the start time of the next physical time slice, it automatically transitions from the normal operation state to the rotation trigger state, thereby generating a time window rotation event. For example, if the time-slice boundary is set to 24 hours, the state machine immediately captures the transition and triggers the time window rotation event the instant the system time jumps from 23:59:59 to 00:00:00 the next day. Through this proactive monitoring mechanism based on the state machine, the switching action of the time window can be ensured to have strict time consistency and atomicity, avoiding data writing errors caused by time judgment delays.

[0093] S302 responds to the time window rotation event, instantiates a time window in memory, rotates it to the new current time window, downgrades the previous current time window to the new historical time window, and marks the previous historical time window as read-only.

[0094] In this embodiment, upon receiving a time window rotation event, the system instantiates a new current time window for writing physical time slices within the new time span. Additionally, the attributes of the current time window, which was previously in a writing state, are modified to historical attributes, making it a historical time window. Writing physical time slices to it is stopped, but the data in its memory is retained for immediate querying. Furthermore, a protective lock is applied to the historical time window before rotation, prohibiting any modification operations, and its memory address mapping is prepared for serialization for subsequent unloading. Through this series of atomic operations, a smooth transition of the index lifecycle in memory is achieved, ensuring the continuity and security of data read / write operations at the moment of rotation.

[0095] S303, designate the new current time window and the new historical time window as active time windows, and designate the historical time window marked as read-only as inactive time windows.

[0096] In this embodiment, the newly instantiated current time window and the newly downgraded historical time window are designated as active time windows. The first physical time slice within the active time window resides in memory to support high-frequency read and write operations. The historical time window marked as read-only is designated as an inactive time window, and the second physical time slice within the inactive time window is removed from the application layer memory. This mechanism enables the system to quickly convert expired historical metadata into an inactive state and release memory resources when facing high cardinality sequence oscillations, fundamentally eliminating the risk of memory overflow caused by the infinite expansion of the global index.

[0097] In one possible implementation, embodiments of this application also provide a scheme for distributed parallel retrieval and reconstruction aggregation of physical time slices in memory and disk. In this regard, a time-series data storage method provided by embodiments of this application further includes the following steps:

[0098] In response to the query command, the system performs distributed parallel retrieval and restoration aggregation on the first physical time shard and the second physical time shard.

[0099] In this embodiment, a query command initiated by a monitoring client or data analysis platform is received. Based on the metadata carried by the command, a specific time-series identifier is located. Then, based on the query time period, parallel retrieval is performed on physical time shards in memory or on disk. The retrieved physical data blocks are then subjected to a reverse decoding process and aggregated to output continuous and complete time-series data results. This distributed parallel retrieval mechanism effectively breaks through the I / O bottleneck of a single storage medium, significantly improving the throughput of queries across long time spans. Simultaneously, the unified restoration and aggregation interface shields the physical differences between the underlying heterogeneous memory and disk storage, ensuring data consistency and transparency from the user's perspective, and achieving efficient end-to-end retrieval from dynamic shard indexing to cascaded compressed data.

[0100] See Figure 4 , Figure 4 This is a further schematic flowchart of a time-series data storage method provided in an embodiment of this application. For example... Figure 4 As shown in the embodiment of this application, a time-series data storage method is provided, wherein the above step of "performing distributed parallel retrieval and restoration aggregation of the first physical time segment and the second physical time segment" may include steps S401 to S403, which are described in detail below.

[0101] S401, parsing to obtain the target timestamp, target metadata, and target aggregation method from the query command.

[0102] In this embodiment, after receiving a query instruction, semantic parsing is performed to extract key retrieval parameters, including a target timestamp, target metadata, and a target aggregation method. The target timestamp represents the time range of the data to be queried, typically expressed as a interval between a start and end time point, used to define the scanning boundaries of subsequent retrieval operations on physical time slices. The target metadata consists of specified metric names and / or tags, serving as the lookup key for the inverted index to filter specific data that conforms to business logic. The target aggregation method is the desired mathematical function type, including but not limited to summation, average, maximum, minimum, or rate calculations, which determines the final processing logic after data reconstruction. This parsing mechanism transforms ambiguous user intent into atomic operation parameters that the storage engine can precisely execute, providing clear input for subsequent parallel retrieval.

[0103] S402, perform parallel retrieval of memory and disk based on the target timestamp to determine the target time series identifier corresponding to the target metadata from the first physical time slice and the second physical time slice using an inverted mapping relationship, and determine the target physical data block based on the target time series identifier.

[0104] In this embodiment, parallel retrieval of memory and disk is performed based on the target timestamp. Specifically, for time segments within the active time window of the target timestamp, the first physical time slice residing in memory is directly accessed; for time segments within the inactive time window of the target timestamp, the second physical time slice on the disk is mapped to the virtual address space as needed through the memory-mapped file mechanism provided by the operating system, thereby avoiding loading the entire historical index into memory.

[0105] When searching the first or second physical time slice, the inverted index mapping relationship can quickly determine the target time series identifier corresponding to the target metadata database, thus using the physical data block identified by the target time series identifier as the target physical data block. This parallel collaboration mechanism between memory and disk leverages the high-speed response of memory to process hot data while utilizing the on-demand loading feature of memory-mapped files to access cold data at low cost, significantly reducing query latency in high-cardinality scenarios.

[0106] S403 decodes the target physical data block to restore the target high-precision floating-point value, and processes the target high-precision floating-point value according to the target aggregation method.

[0107] In this embodiment, the target physical data block is ZSTD decompressed to obtain a variable-length integer stream. Then, LEB128 decoding is used to restore the signed second-order difference value. Next, ZigZag decoding is used to map the unsigned integer back to a signed integer. Subsequently, the inverse accumulation operation of the second-order difference is used to first restore the first-order difference, then restore the original integer value. Finally, the integer is converted back to the target high-precision floating-point value by dividing by a fixed scaling factor. This process ensures that after noise removal and compression, the data can be transparently restored with controllable precision loss.

[0108] Furthermore, the restored high-precision floating-point values ​​are concatenated in memory according to time sequence, and then the target aggregation method is applied for calculation. The final calculation result is then returned to the client. This strategy of transparent restoration followed by aggregation calculation not only ensures the mathematical accuracy of the aggregation result and avoids errors that may occur during direct calculation in the compressed domain, but also significantly reduces network bandwidth consumption, achieving efficient data querying and analysis. Therefore, this design enables the system to maintain extremely low memory usage while providing millisecond-level response speeds and accurate calculation results when facing high-cardinality queries spanning long time periods. This effectively overcomes the technical shortcomings of existing time-series databases, such as high resource consumption and slow response times when processing massive historical data queries.

[0109] Based on the above description, the time-series data storage method provided in this application provides a standardized data source for subsequent processing by acquiring time-series data points containing metadata, timestamps, and high-precision floating-point values. By encoding and segmenting the high-precision floating-point values, the high-information-entropy original data is reduced to compact physical data blocks, significantly reducing storage redundancy. Furthermore, by mapping metadata to time-series identifiers and writing them to the physical time slices to which the timestamps belong, a strong binding between the index structure and physical time is achieved, abandoning the traditional global static index architecture. On this basis, an index rotation mechanism is used to dynamically distinguish between active and inactive time windows, and the corresponding physical time slices are stored differentially in memory or on disk, so that only the currently urgently needed active data resides in memory. This architecture design not only breaks through the performance bottleneck of traditional compression algorithms in handling floating-point noise through concatenated coding, significantly saving disk space and I / O bandwidth, but also completely decouples the system memory usage from the total amount of historical data. This ensures that the system can maintain a stable memory level when facing high cardinality sequence oscillations in a cloud-native environment, effectively avoiding the risk of memory overflow and crashes, and achieving long-term, stable, and low-cost storage of massive time-series data.

[0110] The above describes a time-series data storage method provided by the embodiments of this application. The following describes the apparatus for performing the above-described time-series data storage method.

[0111] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a time-series data storage device provided in an embodiment of this application. Figure 5 As shown in the figure, an embodiment of this application provides a time-series data storage device, comprising:

[0112] The time series data acquisition module 501 is used to acquire the numerical sequence to be written. The numerical sequence consists of time series data points, which include metadata, timestamps, and high-precision floating-point values.

[0113] The time-series data storage module 502 is used to encode and segment high-precision floating-point values ​​to obtain physical data blocks; map metadata to time-series identifiers of physical data blocks, and write the physical data blocks, as well as the inverted mapping relationship between metadata and time-series identifiers, into the physical time slice to which the timestamp belongs; determine active and inactive time windows through index rotation, and store the first physical time slice located in the active time window into memory and the second physical time slice located in the inactive time window into disk.

[0114] In one possible implementation, the time-series data acquisition module 501 is also used for:

[0115] Precision control and truncation of high-precision floating-point values ​​are performed based on metadata to remove random mantissas.

[0116] In one possible implementation, the time-series data storage module 502, used for encoding and segmenting high-precision floating-point values ​​to obtain physical data blocks, is specifically used for:

[0117] The high-precision floating-point value is sequentially converted to integer, subjected to second-order difference operations, positive number mapping, and byte compression to obtain the target value; the target value is then divided into its corresponding physical data blocks according to a preset capacity threshold.

[0118] In one possible implementation, the time-series data storage module 502, used to determine the active and inactive time windows through index rotation, is specifically used for:

[0119] The time window rotation event is triggered by maintaining a state machine in memory; in response to the time window rotation event, a time window is instantiated in memory and rotated to a new current time window, and the current time window before the rotation is downgraded to a new historical time window, and the historical time window before the rotation is marked as read-only; the new current time window and the new historical time window are designated as active time windows, and the historical time window marked as read-only is designated as an inactive time window.

[0120] In one possible implementation, the time-series data storage module 502 is also used for:

[0121] In response to the query command, the system performs distributed parallel retrieval and restoration aggregation on the first physical time shard and the second physical time shard.

[0122] In one possible implementation, the time-series data storage module 502, used for distributed parallel retrieval and reconstruction aggregation of the first physical time slice and the second physical time slice, is specifically used for:

[0123] The system parses and obtains the target timestamp, target metadata, and target aggregation method from the query command; it performs parallel retrieval of memory and disk based on the target timestamp to determine the target time series identifier corresponding to the target metadata from the first physical time slice and the second physical time slice using an inverted mapping relationship, and determines the target physical data block based on the target time series identifier; it decodes the target physical data block to restore the target high-precision floating-point value, and processes the target high-precision floating-point value according to the target aggregation method.

[0124] It should be noted that the detailed functions of each module in the embodiments of this application can be found in the corresponding disclosure of the above-mentioned time-series data storage method embodiments, and will not be repeated here.

[0125] This application also provides an electronic device in its embodiments. See also... Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device in this embodiment may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0126] like Figure 6 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0127] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0128] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the time-series data storage methods provided in this application.

[0129] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the time-series data storage methods provided in this application.

[0130] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0132] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0133] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A time-series data storage method, characterized in that, The time-series data storage method includes: Obtain the numerical sequence to be written, the numerical sequence being composed of time-series data points, the time-series data points including metadata, timestamps, and high-precision floating-point values; The high-precision floating-point values ​​are encoded and segmented to obtain physical data blocks; The metadata is mapped to the time-series identifier of the physical data block, and the physical data block, as well as the inverted mapping relationship between the metadata and the time-series identifier, are written into the physical time slice to which the timestamp belongs; Active and inactive time windows are determined by index rotation, and the first physical time slice located in the active time window is stored in memory, while the second physical time slice located in the inactive time window is stored in disk.

2. The time-series data storage method according to claim 1, characterized in that, Before performing the encoding and block processing of the high-precision floating-point value to obtain physical data blocks, the time-series data storage method further includes: The high-precision floating-point value is subjected to precision control and truncation based on the metadata to remove random mantissas.

3. The time-series data storage method according to claim 1, characterized in that, The process of encoding and segmenting the high-precision floating-point values ​​to obtain physical data blocks includes: The high-precision floating-point value is sequentially subjected to integerization, second-order difference operation, positive number mapping, and byte compression to obtain the target value; The target value is divided into its corresponding physical data blocks according to the preset capacity threshold.

4. The time-series data storage method according to claim 1, characterized in that, The method of determining active and inactive time windows through index rotation includes: The time window rotation event is triggered by maintaining a state machine in memory; In response to the time window rotation event, a time window is instantiated in the memory to rotate into a new current time window, the current time window before rotation is downgraded to a new historical time window, and the historical time window before rotation is marked as read-only. The new current time window and the new historical time window are designated as the active time window, and the historical time window marked as read-only is designated as the inactive time window.

5. The time-series data storage method according to claim 1, characterized in that, The time-series data storage method further includes: In response to the query command, a distributed parallel retrieval and restoration aggregation is performed on the first physical time slice and the second physical time slice.

6. The time-series data storage method according to claim 5, characterized in that, The distributed parallel retrieval and restoration aggregation of the first physical time slice and the second physical time slice includes: The target timestamp, target metadata, and target aggregation method in the query instruction are obtained through parsing. Parallel retrieval of the memory and the disk is performed based on the target timestamp to determine the target time series identifier corresponding to the target metadata from the first physical time slice and the second physical time slice using an inverted mapping relationship, and to determine the target physical data block based on the target time series identifier; The target physical data block is decoded to restore the target high-precision floating-point value, and the target high-precision floating-point value is processed according to the target aggregation method.

7. A time-series data storage device, characterized in that, The time-series data storage device includes: The time-series data acquisition module is used to acquire the numerical sequence to be written. The numerical sequence consists of time-series data points, which include metadata, timestamps, and high-precision floating-point values. The time-series data storage module is used to encode and segment the high-precision floating-point values ​​to obtain physical data blocks; map the metadata to the time-series identifiers of the physical data blocks, and write the physical data blocks, as well as the inverted mapping relationship between the metadata and the time-series identifiers, into the physical time slice to which the timestamp belongs; determine the active time window and the inactive time window through index rotation, and store the first physical time slice located in the active time window into memory and the second physical time slice located in the inactive time window into disk.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the time-series data storage method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the time-series data storage method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the time-series data storage method as described in any one of claims 1 to 6.

Images