Time sequence database query method, system and equipment based on direct calculation of compressed data and medium

Abstract

The invention relates to a time sequence database query method, system and equipment based on direct calculation of compressed data and a medium. The time sequence database query system comprises a client, a query layer and a storage layer, the client is used for sending an SQL query request input by a user to the query layer and receiving a query result returned by the query layer; the query layer is used for reading a target data block from the storage layer according to an SQL query request, constructing the target data block into a compressed state data memory representation by adopting a delay decompression mechanism and a dynamic auxiliary structure management mechanism, and directly executing homomorphic calculation on the generated compressed state data memory representation by utilizing a homomorphic operator; the obtained query result is returned to the client in an uncompressed format; the storage layer is used for storing compressed data blocks, each compressed data block comprises a plurality of compressed data pages, and each compressed data page comprises compressed time sequence data. The method can be widely applied to the technical field of big data processing.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology, specifically relating to a time-series database query method, system, device, and medium based on direct calculation of compressed data. Background Technology

[0002] With the surge in connected devices across IoT, finance, healthcare, and industrial monitoring, real-time data streams are becoming increasingly prevalent, leading to an exponential growth in time-series data. This type of data typically features high-frequency sampling, high redundancy, and strict time alignment, creating an urgent need for efficient storage, management, and querying capabilities. Existing mainstream time-series databases (such as Apache IoTDB, InfluxDB, and OpenTSDB) largely employ compression techniques to save on storage costs and transmission bandwidth. For example, Apache IoTDB, using its proprietary TsFile data format and a two-layer lightweight compression strategy (typically combining run-length encoding (RLE) with the general compression algorithm LZ4), can compress approximately 300 billion data points daily in a railway transportation scenario, reducing the original data volume of over 5TB by about 95%. However, in actual querying, the system still needs to decompress the compressed blocks one by one, significantly increasing query latency and consuming a large amount of memory, which is particularly detrimental to large-scale datasets.

[0003] Homomorphic compression technology can perform calculations directly on compressed data without decompression, significantly reducing query latency, maintaining low memory usage in the compressed state, and providing unified theoretical support for system architecture. Although homomorphic compression technology has been initially applied in text, graph structure, and streaming data scenarios, its research and implementation in time series databases is still lacking. The main challenges are: existing homomorphic compression largely relies on general compression algorithms (such as LZW), failing to be compatible with lightweight algorithms commonly used in time series scenarios (such as RLE, dictionary encoding, Ts_2Diff, etc.); existing homomorphic compression can only cover a small number of basic operations, making it difficult to directly support complex queries dependent on timestamps; and IoT data often involves a large number of null values ​​due to asynchronous sampling, and the compression and rapid maintenance of auxiliary structures such as null value bitmaps have not been adequately addressed.

[0004] In Industrial Internet of Things (IIoT) scenarios, sensors continuously generate data at millisecond or even microsecond frequencies, and this data is typically compressed and stored sequentially in a columnar structure. The business side initiates various queries, such as aggregation, filtering, and transformation, using SQL-like syntax. To perform these operations directly in compressed state to reduce query latency and memory consumption and improve the overall performance of time-series databases, three challenges need to be overcome: First, the lack of a formal query execution framework for compressed time-series data makes it difficult to theoretically verify and implement the "compressed state computation" mechanism in engineering. Second, the range of operators supported is limited; apart from a very small number of simple aggregations that can be achieved using pre-computed summaries, most operations still rely on decompression, disrupting the continuity of the query process and increasing maintenance complexity. Finally, system-level auxiliary structures (such as null bitmaps and lazy deletion markers) are prone to inconsistency during frequent decompression, further amplifying query latency and resource overhead.

[0005] While existing stream processing frameworks attempt to directly compute compressed data within a sliding window, their design prioritizes real-time performance over high compression ratios, and their query semantics are relatively simple, making it difficult to meet the comprehensive needs of time-series databases for large-scale historical data, high compression ratios, and complex timestamp operations. Existing solutions often focus on single-point optimizations (such as metadata caching and specific operator rewriting), lacking a unified theoretical foundation and system architecture, and thus cannot support a complete and efficient query process for compressed time-series data. Therefore, the industry urgently needs a homomorphic compression solution for time-series databases to fully leverage the advantages of HC technology, achieving direct computation on compressed data while maintaining high compression ratios. This would address the shortcomings of existing technologies in terms of theoretical framework, operator coverage, and system-level complexity, thereby comprehensively improving the performance of time-series data management and querying. Summary of the Invention

[0006] To address the aforementioned problems, the purpose of this invention is to provide a method, system, device, and medium for querying time-series databases based on direct computation of compressed data.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a time-series database query system based on direct computation of compressed data, comprising:

[0009] Client-side, query layer, and storage layer;

[0010] The client is used to send the SQL query request entered by the user to the query layer and receive the query results returned by the query layer;

[0011] The query layer is used to read the target compressed data block from the storage layer according to the SQL query request, and then construct the target data block into a compressed data memory representation using a delayed decompression mechanism and a dynamic auxiliary structure management mechanism. The homomorphic operator is used to directly perform homomorphic computation on the generated compressed data memory representation, and the query result is returned to the client in an uncompressed format.

[0012] The storage layer is used to store compressed data blocks, and each compressed data block contains several compressed data pages, and each compressed data page contains compressed time-series data.

[0013] Furthermore, the query layer includes a data structure module and an operator module;

[0014] The data structure module is used to map the target compressed data block in the storage layer into a compressed data memory representation during the loading phase, and maintain two-level positioning indexes: a compressed offset index and a hint index.

[0015] The operator module is used to convert commonly used operators of time series databases into homomorphic operators, and to perform homomorphic computation on the memory representation of compressed data based on compressed offset index and hint index. The resulting query results are returned to the client in uncompressed format.

[0016] Furthermore, the data structure module includes:

[0017] The data reading module is used to read the target compressed data block from the storage layer according to the SQL query request, and write it into the data block cache in a double-compression form;

[0018] The general decompression module is used to sequentially traverse the target compressed data block and use a delayed decompression mechanism to trigger the general decompression of the target compressed data page when it is hit, so as to obtain lightweight compressed data.

[0019] The compressed data memory representation construction module is used to recover null values ​​and deletion information in-situ within lightweight compressed data by relying on dynamic auxiliary structure management, obtain the compressed data memory representation, and construct the compressed offset index and hint index.

[0020] Furthermore, the compressed state data memory representation construction module includes:

[0021] The data partitioning module is used to divide lightweight compressed data into several logical compression units, each of which corresponds to a continuous or related timestamp sequence.

[0022] The dynamic encoding module is used to perform dynamic encoding operations on each logical compression unit based on the lightweight compression algorithm corresponding to the lightweight compressed data, and to obtain several compressed blocks by adopting a dynamic auxiliary structure management mechanism.

[0023] The compressed block encapsulation module is used to store each compressed block as an independent encoded data structure and encapsulate it as a unified list-style object;

[0024] The index construction module is used to organize the encapsulated compressed blocks into an array to form a compressed column, and to construct a compressed offset index that matches the compressed column. At the same time, it initializes the prompt index. The compressed offset index is used to establish a mapping relationship between the logical position and the compressed segment for each compressed block in the compressed column; the prompt index is used to record the position of the last access.

[0025] Furthermore, in the dynamic encoding module, a dynamic auxiliary structure management mechanism is used to perform dynamic encoding operations on each logical compression unit, including:

[0026] In unaligned data, each encoded block in the logical compression unit is directly traversed, and the marked values ​​are skipped according to the deletion list and the running length is adjusted to achieve lazy deletion processing in the compressed state.

[0027] In the aligned data, based on the bitmap and deletion information, the null value region is aligned and divided into multiple null segments according to the compression mode; for continuous null value segments, they are directly encoded as null values ​​and merged into the compressed stream; for non-continuous null value segments, only the affected segments are restored to the original encoding format for subsequent query parsing.

[0028] Furthermore, the operator module includes:

[0029] The retrieval and positioning module is used to retrieve and locate compressed blocks in the compressed column based on the compressed offset index to determine the target location;

[0030] The homomorphic operator execution module is used to construct homomorphic operators and perform homomorphic computation on the compressed data memory representation based on the compressed offset index and the hint index. The resulting query results are returned to the client in uncompressed format, and the hint index is updated at the same time.

[0031] Furthermore, in the retrieval and positioning module, the compressed blocks in the compressed column are retrieved and located according to the compressed offset index to determine the target location, including:

[0032] When a target location request is received, the compressed block number where the target location is located is determined by the compressed offset index, and the corresponding compressed block content is accessed accordingly.

[0033] Update the prompt index to the number of the most recently successfully hit compressed block, and use it as the starting reference position for subsequent accesses;

[0034] When a new target location request arrives, determine whether the compressed block pointed to by the hint index covers the target location of the current target location request:

[0035] If a match is found, the data retrieval is completed directly within that compressed block;

[0036] If a match is not found, the system will choose to continue scanning forward or search for the compressed offset index again from the starting reference position, depending on the positional relationship between the target location and the current suggested index.

[0037] Secondly, the present invention provides a time-series database query method based on direct calculation of compressed data, comprising the following steps:

[0038] The client receives the SQL query request input by the user and sends it to the query layer;

[0039] After the query layer reads the target compressed data block from the storage layer according to the SQL query request, it uses a delayed decompression strategy and a dynamic auxiliary structure management strategy to construct a compressed data memory representation of the target compressed data block, and uses homomorphic operators to directly perform homomorphic computation on the generated compressed data memory representation. The resulting query result is returned to the client in an uncompressed format.

[0040] Thirdly, the present invention provides a computer-readable storage medium for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the method.

[0041] Fourthly, the present invention provides a computing device comprising: one or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, the one or more programs including instructions for performing the method.

[0042] The present invention has the following advantages due to the adoption of the above technical solutions:

[0043] 1. This invention systematically summarizes the complete theoretical system for establishing homomorphic compression in time series databases, connects the entire link of "theory-algorithm-system", and realizes the systematization and standardization of compressed state query.

[0044] 2. The modular compressed temporal data intermediate representation provided by this invention supports various lightweight compression algorithms and multiple types of temporal operators such as filtering, aggregation, and windowing. Compression methods and query operations can be flexibly expanded according to business needs. Through the integration of temporal homomorphic operators, complex temporal queries can be completed without full decompression, significantly reducing query latency and runtime memory usage.

[0045] 3. The optimizations provided by this invention include mechanisms such as dynamic auxiliary coding, lazy deletion, and delayed decompression, ensuring high throughput and low latency performance even as the data scale continues to grow.

[0046] 4. This invention provides a homomorphic query system for time-series data, which is compatible with existing query interfaces, allowing users to achieve performance improvements without modifying the upper-layer application.

[0047] In summary, this invention can be widely applied in the field of big data processing technology, and is particularly suitable for application scenarios that require real-time analysis of massive time series data, such as industrial IoT, financial risk control, smart healthcare, and intelligent manufacturing. Attached Figure Description

[0048] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings:

[0049] Figure 1 This is a schematic diagram of the architecture design of the time-series database query system based on direct computation of compressed data provided by the present invention;

[0050] Figure 2 This is a schematic diagram of the design of a time-series database query system based on direct computation of compressed data provided in an embodiment of the present invention. In the figure, ① represents the import of compressed data blocks from the disk; ② represents the construction of compressed data representation, including delayed decompression and dynamic auxiliary structure management; ③ represents the execution of homomorphic operators, which interact efficiently with the compressed data representation through compressed offset index and hint index.

[0051] Figure 3 (a) and Figure 3 (b) is a schematic diagram of the conventional method provided in the embodiments of the present invention and the sequence scanning method based on delayed decompression of the present invention;

[0052] Figure 4 This is a flowchart illustrating the time-series database query method based on direct calculation of compressed data provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0054] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0055] In some embodiments of the present invention, a time-series database query system based on direct computation of compressed data is provided, which has the following core objectives: First, a theoretical model for homomorphic compression of time-series data is established to achieve an integrated design from the operator layer to the system layer, meeting the comprehensive processing requirements of "high throughput and low latency" for large-scale time-series data; second, through a unified and modular data structure and system-level optimization measures, runtime memory usage is reduced and system scalability is improved; finally, while maintaining a high compression ratio, query operations are performed directly on compressed time-series data to significantly reduce query latency.

[0056] Correspondingly, in other embodiments of the present invention, a method, apparatus, and medium for querying time-series databases based on direct calculation of compressed data are provided.

[0057] To better understand this invention, the theoretical basis of this invention will first be introduced below.

[0058] The homomorphic compression theory proposed in this invention aims to summarize the properties, characteristics, and applicable scope of direct computation techniques for compressed data. Direct computation of compressed data seeks to achieve the same operational results for compressed and uncompressed data without requiring decompression and recompression. A similar concept exists in abstract algebra, namely a homomorphic mapping, which is defined as a mapping that preserves all algebraic structures between the domain and range of an algebraic set. Based on this similarity, this invention proposes the concept of homomorphic compression, treating homomorphism as a novel compression property.

[0059] Applying homomorphic mapping to data compression requires first transforming the data into an algebraic structure. This invention proposes a homomorphic algebraic system for time series data and its query method. Uncompressed and compressed time series are modeled as algebraic systems (Si and Si, respectively). u ,Π) and (S c ,Θ), where S u With S c The data field represents the time series; both Π and Θ contain six basic operators F, J, A, G, E, and S, corresponding to filtering, timestamp concatenation, aggregation, sliding window grouping, expression, and slicing, respectively, forming a one-to-one mapping relationship between them. This invention treats the compression algorithm as a mapping. If for any operator op∈Π and its corresponding op′∈Θ, the following always holds true: This invention will Defined as a homomorphic compression mapping.

[0060] This invention provides a definition for homomorphic time-series database queries based on homomorphic compression mapping. The time-series database query process can be abstracted as a "decompression—restore—query" process, formally represented as follows: When it exists This query is called a homomorphic query; if the query process does not involve any operation op′(u) on the uncompressed segment u, it is a direct homomorphic query; otherwise, it is a partial homomorphic query. If the compression side restores the operation R... c The cost is lower than that of the uncompressed side R u That is, Cost(R) c (c)) <Cost(R u (u)), then R c Defined as a valid recovery; if Cost(Q) is further satisfied... c (c)) <Cost(Q u (u)), then Q c A valid homomorphic query is defined as one that does not meet the criteria for a valid homomorphic query; otherwise, it is considered an invalid homomorphic query.

[0061] Example 1

[0062] like Figure 1 , Figure 2 As shown, this invention provides a time-series database query system based on direct computation of compressed data, which includes:

[0063] The client is used to send the SQL query request entered by the user to the query layer and receive the query results returned by the query layer.

[0064] The query layer is used to read the target compressed data block from the storage layer according to the SQL query request, and then use a delayed decompression strategy and a dynamic auxiliary structure management strategy to construct a compressed data memory representation of the target compressed data block. It also uses homomorphic operators to directly perform homomorphic computation on the generated compressed data memory representation, and returns the query result to the client in an uncompressed format.

[0065] The storage layer is used to store compressed data blocks. Each compressed data block contains several compressed data pages, and each compressed data page contains lightly compressed time-series data.

[0066] Furthermore, the query layer includes a data structure module and an operator module. These two modules work together with the compressed data pages in the storage layer and the SQL query requests from the client to achieve direct calculation and efficient transmission of compressed time-series data.

[0067] To address the data transmission and retrieval overhead issues in homomorphic queries, this embodiment also provides two collaborative optimization techniques: dynamic auxiliary structure management and a delayed decompression mechanism. These two techniques work together to efficiently process null values, deletion markers, and double-layer compressed data pages while maintaining the compressed data structure intact, thereby improving system query performance and reducing resource consumption.

[0068] Specifically, the data structure module is the foundation of the time-series database query system. It is used to map the target compressed data block in the storage layer into a compressed data memory representation by using dynamic auxiliary structure management and delayed decompression mechanism during the loading phase, and maintain two-level positioning indexes: compressed offset index and hint index.

[0069] The operator module converts commonly used time-series database operators such as filtering, joining, aggregation, transformation, and windowing into homomorphic operators. It then performs homomorphic computations on the compressed data memory representation based on compressed offset and hint indexes, returning the query results to the client in uncompressed format. In this embodiment, all homomorphic operators use compressed data memory representation as unified input and output. All intermediate results are passed between operators in compressed data memory representation until the final result is obtained and returned to the client in uncompressed format. The entire process can be completed without explicit decompression, thus meeting the requirements for effective homomorphic queries.

[0070] Furthermore, the data structure module includes:

[0071] The data reading module is used to read the target compressed data block from the storage layer according to the SQL query request and write it into the data block cache in a double-compression form;

[0072] The general decompression module is used to sequentially traverse the target compressed data block and use a delayed decompression mechanism to trigger the general decompression of the target compressed data page when it is hit, so as to obtain lightweight compressed data.

[0073] The compressed data memory representation construction module is used to recover null values ​​and deletion information in-situ within lightweight compressed data by relying on dynamic auxiliary structure management, obtain the compressed data memory representation, and construct the compressed offset index and hint index.

[0074] Furthermore, in the general decompression module, this embodiment adopts a delayed decompression mechanism, which postpones the general decompression operation until the target page is actually accessed.

[0075] like Figure 3The figure illustrates the processing differences between traditional sequence scanning methods and the sequence scanning method based on delayed decompression strategy proposed in this invention when accessing compressed time-series data files. By comparing the data flow and decompression process of the two methods, the figure visually reflects the advantages of the delayed decompression strategy in reducing unnecessary decompression and improving access efficiency.

[0076] In existing methods, time-series databases first use lightweight compression algorithms (such as RLE or Gorilla) for page-level compression, and then uniformly apply a general compression algorithm (such as LZ4) to compress the entire compressed data block. During the query process, even if only a portion of the data is accessed, the entire compressed data block needs to be decompressed, resulting in a significant waste of unnecessary computational resources.

[0077] like Figure 3 (a) illustrates the traditional sequence scanning method. In this method, when the system needs to access a page of data within a compressed data block, even if only a small portion is accessed, the entire compressed page must first be loaded from disk into memory and decompressed. For example, when the system accesses "page 1" in block group 1, it must first decompress the entire compressed block containing "pages 1–5," and then use lightweight decoding (such as RLE decoding) to convert the logical values ​​(e.g., L0:4, L1:9) into the actual raw values ​​(V0:3, V1:8) to obtain the value required by the user. Since the entire compressed block needs to be decompressed for each access, this approach incurs a significant CPU and memory burden, especially when only a portion of the data is accessed, resulting in resource waste.

[0078] Therefore, this invention delays the decompression operation of general compression until the sequence scan phase. Specifically, when reading compressed data blocks, the system retains the double-compression form of the compressed data and only creates a compressed data representation containing metadata. General decompression of the target page is triggered only when the scan cursor is positioned on the target page, and the decompression result is loaded as vector data, avoiding lightweight decoding. For pages that are not accessed, their compression state remains unchanged throughout the entire query process. Figure 3 (b) illustrates the delayed decompression-based sequence scanning method proposed in this invention. Unlike traditional methods, this approach delays the decompression operation of general compression (such as LZ4) during sequence scanning, triggering decompression only when specific page data needs to be read. The figure shows that when the system accesses "page 1," only the compressed data page containing that page is decompressed, avoiding the decompression of other irrelevant pages, thus significantly reducing the decompression workload.

[0079] Furthermore, this method utilizes offset indexes and lightweight compressed data structures within compressed data pages to support page-by-page access and in-page positioning, maintaining decoding accuracy and low latency. Ultimately, the system also reconstructs the original time-series data through decoding operations, but only for the truly necessary portions, achieving minimal data decompression. In addition, to ensure compatibility with upper-level query operators that do not support compressed-state computation, the compressed-state data representation provided by this invention also supports exposing decompressed data views on demand, improving computational flexibility while maintaining interface transparency.

[0080] Furthermore, the compressed data memory representation construction module includes:

[0081] The data partitioning module is used to divide lightweight compressed data into several logical compression units, each of which corresponds to a continuous or related timestamp sequence.

[0082] The dynamic encoding module is used to perform dynamic encoding operations on each logical compression unit based on the lightweight compression algorithm corresponding to the lightweight compressed data, using a dynamic auxiliary structure management mechanism. That is, it restores null values ​​and deleted information in-situ within the lightweight compressed data to obtain several compressed blocks.

[0083] The compressed block encapsulation module is used to store each compressed block as an independent encoded data structure and encapsulate it into a unified list-style object to facilitate subsequent memory management and operation compatibility.

[0084] The index construction module is used to organize compressed blocks into compressed columns in the form of arrays, construct compressed offset indexes that correspond to the compressed columns, and initialize the hint indexes. The compressed offset indexes are used to establish a mapping relationship between the logical position and the compressed segment for each compressed block in the compressed column, so as to provide offset resolution capability. The hint indexes are used to record the temporal locality information of the lightweight compressed data (i.e., record the position of the last access), thereby ensuring the directness of the operator module to the lightweight compressed data.

[0085] Furthermore, in the dynamic encoding module, the present invention adopts a dynamic auxiliary structure management mechanism to realize online reorganization of null values ​​and deletion markers in a dynamic encoding manner while maintaining the compressed state.

[0086] This invention supports two time series data formats: unaligned format (each column has an independent time column) and aligned format (all columns share a single time column). The aligned format typically uses a bitmap to identify null values ​​and a delete list to record logical deletion positions, thus managing and deleting null values ​​without physically modifying the stored content. While this compact layout helps save storage space and improve I / O efficiency, it introduces data access complexity during sequence scanning. Specifically, null values ​​need to be backfilled to their original positions based on the bitmap, and deletion positions need to be compared in real time, disrupting the structural continuity of the compressed data and affecting the efficiency of homomorphic queries.

[0087] To address the aforementioned issues, this invention introduces a dynamic encoding strategy that enables skipping null value insertions and deletions within the compressed data structure without decompressing the entire dataset. In unaligned data, the system directly traverses each encoded block, such as using run-length encoding (RLE), skipping marked values ​​according to the deletion list and adjusting the run length to achieve lazy deletion processing in compressed mode. In aligned data, the system, based on the bitmap and deletion information, aligns and segments the null value region into multiple empty segments according to the RLE mode. For continuous null value segments, the system directly encodes them as null values ​​and merges them into the compressed stream, maintaining structural continuity. For non-continuous null value segments, since they disrupt the compressed structure, the system only restores the affected segments to their original encoding format for subsequent query parsing. Throughout the process, the re-encoding result is written back to the buffer only at the end of the scan; unaccessed data blocks remain in their original compressed state, minimizing intervention during the query process.

[0088] By integrating dynamic auxiliary structure management and delayed compression optimization mechanisms, the time-series database compressed-state query system of this invention achieves a direct query path in the compressed state at both the logical and physical levels. Dynamic auxiliary structure management avoids damage to the compressed structure, while delayed decompression of compressed data blocks minimizes resource consumption during data decompression and transmission.

[0089] Furthermore, such as Figure 4 As shown, the operator module includes:

[0090] The retrieval and positioning module is used to retrieve and locate compressed blocks in the compressed column based on the compressed offset index to determine the target location;

[0091] The homomorphic operator execution module is used to convert commonly used time-series database operators such as filtering, joining, aggregation, transformation, and windowing into homomorphic operators, and to perform homomorphic calculations on the compressed data memory representation based on the compressed offset index and hint index. The resulting query results are returned to the client in uncompressed format.

[0092] Furthermore, in the retrieval and location module, this invention proposes an efficient data access mechanism that combines compressed offset indexes and hint indexes to address the problem of difficult random access to time-series data in memory under compressed conditions. This mechanism achieves rapid location and retrieval of compressed data without the need for decompression by constructing a mapping relationship between logical locations and compressed physical blocks, combined with the locality characteristics of access history.

[0093] During data access, when a target location request is received, the system first determines the compressed block number where the target location is located using the compressed offset index, and then accesses the corresponding compressed block content accordingly. The compressed offset index not only supports fast jump access at the compressed block level, but also allows precise location of the target compressed data item within the compressed block through relative position calculation, avoiding linear scan operations of decompressed columns, effectively improving data access efficiency, and is especially suitable for structured compression formats such as Run-Length Encoding (RLE).

[0094] To further improve the performance of compressed offset indexes in continuous query or sliding window access scenarios, this invention provides a hint index as an optimization method. The hint index stores the number of the most recently successfully hit compressed data block in integer form, serving as the starting reference position for subsequent accesses. When a new request arrives, the system first determines whether the compressed data block indicated by the hint index covers the target position of the current request. If it hits, the system directly completes the data retrieval within that compressed data block; if it misses, it chooses to continue scanning forward or re-search for the compressed offset index from the starting reference position based on the positional relationship between the target position and the current hint index. Once the target position is successfully located, the corresponding hint index is updated for reuse in subsequent accesses.

[0095] This method achieves structured location of compressed state data through compressed offset index and enables fast query initiation with state awareness through hint index, thereby significantly reducing query latency for random access while maintaining the same compression ratio.

[0096] At the implementation level, a generic in-memory data representation object is defined for the upper-layer system. The compressed data representation object inherits from the generic in-memory data representation interface, and a standardized set of data access, slicing, deserialization, and reversal methods are defined to ensure that compressed data can be directly processed by the upper-layer query engine while maintaining encapsulation, avoiding decompression overhead. Write and serialization operations are implemented through auxiliary builder and encoder classes, respectively used to construct compressed blocks and perform in-memory and persistent conversions.

[0097] It is worth noting that this invention features high modularity and abstraction; the specific logic of the compression algorithm is transparent to upper-level operations, thus achieving a unified interface design that is algorithm-independent. Furthermore, since all operations are performed in the compression state, performance losses during decompression and recoding are significantly reduced, resulting in high engineering portability and system integration efficiency.

[0098] Furthermore, the homomorphic operator execution module provides operators that support direct computation in compressed states to achieve efficient homomorphic queries in time series databases. By introducing a core set of structured operators and combining them with compression formats (such as RLE and dictionary encoding), in-situ data processing without decompression is achieved, significantly reducing intermediate computational redundancy and transmission costs. It should be noted that this embodiment is applicable to any lightweight compression algorithm and operator suitable for time series data. For ease of explanation, this embodiment uses six core operator sets and the RLE algorithm as examples, but this is not intended to limit the invention.

[0099] This invention incorporates four coding primitives T of a lightweight time series compression algorithm. delta T repeat T pack T dic , representing difference encoding, repetition encoding, bit packing encoding, and dictionary encoding, respectively. Different lightweight compression algorithms are combined... Implementation, for example, Gorilla by T delta T reprat With T pack constitute.

[0100] In a specific implementation of this invention, six core homomorphic operator sets are defined, including filtering, joining, aggregation, sliding window grouping, expression, and slicing operations. When a query consists of the above operators, the system maps it to a fully homomorphic query, allowing each operation to be completed within the compressed domain; if other operators are included, it is converted to a partially homomorphic query, allowing mixed compressed and decompressed processing. The degree of support for the above-mentioned encoded primitives by the six operators provided by this invention is illustrated in Table 1, where T... repeat T pack T dic Direct homomorphic support for all operators, T delta Conditional support is only available for aggregation and expressions, relying on the summation property.

[0101] Table 1 Operator-Encoding Component Matrix

[0102]

[0103] Filtering operators. Filtering operators support unary predicates, binary comparisons, and regular expressions. For example, for RLE encoding, the system uses a block-by-block traversal mode, calculating each duplicate value only once to avoid redundant checks; this operation is typically pushed into the execution phase during the construction of the compressed data's in-memory representation to reduce memory construction overhead.

[0104] The join operator employs a columnar processing approach, performing matching on each column of data blocks based on timestamp alignment principles. In RLE-encoded columns, the system iterates through the data segments by `run`, directly writing uniform values ​​to constant segments and performing value-level extraction and writing for non-constant segments. This method optimizes the complexity from O(NC) to O(RC) when the number of selected rows far exceeds the number of `RLErun` operations.

[0105] Aggregation operators. Aggregation operators (such as summation, mean, variance, and extrema) incrementally update compressed data using temporary accumulators, eliminating the need for iterative processing of each data point. For example, for RLE-encoded data, the system updates the cumulative state variables n, v, and m based solely on the pattern tuple (n′, v′) when traversing each data block. 2 It enables online calculation of mean and variance. Figure 3 The incremental aggregation calculation process under this method is demonstrated, which effectively avoids computational redundancy after expanding the complete data.

[0106] Expression operators. Expression operators cover arithmetic, logical, and comparison operations. In RLE format, operations are processed on a run basis, and each pattern value is evaluated only once.

[0107] Sliding window grouping operator. This operator is typically used in conjunction with aggregation to divide a compressed column into multiple blocks based on a time window, and then performs aggregation calculations directly within each block without decompression. Because the operation only depends on the time index and the compressed block boundaries, it is suitable for large-scale sliding window analysis scenarios.

[0108] Slicing Operator. The slicing operator supports sequential scanning and batch extraction of long historical sequences. The system quickly locates the starting offset based on the hint index and identifies the compressed segment boundary by combining it with the compressed offset index, thereby accurately extracting sub-region data. During slicing, the hint index is updated to a new location index for reuse in subsequent requests, significantly reducing repeated traversal.

[0109] In summary, the time-series database query system based on direct computation of compressed data proposed in this invention has the following beneficial effects:

[0110] (1) A theoretical framework for homomorphic compression of time series data is proposed. This framework formalizes direct computation into an algebraic homomorphic problem, models timestamp-related operators and complex time series management processes, and provides selection criteria for compression methods and conditions for operator feasibility.

[0111] (2) A unified and modular compressed data memory representation is proposed. This compressed data memory representation supports a variety of lightweight lossless compression algorithms. While maintaining the data compression state, it enables the execution of key time-series database operators such as filtering, aggregation, and sliding window calculation without decompression, and can seamlessly interact with the storage layer and query engine.

[0112] (3) A homomorphic compressed query framework was constructed. This framework implements a homomorphic query operator link on top of a unified compressed temporal data memory representation. Combined with dynamic auxiliary encoding, lazy deletion management and delayed decompression strategies, it efficiently maintains auxiliary structures such as null bitmaps, thereby significantly reducing data access overhead.

[0113] Example 2

[0114] Based on the time-series database query system based on direct calculation of compressed data provided in Embodiment 1, this embodiment provides a time-series database query method based on direct calculation of compressed data, which includes the following steps:

[0115] 1) The client receives the SQL query request input by the user and sends it to the query layer;

[0116] 2) After the query layer reads the target compressed data block from the storage layer according to the SQL query request, it uses a delayed decompression strategy and a dynamic auxiliary structure management strategy to construct a compressed data memory representation of the target compressed data block, and uses homomorphic operators to directly perform homomorphic calculations on the generated compressed data memory representation. The resulting query results are returned to the client in uncompressed format.

[0117] Example 3

[0118] This embodiment provides a processing device corresponding to the time-series database query method based on direct calculation of compressed data provided in Embodiment 2. The processing device can be a client-side processing device, such as a mobile phone, laptop, tablet computer, desktop computer, etc., to execute the method of Embodiment 2.

[0119] The processing device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus to enable communication between them. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the time-series database query method based on direct calculation of compressed data provided in Embodiment 2.

[0120] Preferably, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0121] Preferably, the processor can be any type of general-purpose processor such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation herein.

[0122] Example 4

[0123] The time-series database query method based on direct calculation of compressed data in Embodiment 2 can be specifically implemented as a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the time-series database query method based on direct calculation of compressed data described in Embodiment 2 are loaded.

[0124] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0125] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A time-series database query system based on direct computation of compressed data, characterized in that, include: Client-side, query layer, and storage layer; The client is used to send the SQL query request entered by the user to the query layer and receive the query results returned by the query layer; The query layer is used to read the target compressed data block from the storage layer according to the SQL query request, and then construct the target data block into a compressed data memory representation using a delayed decompression mechanism and a dynamic auxiliary structure management mechanism. The homomorphic operator is used to directly perform homomorphic computation on the generated compressed data memory representation, and the query result is returned to the client in an uncompressed format. The storage layer is used to store compressed data blocks, and each compressed data block contains several compressed data pages, and each compressed data page contains compressed time-series data.

2. The time-series database query system based on direct computation of compressed data as described in claim 1, characterized in that, The query layer includes a data structure module and an operator module; The data structure module is used to map the target compressed data block in the storage layer into a compressed data memory representation during the loading phase, and maintain two-level positioning indexes: a compressed offset index and a hint index. The operator module is used to convert commonly used operators of time series databases into homomorphic operators, and to perform homomorphic computation on the memory representation of compressed data based on compressed offset index and hint index. The resulting query results are returned to the client in uncompressed format.

3. The time-series database query system based on direct computation of compressed data as described in claim 2, characterized in that, The data structure module includes: The data reading module is used to read the target compressed data block from the storage layer according to the SQL query request, and write it into the data block cache in a double-compression form; The general decompression module is used to sequentially traverse the target compressed data block and use a delayed decompression mechanism to trigger the general decompression of the target compressed data page when it is hit, so as to obtain lightweight compressed data. The compressed data memory representation construction module is used to recover null values ​​and deletion information in-situ within lightweight compressed data by relying on dynamic auxiliary structure management, obtain the compressed data memory representation, and construct the compressed offset index and hint index.

4. The time-series database query system based on direct computation of compressed data as described in claim 3, characterized in that, The compressed state data memory representation construction module includes: The data partitioning module is used to divide lightweight compressed data into several logical compression units, each of which corresponds to a continuous or related timestamp sequence. The dynamic encoding module is used to perform dynamic encoding operations on each logical compression unit based on the lightweight compression algorithm corresponding to the lightweight compressed data, and to obtain several compressed blocks by adopting a dynamic auxiliary structure management mechanism. The compressed block encapsulation module is used to store each compressed block as an independent encoded data structure and encapsulate it as a unified list-style object; The index construction module is used to organize the encapsulated compressed blocks into an array to form a compressed column, and to construct a compressed offset index that matches the compressed column. At the same time, it initializes the prompt index. The compressed offset index is used to establish a mapping relationship between the logical position and the compressed segment for each compressed block in the compressed column; the prompt index is used to record the position of the last access.

5. A time-series database query system based on direct computation of compressed data as described in claim 4, characterized in that, In the dynamic encoding module, a dynamic auxiliary structure management mechanism is used to perform dynamic encoding operations on each logical compression unit, including: In unaligned data, each encoded block in the logical compression unit is directly traversed, and the marked values ​​are skipped according to the deletion list and the running length is adjusted to achieve lazy deletion processing in the compressed state. In the aligned data, based on the bitmap and deletion information, the null value region is aligned and divided into multiple null value segments according to the compression mode; for continuous null value segments, they are directly encoded as null values ​​and merged into the compressed stream; for non-continuous null value segments, only the affected segments are restored to the original encoding format for subsequent query parsing.

6. A time-series database query system based on direct computation of compressed data as described in claim 2, characterized in that, The operator module includes: The retrieval and positioning module is used to retrieve and locate compressed blocks in the compressed column based on the compressed offset index to determine the target location; The homomorphic operator execution module is used to construct homomorphic operators and perform homomorphic computation on the compressed data memory representation based on the compressed offset index and the hint index. The resulting query results are returned to the client in uncompressed format, and the hint index is updated at the same time.

7. A time-series database query system based on direct computation of compressed data as described in claim 6, characterized in that, The retrieval and positioning module retrieves and positions compressed blocks in the compressed column based on the compressed offset index to determine the target location, including: When a target location request is received, the compressed block number where the target location is located is determined by the compressed offset index, and the corresponding compressed block content is accessed accordingly. Update the prompt index to the number of the most recently successfully hit compressed block, and use it as the starting reference position for subsequent accesses; When a new target location request arrives, determine whether the compressed block pointed to by the hint index covers the target location of the current target location request: If a match is found, the data retrieval is completed directly within that compressed block; If a match is not found, the system will choose to continue scanning forward or search for the compressed offset index again from the starting reference position, depending on the positional relationship between the target location and the current suggested index.

8. A time-series database query method based on direct computation of compressed data, characterized in that, Includes the following steps: The client receives the SQL query request input by the user and sends it to the query layer; After the query layer reads the target compressed data block from the storage layer according to the SQL query request, it uses a delayed decompression strategy and a dynamic auxiliary structure management strategy to construct a compressed data memory representation of the target compressed data block, and uses homomorphic operators to directly perform homomorphic computation on the generated compressed data memory representation. The resulting query result is returned to the client in an uncompressed format.

9. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the method of claim 8.

10. A computing device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of claim 8.

Images