A time series database query method, system, device and medium based on direct calculation of compressed data

Abstract

The application relates to a time series database query method, system, device and medium based on direct calculation of compressed data, which comprises a client, a query layer and a storage layer; the client is used for sending an SQL query request input by a user to the query layer and receiving a query result returned by the query layer; the query layer is used for constructing a target data block into a compressed state data memory representation by adopting a delayed decompression mechanism and a dynamic auxiliary structure management mechanism after reading the target data block from the storage layer according to the SQL query request, directly performing homomorphic calculation on the generated compressed state data memory representation by using a homomorphic operator, and returning the obtained query result to the client in an uncompressed format; and the storage layer is used for storing compressed data blocks, and each compressed data block contains a plurality of compressed data pages, and each compressed data page contains compressed time series data. The application can be widely applied to the technical field of big data processing.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of big data processing, and particularly relates to a time series database query method, system, device and medium based on compressed data direct calculation. BACKGROUND

[0002] With the explosive growth of connected devices in the Internet of Things, finance, healthcare, and industrial monitoring scenarios, real-time data streams are increasingly popular, and time series data is growing exponentially. Such data usually has characteristics such as high-frequency sampling, high redundancy, and strict time alignment, which determines the urgent need for efficient storage, management, and query capabilities. Existing mainstream time series databases (such as Apache IoTDB, InfluxDB, OpenTSDB, etc.) mostly use compression methods to save storage costs and transmission bandwidth. For example, Apache IoTDB uses its self-developed data format TsFile double-layer lightweight compression strategy (typical algorithm is RLE combined with general compression algorithm LZ4) to compress about 300 billion data points per day in the railway transportation scenario, which can compress the original data volume of more than 5TB by about 95%. However, in the actual query process, the system still needs to decompress the compressed blocks one by one, which significantly increases the query delay and occupies a large amount of memory, which is particularly disadvantageous for large-scale data sets.

[0003] Homomorphic compression technology can directly perform calculations on compressed data without decompression, which can significantly reduce query delay, maintain low memory occupation in the compressed state, and provide unified theoretical support for system architecture. Although homomorphic compression technology has been preliminarily applied in text, graph structure, and stream data scenarios, its research and landing in time series databases are still in the blank. The main difficulties are: existing homomorphic compression mostly relies on general compression algorithms (such as LZW), and cannot be compatible with lightweight algorithms commonly used in time series scenarios (such as RLE, dictionary encoding, Ts_2Diff, etc.); existing homomorphic compression can only cover a small number of basic operations, and it is difficult to directly support complex queries that rely on timestamps; Internet of Things data often contains a large number of null values due to asynchronous sampling, and the compression and fast maintenance of auxiliary structures such as null value bitmap have not been fully solved.

[0004] In the industrial Internet of Things scenario, sensors continuously generate data at a frequency of milliseconds or even microseconds, and usually store them in a columnar structure in sequence. Business side initiates various queries such as aggregation, filtering and transformation with SQL-like syntax. If the above operations are to be completed directly in the compressed state to reduce query delay and memory occupation and improve the performance of time series database as a whole, three challenges need to be overcome: first, there is a lack of formal query execution framework for compressed time series data, making it difficult to verify the theory and engineering of the "compressed state computing" mechanism; second, the range of operator support is limited, and most operations still rely on decompression, which destroys the continuity of the query process and increases the maintenance complexity; finally, the system-level auxiliary structure (such as null bitmap, lazy deletion marker, etc.) is easy to lose consistency in the process of frequent decompression, further amplifying query delay and resource overhead.

[0005] Although the existing stream processing framework attempts to directly calculate compressed data within a sliding window, its design focus is on real-time rather than high compression ratio, and the query semantics are relatively simple, which cannot meet the comprehensive needs of time series database for large-scale historical data, high compression rate and complex timestamp operation. The existing solutions focus on single-point optimization (such as metadata caching, specific operator rewriting), lack a unified theoretical basis and system architecture, and cannot support complete and efficient query process for compressed time series data. Therefore, the industry urgently needs a homomorphic compression solution for time series database to fully utilize the advantages of HC technology, realize direct calculation on compressed data while maintaining high compression rate, and solve the deficiencies of existing technology in theoretical framework, operator coverage and system-level complexity, thereby improving the performance of time series data management and query. SUMMARY

[0006] To solve the above problems, the purpose of the present application is to provide a time series database query method, system, device and medium based on direct calculation of compressed data.

[0007] To achieve the above purpose, the present application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a time series database query system based on direct calculation of compressed data, comprising:

[0009] Client, query layer and storage layer;

[0010] The client is used to send the SQL query request input by the user to the query layer, and receive the query result returned by the query layer;

[0011] The query layer is configured to read target compressed data blocks from the storage layer according to a SQL query request, and to construct the target data blocks into a compressed-state data memory representation by using a lazy decompression mechanism and a dynamic auxiliary structure management mechanism, and to directly perform homomorphic computation on the generated compressed-state data memory representation by using a homomorphic operator, and to return the query result in an uncompressed format to the client.

[0012] The storage layer is configured to store compressed data blocks, and each compressed data block includes a plurality of compressed data pages, and each compressed data page includes compressed time-series data.

[0013] Further, the query layer includes a data structure module and an operator module.

[0014] The data structure module is configured to map target compressed data blocks in the storage layer into a compressed-state data memory representation in a loading stage, and to maintain two-level positioning indexes of a compressed offset index and a hint index.

[0015] The operator module is configured to convert common operators of a time-series database into homomorphic operators, and to perform homomorphic computation on the compressed-state data memory representation based on the compressed offset index and the hint index, and to return the query result in an uncompressed format to the client.

[0016] Further, the data structure module includes:

[0017] A data reading module is configured to read target compressed data blocks from the storage layer according to a SQL query request, and to write the target compressed data blocks into a data block cache in a double-layer compression form.

[0018] A general decompression module is configured to sequentially traverse the target compressed data blocks, and to trigger general decompression of a target compressed data page by using a lazy decompression mechanism when the target compressed data page is hit, to obtain lightweight compressed data.

[0019] A compressed-state data memory representation construction module is configured to restore null values and deletion information in the lightweight compressed data in place by using a dynamic auxiliary structure management mechanism, to obtain a compressed-state data memory representation, and to construct a compressed offset index and a hint index.

[0020] Further, the compressed-state data memory representation construction module includes:

[0021] A data division module is configured to divide the lightweight compressed data into a plurality of logical compression units, and each logical compression unit corresponds to a sequence of continuous or related time stamps.

[0022] A dynamic encoding module is configured to perform dynamic encoding operation on each logical compression unit by using a dynamic auxiliary structure management mechanism based on a lightweight compression algorithm corresponding to the lightweight compressed data, to obtain a plurality of compressed blocks.

[0023] The compressed block packaging module is configured to store each compressed block as an independent encoding data structure and package the compressed blocks as a unified list-type object.

[0024] The index construction module is configured to organize the packaged compressed blocks in an array form to construct a compressed column, construct a compressed offset index matched with the compressed column, and initialize a hint index, wherein the compressed offset index is configured to establish a mapping relationship between a logical position and a compressed segment for each compressed block in the compressed column, and the hint index is configured to record a last accessed position.

[0025] Further, the dynamic encoding module adopts a dynamic auxiliary structure management mechanism to perform dynamic encoding operation on each logical compressed unit, including:

[0026] In the unaligned data, each encoded block in the logical compressed unit is directly traversed, a value marked by the deletion list is skipped, and a run length is adjusted to achieve the lazy deletion processing in the compressed state.

[0027] In the aligned data, the null value region is aligned and divided into multiple null segments according to the compressed mode based on the bitmap and the deletion information; for the continuous null value segment, the null value run is directly encoded and merged into the compressed stream; and for the non-continuous null value segment, only the affected segment is restored to the original encoding format for subsequent query and analysis.

[0028] Further, the operator module includes:

[0029] The retrieval and positioning module is configured to retrieve and position the compressed blocks in the compressed column based on the compressed offset index to determine a target position.

[0030] The homomorphic operator execution module is configured to construct a homomorphic operator, perform homomorphic calculation on the compressed state data memory representation based on the compressed offset index and the hint index, return the query result in an uncompressed format to the client, and update the hint index.

[0031] Further, in the retrieval and positioning module, the compressed blocks in the compressed column are retrieved and positioned based on the compressed offset index to determine a target position, including:

[0032] When a target position request is received, the compressed block number where the target position is located is determined based on the compressed offset index, and the corresponding compressed block content is accessed based on the compressed block number.

[0033] The hint index is updated to the compressed block number of the last successful hit as a starting reference position for subsequent access.

[0034] When a new target position request arrives, it is determined whether the compressed block indicated by the hint index covers the target position of the current target position request.

[0035] If hit, data retrieval is completed directly within the compressed block;

[0036] If not hit, continue scanning forward or re-search the compressed offset index from the starting reference position according to the positional relationship between the target position and the current hint index.

[0037] In a second aspect, the present application provides a time series database query method based on compressed data direct calculation, comprising the following steps:

[0038] The client receives the SQL query request input by the user and sends it to the query layer;

[0039] After the query layer reads the target compressed data block from the storage layer according to the SQL query request, it uses the delayed decompression strategy and dynamic auxiliary structure management strategy to construct the target compressed data block into a compressed state data memory representation, and directly performs homomorphic calculation on the generated compressed state data memory representation using homomorphic operators, and returns the query result in uncompressed format to the client.

[0040] In a third aspect, the present application provides a computer readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the method.

[0041] In a fourth aspect, the present application provides a computing device, comprising one or more processors and a memory, the memory storing one or more programs and being configured to be executed by the one or more processors, the one or more programs including instructions for executing the method.

[0042] The present application has the following advantages due to the above technical solutions:

[0043] 1. The present application systematically summarizes the establishment of a homomorphic compression complete theoretical system in a time series database, breaking through the "theory-algorithm-system" whole link, and realizing the systematization and standardization of compressed state query.

[0044] 2. The modularized compressed state time series data intermediate representation provided by the present application supports multiple lightweight compression algorithms and filtering, aggregation, windowing and other types of time series operators, and can flexibly extend compression methods and query operations according to business needs. Through the integration of time series homomorphic operators, complex time series queries can be completed without full decompression, significantly reducing query delay and runtime memory occupancy.

[0045] 3. The optimization provided by the present application covers dynamic auxiliary encoding, lazy deletion and delayed decompression mechanisms, ensuring that high throughput and low latency performance is maintained even as data size continues to grow.

[0046] 4. The application provides a time series data homomorphic query system, which is compatible with existing query interfaces, and users can obtain performance improvement without modifying upper-layer applications.

[0047] In summary, the application can be widely applied to the technical field of big data processing, and is particularly suitable for industrial Internet of Things, financial risk control, intelligent medical treatment, intelligent manufacturing and other application scenarios with real-time analysis requirements for massive time series data. BRIEF DESCRIPTION OF DRAWINGS

[0048] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for purposes of illustration only and are not to be construed as limiting the application. Throughout the drawings, like reference numerals refer to like parts throughout the several views. In the drawings:

[0049] Figure 1 is a schematic diagram of the architecture design of the time series database query system based on direct calculation of compressed data provided by the application;

[0050] Figure 2 is a schematic diagram of the design of the time series database query system based on direct calculation of compressed data provided by the embodiment of the application, wherein ① represents importing compressed data blocks from a disk; ② represents a compressed state data representation structure, including delayed decompression and dynamic auxiliary structure management; and ③ represents homomorphic operator execution, in which the operator efficiently interacts with the compressed state data representation through compressed offset indexes and prompt indexes;

[0051] Figure 3 (a) and Figure 3 (b) are schematic diagrams of a conventional method and a sequence scanning method based on delayed decompression of the application provided by the embodiment of the application;

[0052] Figure 4 is a flowchart schematic diagram of the time series database query method based on direct calculation of compressed data provided by the embodiment of the application. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions and advantages of the embodiments of the application clearer, the technical solutions of the embodiments of the application will be described clearly and completely below with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the described embodiments of the application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the application.

[0054] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0055] In some embodiments of the present application, a time series database query system based on compressed data direct calculation is provided, which has the following core goals: first, to establish a time series data homomorphic compression theoretical model, realize integrated design from the operator layer to the system layer, and meet the comprehensive processing needs of large-scale time series data "high throughput-low delay"; second, to reduce runtime memory occupation and improve system scalability through unified and modular data structure and system-level optimization measures; and finally, to directly query and operate on compressed time series data under the premise of maintaining high compression ratio, so as to significantly reduce query delay.

[0056] Correspondingly, in some other embodiments of the present application, a time series database query method, device and medium based on compressed data direct calculation are provided.

[0057] In order to better understand the present application, first, the theoretical basis of the present application is introduced as follows.

[0058] The homomorphic compression theory proposed by the present application aims to summarize the properties, characteristics and applicable scope of compressed data direct calculation technology. Compressed data direct calculation seeks to achieve the same operation results on compressed and uncompressed data without decompression and re-compression. In abstract algebra, there is a similar concept, homomorphic mapping, which is defined as a mapping that preserves all algebraic structures between the domain and the range of the algebraic set. Based on this similarity, the present application proposes the concept of homomorphic compression, which regards homomorphism as a novel compression attribute.

[0059] To apply homomorphic mapping to data compression, the data must first be converted into a structure with algebraic awareness. The present application proposes a homomorphic algebraic system for time series data and a query method thereof. Uncompressed time series and compressed time series are modeled as algebraic systems (S u ,Π) and (S c ,Θ) respectively, where S u and S c represent the data domain of time series; both Π and Θ contain six types of basic operators F, J, A, G, E, S, corresponding to filter, timestamp join, aggregation, sliding window grouping, expression and slice respectively, and they form a one-to-one mapping relationship. The present application regards the compression algorithm as a mapping If for any operator op∈Π and its corresponding op′∈Θ, the following equation always holds The present application provides a time series database query system based on compressed data direct calculation. Definition as homomorphic compression mapping.

[0060] The present application provides a homomorphic time series database query based on homomorphic compression mapping. The time series database query process can be abstracted as a "decompression-recovery-query" process, which is formally represented as When there is The query is called homomorphic query; if the operation op'(u) on the uncompressed segment u does not appear at all during the query process, it is a direct homomorphic query, otherwise it is a partial homomorphic query. If the cost of the recovery operation R c on the compressed side is lower than that of R u on the uncompressed side, i.e. Cost(R c (c)) < Cost(R u (u)), R c is defined as effective recovery; if further satisfies Cost(Q c (c)) < Cost(Q u (u)), Q c is an effective homomorphic query, otherwise it is an ineffective homomorphic query.

[0061] Embodiment 1

[0062] As shown in Figure 1 , Figure 2 , the present application provides a time series database query system based on compressed data direct calculation, which comprises:

[0063] A client for sending a SQL query request input by a user to a query layer and receiving a query result returned by the query layer;

[0064] A query layer for constructing a target compressed data block into a compressed state data memory representation by using a delayed decompression strategy and a dynamic auxiliary structure management strategy after reading the target compressed data block from a storage layer according to the SQL query request, and directly performing homomorphic calculation on the generated compressed state data memory representation by using a homomorphic operator to obtain a query result in uncompressed format and return the query result to the client;

[0065] A storage layer for storing compressed data blocks, each of which contains a plurality of compressed data pages, each of which contains time series data compressed in a lightweight manner.

[0066] Further, the query layer includes a data structure module and an operator module, which cooperate with the compressed data pages of the storage layer and the SQL query request of the client to jointly realize direct calculation and efficient transmission of compressed time series data.

[0067] To solve the problem of data transmission and reading overhead in the homomorphic query process, the embodiment further provides two collaborative optimization techniques: dynamic auxiliary structure management and delayed decompression mechanism. The two work together to achieve efficient processing of null values, deletion markers, and double-layer compressed data pages while maintaining the integrity of the compressed data structure, thereby improving system query performance and reducing resource consumption.

[0068] Specifically, the data structure module is the foundation of the time series database query system, used to map the target compressed data block in the storage layer to a compressed state data memory representation using dynamic auxiliary structure management and delayed decompression mechanism during the loading phase, and maintain two-level positioning indexes: compressed offset index and hint index.

[0069] The operator module is used to convert common operators of time series databases such as filtering, joining, aggregating, transforming, and windowing into homomorphic operators, and perform homomorphic calculations on the compressed state data memory representation based on the compressed offset index and hint index. The query results are returned to the client in uncompressed format. In this embodiment, these homomorphic operators all take the compressed state data memory representation as the unified input and output. All intermediate results are still in the form of compressed state data memory representation between operators until the final result is obtained and returned to the client in uncompressed format. The entire process can be completed without explicit decompression, thus meeting the requirements of effective homomorphic query.

[0070] Further, the data structure module includes:

[0071] The data reading module is used to read the target compressed data block from the storage layer according to the SQL query request and write it into the data block cache in double-layer compressed form.

[0072] The general decompression module is used to sequentially traverse the target compressed data block and trigger the general decompression of the target compressed data page when it is hit using the delayed decompression mechanism, obtaining lightweight compressed data.

[0073] The compressed state data memory representation construction module is used to recover null values and deletion information in lightweight compressed data in place relying on dynamic auxiliary structure management, obtaining the compressed state data memory representation, and constructing the compressed offset index and hint index.

[0074] Further, in the general decompression module, the embodiment uses the delayed decompression mechanism to postpone the general decompression operation until the target page is truly accessed.

[0075] As Figure 3The figure illustrates the processing differences between traditional sequence scanning methods and the sequence scanning method based on delayed decompression strategy proposed in this invention when accessing compressed time-series data files. By comparing the data flow and decompression process of the two methods, the figure visually reflects the advantages of the delayed decompression strategy in reducing unnecessary decompression and improving access efficiency.

[0076] In existing methods, time-series databases first use lightweight compression algorithms (such as RLE or Gorilla) for page-level compression, and then uniformly apply a general compression algorithm (such as LZ4) to compress the entire compressed data block. During the query process, even if only a portion of the data is accessed, the entire compressed data block needs to be decompressed, resulting in a significant waste of unnecessary computational resources.

[0077] like Figure 3 (a) illustrates the traditional sequence scanning method. In this method, when the system needs to access a page of data within a compressed data block, even if only a small portion is accessed, the entire compressed page must first be loaded from disk into memory and decompressed. For example, when the system accesses "page 1" in block group 1, it must first decompress the entire compressed block containing "pages 1–5," and then use lightweight decoding (such as RLE decoding) to convert the logical values ​​(e.g., L0:4, L1:9) into the actual raw values ​​(V0:3, V1:8) to obtain the value required by the user. Since the entire compressed block needs to be decompressed for each access, this approach incurs a significant CPU and memory burden, especially when only a portion of the data is accessed, resulting in resource waste.

[0078] Therefore, this invention delays the decompression operation of general compression until the sequence scan phase. Specifically, when reading compressed data blocks, the system retains the double-compression form of the compressed data and only creates a compressed data representation containing metadata. General decompression of the target page is triggered only when the scan cursor is positioned on the target page, and the decompression result is loaded as vector data, avoiding lightweight decoding. For pages that are not accessed, their compression state remains unchanged throughout the entire query process. Figure 3 (b) illustrates the delayed decompression-based sequence scanning method proposed in this invention. Unlike traditional methods, this approach delays the decompression operation of general compression (such as LZ4) during sequence scanning, triggering decompression only when specific page data needs to be read. The figure shows that when the system accesses "page 1," only the compressed data page containing that page is decompressed, avoiding the decompression of other irrelevant pages, thus significantly reducing the decompression workload.

[0079] In addition, the method utilizes offset index in compressed data page and lightweight compressed data structure, supports page access and in-page positioning, and maintains decoding accuracy and low latency. Finally, the system restores the original time series data through decoding operation, but only for the part that is really needed, and realizes data minimization decompression. In addition, to be compatible with upper-layer query operators that do not support compressed state calculation, the compressed state data representation provided by the application also supports on-demand exposure of decompressed data view, which ensures interface transparency and improves calculation flexibility.

[0080] Further, the compressed state data memory representation construction module comprises:

[0081] The data division module is configured to divide the lightweight compressed data into a plurality of logical compressed units, each of which corresponds to a continuous or correlated timestamp sequence.

[0082] The dynamic encoding module is configured to perform dynamic encoding operation on each logical compressed unit based on a lightweight compression algorithm corresponding to the lightweight compressed data, that is, to restore null values and deletion information in place in the lightweight compressed data to obtain a plurality of compressed blocks.

[0083] The compressed block packaging module is configured to store each compressed block as an independent encoded data structure and package it as a unified list object for subsequent memory management and operation compatibility.

[0084] The index construction module is configured to organize and build the compressed blocks into a compressed column in an array form, construct a compressed offset index matched with the compressed column, and initialize a hint index. The compressed offset index is used to establish a mapping relationship between the logical position and the compressed segment for each compressed block in the compressed column to provide offset resolution capability. The hint index is used to record the time locality information of the lightweight compressed data (i.e., record the last accessed position) to ensure the directness of the operator module to the lightweight compressed data.

[0085] Further, in the dynamic encoding module, the application adopts a dynamic auxiliary structure management mechanism to realize online recombination of null values and deletion markers in a dynamic encoding manner while maintaining the compressed state.

[0086] The system of the application supports two time series data formats: a non-aligned format (each column has an independent time column) and an aligned format (each column shares a time column). The aligned format usually uses a bitmap to identify null values and uses a deletion list to record logical deletion positions, so as to complete null value management and deletion operations without physically modifying the storage content. Although such a compact layout helps to save storage space and improve I / O efficiency, it introduces data access complexity in the sequence scanning process. Specifically, null values need to be backfilled to the original positions according to the bitmap, and the deletion positions need to be compared in real time, which destroys the structural continuity of compressed data and affects the efficiency of homomorphic queries.

[0087] To solve the above problems, the application introduces a dynamic encoding strategy to realize null value insertion and deletion skipping in the compressed data structure without decompressing the entire data set. In non-aligned data, the system directly traverses each encoding block, such as run-length encoding (RLE), skips the marked values according to the deletion list, and adjusts the run length to realize lazy deletion processing in the compressed state. In aligned data, the system aligns and divides the null value area into multiple null segments according to the bitmap and deletion information in the RLE mode. For continuous null value segments, the system directly encodes as null runs and merges into the compressed stream to maintain structural continuity; for non-continuous null value segments, the system only restores the affected segments to the original encoding format for subsequent query parsing because they destroy the compressed structure. In the entire process, only the re-encoded results are written back to the buffer at the end of scanning, and the data blocks that are not accessed remain in the original compressed state, realizing minimal intervention during query processing.

[0088] By integrating dynamic auxiliary structure management and delayed compression optimization mechanisms, the time series database query system in the compressed state of the application realizes a direct query path in the compressed state at both logical and physical levels. Dynamic auxiliary structure management avoids damaging the compressed structure, and late decompression of compressed data blocks minimizes resource consumption during data decompression and transmission.

[0089] Further, as shown in Figure 4 , the operator module includes:

[0090] The retrieval and positioning module is configured to retrieve and position the compressed blocks in the compressed column according to the compressed offset index, and determine the target position.

[0091] The homomorphic operator execution module is configured to convert common operators of the time series database, such as filtering, connection, aggregation, transformation, and window, into homomorphic operators, and perform homomorphic calculation on the in-memory representation of the data in the compressed state based on the compressed offset index and the hint index, and return the query result in an uncompressed format to the client.

[0092] Further, in the retrieval and positioning module, the application proposes an efficient data access mechanism combining compressed offset index and hint index to solve the problem of random access of time series data in memory under compression state. The mechanism realizes fast positioning and retrieval of compressed data without decompression by constructing the mapping relationship between logical position and compressed physical block, and combining the locality feature of access history.

[0093] In the data access process, when a target position request is received, the system first determines the compressed block number where the target position is located through the compressed offset index, and accesses the corresponding compressed block content accordingly. The compressed offset index not only supports fast jump access at the compressed block level, but also allows precise positioning of target compressed data items within the compressed block through relative position calculation, avoiding linear scanning operations on decompressed columns, effectively improving data access efficiency, especially for structured compression formats such as run-length encoding (RLE).

[0094] To further improve the performance of compressed offset index in continuous query or sliding window access scenarios, the application provides a hint index as an optimization means. The hint index stores the compressed block number of the last successful hit in integer form as the starting reference position for subsequent access. When a new request arrives, the system first determines whether the compressed data block pointed to by the hint index covers the target position of the current request. If it hits, the system completes data retrieval directly within the compressed data block; if it does not hit, it selects to continue scanning forward or re-finds the compressed offset index from the starting reference position according to the position relationship between the target position and the current hint index. Once the target position is successfully located, the corresponding hint index is updated for reuse in the next access.

[0095] This method realizes structured positioning of compressed state data through compressed offset index, and realizes state-aware fast query start through hint index, thereby greatly reducing the query delay of random access while maintaining the compression rate.

[0096] In terms of implementation, a general memory data representation object is defined for the upper layer system, the compressed state data representation object inherits from the general memory data representation interface, and a set of standardized data access, slicing, deserialization and reverse methods are defined to ensure that compressed data can be directly processed by the upper layer query engine while maintaining encapsulation, avoiding decompression overhead. Write and serialization operations are implemented through auxiliary builder classes and encoder classes, which are used to construct compressed blocks and perform memory and persistent conversion, respectively.

[0097] It is worth pointing out that the present application has the characteristics of high modularity and abstraction, and the specific logic of the compression algorithm is transparent to the upper layer operation, thereby realizing the algorithm-independent unified interface design. In addition, since all operations are completed in the compressed state, the performance loss in the decompression and recoding process is significantly reduced, and the engineering portability and system integration efficiency are high.

[0098] Further, in the homomorphic operator execution module, an operator supporting direct calculation of the compressed state is provided to realize efficient homomorphic queries in the time series database. By introducing a set of core operators with structural awareness, combined with compression formats such as RLE and dictionary encoding, in-place data processing without decompression is realized, thereby significantly reducing intermediate calculation redundancy and transmission cost. It should be noted that the present embodiment is applicable to any lightweight compression algorithm and operator suitable for time series data. For convenience of description, the present embodiment takes a set of six core operators and the RLE algorithm as an example, but this is not intended to limit the present application.

[0099] The present application defines four encoding primitives T delta , T repeat , T pack , T dic , which represent difference encoding, repetition encoding, bit packing encoding and dictionary encoding, respectively. Different lightweight compression algorithms are implemented by combining , for example, Gorilla is composed of T delta , T reprat and T pack .

[0100] In the specific implementation of the present application, a set of six core homomorphic operators is defined, including filtering, joining, aggregation, sliding window grouping, expression and slicing operations. When a query is composed of the above operators, the system maps it to a full homomorphic query, so that each operation is completed in the compressed domain; if it contains other operators, it is converted into a partial homomorphic query, allowing mixed compressed state and decompressed state processing. The support degree of the six operators provided by the present application to the above encoding primitives is illustrated in Table 1, in which T repeat , T pack , T dic are directly homomorphic supported for all operators, and T delta only has conditional support in aggregation and expression depending on the sum property.

[0101] Table 1 Operator-encoding component matrix

[0102]

[0103] Filter operator. Filter operator supports unary predicate, binary comparison and regular expression. For example, for RLE encoding, the system traverses the pattern by block, and each repeated value is calculated only once, avoiding repeated judgment; this operation is usually pushed into the execution stage during the construction of the compressed state data memory representation to reduce the memory construction overhead.

[0104] Join operator. Join operator adopts columnar processing mode, and performs matching on each column data block according to the timestamp alignment principle. In the RLE encoding column, the system traverses the data segment by run, directly writes a uniform value to the constant segment, and performs value level extraction and writing to the non-constant segment. This method can optimize the complexity from O (NC) to O (RC) when the selected row is much more than the RLE run number.

[0105] Aggregation operator. Aggregation operators (such as sum, mean, variance, and extreme value) in compressed data are updated incrementally by temporary accumulators, without iterative processing of each data point. For example, for RLE encoded data, the system updates the cumulative state variables n, v, m 2 , and realizes online calculation of mean and variance when traversing each data block. Figure 3 The incremental aggregation calculation process under this mode is shown, which effectively avoids the calculation redundancy after expanding the complete data.

[0106] Expression operator. Expression operator covers arithmetic, logic and comparison operations. In RLE format, the operation is processed by run, and each mode value is calculated only once.

[0107] Sliding window grouping operator. This operator is usually used in combination with aggregation, which divides the compressed column into multiple blocks according to the time window, and directly performs aggregation calculation in each block without decompression. Because the operation only depends on the time index and the compressed block boundary, it is suitable for large-scale sliding window analysis scenarios.

[0108] Slice operator. Slice operator supports sequential scanning and batch extraction of long history sequences. The system quickly locates the starting offset based on the hint index, and identifies the compressed segment boundary by combining the compressed offset index, so as to accurately extract the sub-region data. During the slicing process, the hint index will be updated to the new position index for subsequent request reuse, which significantly reduces repeated traversal.

[0109] In summary, the time series database query system based on compressed data direct calculation proposed by the present application has the following beneficial effects:

[0110] (1) A homomorphic compression theory framework for time series data is proposed. The framework formalizes direct calculation as an algebraic homomorphism problem, models timestamp-related operators and complex time series management processes, and gives the compressed method selection criteria and operator feasibility determination conditions.

[0111] (2) A unified modular compressed data memory representation is proposed. The compressed data memory representation supports multiple lightweight lossless compression algorithms, and enables filter, aggregate and sliding window computation without decompression, and seamlessly interacts with storage layer and query engine while maintaining data compression state.

[0112] (3) A homomorphic compressed query framework is constructed. The framework implements a homomorphic query operator chain on the basis of the unified compressed state time series data memory representation, and combines dynamic auxiliary coding, lazy deletion management and delayed decompression strategy to efficiently maintain auxiliary structures such as null bitmap, thereby significantly reducing data access overhead.

[0113] Embodiment 2

[0114] Based on the compressed data direct calculation based time series database query system provided in Embodiment 1, correspondingly, the present embodiment provides a compressed data direct calculation based time series database query method, which comprises the following steps:

[0115] 1) The client receives the SQL query request input by the user and sends it to the query layer;

[0116] 2) After the query layer reads the target compressed data block from the storage layer according to the SQL query request, it uses the delayed decompression strategy and dynamic auxiliary structure management strategy to construct the target compressed data block into a compressed state data memory representation, and directly performs homomorphic calculation on the generated compressed state data memory representation using homomorphic operators, and returns the query result in uncompressed format to the client.

[0117] Embodiment 3

[0118] The present embodiment provides a processing device corresponding to the compressed data direct calculation based time series database query method provided in Embodiment 2, which can be a processing device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of Embodiment 2.

[0119] The processing device comprises a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through the bus to complete the communication among each other. The memory stores a computer program executable on the processor, and the processor executes the compressed data direct calculation based time series database query method provided in Embodiment 2 when executing the computer program.

[0120] Preferably, the memory can be a high-speed random access memory (RAM: Random Access Memory), and can also include a non-volatile memory, such as at least one disk memory.

[0121] Preferably, the processor can be a central processing unit (CPU), a digital signal processor (DSP), or various types of general-purpose processors, without limitation.

[0122] Embodiment 4

[0123] The time series database query method based on direct calculation of compressed data in this embodiment 2 can be embodied as a computer program product, which can include a computer readable storage medium, and computer readable program instructions for executing the time series database query method based on direct calculation of compressed data described in this embodiment 2 are loaded on the computer readable storage medium.

[0124] The computer readable storage medium can be a tangible device that maintains and stores instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0125] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0127] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0128] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are generated to realize the computer-implemented processes in the computer or other programmable devices, and the instructions executed in the computer or other programmable devices provide operational steps for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0129] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the above embodiments of the present application have been described in detail, those skilled in the art should understand: the specific embodiments of the present application can be modified or replaced by the equivalent, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered in the protection scope of the claims of the present application.

Claims

1. A time-series database query system based on direct computation of compressed data, characterized in that, include: Client-side, query layer, and storage layer; The client is used to send the SQL query request entered by the user to the query layer and receive the query results returned by the query layer; The query layer is used to read the target compressed data block from the storage layer according to the SQL query request, and then construct the target data block into a compressed data memory representation using a delayed decompression mechanism and a dynamic auxiliary structure management mechanism. The homomorphic operator is used to directly perform homomorphic computation on the generated compressed data memory representation, and the query result is returned to the client in an uncompressed format. The storage layer is used to store compressed data blocks, and each compressed data block contains several compressed data pages, and each compressed data page contains compressed time-series data. The query layer includes a data structure module and an operator module. The data structure module is used to map the target compressed data block in the storage layer into a compressed data memory representation during the loading phase, and maintain two-level positioning indexes: a compressed offset index and a hint index. The operator module is used to convert commonly used operators of time-series databases into homomorphic operators, and perform homomorphic computation on the compressed data memory representation based on the compressed offset index and the hint index. The resulting query result is returned to the client in an uncompressed format. The data structure module includes: a data reading module, used to read target compressed data blocks from the storage layer according to SQL query requests and write them into the data block cache in a double-compression form; a general decompression module, used to sequentially traverse the target compressed data blocks and trigger the general decompression of the target compressed data pages when a delayed decompression mechanism is hit, to obtain lightweight compressed data; and a compressed data memory representation construction module, used to restore null values ​​and deletion information in-situ within the lightweight compressed data based on dynamic auxiliary structure management, to obtain a compressed data memory representation, and to construct a compression offset index and a hint index. The compressed data memory representation construction module includes: a data partitioning module, used to divide the lightweight compressed data into several logical compression units, each logical compression unit corresponding to a continuous or correlated timestamp sequence; a dynamic encoding module, used to perform dynamic encoding operations on each logical compression unit based on the lightweight compression algorithm corresponding to the lightweight compressed data, using a dynamic auxiliary structure management mechanism to obtain several compressed blocks; a compressed block encapsulation module, used to store each compressed block as an independent encoded data structure and encapsulate it into a unified list-style object; and an index construction module, used to organize the encapsulated compressed blocks into compressed columns in the form of an array, construct a compressed offset index corresponding to the compressed columns, and initialize the prompt index, wherein the compressed offset index is used to establish a mapping relationship between the logical position and the compressed segment for each compressed block in the compressed column; and the prompt index is used to record the position of the last access.

2. The time-series database query system based on direct computation of compressed data as described in claim 1, characterized in that, In the dynamic encoding module, a dynamic auxiliary structure management mechanism is used to perform dynamic encoding operations on each logical compression unit, including: In unaligned data, each encoded block in the logical compression unit is directly traversed, and the marked values ​​are skipped according to the deletion list and the running length is adjusted to achieve lazy deletion processing in the compressed state. In the aligned data, based on the bitmap and deletion information, the null value region is aligned and divided into multiple null value segments according to the compression mode; for continuous null value segments, they are directly encoded as null values ​​and merged into the compressed stream; for non-continuous null value segments, only the affected segments are restored to the original encoding format for subsequent query parsing.

3. The time-series database query system based on direct computation of compressed data as described in claim 1, characterized in that, The operator module includes: The retrieval and positioning module is used to retrieve and locate compressed blocks in the compressed column based on the compressed offset index to determine the target location; The homomorphic operator execution module is used to construct homomorphic operators and perform homomorphic computation on the compressed data memory representation based on the compressed offset index and the hint index. The resulting query results are returned to the client in uncompressed format, and the hint index is updated at the same time.

4. The time-series database query system based on direct computation of compressed data as described in claim 3, characterized in that, The retrieval and positioning module retrieves and positions compressed blocks in the compressed column based on the compressed offset index to determine the target location, including: When a target location request is received, the compressed block number where the target location is located is determined by the compressed offset index, and the corresponding compressed block content is accessed accordingly. Update the prompt index to the number of the most recently successfully hit compressed block, and use it as the starting reference position for subsequent accesses; When a new target location request arrives, determine whether the compressed block pointed to by the hint index covers the target location of the current target location request: If a match is found, the data retrieval is completed directly within that compressed block; If a match is not found, the system will choose to continue scanning forward or search for the compressed offset index again from the starting reference position, depending on the positional relationship between the target location and the current suggested index.

5. A time-series database query method based on direct computation of compressed data using the system described in any one of claims 1 to 4, characterized in that, Includes the following steps: The client receives the SQL query request input by the user and sends it to the query layer; After the query layer reads the target compressed data block from the storage layer according to the SQL query request, it uses a delayed decompression strategy and a dynamic auxiliary structure management strategy to construct a compressed data memory representation of the target compressed data block, and uses homomorphic operators to directly perform homomorphic computation on the generated compressed data memory representation. The resulting query result is returned to the client in an uncompressed format.

6. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the method of claim 5.

7. A computing device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of claim 5.

Images