Data processing methods, systems, readable storage media, and computer program products

Abstract

This application discloses a data processing method, system, readable storage medium, and computer program product, belonging to the field of data processing technology. The data processing method includes: acquiring sensor data; compressing the sensor data based on a target dictionary and a target observation matrix to obtain compressed data segments; the compressed data segments include time window information and sparsity characteristic information of the sensor data; storing the compressed data segments according to a pre-constructed index structure in a time-series database, the index structure including a main index layer and an auxiliary index layer, the main index layer using time windows as indexes and the auxiliary index layer using sparsity characteristic information as indexes; upon receiving a user's query request, determining the target time range and target sparsity to be queried, obtaining target compressed data based on the target time range and target sparsity, and decompressing the target compressed data to obtain the target data. This allows for effective querying of the required data.

Description

Technical Field

[0001] This application belongs to the field of big data processing technology, specifically relating to a data processing method, system, readable storage medium, and computer program product. Background Technology

[0002] In related technologies, time-series data acquired by sensors often exhibits sparsity under dynamic operating conditions, meaning sensor data is frequently sparse. Most elements in sparse data are zero or invalid values, therefore, sparse data needs to be compressed before storage to reduce resource consumption. Time-series databases are database systems specifically designed for processing continuous data with timestamps, and are commonly used to store compressed sensor data. However, because the index structure of existing time-series databases typically only includes timestamps, it often leads to difficulties in effectively retrieving the required data. Summary of the Invention

[0003] The purpose of this application is to provide a data processing method, system, readable storage medium, and computer program product to solve the problem that existing time-series databases cannot effectively query the required data.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows: Firstly, a data processing method is provided, including: Sensor data is collected, a target dictionary and a target observation matrix are determined, and the sensor data is compressed based on the target dictionary and the target observation matrix to obtain a compressed data segment; wherein, the compressed data segment includes the time window information and sparsity information of the sensor data; Based on the time window information and the sparsity characteristic information, the compressed data segment is stored according to the index structure pre-built in the time series database; wherein, the index structure includes a main index layer and an auxiliary index layer, the main index layer is indexed by the time window, the auxiliary index layer is the leaf layer of the main index layer, and the auxiliary index layer is indexed by the sparsity characteristic information; Upon receiving a user's query request, the system determines the target time range and target sparsity based on the query request. Then, based on the target time range, target sparsity, and the storage information in the time series database, it obtains the target compressed data and decompresses the target compressed data to obtain the target data.

[0005] Secondly, a data processing system is provided, including: A data preprocessing layer is used to collect sensor data, determine the target dictionary and target observation matrix, and compress the sensor data based on the target dictionary and target observation matrix to obtain compressed data segments; wherein, the compressed data segments include the time window information and sparsity information of the sensor data; The core processing layer is used to store the compressed data segment according to the time window information and the sparse characteristic information, and according to the index structure pre-built in the time series database. The index structure includes a main index layer and an auxiliary index layer. The main index layer is indexed by the time window, and the auxiliary index layer is the leaf layer of the main index layer. The auxiliary index layer is indexed by the sparse characteristic information. The application service layer is used to receive user query requests; The core processing layer is also used to: determine the target time range and target sparsity to be queried according to the query request; obtain target compressed data according to the target time range and target sparsity and the storage information in the time series database; and decompress the target compressed data to obtain the target data.

[0006] Thirdly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0007] Fourthly, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] The solution implemented in this application allows data to be stored using time windows and sparse characteristic information as indexes. This enables the efficient retrieval of required data segments by utilizing sparse characteristic information to quickly exclude data segments that do not meet the conditions when performing time range queries containing sparse characteristic information. This eliminates the need to read and decompress large amounts of invalid data. Attached Figure Description

[0009] Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application; Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the data processing system in a specific embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0012] To facilitate understanding of the embodiments of this application, the following will be described first.

[0013] A Time-Series Database (TSDB) is a database system specifically designed for processing continuous data with timestamps. It includes at least the following core characteristics: (1) The time series data model with timestamp as the key index is adopted, which can support efficient writing and querying of data in time order. It is typically used in scenarios such as sensor data and log data.

[0014] (2) Adopt a data partitioning strategy, usually partitioning and storing data according to time windows, which facilitates data archiving, query optimization and load balancing.

[0015] (3) It adopts aggregate query optimization and has built-in aggregate functions for time series, supporting efficient time range aggregate calculation, such as calculating the average, calculating the maximum value and trend analysis.

[0016] Sparse representation techniques refer to representing raw data as a linear combination of a small number of dictionary atoms through dictionary learning (DL). The core steps include: Step 1: Dictionary Construction: Generate an overcomplete dictionary from the training data for sparse coding; Step 2: Sparse coding: Map the input data to sparse coefficients to satisfy the sparse representation (SR) of the input data in the dictionary. Step 3: Data Reconstruction: Reconstruct the original data using sparse coefficients and a dictionary to achieve data decompression and recovery.

[0017] Common compression techniques for time series databases include lossless compression and lossy compression. Lossless compression refers to compressing data by finding repeating patterns, which is suitable for scenarios with high accuracy requirements, but has limited compression ratio. Lossy compression refers to a series of codes, such as run-length encoding, for numerical time series, which improve the compression ratio by allowing a certain amount of error.

[0018] Optionally, the scenarios applicable to the embodiments of this application include, but are not limited to, dynamic sparse scenarios (DSS).

[0019] The data processing method, system, readable storage medium, and computer program product provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0020] Please see Figure 1 , Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps: Step 11: Collect sensor data, determine the target dictionary and target observation matrix, and compress the sensor data based on the target dictionary and target observation matrix to obtain compressed data segments; the compressed data segments include the time window information and sparsity information of the sensor data.

[0021] Step 12: Based on the time window information and the sparsity characteristic information, store the compressed data segment according to the index structure pre-built in the time series database; wherein, the index structure includes a main index layer and an auxiliary index layer, the main index layer is indexed by the time window, the auxiliary index layer is the leaf layer of the main index layer, and the auxiliary index layer is indexed by the sparsity characteristic information.

[0022] Step 13: Upon receiving a user's query request, determine the target time range and target sparsity to be queried based on the query request. Based on the target time range and target sparsity to be queried and the storage information in the time series database, obtain the target compressed data and decompress the target compressed data to obtain the target data.

[0023] The solution implemented in this application allows data to be stored using time windows and sparse characteristic information as indexes. This enables the efficient retrieval of required data segments by utilizing sparse characteristic information to quickly exclude data segments that do not meet the conditions when performing time range queries containing sparse characteristic information. This eliminates the need to read and decompress large amounts of invalid data.

[0024] In this embodiment, the sensor data can be acquired through sensor nodes and may include data such as temperature, pressure, and vibration. Compression and storage operations are performed on the sensor data within each time window; that is, data compression processing is performed based on the time window. The length of the time window is configurable and is not limited thereto.

[0025] In some embodiments, sensor nodes can acquire signals such as temperature, pressure, and vibration in real time through physical interfaces and generate raw data points with nanosecond-level timestamps. For multi-sensor clusters, a broadcast-based clock synchronization algorithm can be used to eliminate clock skew between sensor nodes, ensuring the consistency of the data time series. For outliers in the sensor data, the 3σ criterion can be used to filter data points that significantly deviate from the mean (such as out-of-range values ​​caused by sensor malfunctions), preventing invalid data from entering subsequent processing.

[0026] Optionally, determining the target dictionary and target observation matrix includes: determining the target dictionary based on a pre-acquired initial dictionary and dictionary updates synchronized from the dictionary training library, i.e., determining the latest version of the dictionary as the target dictionary; determining the number of rows in the target observation matrix based on the sparsity of the sensor data, and determining the target observation matrix based on the number of rows. This can improve the accuracy of data compression.

[0027] In some embodiments, the sparsity k of the corresponding sensor data can be calculated on a time-window basis, where k = number of non-zero elements / total amount of data in the window. Further, the number of rows M of the observation matrix can be determined based on the real-time sparsity k corresponding to each time window. The observation matrix is ​​defined as follows: α is the row number coefficient, an empirical constant verified experimentally based on the characteristics of sensor data, typically ranging from 1.2 to 1.5, used to balance compression ratio and reconstruction accuracy. The number of columns in the observation matrix corresponds to the vector dimension of the compressed data, i.e., the length of the original data participating in compressed sensing modeling within a time window. This length is usually determined by the time window size and sampling interval. With a fixed time window configuration, the number of columns in the observation matrix remains constant; when the time window size or the number of sampling points changes, the number of columns in the observation matrix adjusts accordingly. In contrast, the number of rows in the observation matrix characterizes the projected observation dimension, which is determined in real-time based on the sparsity of the data within the time window. The observation matrix can use a sparse random projection matrix, meaning it only needs to store the positions and values ​​of non-zero elements, reducing matrix storage and multiplication operations.

[0028] In some embodiments, the dictionary is stored in a versioned manner. In the database metadata table (such as DICTIONARY_METADATA), each record contains a dictionary version number consisting of a timestamp and a serial number, an applicable time window, a set of dictionary atoms, training data features, and a checksum. Adjacent dictionary versions are stored incrementally, recording only the differences between atoms, which minimizes storage overhead.

[0029] Optionally, the data processing method in this application embodiment further includes: Under the condition that the first condition is met, sensor data within the N time windows closest to the current time are used as training data, and the training data is used to iteratively optimize dictionary atoms in the current dictionary that do not match the current data sparsity pattern to obtain updated dictionary content; where N is an integer greater than 1; the first condition includes at least one of the following: The difference between the sparsity characteristics of the compressed data segment received at the current time and the training data features of the dictionary version corresponding to the compressed data segment exceeds a first threshold, which can be preset based on actual needs. The interval between the current time and the last dictionary update time exceeds a second threshold, which can be preset based on actual needs.

[0030] In other words, the solution in this application embodiment can dynamically update the dictionary, thereby ensuring the accuracy of data compression. The dictionary update process can be executed in the background thread of the database. It extracts compressed data segments from the historical data cache of the dictionary training library for a certain number of recent time windows, decompresses them, and then uses the improved K-SVD algorithm to perform local atomic dictionary updates, avoiding performance fluctuations caused by full training. The improved K-SVD algorithm is a customized improvement for the kernel-level dictionary management requirements of time-series databases. Specific improvements include: iterative optimization is only performed on dictionary atoms that do not match the current data sparsity pattern (identified by sparsity differences), without needing to update all atoms in the dictionary, thus reducing the amount of computation; during the atomic update process, metadata such as the sparsity and time window features of the current data segment are introduced to ensure that the newly generated atoms are adapted to dynamic sparse scenarios; only local historical data from the most recent N time windows are used as training data instead of the full historical data, and training data is quickly obtained by combining historical data cache, reducing the number of iterations and adapting to the real-time requirements of the database kernel.

[0031] In some embodiments, the primary index layer described above is a time window index layer based on a B+ tree, using the start time of the time window at the millisecond level as the primary key, supporting fast time range traversal. The secondary index layer described above extends the fields in the leaf nodes of the primary index layer, embedding sparsity, dictionary version number (e.g., using a 4-byte integer), and data block storage offset composed of file system path + byte offset, forming composite index entries. This achieves joint positioning based on time and sparsity characteristics, such as locating the time window according to the query time range, filtering invalid entries according to sparsity, and directly locating the corresponding compressed data block based on the data block storage offset.

[0032] In some embodiments, the above-mentioned sparsity characteristic information includes information such as sparsity and dictionary version number.

[0033] In some embodiments, the above index structure further includes a statistical index layer, which stores a summary of data sparsity for each time window and the frequency of dictionary version usage. The statistical index layer is used to assist in retrieving the target compressed data. Thus, a three-layer index structure of "main index layer - auxiliary index layer - statistical index layer" can be formed. The data from the main index layer, auxiliary index layer, and statistical index layer are collectively incorporated into the complete index system, thereby efficiently retrieving the required data.

[0034] For example, statistical summaries of sparsity (such as mean, variance, etc.) and metadata such as dictionary version usage frequency can be generated periodically for each time window. This metadata can be incorporated into the complete index system of the index builder together with the standardized data segment containing time window, sparsity coefficient, and initial dictionary reference. The metadata of the standardized data segment is used to bind the data segment to the dictionary version, and the metadata of the statistical index layer is used to accelerate aggregation queries. Together, they accelerate the execution efficiency of aggregation queries.

[0035] Optionally, the storage format of the compressed data segment in the time-series database may include: a header, a first array, a second array, and a third array. The header stores the version number of the target dictionary, enabling rapid association with the dictionary atom set in the dictionary training library. During decompression, it is not necessary to load the entire dictionary; only the difference atoms corresponding to the version need to be obtained. The first array stores the positions of the non-zero elements of the sparse coefficients corresponding to the sensor data. The second array stores the values ​​of the non-zero elements of the sparse coefficients corresponding to the sensor data. The third array stores the data length of the sensor data, facilitating rapid location of the data length.

[0036] In some embodiments, the process of obtaining the target compressed data includes: first, performing a range search on the time window B+ tree of the time series database based on the target time range to be queried, and obtaining a set of leaf entries corresponding to the candidate time window; second, filtering the sparsity in the obtained set of leaf entries based on the target sparsity, removing leaf entries that do not match the target sparsity to reduce the scanning of invalid data segments; then, for each retained leaf entry, reading the data block storage offset and dictionary version number from it, and directly reading the corresponding compressed data block in the storage layer according to the offset, and determining the dictionary version required for subsequent decompression according to the dictionary version number. After reading the compressed data block, the compressed representation of the sparsity coefficients can be obtained, and the data reconstruction can be completed by combining it with the dictionary corresponding to the determined dictionary version.

[0037] Optionally, determining the target time range and target sparsity based on the query request includes: The query request is parsed to determine the target time range to be queried; that is, the target time range to be queried can be determined by parsing the time conditions in the query request. Based on the stored information in the time-series database, the target sparsity corresponding to the target time range to be queried is determined. The target sparsity can be obtained through statistical analysis or real-time calculation of the stored data; for example, the proportion of non-zero sparsity coefficients in the stored data can be calculated as the target sparsity within the corresponding time range.

[0038] Optionally, before decompressing the target compressed data as described above, the data processing method in this application embodiment may further include: Based on the pre-configured correspondence between the query time range and data sparsity and the observation matrix dimension, the target observation matrix dimension corresponding to the target time range and target sparsity to be queried is determined, and the target observation matrix is ​​constructed based on the target observation matrix dimension.

[0039] The above-mentioned decompression of the target compressed data includes: decompressing the target compressed data using the target observation matrix. This allows for the acquisition of a suitable observation matrix dimension using pre-configured information, thereby improving the decompression accuracy.

[0040] In some embodiments, the observation matrix dimension adjustment strategy can be flexibly configured through a configuration file. The configuration file contains the correspondence between different query time ranges and data sparsity and the dimensions of the observation matrix, as well as related parameter settings, such as the dimension adjustment threshold and step size. This correspondence between different query time ranges and data sparsity and the dimensions of the observation matrix in the configuration file can be predetermined through controlled variable experiments based on historical data samples from similar sensors. For example, first, data sparsity intervals and query accuracy levels are defined. Then, for each sparsity interval-accuracy level combination, the reconstruction error and decompression computation time under different observation matrix dimensions are tested. The dimension value that meets the accuracy requirements and has the lowest computational cost is selected as the optimal dimension for that combination. Subsequently, the query response time and reconstruction error feedback data in actual business scenarios can be collected through the performance monitoring module in the database backend. This correspondence is iteratively corrected, and finally, a stable correspondence is formed and stored in the configuration file. The data processing system can read and update the configuration file in real time during operation, adapting to different application scenarios and changing requirements without modifying the code.

[0041] Optionally, before decompressing the target compressed data as described above, the data processing method in this application embodiment may further include: Based on the target time range to be queried, load the corresponding version of the dictionary from the dictionary training library.

[0042] The above-mentioned decompression of the target compressed data includes: decompressing the target compressed data using the dictionary obtained by loading.

[0043] In summary, this application proposes a method for sparse sensing storage and query optimization of sensor data for time-series databases. The core of this method lies in breaking through the decoupling between traditional time-series databases (TSDBs) and sensor data compression technologies. By embedding sparsity sensing capabilities at the database kernel layer, a collaborative architecture of "edge preprocessing - intelligent database storage - query-driven optimization" is constructed to achieve efficient storage and querying of sensor data. Addressing the pain points of sensor data storage and querying in dynamic sparse scenarios, this application proposes a time-driven dynamic dictionary management mechanism, a composite index system integrating sparse metadata, and a query semantic-aware compression coding strategy, achieving simultaneous improvements in data compression ratio, reconstruction accuracy, storage efficiency, and query speed. Key improvements in this application include: deeply integrating the dictionary learning process with the database transaction system, enabling the database to perceive data sparsity patterns in real time and dynamically adjust storage strategies; integrating sparsity, dictionary version, and other metadata at the index layer to achieve efficient query filtering based on sparsity characteristics; and reducing the computational load on edge devices while improving the overall system's real-time response capabilities through a lightweight edge computing and database collaboration mechanism.

[0044] It should be noted that the data processing method provided in this application embodiment is executed by a data processing system. The data processing system in this application embodiment will be described in detail below.

[0045] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application, as shown below. Figure 2 As shown, the data processing system includes: The data preprocessing layer 21 is used to collect sensor data, determine the target dictionary and target observation matrix, and compress the sensor data based on the target dictionary and target observation matrix to obtain compressed data segments; wherein, the compressed data segments include the time window information and sparsity information of the sensor data; The core processing layer 22 is used to store the compressed data segment according to the time window information and the sparse characteristic information, and according to the index structure pre-built in the time series database; wherein, the index structure includes a main index layer and an auxiliary index layer, the main index layer is indexed by the time window, and the auxiliary index layer is the leaf layer of the main index layer, and the auxiliary index layer is indexed by the sparse characteristic information. Application service layer 23 is used to receive user query requests; The core processing layer 22 is further configured to: determine the target time range and target sparsity to be queried according to the query request; obtain target compressed data according to the target time range and target sparsity and the storage information in the time series database; and decompress the target compressed data to obtain target data.

[0046] The solution implemented in this application allows data to be stored using time windows and sparse characteristic information as indexes. This enables the efficient retrieval of required data segments by utilizing sparse characteristic information to quickly exclude data segments that do not meet the conditions when performing time range queries containing sparse characteristic information. This eliminates the need to read and decompress large amounts of invalid data.

[0047] It should be noted that the aforementioned data preprocessing layer 21, core processing layer 22 and application service layer 23 achieve data and control signal interaction through standardized interfaces, forming an end-to-end sparse sensing data processing closed loop.

[0048] The aforementioned data preprocessing layer 21 can also be called the edge data preprocessing layer, which mainly achieves data preprocessing through edge computing nodes. For example... Figure 3As shown, the edge data preprocessing layer can include a sensor cluster and edge computing nodes. These edge computing nodes include a data acquisition module, a sparsity detection module, and a lightweight encoding module. The edge data preprocessing layer is primarily used to achieve real-time acquisition of sensor data, sparsity detection, and lightweight compression encoding, providing preprocessed structured data segments for the database layer. The interaction logic of the edge data preprocessing layer includes: sensors output raw time-series data at a fixed frequency; the edge computing nodes first perform timestamp calibration and outlier filtering on the acquired sensor data; then, they calculate data sparsity in real-time using a sliding window algorithm to identify dynamic sparsity patterns; subsequently, the lightweight encoding module dynamically adjusts the observation matrix dimension based on the sparsity, adapting the observation matrix dimension to the current data sparsity, optimizing compression efficiency and reconstruction accuracy to adjust the compression strategy, and using a low-complexity algorithm to generate sparsity coefficients. This compresses the raw data into a standardized compressed data segment containing a time window, sparsity coefficients, and an initial dictionary reference, and finally uploads it to the database layer via the edge network. The correspondence between the edge layer adjustment of the observation matrix parameters and the dynamic adaptation of the observation matrix based on the query conditions constitutes the entire collaborative process of data writing and query reading.

[0049] The aforementioned core processing layer 22 can also be referred to as the time-series database core processing layer. For example... Figure 3 As shown, the core processing layer of the time-series database can include a TSDB storage engine and a dictionary training library. The TSDB storage engine includes a data ingestion module, a segmented dictionary manager, an index builder, a query optimizer, and a compression-aware executor. The dictionary training library includes modules such as historical data caching, an incremental learning engine, and a metadata manager. The core processing layer of the time-series database mainly undertakes the core functions of data storage, dynamic dictionary management, index building, and query optimization, and is the key layer for realizing database kernel-level sparsity awareness. The interaction logic of the core processing layer of the time-series database includes: the data ingestion module receives compressed data segments uploaded from the edge layer, triggering the segmented dictionary manager to make dictionary matching and update decisions; the dictionary training library implements incremental dictionary learning based on historical data, and the generated new dictionary version is bound to the data segment through a transaction mechanism; the index builder encodes metadata such as time window, sparsity, and dictionary version into a composite index to support fast data location; the query optimizer parses user query requests and combines the composite index with the compression-aware executor to implement query-driven decompression scheduling.

[0050] The aforementioned application service layer may include a data analysis platform, a real-time monitoring system, and an API interface layer, such as... Figure 3 As shown, it is mainly used to provide users with data query, analysis and visualization services, and interacts with the database layer through standardized APIs to support business needs in multiple scenarios such as real-time monitoring, historical trend analysis and fault diagnosis.

[0051] In some embodiments, the three business processes implemented in this application are as follows: (1) Data writing: Sensor data is compressed by the edge computing node and transmitted to the TSDB storage engine. After the data ingestion module parses the data, it triggers the dictionary update logic. The index builder writes sparse metadata into the composite index. Finally, the data segment and dictionary version are atomically stored in the database storage layer. (2) Data query: The query request obtained by the application service layer is passed to the query optimizer through the API interface. The storage location and dictionary version of the target data segment are quickly located through the composite index. The compression sensing executor dynamically adjusts the decompression strategy according to the query accuracy requirements. The reconstructed data is returned to the application service layer through the API. (3) Update / adjustment process: The segment dictionary manager monitors the data sparsity changes in real time and sends a dictionary update request to the dictionary training library. The query optimizer converts the user query semantics (such as accuracy and time range) into control instructions of the compression sensing executor to realize the dynamic adjustment of the encoding parameters.

[0052] The following is combined with Figure 3 The data processing system in the embodiments of this application will be described in detail. For example... Figure 3 As shown, the data processing system mainly includes an edge data preprocessing layer, a time-series database core processing layer, and an application service layer, which are described below.

[0053] (a) Edge data preprocessing layer, which mainly implements edge data preprocessing and lightweight coding.

[0054] Specifically, the processing operations of the edge data preprocessing layer can include: (1) Sensor data acquisition and calibration.

[0055] Sensor nodes can acquire signals such as temperature, pressure, and vibration in real time through physical interfaces, generating raw data points with nanosecond-level timestamps. Edge computing nodes collect data through polling or interrupt methods, first performing timestamp calibration. For multi-sensor clusters, a broadcast-based clock synchronization algorithm can be used to eliminate clock skew between sensor nodes, ensuring the consistency of the data time series. For outliers in sensor data, the 3σ criterion can be used to filter data points that significantly deviate from the mean (such as out-of-range values ​​caused by sensor malfunctions), preventing invalid data from entering subsequent processing.

[0056] (2) Dynamic sparsity detection and pattern recognition.

[0057] The sparsity detection module employs a sliding window technique with a configurable window length. It calculates the sparsity *k* over each time window, where *k* = the number of non-zero elements / the total amount of data in the window. To identify dynamic sparsity scenarios, the sparsity detection module maintains historical sparsity records for the previous three time windows. When the sparsity of the current time window deviates from the historical mean by more than a certain threshold, it is marked as a "sparse mode mutation," triggering the edge coding module to adjust its compression strategy. The edge coding module is a core submodule of the lightweight coding module and is not included in the... Figure 3 The annotation is separate and its function is embedded in the lightweight encoding module. The above-mentioned compression adjustment strategy is to adjust the dimension of the observation matrix. In addition, the periodic changes in sparsity can be analyzed by Fourier transform to provide a predictive basis for dictionary updates in the database layer.

[0058] (3) Perform sparse coding on the corrected sensor data.

[0059] The lightweight encoding module can use the Orthogonal Matching Pursuit (OMP) algorithm to solve for sparse coefficients, and reduce computational complexity by limiting the number of iterations and optimizing the atomic matching strategy. The specific encoding process includes: S31: Dynamically generate the observation matrix. Specifically, the number of rows M of the observation matrix can be determined based on the real-time sparsity k. α is the row number coefficient of the observation matrix. α is an empirical constant verified experimentally based on the characteristics of sensor data, typically ranging from 1.2 to 1.5, used to balance compression ratio and reconstruction accuracy. The observation matrix can be a sparse random projection matrix, meaning it only needs to store the positions and values ​​of non-zero elements, reducing matrix storage and multiplication operations.

[0060] S32: The sensor data is mapped to sparse coefficients using the current dictionary. The specific iterative process is as follows: the default maximum number of iterations is 1.2k. In each iteration, the inner product of dictionary atoms and residuals (the difference between the reconstructed data of the current iteration and the original input data) is calculated. Only the top 20% of atoms with the absolute value of the inner product are selected for updating. The update process is implemented in two steps: first, the sparse coefficients corresponding to the currently selected atoms are solved based on the least squares method; second, the residuals are updated, i.e., new residual = old residual - selected atom × updated sparse coefficient. Finally, the mapped sparse coefficients are obtained through multiple rounds of iteration. To avoid redundant calculations, the dictionary required by the OMP algorithm comes from: a basic dictionary version pre-trained based on historical monitoring data of the same type of sensor in the time-series database dictionary training library; when the edge computing node starts for the first time, it obtains the initial dictionary from the dictionary training library through the edge network; after the database layer dictionary is updated, the edge computing node obtains the updated dictionary content through an incremental synchronization mechanism to ensure that the dictionary version is consistent with the database. That is, the dictionary is not generated autonomously by the edge, but is obtained by the dictionary training library based on the initial dictionary through optimization and iteration.

[0061] S33: Compressed data segment encapsulation: The non-zero elements of the sparse coefficients are stored separately, and metadata such as the start time of the time window, the initial dictionary version, and the sparsity are added to form binary data, which greatly reduces the number of bytes transmitted compared to the original data.

[0062] (ii) The core processing layer of the time series database mainly realizes data storage and query through the core technology of the time series database storage engine, involving the following related operations.

[0063] (1) Segmented dictionary management and transaction-level version control.

[0064] The segmented dictionary manager is a core component for implementing dynamic sparsity awareness in time-series databases. Its key implementation includes the following: Versioned dictionary storage: In the database metadata table (such as DICTIONARY_METADATA), each record contains a dictionary version number consisting of a timestamp and a serial number, an applicable time window, a set of dictionary atoms, training data features, and a checksum. Adjacent dictionary versions are stored incrementally, recording only the differences between atoms, which can significantly reduce storage overhead.

[0065] Dynamic update triggering mechanism: When a compressed data segment from the edge layer is received, the dictionary version corresponding to the current time window is first queried. If the difference between the sparsity of the data segment and the training data features of the dictionary version exceeds a threshold, or if the time since the last update exceeds a certain number of time windows, a dictionary update is triggered. The dictionary update process can be executed in a background thread of the database. Compressed data segments from the most recent certain number of time windows are extracted from the historical data cache of the dictionary training library. After decompression, a local atomic dictionary update is performed using an improved K-SVD algorithm, avoiding performance fluctuations caused by full training. The improved K-SVD algorithm is a customized improvement for the kernel-level dictionary management requirements of time-series databases. Specific improvements include at least the following: iterative optimization is performed only on dictionary atoms that do not match the current data sparsity pattern (identified by sparsity differences), without needing to update all atoms in the dictionary, thus reducing computational load; during the atom update process, metadata such as the sparsity and time window features of the current data segment are introduced to ensure that the newly generated atoms are adapted to dynamic sparse scenarios; only local historical data from the most recent N time windows are used as training data instead of the entire historical data, and historical data caching is used to quickly obtain training data, reducing the number of iterations and adapting to the real-time requirements of the database kernel.

[0066] Transaction-level binding mechanism: The creation of a new dictionary version is completed within a database transaction and bound to the data segment. Foreign key constraints ensure that the dictionary version referenced by the data segment exists in the metadata table. If a reconstruction error exceeds the threshold during the dictionary update process, the transaction is automatically rolled back, retaining the old dictionary version to ensure data consistency and service availability. For example, when the incremental learning engine detects that the reconstruction error of the new dictionary exceeds the threshold, it can trigger the following processing flow: 1) Immediately terminate the current update transaction and roll back to the previous valid dictionary version; 2) Send a dictionary version rollback instruction to the edge computing node to ensure that subsequent data encoding uses the old dictionary version; 3) Record the exception log, analyze whether the training data contains noise or outliers, automatically filter out abnormal data segments, and re-initiate the update request.

[0067] It should be noted that, to ensure the atomicity of transactions, data segment writes and dictionary version binding operations must be completed within the same database transaction to ensure consistency between the two. This leverages the ACID properties of the database to avoid inconsistencies caused by partial failures. Related checksum verification is primarily achieved by calculating the sparsity coefficient and the checksum of metadata during data segment storage and transmission, and performing integrity verification during reading to detect whether data has been corrupted due to disk errors or network interference.

[0068] (2) Construction and query optimization of time-sparse composite index.

[0069] The index builder deeply integrates traditional time-based indexes with sparse metadata, forming a three-layer index structure: The first layer is the main index layer, which is a time window index layer based on a B+ tree. It uses the start time of the time window at the millisecond level as the primary key and supports fast time range traversal.

[0070] The second layer is the auxiliary index layer. This layer extends the fields in the leaf nodes of the main index layer, embedding sparsity, dictionary version number (e.g., using a 4-byte integer), and data block storage offset composed of file system path + byte offset, forming a composite index entry to achieve joint positioning of time and sparsity characteristics.

[0071] The third layer is the statistical index layer, which can periodically generate statistical summaries of sparsity for each time window (such as mean, variance, etc.) and metadata such as dictionary version usage frequency. This metadata can be incorporated into the complete index system of the index builder together with the standardized data segment containing time window, sparsity coefficient, and initial dictionary reference. The metadata of the standardized data segment is used to bind the data segment to the dictionary version, and the metadata of the statistical index layer is used to accelerate aggregation queries. Together, they accelerate the execution efficiency of aggregation queries.

[0072] The query optimizer utilizes composite indexes to achieve a three-level optimization process: index filtering, data location, and on-demand loading. First, it pushes the sparsity filtering condition to the index layer, directly excluding index entries that do not meet the conditions, reducing the scanning of invalid data segments by over 90%. Next, based on the query time range, it pre-loads the corresponding version of the dictionary from the dictionary training library into the memory buffer, avoiding disk I / O latency during each query. For queries spanning multiple time windows, the task can be broken down into multiple subqueries executed in parallel, leveraging the database's multi-core processor to accelerate data retrieval.

[0073] (3) Storage format optimization and sparse data compression.

[0074] The TSDB storage layer employs a storage format combining columnar and sparse matrix structures, with dedicated data blocks designed for sparsity coefficients. The first step is the storage of non-zero elements. Each data block contains three arrays: a 32-bit integer index array storing the positions of non-zero elements, an array storing the element values, and an array recording the data length at each time point, which facilitates quick location of the length.

[0075] Secondly, there is dictionary-associated storage. The header of each data block can contain a 4-byte dictionary version number, which is quickly associated with the dictionary atom set in the dictionary training library through memory mapping technology. Therefore, when decompressing data, it is not necessary to load the entire dictionary, but only to obtain the difference atoms of the corresponding version.

[0076] Finally, there's the compression algorithm adaptation strategy. For extremely sparse data, run-length encoding can be used to compress the index array, further improving storage efficiency and increasing storage density compared to traditional columnar storage.

[0077] It should be noted that the sparse data compression mentioned here corresponds to a part of the compression method in this scheme. The compression scheme of this application is a composite scheme of sparse coefficient optimized storage, dictionary incremental management and index collaborative filtering. It does not rely solely on index construction, but achieves the compression target through the collaborative efforts of storing only non-zero elements of sparse coefficients, incremental dictionary storage and index layer filtering of invalid data. The core is to optimize both storage and indexing simultaneously by utilizing the characteristics of sparse data.

[0078] (4) Query-driven decompression encoding optimization.

[0079] The compression-aware executor acts as a bridge between query and storage, and can dynamically adjust the decompression strategy according to the query semantics, thereby achieving a balance between compression ratio and reconstruction accuracy.

[0080] Regarding decompression accuracy, it can be categorized from highest to lowest as follows: 1) Level 1: Lossless. This level uses the original observation matrix and a complete dictionary for reconstruction, strictly preserving all sparse coefficient details, with reconstruction error controlled within 0.1%. 2) Level 2: High-precision. This level uses 16-bit integer encoding for sparse coefficients, allowing a 0.5% reconstruction error, and offers a certain improvement in decompression speed compared to the lossless level. 3) Level 3: Low-precision. This level uses dictionary trimming and 8-bit integer encoding for sparse coefficients, allowing a 5% reconstruction error, resulting in a reduction in the amount of data after decompression, suitable for large-scale aggregation queries.

[0081] Optionally, the observation matrix can be dynamically adapted according to query requirements, i.e., the dimensions of the observation matrix can be dynamically adjusted to minimize decompression computation and I / O overhead while meeting query accuracy requirements. It should be noted that the relationship between this dynamic adaptation of the observation matrix and the lightweight encoding module in the edge data preprocessing layer dynamically adjusting the observation matrix parameters based on sparsity is as follows: the edge data preprocessing layer adjusts the compression adaptation at the data writing end, determining the initial dimensions of the observation matrix based on real-time data sparsity, with the core objective of optimizing compression efficiency and reducing storage and transmission costs at the edge; this adjustment, on the other hand, is the decompression adaptation at the query reading end, dynamically adjusting the dimensions of the observation matrix based on user query semantics, with the core objective of balancing query accuracy and decompression efficiency. Furthermore, the sparse data uploaded by the edge layer is stored in a composite index. During this adjustment, the sparse data is read through the index, and the dimensions of the observation matrix are calculated in conjunction with the query accuracy requirements, achieving parameter coordination between writing and querying.

[0082] Optionally, upon receiving a user's query request, the system can first obtain the query time range and data sparsity parameters, and then adjust the dimensions of the observation matrix in real time based on these parameters. The query time range can be determined by parsing the time condition in the query statement. Data sparsity can be obtained through statistical analysis or real-time calculation of the stored data, such as calculating the proportion of non-zero sparsity coefficients in the data. Based on preset dimension adjustment rules or algorithms, and combined with the obtained query time range and data sparsity parameters, a suitable observation matrix dimension is calculated. For example, the observation matrix dimension adjustment strategy can be flexibly configured through a configuration file. The configuration file contains the correspondence between different query time ranges and data sparsity and the observation matrix dimensions, as well as related parameter settings, such as the dimension adjustment threshold and step size. The correspondence between different query time ranges and data sparsity and the observation matrix dimensions in this configuration file can be predetermined through controlled variable experiments based on historical data samples from similar sensors. For example, firstly, the data sparsity intervals and query precision levels are defined. Then, for each sparsity interval-precision level combination, the reconstruction error and decompression computation time under different observation matrix dimensions are tested. The dimension value that meets the precision requirements and has the lowest computational cost is selected as the optimal dimension for that combination. Subsequently, the query response time and reconstruction error feedback data in actual business scenarios can be collected through the database backend performance monitoring module. This correspondence is then iteratively corrected, and a stable correspondence is finally formed and stored in the configuration file. The data processing system can read and update the configuration file in real time during operation, adapting to different application scenarios and changing requirements without modifying the code.

[0083] Optionally, this solution can optimize the caching mechanism. This part of the implementation requires maintaining a variable-capacity cache for recently accessed dictionary versions and data blocks; that is, maintaining a variable-capacity cache space to store recently accessed dictionary versions and data blocks. The dictionary version cache mainly stores optimized dictionaries at different precision levels so that the required dictionary resources can be quickly obtained during decompression. The data block cache stores frequently accessed data blocks to reduce the number of times data is read from the storage medium. The cache capacity is dynamically adjusted according to the system's memory usage and data access frequency. When system memory is sufficient and data access frequency is high, the cache capacity is automatically expanded to store more hot data; when system memory is tight, the cache capacity is appropriately reduced to free up memory resources for other processes. Furthermore, the most recent access time can be recorded for each dictionary version and data block in the cache, and low-frequency data can be evicted using the LRU algorithm.

[0084] It's important to note that the caching mechanism here does not refer to... Figure 3The dictionary training library contains historical data caching, and the two have completely different functions and service goals. The core function of the historical data cache is to store historical sensor data required for dictionary training, providing training samples for incremental learning when the segmented dictionary manager triggers dictionary updates, thus serving the dynamic dictionary update process. The caching mechanism here specifically stores recently accessed dictionary versions and frequently queried data blocks. Its core objective is to reduce dictionary loading latency and data block read I / O overhead during decompression, directly serving the query and decompression process. Furthermore, the cache capacity is dynamically adjusted based on system memory usage and data access frequency.

[0085] In some embodiments, to reduce the computing load on edge devices, the scheme in this application can adopt the following bidirectional collaborative mechanism: 1) Lightweighting on the edge side, such as limiting the maximum number of iterations of the OMP algorithm to avoid response delays caused by complex calculations on edge computing nodes, while using incremental data upload, sending only the difference data between the current window and the previous window to reduce network transmission volume; 2) Intelligentization on the database side, such as: the segmented dictionary manager predicts the sparsity pattern of future data based on the sparsity historical data uploaded from the edge, triggering dictionary updates in advance to reduce the encoding waiting time of edge devices; for disconnected edge computing nodes, the database automatically caches the metadata of unreceived data segments, and prioritizes requesting missing data after the connection is restored to ensure the integrity of the time series.

[0086] In summary, the solution presented in this application, through improvements to the time-series database kernel, indexing system, compression strategy, and edge collaboration mechanism, effectively addresses the core issues of existing technologies in terms of dictionary adaptability, query efficiency, storage optimization, and edge adaptability. Compared to the closest existing technologies (such as database compression schemes based on fixed dictionaries), it has the following significant technical advantages: (1) The following advantages are achieved through the incremental dictionary learning mechanism at the database kernel level: First, there's real-time adaptability. The segmented dictionary manager monitors data sparsity in real time. When a significant change in the sparsity pattern is detected, it proactively extracts the most recent historical data segments from the time-series database and updates only dictionary atoms that don't match the current data using an incremental learning algorithm, rather than training the entire dataset. This on-demand update strategy enables the dictionary to quickly respond to dynamic sparse scenarios, ensuring consistency between data and dictionary versions, avoiding encoding errors from fixed dictionaries, and mitigating the lag of fixed-frequency updates.

[0087] Secondly, there's transaction-level consistency. The process of binding dictionary versions to data segments is embedded in the database transaction system, ensuring atomic operations between the new dictionary version and data storage. When a dictionary update fails, transaction rollback can restore the most recent valid version, avoiding dictionary matching errors during decompression and significantly improving the reliability of data reconstruction.

[0088] (2) The query path is fundamentally changed by constructing a time-sparse composite index system. First, the sparsity filtering conditions are pushed down to the index layer, and the sparse metadata embedded in the index is used to quickly exclude data segments that do not meet the conditions, avoiding the inefficient process of full loading and then memory filtering in traditional solutions. This optimization means that when processing time range queries containing sparse conditions, the database does not need to read and decompress a large amount of invalid data, significantly reducing I / O operations and memory usage. Second, there is a three-layer index structure. The main index layer uses a B+ tree to quickly locate the time window, the auxiliary index layer stores sparsity and dictionary version information, and the statistical index layer maintains the sparse feature summary of each time window, enabling the query optimizer to generate a more accurate execution plan. When necessary, it can be quickly calculated directly through the index summary, avoiding scanning data segments row by row, and greatly improving the execution efficiency of complex queries.

[0089] (3) The storage engine deeply integrates with the characteristics of sparse data. Specifically, the solution in this application abandons the traditional columnar storage method of recording all data, and only stores the positions and values ​​of non-zero elements with sparse coefficients. Combined with the dictionary incremental update technology, it fundamentally reduces the amount of data stored. For dynamic sparse scenarios, this storage format can adapt to data characteristics and avoid the problem of "invalid atomic redundant storage" in fixed dictionary schemes. In addition, the compression encoding parameters are deeply bound to the database storage strategy, so that when writing data, the storage format is dynamically selected according to the real-time sparsity, and when reading data, the decompression strategy is automatically matched according to the query requirements, so as to achieve a balance between storage efficiency and query performance.

[0090] (4) By leveraging lightweight edge processing and intelligent database collaboration, the system load is rationally allocated. Specifically, edge computing nodes employ an approximate greedy algorithm to solve for sparse coefficients, and by limiting the number of iterations and optimizing atomic matching strategies, the complexity of encoding computation is effectively reduced. This lightweight design enables the encoding process to run efficiently on resource-constrained edge devices, avoiding data processing delays or abnormal device power consumption caused by complex calculations. The time-series database proactively undertakes high-load tasks such as dictionary training and version management, while edge nodes only need to upload sparse features and compressed data segments. When the edge device loses connection with the database, the local caching mechanism ensures that data is not lost. After the connection is restored, the database automatically synchronizes the missing data segments, realizing the corresponding collaborative mode and significantly improving the overall reliability and scalability of the system.

[0091] It should be noted that the sensor data in this application embodiment can be data collected in fields such as industrial IoT, smart cities, and smart grids. The improved time-series database in this application embodiment can be deployed on physical machines, virtual machines, containers, private clouds, and / or public clouds, providing secure, stable, and reliable data storage and management services for core systems in key industries, internet business systems, and government and enterprise business systems.

[0092] This application also provides a readable storage medium storing a program or instructions thereon. When the program or instructions are executed by a processor, they can implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0093] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0094] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0095] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0096] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a service classification device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0098] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized by, include: Sensor data is collected, a target dictionary and a target observation matrix are determined, and the sensor data is compressed based on the target dictionary and the target observation matrix to obtain a compressed data segment; wherein, the compressed data segment includes the time window information and sparsity information of the sensor data; Based on the time window information and the sparsity characteristic information, the compressed data segment is stored according to the index structure pre-built in the time series database; wherein, the index structure includes a main index layer and an auxiliary index layer, the main index layer is indexed by the time window, the auxiliary index layer is the leaf layer of the main index layer, and the auxiliary index layer is indexed by the sparsity characteristic information; Upon receiving a user's query request, the system determines the target time range and target sparsity based on the query request. Then, based on the target time range, target sparsity, and the storage information in the time series database, it obtains the target compressed data and decompresses the target compressed data to obtain the target data.

2. The method of claim 1, wherein, The determination of the target dictionary and target observation matrix includes: The target dictionary is determined based on the pre-acquired initial dictionary and the dictionary update content synchronized from the dictionary training library; The number of rows in the target observation matrix is ​​determined based on the sparsity of the sensor data, and the target observation matrix is ​​determined based on the number of rows.

3. The method of claim 1, wherein, The compressed data segment is stored in the time-series database in the following format: a header, a first array, a second array, and a third array; wherein, the header is used to store the version number of the target dictionary, the first array is used to store the position of the non-zero element of the sparse coefficient corresponding to the sensor data, the second array is used to store the value of the non-zero element of the sparse coefficient corresponding to the sensor data, and the third array is used to store the data length of the sensor data.

4. The method according to claim 1 or 2, characterized in that, The method further includes: Under the condition of satisfying the first condition, the sensor data within the N time windows closest to the current time are used as training data, and the training data is used to iteratively optimize the dictionary atoms in the current dictionary that do not match the current data sparsity pattern to obtain the updated dictionary content; where N is an integer greater than 1. The first condition includes at least one of the following: The difference between the sparsity characteristics of the compressed data segment received at the current time and the training data features of the dictionary version corresponding to the compressed data segment exceeds a first threshold. The interval between the current time and the last dictionary update time exceeds the second threshold.

5. The method of claim 1, wherein, The index structure also includes a statistical index layer, which stores a summary of the data sparsity for each time window and the frequency of dictionary version usage. The statistical index layer is used to assist in querying and obtaining the target compressed data.

6. The method according to claim 1, characterized in that, The step of determining the target time range and target sparsity based on the query request includes: Parse the query request to determine the target time range to be queried; Based on the stored information in the time series database, the target sparsity corresponding to the target time range to be queried is determined.

7. The method according to claim 1, characterized in that, Before decompressing the target compressed data, the method further includes: Based on the pre-configured correspondence between the query time range and data sparsity and the observation matrix dimension, the target observation matrix dimension corresponding to the target time range and target sparsity to be queried is determined, and the target observation matrix is ​​constructed based on the target observation matrix dimension. The decompression of the target compressed data includes: The target compressed data is decompressed using the target observation matrix.

8. The method according to claim 1, characterized in that, Before decompressing the target compressed data, the method further includes: Based on the target time range to be queried, load the corresponding version of the dictionary from the dictionary training library; The decompression of the target compressed data includes: The target compressed data is decompressed using the dictionary obtained from the loading process.

9. A data processing system, characterized in that, include: A data preprocessing layer is used to collect sensor data, determine the target dictionary and target observation matrix, and compress the sensor data based on the target dictionary and target observation matrix to obtain compressed data segments; wherein, the compressed data segments include the time window information and sparsity information of the sensor data; The core processing layer is used to store the compressed data segment according to the time window information and the sparse characteristic information, and according to the index structure pre-built in the time series database. The index structure includes a main index layer and an auxiliary index layer. The main index layer is indexed by the time window, and the auxiliary index layer is the leaf layer of the main index layer. The auxiliary index layer is indexed by the sparse characteristic information. The application service layer is used to receive user query requests; The core processing layer is also used to: determine the target time range and target sparsity to be queried according to the query request; obtain target compressed data according to the target time range and target sparsity and the storage information in the time series database; and decompress the target compressed data to obtain the target data.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1 to 8.

11. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 8.

Images