Monitoring data compression and storage method and system driven by time sequence database

Abstract

The invention relates to the technical field of data compression and storage, and discloses a monitoring data compression and storage method driven by a time sequence database, comprising the following steps: S1, preprocessing multi-source monitoring data, and constructing a dynamic tensor comprising a timestamp, a numerical index and a multi-dimensional label; s2, performing dynamic dimension reduction processing on the dynamic tensor to generate a core tensor and a multi-dimensional factor matrix; s3, based on the core tensor and the multi-dimensional factor matrix, determining a compression parameter, a decompression parallelism degree and an index granularity through a joint optimization model; and S4, performing hierarchical coding on residual data generated by the dynamic dimension reduction processing to generate a lightweight residual coding result. Through a dynamic tensor decomposition and incremental updating technology, low storage overhead and real-time dimension expansion capability of streaming monitoring data are realized, the problems of calculation redundancy and storage expansion caused by the fact that the streaming monitoring data cannot adapt to dynamic newly-added tags are solved, and meanwhile, frequent reconstruction cost caused by data dynamic expansion is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression and storage, and in particular to a monitoring data compression and storage method and system driven by a time series database. Background Art

[0002] With the rapid development of the Internet of Things and the Industrial Internet, the time series data generated by the monitoring system presents the typical characteristics of high dimensionality, high cardinality labels, and dynamic streaming growth. Traditional time series data compression and storage technologies are mainly based on scalar or matrix models (such as Delta encoding and dictionary compression). When faced with complex scenarios with multi-dimensional label associations, there are problems such as redundant dimensional storage and insufficient dynamic expansion capabilities. For example, although the static tensor decomposition method used in the existing technology can eliminate some redundancy, it cannot adapt to the dynamic addition of labels or the streaming expansion of data, resulting in frequent global re-decomposition and a surge in computational overhead. In addition, the existing compression algorithms mostly optimize a single indicator in isolation (such as compression rate or decompression speed), lack of global consideration of multi-objective collaboration in resource-constrained environments, and parameter solidification can easily lead to an imbalance between edge bandwidth and cloud storage.

[0003] Existing residual coding schemes typically employ uniform quantization or fixed bit-width allocation, ignoring the varying importance of data features to downstream tasks. This results in loss of valuable information and inefficient bandwidth usage. For example, critical equipment monitoring data and general environmental data contribute significantly to fault prediction, yet traditional methods still compress them with the same bit-width, resulting in both accuracy and efficiency losses.

[0004] In the area of indexing and query optimization, existing technologies often rely on single time indexes or tag inverted lists, making it difficult to support efficient joint queries across multi-dimensional tag combinations and time ranges. In particular, in high-cardinality tag scenarios, traditional index structures are prone to excessive fragmentation and excessive query I / O overhead due to fixed sharding granularity, making them unable to adapt to dynamically changing resource constraints. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a monitoring data compression and storage method and system driven by a time series database, which solves the systematic problems of the technology in dynamic dimensional expansion, multi-indicator collaborative optimization, feature-sensitive compression and cross-dimensional efficient query.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a monitoring data compression and storage method driven by a time series database, comprising the following steps: S1. Preprocess multi-source monitoring data and construct a dynamic tensor containing timestamps, numerical indicators and multi-dimensional labels; S2. Perform dynamic dimensionality reduction processing on the dynamic tensor to generate a core tensor and a multidimensional factor matrix; S3. Based on the core tensor and the multidimensional factor matrix, determine compression parameters, decompression parallelism, and index granularity through a joint optimization model; S4, performing hierarchical coding on the residual data generated by the dynamic dimensionality reduction process to generate a lightweight residual coding result; S5. Constructing a multidimensional joint index based on the layered encoding result, and storing the compressed data and metadata; S6. According to the query request, locate the data through the multi-dimensional joint index and reconstruct the output result.

[0007] Preferably, the step of preprocessing the multi-source monitoring data includes: Divide the original data into preset time windows and align the timestamps to the window start time; Map multidimensional labels to integer indices and build a label-index mapping table; Aggregate numerical indicators by label dimension to generate N-order dynamic tensors.

[0008] Preferably, the step of performing dynamic dimensionality reduction processing on the dynamic tensor includes: Perform tensor decomposition on the initial tensor to generate the core tensor and orthogonal factor matrix; When new labels or data are added, the factor matrix is locally updated by the incremental update algorithm, and the incremental update satisfies: ΔU (n) =E (n) ·Φ (n)T ,Φ (n) =(U (n)T U (n) ) -1 U (n)T ; Among them, ΔU (n) is the factor matrix increment, E (n) is the error matrix.

[0009] Preferably, the objective function of the joint optimization model is: min(λ1·compression ratio term +λ2·decompression time term +λ3·query I / O overhead term); The query I / O overhead item and the index granularity satisfy a nonlinear relationship: Query I / O cost item = γlogγ; γ is the index granularity parameter, and λ1, λ2, and λ3 are weight coefficients.

[0010] Preferably, the step of performing hierarchical coding on the residual data generated by the dynamic dimensionality reduction process includes: Calculate the residual between the dynamic tensor and the dimensionality reduction result at the edge; In the cloud, the residual quantization bit width is dynamically allocated according to the importance of data features. Residuals with high importance are allocated high bit width, and residuals with low importance are allocated low bit width.

[0011] Preferably, the step of constructing a multidimensional joint index from the layered coding results includes: Construct label inverted index based on factor matrix row number; Build a B+ tree index based on timestamp; The index granularity is dynamically adjusted according to the output parameters of the joint optimization model.

[0012] Preferably, the steps of locating data using a multi-dimensional joint index and reconstructing the output result include: Calculate the approximate result using the core tensor and sub-matrix product: Among them, S N is the set of factor matrix row numbers that meet the query conditions; The decoded residual data is superimposed to generate the exact result.

[0013] Preferably, the dynamically allocating residual quantization bit width includes: Calculate the gradient importance of the residual element to the loss function through the surrogate model; Assign high or low bit width based on the gradient importance threshold.

[0014] Preferably, the incremental update algorithm includes: Only the local factor matrix blocks corresponding to the newly added label dimensions are updated; Iteratively optimize the core tensor via the least squares method.

[0015] The monitoring data compression storage system driven by a time series database includes: Data preprocessing module for constructing dynamic tensors; Dynamic dimensionality reduction module, used to generate core tensors and factor matrices; Joint optimization module for determining compression, decompression and indexing parameters; Hierarchical coding module for residual quantization and transmission; Storage and indexing module, used for data persistence and multi-dimensional index construction; Query reconstruction module, used for data positioning and accuracy reconstruction.

[0016] The present invention provides a monitoring data compression and storage method and system driven by a time series database. It has the following beneficial effects: 1. This invention achieves low storage overhead and real-time dimensional expansion for streaming monitoring data through dynamic tensor decomposition and incremental update technology. Compared to existing solutions that use global re-decomposition or static dimensionality locking, this solves the computational redundancy and storage bloat caused by the inability to adapt to dynamically added tags, while also avoiding the frequent reconstruction costs caused by dynamic data expansion.

[0017] 2. This invention leverages multi-objective joint optimization model configuration technology to achieve a global balance between compression ratio, decompression speed, and query efficiency. This addresses the resource allocation imbalance caused by isolated optimization of a single metric in existing technologies. This approach overcomes the performance bottleneck caused by traditional parameter rigidity in heterogeneous edge-cloud environments, effectively adapting to the elastic demands of resource-constrained scenarios.

[0018] 3. This invention utilizes layered residual coding and feature-aware quantization techniques to significantly reduce transmission overhead while ensuring data reconstruction accuracy. Unlike the coarse-grained processing of residual data in existing techniques, this method dynamically allocates bit width based on gradient importance, solving the redundant storage and bandwidth issues caused by traditional methods that ignore feature sensitivity.

[0019] 4. This invention achieves rapid data location across tag and time dimensions through multi-dimensional joint indexing and dynamic granularity adjustment. Compared to existing designs that rely on a single index or fixed sharding strategy, it overcomes the shortcomings of inefficient high-cardinality tag queries and excessive latency in time range searches, while also avoiding the waste of storage resources caused by excessive index fragmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] Example: Please see the attached Figure 1 , an embodiment of the present invention provides a monitoring data compression and storage method driven by a time series database, comprising the following steps: S1, preprocessing multi-source monitoring data to construct a dynamic tensor including timestamps, numerical indicators and multi-dimensional labels; The implementation process of data preprocessing and dynamic tensor construction is as follows: First, we receive monitoring data streams from multiple source devices. The data includes timestamps, numerical indicators, and multi-dimensional labels. Since the timestamps of the original data may have different sampling frequencies, we adopt a time window block alignment strategy. Specifically, we set the time window length to Δt and align the timestamps to the window starting point closest to it. The time alignment formula is defined as: Among them, t epoch is the time reference point (such as the starting point of Unix time), Δt is the preset window length, t aligned : Aligned timestamp. This alignment operation unifies the time dimension of the data and provides structured input for subsequent tensor decomposition.

[0023] Subsequently, the multidimensional tags are encoded and mapped. Since tags such as device IDs and regions have high cardinality, directly storing the original tags would result in dimensionality explosion. For example, a tag-index mapping table M is constructed to map each unique tag combination to an integer index.

[0024] For example, the label combination label combination (device i d=001,region=A) is mapped to tensor coordinates (i=1,j=1). The mapping table is implemented through a hash function or table lookup method, which can convert high-dimensional discrete labels into low-dimensional continuous indices to solve the problem of label sparsity.

[0025] Finally, the numerical indicators are aggregated by the label dimension to construct a dynamic tensor. For data points with the same label combination in the same time window, the numerical indicators are aggregated into a single value. For example, the mean or accumulation is used as the aggregation function to generate an N-order tensor. Its element values are defined as: Among them, K is the number of data points corresponding to the label combination in the time window, v k is the value of the kth data point, X: N-order dynamic tensor, dimensions I1, I2, ..., I N The cardinality of different label types. This aggregation operation can eliminate redundant data within the time window while preserving the semantic relevance of multi-dimensional labels.

[0026] Time window blocking provides time-aligned data blocks for dynamic tensor decomposition, avoiding decomposition errors caused by scattered timestamps; label encoding mapping reduces storage overhead through dimensionality reduction processing and lays the foundation for the dimensional scalability of subsequent tensor decomposition; numerical aggregation operations convert streaming data into structured tensors to adapt to the input requirements of the tensor decomposition algorithm.

[0027] Through the above steps, the spatiotemporal alignment, label dimension compression and numerical aggregation of the original monitoring data can be achieved, and structured input suitable for dynamic tensor decomposition can be generated, providing a basis for subsequent dimensionality reduction processing.

[0028] S2. Perform dynamic dimensionality reduction on the dynamic tensor to generate a core tensor and a multidimensional factor matrix; The implementation process of dynamic incremental tensor decomposition dimensionality reduction is as follows: First, tensor decomposition is performed on the dynamic tensor X generated in step S1 to generate a low-rank approximate representation. For example, the Tucker decomposition model is used to decompose the original tensor into core tensors And the multidimensional factor matrix U(n)U(n). The decomposition goal is to minimize the reconstruction error, which is defined in mathematical form as: in, For the core tensor (R N < N ), is the n-th dimension factor matrix, satisfying the orthogonality constraint U (n)T U (n) = I. This decomposition can compress the original data through low-rank representation and eliminate redundant information between multi-dimensional labels.

[0029] When the tensor dimension is expanded due to the addition of new labels or data blocks, the incremental update algorithm is used to locally update the factor matrix to avoid the computational overhead of global re-decomposition. Specifically, assuming that the new dimension index is i′∈I n , first calculate the expansion matrix of the new data block The error matrix with the original decomposition result: in, is the matrix form of the original core tensor expanded along the nth module, is the matrix of the newly added data block expanded along the nth module, The matrix form of the original core tensor expanded along the nth mode. Subsequently, the factor matrix is updated by the projection matrix Φ(n): ΔU (n) =E (n) ·Φ (n)T ,Φ (n) =(U (n)T U (n) ) -1 U (n)T , where ΔU (n) : Incremental update amount of the factor matrix.

[0030] ​The incremental update is only for the affected local factor matrix block, which can significantly reduce the amount of calculation. Finally, the updated factor matrix is fixed and the core tensor is iteratively optimized by alternating least squares method. The objective function is: ||·|| F : The Frobenius norm of the matrix, used to measure the reconstruction error.

[0031] During the optimization process, each slice of the core tensor is updated modulo-modulo until the residual converges.

[0032] The initial Tucker decomposition provides the basic low-rank structure for subsequent incremental updates. The orthogonality constraint ensures the linear independence of the factor matrix and avoids dimensional redundancy. The incremental update algorithm dynamically adapts to the dimensional expansion of new data or labels through local error calculation and projection operations, forming a closed loop with the dynamic tensor construction in step S1. The iterative optimization of the core tensor ensures that the decomposition result still maintains the minimum reconstruction error after dynamic expansion, providing a stable input for the residual coding in step S4.

[0033] Through dynamic incremental tensor decomposition, multi-dimensional label data can be compressed with a low-rank structure, reducing storage overhead; the incremental update mechanism avoids the computational burden of global re-decomposition and adapts to the dynamic changes of streaming data; core tensor optimization ensures the accuracy and stability of the decomposition results, providing a reliable foundation for subsequent layered coding.

[0034] S3. Based on the core tensor and multidimensional factor matrix, the compression parameters, decompression parallelism, and index granularity are determined through a joint optimization model. First, based on the core tensor generated in step S2 With the factor matrix {U (n) A joint optimization model is constructed. The model aims to balance compression ratio, decompression speed, and query efficiency. Three sets of optimization variables are defined: compression parameter α, decompression parallelism β, and index granularity γ. For example, compression parameter α is mapped to the quantization bit width, decompression parallelism β is mapped to the number of threads or GPU cores, and index granularity γ is mapped to the B+ tree node size.

[0035] The objective function of the optimization model is defined as: min α,β,γ (λ1·f comp (α)+λ2·f decomp (β)+λ3·f query (γ)); Among them, f comp (α) represents the compression rate term, f decomp (β) represents the decompression time term, f query(γ) represents the query I / O cost item. For example, the function of each sub-item is defined as: f query (γ) = γlogγ; Where λ1, λ2, and λ3 are preset weight coefficients, reflecting the priority of different optimization objectives.

[0036] Furthermore, the constraints are defined as: α+β+γ≤C,α>0,β>0,γ>0; Where C is the upper limit of system resources (such as total memory or number of computing units). This constraint ensures that the parameter configuration adapts to the actual hardware resource limitations.

[0037] Subsequently, the Frank-Wolfe algorithm is used to solve the above optimization problem. The specific process includes: gradient calculation: Compute the gradient of the objective function with respect to the optimization variable: Direction search: Solve the descent direction in the feasible domain D = {(α, β, γ) | α + β + γ ≤ C}: Where D: feasible domain, defined by resource constraints This step obtains a closed-form solution through the Lagrange multiplier method.

[0038] Step size update: Determine the step size η through line search k , update the parameters: x k+1 =x k +η k (d k -x k ); Iterate the above steps until the objective function converges and output the optimal parameter α * , β * , γ * ; α: quantization bit width, which determines the compression ratio and precision loss; β: the number of parallel computing units, which affects the decompression speed; γ: the index sharding granularity, which determines the query I / O times.

[0039] The objective function design mathematically quantifies the trade-off between compression rate, decompression speed, and query efficiency, and combines it with the decomposition results of step S2 to ensure that the compression parameters adapt to the data characteristics. The constraints reflect actual resource limitations to avoid parameter configuration exceeding hardware capabilities. The Frank-Wolfe algorithm achieves efficient solution through gradient direction search and feasible domain projection, providing parameter input for residual coding in step S4 and index construction in S5.

[0040] Through the joint optimization model, the optimal parameter configuration can be automatically determined under given resource constraints, balancing data compression rate, decompression speed and query efficiency, providing a quantitative basis for subsequent hierarchical coding and index construction.

[0041] S4, performing hierarchical coding on the residual data generated by the dynamic dimensionality reduction process to generate a lightweight residual coding result; First, dynamic tensors are calculated at the edge The dimensionality reduction result of step S2 The residual is defined as the difference between the original tensor and the low-rank approximation result: The residual contains the high-frequency details lost during the dimensionality reduction process and is used to restore data accuracy in the cloud. For example, the residual R, the core tensor And the factor matrix {U (n)}Upload to the cloud to form a layered encoded data stream.

[0042] Subsequently, feature-aware quantization is performed in the cloud. Since the influence of different elements in the residual data on the model accuracy varies significantly, the gradient importance is used to dynamically allocate the quantization bit width. Specifically, through the lightweight proxy model Calculate the residual element for the preset loss function Gradient importance of: Among them, r i is the residual element, g i is its gradient importance, R: residual tensor, reflecting the information loss in the dimensionality reduction process. According to the gradient importance threshold θ, the quantization bit width is dynamically allocated: g i : The importance of the residual element gradient, which measures its impact on the model accuracy; b i : Dynamically allocated quantization bit width determines data accuracy and compression rate Where b high with b low The high bit width and low bit width are preset respectively. This allocation strategy ensures that the high-importance residuals retain higher precision, while the low-importance residuals are compressed and stored.

[0043] Furthermore, non-uniform quantization is performed on the residual. For example, piecewise uniform quantization is used to divide the residual block into sub-regions, and the quantization parameters are adjusted according to the statistical characteristics of each sub-region. The quantization formula is defined as: Among them, μ and σ are the mean and standard deviation of the residual block respectively, and Round(·) is the rounding function. Stored together with metadata (mean, standard deviation, bit width allocation table).

[0044] The residual calculation is based on the decomposition result of step S2, retaining the difference information between the original data and the low-rank approximation, providing a basis for subsequent accuracy recovery; the gradient importance calculation associates data features with model requirements through a proxy model to ensure that the quantization strategy is adapted to downstream tasks; non-uniform quantization combines statistical characteristics with dynamic bit width allocation to reduce redundant data while retaining key features, forming a synergy with the index construction in step S5.

[0045] Through residual layered coding, lightweight computing can be achieved on the edge, and the cloud can dynamically optimize quantization strategies to balance transmission overhead and data accuracy; feature-aware quantization retains key information and adapts to different downstream task requirements; non-uniform quantization reduces storage space usage and provides compact data representation for multi-dimensional index construction.

[0046] S5. Construct a multidimensional joint index based on the hierarchical encoding result and store the compressed data and metadata; First, based on the quantized residual generated in step S4 and metadata (core tensors Factor matrix {U (n)}、Quantization parameter {b i}), build a hierarchical storage structure. Exemplarily divide the data into metadata layer and data layer: Metadata layer: storing core tensors Factor matrix {U (n)}、Quantization parameter {b i}, used to quickly reconstruct the approximate result; Data layer: stores quantized residuals and the original timestamp sequence, organized into blocks by time windows.

[0047] Furthermore, a multi-dimensional joint index is constructed to speed up the query. The index includes a label inverted index and a time B+ tree index: Label inverted index: Mapping of label combinations to data blocks is established based on the row numbers of the factor matrix. For example, the label combination (device i d=001,region=A) corresponds to the factor matrix U (1) The first row of U (2) Line 1 maps to the core tensor slice An index entry is defined as: Index(·): Label inverted index item, mapping label combinations to data blocks.

[0048] Time B+ tree index: divides data blocks by timestamp range, and each leaf node stores the physical address of the data block corresponding to the time window. The index key is the time window start time t start ,satisfy: Wherein, Δt is the time window length preset in step S1.

[0049] t start : The start time of the time window, used for B+ tree index keys.

[0050] Finally, the index structure is dynamically adjusted based on the index granularity parameter γ output in step S3. For example, the index granularity γ controls the size of the B+ tree node and the scale of the inverted index fragments: When γ is small, the B+ tree nodes contain fewer time windows, reducing the number of query I / Os; When γ is large, the inverted index shards cover more label combinations and reduce the height of the index tree.

[0051] Hierarchical storage reduces redundant data access by separating metadata from quantized residuals, adapting to the hierarchical encoding output of step S4; the label inverted index is directly associated with the factor matrix row number of step S2 to achieve fast filtering of multi-dimensional labels; the temporal B+ tree index is aligned with the time window of step S1 to ensure the efficiency of time range queries; the dynamic index granularity is based on the optimization parameters of step S3, balancing storage overhead and query performance.

[0052] γ: Index granularity parameter that controls the trade-off between storage and query efficiency of the index structure.

[0053] Through hierarchical storage and multi-dimensional joint indexing, efficient data organization and rapid location can be achieved; the label inverted index supports precise filtering of multi-dimensional label combinations, and the time B+ tree index accelerates time range queries; dynamic index granularity adapts to different resource constraints and optimizes storage and query performance.

[0054] S6. According to the query request, locate the data through the multi-dimensional joint index and reconstruct the output results.

[0055] First, the time range and multi-dimensional label conditions in the query request are parsed, and the target data block is located using the multi-dimensional joint index constructed in step S5. For example, for the label condition region = A, the label inverted index is used to obtain the corresponding factor matrix row number set S2, and the time B+ tree index is used to locate the data block that meets the time window [t start ,t end ] is the physical address of the data block.

[0056] Then, based on the core tensor generated in step S2 With the factor matrix subset Calculate the approximate result. For example, the approximate reconstruction formula is defined as: in, The row number in the N-th dimension factor matrix belongs to the set S N submatrix of ; S N : The set of factor matrix row numbers that meet the query conditions in the Nth dimension.

[0057] This approximate calculation uses low-rank structures to quickly generate preliminary results and reduce real-time computing overhead.

[0058] Furthermore, for queries that require accurate results, the quantized residual of step S4 is superimposed First, perform the inverse quantization operation: Where σ and μ are the statistical parameters of the quantized residual block, b i The bit width is dynamically allocated. The residual after inverse quantization This is superimposed with the approximate result to produce the exact output: in: The approximate result tensor is generated by multiplying the core tensor with the submatrix; The exact result tensor contains the complete information after residual superposition.

[0059] Multi-dimensional index positioning combines the label inverted index and time B+ tree index of step S5 to achieve fast data filtering and physical address mapping; the approximate result calculation relies on the decomposition structure of step S2 and uses the low-rank characteristics to accelerate the response; the residual superposition mechanism is based on the layered coding data of step S4 to restore high-frequency detail information to meet accuracy requirements.

[0060] Through a hierarchical response mechanism of approximate calculation and residual superposition, it is possible to dynamically balance response speed and result accuracy according to query requirements; multi-dimensional index positioning reduces irrelevant data access and improves query efficiency; inverse quantization operations restore residual details to ensure that the final output is consistent with the original data.

[0061] Please see the attached Figure 2 , an embodiment of the present invention provides a monitoring data compression storage system driven by a time series database, including: Data preprocessing module for constructing dynamic tensors; Spatiotemporal alignment: Divide the data into preset time windows (e.g., Δt = 1 minute) and align the timestamps to the start of the window. The formula is:

[0062] Tag encoding: Map device IDs, regions, and other tags to integer indexes through hashing, and construct a tag-index mapping table M.

[0063] Tensor aggregation: Aggregate numerical indicators (mean or accumulation) by label dimension to generate N-order dynamic tensors

[0064] Provide structured input for the dynamic dimensionality reduction module to eliminate the temporal discreteness and label sparsity of the original data.

[0065] Dynamic dimensionality reduction module, used to generate core tensors and factor matrices; Initial Tucker decomposition: Decompose the dynamic tensor into a low-rank structure: in is the core tensor, U (n) is the orthogonal factor matrix of each dimension.

[0066] Incremental update: When new data is added, the error matrix E is calculated (n) And locally update the factor matrix: ΔU (n) =E (n) ·Φ (n)T ,Φ (n) =(U (n)T U (n) ) -1 U (n)T ; The data dimension is compressed through low-rank approximation, providing dimensionality reduction results for the joint optimization module.

[0067] Joint optimization module for determining compression, decompression and indexing parameters; Objective function: Define the multi-objective optimization problem: Among them, α (quantization bit width), β (parallelism), and γ (index granularity) are optimization variables.

[0068] Solution algorithm: Use the Frank-Wolfe algorithm to iteratively update the parameters, and the gradient calculation is: The output parameters guide layered encoding and index construction, adapting to resource constraints.

[0069] Hierarchical coding module for residual quantization and transmission; Residual calculation: Generate the residual of the original tensor and the low-rank approximation: Dynamic Quantization: Calculating Residual Gradient Importance via Proxy Models Dynamically allocate bit width: It retains key features and provides lightweight data flow for storage modules.

[0070] Storage and indexing module, used for data persistence and multi-dimensional index construction; Tiered storage: metadata layer storage {U (n)}, quantization parameters; data layer stores quantization residuals

[0071] Multidimensional index: Label inverted index: mapping label combinations to factor matrix row numbers, such as device i d=001→U (1) (1,:).

[0072] Time B+ tree index: divides data blocks by time window, with key t start .

[0073] Dynamically adjust the index granularity based on the optimization parameter γ to balance storage and query performance.

[0074] Query reconstruction module, used for data positioning and accuracy reconstruction.

[0075] Approximate calculation: Generate approximate results using the core tensor and sub-matrix product:

[0076] Exact reconstruction: Dequantize the residual and add it to the approximate result:

[0077] Combined with indexes, data can be quickly located and graded to respond to different accuracy requirements.

[0078] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A time series database driven monitoring data compression and storage method, characterized in that: The following steps are involved: Preprocess multi-source monitoring data and construct dynamic tensors containing timestamps, numerical indicators, and multi-dimensional labels; Performing dynamic dimensionality reduction processing on the dynamic tensor to generate a core tensor and a multidimensional factor matrix; Based on the core tensor and the multidimensional factor matrix, compression parameters, decompression parallelism, and index granularity are determined through a joint optimization model; Performing layered coding on the residual data generated by the dynamic dimensionality reduction process to generate a lightweight residual coding result; Constructing a multidimensional joint index based on the hierarchical encoding result, and storing the compressed data and metadata; According to the query request, the data is located through the multi-dimensional joint index and the output result is reconstructed.

2. The monitoring data compression and storage method driven by a time series database according to claim 1 is characterized in that: The step of preprocessing the multi-source monitoring data includes: Divide the original data into preset time windows and align the timestamps to the window start time; Map multidimensional labels to integer indices and build a label-index mapping table; Aggregate numerical indicators by label dimension to generate N-order dynamic tensors.

3. The monitoring data compression and storage method driven by a time series database according to claim 1, characterized in that: The step of performing dynamic dimensionality reduction processing on the dynamic tensor comprises: Perform tensor decomposition on the initial tensor to generate the core tensor and orthogonal factor matrix; When new labels or data are added, the factor matrix is locally updated using an incremental update algorithm that satisfies: D.U. (n) =E (n) ·F (n)T ,F (n) =(U (n)T U (n) ) -1 U (n)T ; Among them, ΔU (n) is the factor matrix increment, E (n) is the error matrix.

4. The monitoring data compression and storage method driven by a time series database according to claim 1, characterized in that: The objective function of the joint optimization model is: min(λ1·compression ratio term +λ2·decompression time term +λ3·query I / O overhead term); The query I / O overhead item and the index granularity satisfy a nonlinear relationship: Query I / O cost item = γlogγ; γ is the index granularity parameter, and λ1, λ2, and λ3 are weight coefficients.

5. The monitoring data compression and storage method driven by a time series database according to claim 1 is characterized in that: The step of performing hierarchical coding on the residual data generated by the dynamic dimensionality reduction process comprises: Calculate the residual between the dynamic tensor and the dimensionality reduction result at the edge; In the cloud, the residual quantization bit width is dynamically allocated according to the importance of data features. Residuals with high importance are allocated high bit width, and residuals with low importance are allocated low bit width.

6. The monitoring data compression and storage method driven by a time series database according to claim 1, characterized in that: The step of constructing a multi-dimensional joint index from the layered coding results includes: Construct label inverted index based on factor matrix row number; Build a B+ tree index based on timestamp; The index granularity is dynamically adjusted according to the output parameters of the joint optimization model.

7. The monitoring data compression and storage method driven by a time series database according to claim 1, characterized in that: The steps of locating data through multi-dimensional joint indexing and reconstructing output results include: Calculate the approximate result using the core tensor and sub-matrix product: in, is the set of factor matrix row numbers that meet the query conditions; The decoded residual data is superimposed to generate the exact result.

8. The monitoring data compression and storage method driven by a time series database according to claim 5, characterized in that: The dynamically allocating residual quantization bit width includes: Calculate the gradient importance of the residual element to the loss function through the surrogate model; Assign high or low bit width based on the gradient importance threshold.

9. The monitoring data compression and storage method driven by a time series database according to claim 3, characterized in that: The incremental update algorithm includes: Only the local factor matrix blocks corresponding to the newly added label dimensions are updated; Iteratively optimize the core tensor via the least squares method.

10. A time series database driven monitoring data compression storage system, according to any one of claims 1 to 9, characterized in that: include: Data preprocessing module for constructing dynamic tensors; Dynamic dimensionality reduction module, used to generate core tensors and factor matrices; Joint optimization module for determining compression, decompression and indexing parameters; Hierarchical coding module for residual quantization and transmission; Storage and indexing module, used for data persistence and multi-dimensional index construction; Query reconstruction module, used for data positioning and accuracy reconstruction.

Images