Time series database driven monitoring data compression storage method, system

Abstract

The application relates to the technical field of data compression storage, and discloses a time series database driven monitoring data compression storage method, which comprises the following steps: S1, preprocessing multi-source monitoring data, and constructing a dynamic tensor containing a timestamp, a numerical index and a multi-dimensional label; S2, performing dynamic dimension reduction processing on the dynamic tensor to generate a core tensor and a multi-dimensional factor matrix; S3, based on the core tensor and the multi-dimensional factor matrix, determining compression parameters, decompression parallelism and index granularity through a joint optimization model; and S4, performing hierarchical coding on residual data generated by the dynamic dimension reduction processing to generate a light residual coding result. Through dynamic tensor decomposition and incremental updating technology, low storage overhead and real-time dimension expansion capability of streaming monitoring data are realized, the problems of calculation redundancy and storage expansion caused by the incapability of the streaming monitoring data to adapt to dynamic newly-added labels are solved, and the frequent reconstruction cost caused by dynamic expansion of the data is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data compression storage, in particular to a time series database driven monitoring data compression storage method and system. BACKGROUND

[0002] With the rapid development of the Internet of Things and the Industrial Internet, the time series data generated by the monitoring system presents typical characteristics of high dimension, high cardinality label and dynamic stream growth. Traditional time series data compression storage technology is mainly based on scalar or matrix model (such as Delta encoding, dictionary compression), which has problems such as dimension redundancy storage and insufficient dynamic expansion capability when facing complex scenes with multi-dimensional label association. For example, although the static tensor decomposition method in the prior art can eliminate part of the redundancy, it cannot adapt to the dynamic addition of labels or the stream expansion of data, resulting in frequent global re-decomposition and a sharp increase in computing overhead. In addition, the existing compression algorithms mostly optimize a single indicator (such as compression rate or decompression speed) in isolation, lack global consideration of multi-objective coordination in resource-constrained environments, and the fixed parameters are easy to cause imbalance between edge bandwidth and cloud storage.

[0003] In the aspect of residual coding, the existing scheme usually adopts uniform quantization or fixed bit width allocation, ignoring the importance difference of data characteristics to downstream tasks, resulting in loss of high-value information or inefficient bandwidth occupation. For example, the contribution of key device monitoring data and ordinary environment data to fault prediction is significantly different, but the traditional method still compresses them with the same bit width, causing double loss of precision and efficiency.

[0004] In the field of index and query optimization, the existing technology mostly relies on single time index or label inverted list, and it is difficult to support efficient joint query of multi-dimensional label combination and time range. Especially in the high cardinality label scene, the traditional index structure is easy to cause excessive fragmentation or excessive query I / O overhead due to fixed fragmentation granularity, and cannot adapt to the dynamically changing resource constraints. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a time series database driven monitoring data compression storage method and system, which solves the systematic problems in dynamic dimension expansion, multi-indicator collaborative optimization, feature sensitive compression and cross-dimensional efficient query.

[0006] To achieve the above purpose, the present application realizes the following technical scheme: a time series database driven monitoring data compression storage method, comprising the following steps:

[0007] S1, preprocessing multi-source monitoring data to build a dynamic tensor containing timestamp, numerical indicator and multi-dimensional label;

[0008] S2, performing dynamic dimension reduction processing on the dynamic tensor to generate a core tensor and a multi-dimensional factor matrix;

[0009] S3, determining compression parameters, decompression parallelism and index granularity by a joint optimization model based on the core tensor and the multi-dimensional factor matrix;

[0010] S4, performing hierarchical encoding on residual data generated by the dynamic dimension reduction processing to generate a lightweight residual encoding result;

[0011] S5, constructing a multi-dimensional joint index based on the hierarchical encoding result, and storing compressed data and metadata;

[0012] S6, locating data and reconstructing output results through the multi-dimensional joint index according to a query request.

[0013] Preferably, the step of preprocessing the multi-source monitoring data comprises:

[0014] Dividing the original data according to a preset time window, and aligning the timestamp to the window start time;

[0015] Mapping the multi-dimensional label to an integer index, and constructing a label-index mapping table;

[0016] Aggregating the numerical indicators according to the label dimension to generate an N-order dynamic tensor.

[0017] Preferably, the step of dynamically reducing the dimension of the dynamic tensor comprises:

[0018] Performing tensor decomposition on the initial tensor to generate a core tensor and an orthogonal factor matrix;

[0019] When a new label or data is added, updating the factor matrix locally through an incremental update algorithm, wherein the incremental update satisfies: ΔU (n) =E (n) ·Φ (n)T ,Φ (n) =(U (n)T U (n) ) -1 U (n)T ;

[0020] Wherein, ΔU (n) is the factor matrix increment, and E (n) is the error matrix.

[0021] Preferably, the objective function of the joint optimization model is:

[0022] min(λ1·compression rate term+λ2·decompression time term+λ3·query I / O overhead term);

[0023] Wherein, the query I / O overhead term and the index granularity satisfy a nonlinear relationship:

[0024] Query I / O overhead item = γlogγ

[0025] γ is an index granularity parameter, λ1, λ2, λ3 are weight coefficients.

[0026] Preferably, the step of hierarchically encoding the residual data generated by the dynamic dimensionality reduction processing comprises:

[0027] The residual of the dynamic tensor and the dimensionality reduction result is calculated at the edge side.

[0028] The residual quantization bit width is dynamically allocated according to the importance of data characteristics in the cloud, high importance residual is allocated high bit width, and low importance residual is allocated low bit width.

[0029] Preferably, the step of constructing a multi-dimensional joint index based on the hierarchical encoding result comprises:

[0030] Constructing a label inverted index based on the row number of the factor matrix;

[0031] Constructing a B+ tree index based on the timestamp;

[0032] The index granularity is dynamically adjusted according to the output parameters of the joint optimization model.

[0033] Preferably, the step of positioning data and reconstructing the output result based on the multi-dimensional joint index comprises:

[0034] Calculating the approximate result using the core tensor and the sub-matrix product:

[0035]

[0036] Wherein, S N is a set of factor matrix row numbers that meet the query condition;

[0037] The residual data after superposition decoding generates an accurate result.

[0038] Preferably, the step of dynamically allocating residual quantization bit width comprises:

[0039] Calculating the gradient importance of residual elements on the loss function through a proxy model;

[0040] Allocating high bit width or low bit width according to the gradient importance threshold.

[0041] Preferably, the incremental update algorithm comprises:

[0042] Only the local factor matrix block corresponding to the newly added label dimension is updated;

[0043] The core tensor is iteratively optimized by the least square method.

[0044] The time series database driven monitoring data compression storage system comprises:

[0045] a data preprocessing module configured to construct a dynamic tensor;

[0046] a dynamic dimension reduction module configured to generate a core tensor and a factor matrix;

[0047] a joint optimization module configured to determine compression, decompression and indexing parameters;

[0048] a hierarchical encoding module configured to quantize and transmit residual data;

[0049] a storage and indexing module configured to persist data and construct a multi-dimensional index;

[0050] a query reconstruction module configured to locate data and reconstruct precision.

[0051] The present application provides a time series database driven monitoring data compression storage method and system, which has the following beneficial effects:

[0052] 1. The present application realizes low storage overhead and real-time dimension expansion capability of streaming monitoring data through dynamic tensor decomposition and incremental update technology. Compared with the scheme of global re-decomposition or static dimension locking in the prior art, the present application solves the problem of calculation redundancy and storage expansion caused by the inability to adapt to dynamically added labels, and avoids the frequent reconstruction cost caused by dynamic data expansion.

[0053] 2. The present application achieves global balance of compression rate, decompression speed and query efficiency based on the technology of configuring parameters of a multi-objective joint optimization model. In view of the resource allocation imbalance caused by isolated optimization of a single index in the prior art, the present application breaks through the performance bottleneck caused by parameter solidification in the traditional method in the edge-cloud heterogeneous environment, and effectively adapts to the elastic demand of resource limited scenarios.

[0054] 3. The present application adopts hierarchical residual coding and feature-aware quantization technology, which significantly reduces transmission overhead while ensuring data reconstruction accuracy. Unlike the coarse-grained processing mode of uniform coding of residual data in the prior art, the present application solves the problem of redundant storage and bandwidth occupation caused by ignoring feature sensitivity in the traditional method by dynamically allocating bit width according to gradient importance.

[0055] 4. The present application realizes fast data positioning across labels and time dimensions through multi-dimensional joint indexing and dynamic granularity adjustment technology. Compared with the design of relying on a single index or fixed sharding strategy in the prior art, the present application overcomes the defects of low query efficiency of high cardinality labels and high time range retrieval delay, and avoids the waste of storage resources caused by excessive fragmentation of indexes. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 a method flowchart of the present application;

[0057] Figure 2 System architecture diagram of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0059] Embodiment:

[0060] Please refer to the accompanying Figure 1 The embodiment of the present application provides a time series database driven monitoring data compression storage method, which comprises the following steps: S1, preprocessing multi-source monitoring data, and constructing a dynamic tensor comprising a timestamp, a numerical index and a multi-dimensional label;

[0061] The implementation process of data preprocessing and dynamic tensor construction is as follows:

[0062] Firstly, the monitoring data stream from the multi-source device is received, and the data comprises a timestamp, a numerical index and a multi-dimensional label. Since the timestamps of the original data may have different sampling frequencies, a time window block alignment strategy is exemplarily adopted. Specifically, the time window length is set as Δt, and the timestamp is aligned to the nearest window starting point. The time alignment formula is defined as:

[0063]

[0064] Wherein, t epoch is a time reference point (such as the Unix time starting point), Δt is a preset window length, and t aligned is the aligned timestamp. The alignment operation can realize the unification of the time dimension of the data, and provide a structured input for subsequent tensor decomposition.

[0065] Subsequently, the multi-dimensional label is encoded and mapped. Since the device ID, region and other labels have high cardinality characteristics, directly storing the original labels will cause dimension explosion. A label-index mapping table M is exemplarily constructed, and each unique label combination is mapped to an integer index.

[0066] For example, the label combination (device i d = 001, region = A) is mapped to the tensor coordinate (i = 1, j = 1). The mapping table is realized by a hash function or a lookup table method, which can convert high-dimensional discrete labels into low-dimensional continuous indexes, and solve the label sparsity problem.

[0067] Finally, the numerical indicators are aggregated along the label dimension to construct the dynamic tensor. For data points with the same label combination within the same time window, the numerical indicators are aggregated into a single value. Exemplarily, the mean or accumulation is used as the aggregation function to generate an N-order tensor The element value is defined as:

[0068]

[0069] where K is the number of data points corresponding to the label combination within the time window, v k is the numerical value of the kth data point, X: N-order dynamic tensor, dimensions I1, I2, …, I N corresponding to different label types. This aggregation operation can eliminate redundant data within the time window while retaining the semantic association of multi-dimensional labels.

[0070] The time window block provides time dimension alignment data block for dynamic tensor decomposition, avoiding decomposition error caused by timestamp dispersion; label encoding mapping reduces storage overhead through dimension reduction processing, and lays the foundation for dimension scalability of subsequent tensor decomposition; the numerical aggregation operation converts streaming data into structured tensors, adapting to the input requirements of tensor decomposition algorithms.

[0071] Through the above steps, the spatio-temporal alignment, label dimension compression and numerical aggregation of the original monitoring data can be realized, generating a structured input suitable for dynamic tensor decomposition, providing a basis for subsequent dimension reduction processing.

[0072] S2, performing dynamic dimension reduction processing on the dynamic tensor to generate a core tensor and a multi-dimensional factor matrix;

[0073] The implementation process of dynamic incremental tensor decomposition dimension reduction is as follows:

[0074] First, perform tensor decomposition on the dynamic tensor X generated in step S1 to generate a low-rank approximate representation. Exemplarily, the Tucker decomposition model is used to decompose the original tensor into a core tensor and a multi-dimensional factor matrix U(n)U(n). The decomposition target is to minimize the reconstruction error, which is mathematically defined as:

[0075]

[0076] where is the core tensor (R N <<I N ), is the nth-dimensional factor matrix, satisfying the orthogonality constraint U (n)T U (n) = I. This decomposition can compress the original data through low-rank representation and eliminate the redundant information between multi-dimensional labels.

[0077] When a new label or data block causes the tensor dimension to expand, the incremental update algorithm is exemplarily adopted to update the factor matrix locally, avoiding the computational overhead of global re-decomposition. Specifically, assuming that the newly added dimension index is i' ∈ I n , first calculate the error matrix of the new data block unfolding matrix and the original decomposition result:

[0078]

[0079] wherein, is the matrix form of the original core tensor along the nth mode unfolding, is the matrix of the new data block along the nth mode unfolding, is the matrix form of the original core tensor along the nth mode unfolding. Then, update the factor matrix ΔU (n) = E (n) · Φ (n)T , Φ (n) = (U (n)T U (n) ) -1 U (n)T , wherein ΔU (n) : the incremental update amount of the factor matrix.

[0080] This incremental update is only for the affected local factor matrix block, which can significantly reduce the amount of calculation. Finally, fix the updated factor matrix, and optimize the core tensor by the alternating least squares method. The objective function is:

[0081]

[0082] ||·|| F : Frobenius norm of the matrix, used to measure the reconstruction error.

[0083] During the optimization process, each slice of the core tensor is updated by mode by mode until the residual error converges.

[0084] The initial Tucker decomposition provides a low-rank structure for subsequent incremental updates, and the orthogonality constraint ensures the linear independence of the factor matrix, avoiding dimension redundancy; the incremental update algorithm dynamically adapts to the dimension expansion of the newly added data or labels through local error calculation and projection operation, forming a closed loop with the dynamic tensor construction in step S1; the iterative optimization of the core tensor ensures that the decomposition result still maintains the minimum reconstruction error after dynamic expansion, providing a stable input for the residual error coding in step S4.

[0085] By dynamic incremental tensor decomposition, multi-dimensional label data can be compressed in low-rank structure, reducing storage overhead; incremental update mechanism avoids the computational burden of global re-decomposition, and adapts to the dynamic changes of streaming data; core tensor optimization ensures the accuracy and stability of the decomposition result, providing a reliable foundation for subsequent hierarchical coding.

[0086] S3, based on the core tensor and the multi-dimensional factor matrix, determining the compression parameter, the decompression parallelism and the index granularity through a joint optimization model;

[0087] First, based on the core tensor generated in step S2 and the factor matrix {U (n)}, a joint optimization model is constructed. The objective of the model is to balance the compression rate, decompression speed and query efficiency, and three groups of optimization variables are defined: compression parameter a, decompression parallelism b, and index granularity g. Exemplarily, the compression parameter a is mapped to the quantization bit width, the decompression parallelism b is mapped to the number of threads or GPU cores, and the index granularity g is mapped to the B+ tree node size.

[0088] The objective function of the optimization model is defined as:

[0089] min α,β,γ (λ1·f comp (α)+λ2·f decomp (β)+λ3·f query (γ));

[0090] Where f comp (α) represents the compression rate term, f decomp (β) represents the decompression time term, and f query (γ) represents the query I / O overhead term. Exemplarily, each sub-term function is defined as:

[0091] f query (γ)=γlogγ;

[0092] In the formula, λ1, λ2, λ3 are preset weight coefficients, reflecting the priority of different optimization objectives.

[0093] Further, the constraint condition is defined as:

[0094] a+b+g≤C, a>0, b>0, g>0;

[0095] Where C is the upper limit of system resources (such as total memory or number of computing units). This constraint ensures that the parameter configuration adapts to the actual hardware resource limit.

[0096] Subsequently, Frank-Wolfe algorithm is used to solve the above optimization problem. The specific process includes: gradient calculation:

[0097] Compute the gradient of the objective function with respect to the optimization variables:

[0098]

[0099] Direction search: solve the descent direction in the feasible region D = {(a, b, g) | a + b + g < C}:

[0100]

[0101] Where D: feasible region, defined by resource constraints

[0102] This step obtains a closed-form solution by the Lagrange multiplier method.

[0103] Step size update: determine the step size h by line search k , update the parameters:

[0104] x k+1 = x k + h k (d k -x k );

[0105] Iterate the above steps until the objective function converges, output the optimal parameters a * , b * , g * ;

[0106] a: quantization bit width, determines compression ratio and accuracy loss, b: number of parallel computing units, affects decompression speed, g: index sharding granularity, determines query I / O frequency.

[0107] The objective function design quantifies the trade-off between compression ratio, decompression speed, and query efficiency in mathematical form, combined with the decomposition result of step S2, to ensure that the compression parameters adapt to the data characteristics; the constraint conditions reflect the actual resource limitations to avoid parameter configuration exceeding the hardware capabilities; the Frank-Wolfe algorithm realizes efficient solution through gradient direction search and feasible region projection, providing parameter input for residual encoding in step S4 and index construction in S5.

[0108] Through joint optimization model, the optimal parameter configuration can be automatically determined under given resource constraints, balancing data compression ratio, decompression speed, and query efficiency, providing quantitative basis for subsequent hierarchical encoding and index construction.

[0109] S4, hierarchical encoding of residual data generated by dynamic dimensionality reduction processing to generate lightweight residual encoding results;

[0110] First, compute the dynamic tensor at the edge with the dimensionality reduction result of step S2 The residual is defined as the difference between the original tensor and the low-rank approximation result:

[0111]

[0112] The residual contains the high-frequency detail information lost in the dimensionality reduction process, which is used to restore the data accuracy in the cloud. Exemplarily, the residual R, core tensor and factor matrix {U (n)} are uploaded to the cloud to form a hierarchical encoding data stream.

[0113] Subsequently, feature-aware quantization is performed in the cloud. Since the different elements in the residual data have significantly different effects on model accuracy, a gradient importance dynamic allocation quantization bit width is exemplarily adopted. Specifically, the gradient importance of the residual element to the preset loss function is calculated by a lightweight proxy model :

[0114]

[0115] where r i is the residual element, g i is its gradient importance, R: residual tensor, reflecting the information loss in the dimensionality reduction process. According to the gradient importance threshold θ, the quantization bit width is dynamically allocated:

[0116]

[0117] g i : residual element gradient importance, measuring its impact on model accuracy;

[0118] b i : dynamically allocated quantization bit width, determining data accuracy and compression rate

[0119] where b high and b low are the preset high bit width and low bit width, respectively. This allocation strategy ensures that high importance residuals retain higher accuracy and low importance residuals are compressed for storage.

[0120] Further, non-uniform quantization is performed on the residual. Exemplarily, piecewise uniform quantization is adopted, which divides the residual block into sub-regions, and adjusts the quantization parameter according to the statistical characteristics of each sub-region. The quantization formula is defined as:

[0121]

[0122] where μ and σ are the mean and standard deviation of the residual block, respectively, and Round(·) is the rounding function. The quantized residual is stored together with the metadata (mean, standard deviation, bit width allocation table).

[0123] Residual calculation is based on the decomposition result of step S2, which retains the difference information between the original data and the low-rank approximation, providing a basis for subsequent precision recovery; gradient importance calculation correlates data characteristics with model requirements through a proxy model, ensuring that the quantization strategy adapts to downstream tasks; non-uniform quantization combines statistical characteristics and dynamic bit width allocation to reduce redundant data while preserving key features, forming a synergy with the index construction of step S5.

[0124] Through residual hierarchical encoding, lightweight computation can be achieved at the edge, dynamic optimization of quantization strategy in the cloud, balancing transmission overhead and data precision; feature-aware quantization preserves key information, adapting to different downstream task requirements; non-uniform quantization reduces storage space occupation, providing compact data representation for multi-dimensional index construction.

[0125] S5, constructing a multi-dimensional joint index based on the hierarchical encoding result, and storing compressed data and metadata;

[0126] First, based on the quantized residuals generated in step S4 and metadata (core tensor factor matrix {U (n)}, quantization parameter {b i}), a hierarchical storage structure is constructed. The data is divided into metadata layer and data layer, for example:

[0127] Metadata layer: stores core tensor factor matrix {U (n)}, quantization parameter {b i}, used for fast reconstruction of approximate results;

[0128] Data layer: stores quantized residuals and original timestamp sequence, organized in time window blocks.

[0129] Further, a multi-dimensional joint index is constructed to speed up the query. The index includes label inverted index and time B+ tree index:

[0130] Label inverted index: based on the row number of factor matrix, the mapping of label combination to data block is established. For example, label combination (device i d = 001, region = A) corresponds to the 1st row of U (1) and the 1st row of U (2) , which is mapped to core tensor slice Index item is defined as:

[0131]

[0132] Index(·): label inverted index item, mapping label combination to data block.

[0133] Time B+ tree index: data blocks are divided by timestamp range, and each leaf node stores the physical address of data blocks in the corresponding time window. The index key is the start time t of the time window start , satisfying:

[0134]

[0135] where Δt is the preset time window length in step S1.

[0136] t start : the start time of the time window, used as the B+ tree index key.

[0137] Finally, according to the index granularity parameter γ output by step S3, the index structure is dynamically adjusted. Exemplarily, the index granularity γ controls the B+ tree node size and the size of the inverted index shard:

[0138] When γ is small, the B+ tree node contains fewer time windows, reducing the number of query I / O;

[0139] When γ is large, the inverted index shard covers more label combinations, reducing the height of the index tree.

[0140] Hierarchical storage reduces redundant data access by separating metadata and quantized residuals, adapting to the hierarchical encoding output of step S4; label inverted index directly associates with the factor matrix row number of step S2, realizing fast filtering of multi-dimensional labels; time B+ tree index aligns with the time window of step S1, ensuring the efficiency of time range query; dynamic index granularity is based on the optimization parameters of step S3, balancing storage overhead and query performance.

[0141] γ: index granularity parameter, controlling the storage and query efficiency trade-off of the index structure.

[0142] Through hierarchical storage and multi-dimensional joint index, efficient organization and fast positioning of data can be realized; label inverted index supports accurate filtering of multi-dimensional label combinations, time B+ tree index accelerates time range query; dynamic index granularity adapts to different resource constraints, optimizing storage and query performance.

[0143] S6, according to the query request, locating data through multi-dimensional joint index and reconstructing the output result.

[0144] First, parse the time range and multi-dimensional label conditions in the query request, and locate the target data block through the multi-dimensional joint index constructed in step S5. Exemplarily, for the label condition region = A, use the label inverted index to get the corresponding factor matrix row number set S2, and locate the data block physical address that satisfies the time window [t start , t end ] through the time B+ tree index.

[0145] Subsequently, the core tensor generated in step S2 is multiplied by the sub-matrix with the subset of factor matrix The approximate result is calculated. Exemplarily, the approximate reconstruction formula is defined as:

[0146]

[0147] wherein, is the row number of the Nth dimension factor matrix belonging to the set S N sub-matrix;

[0148] S N : the set of row numbers of the Nth dimension factor matrix satisfying the query condition.

[0149] The approximate calculation utilizes the low-rank structure to quickly generate a preliminary result, reducing the real-time calculation overhead.

[0150] Further, for a query requiring an accurate result, the quantized residual of step S4 is superimposed First, the inverse quantization operation is performed:

[0151]

[0152] In the formula, σ and μ are statistical parameters of the quantized residual block, b i is the dynamically allocated bit width. The residual after inverse quantization is superimposed with the approximate result to generate an accurate output:

[0153]

[0154] wherein: The approximate result tensor is generated by the multiplication of the core tensor and the sub-matrix;

[0155] The accurate result tensor contains complete information after residual superposition.

[0156] The multi-dimensional index positioning combines the label inverted index and the time B+ tree index of step S5 to realize fast data filtering and physical address mapping; the approximate result calculation relies on the decomposition structure of step S2 to accelerate the response by utilizing the low-rank characteristic; the residual superposition mechanism is based on the hierarchical encoding data of step S4 to restore high-frequency detail information, meeting the accuracy requirement.

[0157] Through the hierarchical response mechanism of approximate calculation and residual superposition, the response speed and result accuracy can be dynamically balanced according to the query requirement; the multi-dimensional index positioning reduces irrelevant data access, improving the query efficiency; the inverse quantization operation restores the residual details, ensuring the consistency of the final output with the original data.

[0158] Please refer to the attached Figure 2The embodiment of the application provides a time series database driven monitoring data compression storage system, comprising:

[0159] A data preprocessing module is configured to construct a dynamic tensor.

[0160] Space-time alignment: data is divided according to a preset time window (for example, Δt = 1 minute), and a timestamp is aligned to a starting point of the window, and a formula is as follows:

[0161] Label coding: labels such as device IDs and areas are mapped into integer indexes through hash mapping, and a label-index mapping table M is constructed.

[0162] Tensor aggregation: numerical indicators (mean or accumulation) are aggregated according to a label dimension, and an N-order dynamic tensor is generated.

[0163] The dynamic dimension reduction module is provided with a structured input, and the time dispersion and label sparsity of original data are eliminated.

[0164] A dynamic dimension reduction module is configured to generate a core tensor and a factor matrix.

[0165] Initial Tucker decomposition: the dynamic tensor is decomposed into a low-rank structure: wherein is the core tensor, U (n) is an orthogonal factor matrix of each dimension.

[0166] Incremental update: when new data is added, an error matrix E (n) is calculated, and a factor matrix is locally updated as follows:

[0167] ΔU (n) = E (n) · Φ (n)T , Φ (n) = (U (n)T U (n) ) -1 U (n)T ;

[0168] The data dimension is compressed through low-rank approximation, and a dimension reduction result is provided for a joint optimization module.

[0169] A joint optimization module is configured to determine compression, decompression and index parameters.

[0170] Objective function: a multi-objective optimization problem is defined as follows:

[0171]

[0172] Wherein, α (quantization bit width), β (parallelism) and γ (index granularity) are optimization variables.

[0173] Solving algorithm: Frank-Wolfe algorithm is used to update parameters iteratively, and the gradient calculation is:

[0174]

[0175] Output parameter guidance layered coding and index construction, adapt to resource constraints.

[0176] Layered coding module, for residual quantization and transmission;

[0177] Residual calculation: generate the residual of the original tensor and the low rank approximation:

[0178]

[0179] Dynamic quantization: calculate the importance of residual gradient through proxy model Dynamic allocation bit width:

[0180]

[0181] Preserve key features to provide lightweight data flow for storage module.

[0182] Storage and index module, for data persistence and multi-dimensional index construction;

[0183] Layered storage: metadata layer storage {U (n)}, quantization parameters; data layer storage quantized residual

[0184] Multi-dimensional index: label inverted index: map label combination to factor matrix row number, such as device i d=001→U (1) (1, :).

[0185] Time B+ tree index: divide data block according to time window, key is t start .

[0186] Based on the optimization parameter γ, dynamically adjust the index granularity, balance the storage and query performance.

[0187] Query reconstruction module, for data positioning and precision reconstruction.

[0188] Approximate calculation: use core tensor and submatrix product to generate approximate result:

[0189]

[0190] Accurate reconstruction: after inverse quantization of residual, add to the approximate result:

[0191]

[0192] Quickly locate data by index, hierarchical response to different accuracy requirements.

[0193] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely divergences of the principles and spirit of the application and that numerous modifications, changes, substitutions, and alterations can be made thereto without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

Claims

1. A time-series database-driven method for compressing and storing monitoring data, characterized in that, Includes the following steps: Preprocess the multi-source monitoring data to construct a dynamic tensor containing timestamps, numerical indicators, and multi-dimensional labels; The dynamic tensor is subjected to dynamic dimensionality reduction to generate a core tensor and a multidimensional factor matrix; Based on the core tensor and multidimensional factor matrix, compression parameters, decompression parallelism, and index granularity are determined through a joint optimization model. The compression parameters are used to determine the residual quantization bit width, and the residual data generated by the dynamic dimensionality reduction process is hierarchically encoded to generate a lightweight residual encoding result. The index structure is adjusted using the index granularity to construct a multidimensional joint index based on the hierarchical encoding result, and the lightweight residual encoding result as compressed data and the core tensor, multidimensional factor matrix, and quantization bit width as metadata are stored, wherein the quantization bit width is obtained by mapping the compression parameters. Based on the query request, the data is located and the output result is reconstructed using the multidimensional composite index.

2. The time-series database-driven monitoring data compression and storage method according to claim 1, characterized in that, The steps for preprocessing multi-source monitoring data include: Divide the raw data into preset time windows and align the timestamps to the start time of the window. Map multidimensional labels to integer indices to construct a label-index mapping table; Aggregate numerical metrics by label dimension to generate an N-order dynamic tensor.

3. The time-series database-driven monitoring data compression and storage method according to claim 1, characterized in that, The steps for performing dynamic dimensionality reduction on the dynamic tensor include: Perform tensor decomposition on the initial tensor to generate the core tensor and orthogonal factor matrix; When new labels or data are added, the factor matrix is ​​locally updated using an incremental update algorithm, wherein the incremental update satisfies: ； in, For the factor matrix increment, This is the error matrix.

4. The time-series database-driven monitoring data compression and storage method according to claim 1, characterized in that, The objective function of the joint optimization model is: ； Wherein, the query The relationship between overhead items and index granularity is non-linear: ； For index granularity parameters, These are the weighting coefficients.

5. The time-series database-driven monitoring data compression and storage method according to claim 1, characterized in that, The steps for hierarchical encoding of the residual data generated by the dynamic dimensionality reduction process include: Calculate the residual between the dynamic tensor and the dimensionality reduction result at the edge; In the cloud, residual quantization bit width is dynamically allocated based on the importance of data characteristics; high-importance residuals are allocated high bit width, and low-importance residuals are allocated low bit width.

6. The time-series database-driven monitoring data compression and storage method according to claim 1, characterized in that, The steps for constructing a multidimensional joint index from the hierarchical encoding results include: Construct a label inverted index based on the row numbers of the factor matrix; Build a B+ tree index based on timestamps; The index granularity is dynamically adjusted based on the output parameters of the joint optimization model.

7. The time-series database-driven monitoring data compression and storage method according to claim 1, characterized in that, The steps for locating data and reconstructing the output results using the multidimensional composite index include: The approximate result is calculated using the product of the core tensor and the submatrix: ； in, The set of row numbers of the factor matrix that satisfy the query conditions; The decoded residual data is superimposed to generate accurate results.

8. The time-series database-driven monitoring data compression and storage method according to claim 5, characterized in that, The dynamically allocated residual quantization bit width includes: The gradient importance of residual elements with respect to the loss function is calculated using a surrogate model. High or low bit width is assigned based on gradient importance threshold.

9. The time-series database-driven monitoring data compression and storage method according to claim 3, characterized in that, The incremental update algorithm includes: Only the local factor matrix blocks corresponding to the newly added label dimensions are updated; The core tensor is optimized iteratively using the least squares method.

10. A time-series database-driven monitoring data compression and storage system, comprising the time-series database-driven monitoring data compression and storage method according to any one of claims 1-9, characterized in that, include: The data preprocessing module is used to construct dynamic tensors; The dynamic dimensionality reduction module is used to generate the core tensor and factor matrix; The joint optimization module is used to determine compression, decompression, and indexing parameters; Layered coding module, used for residual quantization and transmission; The storage and indexing module is used for data persistence and multidimensional index building; The query reconstruction module is used for data location and accuracy reconstruction.

Images