Large data compression storage and rapid retrieval method and system

Abstract

The invention relates to the field of data storage, and discloses a big data compression storage and rapid retrieval method and system.The method comprises the following steps that dimension weight vectors are dynamically generated based on multi-dimensional information entropy, and rotating Hilbert codes are generated by adjusting the direction of a space filling curve; adaptively determining the number of segments and an error threshold value by using a time sequence prediction model, and compressing and storing a coding residual error by using a linear model; a double-layer collaborative index structure containing a space bitmap index and a value domain B + tree index is constructed, and local index reconstruction during weight change is achieved through an incremental updating mechanism; mapping a query range into a coding interval, screening candidate data segments in combination with an index, and fusing model parameters and residual decoding to return a result; newly added data code correction and expired data elimination are asynchronously processed, and the storage space consistency is optimized. According to the method, the unification of high compression ratio, quick retrieval and self-adaptive updating is realized.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a method and system for compressing, storing and quickly retrieving large data. Background Art

[0002] In the field of data storage technology, with the popularization of the Internet of Things and smart terminal devices, the real-time processing of multi-dimensional spatiotemporal data faces multiple challenges, including a surge in data size, increased dynamic correlation between dimensions, and increased real-time query requirements. Traditional encoding methods based on static space-filling curves usually fix the order of dimension expansion and are difficult to adapt to the temporal evolution of data distribution characteristics, resulting in a weakening of the spatial locality of the encoded values ​​and a large amount of redundant information generated during the compression process. Existing indexing mechanisms mostly use a single structure to organize encoded data. In dynamic weight scenarios, index updates require global reconstruction, causing storage jitter and query performance degradation. At the same time, there is a lack of adaptive control of the time-varying characteristics of the data stream during data segmentation compression, and fixed error thresholds can easily lead to inefficient segmentation, resulting in a contradiction between storage fragmentation and query efficiency.

[0003] In addition, the existing system lacks a collaborative management mechanism with the index structure in terms of expired data elimination and storage space recovery. Frequent data updates lead to a decrease in storage resource utilization, making it difficult to support the long-term demand for efficient data services.

[0004] The above technical bottlenecks restrict the simultaneous improvement of storage compression efficiency and retrieval real-time performance in large-scale dynamic data scenarios. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a method and system for big data compression storage and rapid retrieval, which solves the technical problems of high coding redundancy, high index update overhead, severe storage fragmentation and low query efficiency caused by changes in dimensional correlation in dynamic data scenarios.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for compressing, storing and quickly retrieving large data, comprising the following steps:

[0007] S1, dynamically generating a dimension weight vector based on the multi-dimensional information entropy in the data stream, adjusting the dimension priority of the space filling curve according to the dimension weight vector, and generating a rotated Hilbert code value for each data point;

[0008] S2, using a time series prediction model to predict the number of data segment divisions, dynamically calculating a segmentation error threshold based on the predicted number of data segment divisions, segmenting the sorted data based on the rotated Hilbert code values, and storing piecewise linear model parameters used to describe each of the data segments and a residual coding result of each of the data segments;

[0009] S3, constructing a spatial bitmap index based on the distribution range of the rotated Hilbert code value, and constructing a range B+ tree index based on the code interval endpoint value of each data segment, and triggering an incremental update of the spatial bitmap index and the range B+ tree index when it is detected that the change of the dimension weight vector exceeds a preset threshold;

[0010] S4, mapping the query range into a rotated Hilbert coding interval, using the spatial bitmap index to screen candidate data segments, locating the target data segment through the range B+ tree index, and returning the data within the query range by combining the piecewise linear model parameters and the residual coding result;

[0011] S5, dynamically modifying the dimension weight vector and the rotated Hilbert code value according to the newly added data, eliminating expired data and synchronously updating the spatial bitmap index and the range B+ tree index.

[0012] Preferably, in step S1, the step of generating the rotated Hilbert code value of each data point includes:

[0013] S11, for each dimension in the data stream, divide the corresponding spatial range into the same number of uniform grids, and count the number of data points in each grid of each dimension;

[0014] S12, for the dth dimension, calculate the data distribution probability of the dth dimension in the bth grid The calculation formula for the data distribution probability is:

[0015]

[0016] in, Indicates the amount of data in the bth grid of the dth dimension; N is the total number of all data points in the data stream in the current time window; b∈[1,B], B is the preset number of grid divisions;

[0017] S13, based on the calculated data distribution probability, calculate the information entropy S of the d-th dimension d , the calculation formula of the information entropy is:

[0018]

[0019] S14, based on the calculated information entropy values ​​of each dimension, generate an n-dimensional dimension weight vector w=(w1,w2,....,w n ); Among them, the weight value w corresponding to the d-th dimension d Calculated by the following formula:

[0020]

[0021] Among them, S irepresents the information entropy of the i-th dimension;

[0022] S15, based on the generated dimension weight vector w, adjusting the dimension access priority of the Hilbert space filling curve when processing multidimensional data, and for the dimension with the highest weight value, extending the Hilbert space filling curve along the dimensional direction of the dimension with the highest weight value;

[0023] S16, generating a corresponding rotational Hilbert code value for each data point in the data stream according to the Hilbert space filling curve after the dimension priority is adjusted.

[0024] Preferably, in step S2, the time series prediction model is an autoregressive integrated moving average model, and the step of using the time series prediction model to predict the number of data segment divisions includes:

[0025] S21, obtain the number of data segments actually observed in a series of historical time windows to form the original time series, recorded as {k1, k2, ...., k t}, for the original sequence {k s} Perform d-order difference operation to obtain a stable time series {Δ d k t}, where k s represents the actual number of segments of the sth historical time window, t is the serial number of the latest historical time window currently observed, Δ d k t represents the d-order difference value of the original sequence in the t-th window, where d is the preselected difference order;

[0026] S22, based on the obtained stationary time series {Δ d k t}, establish an ARIMA (p, d, q) model, the expression of the ARIMA (p, d, q) model is:

[0027]

[0028] Among them, Δ d k t represents the d-order difference value of the original sequence in the t-th time window, p is the order of the autoregressive term of the model, q is the order of the moving average term of the model, φ i is the autoregressive coefficient, i=1,2,...,p, θ j is the moving average coefficient, j=1,2,...,q,∈t is the white noise residual term of the t-th window, Δ d k t-i is the d-order difference value of the original sequence in the ti-th time window, ∈ t-j is the white noise residual term in the tj-th time window;

[0029] S23, using the established ARIMA (p, d, q) model and the next window division number prediction formula, predict the number of data segment divisions in the next time window, the next window division number prediction formula is:

[0030]

[0031] in, The number of segments to divide the forecast data into for the next time window.

[0032] Preferably, the step of predicting the number of data segment divisions using the time series prediction model further includes the step of dynamically adjusting the moving average coefficient in the ARIMA (p, d, q) model according to the number of data segment divisions actually occurring in the period after each time window processing period ends, specifically:

[0033] S24, for the currently ended time window, obtain the actual number of data segment divisions k that actually occurred in the current window real ;

[0034] S25, obtaining the number of predicted data segments generated by the ARIMA (p, d, q) model in the previous time window for the same current window

[0035] S26, calculating the actual number of data segment divisions k real Divide the number of segments with the forecast data The deviation Δk between them is calculated as follows:

[0036] S27, obtaining the dimension weight vector w calculated for the current window current , and the dimension weight vector w calculated from the previous time window of the current window previous ;

[0037] S28, calculating the learning rate η for subsequent correction of the moving average coefficient, wherein the calculation formula of the learning rate η is:

[0038] η=β·||w current -w previous ||1+γ

[0039] Among them, w current is the dimension weight vector of the current window; previous is the dimension weight vector of the previous window; ||w current -w previous ||1 represents the w current With the w previousThe Manhattan norm between β and γ; β and γ are preset adjustment coefficients;

[0040] S29, for each moving average coefficient θ in the ARIMA (p, d, q) model j , the coefficient is dynamically corrected using the calculated prediction deviation Δk and the calculated learning rate η, and the corrected moving average coefficient Calculated by the following formula:

[0041]

[0042] in, is the value of the jth moving average coefficient before this correction operation is performed, The new value of the j-th moving average coefficient after this correction operation will be used in the forecast calculation of the ARIMA (p, d, q) model in the subsequent time window.

[0043] Preferably, in step S3, the step of constructing a spatial bitmap index according to the distribution range of the rotated Hilbert code value includes:

[0044] S31, determining a coding range according to the rotation Hilbert coding values ​​of all data points in the current time window, and calculating the maximum coding value h of all the rotation Hilbert coding values ​​in the current time window. max =max{h i |i∈W k} and the minimum encoding value h min =min{h i |i∈W k}, where W k Represents the data set of the kth time window;

[0045] S32, based on the calculated h max and h min , calculate the bitmap resolution parameter m by the following formula:

[0046]

[0047] S33, divide the entire data space composed of the rotated Hilbert code values ​​into 2 m ×2 m Grid, define each grid corresponding to a unique integer encoding subinterval in g∈[1,2 m ×2 m ] is the grid number;

[0048] S34, initialize a size of 2 m ×2m The bitmap matrix B is created, and the initial values ​​of all elements in the matrix are set to 0;

[0049] S35, for each data segment formed in step S2, obtain its corresponding rotation Hilbert coding interval

[0050] S36, traverse each grid in the bitmap matrix, if there is at least one data segment S j The coding interval The interval between the current grid grid g If there is an intersection, the element corresponding to the grid g in the bitmap matrix is ​​marked as 1, otherwise the element is marked as 0.

[0051] Preferably, in step S3, the triggering condition for the incremental update of the spatial bitmap index and the range B+ tree index is:

[0052] When the change in the dimension weight vector Δw=||w new -w old When ||2 exceeds the threshold δ, an incremental update operation is performed, where w new is the weight vector of the current window, w old is the weight vector of the previous window, ||·||2 represents the Euclidean norm;

[0053] The threshold δ is adaptively adjusted according to the historical variation of the weight vector. The calculation formula of the threshold δ is:

[0054]

[0055] Among them, λ d is the weight change sensitivity coefficient of the d-th dimension, is the weight value of the dth dimension of the previous window;

[0056] The incremental update operation includes: recalculating the rotated Hilbert code values ​​affected by the weight change, generating a new spatial bitmap index only for the area where the code value changes, and updating the code interval endpoint values ​​of the corresponding data segment in the value range B+ tree index.

[0057] Preferably, in step S4, the steps of mapping the range to be queried into a rotated Hilbert coding interval, screening candidate data segments using the spatial bitmap index, locating the target data segment using the range B+ tree index, and returning the data within the query range in combination with the piecewise linear model parameters and the residual coding result include:

[0058] S41, receiving the multidimensional space range to be queried defined by the user QR = [x l ,x u ]×[y l,y u ], mapping the spatial range to be queried into a rotation Hilbert coding interval [h l ,h u ], where h l is the lower left corner of QR (x l ,y l ) corresponding to the encoding value, h u is the upper right corner point (x u ,y u ) The corresponding code value, the mapping rule is consistent with the calculation method of the generated rotation Hilbert code value;

[0059] S42, using the constructed spatial bitmap index and combining the rotation Hilbert coding interval [h l ,h u ]Filter the candidate data segment set The candidate data segment set Defined as Among them, S j For a data segment, and Data segment S j The minimum and maximum code values ​​of the corresponding rotated Hilbert code values;

[0060] S43, searching through the value range B+ tree index The storage address of all data segments in the Do the following:

[0061] 1) Extract the data stored in data segment S j The linear model parameters (a j ,b j ), calculate the data stored in the data segment S j The actual spatial range covered by the linear model (x,y)=Hilbert_Decode(h)};

[0062] Where Hilbert_Decode(h) represents the function of inversely solving the spatial coordinates from the encoded value h;

[0063] 2) According to the spatial range of the data segment The relationship between the spatial range QR to be queried is used for conditional judgment and data extraction:

[0064] Condition 1: If the spatial range to be queried is determined Then directly process the data segment S j To return all the original data points contained within it;

[0065] Condition 2: If condition 1 is not met, the decoded data is stored in data segment S j The residual coding result q j , for each by using the piecewise linear model parameters (a j ,b j ) and the decoded residual coding result q j The original data point (x p ,y p ), if the reconstructed original data point (x p ,y p ) satisfies the boundary conditions x of the query space range QR l ≤x p ≤x u And y l ≤y p ≤y u , then the data point (x p ,y p ) is added to the final query result set.

[0066] Preferably, in step S5, the step of eliminating expired data and synchronously updating the spatial bitmap index and the range B+ tree index includes:

[0067] S51, record the latest update timestamp for each data segment S0 in the system

[0068] S52, periodically or according to a predetermined strategy, check all data segments. If for a certain data segment S j , when it meets the conditions, mark the data segment S j is the failure segment, where T expired is the preset expiration time threshold, t current The current time of the system;

[0069] S53, for each invalid data segment, mark the leaf node corresponding to the invalid segment as a logically deleted state in the value range B+ tree index, and retain its encoding interval endpoint value But the updated storage pointer is an invalid address;

[0070] S54: In the spatial bitmap index, all bitmap grids that intersect with the coding interval of the invalid data segment are identified. For each bitmap grid, check whether it intersects with the coding interval of any other data segment that is not marked as invalid. If the bitmap grid no longer intersects with the coding interval of any valid data segment after removing the association with the invalid data segment, update the bitmap flag B(g) corresponding to the bitmap grid to 0; otherwise, if the bitmap grid is still associated with other valid data segments, its flag B(g) remains 1.

[0071] S55, when the storage space utilization rate of the failed segment is lower than a preset threshold, physical deletion and storage merging operations are performed: continuous free storage blocks are merged into a larger available space, and the storage pointer address of the affected node in the range B+ tree index is updated.

[0072] The present invention also provides a large data compression storage and rapid retrieval system, comprising:

[0073] A dynamic encoding module is used to receive real-time data streams, calculate the information entropy of each dimension and generate a dimension weight vector, adjust the dimension priority of the space filling curve according to the dimension weight vector, generate a rotated Hilbert code value, and transmit the encoding result to the segment compression module;

[0074] The segmented compression module is connected to the dynamic coding module and includes an ARIMA prediction unit, a dynamic threshold calculation unit, and a residual coding unit, where:

[0075] The ARIMA prediction unit predicts the number of segments in the next time window based on the historical segment number sequence;

[0076] The dynamic threshold calculation unit dynamically adjusts the error threshold based on the predicted number of segments to control the granularity of data segment division;

[0077] The residual coding unit compresses and encodes the segmented linear model parameters and residual data, and sends the compression results to the collaborative indexing module;

[0078] The collaborative indexing module communicates bidirectionally with the segment compression module and the dynamic encoding module, and includes a bitmap generator, a B+ tree manager, and an incremental update controller, among which:

[0079] The bitmap generator constructs a spatial bitmap index according to the distribution range of the rotation Hilbert code value;

[0080] The B+ tree manager constructs a value range B+ tree index based on the encoding interval endpoint values ​​of the data segment;

[0081] The incremental update controller monitors the changes in the dimension weight vector and triggers local reconstruction of the index when the change exceeds a threshold;

[0082] The query engine is connected to the collaborative index module, receives external query requests and maps the spatial range to the coding interval, uses the spatial bitmap index and the range B+ tree index to screen candidate data segments, and returns the query data in combination with the linear model parameters and the residual decoding results;

[0083] The data update agent interacts with the dynamic encoding module, collaborative indexing module, and segment compression module to handle encoding update operations for newly added data, eliminate expired data segments, and maintain consistency between indexes and storage space.

[0084] Preferably, the incremental update controller has a built-in weight change monitoring channel, which receives the weight vector change output by the dynamic encoding module in real time, and sends a local reconstruction instruction to the bitmap generator and the B+ tree manager when the weight vector change exceeds a preset threshold;

[0085] The leaf nodes of the B+ tree manager contain a bidirectional pointer structure:

[0086] The forward pointer associates the encoding interval endpoint value with the storage address;

[0087] The back pointer associates the storage address with the corresponding space bitmap grid mark;

[0088] The bidirectional pointer supports fast location of storage blocks according to the encoding value and updates the bitmap index status in the reverse direction.

[0089] The present invention provides a method and system for compressing, storing, and rapidly retrieving large amounts of data. This method has the following beneficial effects:

[0090] 1. This invention significantly improves data compression efficiency through a dynamic encoding mechanism driven by multidimensional information entropy. Based on real-time calculation of the information entropy distribution of data in each dimension, a weight vector is generated. The dimensional expansion direction of the Hilbert curve is dynamically adjusted to cluster data in high-resolution dimensions more closely, effectively reducing spatial redundancy after encoding. Combined with the residual coding technology of the segmented compression module, this method reduces storage space while ensuring accuracy.

[0091] 2. This invention significantly optimizes query response speed through a spatiotemporal collaborative indexing architecture and a model-based prejudgment mechanism. A spatial bitmap index rapidly screens candidate data segments, while a B+ tree index precisely locates physical storage locations. Combined with linear model parameters that precompute the spatial coverage of data segments, this method can skip residual decoding of invalid data segments, reducing disk I / O and computational overhead, and achieving a tiered improvement in query efficiency.

[0092] 3. This invention enhances the system's dynamic adaptability through the weight-sensitive mechanism of the incremental update controller. By monitoring the magnitude of changes in dimension weight vectors in real time, local index reconstruction is triggered only when the weight change exceeds a threshold, avoiding the overhead of global index reconstruction. Combined with the bidirectional pointer design of B+ tree child nodes, this ensures the atomicity of the index update process and storage structure adjustments, maintaining system stability in high-frequency data update scenarios.

[0093] 4. This invention achieves efficient utilization of storage resources through a data lifecycle management strategy. The data update agent uses a collaborative mechanism that combines logical deletion marking with physical storage merging, maintains invalid segment marking information in the B+ tree index, and asynchronously executes storage space merging tasks. Combined with the storage fragmentation prediction capabilities of the dynamic threshold calculation unit, this effectively balances storage space utilization and system maintenance costs.

[0094] 5. This invention leverages a modular asynchronous processing architecture to ensure system reliability. The query engine and data update agent use a non-blocking communication mode. Key operations are recorded in transaction logs. Combined with the incremental update controller's delay window mechanism, it supports operation rollback when index updates fail, ensuring data consistency. A load-aware task scheduler dynamically adjusts resource allocation, improving system fault tolerance and overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0096] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0097] Please see the attached Figure 1 The present invention provides a big data compression storage and fast retrieval method, which realizes efficient data compression and fast range query through dynamic dimension weight allocation, spatiotemporal collaborative indexing and prediction segmentation mechanism.

[0098] like Figure 1 As shown, the big data compression storage and rapid retrieval method may include the following steps:

[0099] S1. Dynamically generate dimension weights based on the multidimensional information entropy in the data stream, and adjust the dimension priority of the space filling curve accordingly to generate the rotational Hilbert code;

[0100] S2. Divide the data into segments through time series prediction, dynamically set the segmentation error threshold, sort and segment the encoded data, and record the linear model and residual code;

[0101] S3: Build a spatial bitmap index based on the code value and a B+ tree index based on the data value. When the dimension weight change exceeds the threshold, an incremental update of the index is triggered.

[0102] S4. Map the query range to a coding interval, filter candidate segments using the spatial index, locate the target segment using the B+ tree, and restore the query results based on the model.

[0103] S5. The weight changes caused by the new data will update the code and index, eliminate expired data and synchronously maintain the index structure.

[0104] The following is a detailed description of each step in the method of the present invention, which comprehensively explains the specific implementation principles, technical details and processes of each step.

[0105] In step S1, the present invention dynamically analyzes the multi-dimensional distribution characteristics of the data stream, generates a weight vector reflecting the degree of concentration of each dimensional data set, and adjusts the encoding priority of the space filling curve accordingly to achieve efficient spatial encoding of the data. The specific implementation process includes:

[0106] First, the multidimensional space range in the data stream is divided into uniform grids according to preset rules. For each dimension d∈[1,n], where n represents the total number of dimensions, the corresponding data space range is Divide into B equal-width intervals, each interval is called a grid. Count the number of data points in the bth grid of the dth dimension in the current time window Where b∈[1,B]. Calculate the grid data distribution probability of each dimension based on statistical results in represents the amount of data in the bth grid of the dth dimension; N is the total number of all data points in the data stream in the current time window.

[0107] Then, the information entropy of each dimension is calculated based on the distribution probability. For the dth dimension, its information entropy S d Calculated by the following formula:

[0108]

[0109] Among them, when When, define This formula quantifies the uniformity of data distribution in dimension d: when the data set is distributed in a small number of grids, S d The value is low; when the data is evenly distributed, S d The value is higher.

[0110] Generate the dimension weight vector w=(w1,w2,...,w n ). Among them, the weight w of the d-th dimension d Determined by the following normalization formula:

[0111]

[0112] Among them, S i represents the information entropy of the i-th dimension;

[0113] The weight vector reflects the ability of different dimensions to distinguish the data space distribution. The weight value w d The larger it is, the more uneven the data distribution of dimension d is, and it should be given higher priority in spatial encoding.

[0114] The dimension priority of the space filling curve is adjusted based on the weight vector. Specifically, based on the generated dimension weight vector w, the dimension access priority of the Hilbert space filling curve when processing multidimensional data is adjusted, and the dimension with the highest information entropy weight is selected as the main expansion direction of the Hilbert curve. According to the Hilbert space filling curve after the dimension priority adjustment, the corresponding rotation Hilbert coding value is generated for each data point in the data stream. For n-dimensional space, the generation rule of the adjusted rotation Hilbert curve is: in the process of recursive segmentation of the curve, space division is preferentially performed along the dimension with the highest weight value. This adjustment enables the data points to be clustered in the coding space according to the distribution characteristics of the high-discrimination dimension, thereby improving the subsequent data compression efficiency.

[0115] Generate the rotation Hilbert code value according to the adjusted space filling curve. For each data point x=(x1,x2,...,x n ), map it to the n-dimensional grid coordinate system, and calculate its corresponding encoding value according to the recursive segmentation order of the curve The encoding value generation process satisfies the following: when the weight vector w changes, the encoding value h(x) of the same spatial point will be dynamically updated by adjusting the recursive segmentation order of the curve, thereby adapting to changes in data distribution characteristics.

[0116] Preferably, the value of the grid division number B is related to the data size, and can be calculated based on the data volume N of the current time window. Adaptive adjustment. This strategy ensures computational efficiency while avoiding distortions in information entropy assessment caused by overly dense or sparse grids. Furthermore, when generating rotational Hilbert codes, for dimensions with the same weight, the first dimension in the original dimension order is prioritized as the secondary priority, ensuring the determinism of the encoding rules.

[0117] In step S2, the present invention dynamically determines the data segmentation strategy based on the time series prediction model, predicts the number of segments in the next time window and adaptively adjusts the error threshold to achieve adaptive compression and structured storage of the rotated Hilbert coded data. The specific implementation process includes:

[0118] First, a time series model of the number of historical segmentations is constructed. The number of data segments actually observed in a series of historical time windows is obtained to form the original time series, which is recorded as {k1, k2, ...., k t}, where t represents the sequence number of the current time window. s} Perform d-order difference processing to eliminate non-stationarity and obtain the stationary series {Δ d k t}. The difference operation is realized by recursive calculation:

[0119] Δ d k t =Δ d-1 k t -Δ d-1 k t-1 (Initial value Δ 1 k t =k t -k t-1 )

[0120] The ARIMA (p, d, q) model is established based on the stationary series, and its mathematical expression is defined as:

[0121]

[0122] Among them, Δ d k t represents the d-order difference value of the original sequence in the t-th time window, p is the order of the autoregressive term of the model, q is the order of the moving average term of the model, φ i is the autoregressive coefficient, i=1,2,...,p, θ j is the moving average coefficient, j=1,2,...,q,∈ t is the white noise residual term of the t-th window, Δ d k t-i is the d-order difference value of the original sequence in the ti-th time window, ∈ t-j is the white noise residual term in the tj-th time window, is the white noise residual term that obeys the normal distribution.

[0123] Predict the number of segments for the next window based on the model The calculation formula is:

[0124]

[0125] This predicted value comprehensively considers the trend of the historical segment quantity and the residual fluctuation characteristics, providing a benchmark for subsequent error threshold calculation.

[0126] Based on the number of predicted segments Dynamically determine the maximum allowable fitting error ∈max Specifically, based on the total amount of data N in the current time window, the error threshold is calculated using the following formula:

[0127]

[0128] This threshold is used to control the granularity of data segment division: when the number of predicted segments increases, ∈ max Lowering the error tolerance will make the data segmentation more refined; otherwise, the error tolerance will be relaxed to reduce the number of segments.

[0129] The error threshold is used to segment the rotated Hilbert coded data. After all data points in the current time window are sorted in ascending order by the code value, the sliding window method is used for segmentation: starting from the first data point, the window range is gradually expanded until the linear fitting residual of the data points in the window satisfies:

[0130]

[0131] Among them, (a, b) are the linear regression parameters of the window data, h p is the encoding value of the data point. When the window expansion exceeds the error threshold, the current window is marked as a data segment and its linear model parameters (a j ,b j ), and store the residual coding results q of all data points j ={h p -(a j p+b j )}.

[0132] Preferably, a dynamic correction mechanism is introduced in the process of updating the model parameters. real and predicted value There is a bias When the moving average coefficient θ j Make online adjustments:

[0133]

[0134] in, is the value of the jth moving average coefficient before this correction operation is performed, The new value of the j-th moving average coefficient after this correction operation will be used in the forecast calculation of the ARIMA (p, d, q) model in the subsequent time window;

[0135] The learning rate η is associated with the amount of change in the dimension weight vector, and its calculation satisfies:

[0136] η=β·||w current -w previous ||1+γ

[0137] Among them, w current is the dimension weight vector of the current window; previous is the dimension weight vector of the previous window; ||w current -w previous ||1 represents the w current With the w previous The Manhattan norm between β and γ is the preset adjustment coefficient. This mechanism enables the model to adaptively adjust the prediction parameters according to changes in data distribution characteristics.

[0138] Preferably, the segmentation results and the residual coding data are stored in the form of key-value pairs, and each data segment S j The records contain:

[0139] Linear model parameters (a j ,b j );

[0140] Residual coding set q j The compressed storage address of

[0141] Coded value interval endpoints

[0142] This storage structure provides the data foundation for subsequent index construction and fast query.

[0143] In step S3, the present invention realizes fast retrieval and adaptive maintenance of data storage structure by collaboratively constructing spatial bitmap index and range B+ tree index and designing dynamic incremental update mechanism. The specific implementation process includes:

[0144] First, the coding range is determined based on the rotation Hilbert coding values ​​of all data points in the current time window. The maximum coding value h of all rotation Hilbert coding values ​​in the current time window is calculated. max =max{h p |p∈W k} and the minimum encoding value h min =min{h p |p∈W k}, where W k Represents the data set of the kth time window. The bitmap resolution m is dynamically determined based on the encoding range, and its calculation formula is:

[0145]

[0146] This resolution allows the coding space to be divided into 2 m ×2 m Each grid corresponds to a unique integer encoding interval, which ensures index accuracy while avoiding excessive expansion of storage space.

[0147] Then, define the grid coding interval division rule. Each grid g∈[1,2 2m The corresponding coding interval endpoints are calculated using the following formula:

[0148]

[0149] in, Indicates the encoding interval step of a single grid. Through this division method, the encoding space is evenly mapped to the bitmap matrix Enables visual representation of data distribution.

[0150] During the bitmap index construction process, for each data segment S j The coding interval Perform spatial relationship judgment. If there is a grid g that satisfies The flag of the corresponding position in the bitmap matrix is ​​set to 1, that is, B(g) = 1; otherwise, it remains 0. This operation quickly screens candidate data segments through bit operations, significantly improving the efficiency of range queries.

[0151] Furthermore, a range B+ tree index is constructed to support accurate data location. As the key value, the following tuple is stored in the B+ tree leaf node:

[0152]

[0153] where {g k} represents the set of bitmap grid numbers associated with the data segment. The ordered hierarchical structure of the B+ tree allows for quick retrieval of the storage location of the data segment covering a specific coding interval.

[0154] When a significant change in the dimension weight vector is detected, the incremental update mechanism of the index is triggered. Define the weight change Δw = ||w new -w old ||2, where w new and w old are the weight vectors of the current and previous windows respectively, and ||·||2 represents the Euclidean norm. If Δw exceeds the adaptive threshold δ, the following update operation is performed:

[0155]

[0156] Among them, λ d is the weight change sensitivity coefficient of the d-th dimension, is the weight value of the dth dimension in the previous window. This threshold is dynamically adjusted based on the historical weight fluctuation characteristics to avoid frequent invalid updates.

[0157] The incremental update operation involves recalculating the rotated Hilbert code values ​​affected by the weight change and generating a new spatial bitmap index only for the region where the code value has changed. For the range B+ tree index, the code interval endpoints of the affected data segment are updated. The query efficiency is maintained by adjusting the balanced tree structure. This mechanism significantly reduces the computational overhead of global reconstruction while ensuring index consistency.

[0158] Preferably, a lazy marking strategy is employed during bitmap index updates. For data points whose code values ​​shift due to weight changes, their original grid markers are reset to 0 only after confirming that no other valid data segments are overwriting them, thus avoiding index jitter caused by excessive updates. Furthermore, B+ tree index updates are recorded in transaction logs, ensuring that data consistency can be restored through log rollbacks in the event of system anomalies.

[0159] In step S4, the present invention achieves efficient retrieval of data within the query range through a dual-layer index collaborative filtering and model-driven data restoration mechanism. The specific implementation process includes:

[0160] First, receive the multidimensional space range to be queried defined by the user, and convert the space range to be queried QR = [x l ,x u ]×[y l ,y u ] is mapped to a rotation Hilbert coding interval. According to the coding rule defined in step S1, the lower left corner point (x l ,y l ) corresponding code value h l and the upper right corner (x u ,y u ) corresponding code value h u , forming a coding interval [h l ,h u ]. The mapping process is implemented by calling the same space-filling curve recursive segmentation algorithm as the encoding generation stage to ensure strict consistency between the query range and the data encoding rules.

[0161] Use the spatial bitmap index to filter the candidate data segment set. Traverse all grids g marked as 1 in the bitmap matrix B. If the corresponding encoding interval With [h l ,h u ] If there is an intersection, the data segment associated with the grid is added to the candidate set The specific judgment conditions are:

[0162]

[0163] This step quickly filters out irrelevant data segments through the bitmap, significantly reducing the scope of subsequent processing.

[0164] The physical storage address of the candidate data segment is located by the value range B+ tree index. For the query key, perform a range search operation in the B+ tree to obtain The storage location information of all data segments in the. Extract the stored linear model parameters (a j ,b j ) and calculate the prediction range covered by the model:

[0165]

[0166] Here, Hilbert_Decode(h) represents a function for inversely solving spatial coordinates from the code value h, and its implementation logic is the inverse operation of the code generation process in step S1.

[0167] Specifically, the linear model parameters (a j ,b j ) and the forecast range The association logic is specifically manifested as follows:

[0168] Each data segment S j In the process of segmented compression, the rotation Hilbert code values ​​of its internal data points are fitted by the linear regression model to obtain the form h p =a j p+b j +e p The expression of where p is the serial number of the data point in the segment, e p is the residual.

[0169] Model parameters (a j ,b j ) implies the approximate distribution law of the data segment in the coding space:

[0170] a j Indicates the linear growth rate of the code value as the data point sequence number changes, reflecting the distribution density of the data in the code space;

[0171] b j Indicates the initial offset of the linear model, corresponding to the encoding value reference of the starting position of the data segment.

[0172] Based on this, data segment S j The encoding interval endpoints and It can be calculated as follows:

[0173]

[0174] Among them, |S j| is the total number of data points in the segment. Parameter (a j ,b j ) and the residual set q j ={e p}Together define the complete encoding information of the data segment.

[0175] During the query processing phase, the prediction range is quickly calculated using parameters The boundaries:

[0176]

[0177] This range covers the linear model prediction space area when the residual is not considered, providing a quick prediction basis for query range coverage determination.

[0178] Furthermore, the query range coverage determination is performed on each data segment. If the query range QR is completely contained in Satisfaction All data points in the data segment are directly returned without decoding the residual code. This judgment is based on the spatial inclusion relationship between the prediction range and the query range of the linear model, and the model parameters are used to quickly skip the precise verification step.

[0179] When the model coverage cannot completely contain the query range, the residual encoding result q j Decoding operation of data segment S j Each data point (x p ,y p )∈S j , according to its residual e p ∈q j Restore the original encoding value:

[0180] h p =a j p+b j +e p

[0181] Then calculate its spatial coordinates (x p ,y p ). If the coordinates satisfy x l ≤x p ≤x u And y l ≤y p ≤y u , then add it to the final query result set.

[0182] Preferably, a batch processing optimization strategy is employed during the residual decoding process. For continuously stored residual data blocks, SIMD instructions are used to parallelize the encoded values ​​of multiple data points, accelerating the coordinate transformation process. Furthermore, a spatial pruning algorithm is employed during the coordinate verification phase to prioritize points that clearly fall outside the query range, reducing unnecessary numerical comparisons.

[0183] Query result sets are returned via streaming, dynamically merging partial results as segments are processed, reducing memory usage and improving response speed. Query results spanning multiple segments are sorted in ascending order by code value and output, maintaining local order on the space-filling curve.

[0184] In step S5, the present invention uses dynamic data update and storage optimization mechanisms to achieve real-time adaptation of the system to changes in data distribution and effective management of storage resources. The specific implementation process includes:

[0185] First, process the dimension weight changes caused by the new data. When the new data arrives, re-execute the dimension weight calculation process of step S1 to generate the updated weight vector w new If the weight change Δw=||w new -w prev If ||2 exceeds the preset threshold δ, the following actions are triggered:

[0186] 1. According to w new Adjust the direction of the space-filling curve and recalculate the rotation Hilbert encoding values ​​of all data points;

[0187] 2. Perform the incremental index update of step S3 for the area where the code value changes to ensure the consistency of the spatial bitmap index and the range B+ tree index.

[0188] Secondly, implement the expired data elimination strategy. j Record its last updated timestamp When the system current time t current satisfy:

[0189]

[0190] When , the data segment is marked as an invalid segment. expired It is the preset expiration time threshold, and its value is related to the data update frequency.

[0191] At the index update level, perform logical deletion operations on the failed segments:

[0192] In the range B+ tree index, S j The corresponding leaf node is marked as logically deleted, and its encoding interval endpoint is retained. However, the storage pointer is set to an invalid address. This operation is achieved by modifying the node metadata field to avoid damage to the index structure caused by immediate deletion.

[0193] In the spatial bitmap index, traverse all If there is an intersection grid g, and grid g is no longer associated with any valid data segment, then B(g) is set to 0. This determination is made by querying whether there is another valid segment encoding interval covering the B+ tree. accomplish.

[0194] Furthermore, physical consolidation of storage space is performed. Calculate the storage space utilization rate occupied by the failed segment When ρ<τ (τ is the preset threshold), the merging operation is started:

[0195] 1. Scan the physical storage area and identify a continuous sequence of free storage blocks {B1, B2, ..., B k};

[0196] 2. Merge adjacent free blocks into a block with a capacity of New storage block;

[0197] 3. Update the storage pointer address of the affected node in the range B+ tree index to ensure that the mapping relationship between the index and the storage location is correct.

[0198] Preferably, a delay window mechanism is introduced between logical deletion and physical merging. The storage block marked as invalid is delay The recoverable state remains within the time window. If a data rollback request is received during this period, the data segment and associated indexes can be restored through the transaction log. This mechanism is implemented by maintaining the transaction ID and timestamp of the delete operation, enhancing the system's fault tolerance.

[0199] Storage optimization and index update operations are executed through asynchronous task queues to avoid blocking real-time data processing. The task queue scheduler dynamically adjusts the execution priority of merge operations based on system load. When the storage fragmentation level exceeds a critical value, the merge task of the highly fragmented area is triggered first. The fragmentation level measurement formula is:

[0200]

[0201] When F>φ (φ is the system preset threshold), it is determined to be a high-fragmentation area.

[0202] In general, the present invention achieves efficient data processing through dynamic dimension weight allocation and spatiotemporal collaborative indexing technology. Its overall process includes: dynamically generating dimension weight vectors based on the information entropy of multidimensional data streams, adjusting the direction of the space filling curve to generate rotational Hilbert codes; dynamically determining the number of data segments and the error threshold using a time series prediction model, sorting the encoded data, and compressing the storage parameters and residuals according to the linear model; constructing a two-layer index structure (spatial bitmap index and value range B+ tree index) and designing an incremental update mechanism, and locally reconstructing the index when the dimension weight change exceeds the threshold; mapping the query range to the encoding interval, collaboratively screening candidate data segments through the index, and combining the linear model with residual decoding to accurately return the results; dynamically correcting the encoding rules and index status according to the newly added data, asynchronously eliminating expired data and optimizing the storage space, forming a closed-loop processing link of data compression, index construction, query optimization and dynamic update.

[0203] The big data compression storage and rapid retrieval system described below and the big data compression storage and rapid retrieval method described above can be referenced to each other.

[0204] The present invention also provides a big data compression storage and rapid retrieval system, which consists of a dynamic encoding module, a segmented compression module, a collaborative indexing module, a query engine and a data update agent. The modules work together to achieve data compression storage and efficient retrieval.

[0205] The dynamic encoding module receives the data stream in real time and generates a dynamic weight vector by calculating the information entropy of the data distribution in each dimension. Based on the weight vector, it adjusts the dimensional expansion direction of the Hilbert curve to generate a rotated Hilbert code value for each data point. This module transmits the encoding result to the segmentation compression module, triggering the data segmentation process.

[0206] The segment compression module includes an ARIMA prediction unit, a dynamic threshold calculation unit, and a residual coding unit. The ARIMA prediction unit predicts the number of segments in the next time window based on the historical segment count sequence. The dynamic threshold calculation unit generates an error threshold based on the predicted value to control the data segment granularity. The residual coding unit performs a linear fit on the sorted data segments, stores the model parameters, and compresses the residual data. The compression results are transmitted to the collaborative indexing module to drive index construction.

[0207] The collaborative indexing module consists of a bitmap generator, a B+ tree manager, and an incremental update controller. The bitmap generator constructs a spatial bitmap index based on the distribution of encoding values, marking the spatial coverage relationship between data segments and grids. The B+ tree manager constructs an ordered index using the endpoints of the data segment encoding interval as key values. Its leaf nodes associate storage addresses with bitmap grid states through bidirectional pointers. The incremental update controller monitors changes in the weight vector in real time. When the change exceeds a threshold, it triggers a partial reconstruction of the bitmap index and an update of the B+ tree nodes to ensure synchronization between the index and the data distribution.

[0208] The query engine receives external query requests, maps the spatial range to a coded interval, uses bitmap indexes to select candidate data segments, and locates the storage location using a B+ tree. For candidate segments, it quickly determines spatial coverage relationships using linear model parameters. For segments that are not fully covered, it decodes residual data and verifies coordinates, ultimately returning accurate query results.

[0209] The data update agent asynchronously handles new data additions and expired data maintenance. New data triggers weight recalculation and incremental index updates in the dynamic encoding module. Expired data segments are marked as logically deleted in the B+ tree, and their associated bitmap grid states are simultaneously cleared. Physical storage space merge operations are executed via an asynchronous task queue, merging free blocks and updating index storage pointers to maintain storage efficiency. The system's modular design and asynchronous communication mechanisms enable efficient coordination of data compression, index optimization, and dynamic updates.

[0210] The system of this embodiment can be used to execute the above method embodiments, and its principles and technical effects are similar, so they will not be repeated here.

[0211] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for compressing, storing and quickly retrieving large amounts of data, characterized in that: The following steps are involved: S1, dynamically generating a dimension weight vector based on the multi-dimensional information entropy in the data stream, adjusting the dimension priority of the space filling curve according to the dimension weight vector, and generating a rotated Hilbert code value for each data point; S2, using a time series prediction model to predict the number of data segment divisions, dynamically calculating a segmentation error threshold based on the predicted number of data segment divisions, segmenting the sorted data based on the rotated Hilbert code values, and storing piecewise linear model parameters used to describe each of the data segments and a residual coding result of each of the data segments; S3, constructing a spatial bitmap index based on the distribution range of the rotated Hilbert code value, and constructing a range B+ tree index based on the code interval endpoint value of each data segment, and triggering an incremental update of the spatial bitmap index and the range B+ tree index when it is detected that the change of the dimension weight vector exceeds a preset threshold; S4, mapping the query range into a rotated Hilbert coding interval, using the spatial bitmap index to screen candidate data segments, locating the target data segment through the range B+ tree index, and returning the data within the query range by combining the piecewise linear model parameters and the residual coding result; S5, dynamically modifying the dimension weight vector and the rotated Hilbert code value according to the newly added data, eliminating expired data and synchronously updating the spatial bitmap index and the range B+ tree index.

2. The method for compressing, storing and rapidly retrieving large data according to claim 1, characterized in that: In step S1, the step of generating the rotated Hilbert code value of each data point includes: S11, for each dimension in the data stream, divide the corresponding spatial range into the same number of uniform grids, and count the number of data points in each grid of each dimension; S12, for the dth dimension, calculate the data distribution probability of the dth dimension in the bth grid The calculation formula for the data distribution probability is: in, Indicates the amount of data in the bth grid of the dth dimension; N is the total number of all data points in the data stream in the current time window; b∈[1,B], B is the preset number of grid divisions; S13, based on the calculated data distribution probability, calculate the information entropy S of the d-th dimension d , the calculation formula of the information entropy is: S14, based on the calculated information entropy values ​​of each dimension, generate an n-dimensional dimension weight vector w=(w1,w2,...,w n ); Among them, the weight value w corresponding to the d-th dimension d Calculated by the following formula: Among them, S i represents the information entropy of the i-th dimension; S15, based on the generated dimension weight vector w, adjusting the dimension access priority of the Hilbert space filling curve when processing multidimensional data, and for the dimension with the highest weight value, extending the Hilbert space filling curve along the dimensional direction of the dimension with the highest weight value; S16, generating a corresponding rotational Hilbert code value for each data point in the data stream according to the Hilbert space filling curve after the dimension priority is adjusted.

3. The method for compressing, storing and rapidly retrieving large data according to claim 1, wherein: In step S2, the time series prediction model is an autoregressive integrated moving average model, and the step of using the time series prediction model to predict the number of data segment divisions includes: S21, obtain the number of data segments actually observed in a series of historical time windows to form an original time series, recorded as {k1, k2, ..., kt}, and s } Perform d-order difference operation to obtain a stable time series {Δ d k t }, where k s represents the actual number of segments of the sth historical time window, t is the serial number of the latest historical time window currently observed, Δ d k t represents the d-order difference value of the original sequence in the t-th window, where d is the preselected difference order; S22, based on the obtained stationary time series {Δ d k t }, establish an ARIMA (p, d, q) model, the expression of the ARIMA (p, d, q) model is: Among them, Δ d k t represents the d-order difference value of the original sequence in the t-th time window, p is the order of the autoregressive term of the model, q is the order of the moving average term of the model, φ i is the autoregressive coefficient, i=1,2,...,p, θ j is the moving average coefficient, j=1,2,...,q,∈ t is the white noise residual term of the t-th window, Δ d k t-i is the d-order difference value of the original sequence in the ti-th time window, ∈ t-j is the white noise residual term in the tj-th time window; S23, using the established ARIMA (p, d, q) model and the next window division number prediction formula, predict the number of data segment divisions in the next time window, the next window division number prediction formula is: in, The number of segments to divide the forecast data into for the next time window.

4. The method for compressing, storing and rapidly retrieving large data according to claim 3, wherein: The step of using the time series prediction model to predict the number of data segment divisions also includes the step of dynamically adjusting the moving average coefficient in the ARIMA (p, d, q) model according to the number of data segment divisions actually occurring in the period after each time window processing period ends, specifically: S24, for the currently ended time window, obtain the actual number of data segment divisions k that actually occurred in the current window real ; S25, obtaining the number of predicted data segments generated by the ARIMA (p, d, q) model in the previous time window for the same current window S26, calculating the actual number of data segment divisions k real Divide the number of segments with the forecast data The deviation Δk between them is calculated as follows: S27, obtaining the dimension weight vector w calculated for the current window current , and the dimension weight vector w calculated from the previous time window of the current window previous ; S28, calculating the learning rate η for subsequent correction of the moving average coefficient, wherein the calculation formula of the learning rate η is: η=β·||w current -w previous ||1+c Among them, w current is the dimension weight vector of the current window; previous is the dimension weight vector of the previous window; ||w current -w previous ||1 represents the w current With the w previous The Manhattan norm between β and γ; β and γ are preset adjustment coefficients; S29, for each moving average coefficient θ in the ARIMA (p, d, q) model j , the coefficient is dynamically corrected using the calculated prediction deviation Δk and the calculated learning rate η, and the corrected moving average coefficient Calculated by the following formula: in, is the value of the jth moving average coefficient before this correction operation is performed, The new value of the j-th moving average coefficient after this correction operation will be used in the forecast calculation of the ARIMA (p, d, q) model in the subsequent time window.

5. The method for compressing, storing and rapidly retrieving large data according to claim 1, wherein: In step S3, the step of constructing a spatial bitmap index according to the distribution range of the rotated Hilbert code value includes: S31, determining a coding range according to the rotation Hilbert coding values ​​of all data points in the current time window, and calculating the maximum coding value h of all the rotation Hilbert coding values ​​in the current time window. max =max{h i |i∈W k } and the minimum encoding value h min =min{h i |i∈W k }, where W k Represents the data set of the kth time window; S32, based on the calculated h max and h min , calculate the bitmap resolution parameter m by the following formula: S33, divide the entire data space composed of the rotated Hilbert code values ​​into 2 m ×2 m Grid, define each grid corresponding to a unique integer encoding subinterval in g∈[1,2 m ×2 m ] is the grid number; S34, initialize a size of 2 m ×2 m The bitmap matrix B is created, and the initial values ​​of all elements in the matrix are set to 0; S35, for each data segment formed in step S2, obtain its corresponding rotation Hilbert coding interval S36, traverse each grid in the bitmap matrix, if there is at least one data segment S j The coding interval The interval between the current grid grid g If there is an intersection, the element corresponding to the grid g in the bitmap matrix is ​​marked as 1, otherwise the element is marked as 0.

6. The method for compressing, storing and rapidly retrieving large data according to claim 5, characterized in that: In step S3, the triggering conditions for the incremental update of the spatial bitmap index and the range B+ tree index are: When the change in the dimension weight vector Δw=||w new -w old When ||2 exceeds the threshold δ, an incremental update operation is performed, where w new is the weight vector of the current window, w old is the weight vector of the previous window, ||·||2 represents the Euclidean norm; The threshold δ is adaptively adjusted according to the historical variation of the weight vector. The calculation formula of the threshold δ is: Among them, λ d is the weight change sensitivity coefficient of the d-th dimension, is the weight value of the dth dimension of the previous window; The incremental update operation includes: recalculating the rotated Hilbert code values ​​affected by the weight change, generating a new spatial bitmap index only for the area where the code value changes, and updating the code interval endpoint values ​​of the corresponding data segment in the value range B+ tree index.

7. The method for compressing, storing and rapidly retrieving large data according to claim 1, wherein: In step S4, the steps of mapping the query range into a rotated Hilbert coding interval, screening candidate data segments using the spatial bitmap index, locating the target data segment using the range B+ tree index, and returning the data within the query range by combining the piecewise linear model parameters and the residual coding result include: S41, receiving the multidimensional space range to be queried defined by the user QR = [x l ,x u ]×[y l ,y u ], mapping the spatial range to be queried into a rotation Hilbert coding interval [h l ,h u ], where h l is the lower left corner of QR (x l ,y l ) corresponding to the encoding value, h u is the upper right corner point (x u ,y u ) The corresponding code value, the mapping rule is consistent with the calculation method of the generated rotation Hilbert code value; S42, using the constructed spatial bitmap index and combining the rotation Hilbert coding interval [h l ,h u ]Filter the candidate data segment set The candidate data segment set Defined as Among them, S j For a data segment, and Data segment S j The minimum and maximum code values ​​of the corresponding rotated Hilbert code values; S43, searching through the value range B+ tree index The storage address of all data segments in the Do the following: 1) Extract the data stored in data segment S j The linear model parameters (a j ,b j ), calculate the data stored in the data segment S j The actual spatial range covered by the linear model (x,y)=Hilbert_Decode(h)}; Where Hilbert_Decode(h) represents the function of inversely solving the spatial coordinates from the encoded value h; 2) According to the spatial range of the data segment The relationship between the spatial range QR to be queried is used for conditional judgment and data extraction: Condition 1: If the spatial range to be queried is determined Then directly process the data segment S j To return all the original data points contained within it; Condition 2: If condition 1 is not met, the decoded data is stored in data segment S j The residual coding result q j , for each by using the piecewise linear model parameters (a j ,b j ) and the decoded residual coding result q j The original data point (x p ,y p ), if the reconstructed original data point (x p ,y p ) satisfies the boundary conditions x of the query space range QR l ≤x p ≤x u And y l ≤y p ≤y u , then the data point (x p ,y p ) is added to the final query result set.

8. The method for compressing, storing and rapidly retrieving large data according to claim 1, wherein: In step S5, the step of eliminating expired data and synchronously updating the spatial bitmap index and the range B+ tree index includes: S51, for each data segment in the system. j Record its latest update timestamp S52, periodically or according to a predetermined strategy, check all data segments. If for a certain data segment S j , which meets the condition When the data segment S is marked j is the failure segment, where T expired is the preset expiration time threshold, t current The current time of the system; S53, for each invalid data segment, mark the leaf node corresponding to the invalid segment as a logically deleted state in the value range B+ tree index, and retain its encoding interval endpoint value But the updated storage pointer is an invalid address; S54: In the spatial bitmap index, all bitmap grids that intersect with the coding interval of the invalid data segment are identified. For each bitmap grid, check whether it intersects with the coding interval of any other data segment that is not marked as invalid. If the bitmap grid no longer intersects with the coding interval of any valid data segment after removing the association with the invalid data segment, update the bitmap flag B(g) corresponding to the bitmap grid to 0; otherwise, if the bitmap grid is still associated with other valid data segments, its flag B(g) remains 1. S55, when the storage space utilization rate of the failed segment is lower than a preset threshold, physical deletion and storage merging operations are performed: continuous free storage blocks are merged into a larger available space, and the storage pointer address of the affected node in the range B+ tree index is updated.

9. A big data compression storage and rapid retrieval system for implementing the method according to any one of claims 1 to 8, characterized in that: include: A dynamic encoding module is used to receive real-time data streams, calculate the information entropy of each dimension and generate a dimension weight vector, adjust the dimension priority of the space filling curve according to the dimension weight vector, generate a rotated Hilbert code value, and transmit the encoding result to the segment compression module; The segmented compression module is connected to the dynamic coding module and includes an ARIMA prediction unit, a dynamic threshold calculation unit, and a residual coding unit, where: The ARIMA prediction unit predicts the number of segments in the next time window based on the historical segment number sequence; The dynamic threshold calculation unit dynamically adjusts the error threshold based on the predicted number of segments to control the granularity of data segment division; The residual coding unit compresses and encodes the segmented linear model parameters and residual data, and sends the compression results to the collaborative indexing module; The collaborative indexing module communicates bidirectionally with the segment compression module and the dynamic encoding module, and includes a bitmap generator, a B+ tree manager, and an incremental update controller, among which: The bitmap generator constructs a spatial bitmap index according to the distribution range of the rotation Hilbert code value; The B+ tree manager constructs a value range B+ tree index based on the encoding interval endpoint values ​​of the data segment; The incremental update controller monitors the changes in the dimension weight vector and triggers local reconstruction of the index when the change exceeds a threshold; The query engine is connected to the collaborative index module, receives external query requests and maps the spatial range to the coding interval, uses the spatial bitmap index and the range B+ tree index to screen candidate data segments, and returns the query data in combination with the linear model parameters and the residual decoding results; The data update agent interacts with the dynamic encoding module, collaborative indexing module, and segment compression module to handle encoding update operations for newly added data, eliminate expired data segments, and maintain consistency between indexes and storage space.

10. The big data compression storage and rapid retrieval system according to claim 9, characterized in that: The incremental update controller has a built-in weight change monitoring channel, which receives the weight vector change output by the dynamic encoding module in real time. When the weight vector change exceeds a preset threshold, it sends a local reconstruction instruction to the bitmap generator and the B+ tree manager; The leaf nodes of the B+ tree manager contain a bidirectional pointer structure: The forward pointer associates the encoding interval endpoint value with the storage address; The back pointer associates the storage address with the corresponding space bitmap grid mark; The bidirectional pointer supports fast location of storage blocks according to the encoding value and updates the bitmap index status in the reverse direction.

Images