A big data compression storage and fast retrieval method and system

Abstract

The application relates to the field of data storage, and discloses a big data compression storage and fast retrieval method and system, which comprises the following steps: dynamically generating a dimension weight vector based on multi-dimensional information entropy, and generating a rotating Hilbert code by adjusting the direction of a space-filling curve; adaptively determining the number of segments and an error threshold by using a time series prediction model, and compressively storing coding residual errors by using a linear model; constructing a double-layer collaborative index structure comprising a space bitmap index and a value range B+ tree index, and realizing local index reconstruction when the weight changes through an incremental updating mechanism; mapping a query range to a coding interval, screening candidate data segments in combination with the index, and returning a result by fusing model parameters and residual error decoding; asynchronously processing new data coding correction and obsolete data elimination, and optimizing storage space consistency. The application realizes the unification of high compression rate, fast retrieval and adaptive update.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a method and system for big data compression storage and fast retrieval. Background Technology

[0002] In the field of data storage technology, with the popularization of the Internet of Things and smart terminal devices, the real-time processing of multidimensional spatiotemporal data faces multiple challenges, including a surge in data volume, enhanced dynamic correlation of dimensions, and increased requirements for real-time queries. Traditional encoding methods based on static space-filling curves typically fix the order of dimension expansion, making it difficult to adapt to the temporal evolution of data distribution characteristics. This leads to weakened locality of the encoded value space and the generation of a large amount of redundant information during compression. Existing indexing mechanisms mostly use a single structure to organize encoded data. In dynamic weight scenarios, index updates require global reconstruction, causing storage jitter and degraded query performance. At the same time, the lack of adaptive control over the time-varying characteristics of data flow during data segmentation compression, coupled with fixed error thresholds, easily leads to inefficient segmentation, creating a contradiction between storage fragmentation and query efficiency.

[0003] In addition, the existing system lacks a collaborative management mechanism with the index structure for expired data eviction and storage space reclamation. Frequent data updates lead to a decrease in storage resource utilization, making it difficult to support the demand for high-efficiency data services in the long term.

[0004] The aforementioned technical bottlenecks have limited the simultaneous improvement of storage compression efficiency and retrieval real-time performance in large-scale dynamic data scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for big data compression storage and fast retrieval, which solves the technical problems of high coding redundancy, large index update overhead, severe storage fragmentation, and low query efficiency caused by changes in dimensional correlation in dynamic data scenarios.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for big data compression storage and fast retrieval, comprising the following steps:

[0007] S1, dynamically generate dimension weight vectors based on multi-dimensional information entropy in the data stream, adjust the dimension priority of the space filling curve according to the dimension weight vectors, and generate the rotation Hilbert code value of each data point;

[0008] S2, use a time series prediction model to predict the number of data segments, dynamically calculate the segmentation error threshold based on the predicted number of data segments, segment the sorted data based on the rotated Hilbert code value, and store the segmented linear model parameters and residual coding results of each data segment to describe each data segment.

[0009] S3, construct a spatial bitmap index based on the distribution range of the rotated Hilbert encoded values, and construct a value domain B+ tree index by combining the encoding interval endpoint values ​​of each data segment. When the change in the dimension weight vector exceeds a preset threshold, the incremental update of the spatial bitmap index and the value domain B+ tree index is triggered.

[0010] S4, map the query range to a rotated Hilbert encoding interval, use the spatial bitmap index to filter candidate data segments, locate the target data segment through the value domain B+ tree index, and return the data within the query range by combining the piecewise linear model parameters and the residual encoding result;

[0011] S5. Dynamically correct the dimension weight vector and rotation Hilbert encoding value based on the newly added data, eliminate expired data, and synchronously update the spatial bitmap index and value range B+ tree index.

[0012] Preferably, in step S1, the step of generating the rotated Hilbert code value for each data point includes:

[0013] S11, for each dimension in the data stream, divide the corresponding spatial range into the same number of uniform grids, and count the number of data points in each grid of each dimension;

[0014] S12, For the d-th dimension, calculate the data distribution probability of the d-th dimension within the b-th grid. The formula for calculating the probability distribution of the data is:

[0015]

[0016] in, This represents the amount of data in the b-th grid of the d-th dimension; N is the total number of all data points in the data stream within the current time window; b∈[1,B], where B is the preset number of grid divisions;

[0017] S13, Based on the calculated data distribution probability, calculate the information entropy S of the d-th dimension. d The formula for calculating the information entropy is:

[0018]

[0019] S14, Based on the calculated information entropy values ​​of each dimension, generate an n-dimensional dimension weight vector w = (w1, w2, ..., w...). n ); where the weight value w corresponding to the d-th dimension. d Calculated using the following formula:

[0020]

[0021] Among them, S iThis represents the information entropy of the i-th dimension;

[0022] S15, Based on the generated dimension weight vector w, adjust the dimension access priority of the Hilbert space filling curve when processing multidimensional data. For the dimension with the highest weight value, extend the Hilbert space filling curve along the dimension direction of the dimension with the highest weight value.

[0023] S16, based on the Hilbert space-filling curve adjusted for dimensional priority, generate the corresponding rotated Hilbert code value for each data point in the data stream.

[0024] Preferably, in step S2, the time series prediction model is an autoregressive integral moving average model, and the step of using the time series prediction model to predict the number of data segment divisions includes:

[0025] S21, Obtain the number of data segments actually observed within a series of historical time windows, forming the original time series, denoted as {k1,k2,....,k t}, for the original sequence {k s Performing d-order difference operations yields a stationary time series {Δ}. d k t}, where k s Δ represents the actual number of segments in the s-th historical time window, t is the sequence number of the latest historical time window observed, and Δ d k t This represents the d-th order difference value of the original sequence in the t-th window, where d is the pre-selected difference order;

[0026] S22, based on the obtained stationary time series {Δ d k t}, establish an ARIMA(p,d,q) model, the expression of which is:

[0027]

[0028] Where, Δ d k t Let d represent the d-th order difference of the original sequence at time window t, p be the order of the autoregressive term in the model, q be the order of the moving average term in the model, and φ be the order of the moving average term in the model. i Here are the autoregressive coefficients, i = 1, 2, ..., p, θ j Let be the moving average coefficient, j = 1, 2, ..., q, ∈ t, and Δ be the white noise residual term for the t-th window. d k t-i Let be the d-th order difference value of the original sequence in the ti-th time window, ∈ t-j For the white noise residual term in the tj-th time window;

[0029] S23, using the established ARIMA(p,d,q) model and the next window partitioning quantity prediction formula, predict the number of data segment partitions for the next time window. The next window partitioning quantity prediction formula is as follows:

[0030]

[0031] in, The number of segments to divide the forecast data for the next time window.

[0032] Preferably, the step of predicting the number of data segment divisions using a time series forecasting model further includes a step of dynamically adjusting the moving average coefficient in the ARIMA(p,d,q) model based on the actual number of data segment divisions that occurred within each time window processing period after the end of that period, specifically:

[0033] S24, For the currently ended time window, obtain the actual number of data segment divisions k that actually occurred within the current window. real ;

[0034] S25, obtain the number of predicted data segments generated by the ARIMA(p,d,q) model in the previous time window for the same current window.

[0035] S26, Calculate the actual data segment division number k real Number of segments to be divided with the predicted data The deviation Δk between them is calculated using the following formula:

[0036] S27, Obtain the dimension weight vector w calculated for the current window. current And the dimension weight vector w calculated from the previous time window of the current window. previous ;

[0037] S28, Calculate the learning rate η used for subsequent adjustments to the moving average coefficients. The formula for calculating the learning rate η is as follows:

[0038] η=β·||w current -w previous ||1+γ

[0039] Among them, w current w is the dimension weight vector of the current window; previous Let w be the dimension weight vector of the previous window; current -w previous ||1 represents the w current With the w previousThe Manhattan norm between; β and γ are preset adjustment coefficients;

[0040] S29, for each moving average coefficient θ in the ARIMA(p,d,q) model j The coefficient is dynamically corrected using the calculated prediction bias Δk and the calculated learning rate η, resulting in a corrected moving average coefficient. Calculated using the following formula:

[0041]

[0042] in, Let be the value of the j-th moving average coefficient before this correction operation was performed. This is the new value of the j-th moving average coefficient after this correction operation. This new value will be used in the ARIMA(p,d,q) model's prediction calculations for subsequent time windows.

[0043] Preferably, in step S3, the step of constructing a spatial bitmap index based on the distribution range of the rotated Hilbert encoded values ​​includes:

[0044] S31, determine the encoding range based on the rotated Hilbert encoding values ​​of all data points within the current time window, and calculate the maximum encoding value h of all rotated Hilbert encoding values ​​within the current time window. max =max{h i |i∈W k} and minimum encoded value h min =min{h i |i∈W k}, where W k This represents the data set for the k-th time window;

[0045] S32, based on the calculated h max and h min The resolution parameter m of the bitmap is calculated using the following formula:

[0046]

[0047] S33 divides the overall data space consisting of rotated Hilbert encoded values ​​into 2 m ×2 m The grid defines a unique integer-encoded sub-range for each grid. in g∈[1,2 m ×2 m ] represents the grid number;

[0048] S34, initialize a space with a size of 2 m ×2m The bitmap matrix B is generated, and the initial value of all elements in the matrix is ​​set to 0;

[0049] S35, for each data segment formed in step S2, obtain its corresponding rotated Hilbert encoding interval.

[0050] S36, traverse each grid in the bitmap matrix, if there exists at least one data segment S j encoding range The interval between the current grid grid g If an intersection exists, the element in the bitmap matrix corresponding to grid g is marked as 1; otherwise, the element is marked as 0.

[0051] Preferably, in step S3, the triggering condition for the incremental update of the spatial bitmap index and the value domain B+ tree index is:

[0052] When the change in the dimension weight vector Δw = ||w new -w old When ||2 exceeds the threshold δ, an incremental update operation is performed, where w new w is the weight vector of the current window. old Let ||·||2 be the weight vector of the previous window, and ||·||2 represent the Euclidean norm.

[0053] The threshold δ is adaptively adjusted based on the historical change magnitude of the weight vector, and the formula for calculating the threshold δ is as follows:

[0054]

[0055] Where, λ d Let be the sensitivity coefficient for weight changes in the d-th dimension. This represents the weight value of the d-th dimension of the previous window;

[0056] The incremental update operation includes: recalculating the rotated Hilbert code value affected by the weight change, generating a new spatial bitmap index only for the region where the code value changes, and updating the endpoint value of the corresponding data segment's code interval in the value domain B+ tree index.

[0057] Preferably, in step S4, the steps of mapping the query range to a rotated Hilbert encoding interval, filtering candidate data segments using the spatial bitmap index, locating the target data segment using the value range B+ tree index, and returning the data within the query range by combining the piecewise linear model parameters and the residual encoding result include:

[0058] S41, Receive the user-defined multidimensional spatial range to be queried QR = [x l ,x u ]×[y l,y u The spatial range to be queried is mapped to a rotated Hilbert encoding interval [h]. l ,h u ], where h l The bottom left corner of QR (x l ,y l The corresponding encoded value, h u The upper right corner point (x u ,y u The corresponding encoded value follows the same mapping rule as the calculation method for the generated rotated Hilbert encoded value;

[0059] S42, using the constructed spatial bitmap index, and combining it with the rotated Hilbert encoding interval [h l ,h u Filter the set of candidate data segments The candidate data segment set Defined as Among them, S j For a data segment, and Data segment S j The minimum and maximum coded values ​​of the corresponding rotated Hilbert codes;

[0060] S43, retrieve via the value range B+ tree index The storage addresses of all data segments in the data, for each data segment Perform the following operations:

[0061] 1) Extract data stored in data segment S j Linear model parameters (a) j ,b j ), calculate the data stored in data segment S j The actual spatial range covered by the linear model (x,y)=Hilbert_Decode(h)};

[0062] Here, Hilbert_Decode(h) represents the function that deciphers the spatial coordinates from the encoded value h;

[0063] 2) Based on the spatial range of the data segment The relationship between the query range QR and the query result is used to perform conditional judgments and data extraction.

[0064] Condition 1: If the range of the space to be queried is determined... Then directly process data segment S j To return all the original data points contained within it;

[0065] Condition 2: If condition 1 is not met, then the decoded data is stored in data segment S. j The residual coding result q j For each, by using the piecewise linear model parameters (a) j ,b j and the residual encoding result q from the decoded data. j The original data points (x) are reconstructed from the corresponding residual values ​​obtained in the middle. p ,y p If the reconstructed original data point (x) p ,y p The boundary conditions x satisfy the query space range QR. l ≤x p ≤x u And y l ≤y p ≤y u Then the data point (x) p ,y p Add it to the final query result set.

[0066] Preferably, in step S5, the step of eliminating expired data and synchronously updating the spatial bitmap index and the value range B+ tree index includes:

[0067] S51 records the latest update timestamp for each data segment S0 in the system.

[0068] S52, periodically or according to a predetermined strategy, check all data segments, if for a certain data segment S j When the condition is met, the data segment S is marked. j This is the failed segment, where T expired t is the preset expiration time threshold. current The current system time;

[0069] S53, for each failed data segment, in the value domain B+ tree index, the leaf node corresponding to the failed segment is marked as logically deleted, and its encoding interval endpoint value is retained. However, the memory pointer is updated to an invalid address;

[0070] S54, in the spatial bitmap index, identify all bitmap grids that intersect with the coding interval of the failed data segment. For the bitmap grid, check whether it still intersects with the coding interval of any other data segment that has not been marked as failed. If the bitmap grid no longer intersects with the coding interval of any valid data segment after removing the association with the failed data segment, then update the bitmap tag B(g) corresponding to the bitmap grid to 0; otherwise, if the bitmap grid is still associated with other valid data segments, then its tag B(g) remains 1.

[0071] S55, when the storage space utilization rate occupied by the failed segment is lower than the preset threshold, perform physical deletion and storage merging operations: merge consecutive free storage blocks into a larger capacity available space, and update the storage pointer address of the affected node in the value domain B+ tree index.

[0072] This invention also provides a big data compression storage and fast retrieval system, comprising:

[0073] The dynamic encoding module is used to receive real-time data streams, calculate the information entropy of each dimension and generate a dimension weight vector, adjust the dimension priority of the space filling curve according to the dimension weight vector, generate rotated Hilbert encoded values, and transmit the encoding results to the segmented compression module.

[0074] The segmented compression module, connected to the dynamic coding module, includes an ARIMA prediction unit, a dynamic threshold calculation unit, and a residual coding unit, wherein:

[0075] The ARIMA prediction unit predicts the number of segments in the next time window based on the historical segment number sequence;

[0076] The dynamic threshold calculation unit dynamically adjusts the error threshold based on the predicted number of segments to control the granularity of data segment division.

[0077] The residual encoding unit compresses and encodes the segmented linear model parameters and residual data, and sends the compression result to the collaborative indexing module.

[0078] The collaborative indexing module communicates bidirectionally with the segmentation compression module and the dynamic encoding module. It includes a bitmap generator, a B+ tree manager, and an incremental update controller, among which:

[0079] The bitmap generator constructs a spatial bitmap index based on the distribution range of the rotated Hilbert encoded values;

[0080] The B+ tree manager constructs a value-domain B+ tree index based on the endpoint values ​​of the encoded interval of the data segment;

[0081] The incremental update controller monitors changes in the dimension weight vector and triggers a local reconstruction of the index when the change exceeds a threshold.

[0082] The query engine, connected to the collaborative index module, receives external query requests and maps spatial ranges to encoded intervals. It uses the spatial bitmap index and the value domain B+ tree index to filter candidate data segments and returns query data by combining linear model parameters and residual decoding results.

[0083] The data update agent interacts with the dynamic encoding module, collaborative indexing module, and segmented compression module to handle encoding update operations for newly added data, evict expired data segments, and maintain the consistency between the index and storage space.

[0084] Preferably, the incremental update controller has a built-in weight change monitoring channel to receive the weight vector change output by the dynamic encoding module in real time. When the weight vector change exceeds a preset threshold, it sends a local reconstruction instruction to the bitmap generator and the B+ tree manager.

[0085] The leaf nodes of the B+ tree manager contain a bidirectional pointer structure:

[0086] The forward pointer associates the endpoint value of the encoding interval with the storage address;

[0087] Back pointers are associated with memory addresses and corresponding space bitmap grid markers;

[0088] The bidirectional pointers enable quick location of storage blocks based on encoded values ​​and reverse update of the bitmap index state.

[0089] This invention provides a method and system for big data compression, storage, and rapid retrieval. It offers the following advantages:

[0090] 1. This invention significantly improves data compression efficiency through a dynamic encoding mechanism driven by multidimensional information entropy. It generates weight vectors based on real-time calculation of the information entropy of data distribution across various dimensions, dynamically adjusting the dimensional expansion direction of the Hilbert curve. This results in denser clustering of data in high-discrimination dimensions, effectively reducing spatial redundancy after encoding. Combined with residual encoding technology in the segmented compression module, it reduces storage space usage while maintaining accuracy.

[0091] 2. This invention significantly optimizes query response speed through a spatiotemporal collaborative index architecture and a model-based prediction mechanism. Spatial bitmap indexes quickly filter candidate data segments, while B+ tree indexes accurately locate physical storage positions. Combined with the pre-calculation of the spatial coverage of data segments using linear model parameters, the residual decoding process for invalid data segments can be skipped, reducing disk I / O and computational overhead, thus achieving a hierarchical improvement in query efficiency.

[0092] 3. This invention enhances the dynamic adaptability of the system based on the weight-sensitive mechanism of the incremental update controller. By monitoring the change amplitude of the dimension weight vector in real time, local index reconstruction is triggered only when the weight change exceeds the threshold, avoiding the overhead of global index reconstruction. Combined with the bidirectional pointer design of B+ leaf nodes, the atomicity of the index update process and storage structure adjustment is ensured, maintaining the stability of the system in high-frequency data update scenarios.

[0093] 4. This invention achieves efficient utilization of storage resources through a data lifecycle management strategy. The data update agent adopts a collaborative mechanism of logical deletion marking and physical storage merging. It maintains invalid segment marking information in the B+ tree index, asynchronously executes storage space merging tasks, and combines the storage fragmentation prediction capability of the dynamic threshold calculation unit to effectively balance storage space utilization and system maintenance costs.

[0094] 5. This invention leverages a modular asynchronous processing architecture to ensure system reliability. The query engine and data update agent employ a non-blocking communication mode, critical operations are recorded through transaction logs, and the incremental update controller's delayed window mechanism supports operation rollback in case of index update failure, ensuring data consistency. Simultaneously, a load-aware task scheduler dynamically adjusts resource allocation, enhancing system fault tolerance and overall performance. Attached Figure Description

[0095] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0096] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0097] Please see the appendix Figure 1 This invention provides a method for big data compression, storage, and fast retrieval, which achieves efficient data compression and fast range query through dynamic dimension weight allocation, spatiotemporal collaborative indexing, and predictive segmentation mechanism.

[0098] like Figure 1 As shown, this big data compression storage and fast retrieval method may include the following steps:

[0099] S1. Dynamically generate dimension weights based on the multidimensional information entropy in the data stream, and adjust the dimension priority of the space filling curve accordingly to generate a rotated Hilbert code.

[0100] S2. Divide the data into segments by predicting the number of segments through time series, dynamically set the segmentation error threshold, sort and segment the encoded data, and record the linear model and residual coding.

[0101] S3. Construct a spatial bitmap index based on encoded values ​​and a B+ tree index based on data values. Trigger incremental index updates when the dimension weight changes exceed the threshold.

[0102] S4. Map the query range to a coding interval, filter candidate segments using spatial index, locate the target segment using B+ tree, and restore the query results using the model.

[0103] S5. Weight changes caused by new data will update the encoding and index, while expired data will be eliminated and the index structure will be maintained synchronously.

[0104] The following is a detailed description of each step in the method of the present invention, providing a comprehensive explanation of the specific implementation principles, technical details, and processes of each step.

[0105] In step S1, this invention dynamically analyzes the multi-dimensional distribution characteristics of the data stream to generate a weight vector reflecting the degree of data concentration in each dimension, and adjusts the encoding priority of the space-filling curve accordingly to achieve efficient spatial encoding of the data. The specific implementation process includes:

[0106] First, the multidimensional spatial range in the data stream is divided into a uniform grid according to a preset rule. For each dimension d∈[1,n], where n represents the total number of dimensions, its corresponding data spatial range is... The data is divided into B equal-width intervals, each interval being called a grid. The number of data points in the b-th grid for the d-th dimension within the current time window is counted. Where b∈[1,B]. The probability distribution of grid data in each dimension is calculated based on the statistical results. in This represents the amount of data in the b-th grid of the d-th dimension; N is the total number of all data points in the data stream within the current time window.

[0107] Subsequently, the information entropy of each dimension is calculated based on the probability distribution. For the d-th dimension, its information entropy S d Calculated using the following formula:

[0108]

[0109] Among them, when When, define This formula quantifies the uniformity of data distribution along dimension d: when the data is concentrated in a few grid cells, S d The value is low; when the data is uniformly distributed, S d The value is relatively high.

[0110] Generate a dimension weight vector w = (w1, w2, ..., w2) based on the calculated information entropy value. n ), where the weight w of the d-th dimension d Determined by the following normalization formula:

[0111]

[0112] Among them, S i This represents the information entropy of the i-th dimension;

[0113] This weight vector reflects the ability of different dimensions to distinguish the spatial distribution of the data. Weight value w d The larger the value of d, the more uneven the data distribution in dimension d, and the higher the priority it should be given in spatial coding.

[0114] The dimensional priority of the space-filling curve is adjusted based on the weight vector. Specifically, based on the generated dimensional weight vector w, the dimensional access priority of the Hilbert space-filling curve when processing multidimensional data is adjusted. The dimension with the highest information entropy weight is selected as the main expansion direction of the Hilbert curve. According to the Hilbert space-filling curve after dimensional priority adjustment, a corresponding rotated Hilbert code value is generated for each data point in the data stream. For n-dimensional space, the adjusted rotated Hilbert curve generation rule is: during the curve recursive segmentation process, spatial partitioning is preferentially performed along the dimension with the highest weight value. This adjustment allows data points to cluster in the encoding space according to the distribution characteristics of high-discrimination dimensions, thereby improving the efficiency of subsequent data compression.

[0115] The rotated Hilbert code value is generated based on the adjusted space-filling curve. For each data point x = (x1, x2, ..., x... n This is done by mapping the curve to an n-dimensional grid coordinate system and calculating its corresponding encoded value according to the recursive segmentation order of the curve. The generation process of the encoded value satisfies the following: when the weight vector w changes, the encoded value h(x) of the same spatial point will be dynamically updated by adjusting the recursive segmentation order of the curve, thereby adapting to the changes in the data distribution characteristics.

[0116] Preferably, the value of the grid number B is related to the data size and can be determined based on the data volume N in the current time window. Adaptive adjustment. This strategy ensures computational efficiency while avoiding distortion of information entropy evaluation due to excessively dense or sparse grids. Furthermore, when generating rotated Hilbert codes, for dimensions with the same weight values, the earlier dimension in the original dimension order is prioritized as the secondary priority partitioning direction, ensuring the determinism of the encoding rules.

[0117] In step S2, this invention dynamically determines the data segmentation strategy based on a time series prediction model. By predicting the number of segments in the next time window and adaptively adjusting the error threshold, it achieves adaptive compression and structured storage of rotated Hilbert-coded data. The specific implementation process includes:

[0118] First, a time series model of the historical segment count is constructed. The actual observed data segment counts within a series of historical time windows are obtained to form the original time series, denoted as {k1, k2, ..., kk}. t}, where t represents the sequence number of the current time window. For the original sequence {k s} Perform d-order differencing to eliminate nonstationarity, resulting in a stationary sequence {Δ} d k t Difference operations are implemented through recursive calculations:

[0119] Δ d k t =Δ d-1 k t -Δ d-1 k t-1 (initial value Δ) 1 k t =k t -k t-1 )

[0120] The mathematical expression for establishing an ARIMA(p,d,q) model based on stationary sequences is defined as follows:

[0121]

[0122] Where, Δ d k t Let d represent the d-th order difference of the original sequence at time window t, p be the order of the autoregressive term in the model, q be the order of the moving average term in the model, and φ be the order of the moving average term in the model. i Here are the autoregressive coefficients, i = 1, 2, ..., p, θ j Let j = 1, 2, ..., q, ∈ t Let Δ be the white noise residual term for the t-th window. d k t-i Let be the d-th order difference value of the original sequence in the ti-th time window, ∈ t-j For the white noise residual term in the tj-th time window, This is the white noise residual term that follows a normal distribution.

[0123] The model predicts the number of segments for the next window. The formula is:

[0124]

[0125] This prediction takes into account both the trend of historical segmentation and the fluctuation characteristics of residuals, providing a benchmark for subsequent error threshold calculations.

[0126] Based on the predicted number of segments Dynamically determine the maximum allowable fitting error ∈max Specifically, based on the total amount of data N in the current time window, the error threshold is calculated using the following formula:

[0127]

[0128] This threshold is used to control the granularity of data segmentation: as the number of predicted segments increases, ∈ max Lowering the tolerance level encourages more precise data segmentation; conversely, increasing the tolerance level relaxes the error tolerance to reduce the number of segments.

[0129] The rotated Hilbert encoded data is segmented using an error threshold. After sorting all data points within the current time window in ascending order of their encoded values, a sliding window method is used for segmentation: starting from the first data point, the window range is gradually expanded until the linear fitting residuals of the data points within the window satisfy the following condition:

[0130]

[0131] Where (a,b) are the linear regression parameters for the window data, h p This represents the encoded value of the data point. When the window expands beyond the error threshold, the current window is marked as a data segment, and its linear model parameters (a) are recorded. j ,b j ), and store the residual coding results q of all data points. j ={h p -(a j ·p+b j )}.

[0132] Preferably, a dynamic correction mechanism is introduced during the model parameter update process. When the actual number of segments k... real Compared with the predicted value There is a deviation When, for the moving average coefficient θ j Make adjustments online:

[0133]

[0134] in, Let be the value of the j-th moving average coefficient before this correction operation was performed. This is the new value of the j-th moving average coefficient after this correction operation. This new value will be used in the prediction calculation of the ARIMA(p,d,q) model in subsequent time windows.

[0135] The learning rate η is related to the change in the dimension weight vector, and its calculation satisfies:

[0136] η=β·||w current -w previous ||1+γ

[0137] Among them, w current w is the dimension weight vector of the current window; previous Let w be the dimension weight vector of the previous window; current -w previous ||1 represents the w current With the w previous The Manhattan norm is defined as β and γ; β and γ are preset adjustment coefficients. This mechanism enables the model to adaptively adjust the prediction parameters according to changes in the data distribution characteristics.

[0138] Preferably, the segmentation results and residual coded data are stored in key-value pair format, with each data segment S... j The records contain:

[0139] Linear model parameters (a) j ,b j );

[0140] Residual coding set q j The compressed storage address;

[0141] Endpoints of the encoded value range

[0142] This storage structure provides the data foundation for subsequent index building and fast querying.

[0143] In step S3, this invention achieves rapid retrieval and adaptive maintenance of the data storage structure by collaboratively constructing a spatial bitmap index and a value domain B+ tree index, and by designing a dynamic incremental update mechanism. The specific implementation process includes:

[0144] First, determine the encoding range based on the rotated Hilbert encoding values ​​of all data points within the current time window. Then, calculate the maximum encoded value h of all rotated Hilbert encoding values ​​within the current time window. max =max{h p |p∈W k} and minimum encoded value h min =min{h p |p∈W k}, where W k Let m represent the data set for the k-th time window. The bitmap resolution m is dynamically determined based on the encoding range, and its calculation formula is:

[0145]

[0146] This resolution results in the coding space being divided into 2. m ×2 m Each grid corresponds to a unique integer encoding range, ensuring index accuracy while avoiding excessive storage space expansion.

[0147] Subsequently, the grid encoding interval partitioning rules are defined. Each grid g∈[1,2] 2m The corresponding encoding interval endpoints are calculated using the following formula:

[0148]

[0149] in, This represents the coding interval step size of a single grid. Through this partitioning method, the coding space is uniformly mapped to the bitmap matrix. To achieve a visual representation of data distribution.

[0150] During the bitmap index construction process, for each data segment S j encoding range Perform spatial relation determination. If a grid g satisfies... Then, the marker at the corresponding position in the bitmap matrix is ​​set to 1, i.e., B(g) = 1; otherwise, it remains 0. This operation quickly filters candidate data segments through bitwise operations, significantly improving the efficiency of range queries.

[0151] Furthermore, a value-range B+ tree index is constructed to support precise data location. This is based on the endpoints of the encoded intervals of each data segment. As key-value pairs, the following tuples are stored in the leaf nodes of the B+ tree:

[0152]

[0153] Where {g k} represents the set of bitmap grid numbers associated with this data segment. The ordered hierarchical structure of a B+ tree allows for rapid retrieval of the storage location of data segments covering a specific coding interval.

[0154] When a significant change in the dimension weight vector is detected, the incremental index update mechanism is triggered. The weight change is defined as Δw = ||w||. new -w old ||2, where w new and w old These are the weight vectors for the current and previous windows, respectively, where ||·||² represents the Euclidean norm. If Δw exceeds the adaptive threshold δ, the following update operation is performed:

[0155]

[0156] Where, λ d Let be the sensitivity coefficient for weight changes in the d-th dimension. This is the weight value of the d-th dimension of the previous window. This threshold is dynamically adjusted based on historical weight fluctuation characteristics to avoid frequent and invalid updates.

[0157] Incremental update operations involve recalculating the rotated Hilbert encoded values ​​affected by weight changes and generating new spatial bitmap indexes only for regions where encoded values ​​have changed. For range B+ tree indexes, the endpoints of the encoded intervals of the affected data segments are updated. It maintains query efficiency by adjusting the balanced tree structure. This mechanism significantly reduces the computational overhead of global reconstruction while ensuring index consistency.

[0158] Preferably, a lazy marking strategy is adopted during the bitmap index update process. For data points whose encoded values ​​migrate due to weight changes, their original grid markers are only set to 0 when it is confirmed that no other valid data segments cover them, thus avoiding index thrashing caused by excessive updates. Simultaneously, B+ tree index update operations are recorded through the transaction log, ensuring that data consistency can be restored through log rollback in the event of system anomalies.

[0159] In step S4, this invention achieves efficient retrieval of data within the query range through a two-layer index collaborative filtering and model-driven data restoration mechanism. The specific implementation process includes:

[0160] First, receive the user-defined multidimensional spatial range to be queried, and set the spatial range to be queried QR = [x l ,x u ]×[y l ,y u The mapping is performed as a rotated Hilbert encoding interval. Based on the encoding rules defined in step S1, the lower left corner point (x) of the query range is calculated. l ,y l The corresponding encoded value h l and the top right corner point (x u ,y u The corresponding encoded value h u , forming the coding interval [h l ,h u This mapping process is implemented by calling the same space-filling curve recursive segmentation algorithm as the encoding generation stage, ensuring strict consistency between the query range and the data encoding rules.

[0161] Filter the candidate data segment set using spatial bitmap indexing. Traverse all grids g marked as 1 in the bitmap matrix B, if their corresponding encoding intervals... With [h l ,h u If there is an intersection, the data segment associated with that grid is added to the candidate set. The specific judgment criteria are as follows:

[0162]

[0163] This step uses bitmaps to quickly filter out irrelevant data segments, significantly narrowing the scope of subsequent processing.

[0164] The physical storage address of the candidate data segment is located using a B+ tree index of the value range. The endpoints of the encoded interval are used as the reference. Using the query key, perform a range search operation in the B+ tree to retrieve... The storage location information for all data segments. For each data segment... Extract its stored linear model parameters (a) j ,b j And calculate the prediction range covered by the model:

[0165]

[0166] Hilbert_Decode(h) represents the function that deciphers the spatial coordinates from the encoded value h, and its implementation logic is the inverse operation of the encoding generation process in step S1.

[0167] Specifically, the linear model parameters (a) j ,b j ) and prediction range The specific manifestation of the association logic is as follows:

[0168] Each data segment S j During the segmented compression process, the rotated Hilbert codes of its internal data points are fitted using a linear regression model to obtain a form like h. p =a j ·p+b j +e p The expression, where p is the index of the data point within the segment, e p This is the residual.

[0169] Model parameters (a) j ,b j This implies an approximate distribution pattern of the data segment in the coding space:

[0170] a j It represents the linear growth rate of the encoded value as the data point number changes, reflecting the distribution density of the data in the encoding space;

[0171] b j This represents the initial offset of the linear model, corresponding to the coded value reference at the beginning of the data segment.

[0172] Based on this, data segment S j encoding interval endpoints and It can be calculated in the following ways:

[0173]

[0174] Among them, |S j| represents the total number of data points within the segment. Parameter (a) j ,b j ) and residual set q j ={e p Together, they define the complete encoding information of the data segment.

[0175] During the query processing phase, the prediction range is quickly calculated using parameters. Boundary:

[0176]

[0177] This range covers the prediction space of the linear model without considering residuals, providing a quick basis for determining the coverage of the query range.

[0178] Furthermore, a query range coverage determination is performed on each data segment. If the query range QR is completely contained within... That is, satisfy This directly returns all data points within the data segment without needing to decode the residual encoding. This determination is based on the spatial inclusion relationship between the prediction range and the query range of the linear model, using model parameters to quickly skip the precise verification step.

[0179] When the model coverage cannot fully encompass the query range, the residual encoding result q is executed. j Decoding operation for data segment S. j Each data point (x) p ,y p )∈S j According to its residual e p ∈q j Restore the original encoded value:

[0180] h p =a j ·p+b j +e p

[0181] Then, its spatial coordinates (x) are calculated using Hijbert_Decode(h). p ,y p If the coordinate satisfies x l ≤x p ≤x u And y l ≤y p ≤y u If so, then add it to the final query result set.

[0182] Preferably, a batch processing optimization strategy is adopted during the residual decoding process. For contiguously stored residual data blocks, the encoded values ​​of multiple data points are calculated in parallel using SIMD instructions to restore the original data, accelerating the coordinate transformation process. Simultaneously, a spatial pruning algorithm is used during the coordinate verification stage to prioritize the exclusion of coordinate points that are significantly outside the query range, reducing unnecessary numerical comparison operations.

[0183] The query result set is returned via streaming, dynamically merging parts of the result during data segment processing to reduce memory usage and improve response speed. For query results spanning multiple data segments, they are output in ascending order of their encoded values, maintaining local ordering of the data along the space-fill curve.

[0184] In step S5, the present invention achieves real-time adaptation to changes in data distribution and effective management of storage resources through a dynamic data update and storage optimization mechanism. The specific implementation process includes:

[0185] First, address the changes in dimension weights caused by new data. When new data arrives, re-execute the dimension weight calculation process in step S1 to generate the updated weight vector w. new If a weight change Δw = ||w is detected new -w prev If ||2 exceeds the preset threshold δ, the following operation will be triggered:

[0186] 1. According to w new Adjust the direction of the space fill curve and recalculate the rotation Hilbert code values ​​for all data points;

[0187] 2. Perform incremental index update in step S3 for regions where the encoded value changes to ensure consistency between the spatial bitmap index and the value domain B+ tree index.

[0188] Secondly, implement an expired data eviction policy. For each data segment S j Record its last update timestamp When the system's current time is t current satisfy:

[0189]

[0190] When this happens, the data segment is marked as a failed segment. Where T... expired This is a preset expiration time threshold, the value of which is related to the data update frequency.

[0191] At the index update level, perform logical deletion operations on invalid segments:

[0192] In the value range B+ tree index, S j The corresponding leaf node is marked as logically deleted, while its encoding range endpoints are retained. However, the storage pointer is set to an invalid address. This operation is achieved by modifying the node metadata fields, avoiding index structure corruption caused by immediate deletion.

[0193] In the spatial bitmap index, traverse all instances of... If grids g have intersections, and grid g is no longer associated with any valid data segment, then B(g) is set to 0. This determination is made by checking whether there are other valid segments whose encoded intervals cover each other in the B+ tree. accomplish.

[0194] Further, perform physical merging of storage space. Calculate the storage space utilization rate of the failed segments. When ρ < τ (τ is a preset threshold), the merging operation is initiated:

[0195] 1. Scan the physical storage area to identify a continuous sequence of free storage blocks {B1, B2, ..., B}. k};

[0196] 2. Merge adjacent free blocks into a single block with a capacity of [capacity value missing]. The new storage block;

[0197] 3. Update the storage pointer addresses of the affected nodes in the value domain B+ tree index to ensure that the mapping relationship between the index and the storage location is correct.

[0198] Preferably, a delay window mechanism is introduced between logical deletion and physical merging. Storage blocks marked as invalid are deleting in T... delay The system remains in a recoverable state within the time window. If a data rollback request is received during this period, the data segments and associated indexes can be recovered via the transaction log. This mechanism enhances system fault tolerance by maintaining the transaction ID and timestamp of the delete operation.

[0199] Storage optimization and index update operations are executed asynchronously via a task queue to avoid blocking real-time data processing. The task queue scheduler dynamically adjusts the execution priority of merge operations based on system load. When storage fragmentation exceeds a critical threshold, merge tasks for highly fragmented areas are triggered first. The fragmentation level is measured using the following formula:

[0200]

[0201] When F > φ (φ is the system's preset threshold), it is determined to be a high-fragmentation region.

[0202] In summary, this invention achieves efficient data processing through dynamic dimensional weight allocation and spatiotemporal collaborative indexing technology. Its overall process includes: dynamically generating dimensional weight vectors based on the information entropy of multidimensional data streams; adjusting the direction of the space-filling curve to generate rotated Hilbert codes; dynamically determining the number of data segments and error thresholds using a time series prediction model; sorting the coded data and compressing and storing parameters and residuals using a linear model; constructing a two-layer index structure (spatial bitmap index and value domain B+ tree index) and designing an incremental update mechanism, locally reconstructing the index when dimensional weight changes exceed the threshold; mapping the query range to a coding interval; collaboratively filtering candidate data segments through the index; and accurately returning results by combining a linear model and residual decoding; dynamically correcting coding rules and index status based on new data; asynchronously eliminating expired data and optimizing storage space, forming a closed-loop processing chain of data compression, index construction, query optimization, and dynamic updates.

[0203] The big data compression storage and fast retrieval system described below can be referred to in correspondence with the big data compression storage and fast retrieval system method described above.

[0204] This invention also provides a big data compression storage and fast retrieval system, which consists of a dynamic encoding module, a segmented compression module, a collaborative index module, a query engine, and a data update agent. The modules work together to achieve data compression storage and efficient retrieval.

[0205] The dynamic encoding module receives the data stream in real time, generates a dynamic weight vector by calculating the information entropy of the data distribution in each dimension, and adjusts the dimensional expansion direction of the Hilbert curve based on the weight vector to generate a rotated Hilbert encoded value for each data point. This module transmits the encoding result to the segmentation compression module, triggering the data segmentation process.

[0206] The segmented compression module incorporates an ARIMA prediction unit, a dynamic threshold calculation unit, and a residual encoding unit. The ARIMA prediction unit predicts the number of segments for the next time window based on the historical segment count sequence. The dynamic threshold calculation unit generates an error threshold based on the predicted values ​​to control the granularity of data segmentation. The residual encoding unit performs linear fitting on the sorted data segments, stores the model parameters, and compresses the residual data. The compression results are then transmitted to the collaborative indexing module to drive index construction.

[0207] The collaborative indexing module includes a bitmap generator, a B+ tree manager, and an incremental update controller. The bitmap generator constructs a spatial bitmap index based on the distribution of encoded values, marking the spatial coverage relationship between data segments and the grid. The B+ tree manager constructs an ordered index using the endpoints of the encoded intervals of data segments as keys, and its leaf nodes are linked to storage addresses and bitmap grid states via bidirectional pointers. The incremental update controller monitors changes in the weight vector in real time, triggering partial reconstruction of the bitmap index and updates to the B+ tree nodes when the changes exceed a threshold, ensuring that the index is synchronized with the data distribution.

[0208] The query engine receives external query requests, maps the spatial range to encoded intervals, filters candidate data segments using a bitmap index, and locates the storage position using a B+ tree. For candidate segments, it quickly determines the spatial coverage relationship by combining linear model parameters, decodes residual data and verifies coordinates for segments that cannot be completely covered, and finally returns accurate query results.

[0209] The data update agent asynchronously handles the maintenance of new and expired data. New data triggers weight recalculation and incremental index updates in the dynamic encoding module; expired data segments are marked for logical deletion in the B+ tree, and their associated bitmap grid state is synchronously cleared. Physical storage space merging operations are executed through an asynchronous task queue, merging free blocks and updating index storage pointers to maintain storage efficiency. Through modular design and asynchronous communication mechanisms, the system achieves efficient coordination between data compression, index optimization, and dynamic updates.

[0210] The system in this embodiment can be used to execute the above method embodiments, and its principle and technical effect are similar, so they will not be described again here.

[0211] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for big data compression, storage, and fast retrieval, characterized in that, Includes the following steps: S1, dynamically generate dimension weight vectors based on the multi-dimensional information entropy in the data stream, adjust the dimension priority of the Hilbert space filling curve according to the dimension weight vectors, and generate the rotated Hilbert code value of each data point; S2, use a time series prediction model to predict the number of data segments, dynamically calculate the segmentation error threshold based on the predicted number of data segments, segment the sorted data based on the rotated Hilbert code value, and store the segmented linear model parameters and residual coding results of each data segment to describe each data segment. S3, construct a spatial bitmap index based on the distribution range of the rotated Hilbert encoded values, and construct a value domain B+ tree index by combining the endpoint values ​​of the rotated Hilbert encoded intervals of each data segment. When the change in the dimension weight vector exceeds a preset threshold, the incremental update of the spatial bitmap index and the value domain B+ tree index is triggered. S4, map the query range to a rotated Hilbert encoding interval, use the spatial bitmap index to filter candidate data segments, locate the target data segment through the value domain B+ tree index, and return the data within the query range by combining the piecewise linear model parameters and the residual encoding result; S5. Dynamically correct the dimension weight vector and rotation Hilbert encoding value based on the newly added data, eliminate expired data, and synchronously update the spatial bitmap index and value range B+ tree index.

2. The big data compression storage and fast retrieval method according to claim 1, characterized in that, In step S1, the step of generating the rotated Hilbert code value for each data point includes: S11, for each dimension in the data stream, divide the corresponding spatial range into the same number of uniform grids, and count the number of data points in each grid of each dimension; S12, for the first Calculate the dimension. The dimension in the first Data distribution probability within each grid The formula for calculating the probability distribution of the data is: ; in, Indicates the first The dimension in the first The amount of data within each grid; This represents the total number of all data points in the data stream within the current time window. , The preset number of grid divisions; S13, Based on the calculated data distribution probability, calculate the first... Information entropy in each dimension The formula for calculating the information entropy is: ; S14, based on the calculated information entropy values ​​of each dimension, generate a... Dimensional weight vector Among them, the first The weight values ​​corresponding to each dimension Calculated using the following formula: ; in, Indicates the first Information entropy in one dimension; S15, based on the generated dimension weight vector Adjust the dimension access priority of the Hilbert space filling curve when processing multidimensional data. For the dimension with the highest weight value, extend the Hilbert space filling curve along the dimension direction of the dimension with the highest weight value. S16, based on the Hilbert space-filling curve adjusted for dimensional priority, generate the corresponding rotated Hilbert code value for each data point in the data stream.

3. The big data compression storage and fast retrieval method according to claim 1, characterized in that, In step S2, the time series prediction model is an autoregressive integral moving average model, and the step of using the time series prediction model to predict the number of data segment divisions includes: S21, obtain the number of data segments actually observed within a series of historical time windows, forming the original time series, denoted as... For the original time series conduct The order difference operation yields a stationary time series. ,in, Indicates the first The actual number of segments in each historical time window This is the sequence number of the latest historical time window that has been observed so far. Indicates the original time series at the 1st... Each window Step difference fraction, The pre-selected difference order; S22, based on the obtained stationary time series Establish an ARIMA(p,d,q) model, the expression of which is: ; in, Indicates the original time series at the 1st... a time window Step difference fraction, The order of the autoregressive term in the model. The order of the moving average term in the model. These are the autoregressive coefficients. , The moving average coefficient is... , For the first White noise residuals for each window, For the original time series in the 1st a time window Step difference fraction, In the first White noise residuals for each time window; S23, using the established ARIMA(p,d,q) model and the prediction formula for the number of data segments in the next time window, predict the number of data segments in the next time window. The prediction formula for the number of data segments in the next time window is as follows: ; in, The number of segments to divide the forecast data for the next time window.

4. The big data compression storage and fast retrieval method according to claim 3, characterized in that, The step of predicting the number of data segment partitions using a time series forecasting model further includes dynamically adjusting the moving average coefficient in the ARIMA(p,d,q) model based on the actual number of data segment partitions that occurred within each time window processing period after the end of that period. Specifically: S24, For the currently ended time window, obtain the actual number of data segment divisions that actually occurred within the current time window. ; S25, obtain the number of predicted data segments generated by the ARIMA(p,d,q) model in previous time windows for the same current time window. ; S26, Calculate the actual number of data segment divisions. Number of segments to be divided with the predicted data Deviation between The deviation The calculation formula is: ; S27, Obtain the dimension weight vector calculated for the current time window. The dimension weight vector calculated from the previous time window of the current time window. ; S28, Calculate the learning rate used for subsequent adjustments to the moving average coefficients. The learning rate The calculation formula is: ; in, The dimension weight vector of the current time window; The dimension weight vector of the previous time window; Indicates the With the The Manhattan norm between; and The preset adjustment coefficient; S29, for each moving average coefficient in the ARIMA(p,d,q) model Using the calculated deviation and the calculated learning rate The coefficient is dynamically adjusted, resulting in a corrected moving average coefficient. Calculated using the following formula: ; in, For the first The value of each moving average coefficient before the current correction operation was performed. For the first The new values ​​of the moving average coefficients after this correction operation will be used in the ARIMA(p,d,q) model's prediction calculations for subsequent time windows.

5. The big data compression storage and fast retrieval method according to claim 1, characterized in that, In step S3, the step of constructing a spatial bitmap index based on the distribution range of the rotated Hilbert encoded values ​​includes: S31, determine the encoding range based on the rotated Hilbert encoding values ​​of all data points within the current time window, and calculate the maximum encoding value of all rotated Hilbert encoding values ​​within the current time window. and minimum encoded value ,in, Indicates the first A data set for a time window The first in the current time window Rotated Hilbert encoded values ​​of each data point; S32, based on calculation and The resolution parameter of the bitmap is calculated using the following formula. : ； S33 divides the overall data space consisting of rotated Hilbert encoded values ​​into The grid defines a unique integer-encoded sub-range for each grid. ,in ; ; ; For grid number; S34, initialize a size of bitmap matrix And set the initial value of all elements in the matrix to 0; S35, for each data segment formed in step S2 Obtain its corresponding rotated Hilbert encoding range. ; S36, traverse each grid in the bitmap matrix, if there is at least one data segment Rotational Hilbert encoding range With the current grid Integer encoded sub-interval If there is an intersection, then the corresponding grid in the bitmap matrix will be... The element is marked as 1 if it is not 1, otherwise the element is marked as 0.

6. The big data compression storage and fast retrieval method according to claim 5, characterized in that, In step S3, the triggering condition for the incremental update of the spatial bitmap index and the value domain B+ tree index is: When the change in the dimension weight vector Exceeding the preset threshold At that time, an incremental update operation is performed, where This is the weight vector for the current time window. This is the weight vector of the previous time window. Represents the Euclidean norm; The preset threshold The preset threshold is adaptively adjusted based on the historical change magnitude of the weight vector. The calculation formula is: ; in, For the first Sensitivity coefficients for weight changes in each dimension. For the previous time window Weight values ​​for each dimension, Dimension weight vector The total dimension, representing the total dimension of the data flow; The incremental update operation includes: recalculating the rotated Hilbert encoded values ​​affected by the weight changes, generating new spatial bitmap indices only for regions where the rotated Hilbert encoded values ​​have changed, and updating the corresponding data segments in the value domain B+ tree index. The endpoint values ​​of the rotated Hilbert-encoded interval.

7. The big data compression storage and fast retrieval method according to claim 1, characterized in that, In step S4, the steps of mapping the query range to a rotated Hilbert encoding interval, filtering candidate data segments using the spatial bitmap index, locating the target data segment using the value range B+ tree index, and returning the data within the query range by combining the piecewise linear model parameters and the residual encoding result include: S41, Receive the user-defined multidimensional spatial range to be queried The multidimensional spatial range to be queried is mapped to a rotated Hilbert encoding interval. ,in for Lower left corner point The corresponding encoded value, The top right corner The corresponding encoded values ​​are mapped in the same way as the generated rotated Hilbert encoded values. S42, using the constructed spatial bitmap index, and combining it with the rotated Hilbert encoding range Filtering candidate data segment set The candidate data segment set Defined as ,in, For a data segment, and Data segments The minimum and maximum coded values ​​of the corresponding rotated Hilbert codes; S43, retrieve via the value range B+ tree index All data segments The storage address for each data segment Perform the following operations: 1) Extract data stored in the data segment Linear model parameters Calculate the data stored in the data segment The actual spatial range covered by the linear model , ; in, Indicates from the encoded value A function that solves for spatial coordinates inversely; 2) Based on the data segment Actual spatial range With the range of the multidimensional space to be queried Based on the relationships between them, conditional judgments and data extraction are performed: Condition 1: If the range of the multidimensional space to be queried is determined... Then process the data segment directly. To return all the original data points contained within it; Condition 2: If condition 1 is not met, then the decoded data is stored in the data segment. residual coding results For each, by using the piecewise linear model parameters and the residual encoding result from the decoded data The original data points are reconstructed from the corresponding residual values ​​obtained in the process. If the original data points to be reconstructed Satisfy the range of the multidimensional space to be queried Boundary conditions and Then the data point Add to the final query result set.

8. The big data compression storage and fast retrieval method according to claim 1, characterized in that, In step S5, the step of eliminating expired data and synchronously updating the spatial bitmap index and the value range B+ tree index includes: S51 represents each data segment in the system. Record its latest update timestamp ; S52, periodically or according to a predetermined strategy, checks all data segments. If for a certain data segment It meets the conditions When, mark the data segment. This is a failed data segment, in which The preset expiration time threshold, The current system time; S53, for each failed data segment, in the value domain B+ tree index, the leaf node corresponding to the failed data segment is marked as logically deleted, and its rotated Hilbert encoded interval endpoint value is retained. However, the memory pointer is updated to an invalid address; S54, in the spatial bitmap index, identify all bitmap grids that intersect with the rotated Hilbert encoding interval of the failed data segment. For the bitmap grid, check whether the bitmap grid still intersects with the rotated Hilbert encoding interval of any other data segment that has not been marked as failed. If the bitmap grid no longer intersects with the rotated Hilbert encoding interval of any valid data segment after removing the association with the failed data segment, then update the bitmap tag B(g) corresponding to the bitmap grid to 0; otherwise, if the bitmap grid is still associated with other valid data segments, then its tag B(g) remains 1. S55, when the storage space utilization rate occupied by the failed data segment is lower than the preset threshold, perform physical deletion and storage merging operations: merge consecutive free storage blocks into a larger available space, and update the storage pointer address of the affected node in the value domain B+ tree index.

9. A big data compression storage and fast retrieval system, used to implement the method as described in any one of claims 1-8, characterized in that, include: The dynamic encoding module is used to receive real-time data streams, calculate the information entropy of each dimension and generate a dimension weight vector, adjust the dimension priority of the Hilbert space filling curve according to the dimension weight vector, generate rotated Hilbert encoded values, and transmit the encoding results to the segmented compression module. The segmented compression module, connected to the dynamic coding module, includes an ARIMA prediction unit, a dynamic threshold calculation unit, and a residual coding unit, wherein: The ARIMA prediction unit predicts the number of segments in the next time window based on the historical segment number sequence; The dynamic threshold calculation unit dynamically adjusts the segmentation error threshold based on the predicted number of segments to control the granularity of data segment division. The residual coding unit generates piecewise linear model parameters and residual coding results, and sends the compression results to the collaborative indexing module; The collaborative indexing module communicates bidirectionally with the segmentation compression module and the dynamic encoding module. It includes a bitmap generator, a B+ tree manager, and an incremental update controller, among which: The bitmap generator constructs a spatial bitmap index based on the distribution range of the rotated Hilbert encoded values; The B+ tree manager constructs a range B+ tree index based on the endpoint values ​​of the rotated Hilbert-coded intervals of the data segment; The incremental update controller monitors changes in the dimension weight vector and triggers a local reconstruction of the index when the change exceeds a preset threshold. The query engine, connected to the collaborative index module, receives external query requests and maps the multi-dimensional spatial range to be queried into a rotated Hilbert coding interval. It uses the spatial bitmap index and the value domain B+ tree index to filter candidate data segments and returns the query data by combining the piecewise linear model parameters and the residual coding results. The data update agent interacts with the dynamic encoding module, collaborative indexing module, and segmented compression module to handle encoding update operations for newly added data, evict expired data segments, and maintain the consistency between the index and storage space.

10. The big data compression storage and fast retrieval system according to claim 9, characterized in that, The incremental update controller has a built-in weight change monitoring channel, which receives the weight vector change output by the dynamic encoding module in real time. When the weight vector change exceeds a preset threshold, it sends a local reconstruction instruction to the bitmap generator and the B+ tree manager. The leaf nodes of the B+ tree manager contain a bidirectional pointer structure: The forward pointer is associated with the endpoint values ​​of the rotated Hilbert-encoded interval and their storage addresses; Back pointers are associated with memory addresses and corresponding space bitmap grid markers; The bidirectional pointers enable rapid location of storage blocks based on the rotated Hilbert code value and reverse update of the bitmap index state.

Images