Efficient data compression storage algorithm based on adaptive entropy optimization model

Abstract

The application provides a high-efficiency data compression storage algorithm based on an adaptive entropy optimization model, comprising the following steps: inputting a data set and parameters; initialization; data block processing; loop processing each data block, calculating an entropy value, adjusting a weight, calculating a weighted frequency, updating a dynamic dictionary, encoding compression, updating the weight; judging whether there is an unprocessed data block; when there is an unprocessed data block, returning to the previous step; when there is no unprocessed data block, continuing the next step; merging all encoding sequences; and outputting a result. Through the adaptive entropy optimization model and the progressive block optimization strategy, the application can dynamically adjust a compression scheme according to a data entropy value in real time, thereby improving the decompression speed and storage efficiency while ensuring the compression rate.

Description

Technical Field

[0001] The present invention belongs to the field of data compression and storage optimization, and in particular relates to an efficient adaptive compression storage algorithm applied to multimodal and large-scale data. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, the demand for data storage and transmission across various industries is growing exponentially. Traditional data compression algorithms, such as Huffman coding and the LZW algorithm, suffer from low compression rates, poor adaptability, and inefficiency when processing large-scale, multi-dimensional, and high-entropy datasets. In particular, when dealing with high-entropy or low-redundancy data, traditional algorithms offer suboptimal compression results and fail to meet the demands of practical applications. Therefore, researching a data compression algorithm that can adaptively adjust and balance compression rates and computational efficiency is of great practical significance. Summary of the Invention

[0003] The present invention aims to provide an efficient data compression and storage algorithm based on an adaptive entropy optimization model, aiming to address the deficiencies in the prior art. By constructing an adaptive entropy optimization model, a fine entropy analysis and weight adjustment of the data set are performed, and combined with an improved dynamic dictionary coding, efficient data compression and storage are achieved.

[0004] To achieve the above objectives, the present invention proposes the following technical solutions:

[0005] An efficient data compression storage algorithm based on an adaptive entropy optimization model includes the following steps:

[0006] Step 1: Input the data set and parameters: data set D = {d1, d2, ..., d N}; Parameters include: block size B, learning rate η, regularization parameter λ;

[0007] Step 2, Initialization:

[0008] (1) Initialize the weight vector W, initial value

[0009] (2) Initialize the dynamic dictionary Dynamic Dictionary Empty or contains common symbols;

[0010] Step 3: Data block processing: Divide D into Data blocks D k ;

[0011] Step 4: Loop through each data block D k , as follows:

[0012] a. Calculate entropy: (1) For each data element di ∈D k , calculate the entropy H(d i ); (2) For data block D k , calculate the average entropy H(D k );

[0013] b. Adjust the weight: According to the entropy value and the average entropy, adjust the entropy weight w i , set w i is α or β;

[0014] c. Calculate weighted frequency: Calculate weighted frequency f i =w i × frequency (di), where f i is the weighted frequency; frequency (d i ) is d i In D k The number of occurrences in ;

[0015] d. Update dynamic dictionary: according to weighted frequency f i Update dynamic dictionary

[0016] e. Encoding compression: using updated dictionary To D k Encode and get the encoding sequence C k ;

[0017] f. Update weights:

[0018] Iteratively update w according to the formula i ,

[0019] Step 5: Determine whether there are any unprocessed data blocks: if there are any unprocessed data blocks, return to step 4; if there are no unprocessed data blocks, proceed to the next step;

[0020] Step 6, merge all coding sequences C: Merge coding sequences

[0021] Step 7, output results: compressed data C and updated dictionary Save the updated dictionary

[0022] The efficient data compression and storage algorithm proposed in this paper, based on an adaptive entropy optimization model, achieves efficient compression and storage of large-scale, multi-dimensional, high-entropy data by analyzing the entropy values ​​of the data set and adjusting the weights, combined with improved dynamic dictionary coding. Specifically, it has the following beneficial effects:

[0023] (1) Strong adaptability: The algorithm can dynamically adjust according to the characteristics of the data and adapt to different types of data sets.

[0024] (2) High compression rate: effectively reduce data redundancy and improve compression rate.

[0025] (3) High computational efficiency: Block processing and optimized encoding algorithms reduce computational complexity and can significantly improve computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the efficient data compression and storage algorithm based on the adaptive entropy optimization model described in the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The parameters of the present invention are: D is the input data set, which contains N data elements d i .d i is the i-th data element. N is the total number of data elements in the data set D. M is the total number of symbols that may appear in the data set. ik For i The probability of symbol k appearing in . i ) is the data element d i The entropy value. W is the weight vector, W=w1,w2,…,w N}. w i is the data element d i The weight of λ is the regularization parameter, which is used to balance the influence of weighted entropy and regularization term. R(W) is the regularization term, which is usually η is the learning rate, which controls the step size of weight update. t is the index of the number of iterations. B is the block size, and each data block contains B data elements. K is the total number of data blocks, D k is the kth data block. |D k | is the data block D k The number of data elements in H(D k ) is the data block D k α is a preset higher weight parameter, which is assigned to d with entropy values ​​lower than or equal to the average entropy. i β is a preset lower weight parameter, which is assigned to the d with entropy values ​​higher than the average entropy. i . is a dynamic dictionary used to store symbol sequences and their codes. i The symbol or symbol combination that appears in s. c is the code, which represents the code value assigned to the symbol sequence s. C is the compressed code sequence set. k is the encoding sequence of the kth data block.

[0029] As Figure 1 shown, the high-efficiency data compression storage algorithm based on adaptive entropy optimization model of the present application includes the following steps:

[0030] 1. Input data set and parameters:

[0031] Data set D = {d1, d2, …, d N};

[0032] Parameters include: block size B, learning rate η, regularization parameter λ.

[0033] 2. Initialization:

[0034] (1) Initialize weight vector W, initial value

[0035] (2) Initialize dynamic dictionary Dynamic dictionary is empty or contains common symbols.

[0036] 3. Data block processing:

[0037] Divide the data set D into data blocks D k according to the block size B.

[0038] 4. Loop through each data block D k :

[0039] a. Calculate the entropy value:

[0040] (1) For each data element d i ∈ D k , calculate the entropy H(d i );

[0041]

[0042] where: p ik is the probability of symbol k appearing in d i ; M is the total number of possible symbols.

[0043] (2) For data block D k , calculate the average entropy H(D k ).

[0044]

[0045] b. Adjust the weight:

[0046] According to the comparison of entropy value and average entropy, adjust the weight w i , set w i to α or β.

[0047]

[0048] Where α>β≥0.

[0049] c. Calculate weighted frequency:

[0050] Calculate the weighted frequency f i =w i × frequency (d i ), where f i is the weighted frequency; frequency (d i ) is d i In D k The number of occurrences in .

[0051] d. Update the dynamic dictionary:

[0052] According to the weighted frequency f i Update dynamic dictionary Symbol sequences with higher weights are preferentially added to the dictionary or assigned shorter codes.

[0053] e. Encoding compression:

[0054] Using an updated dictionary To D k Encode and get the encoding sequence C k . Weight w i Affects the encoding length, the higher the weight, the shorter the encoding.

[0055] f. Update weights:

[0056] Iteratively update w according to the formula i ,

[0057] 5. Determine whether there are unprocessed data blocks

[0058] When there are unprocessed data blocks, return to step 4; when there are no unprocessed data blocks, proceed to the next step.

[0059] 6. Merge all coding sequences C

[0060] Merge coding sequences

[0061] 7. Output results

[0062] Compressed data C and updated dictionary Save the updated dictionary

[0063] The core of this invention lies in analyzing and optimizing the entropy values ​​in a dataset through an adaptive entropy optimization model, dynamically adjusting the weights of data elements to improve compression efficiency and storage performance. By processing data in blocks and applying differentiated compression strategies for blocks with different entropy characteristics, combined with improved dynamic dictionary coding, this method achieves efficient compression of large-scale, multi-dimensional, high-entropy data.

[0064] The optimization objective function used by the adaptive entropy optimization model of the present invention is as follows:

[0065]

[0066] The derivation optimization algorithm used by the adaptive entropy optimization model of the present invention is as follows:

[0067] For the objective function F(W,D) i Find the partial derivatives:

[0068] Solve the weight update formula:

[0069] To ensure that the weights are non-negative, an activation function is used:

[0070] The present invention responds to data pattern changes through a dynamic entropy model and a progressive block optimization strategy, can efficiently compress multimodal data, and is suitable for data storage and transmission optimization in cloud computing, edge computing, and distributed systems.

Claims

1. An efficient data compression and storage algorithm based on an adaptive entropy optimization model, characterized in that: The following steps are involved: Step 1: Input the data set and parameters: data set D = {d1, d2, ..., d N }, N is the number of data elements; parameters include: block size B, learning rate η, regularization parameter λ; Step 2, Initialization: (1) Initialize the weight vector W, initial value (2) Initialize the dynamic dictionary Dynamic Dictionary Empty or contains common symbols; Step 3: Data block processing: Divide the data set D into blocks of size B. Data blocks D k ; Step 4: Loop through each data block D k , as follows: a. Calculate entropy: (1) For each data element d i ∈D k , calculate the entropy H(d i ); (2) For data block D k , calculate the average entropy H(D k ); b. Adjust the weight: According to the entropy value and the average entropy, adjust the entropy weight w i , set w i is α or β; Among them, α>β≥0; c. Calculate weighted frequency: Calculate weighted frequency f i =w i × frequency (d i ), where fi is the weighted frequency; frequency (d i ) is d i In D k The number of occurrences in ; d. Update dynamic dictionary: according to weighted frequency f i Update dynamic dictionary e. Encoding compression: using updated dictionary To D k Encode and get the encoding sequence C k ; f. Update weights: Iteratively update w according to the formula i , Step 5: Determine whether there are any unprocessed data blocks: if there are any unprocessed data blocks, return to step 4; if there are no unprocessed data blocks, proceed to the next step; Step 6, merge all coding sequences C: Merge coding sequences Step 7, output results: compressed data C and updated dictionary Save the updated dictionary 2. The efficient data compression and storage algorithm based on the adaptive entropy optimization model according to claim 1 is characterized in that: In the step 4, Entropy H(d i ) is calculated as: Among them, p ik For i The probability of the symbol k appearing; M is the total number of possible symbols; The calculation formula of average entropy H(Dk) is:

3. The efficient data compression and storage algorithm based on the adaptive entropy optimization model according to claim 1 is characterized in that: The optimization objective function adopted by the adaptive entropy optimization model is: Among them, R(W) is the regularization term, 4. The efficient data compression and storage algorithm based on the adaptive entropy optimization model according to claim 1 is characterized in that: The derivation optimization algorithm adopted by the adaptive entropy optimization model is as follows: For the objective function F(W, D) w i Find the partial derivatives: Solve the weight update formula: To ensure that the weights are non-negative, an activation function is used:

Images