Efficient data compression storage algorithm based on adaptive entropy optimization model

Abstract

The invention provides an efficient data compression storage algorithm based on a self-adaptive entropy optimization model. The efficient data compression storage algorithm comprises the steps of inputting a data set and parameters; performing initialization; carrying out data block processing; circularly processing each data block, calculating an entropy value, adjusting a weight, calculating a weighting frequency, updating a dynamic dictionary, coding and compressing, and updating the weight; judging whether unprocessed data blocks exist or not, and returning to the previous step when the unprocessed data blocks exist; when the unprocessed data blocks do not exist, continuing the next step; combining all the coding sequences; and outputting a result. Through the adaptive entropy optimization model and the progressive block optimization strategy, the compression scheme can be dynamically adjusted in real time according to the data entropy value, and the decompression speed and the storage efficiency are improved while the compression rate is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of data compression and storage optimization, and in particular relates to an efficient adaptive compression storage algorithm applied to multi-modal and large-scale data. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, the demand for data storage and transmission in various industries has grown exponentially. Traditional data compression algorithms, such as Huffman coding and LZW algorithm, have problems such as low compression rate, poor adaptability, and low efficiency when processing large-scale, multi-dimensional, and high-entropy data sets. Especially when facing high-entropy or low-redundancy data, the compression effect of traditional algorithms is not ideal and cannot meet the needs of practical applications. Therefore, it is of great practical significance to study a data compression algorithm that can adaptively adjust and take into account both compression rate and computational efficiency. Summary of the invention

[0003] The present invention aims to provide an efficient data compression and storage algorithm based on an adaptive entropy optimization model, aiming to address the deficiencies in the prior art. By constructing an adaptive entropy optimization model, a fine entropy analysis and weight adjustment of the data set are performed, and combined with improved dynamic dictionary coding, efficient data compression and storage are achieved.

[0004] To achieve the above object, the present invention proposes the following technical solutions:

[0005] An efficient data compression storage algorithm based on an adaptive entropy optimization model comprises the following steps:

[0006] Step 1: Input the data set and parameters: data set D = {d1, d2, …, d N}; Parameters include: block size B, learning rate η, regularization parameter λ;

[0007] Step 2, Initialization:

[0008] (1) Initialize the weight vector W, initial value

[0009] (2) Initialize the dynamic dictionary Dynamic dictionary is empty or contains common symbols;

[0010] Step 3: Data block processing: Divide D into Data block D k ;

[0011] Step 4: loop through each data block D k , as follows:

[0012] a. Calculate the entropy value: (1) For each data element di ∈D k , calculate the entropy H(d i ); (2) For data block D k , calculate the average entropy H(D k );

[0013] b. Adjust weight: According to the entropy value and the average entropy, adjust the entropy weight w i , set w i is α or β;

[0014] c. Calculate the weighted frequency: Calculate the weighted frequency f i =w i × frequency (di), where f i is the weighted frequency; frequency (d i ) is d i In D k The number of occurrences in ;

[0015] d. Update dynamic dictionary: according to weighted frequency f i Update dynamic dictionary

[0016] e. Encoding compression: using updated dictionary Right D k Encode and get the encoding sequence C k ;

[0017] f. Update weights:

[0018] Iteratively update w according to the formula i ,

[0019] Step 5, determine whether there are unprocessed data blocks: if there are unprocessed data blocks, return to step 4; if there are no unprocessed data blocks, proceed to the next step;

[0020] Step 6: Merge all coding sequences C: Merge coding sequences

[0021] Step 7, output results: compressed data C and updated dictionary Save the updated dictionary

[0022] The efficient data compression and storage algorithm based on the adaptive entropy optimization model proposed in this invention realizes the efficient compression and storage of large-scale, multi-dimensional, high-entropy data by analyzing the entropy value and adjusting the weight of the data set, combined with improved dynamic dictionary coding. The specific beneficial effects are as follows:

[0023] (1) Strong adaptability: The algorithm can dynamically adjust according to data characteristics and adapt to different types of data sets.

[0024] (2) High compression rate: effectively reduce data redundancy and improve compression rate.

[0025] (3) High computational efficiency: Block processing and optimized encoding algorithms reduce computational complexity and can significantly improve computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the efficient data compression storage algorithm based on the adaptive entropy optimization model described in the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The parameters of the present invention are as follows: D is the input data set, which contains N data elements d i .d i is the i-th data element. N is the total number of data elements in the data set D. M is the total number of symbols that may appear in the data set. ik For i The probability of symbol k appearing in i ) is the data element d i The entropy value of . W is the weight vector, W = w1, w2, …, w N}. i is the data element d i λ is the regularization parameter, which is used to balance the influence of weighted entropy and regularization term. R(W) is the regularization term, which is usually η is the learning rate, which controls the step size of weight update. t is the index of the number of iterations. B is the block size, and each data block contains B data elements. K is the total number of data blocks, D k is the kth data block. |D k | is the data block D k The number of data elements in H(D k ) is the data block D k α is a preset higher weight parameter, which is assigned to the d values ​​with entropy values ​​lower than or equal to the average entropy. i β is a preset lower weight parameter, which is assigned to the d with entropy values ​​higher than the average entropy. i . is a dynamic dictionary used to store symbol sequences and their codes. s is a symbol sequence. i The symbol or symbol combination that appears in s. c is the code, which represents the code value assigned to the symbol sequence s. C is the set of compressed code sequences. k is the encoding sequence of the kth data block.

[0029] like Figure 1 As shown, the efficient data compression storage algorithm based on the adaptive entropy optimization model of the present invention comprises the following steps:

[0030] 1. Input data set and parameters:

[0031] Dataset D = {d1, d2, …, d N};

[0032] Parameters include: block size B, learning rate η, and regularization parameter λ.

[0033] 2. Initialization:

[0034] (1) Initialize the weight vector W, initial value

[0035] (2) Initialize the dynamic dictionary Dynamic dictionary Either empty or contains common symbols.

[0036] 3. Data block processing:

[0037] Divide the dataset D into blocks of size B. Data block D k .

[0038] 4. Loop through each data block D k :

[0039] a. Calculate the entropy value:

[0040] (1) For each data element d i ∈D k , calculate the entropy H(d i );

[0041]

[0042] Where: p ik For i The probability of symbol k appearing in ; M is the total number of possible symbols.

[0043] (2) For data block D k , calculate the average entropy H(D k ).

[0044]

[0045] b. Adjust weights:

[0046] According to the entropy value and the average entropy, adjust the weight w i , set w i is α or β.

[0047]

[0048] Where α>β≥0.

[0049] c. Calculate weighted frequency:

[0050] Calculate the weighted frequency f i =w i × frequency (d i ), where f i is the weighted frequency; frequency (d i ) is d i In D k The number of occurrences in .

[0051] d. Update the dynamic dictionary:

[0052] According to the weighted frequency f i Update dynamic dictionary Symbol sequences with higher weights are preferentially added to the dictionary or assigned shorter codes.

[0053] e. Encoding compression:

[0054] Using an updated dictionary Right D k Encode and get the encoding sequence C k . Weight w i Affects the length of the code, the higher the weight, the shorter the code.

[0055] f. Update weights:

[0056] Iteratively update w according to the formula i ,

[0057] 5. Determine whether there are unprocessed data blocks

[0058] When there are unprocessed data blocks, return to step 4; when there are no unprocessed data blocks, proceed to the next step.

[0059] 6. Merge all coding sequences C

[0060] Merge coding sequences

[0061] 7. Output results

[0062] Compressed data C and updated dictionary Save the updated dictionary

[0063] The core of the present invention is to analyze and optimize the entropy value in the data set through an adaptive entropy optimization model, dynamically adjust the weight of the data elements, and thus improve the compression efficiency and storage performance. By processing the data in blocks, using differentiated compression strategies for data blocks with different entropy characteristics, combined with improved dynamic dictionary coding, efficient compression of large-scale, multi-dimensional, high-entropy data is achieved.

[0064] The optimization objective function adopted by the adaptive entropy optimization model of the present invention is as follows:

[0065]

[0066] The derivation optimization algorithm adopted by the adaptive entropy optimization model of the present invention is as follows:

[0067] For the objective function F(W,D) i Find partial derivatives:

[0068] Solve the weight update formula:

[0069] To ensure that the weights are non-negative, an activation function is used:

[0070] The present invention responds to data pattern changes through a dynamic entropy model and a progressive block optimization strategy, can efficiently compress multimodal data, and is suitable for data storage and transmission optimization in cloud computing, edge computing, and distributed systems.

Claims

1. An efficient data compression storage algorithm based on an adaptive entropy optimization model, characterized in that: The following steps are involved: Step 1: Input the data set and parameters: data set D = {d1, d2, ..., d N }, N is the number of data elements; parameters include: block size B, learning rate η, regularization parameter λ; Step 2, Initialization: (1) Initialize the weight vector W, initial value (2) Initialize the dynamic dictionary Dynamic dictionary is empty or contains common symbols; Step 3: Data block processing: Divide the data set D into blocks of size B. Data block D k ; Step 4: loop through each data block D k , as follows: a. Calculate the entropy value: (1) For each data element d i ∈D k , calculate the entropy H(d i ); (2) For data block D k , calculate the average entropy H(D k ); b. Adjust weight: According to the entropy value and the average entropy, adjust the entropy weight w i , set w i is α or β; Among them, α>β≥0; c. Calculate the weighted frequency: Calculate the weighted frequency f i =w i × frequency (d i ), where fi is the weighted frequency; frequency (d i ) is d i In D k The number of occurrences in ; d. Update dynamic dictionary: according to weighted frequency f i Update dynamic dictionary e. Encoding compression: using updated dictionary Right D k Encode and get the encoding sequence C k ; f. Update weights: Iteratively update w according to the formula i , Step 5, determine whether there are unprocessed data blocks: if there are unprocessed data blocks, return to step 4; if there are no unprocessed data blocks, proceed to the next step; Step 6: Merge all coding sequences C: Merge coding sequences Step 7, output results: compressed data C and updated dictionary Save the updated dictionary 2. The efficient data compression storage algorithm based on the adaptive entropy optimization model according to claim 1 is characterized in that: In step 4, Entropy H(d i ) is calculated as: Among them, p ik For i The probability of symbol k appearing; M is the total number of possible symbols; The calculation formula of average entropy H(Dk) is:

3. The efficient data compression storage algorithm based on the adaptive entropy optimization model according to claim 1 is characterized in that: The optimization objective function adopted by the adaptive entropy optimization model is: Among them, R(W) is the regularization term, 4. The efficient data compression storage algorithm based on the adaptive entropy optimization model according to claim 1 is characterized in that: The derivation optimization algorithm adopted by the adaptive entropy optimization model is as follows: For the objective function F(W, D) on w i Find partial derivatives: Solve the weight update formula: To ensure that the weights are non-negative, an activation function is used:

Images