Fractal data compression and storage method and system and related equipment

Abstract

The invention relates to the technical field of data processing, and discloses a fractal data compression and storage method and system and related equipment thereof. The method comprises the following steps: firstly, performing multi-level analysis on input data, identifying self-similarity structures on different scales, and generating self-similarity descriptors; then dynamically dividing the data into optimal compression units based on self-similarity descriptors, and designing an optimal fractal transformation function set for each partition; performing multi-level coding on the residual error after fractal transformation, and performing adaptive entropy coding by using context information to obtain a compressed data stream; and finally, progressive decoding and quality-controllable reconstruction of the compressed data are realized. A performance analysis and online optimization mechanism is also introduced, and the system performance can be continuously improved. According to the method, the compression ratio and the reconstruction quality can be remarkably improved, the encoding and decoding complexity and the memory overhead are reduced, and the method can be widely applied to data storage and transmission of Internet of Things equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a fractal data compression storage method, system and related equipment. Technical Background

[0002] With the rapid development of IoT technology, more and more devices are connected to the network, generating massive amounts of data. However, IoT devices usually have limited storage and computing resources, making it difficult to effectively store and transmit raw data. Therefore, it is urgent to develop efficient data compression technology that can minimize the amount of data while ensuring the quality of reconstruction, in order to adapt to the characteristics and needs of IoT applications.

[0003] Traditional data compression methods, such as predictive coding and transform coding, mainly use the spatial correlation and statistical redundancy of data for compression. For example, JPEG uses discrete cosine transform to convert images into frequency domain coefficients, and then quantizes and entropy codes them; PNG uses the LZ77 algorithm to eliminate repeated sequences in the data. However, these methods usually only consider the data characteristics of a single scale and it is difficult to fully explore multi-scale structural information. In addition, most of them use fixed coding strategies and cannot adaptively adjust the compression parameters according to the data content, resulting in limited compression performance.

[0004] In recent years, compression methods based on fractal theory have received widespread attention. Fractal compression uses the self-similarity of data at different scales to approximate the original data through an iterative function system. Compared with traditional methods, fractal compression has higher compression ratio and reconstruction quality, and is particularly suitable for images with rich textures and complex multidimensional data. However, existing fractal compression algorithms still have the following shortcomings:

[0005] (1) Insufficient extraction of self-similar features. Most algorithms only consider the similarity of local regions, ignoring the global structural information, making it difficult to accurately characterize the multi-scale characteristics of the data.

[0006] (2) Suboptimal data partitioning strategy: The traditional fixed block size partitioning method does not fully consider the data content, resulting in redundant compression units and low coding efficiency.

[0007] (3) The transformation function design is not flexible enough. The simple geometric transformation function form limits the ability to express complex structures and affects the reconstruction quality.

[0008] (4) The coding efficiency of residual information is low. There is a lack of effective coding methods for the transformed residual data, resulting in a loss of compression ratio.

[0009] (5) Lack of adaptive optimization mechanism. Fixed encoding and decoding parameters are difficult to adapt to dynamically changing data characteristics and application requirements, and cannot achieve continuous performance improvement.

[0010] In summary, the existing technology is difficult to meet the actual needs of IoT data compression, and a new efficient, adaptive and robust compression storage method is urgently needed. The present invention is proposed in this context, aiming to solve the above technical problems and provide a fractal data compression storage method and system for IoT. Summary of the invention

[0011] The present invention proposes a fractal data compression storage method, system and related equipment to solve the problems existing in the prior art, such as insufficient self-similar feature extraction, suboptimal data partitioning strategy, inflexible transformation function design, low residual information encoding efficiency and lack of adaptive optimization mechanism.

[0012] In a first aspect, an embodiment of the present invention provides a method for compressing and storing fractal data, comprising:

[0013] Step 101: Perform multi-level analysis on the input data, identify self-similar structures at different scales, and generate self-similarity descriptors;

[0014] Step 102: dynamically divide the data into optimal compression units based on the self-similarity descriptor to obtain an optimized partitioning scheme;

[0015] Step 103: According to the optimized partitioning scheme, an optimal fractal transformation function set is designed for each partition to obtain a fractal transformation set; wherein, for any pixel x in the partition block, the transformed pixel is:

[0016] f m (x) = a m ·[s m ·R(θ m )·x+o m ]+b m

[0017] Among them, f m (x) is the transformed pixel, θ m is the rotation angle, R(θ m ) is the rotation matrix, a m 、s m , o m and b m They are brightness scale factor, geometric scale factor, offset vector and brightness offset respectively;

[0018] Step 104: performing multi-level coding on the residual after fractal transformation according to the fractal transformation set to obtain a hierarchical residual code stream;

[0019] Step 105: performing efficient entropy coding using context information of the data according to the hierarchical residual code stream to obtain a compressed data stream;

[0020] Step 106: According to the compressed data stream, progressive decoding and quality-controlled reconstruction of the data are implemented to obtain reconstructed data;

[0021] Step 107: Analyze system performance and continuously optimize each module based on the reconstruction data and the whole system operation data to obtain a final optimization strategy.

[0022] Furthermore, the step 101 includes:

[0023] a. Divide the input data X into n=2s-1 sub-blocks Where s is the division scale, s = 1, 2, ..., S;

[0024] b. For each sub-block Extract the mean Standard Deviation and gradient histogram Construct the feature matrix F (s) ;

[0025] c. Calculate F (s) Neutron Block and The Euclidean distance between Right now:

[0026]

[0027] in represents the L2 norm;

[0028] d. Define the similarity threshold as ε, if They are considered similar, and thus a self-similar graph G is constructed. (s) And extract all complete subgraphs as self-similar patterns.

[0029] Furthermore, the step 102 obtains the optimized partitioning scheme in the following manner:

[0030] a. generating an initial partitioning scheme P0 comprising K+K′ partition blocks based on the self-similarity descriptor, wherein K is the number of self-similar mode sub-blocks and K′ is the number of non-self-similar mode sub-blocks;

[0031] b. For each partition block B k Evaluate its compression cost C(B k ) and reconstruction cost R(B k ), both of which satisfy:

[0032] C(B k )=w c1 f1(B k )+w c2 f2(B k )+…

[0033] R(B k )=w r1 g1(B k )+w r2 g2(B k )+…

[0034] where f (·) and g (·) is the cost feature, w ci and w ri is the corresponding weight;

[0035] c. Iteratively optimize P0 through heuristic strategies, with the optimization goal of: min C(P), R(P) ≤ R0, and organize the optimized partition scheme P into a quadtree structure, where min C(P) represents minimizing the compression cost C(P) of the current partition scheme P, R(P) represents the reconstruction cost of the current partition scheme P, and R0 is the reconstruction cost constraint.

[0036] Furthermore, the partition block B in step 103 k The optimal fractal transformation function f k Obtained through:

[0037] a. For the candidate function f m , calculate f m (B k ) and B k The mean square error between and f m The encoding length

[0038] b. Take the function with the minimum rate distortion cost as the optimal transformation f k : Where λ∈[0,1] is the balancing factor;

[0039] c. Consider the consistency of adjacent partition block transformations to reduce the bit rate and further optimize f k , the optimization goal is: where Σ k MSE k is the mean square error MSE of all blocks k k The sum of μ is the consistency weight, and ||f k -f j ||Measures the difference between two transformation functions;

[0040] d. Perform 32-bit fixed-point quantization encoding on the optimized transformation function parameters.

[0041] Furthermore, the step 104 includes:

[0042] a. Calculate each partition block B k The residual signal e k :e k =B k -f k (B k ), where f k For B k The best fractal transformation function of ;

[0043] b. For the residual signal e k Perform L-layer wavelet decomposition to obtain the wavelet coefficients of each layer in is the wavelet coefficient of the lth layer, DWT(·) represents discrete wavelet transform;

[0044] c. For high frequency wavelet coefficients The high frequency sparse coefficient vector Represented as its wavelet dictionary matrix D (lH) The sparse linear combination of:

[0045] in is the high frequency sparse coefficient, for The L0 norm of , T is the sparsity constraint;

[0046] d. For low frequency wavelet coefficients Adaptive arithmetic coding is used to convert the low-frequency coefficient vector Encoded as low frequency encoding result c (lG) ;

[0047] e. For high frequency sparse coefficient vector The high frequency coding result c is obtained by using the entropy coding method combining run-length coding and Huffman coding. (lH) ;

[0048] f. Based on Lagrangian optimization allocation of each layer rate:

[0049] {R (1) , R (2) , ..., R (L)}=argmin D tot +λ·(R (1) +R (2) +...+R (L) -R (tot) ), where D tot is the total distortion, R (l) is the corresponding bit rate allocation of the lth layer, λ is the Lagrange multiplier, and the total distortion can be expressed as the weighted sum of the distortions of each layer: D tot =∑w l ·D l , where Dl is the reconstruction distortion of the l-th layer coefficient, w l is the scale weight.

[0050] Furthermore, the entropy coding process of step 105 includes:

[0051] a. Extract the lth layer residual data c (l) The context feature x (l) , establish the conditional probability model p(c (l) |x (l) );

[0052] b. Treating Coding Coefficients Extracting its quantization context And find the corresponding probability model parameters Compute conditional probability estimates: Where f(·) is the parameterized form of the probability model;

[0053] c. Based on the conditional probability represent The probability estimate of Perform arithmetic coding and update the coding interval to

[0054] d. Utilization Update Context The corresponding probability model parameters Implement adaptive learning.

[0055] Furthermore, the decoding and reconstruction process of step 106 includes:

[0056] a. Iterative calculation of fractal coding parameters {f k |k=1,2,...,K}, reconstruct the base layer approximate data X (0) ;

[0057] b. Decode the residual data layer by layer c (l) , get the quantized residual data right Perform inverse quantization and inverse transformation to obtain the residual reconstruction decoding result The high-frequency sparse coefficients are reconstructed as:

[0058] The low frequency coefficients are reconstructed as:

[0059] c. Decode the result and the upper reconstruction result X (l-1) Superposition, to obtain the layer-by-layer corrected reconstruction result:

[0060] d. Calculate the objective distortion metric Dl , when D l ≥Q th When , stop decoding and output the current reconstruction result X (l) , where Q th is the quality threshold.

[0061] Furthermore, the performance analysis and optimization process of step 107 includes:

[0062] a. Collect and upload objective performance indicators, including compression ratio, peak signal-to-noise ratio, structural similarity, encoding and decoding time, and memory usage;

[0063] b. To address the bottleneck of compression ratio, optimize the fractal coding transformation function, residual coding strategy and entropy coding model; to address the bottleneck of quality, improve the accuracy of fractal coding, optimize residual data correction and improve the quality assessment model; to address the bottleneck of encoding and decoding efficiency, reduce the complexity of fractal coding, reduce arithmetic coding overhead and use hardware acceleration; to address the bottleneck of memory, compress fractal coding parameters, reduce residual data cache and optimize probability model storage;

[0064] c. Based on machine learning paradigms such as self-supervised learning, continuously optimize the encoding and decoding algorithms, and update them collaboratively with the cloud and the edge.

[0065] In a second aspect, an embodiment of the present invention provides a fractal data compression storage system, the system comprising:

[0066] a. Multi-scale self-similarity analysis module, used to perform multi-level analysis on input data, identify self-similar structures, and generate self-similarity descriptors;

[0067] b. An adaptive partition optimization module, used to dynamically partition data based on the self-similarity descriptor to obtain an optimized partitioning scheme;

[0068] c. A fractal transformation coding module, which is used to design an optimal fractal transformation function set for each partition to obtain a fractal coding result;

[0069] d. A hierarchical residual coding module, used to perform multi-level coding on the residual of the fractal coding to obtain a hierarchical residual code stream;

[0070] e. A context-adaptive entropy coding module, which is used to perform entropy coding in combination with the context information of the residual data to obtain a compressed data stream;

[0071] f. Hierarchical decoding and quality reconstruction module, used to decode compressed data and reconstruct layer by layer to obtain a reconstruction result that meets quality requirements;

[0072] g. Self-optimizing feedback loop module, used to analyze system performance, adjust parameters of each module, and output dynamic optimization strategies.

[0073] In a third aspect, an embodiment of the present invention provides an Internet of Things device, comprising a processor and a memory, wherein the processor is used to execute a computer program stored in the memory, and when the program is running, the processor executes any fractal data compression and storage method provided in the specification of the present invention.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. Multi-scale self-similar feature extraction fully mines the structural information of the data to obtain a more accurate similarity description, providing an optimization basis for subsequent compression coding.

[0076] 2. Adaptive data partitioning strategy, dynamically adjust compression units according to content characteristics, minimize coding redundancy, and significantly improve compression efficiency.

[0077] 3. Flexible fractal transformation function design, through parameterized model and optimization solution, adaptively approximates complex structures and improves reconstruction quality.

[0078] 4. An efficient residual signal coding scheme, combining multi-scale decomposition and compressed sensing technology, fully utilizes the sparse characteristics of data and minimizes redundant information.

[0079] 5. Context-adaptive entropy coding dynamically updates the probability model, continuously improves coding efficiency, and adapts to dynamic changes in data.

[0080] 6. Progressive decoding and quality control, supporting different reconstruction levels, obtaining the best results under coding constraints, and flexibly meeting application requirements.

[0081] 7. Performance self-optimization and cloud-edge collaboration continuously tap performance potential through data-driven parameter tuning and algorithm evolution to achieve end-to-end global optimization.

[0082] In summary, the fractal data compression storage method and system provided by the present invention can significantly improve the compression ratio and reconstruction quality, reduce the encoding and decoding complexity and memory overhead, have strong adaptability and practical value, can be widely used in data storage and transmission of Internet of Things devices, and have good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those skilled in the art, other relevant drawings can be obtained based on these drawings without creative work.

[0084] Figure 1A flowchart of the steps of the fractal data compression and storage method for IoT devices;

[0085] Figure 2 Module diagram of fractal data compression and storage system for IoT devices;

[0086] Figure 3 Schematic diagram of the structure of the fractal data compression storage system running on IoT devices. Specific implementation plan

[0087] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments described here are only a part of the many embodiments of the present invention, not all cases. Those of ordinary skill in the art can make other various variations and implementations of the present invention based on these embodiments without paying creative work, and these variations and implementations are also within the scope of the protection required by the present invention. The flow chart is only an exemplary description, and does not mean that all content and steps must be included, and the order of steps is not immutable. Certain steps can be further subdivided, combined or partially merged, and the specific execution order can be flexibly adjusted according to actual conditions. The terms used in this specification, such as "one", "an" and "the" in the singular form, include singular and plural forms unless the context clearly indicates that it is plural. The use of these terms is only to describe specific embodiments, and is not intended to limit the scope of protection of the present invention.

[0088] The present invention relates to a fractal data compression storage method, system and related equipment. The method can be run in various terminal Internet of Things devices, such as tablet computers, laptop computers, desktop computers, PDAs, wearable devices, etc. The terminal Internet of Things device running the method can be either a single server or a server cluster. Several embodiments will be described in detail below with reference to the accompanying drawings.

[0089] like Figure 1 As shown, the fractal data compression storage method 100 proposed by the present invention includes steps 101 to 107, which are specifically as follows:

[0090] Step 101 , performing multi-level analysis on the input data, identifying self-similar structures at different scales, and generating self-similarity descriptors.

[0091] Exemplarily, the data X of the input device is segmented. Let the data length be N, and the segmentation scale be s (s=1, 2, ..., S), then the sub-block length at each scale is We can use a 50% overlapping sliding window to divide X into n sub-blocks, denoted as After segmentation, a total of n = 2s-1 sub-blocks are obtained. Extract its statistical and geometric features. Statistical features include mean and standard deviation The geometric features use the local gradient histogram (Histogram of oriented gradients, HOG) descriptor After feature extraction is completed, all sub-block features of the sth scale constitute the feature matrix where f i (s) Represents the feature representation of the i-th sub-block at the s-th scale.

[0092] For example, within each scale s, we calculate the similarity between all sub-blocks. The similarity can be measured using the Euclidean distance d(·), that is:

[0093]

[0094] in represents the L2 norm.

[0095] For example, the similarity threshold is defined as ε. Then it is called and They are similar within the ε error range. Through the similarity relationship, we can get a self-similar graph G with n nodes and m edges. (s) In G (s) In , all complete subgraphs (clusters) are extracted, and each cluster is regarded as a self-similar pattern. Finally, all self-similar patterns at s scales are recorded in the self-similarity descriptor.

[0096] Step 102: dynamically divide the data into optimal compression units based on the self-similarity descriptor to obtain an optimized partitioning scheme.

[0097] Exemplarily, an initial partitioning scheme P0 is generated according to the self-similarity descriptor. Specifically, for each self-similar pattern M k (k=1, 2, ..., K), generate a partition block B k , including M k At the same time, the sub-blocks that do not belong to any self-similar pattern are divided into separate partition blocks. The initial partition scheme P0 contains K+K′ partition blocks, where K′ is the number of non-self-similar pattern sub-blocks.

[0098] For example, for each partition block B k , we evaluate its compression cost C(B k ) and reconstruction cost R(B k ). Compression cost C(B k) measures the difficulty of compressing data in a partition block, which is related to the statistical and geometric characteristics of the data in the block, the spatial correlation of the data in the block, and the size of the block, such as the mean, standard deviation, gradient distribution, and the larger the block, the higher the compression cost. k ) measures the difficulty of decoding and reconstruction of partitioned blocks, mainly considering the visual importance of the data within the block, the correlation between the data within the block and the adjacent blocks, and the size of the block, such as texture, edge, cross-block prediction, and the smaller the block, the higher the reconstruction cost.

[0099] For example, C(B k ) and R(B k ) can be modeled as a weighted sum, namely:

[0100] C(B k )=w c1 f1(B k )+w c2 f2(B k )+…

[0101] R(B k )=w r1 g1(B k )+w r2 g2(B k )+…

[0102] Among them, f (·) and g (·) is the cost feature, w ci and w ri is the corresponding weight.

[0103] For example, the initial partition scheme P0 is not necessarily optimal, so the partition block boundaries need to be adjusted to reduce the total cost. Let P represent the current partition scheme, C(P) and R(P) be the compression cost and reconstruction cost respectively, and the optimization goal is:

[0104] minC(P),R(R)≤R0

[0105] Among them, min C(P) represents minimizing the compression cost C(P) of the current partition scheme P, R(P) represents the reconstruction cost of the current partition scheme P, and R0 is the reconstruction cost constraint. Adopt heuristic strategy to iteratively optimize the partition scheme: k Try to merge with its adjacent blocks to get a new block B` k ; Calculate the new block B` k and the remainder P\B k cost and resource consumption. If the combined total cost C(B` k )+C(P\B k )) is less than the cost C(P) of the current solution and the total resource consumption R(B` k)+R(P\B k )≤R0, then update the current plan: P=(P\B k )∪B` k .

[0106] Exemplarily, in order to facilitate subsequent encoding, the optimized partition scheme P is organized into a quadtree structure. Each leaf node corresponds to a partition block, and non-leaf nodes represent the merging of sub-blocks. At this point, the adaptive partition optimization module outputs the optimal partition scheme P and its hierarchical representation H(P), ready for subsequent encoding.

[0107] Step 103: design an optimal fractal transformation function set for each partition according to the optimization partitioning scheme of the target device to obtain a fractal transformation set.

[0108] For example, for partition block B k , we start from the family of functions F = {f1, f2, ..., f M}Select the best transformation function f k To approximate B k . Among them, each candidate function f m (m=1, 2, .., M) is defined by the following parameters: geometric transformation parameters, including the rotation angle θ m , geometric scale factor s m and the offset vector o m ; Brightness transformation parameters, including brightness scale factor a m and brightness offset b m .

[0109] Specifically, for any pixel x in the partition block, the transformed pixel is:

[0110] f m (x) = a m ·[s m ·R(θ m )·x+o m ]+b m

[0111] Where R(θ m ) is the rotation matrix, f m (x) is the transformed pixel. The function family F can be generated by traversing the discrete parameter space, for example: θ m ∈{0°, 90°, 180°, 270°}; s m ∈{0.5, 1, 2}; m ∈{(0,0),(0,1),(1,0),(1,1)};a m ∈{0.5, 1, 1.5}; b m ∈{-255, -127, 0, 127, 255}.

[0112] For example, for each candidate function f m , evaluate its encoding B k First, calculate f m (B k ) and B k The mean square error (MSE) of Calculate f m The encoding length These include: the encoding length of the transform parameters, the residual image (B k -f m (B k )) encoding length; finally, the function with the smallest rate-distortion cost is selected as the optimal transformation f k ,Right now: Where λ∈[0,1] is a balancing factor used to control distortion (through mean square error Measure) and length The weight of each partition block B k The best transformation function f k After that, the entire transformation function set {f1, f2, ..., f k}.

[0113] Specifically, the consistency of transformations of adjacent partition blocks is considered to reduce the bit rate. For example, if the transformation parameters of two adjacent blocks are very close, they can be merged into a larger block to reduce the coding overhead. Let N(B k ) means B k For a set of adjacent blocks in space, the optimization objective is:

[0114]

[0115] where ∑ k MSE k is the mean square error MSE of all blocks k k The sum of; μ is the consistency weight, ||f k -f j ||Measure the difference between two transformation functions (such as the L1 norm of the parameter difference). Iterate and optimize the function set through the local search strategy, for example: for each B k , evaluate N(B k ) j (B k adjacent elements), the original block B k And to B k The new block (B k ∪B j ), the objective function value f(B k ∪B j ) is smaller than the original block Bk The objective function value f(B k ), then recalculate the new fast objective function value f k , at this time f(B k ) is equal to the objective function value f of the new block k This process is repeated iteratively for each element and its neighbors until the objective function value cannot be further reduced by merging.

[0116] In order to further compress the transformation function parameters, the parameters are quantized and encoded. k , quantized to 2 bits in {0°, 90°, 180°, 270°}; for the scale factor s k and a k , quantize it into 8-bit fixed-point number; for the offset vector o k , quantize it into an 8-bit integer; for the offset b k , using 128-level uniform quantization. After quantization, each transformation function f k It can be represented by 32 bits (without considering the redundancy between parameters).

[0117] Exemplarily, at this point, the fractal transform coding module outputs K transform functions {f1, f2, ..., f k}, achieving a compact representation of the partitioned blocks. At the same time, the transformed residual image (B1-B1-f1(B1), B1-f2(B2)) is also obtained. In practical applications, the computational complexity of fractal coding is mainly reflected in the search of transformation parameters and similarity measurement. In order to reduce the complexity, the following optimization strategies can be considered: Simplify the transformation parameter search: When searching for the optimal iterative function system (IFS) parameters, hierarchical search, heuristic search and other strategies can be used to avoid exhausting all possible parameter combinations; Optimize the similarity measurement: When calculating the similarity between fractal blocks, a fast similarity estimation algorithm can be used, such as hash-based, feature-based methods, etc., to reduce unnecessary repeated calculations.

[0118] Step 104 , performing multi-level coding on the residual after fractal transformation according to the fractal transformation set of the target device to obtain a hierarchical residual code stream.

[0119] For example, each partition block B is calculated k The residual signal e k :e k =B k -f k (B k ), where f k For B k The optimal fractal transformation function of kPerform multi-scale decomposition. Commonly used decomposition tools include wavelet transform, Laplace pyramid, etc.

[0120] For example, taking wavelet transform as an example, k Perform L-layer wavelet decomposition: in is the wavelet coefficient of the lth layer, and DWT(·) represents discrete wavelet transform.

[0121] For example, for each layer of wavelet coefficients Different coding strategies are used for high-frequency wavelet coefficients. Sparse coding based on compressed sensing is used, that is, the high-frequency sparse coefficient vector Represented as its wavelet dictionary matrix D (lH) The sparse linear combination of:

[0122] in is the high frequency sparse coefficient, for The L0 norm (number of non-zero elements) of the wavelet dictionary matrix D is the sparsity constraint, which limits the maximum number of non-zero elements of the sparse coefficients. (lH) It can be obtained from the training data through dictionary learning algorithms such as singular value decomposition (K-SVD) to adapt to the statistical characteristics of natural images. Adaptive arithmetic coding (Advanced Audio Coding, AAC) is used to directly convert the low-frequency coefficient vector Encoded as low frequency encoding result c (lG) , for the high-frequency sparse coefficient vector The entropy coding method combining run-length encoding and Huffman coding is used to obtain the high-frequency coding result c (lH) Unlike traditional arithmetic coding, AAC adaptively adjusts the probability model according to the coded coefficients to better match the local statistical characteristics. In the coding process, it is necessary to reasonably allocate the coding bit rate between different partition blocks and different wavelet scales. tot is the total coding bit rate, and the bit rate of each layer is allocated based on Lagrangian optimization: {R (1) , R (2) , ..., R (L)}=argmin D tot +λ·(R (1) +R (2) +...+R (L) -R (tot) ), where D tot is the total distortion, R (l)is the corresponding bit rate allocation of the lth layer, λ is the Lagrange multiplier, and the total distortion can be expressed as the weighted sum of the distortions of each layer: D tot =∑w l ·D l , where D l is the reconstruction distortion of the l-th layer coefficient, w l is the scale weight.

[0123] For example, after determining the bit rate constraints of each layer, the bit rate is further allocated among different partition blocks in the same layer. The weighted allocation strategy can be designed based on factors such as the visual saliency and texture complexity of the block; finally, the hierarchical residual coding module outputs the layered coded residual data stream: {(c (1G) ,c (1H) ,R (1) ),(c (2G) ,c (2H) ,R (2) ),…,(c (LG) ,c (LH) ,R (L) ), where each tuple (c (lG) ,c (lH) ,R (l) ) contains the low-frequency coding result c of the l-th layer residual data (lG) , high frequency coding result c (lH) And the corresponding code rate allocation R (l) .

[0124] Step 105 , according to the hierarchical residual code stream of the target device, efficient entropy coding is performed using the context information of the data to obtain a compressed data stream.

[0125] Exemplarily, a multi-scale context model is constructed based on the statistical characteristics of the residual data stream. (l) , extract its context feature x (l) ; Based on context features, establish a conditional probability model p(c (l) |x (l) ), describing the residual data c based on the lth layer (l) The obtained l-th layer context feature x (l) The statistical distribution of .

[0126] It should be noted that, in the encoding stage, the conditional probability of the residual coefficient to be encoded is estimated based on the actual extracted context features. Extracting its quantization context And find the corresponding probability model parameters Furthermore, the conditional probability estimate is calculated:

[0127] Where f(·) is the parameterized form of the probability model. For example, for the Gaussian mixture model: where w i is the mixing weight, and N(·) is the Gaussian distribution function.

[0128] Furthermore, based on the estimated conditional probability, arithmetic coding is used to entropy encode the residual coefficients. represent The probability estimate of , arithmetic coding is implemented by scaling the probability interval, and the updated coding interval is: The initial interval is [0, 1]. Each time a coefficient is encoded, the interval length will be reduced To avoid interval underflow, an adaptive strategy is used to dynamically adjust the interval. Specifically, according to the current interval length, the corresponding limited precision (such as 32 bits or 64 bits) is selected, and the interval is enlarged when necessary.

[0129] In addition, a context-based probability update mechanism is also adopted, that is, after encoding, the actual encoding symbol is used to update the coefficients to be encoded. Update its context The corresponding probability model parameters During decoding, the probability model parameters are updated synchronously to ensure the consistency of encoding and decoding. Finally, the output of arithmetic coding is organized into the final bitstream by layer: (l) , b (2) , ..., b (L)}, where b (l) is the coded bit stream of layer l. At the same time, it is necessary to encode the statistical side information of the residual data of each layer, such as the probability model parameters, as the prior knowledge required for decoding. The statistical side information can be processed by fixed length coding or explicit transmission of model parameters.

[0130] Step 106: Implement progressive decoding and quality-controlled reconstruction of data according to the compressed data stream of the target device to obtain reconstructed data.

[0131] Exemplarily, the encoding information and statistical information of the residual data of each layer are parsed from the compressed data stream. The encoding information includes: fractal encoding parameters: {f k |k=1,2,...,K}, that is, the transformation function of each partition block; entropy coded bit stream: {b (l) , b (2) , ..., b (L)}, that is, the arithmetic coding output of the residual data of each layer. The statistical side information includes: context model parameters: θ (l), i.e., the conditional probability model parameters of the residual data of each layer; other auxiliary information, such as block information, wavelet basis selection, quantization step size, etc. According to the predefined syntax and data format of the coding sequence (CDS), this information is correctly extracted and interpreted.

[0132] Exemplarily, using fractal coding parameters, the approximate data of the base layer is first reconstructed. k , applying its transformation function f k Iteratively generate approximate blocks: in is any initial image block, and const is a constant. As the number of iterations increases, It will quickly converge to f k The fixed point of the original partition block B k Highly approximate. All reconstructed partition blocks are spliced ​​together to obtain the base layer approximate data X (0) On the basis of the base layer, the residual data is decoded layer by layer and used to correct the reconstruction result of each layer to obtain a higher quality approximation. (l) , using side information to reconstruct the context probability model p(c (l) |x (l) ), using an arithmetic decoder combined with p (l) Unzip b (l) , get the quantized residual data Further, Perform inverse quantization and inverse transformation to obtain the residual reconstruction decoding result of the lth layer Specifically, for the high-frequency residual layer (lH=1, 2, ..., L-1), Perform inverse quantization to obtain the reconstruction of sparse coefficients Then use the dictionary D of the lth layer (lH) right Perform sparse reconstruction and obtain For the low-frequency residual layer (lG = L), directly Dequantize and get Formally, this process can be expressed as:

[0133]

[0134] Furthermore, the decoding result and the upper reconstruction result X (l-1) Superposition, we get the modified reconstruction of layer l: in for The inverse transform is the inverse wavelet transform or the inverse Laplace transform. In this way, the decoding and correction are performed layer by layer until the preset reconstruction layer number l is reached, and the final reconstruction result X is obtained. (l) .

[0135] Furthermore, in order to meet the needs of different application scenarios, a quality assessment and control mechanism is introduced. Quality assessment aims to quantify the degree of distortion of the reconstruction results. Commonly used distortion metrics include: Mean Square Error (MSE), Structural Similarity (SSIM), and Visual Information Fidelity (VIF).

[0136] According to the reconstruction result X (l) Calculate the corresponding distortion measure D with the original data X l and with the preset quality threshold Q th Compare. When D l ≥Q th , the decoding process is stopped and the current reconstruction result is output.

[0137] In addition, the reconstruction quality and decoding complexity can be adjusted by controlling the number of decoding layers l. The higher the number of layers, the better the reconstruction quality, but the greater the computational overhead. The optimal l value can be adaptively selected according to application requirements, such as energy budget, delay constraint, etc. The hierarchical decoding and quality reconstruction module outputs a reconstruction result X that meets the quality requirements. l , achieving efficient and controllable restoration from compressed code stream to original data. In practical applications, the time and space complexity of arithmetic coding mainly depends on the storage and update of the context model. In order to reduce the complexity, the following optimization strategies can be considered: Optimize context model storage: When designing the context model, compact data structures such as sparse representation and hash index can be used to reduce memory usage; Simplify interval updates: In the interval update process of arithmetic coding, approximate calculations, table lookup and other methods can be used to avoid unnecessary high-precision operations.

[0138] Step 107 , analyzing system performance and continuously optimizing each module according to the reconstruction data of the target device and the whole system operation data, to obtain a final optimization strategy.

[0139] For example, a set of performance indicators is designed to comprehensively evaluate the encoding and decoding performance of the system. It mainly includes: compression ratio: the ratio of the amount of compressed data to the amount of original data, which measures the data compression effect; reconstruction quality: objective performance indicators such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) to measure the degree of distortion of the reconstruction result; coding efficiency: the time or computing resources required to complete the coding task, which measures the complexity of the coding process; decoding efficiency: the time or computing resources required to complete the decoding task, which measures the complexity of the decoding process; memory usage: the storage space required in the encoding and decoding process, which measures the memory overhead of the system. A performance monitoring module is embedded in the codec to collect and record the above indicators in real time. At the same time, the indicator data is uploaded to the cloud for long-term statistical analysis.

[0140] Exemplarily, based on the collected performance data, we diagnose system bottlenecks and locate key points for performance optimization. Specifically, we analyze performance bottlenecks from the following aspects: Compression ratio bottleneck: whether the compression ratio meets the application requirements and whether there is room for improvement; Quality bottleneck: whether the reconstruction quality meets the application requirements and whether there is room for improvement; Coding efficiency bottleneck: whether the encoding speed meets the real-time requirements and whether there is a possibility of acceleration; Decoding efficiency bottleneck: whether the decoding speed meets the real-time requirements and whether there is room for acceleration; Memory bottleneck: whether the memory usage exceeds the device limit and whether there is room for saving. In response to the above bottlenecks, we use data mining technologies such as statistical inference and machine learning to analyze the relationship between performance indicators and encoding and decoding parameters and discover the key influencing factors for performance optimization.

[0141] Furthermore, after finding the performance bottleneck, we can take corresponding strategies to dynamically adjust the encoding and decoding parameters. For the compression ratio bottleneck, we can improve the compression performance by the following means: optimizing the fractal coding transformation function, such as increasing function parameters, expanding function classes, etc.; optimizing the residual coding strategy, such as adjusting the wavelet basis selection, quantization step size, etc.; optimizing the entropy coding model, such as increasing the context dimension, updating the probability estimation method, etc. For the quality bottleneck, we can improve the reconstruction effect by the following means: improving the fractal coding accuracy, such as reducing the quantization step size, increasing the number of iterations, etc.; optimizing the residual data correction, such as selecting high-order approximation methods, edge-sensitive filtering, etc.; improving the quality assessment model, such as introducing visual attention mechanisms, local quality perception, etc. For the codec efficiency bottleneck, we can take the following acceleration measures: reducing the complexity of fractal coding, such as simplifying the transformation parameter search, optimizing the similarity metric, etc.; reducing the arithmetic coding overhead, such as optimizing the context model storage, simplifying the interval update, etc.; using hardware acceleration, such as single instruction multiple data (SIMD) instruction optimization, GPU parallel computing, etc. For memory bottlenecks, we can use the following methods to control storage overhead: compress fractal coding parameters, such as parameter quantization, Huffman coding, etc.; reduce residual data cache, such as streaming processing, blocking strategy, etc.; optimize probability model storage, such as sparse representation, parameter pruning, etc.

[0142] The above tuning strategies are encapsulated as a parameter adaptive adjustment algorithm to optimize the encoding and decoding parameters in real time based on performance feedback. Through iterative optimization, the system performance is continuously improved and converged to the optimal state under application constraints.

[0143] In the long run, simple parameter tuning is difficult to cope with changes in data and application requirements. In pursuit of continuous performance improvement, an online algorithm update mechanism is introduced. Specifically, the following strategies are used to dynamically update the codec algorithm: Fractal coding algorithm: adaptively adjust the transformation function form, similarity measurement, etc. through autonomous learning; Residual coding algorithm: dynamically optimize wavelet basis selection, dictionary learning, etc. through incremental learning; Entropy coding algorithm: adaptively select the optimal context model, probability estimation method, etc. through reinforcement learning; Quality assessment algorithm: online update of visual perception model, attention prediction model, etc. through transfer learning. Comprehensively use unsupervised learning paradigms such as self-supervised learning and active learning to adaptively learn and improve codec algorithms from data streams. At the same time, the cloud-edge collaborative optimization framework is also integrated to optimize the codec through continuous learning in the cloud, and the updates are synchronized to the terminal device regularly. The self-optimization feedback loop module outputs an optimization strategy based on long-term data accumulation and algorithm evolution, realizing continuous self-optimization of the codec system.

[0144] like Figure 2As shown, the fractal data compression and storage system of the present invention includes the following modules: a multi-scale self-similarity analysis module (S201), an adaptive partition optimization module (S202), a fractal transform coding module (S203), a hierarchical residual coding module (S204), a context adaptive entropy coding module (S205), a hierarchical decoding and quality reconstruction module (S206) and a self-optimization feedback loop module (S207).

[0145] The data flow between the modules is as follows: the original device data is first input into the multi-scale self-similarity analysis module to extract the self-similar features; then, the adaptive partition optimization module generates the optimal data partition according to the self-similar features; the fractal transform coding module transforms and encodes each partition; the hierarchical residual coding module performs hierarchical encoding on the transformed residual signal; the context adaptive entropy coding module performs entropy coding and outputs a compressed data stream; the hierarchical decoding and quality reconstruction module reads the compressed data and decodes and reconstructs it; finally, the self-optimization feedback loop module analyzes the system performance, tunes the parameters of each module, and outputs the optimization strategy.

[0146] like Figure 3 As shown, the Internet of Things (IoT) device 300 is mainly composed of a processor 301 and a memory 302, which are connected via a bus 303, which can be an I2C (Inter-Integrated Circuit) bus or the like.

[0147] Specifically, the processor 301 undertakes computing and control functions and is the core of the operation of the Internet of Things (IoT) device. The processor 301 can be a low-power microcontroller chip of the Internet of Things, or other general-purpose processors, digital signal processors (Digital signal processing, DSP), application-specific integrated circuits (Application-Specific Integrated Circuit, ASIC), field-programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor here can be a microprocessor or any conventional processor.

[0148] The memory 302 can be in various forms such as a Flash chip, a read-only memory (ROM), a magnetic disk, an optical disk, a USB flash drive or a mobile hard disk.

[0149] It should be noted that Figure 3 The structure shown in the figure is only a partial structural diagram related to the embodiment of the present invention, and does not limit the Internet of Things (IoT) device to which the solution is applied. The actual Internet of Things (IoT) device may include more or fewer components, or merge some components, or use different component arrangements.

[0150] It is worth noting that the main function of the processor 301 is to execute the computer programs stored in the memory 302. When running these programs, the processor can implement the fractal data compression storage method proposed in the embodiment of the present invention.

[0151] In a specific embodiment of the present invention, the processor is designed to execute a computer program stored in the memory. When the program is running, the processor will complete the following steps:

[0152] a. Perform multi-level analysis on the input data to identify self-similar structures at different scales and generate self-similarity descriptors.

[0153] b. Based on the self-similarity descriptor, dynamically divide the data into optimal compression units to obtain the optimized differentiation scheme.

[0154] c. According to the optimization partitioning scheme of the target device, design an optimal fractal transformation function set for each partition to obtain a fractal transformation set.

[0155] d. According to the fractal transformation set of the target device, the residual after fractal transformation is multi-level encoded to obtain a hierarchical residual code stream.

[0156] e. According to the hierarchical residual code stream of the target device, efficient entropy coding is performed using the context information of the data to obtain a compressed data stream.

[0157] f. According to the compressed data stream of the target device, progressive decoding of the data and reconstruction with controllable quality are implemented to obtain reconstructed data.

[0158] g. Analyze system performance and continuously optimize each module based on the reconstruction data of the target device and the operation data of the entire system to obtain the final optimization strategy.

[0159] The embodiment of the present invention also provides a computer-readable storage medium, in which one or more programs are stored. These programs can be executed by one or more processors to implement the steps of any fractal data compression storage method described in the embodiment of the present invention. The storage medium can be an internal storage unit of the aforementioned Internet of Things (IoT) device, such as a hard disk or memory. It can also be an external storage device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. It should be understood by those skilled in the art that the steps in the above method, the functional modules in the system or device can be implemented by software, firmware, hardware or a combination thereof. In hardware implementation, the division of each functional module does not necessarily correspond to the division of physical components. A physical component may perform multiple functions, or a function may be completed by multiple physical components in collaboration. These components can be software executed by a processor (such as a CPU, DSP or microprocessor), or hardware implementation, or an integrated circuit (such as: Application-specific integrated circuit Application-specific integrated circuit, ASIC). These software can be distributed on computer-readable media, including computer storage media (non-temporary) and communication media (temporary). Computer storage media include but are not limited to RAM, ROM, rewritable memory chips (Electrically Erasable Programmable Read-Only Memory, EEPROM), flash memory, CD-ROM, DVD, magnetic tape, disk, etc., which are used to store information. Communication media include any media that transmit computer-readable instructions, data structures, etc. in modulated data signals.

Claims

1. A method for compressing and storing fractal data, characterized in that: The steps include: Step 101: Perform multi-level analysis on the input data, identify self-similar structures at different scales, and generate self-similarity descriptors; Step 102: dynamically divide the data into optimal compression units based on the self-similarity descriptor to obtain an optimized partitioning scheme; Step 103: According to the optimized partitioning scheme, an optimal fractal transformation function set is designed for each partition to obtain a fractal transformation set; wherein, for any pixel x in the partition block, the transformed pixel is: f m (x)=a m ·[s m ·R(θ m )·x+o m ]+b m Among them, f m (x) is the transformed pixel, θ m is the rotation angle, R(θ m ) is the rotation matrix, a m 、s m , o m and b m They are brightness scale factor, geometric scale factor, offset vector and brightness offset respectively; Step 104: performing multi-level coding on the residual after fractal transformation according to the fractal transformation set to obtain a hierarchical residual code stream; Step 105: performing efficient entropy coding using context information of the data according to the hierarchical residual code stream to obtain a compressed data stream; Step 106: According to the compressed data stream, progressive decoding and quality-controlled reconstruction of the data are implemented to obtain reconstructed data; Step 107: Analyze system performance and continuously optimize each module based on the reconstruction data and the whole system operation data to obtain a final optimization strategy.

2. The method for compressing and storing fractal data according to claim 1, characterized in that: The step 101 includes: a. Divide the input data X into n=2s-1 sub-blocks Where s is the division scale, s = 1, 2, ..., S; b. For each sub-block Extract the mean Standard Deviation and gradient histogram Construct the feature matrix F(s); c. Calculate the F(s) neutron block and The Euclidean distance between Right now: in represents the L2 norm; d. Define the similarity threshold as ε, if They are considered similar, and thus a self-similar graph G is constructed. (s) And extract all complete subgraphs as self-similar patterns.

3. The method for compressing and storing fractal data according to claim 1, characterized in that: The step 102 obtains the optimized partitioning scheme in the following manner: a. generating an initial partitioning scheme P0 comprising K+K′ partition blocks based on the self-similarity descriptor, wherein K is the number of self-similar mode sub-blocks and K′ is the number of non-self-similar mode sub-blocks; b. For each partition block B k Evaluate its compression cost C(B k ) and reconstruction cost R(B k ), both of which satisfy: C(B k )=W c1 ·f1(B k )+w c2 ·f2(B k )+… R(B k )=w r1 ·g1(B k )+w r2 ·g2(B k )+… where f (.) and g (.) is the cost feature, w ci and w ri is the corresponding weight; c. Iteratively optimize P0 through heuristic strategies, with the optimization goal of: min C(P), R(P) ≤ R0, and organize the optimized partition scheme P into a quadtree structure, where min C(P) represents minimizing the compression cost C(P) of the current partition scheme P, R(P) represents the reconstruction cost of the current partition scheme P, and R0 is the reconstruction cost constraint.

4. The method for compressing and storing fractal data according to claim 1, characterized in that: In step 103, partition block B k The optimal fractal transformation function fk is obtained as follows: a. For the candidate function f m , calculate f m (B k ) and B k The mean square error between and f m The encoding length b. Take the function with the minimum rate-distortion cost as the optimal transformation fk: Where λ∈[0,1] is the balancing factor; c. Consider the consistency of adjacent partition block transformations to reduce the bit rate and further optimize f k , the optimization goal is: where ∑ k MSE k is the mean square error MSE of all blocks k k The sum of μ is the consistency weight, and ||f k -f j ||Measures the difference between two transformation functions; d. Perform 32-bit fixed-point quantization encoding on the optimized transformation function parameters.

5. The method for compressing and storing fractal data according to claim 1, characterized in that: The multi-level encoding process in step 104 specifically includes: a. Calculate each partition block B k The residual signal e k :e k =B k -f k (B k ), where f k For B k The best fractal transformation function of ; b. For the residual signal e k Perform L-layer wavelet decomposition to obtain the wavelet coefficients of each layer in is the wavelet coefficient of the lth layer, DWT(·) represents discrete wavelet transform; c. For high frequency wavelet coefficients The high frequency sparse coefficient vector Represented as its wavelet dictionary matrix D (lH) The sparse linear combination of: ,in is the high frequency sparse coefficient, for The L0 norm of , T is the sparsity constraint; d. For low frequency wavelet coefficients Adaptive arithmetic coding is used to convert the low-frequency coefficient vector Encoded as low frequency encoding result c (lG) ; e. For high frequency sparse coefficient vector The high frequency coding result c is obtained by using the entropy coding method combining run-length coding and Huffman coding. (lH) ; f. Based on Lagrangian optimization allocation of each layer rate: {R (1) , R (2) , …, R (L) }=argmin D tot +λ·(R (1) +R (2) +…+R (L) -R (tot) ), where D tot is the total distortion, R (l) is the corresponding bit rate allocation of the lth layer, λ is the Lagrange multiplier, and the total distortion can be expressed as the weighted sum of the distortions of each layer: D tot =∑w l ·D l , where D l is the reconstruction distortion of the lth layer coefficients, w l is the scale weight.

6. The method for compressing and storing fractal data according to claim 1, characterized in that: The entropy coding process of step 105 specifically includes: a. Extract the lth layer residual data c (l) The context feature x (l) , establish the conditional probability model p(c (l) |x (l) ); b. Treating Coding Coefficients Extracting its quantization context And find the corresponding probability model parameters Compute conditional probability estimates: Where f(·) is the parameterized form of the probability model; c. Based on the conditional probability represent The probability estimate of Perform arithmetic coding and update the coding interval to d. Utilization Update Context The corresponding probability model parameters Implement adaptive learning.

7. The method for compressing and storing fractal data according to claim 1, characterized in that: The decoding and reconstruction process of step 106 specifically includes: a. Iterative calculation of fractal coding parameters {f k |k=1,2,...,K}, reconstruct the base layer approximate data X (0) ; b. Decode the residual data layer by layer c (l) , get the quantized residual data right Perform inverse quantization and inverse transformation to obtain the residual reconstruction decoding result The high-frequency sparse coefficients are reconstructed as: The low frequency coefficients are reconstructed as: c. Decode the result Superimposed with the upper layer reconstruction result X(l-1), we get the layer-by-layer corrected reconstruction result: d. Calculate the objective distortion metric D l , when D l ≥Q th When , stop decoding and output the current reconstruction result X (l) , where Q th is the quality threshold.

8. The method for compressing and storing fractal data according to claim 1, characterized in that: The performance analysis and optimization process of step 107 specifically includes: a. Collect and upload objective performance indicators, including compression ratio, peak signal-to-noise ratio, structural similarity, encoding and decoding time, and memory usage; b. To address the bottleneck of compression ratio, optimize the fractal coding transformation function, residual coding strategy and entropy coding model; to address the bottleneck of quality, improve the accuracy of fractal coding, optimize residual data correction and improve the quality assessment model; to address the bottleneck of encoding and decoding efficiency, reduce the complexity of fractal coding, reduce arithmetic coding overhead and use hardware acceleration; to address the bottleneck of memory, compress fractal coding parameters, reduce residual data cache and optimize probability model storage; c. Based on machine learning paradigms such as self-supervised learning, continuously optimize the encoding and decoding algorithms, and update them collaboratively with the cloud and the edge.

9. A fractal data compression storage system, characterized in that: The method for implementing any one of claims 1 to 8 comprises: a. Multi-scale self-similarity analysis module, used to perform multi-level analysis on input data, identify self-similar structures, and generate self-similarity descriptors; b. An adaptive partition optimization module, used to dynamically partition data based on the self-similarity descriptor to obtain an optimized partitioning scheme; c. A fractal transformation coding module, which is used to design an optimal fractal transformation function set for each partition to obtain a fractal coding result; d. A hierarchical residual coding module, used to perform multi-level coding on the residual of the fractal coding to obtain a hierarchical residual code stream; e. A context-adaptive entropy coding module, which is used to perform entropy coding in combination with the context information of the residual data to obtain a compressed data stream; f. Hierarchical decoding and quality reconstruction module, used to decode compressed data and reconstruct layer by layer to obtain a reconstruction result that meets quality requirements; g. Self-optimizing feedback loop module, used to analyze system performance, adjust parameters of each module, and output dynamic optimization strategies.

10. An Internet of Things device, comprising a processor and a memory, characterized in that: The processor is used to execute a computer program stored in the memory. When the program is running, the processor executes the fractal data compression and storage method according to any one of claims 1 to 8.

Images