Fractal data compression storage method, system and related device

Abstract

The application relates to the technical field of data processing, and discloses a fractal data compression storage method, a fractal data compression storage system and related equipment thereof. The method comprises the following steps: firstly, performing multi-level analysis on input data, identifying self-similar structures on different scales, and generating self-similarity descriptors; then, dynamically dividing the data into optimal compression units based on the self-similarity descriptors, and designing a best set of fractal transformation functions for each partition; next, performing multi-level coding on the residual error after fractal transformation, and performing adaptive entropy coding by using context information to obtain a compressed data stream; finally, realizing progressive decoding of the compressed data and quality-controllable reconstruction. The application also introduces a performance analysis and online optimization mechanism, which can continuously improve the system performance. The application can significantly improve the compression ratio and reconstruction quality, reduce the coding and decoding complexity and memory overhead, and can be widely applied to data storage and transmission of Internet of Things devices.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a fractal data compression and storage method, system, and related equipment. Technical Background

[0002] With the rapid development of IoT technology, more and more devices are connecting to the network, generating massive amounts of data. However, IoT devices typically have limited storage and computing resources, making it difficult to effectively store and transmit raw data. Therefore, there is an urgent need to develop efficient data compression technologies to minimize data volume while ensuring reconstruction quality, in order to adapt to the characteristics and needs of IoT applications.

[0003] Traditional data compression methods, such as predictive coding and transform coding, primarily utilize the spatial correlation and statistical redundancy of data for compression. For example, JPEG uses discrete cosine transform to convert the image into frequency domain coefficients before quantization and entropy coding; PNG uses the LZ77 algorithm to eliminate repeating sequences in the data. However, these methods typically only consider data characteristics at a single scale, making it difficult to fully exploit multi-scale structural information. Furthermore, most of them employ fixed coding strategies, failing to adaptively adjust compression parameters based on data content, thus limiting compression performance.

[0004] In recent years, compression methods based on fractal theory have received widespread attention. Fractal compression utilizes the self-similarity of data at different scales to approximate the original data through an iterative function system. Compared with traditional methods, fractal compression achieves higher compression ratios and reconstruction quality, making it particularly suitable for textured images and complex multidimensional data. However, existing fractal compression algorithms still have the following shortcomings:

[0005] (1) Insufficient extraction of self-similar features. Most algorithms only consider the similarity of local regions and ignore global structural information, making it difficult to accurately characterize the multi-scale features of the data.

[0006] (2) Inadequate data partitioning strategy. Traditional fixed block size partitioning does not fully consider data content, resulting in redundant compression units and low encoding efficiency.

[0007] (3) The transformation function design is not flexible enough. The simple geometric transformation function form limits the ability to express complex structures and affects the reconstruction quality.

[0008] (4) The encoding efficiency of residual information is not high. There is a lack of effective encoding methods for the transformed residual data, resulting in a loss of compression ratio.

[0009] (5) Lack of adaptive optimization mechanism. Fixed encoding and decoding parameters are difficult to adapt to dynamically changing data characteristics and application requirements, and cannot achieve continuous performance improvement.

[0010] In summary, existing technologies are insufficient to meet the practical needs of IoT data compression, necessitating a new, efficient, adaptive, and robust compression and storage method. This invention is proposed against this backdrop, aiming to solve the aforementioned technical problems and provide a fractal data compression and storage method and system for the IoT. Summary of the Invention

[0011] This invention proposes a fractal data compression and storage method, system, and related equipment to address the problems in existing technologies, such as insufficient extraction of self-similar features, suboptimal data partitioning strategies, inflexible transformation function design, low efficiency in residual information encoding, and lack of adaptive optimization mechanisms.

[0012] In a first aspect, embodiments of the present invention provide a fractal data compression and storage method, comprising:

[0013] Step 101: Perform multi-level analysis on the input data to identify self-similar structures at different scales and generate self-similar descriptors;

[0014] Step 102: Based on the self-similarity descriptor, dynamically divide the data into optimal compression units to obtain an optimized partitioning scheme;

[0015] Step 103: Based on the optimized partitioning scheme, design the optimal set of fractal transformation functions for each partition to obtain the fractal transformation set; wherein, for any pixel x within a partition block, its transformed pixel is:

[0016] f m (x)=a m ·[s m ·R(θ m )·x+o m ]+b m

[0017] Among them, f m (x) represents the transformed pixel, θ m Let R(θ) be the rotation angle. m Let ) be a rotation matrix, and a m s m o m and b m These are the brightness scale factor, geometric scale factor, offset vector, and brightness offset, respectively.

[0018] Step 104: Based on the fractal transformation set, perform multi-level encoding on the residual after fractal transformation to obtain a hierarchical residual code stream;

[0019] Step 105: Based on the hierarchical residual bitstream, perform efficient entropy coding using the context information of the data to obtain a compressed data stream;

[0020] Step 106: Based on the compressed data stream, perform progressive decoding and quality-controlled reconstruction of the data to obtain reconstructed data;

[0021] Step 107: Based on the reconstructed data and the overall system operation data, analyze the system performance and continuously optimize each module to obtain the final optimization strategy.

[0022] Further, step 101 includes:

[0023] a. Divide the input data X into n = 2s-1 sub-blocks. Where s is the division scale, s = 1, 2, ..., S;

[0024] b. For each sub-block Extract the mean Standard deviation and gradient histogram Constructing the characteristic matrix F (s) ;

[0025] c. Calculate F (s) Neutron block and Euclidean distance Right now:

[0026]

[0027] in Represents the L2 norm;

[0028] d. Define the similarity threshold as ε. If They are then considered similar, and a self-similar graph G is constructed accordingly. (s) And extract all complete subgraphs as self-similar patterns.

[0029] Further, step 102 obtains the optimized partitioning scheme in the following manner:

[0030] a. Generate an initial partitioning scheme P0 containing K+K′ partition blocks based on the self-similarity descriptor, where K is the number of self-similarity pattern sub-blocks and K′ is the number of non-self-similarity pattern sub-blocks;

[0031] b. For each partition block B k Assess its compression cost C(B) k ) and reconstruction cost R(B k Both satisfy:

[0032] C(B k ) = w c1 ·f1(B k )+w c2 ·f2(B k )+…

[0033] R(B k ) = w r1 ·g1(B k )+w r2 ·g2(B k )+…

[0034] Where f (·) and g (·) As a cost feature, w ci and w ri For the corresponding weights;

[0035] c. Iteratively optimize P0 using a heuristic strategy. The optimization objective is: min C(P), R(P)≤R0. Organize the optimized partitioning scheme P into a quadtree structure, where min C(P) represents minimizing the compression cost C(P) of the current partitioning scheme P, R(P) represents the reconstruction cost of the current partitioning scheme P, and R0 is the reconstruction cost constraint.

[0036] Further, in step 103, partition block B k The optimal fractal transformation function f k Obtained through the following methods:

[0037] a. For candidate function f m Calculate f m (B k ) and B k Mean square error between and f m Encoding length

[0038] b. The function with the lowest rate-distortion cost is selected as the optimal transformation f. k : Where λ∈[0,1] is the balance factor;

[0039] c. Consider the consistency of transitions between adjacent partition blocks to reduce the bit rate and further optimize f. k The optimization objective is: Where Σ k MSE k The mean square error (MSE) of all blocks k k The sum, μ is the consistency weight, ||f k -f j ||Measures the difference between two transformation functions;

[0040] d. Perform 32-bit fixed-point quantization encoding on the optimized transform function parameters.

[0041] Further, step 104 includes:

[0042] a. Calculate B for each partition block. k The residual signal e k :e k =B k -f k (B k ), where f k For B k The optimal fractal transformation function;

[0043] b. For the residual signal e k Perform L-level wavelet decomposition to obtain the wavelet coefficients of each level. in The coefficients are the wavelet coefficients of the l-th layer, and DWT(·) represents the discrete wavelet transform;

[0044] c. For high-frequency wavelet coefficients high-frequency sparse coefficient vector It is represented by its wavelet dictionary matrix D (lH) Sparse linear combinations of the following:

[0045] in For high-frequency sparse coefficients, for The L0 norm, where T is the sparsity constraint;

[0046] d. For low-frequency wavelet coefficients Adaptive arithmetic coding is used to transform the low-frequency coefficient vector The encoding result is low-frequency encoding c (lG) ;

[0047] e. For high-frequency sparse coefficient vectors The high-frequency coding result c is obtained by using an entropy coding method that combines run-length coding and Huffman coding. (lH) ;

[0048] f. Allocation of rates for each level based on Lagrange optimization:

[0049] {R (1) R (2) , ..., R (L)}=argmin D tot +λ·(R (1) +R (2) +...+R (L) -R (tot) ), where D tot For total distortion, R (l) For the corresponding code rate allocation of layer l, λ is the Lagrange multiplier, and the total distortion can be expressed as the weighted sum of the distortions of each layer: D tot =∑w l ·D l Dl For the reconstruction distortion of the coefficients in the l-th layer, w l For scale weights.

[0050] Furthermore, the entropy encoding process in step 105 includes:

[0051] a. Extract the residual data of the l-th layer c (l) Contextual features x (l) Establish a conditional probability model p(c) (l) |x (l) );

[0052] b. Coding coefficients Extract its quantization context And find the corresponding probability model parameters Calculate the conditional probability estimate: Where f(·) is the parameterized form of the probability model;

[0053] c. Based on conditional probability represent The probability estimate for Perform arithmetic encoding and update the encoding range as follows:

[0054] d. Utilize Update Context Corresponding probabilistic model parameters Achieve adaptive learning.

[0055] Furthermore, the decoding and reconstruction process in step 106 includes:

[0056] a. Iteratively calculate fractal encoding parameters {f k |k=1,2,...,K}, reconstruct the approximate data X of the base layer. (0) ;

[0057] b. Decode the residual data layer by layer. (l) Quantized residual data were obtained. right Perform inverse quantization and inverse transform to obtain the residual reconstruction decoding result. The high-frequency sparse coefficients are reconstructed as follows:

[0058] The low-frequency coefficients are reconstructed as follows:

[0059] c. Decode the result Compared with the upper-level reconstruction result X (l-1) By superimposing the layers, we obtain the layer-by-layer correction and reconstruction results:

[0060] d. Calculate the objective distortion measure Dl When D l ≥Q th When the time is up, stop decoding and output the current reconstruction result X. (l) Q th This is the quality threshold.

[0061] Furthermore, the performance analysis and optimization process in step 107 includes:

[0062] a. Collect and upload objective performance metrics, including compression ratio, peak signal-to-noise ratio, structural similarity, encoding / decoding time, and memory usage;

[0063] b. To address the compression ratio bottleneck, optimize the fractal coding transformation function, residual coding strategy, and entropy coding model; to address the quality bottleneck, improve fractal coding accuracy, optimize residual data correction, and improve the quality assessment model; to address the encoding / decoding efficiency bottleneck, reduce fractal coding complexity, reduce arithmetic coding overhead, and utilize hardware acceleration; to address the memory bottleneck, compress fractal coding parameters, reduce residual data caching, and optimize probabilistic model storage.

[0064] c. Based on machine learning paradigms such as self-supervised learning, continuously optimize the encoding and decoding algorithms and update them collaboratively between the cloud and the edge.

[0065] In a second aspect, embodiments of the present invention provide a fractal data compression and storage system, the system comprising:

[0066] a. Multi-scale self-similarity analysis module, used to perform multi-level analysis on input data, identify self-similar structures, and generate self-similar descriptors;

[0067] b. An adaptive partitioning optimization module, used to dynamically partition data based on the self-similarity descriptor to obtain an optimized partitioning scheme;

[0068] c. Fractal transformation coding module, used to design the optimal set of fractal transformation functions for each partition and obtain fractal coding results;

[0069] d. Hierarchical residual coding module, used to perform multi-level coding on the residuals of fractal coding to obtain hierarchical residual bitstream;

[0070] e. Context-adaptive entropy coding module, used to combine the context information of residual data to perform entropy coding and obtain compressed data stream;

[0071] f. The hierarchical decoding and quality reconstruction module is used to decode compressed data and reconstruct it layer by layer to obtain a reconstruction result that meets the quality requirements;

[0072] g. Self-optimizing feedback loop module, used to analyze system performance, adjust the parameters of each module, and output dynamic optimization strategies.

[0073] Thirdly, embodiments of the present invention provide an Internet of Things (IoT) device, including a processor and a memory, wherein the processor is used to execute a computer program stored in the memory, and when the program is running, the processor executes any of the fractal data compression and storage methods provided in this specification.

[0074] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0075] 1. Multi-scale self-similarity feature extraction fully explores the structural information of the data, obtains a more accurate similarity description, and provides an optimization foundation for subsequent compression coding.

[0076] 2. Adaptive data partitioning strategy: dynamically adjusts compression units based on content characteristics, minimizes coding redundancy, and significantly improves compression efficiency.

[0077] 3. Flexible fractal transformation function design, through parameterized model and optimized solution, adaptively approximates complex structures and improves reconstruction quality.

[0078] 4. An efficient residual signal coding scheme, combined with multi-scale decomposition and compressed sensing technology, makes full use of the sparsity characteristics of the data and minimizes redundant information.

[0079] 5. Context-adaptive entropy coding dynamically updates the probability model, continuously improving coding efficiency and adapting to dynamic changes in data.

[0080] 6. Progressive decoding and quality control support different reconstruction levels, achieving optimal results under coding constraints and flexibly meeting application needs.

[0081] 7. Performance self-optimization and cloud-edge collaboration: Through data-driven parameter tuning and algorithm evolution, continuously explore performance potential and achieve end-to-end global optimization.

[0082] In summary, the fractal data compression and storage method and system provided by this invention can significantly improve the compression ratio and reconstruction quality, reduce encoding and decoding complexity and memory overhead, and has strong adaptability and practical value. It can be widely used in data storage and transmission of IoT devices and has good application prospects. Attached Figure Description

[0083] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1Flowchart of a method for compressing and storing fractal data from IoT devices;

[0085] Figure 2 Module diagram of a fractal data compression and storage system for IoT devices;

[0086] Figure 3 A schematic diagram of the structure of an IoT device running a fractal data compression and storage system. Detailed Implementation Plan

[0087] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the embodiments described herein are only a part of the numerous embodiments of the present invention, and not all of them. Those skilled in the art can make various other modifications and implementations of the present invention based on these embodiments without creative effort, and these modifications and implementations are also within the scope of protection claimed by the present invention. Flowcharts are merely illustrative and do not necessarily represent a complete list of all contents and steps, nor is the order of steps fixed. Some steps can be further subdivided, combined, or partially merged, and the specific execution order can be flexibly adjusted according to actual circumstances. Terms used in this specification, such as the singular forms "a," "an," and "the," include both singular and plural forms unless the context explicitly indicates that it is plural. The use of these terms is solely for describing specific embodiments and is not intended to constitute any limitation on the scope of protection of the present invention.

[0088] This invention relates to a fractal data compression and storage method, system, and related equipment. The method can run on various terminal IoT devices, such as tablets, laptops, desktops, PDAs, and wearable devices. The terminal IoT device running this method can be a single server or a server cluster. Several embodiments will be described in detail below with reference to the accompanying drawings.

[0089] like Figure 1 As shown, the fractal data compression and storage method 100 proposed in this invention includes steps 101 to 107, as detailed below:

[0090] Step 101: Perform multi-level analysis on the input data to identify self-similar structures at different scales and generate self-similarity descriptors.

[0091] For example, the data X from the input device is segmented. Let the data length be N, and the segmentation scale be s (s = 1, 2, ..., S), then the length of the sub-block under each scale is... A sliding window with a 50% overlap ratio can be used to divide X into n sub-blocks, denoted as . After segmentation, a total of n = 2s-1 sub-blocks are obtained. Next, for each sub-block... Extract its statistical and geometric features. Statistical features include the mean. and standard deviation Geometric features are descriptors using the Histogram of Oriented Gradients (HOG). After feature extraction is completed, the features of all sub-blocks at scale s constitute the feature matrix. Where f i (s) This represents the feature representation of the i-th sub-block at scale s.

[0092] For example, within each scale s, we calculate the pairwise similarity between all sub-blocks. The similarity can be measured using the Euclidean distance d(·), i.e.:

[0093]

[0094] in This represents the L2 norm.

[0095] For example, the similarity threshold is defined as ε. Then it is called and They are similar within the ε error range. Through similarity relations, a self-similar graph G with n nodes and m edges can be obtained. (s) In G (s) In the process, all complete subgraphs (clusters) are extracted, and each clique is considered a self-similar pattern. Finally, all self-similar patterns at s scales are recorded in the self-similarity descriptor.

[0096] Step 102: Based on the self-similarity descriptor, dynamically divide the data into optimal compression units to obtain an optimized partitioning scheme.

[0097] For example, an initial partitioning scheme P0 is generated based on the self-similarity descriptor. Specifically, for each self-similarity pattern M... k (k = 1, 2, ..., K), generate a partition block B. k , including M k All data sub-blocks within the same partition scheme. Simultaneously, sub-blocks not belonging to any self-similar pattern are divided into separate partition blocks. The initial partition scheme P0 contains K+K′ partition blocks, where K′ is the number of non-self-similar pattern sub-blocks.

[0098] For example, for each partition block B k We evaluate its compression cost C(B) k ) and reconstruction cost R(B k Compression cost C(B) kThe difficulty of compressing data within a block is measured by factors such as the statistical and geometric characteristics of the data, spatial correlation within the block, and block size, including mean, standard deviation, gradient distribution, and the fact that larger blocks generally have higher compression costs. The reconstruction cost R(B) is also considered. k The difficulty of decoding and reconstructing blocks is mainly measured by considering the visual importance of data within a block, the correlation between data within a block and adjacent blocks, and the size of the block, such as texture, edges, cross-block prediction, and the higher the reconstruction cost for smaller blocks.

[0099] For example, C(B) k ) and R(B k This can be modeled using a weighted summation method, i.e.:

[0100] C(B k ) = w c1 ·f1(B k )+w c2 ·f2(B k )+…

[0101] R(B k ) = w r1 ·g1(B k )+w r2 ·g2(B k )+…

[0102] Among them, f (·) and g (·) As a cost feature, w ci and w ri For the corresponding weights.

[0103] For example, the initial partitioning scheme P0 may not be optimal, therefore the partition block boundaries need to be adjusted to reduce the total cost. Let P denote the current partitioning scheme, and C(P) and R(P) be the compression cost and reconstruction cost, respectively. The optimization objective is:

[0104] minC(P), R(R)≤R0

[0105] Where, min C(P) represents minimizing the compression cost C(P) of the current partitioning scheme P, R(P) represents the reconstruction cost of the current partitioning scheme P, and R0 is the reconstruction cost constraint. A heuristic strategy is used to iteratively optimize the partitioning scheme: B... k It attempts to merge with its adjacent blocks to obtain a new block B' k ; Calculate the new block B` k and the remaining P\B k The costs and resource consumption. If the total cost after the merger is C(B'), then... k )+C(P\B k The cost of the current solution is less than C(P) and the total resource consumption is R(B). k)+R(P\B k If )≤R0, then update the current solution: P=(P\B k )∪B` k .

[0106] For example, to facilitate subsequent encoding, the optimized partitioning scheme P is organized into a quadtree structure. Each leaf node corresponds to a partition block, and non-leaf nodes represent the merging of sub-blocks. At this point, the adaptive partitioning optimization module outputs the optimal partitioning scheme P and its hierarchical representation H(P), preparing for subsequent encoding.

[0107] Step 103: Based on the optimized differentiation scheme of the target device, design the optimal set of fractal transformation functions for each partition to obtain the fractal transformation set.

[0108] For example, for partition block B k We start from the family of functions F = {f1, f2, ..., f...} M Select the optimal transformation function f in} k To approximate B k Among them, each candidate function f m (m = 1, 2, ..., M) is defined by the following parameters: geometric transformation parameters, including the rotation angle θ. m Geometric scale factor s m and offset vector o m Brightness transformation parameters, including the brightness scale factor a m and brightness offset b m .

[0109] Specifically, for any pixel x within the partition block, its transformed pixel is:

[0110] f m (x)=a m ·[s m ·R(θ m )·x+o m ]+b m

[0111] Where R(θ) m f is the rotation matrix. m (x) represents the transformed pixel. The function family F can be generated by traversing the discrete parameter space, for example: θ m ∈{0°, 90°, 180°, 270°}; s m ∈{0.5, 1, 2}; o m ∈{(0,0),(0,1),(1,0),(1,1)};a m ∈{0.5, 1, 1.5}; b m ∈{-255, -127, 0, 127, 255}.

[0112] For example, for each candidate function f m Evaluate its encoding B k The efficiency. First, calculate f. m (B k ) and B k Mean squared error (MSE): Calculate f m Encoding length This includes: the encoding length of the transform parameters, the residual image (B... k -f m (B k The encoding length of f is determined; finally, the function with the lowest rate-distortion cost is selected as the optimal transform f. k ,Right now: Where λ∈[0,1] is the balance factor, used to control distortion (through mean square error). (measurement) and length The weights; in obtaining each partition block B k The optimal transformation function f k Then, the entire transformation function set {f1, f2, ..., f} is further optimized. k}

[0113] Specifically, consider the consistency of transforms between adjacent blocks to reduce code rate. For example, if the transform parameters of two adjacent blocks are very similar, they can be merged into a larger block to reduce coding overhead. Let N(B k ) represents B k Given a set of spatially adjacent blocks, the optimization objective is:

[0114]

[0115] Where ∑ k MSE k The mean square error (MSE) of all blocks k k The sum; μ is the consistency weight, ||f k -f j ||Measure the difference between two transformation functions (e.g., the L1 norm of the parameter difference). Iteratively optimize the function set through a local search strategy, for example: for each B k , evaluate N(B) k All B in ) j (B k (neighboring elements), the original block B k and to B k The newly formed block (B) k ∪B j The objective function value f(B) corresponding to the new block k ∪B j (smaller than the original block B)k The objective function value f(B) k If the objective function value f is recalculated, then the new objective function value f is recalculated. k At this time, f(B) k The objective function value f of the new block is equal to the objective function value f. k This process iterates over each element and its neighbors until the objective function value can no longer be reduced by merging.

[0116] To further compress the transformation function parameters, the parameters are quantized and encoded for the rotation angle θ. k Quantized to 2 bits in {0°, 90°, 180°, 270°}; for scale factor s k and a k Quantize it into an 8-bit fixed-point number; for the offset vector o k Quantize it into an 8-bit integer; for offset b k 128-level uniform quantization is used. After quantization, each transformation function f k It can be represented using 32 bits (without considering redundancy between parameters).

[0117] For example, up to this point, the fractal transformation encoding module outputs K transformation functions {f1, f2, ..., f...} k This achieves a compact representation of the partitioned blocks. Simultaneously, the transformed residual images (B1-B1-f1(B1), B1-f2(B2)) are also obtained. In practical applications, the computational complexity of fractal coding mainly lies in the search for transformation parameters and similarity measurement. To reduce complexity, the following optimization strategies can be considered: Simplify the search for transformation parameters: When searching for the optimal Iterated Function System (IFS) parameters, hierarchical search, heuristic search, and other strategies can be adopted to avoid exhaustively enumerating all possible parameter combinations; Optimize the similarity measurement: When calculating the similarity between fractal blocks, fast similarity estimation algorithms, such as hash-based and feature-based methods, can be used to reduce unnecessary redundant calculations.

[0118] Step 104: Based on the fractal transformation set of the target device, perform multi-level encoding on the residual after fractal transformation to obtain a hierarchical residual code stream.

[0119] For example, calculate each partition block B k The residual signal e k :e k =B k -f k (B k ), where f k For B k The optimal fractal transformation function; further, for the residual signal e kPerform multi-scale decomposition. Commonly used decomposition tools include wavelet transform and Laplace pyramid.

[0120] For example, taking wavelet transform as an example, for e k Perform L-level wavelet decomposition: in Let be the wavelet coefficients of the l-th layer, and DWT(·) denote the discrete wavelet transform.

[0121] For example, for each layer of wavelet coefficients Different coding strategies are used for high-frequency wavelet coefficients. A sparse coding method based on compression sensing is adopted, that is: the high-frequency sparse coefficient vector is... It is represented by its wavelet dictionary matrix D (lH) Sparse linear combinations of the following:

[0122] in For high-frequency sparse coefficients, for The L0 norm (number of non-zero elements) of the wavelet coefficients is given by T, which is a sparsity constraint limiting the maximum number of non-zero elements in the sparse coefficients. The wavelet dictionary matrix D is also given. (lH) Dictionary learning algorithms such as Singular Value Decomposition (K-SVD) can be used to obtain wavelet coefficients from training data to adapt to the statistical characteristics of natural images. For low-frequency wavelet coefficients... Adaptive arithmetic coding (AAC) is used to directly convert the low-frequency coefficient vector The encoding result is low-frequency encoding c (lG) For high-frequency sparse coefficient vectors An entropy coding method combining run-length encoding and Huffman coding is used to obtain the high-frequency coding result c. (lH) Unlike traditional arithmetic coding, AAC adaptively adjusts the probability model based on the encoded coefficients to better match local statistical characteristics. During the coding process, the coding bit rate needs to be reasonably allocated across different partitions and wavelet scales, allowing R to... tot To determine the total coding bit rate, the bit rate for each layer is allocated based on Lagrange optimization: {R (1) R (2) , ..., R (L)}=argmin D tot +λ·(R (1) +R (2) +...+R (L) -R (tot) ), where D tot For total distortion, R (l)For the corresponding code rate allocation of layer l, λ is the Lagrange multiplier, and the total distortion can be expressed as the weighted sum of the distortions of each layer: D tot =∑w l ·D l D l For the reconstruction distortion of the coefficients in the l-th layer, w l For scale weights.

[0123] For example, after determining the bitrate constraints for each layer, the bitrate is further allocated among different partition blocks within the same layer. A weighted allocation strategy can be designed using factors such as the visual saliency and texture complexity of the blocks; finally, the hierarchical residual coding module outputs the hierarchically encoded residual data stream: {(c (1G) ,c (1H) ,R (1) ),(c (2G) ,c (2H) ,R (2) ),…,(c (LG) ,c (LH) ,R (L) ), where each tuple (c (lG) ,c (lH) ,R (l) The low-frequency coding result c contains the residual data of layer l. (lG) High-frequency coding result c (lH) and the corresponding bitrate allocation R (l) .

[0124] Step 105: Based on the hierarchical residual bitstream of the target device, perform efficient entropy coding using the context information of the data to obtain a compressed data stream.

[0125] For example, a multi-scale context model is constructed based on the statistical characteristics of the residual data stream. For the l-th layer residual data c... (l) Extract its contextual features x (l) Based on contextual features, a conditional probability model p(c) is established. (l) |x (l) ), describing the residual data c based on the l-th layer. (l) The derived contextual features x of the l-th layer (l) Statistical distribution.

[0126] It should be noted that during the encoding stage, the conditional probability of the residual coefficients to be encoded is estimated based on the actual extracted context features. For the nth coefficient to be encoded... Extract its quantization context And find the corresponding probability model parameters Further, calculate the conditional probability estimate:

[0127] Where f(·) is the parameterized form of the probabilistic model. For example, for a Gaussian mixture model: Where w i For mixed weights, N(·) is the Gaussian distribution function.

[0128] Furthermore, based on the estimated conditional probabilities, arithmetic coding is used to entropy-encode the residual coefficients. Let the conditional probabilities... represent The probability estimate is obtained, and arithmetic coding is performed by scaling the probability interval. The updated coding interval is: The initial interval is [0, 1]. The interval length decreases as each coefficient is encoded. To avoid interval underflow, an adaptive strategy is used to dynamically adjust the interval. Specifically, based on the current interval length, a corresponding finite precision (such as 32-bit or 64-bit) representation is selected, and the interval is enlarged as necessary.

[0129] In addition, a context-based probabilistic update mechanism is adopted, that is: after encoding, the coefficients to be encoded are updated using the actual encoded symbols. Update its context Corresponding probabilistic model parameters During decoding, the probability model parameters are updated synchronously to ensure encoding and decoding consistency. Finally, the output of the arithmetic encoding is organized layer by layer into the final bitstream: {b (l) b (2) , ..., b (L)}, where b (l) This is the encoded bitstream for layer l. Simultaneously, statistical information from the residual data of each layer, such as probabilistic model parameters, needs to be encoded as prior knowledge required for decoding. This statistical information can be processed using fixed-length encoding or explicit transmission of model parameters.

[0130] Step 106: Based on the compressed data stream of the target device, perform progressive decoding and quality-controlled reconstruction of the data to obtain reconstructed data.

[0131] For example, the encoding information and statistical information of the residual data at each layer are parsed from the compressed data stream. The encoding information includes: fractal encoding parameters: {f k |k=1,2,...,K}, that is, the transformation function of each partition block; entropy-coded bitstream: {b (l) b (2) , ..., b (L)}, which is the arithmetic encoding output of the residual data at each layer. Statistical information includes: context model parameters: θ (l)This includes the conditional probability model parameters of the residual data at each layer; and other auxiliary information such as block information, wavelet basis selection, and quantization step size. Based on the predefined syntax and data format of the coding sequence (CDS), this information is correctly extracted and interpreted.

[0132] For example, using fractal coding parameters, the approximate data of the base layer is first reconstructed. For each original partition block B k Apply its transformation function f k Iterative generation of approximate blocks: in Let `const` be any initial image patch. As the number of iterations increases... It will converge quickly to f k The fixed point, and the original partition block B k High approximation. By concatenating all the reconstructed partition blocks, we obtain the approximate base layer data X. (0) Building upon the base layer, the residual data is decoded layer by layer, and used to correct the reconstruction results of each layer to obtain a higher quality approximation. For the residual data c of the l-th layer... (l) Reconstructing the contextual probability model p(c) using side information (l) |x (l) ), using an arithmetic decoder combined with p (l) Decompress b (l) Quantized residual data were obtained. Furthermore, on Perform inverse quantization and inverse transform to obtain the residual reconstruction decoding result of the l-th layer. Specifically, for the high-frequency residual layer (lH = 1, 2, ..., L-1), for Perform dequantization to obtain the reconstructed sparse coefficients. Then, using the dictionary D of the l-th level... (lH) right Perform sparse reconstruction to obtain For low-frequency residual layers (lG=L), directly to Perform inverse quantization to obtain Formally, this process can be represented as:

[0133]

[0134] Furthermore, the decoding results Compared with the upper-level reconstruction result X (l-1) By superimposing the layers, we obtain the corrected reconstruction of the l-th layer: in for The inverse transform, such as the wavelet inverse transform or the Laplace inverse transform, is used. This process is repeated layer by layer, decoding and correcting until the preset number of reconstruction layers, l, is reached, yielding the final reconstruction result X. (l) .

[0135] Furthermore, to meet the needs of different application scenarios, a quality assessment and control mechanism is introduced. Quality assessment aims to quantify the degree of distortion in the reconstruction results. Commonly used distortion measures include: Mean Square Error (MSE), Structural Similarity (SSIM), and Visual Information Fidelity (VIF).

[0136] Based on the reconstruction result X (l) Calculate the corresponding distortion measure D with the original data X. l and with the preset quality threshold Q th Comparison. When D l ≥Q th When the time is reached, stop the decoding process and output the current reconstruction result.

[0137] Furthermore, the reconstruction quality and decoding complexity can be adjusted by controlling the number of decoding layers, l. A higher number of layers results in better reconstruction quality, but also greater computational overhead. The optimal value of l can be adaptively selected based on application requirements, such as energy budget and latency constraints. The hierarchical decoding and quality reconstruction module outputs a reconstruction result X that meets the quality requirements. l This achieves efficient and controllable restoration from compressed bitstreams to raw data. In practical applications, the time and space complexity of arithmetic coding mainly depends on the storage and updating of the context model. To reduce complexity, the following optimization strategies can be considered: Optimize context model storage: When designing the context model, compact data structures such as sparse representation and hash indexes can be used to reduce memory usage; Simplify interval updates: During the interval update process of arithmetic coding, approximate calculations and table lookups can be used to avoid unnecessary high-precision calculations.

[0138] Step 107: Based on the reconstruction data of the target device and the overall system operation data, analyze the system performance and continuously optimize each module to obtain the final optimization strategy.

[0139] For example, a set of performance metrics is designed to comprehensively evaluate the system's encoding and decoding performance. These mainly include: Compression ratio: the ratio of compressed data to original data, measuring the data compression effect; Reconstruction quality: objective performance metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), measuring the degree of distortion in the reconstruction result; Encoding efficiency: the time or computational resources required to complete the encoding task, measuring the complexity of the encoding process; Decoding efficiency: the time or computational resources required to complete the decoding task, measuring the complexity of the decoding process; Memory usage: the storage space required during encoding and decoding, measuring the system's memory overhead. A performance monitoring module is embedded in the codec to collect and record the above metrics in real time. Simultaneously, the metric data is uploaded to the cloud for long-term statistical analysis.

[0140] For example, based on the collected performance data, system bottlenecks are diagnosed, and key points for performance optimization are identified. Specifically, we analyze performance bottlenecks from the following aspects: Compression ratio bottleneck: Does the compression ratio meet application requirements, and is there room for improvement? Quality bottleneck: Does the reconstruction quality meet application needs, and is there room for improvement? Encoding efficiency bottleneck: Does the encoding speed meet real-time requirements, and is there potential for acceleration? Decoding efficiency bottleneck: Does the decoding speed meet real-time requirements, and is there room for acceleration? Memory bottleneck: Does memory usage exceed device limits, and is there space for saving memory? For these bottlenecks, statistical inference, machine learning, and other data mining techniques are used to analyze the correlation between performance indicators and encoding / decoding parameters, and to discover key influencing factors for performance optimization.

[0141] Furthermore, after identifying performance bottlenecks, corresponding strategies are adopted to dynamically adjust encoding and decoding parameters. For compression ratio bottlenecks, we can improve compression performance through the following means: optimizing fractal coding transformation functions, such as increasing function parameters and expanding function classes; optimizing residual coding strategies, such as adjusting wavelet basis selection and quantization step size; and optimizing entropy coding models, such as increasing context dimension and updating probability estimation methods. For quality bottlenecks, we can improve reconstruction results through the following means: improving fractal coding accuracy, such as reducing quantization step size and increasing the number of iterations; optimizing residual data correction, such as using higher-order approximation methods and edge-sensitive filtering; and improving quality evaluation models, such as introducing visual attention mechanisms and local quality perception. For encoding and decoding efficiency bottlenecks, we can take the following acceleration measures: reducing fractal coding complexity, such as simplifying transformation parameter search and optimizing similarity metrics; reducing arithmetic coding overhead, such as optimizing context model storage and simplifying interval updates; and utilizing hardware acceleration, such as Single Instruction Multiple Data (SIMD) instruction optimization and GPU parallel computing. To address memory bottlenecks, we can control storage overhead using the following methods: compress fractal coding parameters, such as parameter quantization and Huffman coding; reduce residual data caching, such as streaming processing and block partitioning strategies; and optimize probabilistic model storage, such as sparse representation and parameter pruning.

[0142] The above optimization strategies are encapsulated into a parameter adaptive adjustment algorithm, which optimizes the encoding and decoding parameters in real time based on performance feedback. Through iterative optimization, the system performance is continuously improved and converges to the optimal state under application constraints.

[0143] In the long run, simple parameter tuning is insufficient to address changes in data and application requirements. To pursue continuous performance improvement, an online algorithm update mechanism is introduced. Specifically, the following strategies are used to dynamically update the encoding and decoding algorithms: fractal coding algorithm: adaptively adjusting the transformation function form and similarity metric through autonomous learning; residual coding algorithm: dynamically optimizing wavelet basis selection and dictionary learning through incremental learning; entropy coding algorithm: adaptively selecting the optimal context model and probability estimation method through reinforcement learning; quality assessment algorithm: updating the visual perception model and attention prediction model online through transfer learning. Unsupervised learning paradigms such as self-supervised learning and active learning are comprehensively utilized to adaptively learn and improve the encoding and decoding algorithms from the data stream. Simultaneously, a cloud-edge collaborative optimization framework is integrated to continuously learn and optimize the codec in the cloud and periodically synchronize updates to terminal devices. The self-optimization feedback loop module outputs optimization strategies based on long-term data accumulation and algorithm evolution, achieving continuous self-optimization of the encoding and decoding system.

[0144] like Figure 2As shown, the fractal data compression and storage system of the present invention includes the following modules: a multi-scale self-similarity analysis module (S201), an adaptive partitioning optimization module (S202), a fractal transformation encoding module (S203), a hierarchical residual encoding module (S204), a context adaptive entropy encoding module (S205), a hierarchical decoding and quality reconstruction module (S206), and a self-optimizing feedback loop module (S207).

[0145] The data flow between modules is as follows: Raw device data is first input into the multi-scale self-similarity analysis module to extract self-similarity features; then, the adaptive partitioning optimization module generates the optimal data partitions based on the self-similarity features; the fractal transformation encoding module performs transformation encoding on each partition; the hierarchical residual encoding module performs hierarchical encoding on the transformed residual signal; the context adaptive entropy encoding module performs entropy encoding and outputs a compressed data stream; the hierarchical decoding and quality reconstruction module reads the compressed data and decodes and reconstructs it; finally, the self-optimization feedback loop module analyzes the system performance, optimizes the parameters of each module, and outputs optimization strategies.

[0146] like Figure 3 As shown, the Internet of Things (IoT) device 300 mainly consists of a processor 301 and a memory 302, which are connected by a bus 303, which can be an I2C (inter-integrated circuit) bus, etc.

[0147] Specifically, processor 301 performs computing and control functions and is the core of Internet of Things (IoT) device operation. Processor 301 can be a low-power IoT microcontroller chip, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor here can be a microprocessor or any conventional processor.

[0148] The memory 302 can take various forms, such as Flash chip, read-only memory (ROM), disk, optical disk, USB flash drive or portable hard drive.

[0149] It should be noted that, Figure 3 The structures shown are merely schematic diagrams relating to embodiments of the present invention and do not constitute a limitation on Internet of Things (IoT) devices that apply this solution. Actual IoT devices may contain more or fewer components, or combine some components, or use different component arrangements.

[0150] It is worth noting that the main function of the processor 301 is to execute computer programs stored in the memory 302. When running these programs, the processor is able to implement the fractal data compression and storage method proposed in this embodiment of the invention.

[0151] In one specific embodiment of the present invention, the processor is designed to execute a computer program stored in memory. When the program runs, the processor will perform the following operational steps:

[0152] a. Perform multi-level analysis on the input data to identify self-similar structures at different scales and generate self-similar descriptors.

[0153] b. Based on self-similarity descriptors, the data is dynamically divided into optimal compression units to obtain an optimized differentiation scheme.

[0154] c. Based on the optimized differentiation scheme of the target device, design the optimal set of fractal transformation functions for each partition to obtain the fractal transformation set.

[0155] d. Based on the fractal transformation set of the target device, perform multi-level encoding on the residual after fractal transformation to obtain a hierarchical residual code stream.

[0156] e. Based on the hierarchical residual bitstream of the target device, perform efficient entropy coding using the context information of the data to obtain a compressed data stream.

[0157] f. Based on the compressed data stream of the target device, perform progressive decoding and quality-controlled reconstruction of the data to obtain reconstructed data.

[0158] g. Based on the reconstructed data of the target device and the overall system operation data, analyze the system performance and continuously optimize each module to obtain the final optimization strategy.

[0159] This invention also provides a computer-readable storage medium storing one or more programs. These programs can be executed by one or more processors to implement the steps of any of the fractal data compression storage methods described in this invention. The storage medium can be an internal storage unit of the aforementioned Internet of Things (IoT) device, such as a hard disk or memory. It can also be an external storage device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD card), flash memory card, etc. Those skilled in the art should understand that the steps in the above methods, and the functional modules in the system or apparatus, can be implemented by software, firmware, hardware, or a combination thereof. In hardware implementation, the division of functional modules does not necessarily correspond to the division of physical components. A physical component may perform multiple functions, or a function may be completed collaboratively by multiple physical components. These components can be software executed by a processor (such as a CPU, DSP, or microprocessor), or they can be hardware implementations, or integrated circuits (such as application-specific integrated circuits, ASICs). This software can be distributed on a computer-readable medium, including computer storage media (non-transitory) and communication media (transitory). Computer storage media include, but are not limited to, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory, CD-ROM, DVD, magnetic tape, and hard disks, used for storing information. Communication media include any medium that transmits computer-readable instructions, data structures, etc., in modulated data signals.

Claims

1. A fractal data compression storage method, characterized by, The method comprises the following steps: Step 101: multi-level analysis of input data to identify self-similar structures at different scales, and generate self-similarity descriptors, including: a. divide the input data X into n = 2s-1 sub-blocks i = 1, 2,..., n, where s is a division scale, s = 1, 2,..., S; b. for each sub-block extract mean standard deviation and gradient histogram constitute feature matrix F (s) ; c. Compute F (s) Neutron block With Euclidean distance between That is: wherein denotes the L2 norm; d. Define a similarity threshold ε, and say that two graphs G1, G2 are similar if then they are considered similar, thus constructing the self-similar graph G (s) and extracting all complete subgraphs as self-similar patterns; Step 102: based on the self-similarity descriptors, dynamically divide the data into optimal compression units to obtain an optimized partition scheme, including: a. Based on the self-similarity descriptors, generate an initial partition scheme P0 containing K+K' partition blocks, where K is the number of self-similar pattern subblocks and K' is the number of non-self-similar pattern subblocks; b. For each partition block B k evaluating its compression cost C(B k ) and reconstruction cost R(B k ), both satisfying: C(B k ) = w c1 ·f1(B k )+ w c2 ·f2(B k )+... R(B k ) = w r1 ·g1(B k )+ w r2 ·g2(B k )+ … where f (·) and g (·) are cost features, w ci and w ri are corresponding weights; c. Iteratively optimize P0 by heuristic strategy, with the optimization goal being min C(P), R(P)≤R0, and organize the optimized partition scheme P into a quadtree structure, where min C(P) represents the minimum compression cost C(P) of the current partition scheme P, R(P) represents the reconstruction cost of the current partition scheme P, and R0 is the reconstruction cost constraint; Step 103: design the best set of fractal transformation functions for each partition according to the optimized partition scheme to obtain a set of fractal transformations; wherein for any pixel x in a partition block, the transformed pixel is: f m (x) = a m · [s m · R(θ m ) · x + o m ]+ b m wherein f m (x) is the transformed pixel, θ m is the rotation angle, R(θ m ) is the rotation matrix, a m , s m , o m and b m are the luminance scale factor, the geometric scale factor, the offset vector and the luminance offset, respectively; Step 104: multi-level encoding of the fractal-transformed residual according to the set of fractal transformations to obtain a hierarchical residual code stream, the multi-level encoding process specifically including: a. Compute the residual signal e of each partition block B k k : e k = B k - f k (B k ), where f k is the best fractal transform function of B k ;​ b. the residual signal e k performing L-level wavelet decomposition on the residual signal e wherein is the lth-level wavelet coefficient, and DWT( ) denotes a discrete wavelet transform. c. For high frequency wavelet coefficients (lH= 1,2,...,L-1), the high frequency sparse coefficient vector is represented as a sparse linear combination of its wavelet dictionary matrix D (lH) : where is the high frequency sparse coefficient, is the L0 norm of and T is the sparsity constraint; d. low frequency wavelet coefficients Using adaptive arithmetic coding, the low frequency coefficient vector is encoded as a low frequency coding result c (lG) ; e. to the high frequency sparse coefficient vector The high frequency coding result c is obtained by using the entropy coding mode combining run-length coding and Huffman coding (lH) ; f. Assigning rate to each layer based on Lagrangian optimization: {R (1) ,R (2) ,...,R (L)} = argmin D tot + λ · (R (1) + R (2) +... + R (L) - R (tot) ), where D tot is the total distortion, R (l) is the rate allocation of the l-th layer, λ is the Lagrangian multiplier, and the total distortion can be expressed as a weighted sum of the layer distortion: D tot =∑w l ·D l , where D l is the reconstructed distortion of the l-th layer coefficients, and w l is the scaling weight. Step 105: efficient entropy encoding of the hierarchical residual code stream using the context information of the data to obtain a compressed data stream, the entropy encoding process specifically including: a. extract the first layer residual data c (l) the context feature x (l) , and establish a conditional probability model p(c (l) | x (l) ). b. To encode coefficients extracting its quantization context and finding the corresponding probability model parameters Computing the conditional probability estimate: where f(·) is the parametrized form of the probability model; c. based on the conditional probability represent the probability estimate value, arithmetic coding is performed on , and the coding interval is updated to d. Utilizing updating context corresponding probability model parameters implementing adaptive learning; Step 106: progressive decoding and quality-controllable reconstruction of the data according to the compressed data stream to obtain reconstructed data, the decoding and reconstruction process specifically including: a. Iteratively compute fractal coding parameters {f k |k = 1, 2,..., K}, reconstruct the base layer approximation data X (0) ; b. Decoding residual data c layer by layer (l) , obtaining quantized residual data Performing inverse quantization and inverse transform on , obtaining residual reconstruction decoding result Wherein the high-frequency sparse coefficient reconstruction is: lH=1, 2,..., L-1; and the low-frequency coefficient reconstruction is: lG=L; c. the decoding result with the upper layer reconstruction result X (l-1) superimposed, resulting in a layer-wise modified reconstruction result: d. Compute the objective distortion measure D l When D l ≥ Q th , stop decoding and output the current reconstruction X (l) where Q th is a quality threshold. Step 107: analyze system performance and continuously optimize each module based on the reconstructed data and the overall system running data to obtain the final optimization strategy.

2. The fractal data compression storage method of claim 1, wherein, In the step 103, the partition block B k The optimal fractal transform function f k is obtained by a. for a candidate function f m , compute the mean squared error m between f k and B k , denoted as E and the encoding length of f m ​ b. a function of rate-distortion cost minimization as the best transform f k : where λ ∈ [0, 1] is a balancing factor; c. Consider the consistency of neighboring partition blocks to reduce the code rate, further optimize f k The optimization goal is: Where ∑ k MSE k is the sum of the mean square error MSE k of all blocks k, μ is the consistency weight, and ||f k -f j || measures the difference between the two transform functions; d. 32-bit fixed-point quantization encoding of the optimized transformation function parameters.

3. The fractal data compression storage method of claim 1, wherein, The performance analysis and optimization process of step 107 specifically includes: a. Collect and upload objective performance indicators, including compression ratio, peak signal-to-noise ratio, structural similarity, encoding and decoding time, and memory usage; b. For compression ratio bottlenecks, optimize fractal encoding transformation functions, residual encoding strategies, and entropy encoding models; for quality bottlenecks, improve fractal encoding accuracy, optimize residual data correction, and improve quality evaluation models; for encoding and decoding efficiency bottlenecks, reduce fractal encoding complexity, reduce arithmetic encoding overhead, and utilize hardware acceleration; for memory bottlenecks, compress fractal encoding parameters, reduce residual data cache, and optimize probability model storage; c. Based on machine learning paradigms such as self-supervised learning, continuously optimize the encoding and decoding algorithm, and update it collaboratively in the cloud and at the edge.

4. A fractally data compression storage system characterized by, For implementing the method of any one of claims 1-3, comprising: a. a multi-scale self-similarity analysis module for multi-level analysis of input data to identify self-similar structures and generate self-similarity descriptors; b. an adaptive partition optimization module for dynamically dividing data based on the self-similarity descriptors to obtain an optimized partition scheme; c. a fractal transformation encoding module for designing the best set of fractal transformation functions for each partition to obtain fractal encoding results; d. a hierarchical residual coding module for multi-level coding of the fractally coded residual to obtain a hierarchical residual bitstream; e. a context adaptive entropy coding module for entropy coding of the residual data in combination with context information to obtain a compressed data stream; f. a hierarchical decoding and quality reconstruction module for decoding the compressed data and reconstructing layer by layer to obtain a reconstructed result meeting the quality requirement; g. a self-optimizing feedback loop module for analyzing system performance, adjusting parameters of each module, and outputting a dynamic optimization strategy.

5. An Internet of Things device comprising a processor and a memory, characterized in that, The processor is configured to execute a computer program stored in the memory, and when the program runs, the processor executes the fractal data compression storage method of any one of claims 1-3.

Images