A data compression method and system based on neural networks

Abstract

This invention belongs to the field of computer data compression technology, specifically relating to a data compression method and system based on neural networks. This method achieves efficient and lossless compression of various types of file data by constructing a complete closed-loop compression process covering data reading, feature modeling, expert selection, and encoding output. Specifically, this invention utilizes byte-level embedding mapping to unify the input format, captures short-range and long-range contextual features through a selective state-space model, and then combines a sparse expert hybrid mechanism to achieve dynamic feature selection and adaptive parameter expansion. Finally, entropy encoding is used to convert the predicted byte probability distribution into a compact bitstream, thereby completing data compression and lossless restoration, significantly improving compression accuracy and model versatility.

Description

Technical Field

[0001] This invention relates to the field of computer data compression, and in particular to a data compression method and system based on neural networks. Background Technology

[0002] In today's data-driven world, the volume of digital information is growing at an unprecedented rate. With such massive data volumes, even a 1% improvement in compression rate can result in significant cost savings and energy reductions in storage, transmission, and archiving systems operating at petabyte (PB) or exabyte (EB) levels. At the same time, data center and cloud service providers face increasingly severe challenges, including storage hardware expansion, bandwidth bottlenecks, and rising operating costs. In many mission-critical applications, such as database backup, medical imaging, financial transaction logs, and scientific data archiving, lossless compression is essential because even the smallest loss of information can have serious consequences.

[0003] Traditional lossless compression algorithms, such as Gzip, 7z, and Zstd, have been widely used for various data types.

[0004] Gzip employs the DEFLATE algorithm, which combines LZ77 dictionary compression and Huffman coding. LZ77 uses a sliding window technique to find repeating byte sequences, while Huffman coding is used to entropy encode the remaining data, thus achieving compression. Gzip's advantages lie in its simplicity and speed, but due to limitations in dictionary and window size, its compression ratio is relatively low, especially when processing large-scale or highly redundant data, where its performance is inferior to other methods.

[0005] 7z uses the LZMA (Lempel-Ziv-Markov Chain Algorithm), which combines large dictionary compression with a probabilistic model based on Markov chains to further improve the compression ratio. LZMA uses a larger dictionary and various entropy coding techniques (such as Range coding), making its compression ratio significantly higher than Gzip. However, this also requires more memory and computing resources, resulting in slower compression speed, especially when processing large files, where the performance bottleneck is more obvious.

[0006] Zstandard employs a dictionary-based compression and Huffman coding approach, combined with block-level parallel compression technology, supporting dynamic adjustment of compression ratio and speed. During its design, Zstandard optimized algorithms to maintain fast compression and decompression speeds even at high compression ratios, and its multi-threading support ensures superior performance in modern hardware environments. While Zstandard offers better compression efficiency and speed, its implementation is relatively complex, requiring more meticulous parameter tuning to achieve optimal performance.

[0007] In summary, existing methods typically rely on hand-designed statistical models or limited contextual assumptions, resulting in limited performance when dealing with complex and multimodal data. To address this, some systems attempt to combine multiple dedicated compressors to process different types of data; however, this approach not only suffers from scalability issues but also introduces data privacy risks, as its performance depends on in-depth analysis of data characteristics. Existing methods exhibit significant shortcomings in compression limits, scalability, and privacy protection, making it difficult to meet the future demands for large-scale data storage, transmission, and processing. Summary of the Invention

[0008] The purpose of this invention is to provide a data compression method based on neural networks. By constructing a complete closed-loop compression process covering data reading, feature modeling, expert selection, and encoding output, this method achieves efficient and lossless compression of various types of file data, thereby solving at least one of the aforementioned problems in the prior art.

[0009] This invention discloses a data compression method based on neural networks, the method comprising:

[0010] Step S1: Read the file to be compressed in the form of a byte sequence, map each byte to a continuous dense vector through an embedding operation to obtain an embedding vector, and use rotation position encoding to capture the relative position information of the embedding vector;

[0011] Step S2: Using a selective state-space model, feature information is extracted from the embedding vector obtained by mapping the byte sequence, capturing short-range and long-range patterns in the input byte sequence to achieve scalable and highly expressive context modeling;

[0012] Step S3: Perform expert selection through a gating mechanism, calculate the score of each feature for each expert, and assign each feature to a few sparse experts with the highest scores for processing;

[0013] Step S4: Perform feature fusion operation through gating mechanism, and perform weighted summation of the results obtained after each feature information is processed by several selected sparse experts to obtain the overall feature fusion result, and generate a prediction of the probability distribution of subsequent bytes based on the overall feature fusion result;

[0014] Step S5: Obtain the byte sequence to be compressed, and use the entropy encoder to compress the byte sequence to be compressed into a compact bit stream representation based on the prediction of the probability distribution of subsequent bytes obtained from the model, thereby achieving lossless compression of data of any file type.

[0015] The present invention also provides a neural network-based data compression system, the system being used to perform the method described above, the system comprising:

[0016] The first module is used to read the file to be compressed in the form of a byte sequence, map each byte to a continuous dense vector through an embedding operation to obtain an embedding vector, and use rotation position encoding to capture the relative position information of the embedding vector;

[0017] The second module is used to extract feature information from the embedding vector obtained by mapping the byte sequence using a selective state-space model, capture short-range and long-range patterns in the input byte sequence, and realize scalable and highly expressive context modeling.

[0018] The third module is used to: perform expert selection operations through a gating mechanism, calculate the score of each feature information for each expert, and assign each feature information to a few sparse experts with the highest scores for processing;

[0019] The fourth module is used to: perform feature fusion operation through a gating mechanism, perform weighted summation of the results obtained after each feature information is processed by several selected sparse experts to obtain the overall feature fusion result, and generate a prediction of the probability distribution of subsequent bytes based on the overall feature fusion result;

[0020] The fifth module is used to obtain the byte sequence to be compressed, and use the entropy encoder to compress the byte sequence to be compressed into a compact bit stream representation based on the prediction of the probability distribution of subsequent bytes obtained from the model, thereby achieving lossless compression of data of any file type.

[0021] Compared with the prior art, the present invention has at least one of the following technical effects:

[0022] (1) Input unification and generalization enhancement: By using byte-level embedding mapping and rotation position encoding mechanism, the original byte sequences of different types of files are unified into learnable continuous feature representations, which significantly improves the model's cross-format generalization ability and input consistency;

[0023] (2) Context capture and efficient modeling: The selective state space model is used to fuse convolution and state recursion structure, which can capture short-range and long-range dependencies at the same time, realize efficient modeling of complex data distribution, and improve information compression rate and expressive ability compared with traditional sequence model;

[0024] (3) Sparse expert hybridization and parameter adaptive expansion: Dynamic expert selection and sparse routing are achieved through gating mechanism, so that the model can achieve a balance between computational efficiency and expressive power, while supporting expert function specialization and parameter scale expansion, and improving adaptability to heterogeneous data.

[0025] (4) High-precision entropy coding and lossless restoration: Based on the probability distribution prediction entropy coding strategy, it effectively approaches the theoretical optimal compression limit and achieves a higher compression ratio while ensuring lossless restoration;

[0026] (5) Modular and scalable system deployment: The overall system includes the first to fifth modules, where the first module is used for input embedding, the second module is used for feature extraction, the third and fourth modules are used for sparse expert hybridization, and the fifth module is used for entropy coding. The system has a clear structure and standardized interfaces, which facilitates rapid deployment on general computing platforms and hardware acceleration environments.

[0027] In summary, this invention introduces a neural network-driven general lossless compression framework, forming an efficient closed loop in stages such as unified input, context modeling, expert dynamic selection, and entropy encoding output. This enables accurate lossless compression and high-fidelity restoration of various types of file data. It is a general lossless data compression method that combines high compression ratio, strong generalization ability, and scalable architecture, providing an efficient solution for intelligent compression and secure storage of various types of files. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating a data compression method based on a neural network according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the structure of a data compression system based on a neural network according to another embodiment of the present invention;

[0031] Figure 3 This is a system module architecture diagram of an embodiment of the present invention. Detailed Implementation

[0032] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0033] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0034] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0035] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0036] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0038] In view of the shortcomings of traditional lossless compression algorithms in terms of compression limits, scalability, and privacy protection, there is an urgent need to design a general-purpose lossless compressor that can achieve robust compression on diverse data types, while taking into account engineering scalability and privacy protection, in order to cope with the diverse and ever-growing data scale in the future.

[0039] This invention provides a neural network-based data compression method to achieve efficient and lossless compression of various types of file data. For example... Figure 1 As shown, the method includes:

[0040] Step S1: Read the file to be compressed in the form of a byte sequence, map each byte to a continuous dense vector through an embedding operation to obtain an embedding vector, and use rotation position encoding to capture the relative position information of the embedding vector;

[0041] Step S2: Using a selective state-space model, feature information is extracted from the embedding vector obtained by mapping the byte sequence, capturing short-range and long-range patterns in the input byte sequence to achieve scalable and highly expressive context modeling;

[0042] Step S3: Perform expert selection through a gating mechanism, calculate the score of each feature for each expert, and assign each feature to a few sparse experts with the highest scores for processing;

[0043] Step S4: Perform feature fusion operation through gating mechanism, and perform weighted summation of the results obtained after each feature information is processed by several selected sparse experts to obtain the overall feature fusion result, and generate a prediction of the probability distribution of subsequent bytes based on the overall feature fusion result;

[0044] Step S5: Obtain the byte sequence to be compressed, and use the entropy encoder to compress the byte sequence to be compressed into a compact bit stream representation based on the prediction of the probability distribution of subsequent bytes obtained from the model, thereby achieving lossless compression of data of any file type.

[0045] In the field of sequence modeling, the Transformer model is currently the mainstream. When seeking stronger expressive power, the conventional approach is to use larger and deeper Transformer models. However, this application's embodiment, in addressing the problem of improving general lossless compression performance, breaks through existing path dependencies by employing a selective state-space model as the core feature extractor and deeply integrating it with a sparse expert hybrid module. The selective state-space model is responsible for efficiently processing long sequences and filtering information, solving efficiency and long-range dependency issues. The sparse expert module is responsible for cost-effectively expanding the model capacity based on the high-quality features extracted by the selective state-space model to achieve multi-functional adaptation, solving the problems of versatility and expressive power. The combination of these two approaches forms a unified architecture with clear functional division and synergistic amplification of effects.

[0046] Furthermore, this embodiment constructs an end-to-end neural network system that directly maps the original byte sequence to the compressed bitstream. Utilizing a selective state-space model and a sparse expert hybrid module, it dynamically and deeply "understands" the intrinsic structure and semantic context of the data, thereby generating extremely accurate byte-level probability distribution predictions. Traditional compressors (such as Gzip / LZ77) are based on shallow statistical models that find and replace duplicate strings. Even Zstd's model is hand-designed. This embodiment achieves a fundamental paradigm shift: the compression process is the process by which the model understands the data and predicts the future.

[0047] To achieve versatility, the conventional approach is to combine multiple dedicated compressors, requiring external, explicit file type analysis steps. This application's embodiment employs byte-level embedding and rotational position encoding, providing a unified, format-independent input interface for all data types. The entire system does not perform any pre-processing file type analysis; instead, it implicitly learns data type characteristics during numerical computation, rather than outputting a readable label. Therefore, privacy can be fundamentally protected by eliminating external analysis steps.

[0048] In some embodiments, step S1 specifically includes:

[0049] Determine the embedding dimension q for each byte and the hidden layer dimension h of the model;

[0050] The file to be compressed is read as a byte sequence to obtain a byte stream sequence S, where each byte... A learnable embedding matrix is ​​mapped to a low-dimensional embedding vector. , This represents the i-th byte in S;

[0051] The low-dimensional embedding vector The vectors are concatenated to form a high-dimensional embedding vector. ,in, It is an integer factor of h.

[0052] For example, it can be adopted as follows Figure 3 The top-left input embedding module, i.e., the input embedding layer, performs the embedding operation. For example, the byte sequence of the file to be compressed is S=[S1,S2,…,S…]. T ], where S i ∈[0,255] represents the i-th byte, and T is the sequence length. The value is typically 256 or 512, but each byte has a sign. The range of values ​​is That is, there are only 256 states, which are directly mapped to Multidimensional vectors can lead to redundancy in the target space representation. Therefore, it is necessary to consider mapping each byte to a smaller embedding dimension first. ,satisfy ,Right now Can be Divisible by.

[0053] Then, a vectorized mapping of byte symbols is performed to read the input file byte by byte sequence, with each byte symbol... Mapped to a low-dimensional vector through a learnable embedding matrix. This operation transforms discrete byte symbols into learnable continuous feature representations, avoiding task-dependent word segmentation and improving generalization ability across different file types.

[0054] This embodiment reads the input file byte by byte, replacing the various word segmentation / blocking rules in different fields. This avoids task-dependent word segmentation processing. For any type of file, the symbol table is a 256-bit symbol table because a byte can only have 256 states. By transforming discrete byte symbols into learnable continuous feature representations, bytes in similar contexts are grouped closer together in the vector space. Combined with positional encoding and subsequent sequence modeling, the model can learn commonalities independent of file type, such as repeated segments.

[0055] The specific representation of the embedding matrix is ​​as follows:

[0056] ;

[0057] By using table lookup operations or matrix multiplication, each byte symbol can be... Mapped to a low-dimensional vector :

[0058] .

[0059] In some embodiments, capturing the relative position information of the embedded vector using rotational position encoding includes:

[0060] Introducing a position-dependent rotation matrix into the embedding space, specifically represented as:

[0061] ;

[0062] in Represents the embedding vector The result after rotational position encoding Represents the rotation matrix. Representing rotation matrix and Dimensions express Location information;

[0063] The rotation matrix It consists of a diagonal matrix composed of several two-dimensional rotation submatrices, specifically represented as:

[0064] ;

[0065] Each submatrix and frequency They are defined as follows:

[0066] ;

[0067] .

[0068] This design enables phase rotation at different frequencies across different dimensions, thereby achieving multi-scale position information encoding. Position encoding is applied to each... This embodiment enables the model to capture relative positional information with lower complexity and has better generalization ability when processing long sequences.

[0069] In some embodiments, the low-dimensional embedding vector The vectors are concatenated to form a high-dimensional embedding vector. ,include:

[0070] Use a sliding window to extract subsequences from the low-dimensional embedding vector sequence encoded by rotational position;

[0071] The extracted subsequences are concatenated to form a high-dimensional embedding vector. .

[0072] This embodiment performs structured splicing with low-dimensional embedding. For example, a sliding window is used to extract subsequences from the position-encoded vector sequence. Let the length of the sliding window be... The step size is 1. It is understood that the step size here can also be set to other data. In this embodiment, the explanation is based on a step size of 1. For each window's starting position i (from 1 to...), ... ), retrieve from the window vectors And concatenate them into a high-dimensional vector:

[0073] ;

[0074] in ,Right now Typically the hidden layer dimension One factor. Optionally, the length of the output sequence. This structured concatenation operation aligns the input feature dimensions with the model's hidden dimensions, maintaining the compactness of the input features while aligning the internal dimensions of the model, thus facilitating computation and information transfer in subsequent network layers.

[0075] In some embodiments, step S2 specifically includes:

[0076] S201, after being processed by the linear expansion layer, the input is mapped to twice the internal dimension and split into two signals, one for backbone computation and the other for generating the gated modulation signal;

[0077] S202, one signal used for backbone computation is used to extract short-range temporal features through one-dimensional depthwise separable convolution, and the convolution output is activated by SiLU to obtain nonlinearly enhanced intermediate features;

[0078] S203 utilizes selective state-space modeling to capture long-range dependencies in the input sequence. State-space parameters are generated by a linear projection layer, and step-size parameters are obtained by projection through the linear layer and activation via softplus. A stable diagonal state matrix and direct-connected terms are defined to establish a state recursion relationship at discrete time step t and calculate the state output. The recursion is calculated through a parallel scanning mechanism to achieve sequence modeling.

[0079] S204 is used to generate a gated modulation signal. One of the signals is activated by SiLU to generate dynamic weights, which are then multiplied element-wise with the state output to achieve feature selection and dynamic fusion, thereby obtaining the feature information of the model output.

[0080] For example, it can be adopted as follows Figure 3 In the upper right corner, the feature extraction module is shown, i.e. Figure 3 The selective state-space modeling part of the process involves taking the input data and performing a series of linear transformations, convolutions, state-space modeling, and gating fusion before outputting the result.

[0081] Specifically, we first perform input and linear expansion operations. Let the input be... ,in For batch size, For sequence length, The input feature dimension is [value]. After processing by the linear expansion layer, the input is mapped to twice the internal dimension and split into two signals:

[0082] ;

[0083] in , The former refers to the expansion coefficients, used for backbone calculations, and the latter for generating the gated modulation signal. Here, D represents the hidden layer dimension h of the model, and D = q * j.

[0084] In this embodiment, the sequence of high-dimensional vectors obtained after processing the file data of multiple files to be compressed in step S1 is assembled. The sequence length of the high-dimensional vectors obtained after processing each file in step S1 is L, and the feature dimension is D. The batch size is determined by the number of files to be compressed at one time. To assemble, we obtain the dimension as ( Input tensors (L, D) .

[0085] Then, convolution is performed to extract local features. The main branches extract short-range temporal features using one-dimensional depthwise separable convolution, with a convolution kernel size of [size missing]. This is done to capture local dependencies. The convolutional output is then activated by SiLU to obtain non-linearly enhanced intermediate features:

[0086] .

[0087] Continue with selective state-space modeling. This stage is used to capture long-range dependencies in the input sequence. State-space parameters are generated by the linear projection layer:

[0088] ;

[0089] in , , representing the step size latent variable, the state input matrix, and the state output matrix, respectively. It is a rank parameter. For the state dimension.

[0090] The step size parameter is obtained by projecting a linear layer and then activating it via softplus:

[0091] .

[0092] Simultaneously define a stable diagonal state matrix A and directly connected terms F:

[0093] .

[0094] at discrete time step The state recursion relationship is as follows:

[0095] ;

[0096] in, The parameter representing the step size of time step t. This represents the state input matrix at time step t. Represents intermediate features at time step t;

[0097] The output is calculated as follows:

[0098] ;

[0099] in, Indicates time step The hidden state, This represents the state output matrix at time step t. This represents element-wise multiplication. Calculating the above recursion using a parallel scanning mechanism allows for efficient sequence modeling while maintaining linear complexity.

[0100] The aforementioned selective spatial state model can extract feature information from the embedding vector obtained by mapping byte sequences, capture short-range and long-range patterns in the original input byte sequences, and achieve scalable and highly expressive context modeling.

[0101] Next, gating fusion and output mapping are performed. The gating branch signals are activated by SiLU to generate dynamic weights:

[0102] ;

[0103] Then, it is multiplied element-wise with the state output to achieve feature selection and dynamic fusion:

[0104] ;

[0105] in This is the module output. This process is equivalent to learning a weight that changes with the input for each channel at each time step, and then feeding the weighted information into the output space. This allows the model to make learnable, fine-grained decisions about when to emphasize which features and when to suppress which features; thus, the model's ability to distinguish between channel importance and temporal importance is stronger. In other words, this embodiment enhances the model's ability to distinguish the importance of different channels and temporal features.

[0106] In some embodiments, the input in step S201 is obtained by mean square normalization of the original input, and then input into the selective state space model for feature transformation and extraction. The output of the selective state space model is superimposed on the original input in the form of residual connection.

[0107] This embodiment combines normalization and residual mechanisms to achieve efficient and stable sequence modeling.

[0108] enter First, it undergoes mean square normalization (RMSNorm) processing:

[0109] ;

[0110] It is a very small positive constant, used to prevent numerical instability or gradient explosion caused by a denominator of 0 or too small, and is used for numerical stabilization.

[0111] The input is then fed into a selective state-space model for feature transformation and extraction. The module output is then superimposed on the original input using a residual connection.

[0112] ;

[0113] that is .

[0114] This embodiment maintains consistent feature scales on the one hand, and uses a very small positive constant for the residual path on the other hand to prevent numerical instability or gradient explosion caused by a denominator of 0 or too small, thus performing numerical stabilization.

[0115] The gating mechanism in step S3 is as follows Figure 3 As shown in the right half of the sparse mixing expert module in the lower middle section, in some embodiments, step S3 specifically includes:

[0116] Determine the number of experts and the number of sparse experts actually activated for each feature;

[0117] The gating mechanism calculates each expert's raw score through a linear transformation;

[0118] The highest scorer is retained using Top-K operations. The score corresponding to each expert and index ;

[0119] The scores of the unselected experts were set to Then, Softmax normalization is performed to obtain a sparse weight vector.

[0120] In this embodiment, the gating mechanism requires dynamic and sparse expert selection for each feature, thereby expanding the parameter scale while maintaining inference efficiency and significantly enhancing the model's predictive ability. Each expert in the gating mechanism... It is a two-layer feedforward neural network with ReLU activation function:

[0121] ;

[0122] in It is the input feature vector, and the dimension of the output of step S2 is ( tensor of ) One of the D-dimensional vectors in the array. It is the hidden layer dimension, equivalent to D and h. The value of is from 1 to n, where n is the number of experts.

[0123] First, the number of experts needs to be determined. And the number of sparse experts actually activated for each feature. This is to improve the model's ability to process heterogeneous data without introducing excessive computational costs.

[0124] The gating mechanism then calculates the raw score for each expert through a linear transformation:

[0125] ;

[0126] It is the weight matrix of the gated linear mapping, used to map the input feature x to n expert score spaces; It is the unnormalized expert score vector calculated from the input feature x, that is, the original score of each expert before being selected.

[0127] Then, the highest-scoring sample is retained using the Top-K algorithm. The score corresponding to each expert and index :

[0128] ;

[0129] Let I be an n-dimensional vector, representing the scores of n experts. Each dimension is either 0 or 1. The dimension being 1 indicates that the first dimension is 1. The first expert is one of the m experts with the highest scores. A dimension being 0 indicates that the [number]th dimension is [value]. The expert is not one of the m experts with the highest scores.

[0130] To achieve sparsity, the scores of unselected experts are set to 0. Then, Softmax normalization is performed:

[0131] ;

[0132] At this time, it was obtained It is a sparse weight vector, with only There are n non-zero elements, and the sum of these elements is n. This indicates that the feature was selected. Each expert and their corresponding weight.

[0133] The feature fusion operation described in step S4 is as follows: Figure 3 The left half of the sparse hybrid expert module below shows its final output, which is a weighted sum of the results from these experts:

[0134] ;

[0135] in It is the first The gating mechanism employs a sparse expert gating system. During compression and decompression, all features in the input batch are processed uniformly. It calculates the routing index for each feature and distributes different features to the corresponding expert networks based on the selection results. Each expert independently processes its assigned input data, and the outputs of all experts are then weighted and aggregated according to their respective gating weights to form the final output tensor. This sparse expert gating mechanism allows the model to freely expand the number of experts without significantly increasing runtime. Each feature activates only a small number of experts, achieving a good balance between computational efficiency and prediction accuracy. Furthermore, different experts specialize for different data types during training. This mechanism significantly improves the model's generalization ability when handling heterogeneous data, which is one of the key conditions for achieving universal lossless compression.

[0136] The entropy encoder described in step S5 is as follows: Figure 3 As shown in the lower left corner, this is a lossless compression algorithm based on probability distribution. Its core idea is to utilize the statistical characteristics of symbol occurrence, encoding high-probability symbols into shorter bit sequences and low-probability symbols into longer bit sequences, thereby maximizing the approximation of the optimal compression limit in information theory. Its working principle is based on interval partitioning and probability accumulation: the entire encoding process operates within intervals... The process proceeds by mapping each input symbol to a sub-interval based on the conditional probability predicted by the model. As symbols are input, the interval is continuously narrowed and approaches a specific range, ultimately representing the entire symbol sequence with a real number (or its binary representation) within that interval. During decoding, lossless reconstruction can be achieved simply by using the same probability model to infer each symbol step in reverse. The entropy encoder employs an integer interval encoding scheme instead of a real number interval, solving problems such as insufficient floating-point precision, slower floating-point computation efficiency compared to integers, and lack of hardware support for floating-point operations as the number of symbols increases and the interval length approaches zero in the floating-point interval. This results in a high-precision, low-latency, and hardware-friendly lossless compression coding system.

[0137] The core idea of ​​this embodiment is to construct a lossless compression closed loop driven by a neural network, balancing versatility and engineering deployability. It unifies input representation through byte-level embedding and rotational position encoding, employs a selective state-space model to capture short-range and long-range context, and utilizes sparse expert hybridization to achieve dynamic feature routing and adaptive parameter expansion. Finally, high-precision entropy encoding maps the predicted byte conditional probabilities into a compact bitstream. This embodiment balances model generalization, computational and storage overhead, and privacy protection: it reduces the complexity of relative position modeling through low-dimensional structured embedding and concatenation to align hidden layer dimensions; it captures long-range dependencies while maintaining linear time complexity through selective state-space modules; and it achieves a trade-off between parameter size and inference efficiency by activating only a small number of experts through sparse gating, thereby significantly reducing deployment costs and energy consumption. Simultaneously, the modular architecture facilitates layered deployment and fine-grained optimization on general-purpose computing platforms or hardware accelerators. For the first time, byte-level embedding, rotational position encoding, selective state-space modeling, and sparse experts are combined systematically for lossless compression. This not only surpasses traditional algorithms that rely on fixed statistical assumptions in terms of compression ratio and generalization ability, but also provides a brand-new systematic solution in terms of scalability, privacy preservation, and engineering practicality.

[0138] Another embodiment of the present invention also provides a neural network-based data compression system, the system being used to perform the neural network-based data compression method as described above, such as... Figure 2 As shown, the system includes:

[0139] The first module is used to read the file to be compressed in the form of a byte sequence, map each byte to a continuous dense vector through an embedding operation to obtain an embedding vector, and use rotation position encoding to capture the relative position information of the embedding vector;

[0140] The second module is used to extract feature information from the embedding vector obtained by mapping the byte sequence using a selective state-space model, capture short-range and long-range patterns in the input byte sequence, and realize scalable and highly expressive context modeling.

[0141] The third module is used to: perform expert selection operations through a gating mechanism, calculate the score of each feature information for each expert, and assign each feature information to a few sparse experts with the highest scores for processing;

[0142] The fourth module is used to: perform feature fusion operation through a gating mechanism, perform weighted summation of the results obtained after each feature information is processed by several selected sparse experts to obtain the overall feature fusion result, and generate a prediction of the probability distribution of subsequent bytes based on the overall feature fusion result;

[0143] The fifth module is used to obtain the byte sequence to be compressed, and use the entropy encoder to compress the byte sequence to be compressed into a compact bit stream representation based on the prediction of the probability distribution of subsequent bytes obtained from the model, thereby achieving lossless compression of data of any file type.

[0144] Another embodiment of the present invention provides a data compression system based on a neural network. The system is used to perform the data compression method based on a neural network as described above. The system includes an input embedding module, a feature extraction module, a sparse expert hybrid module, and an entropy encoder module. The input embedding module is used to perform step S1, the feature extraction module is used to perform step S2, the sparse expert hybrid module is used to perform steps S3 and S4, and the entropy encoder module is used to perform step S5.

[0145] The effectiveness verification of this invention is based on the proposed general lossless data compression framework for neural networks. Systematic experimental evaluations were conducted on general lossless data compression tasks, covering multi-source datasets of both homogeneous and heterogeneous data types to verify its universality and compression performance in multimodal environments. Specifically, the experiments selected four homogeneous datasets: text (Enwik8), audio (ESC-50), images (ImageNet), and floating-point numbers (FPDouble); and also included two heterogeneous benchmark datasets: Calgary and Canterbury, which contain various file types such as source code, log files, and binary data, respectively, enabling a comprehensive examination of the algorithm's adaptability and robustness in complex data environments.

[0146] In terms of performance evaluation, compression ratio (CR) is the core metric, defined as the ratio of the original file size to the compressed file size; a higher value indicates better compression performance. The comparison method includes three traditional lossless compression algorithms—Gzip, 7z, and Zstd (all using the highest compression level).

[0147] The lossless compression framework of this invention employs the Adam optimizer with a learning rate of 0.001 and a batch size of 512 during training. Each byte is embedded as a 64-dimensional vector, and prediction is performed using a context window of length 32. The hidden and feedforward layers of the model have dimensions of 256 and 4096, respectively, achieving high-dimensional concatenation of inputs for every four tokens. The Sparse Expert Hybridization (SMoE) part contains eight experts, with only two experts activated for each input feature to achieve dynamic sparse routing. All experiments were performed on a single NVIDIA GeForce RTX 4090 (24GB) GPU.

[0148] This experimental design verifies the robustness and efficiency of the lossless compression framework proposed in this invention under various data environments, demonstrating its comprehensive advantages of high compression ratio, strong generalization ability and scalability in lossless compression tasks, and laying a technical foundation for the practical deployment of general intelligent compression systems.

[0149] Table 1 summarizes the compression ratio results on homogeneous and heterogeneous datasets. On homogeneous datasets, this invention demonstrates superior compression performance compared to all other algorithms in all tests. For example, on the Enwik8 dataset, this invention achieves a compression ratio of 4.66:1, which is 70%, 17%, and 20% higher than Gzip, 7z, and Zstd, respectively; similar trends were observed on the ESC-50 and ImageNet datasets. This invention can effectively model long-range dependencies and nonlinear perceptual structures, features that traditional methods cannot capture, fully demonstrating its advantages in processing content with specific modal structures.

[0150] Table 1 Compression ratio results on homogeneous and heterogeneous datasets

[0151]

[0152] In comparison, on heterogeneous datasets such as Calgary and Canterbury, this invention still outperforms all other algorithms. These datasets contain diverse content, making compression more challenging. However, this invention, relying on byte-level modeling and a dynamic expert routing mechanism, achieves effective generalization without depending on domain-specific heuristics. For example, on the Calgary dataset, this invention achieves a compression rate improvement of 19% compared to Gzip and 8% compared to 7z.

[0153] In summary, this invention demonstrates significant performance improvements on both homogeneous and heterogeneous datasets, fully proving the strong generalization ability and adaptability of the method under diverse data distributions.

[0154] Table 2 shows the ablation experiment results on the Enwik8 dataset (original file size is 95.37MB).

[0155]

[0156] To evaluate the contribution of the three key components (input embedding, feature extraction, and sparse expert hybrid) to the overall performance, ablation experiments were conducted on the Enwik8 dataset, removing one component at a time to observe its impact on compression. Table 2 shows the compression results under different conditions.

[0157] The performance degradation was most pronounced after removing the feature extraction component, with the compressed file size increasing by over 5MB. This indicates that this component plays a crucial role in modeling long-range dependencies and sequence structure, which are essential foundations for achieving accurate estimations in byte-level probabilistic prediction.

[0158] Removing the input embedding also significantly degrades performance because it prevents the model from learning continuous representations of bytes and their positional relationships. Without this part, the model can only process raw byte indices directly, resulting in limited expressive power and an inability to fully capture the potential relationships between bytes.

[0159] When the sparse expert hybrid module is removed, the compressed volume increases slightly, but the increase is relatively small, indicating that its main function is to enhance model capacity and functional specialization. Although it is not a core sequence modeling component, the sparse expert hybrid module helps improve the model's generalization ability and parameter utilization efficiency when dealing with diverse domains or large-scale datasets.

[0160] These results validate the importance of the three core components and demonstrate that the lossless compression architecture of this invention features modularity and collaborative design, with each component working together within the overall framework to achieve high-efficiency lossless compression performance.

[0161] This invention also demonstrates significant advantages during the deployment phase: the end-to-end closed loop consists of input embedding, selective state space, sparse experts, and entropy coding, resulting in a simple path, stable operators, and the ability for streaming inference on a single GPU, significantly reducing GPU memory and CPU usage; integer range entropy coding avoids high-cost floating-point operations, enabling efficient operation on both the client and server sides; it supports batch and block-level parallelism and can be seamlessly integrated into existing storage and transmission links without intrusive modifications, verifying its engineering feasibility and deployment efficiency.

[0162] In summary, this invention is a general lossless compression method with high compression ratio, strong generalization ability, low deployment cost, and minimal impact on system latency and throughput. Its modular architecture and sparse computing characteristics take into account privacy protection, scalability, and energy efficiency, and can operate stably in multimodal and multi-scale data scenarios, providing an efficient, reliable, and scalable intelligent compression solution for data centers, cloud services, and edge devices.

[0163] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A data compression method based on neural networks, characterized in that, The method includes: Step S1: Read the file to be compressed in the form of a byte sequence, map each byte to a continuous dense vector through an embedding operation to obtain an embedding vector, and use rotation position encoding to capture the relative position information of the embedding vector; The step of capturing the relative position information of the embedded vector using rotational position encoding includes: Introducing a position-dependent rotation matrix into the embedding space, specifically represented as: in Represents the embedding vector The result after rotational position encoding Represents the rotation matrix. Representing rotation matrix and Dimensions express Location information; The rotation matrix It consists of a diagonal matrix composed of several two-dimensional rotation submatrices, specifically represented as: Each submatrix and frequency They are defined as follows: Step S2: Using a selective state-space model, feature information is extracted from the embedding vector obtained by mapping the byte sequence, capturing short-range and long-range patterns in the input byte sequence to achieve scalable and highly expressive context modeling; Step S3: Perform expert selection through a gating mechanism, calculate the score of each feature for each expert, and assign each feature to a few sparse experts with the highest scores for processing; Step S4: Perform feature fusion operation through gating mechanism, and perform weighted summation of the results obtained after each feature information is processed by several selected sparse experts to obtain the overall feature fusion result, and generate a prediction of the probability distribution of subsequent bytes based on the overall feature fusion result; Step S5: Obtain the byte sequence to be compressed, and use the entropy encoder to compress the byte sequence to be compressed into a compact bit stream representation based on the prediction of the probability distribution of subsequent bytes obtained from the model, thereby achieving lossless compression of data of any file type.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: Determine the embedding dimension q for each byte and the hidden layer dimension h of the model; The file to be compressed is read as a byte sequence to obtain a byte stream sequence S, where each byte... A learnable embedding matrix is ​​mapped to a low-dimensional embedding vector. , This represents the i-th byte in S; The low-dimensional embedding vector The vectors are concatenated to form a high-dimensional embedding vector. ,in, It is an integer factor of h.

3. The method according to claim 2, characterized in that, The low-dimensional embedding vector The vectors are concatenated to form a high-dimensional embedding vector. ,include: Use a sliding window to extract subsequences from the low-dimensional embedding vector sequence encoded by rotational position; The extracted subsequences are concatenated to form a high-dimensional embedding vector. .

4. The method according to claim 3, characterized in that, Step S3 specifically includes: Determine the number of experts and the number of sparse experts actually activated for each feature; The gating mechanism calculates each expert's raw score through a linear transformation; The highest scorer is retained using Top-K operations. The score corresponding to each expert and index ; The scores of the unselected experts were set to Then, Softmax normalization is performed to obtain a sparse weight vector.

5. A data compression system based on a neural network, characterized in that, The system is used to perform the method as described in any one of claims 1 to 4, the system comprising: The first module is used to read the file to be compressed in the form of a byte sequence, map each byte to a continuous dense vector through an embedding operation to obtain an embedding vector, and use rotation position encoding to capture the relative position information of the embedding vector; The step of capturing the relative position information of the embedded vector using rotational position encoding includes: Introducing a position-dependent rotation matrix into the embedding space, specifically represented as: in Represents the embedding vector The result after rotational position encoding Represents the rotation matrix. Representing rotation matrix and Dimensions express Location information; The rotation matrix It consists of a diagonal matrix composed of several two-dimensional rotation submatrices, specifically represented as: Each submatrix and frequency They are defined as follows: The second module is used to extract feature information from the embedding vector obtained by mapping the byte sequence using a selective state-space model, capture short-range and long-range patterns in the input byte sequence, and realize scalable and highly expressive context modeling. The third module is used to: perform expert selection operations through a gating mechanism, calculate the score of each feature information for each expert, and assign each feature information to a few sparse experts with the highest scores for processing; The fourth module is used to: perform feature fusion operation through a gating mechanism, perform weighted summation of the results obtained after each feature information is processed by several selected sparse experts to obtain the overall feature fusion result, and generate a prediction of the probability distribution of subsequent bytes based on the overall feature fusion result; The fifth module is used to obtain the byte sequence to be compressed, and use the entropy encoder to compress the byte sequence to be compressed into a compact bit stream representation based on the prediction of the probability distribution of subsequent bytes obtained from the model, thereby achieving lossless compression of data of any file type.

Images