A universal lossless data compression method based on multimodal feature fusion

Abstract

The present invention discloses a universal lossless data compression method based on multimodal feature fusion, comprising: collecting multimodal original data, extracting type features and content features; constructing a multimodal feature fusion model based on a feature fusion module, a liquid neural network, and a Transformer module, inputting the type features and content features into the multimodal feature fusion model for fusion, performing conditional probability prediction on the fusion features, and outputting the predicted probability distribution of each symbol; compressing the byte sequence of the original data using an arithmetic coding method; introducing a small batch online training mechanism during the compression process to complete the updating of the multimodal feature fusion model; the present invention provides an efficient, flexible, and highly adaptable universal data compression solution, which can not only significantly improve the compression efficiency of diversified data in scenarios such as digital twin systems, but also maintain long-term stable performance in dynamically changing environments, and has broad application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of computers and electronic information engineering, and in particular to a universal lossless data compression method based on multimodal feature fusion. Background Art

[0002] In digital twin systems, virtual models must mirror physical entities in real time to achieve accurate perception, simulation, and optimization of the real world. Digital twin systems are widely used in a variety of fields, including manufacturing, energy, transportation, and smart cities. They can significantly improve operational efficiency, reduce failure rates, and enhance system flexibility and intelligence. With the increasing interaction between the physical and virtual worlds, digital twin systems face the challenge of real-time processing and transmission of massive amounts of diverse data. Different data sources (such as text, images, audio, video, and sensor data) have unique statistical characteristics and structural patterns, and require low-latency, high-reliability data synchronization, which places higher demands on data compression technology.

[0003] Most existing data compression technologies are optimized for specific types of data and can achieve good results when processing a single data type (such as images or text). However, when faced with multi-source heterogeneous data, they often suffer from insufficient adaptability and reduced compression efficiency. For example, deep compression methods for images severely reduce the compression rate when processing audio or text data. Traditional general-purpose compression algorithms, such as Gzip and 7zip, while adaptable to a certain extent, have difficulty meeting the real-time and high-efficiency requirements of digital twin systems in terms of compression rate and speed. In addition, while some neural network-based compression methods have made progress in specific tasks (such as video compression), they lack a unified modeling mechanism for processing multimodal data, making it difficult to take into account the characteristics of different data types, resulting in limited overall compression performance.

[0004] In order to meet the actual needs of data diversity, large volume and rapid changes in digital twin systems, a compression technology is needed that can adapt to multiple data types, have high prediction accuracy and high compression ratio, and can be adjusted in real time according to the data flow. Summary of the Invention

[0005] The purpose of the present invention is to provide a universal lossless data compression method based on multimodal feature fusion, which can achieve efficient unified modeling for multiple data types and realize lossless compression through arithmetic coding, effectively improving the data transmission efficiency and real-time response capability in the digital twin system, and making up for the shortcomings of existing methods in universality, compression efficiency and adaptability.

[0006] To achieve the above functions, the present invention designs a universal lossless data compression method based on multimodal feature fusion, which performs the following steps S1 to S4 to complete the fusion and compression of multiple types of data:

[0007] Step S1: Collect various types of raw data, including text, audio, images, and sensor data, preprocess each type of raw data, and extract type features and content features for each type of preprocessed data. Type features describe the overall statistical characteristics of the data, while content features retain the structural information of the raw data.

[0008] Step S2: Based on the cross-attention module, liquid neural network, and Transformer module, a multimodal feature fusion model is constructed and trained. The type feature and content feature are input into the multimodal feature fusion model for fusion to obtain the fusion feature of the type feature and content feature. Conditional probability prediction is performed for each symbol of the fusion feature, and the predicted probability distribution of each symbol is output;

[0009] Step S3: Based on the predicted probability distribution of each symbol, an arithmetic coding method is used to dynamically allocate a coding interval according to the predicted probability distribution of each symbol, and the byte sequence of the original data is compressed;

[0010] Step S4: Introduce a small batch online training mechanism in the compression process, perform back propagation and gradient update based on the negative log-likelihood loss between the predicted probability distribution of each symbol and the true symbol, complete the update of the multimodal feature fusion model, and compress various types of raw data.

[0011] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0012] The present invention proposes a universal lossless data compression method based on multimodal feature fusion, which has significant technical advantages. Through multimodal feature extraction and fusion, the present invention can simultaneously process multiple data types, such as text, audio, video and sensor data, and perform efficient compression under the same framework, avoiding the limitations of traditional methods for a single data type. The method proposed in the present invention improves the prediction accuracy of the model by deeply fusing the statistical characteristics and structural characteristics of the data, thereby effectively reducing the bit rate during the compression process and significantly improving the compression efficiency. In addition, the online adaptive training mechanism introduced by the present invention enables the model to self-adjust when the data stream changes in real time, maintaining efficient compression performance, and is particularly suitable for real-time data processing and applications in dynamic environments. Through arithmetic coding technology, lossless compression is achieved, ensuring the integrity and accuracy of the data, and is suitable for scenarios with strict requirements for data recovery. Overall, the present invention provides an efficient, flexible and highly adaptable universal data compression solution, which can not only significantly improve the compression efficiency of diversified data in scenarios such as digital twin systems, but also maintain long-term stable performance in dynamically changing environments, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flowchart of a general lossless data compression method based on multimodal feature fusion provided according to an embodiment of the present invention;

[0014] Figure 2 1 is a schematic diagram of a process for extracting and fusing type features and content features according to an embodiment of the present invention;

[0015] Figure 3 1 is a schematic diagram of a model structure of a general lossless data compression method based on multimodal feature fusion provided according to an embodiment of the present invention;

[0016] Figure 4 is a graph showing how the training loss and validation loss change with the number of training rounds according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0018] In digital twin systems, as the frequency of interaction between the physical world and virtual models continues to increase, the system must process the real-time transmission and updating of multiple data types, including video, audio, images, text, and sensors. This data exhibits significant heterogeneity in format, distribution, and structure, and places extremely high demands on low latency, high accuracy, and lossless restoration. Traditional compression methods are often optimized for a single data type and are difficult to adapt to multi-source data fusion scenarios. This reduces compression efficiency and increases transmission latency, severely restricting the performance of digital twin systems.

[0019] The embodiment of the present invention provides a general lossless data compression method based on multimodal feature fusion, referring to Figure 1 , execute the following steps S1 to S4 to complete the fusion and compression of multiple types of data:

[0020] Step S1: Collect various types of raw data, including text, audio, images, and sensor data, preprocess each type of raw data, and extract type features and content features for each type of preprocessed data. Type features describe the overall statistical characteristics of the data, while content features retain the structural information of the raw data.

[0021] Reference Figure 2 , the specific steps of step S1 are as follows:

[0022] Step S1.1: Perform unified preprocessing on various types of raw data, standardize all types of raw data and convert them into byte sequences to adapt to the subsequent unified modeling process; the byte sequences are processed in sequence through the embedding layer, normalization layer, and linear layer;

[0023] Step S1.2: extracting content features and three types of features from the processed byte sequence;

[0024] The extracted type features include data entropy, discrete cosine transform (DCT) coefficients, and wavelet transform energy distribution (DWT);

[0025] Among them, data entropy uses Shannon entropy to count the distribution frequency of each byte and calculate the entropy value to measure data complexity. The calculation is as follows:

[0026] ;

[0027] in, It is i input bytes The probability of occurrence in the data, Represents data entropy, with a maximum entropy value of 8;

[0028] The discrete cosine transform coefficients are used to capture the frequency domain characteristics of the data and extract the low-frequency coefficients in the frequency domain to form the spectrum characteristics. The calculation is as follows:

[0029] ;

[0030] Where, For the k discrete cosine transform coefficients, k =0, 1, ..., N -1, is the discrete sequence n samples, N is the length of the discrete sequence;

[0031] The wavelet transform decomposes the signal into a multi-scale energy distribution to capture local variation characteristics. The wavelet transform decomposes the signal into multiple frequency sub-bands and extracts energy features. All types of features are standardized to a distribution with a mean of 0 and a standard deviation of 1. The calculation of the wavelet transform energy distribution is as follows:

[0032] ;

[0033] in, z is the wavelet transform energy distribution, x Represents input data, is the mean of the data, is the standard deviation, is a constant to prevent division by zero.

[0034] The content features extracted for various types of raw data are to truncate the byte sequence of the raw data to a preset maximum length. , ensuring that the feature dimensions are consistent. Through this step, different types of data are uniformly mapped to a standardized feature space, which is conducive to subsequent unified processing.

[0035] Step S1.3: The three types of features are normalized and concatenated into a fixed-length type vector. To achieve adaptive weighted fusion of the three types of features, the model uses a multi-head self-attention mechanism, using the three types of features as query, key, and value, respectively. Self-attention calculations are performed, outputting a unified fusion representation of the type features and forming a weighted representation of each type of feature in the current byte sequence. This approach allows the model to dynamically learn the impact of each type of statistical feature during training.

[0036] Step S2: Refer to Figure 3Based on the cross-attention module, Liquid Neural Networks (LNN), and Transformer module, a multimodal feature fusion model is constructed and trained. The type features and content features are input into the multimodal feature fusion model for fusion to obtain the fusion features of the type features and content features. Conditional probability prediction is performed for each symbol of the fusion feature, and the predicted probability distribution of each symbol is output;

[0037] The specific steps of step S2 are as follows:

[0038] Step S2.1: Map the fused representation of the type features to the same dimensional space as the content features and input them into the cross-attention module together with the content features to perform cross-modal information fusion. The cross-attention module uses the content features as queries and the type features as keys and values, performs feature interaction between the type features and content features through the cross-attention mechanism, and outputs the fused features.

[0039] Step S2.2: Use the gated fusion unit to dynamically weight the original content features and the fusion features, as shown in the following formula:

[0040] ;

[0041] in, is the gated fusion unit at the current time step t The weighted fusion features of the output, is the current time step t The fusion gating matrix, , is the cross attention module at the current time step t The output fusion features, is the current time step t Original content characteristics; Represents the Sigmoid activation function, which is used to normalize the gate matrix value so that its value range is limited to between 0 and 1; W is a trainable weight parameter matrix used to adjust the cross attention module at the current time step. t Output fusion features With the current time step t Original content features Perform linear mapping to generate the current time step t Fusion gating matrix . Dynamic weighted fusion of features is achieved through gating mechanism;

[0042] Step S2.3: Input the fusion features output by the gated fusion unit into the liquid neural network (LNN). The liquid neural network models short-term time series information through nonlinear gated state transitions. The liquid neural network uses recursive neural units to dynamically adjust the timing characteristics. Its state update formula is as follows:

[0043] ;

[0044] in, is the current time step t The hidden state of is the previous time step t A hidden state of -1, is the current time step t The update rate of the gated fusion unit is determined by the previous state and the current input. It can sense local input changes and has a natural advantage in capturing sudden changes, pulses or edge information.

[0045] Step S2.4: In order to further improve the modeling adaptability under different modal data, a modal control mechanism is introduced, as shown in the following formula:

[0046] ;

[0047] in, Indicates that at time step t The hidden state after fusion control and normalization operation, is the current time step t The fusion ratio control gate is obtained by linearly mapping the fusion type features. LayerNorm Representation layer normalization;

[0048] Step S2.5: The output sequence of the Liquid Neural Network is fed into the Transformer module. With the help of positional encoding, a multi-head self-attention mechanism is used to model long-range dependencies. A feedforward network is then used to extract high-level contextual semantics. The Transformer module excels at capturing global structural relationships across positions and scales, complementing the Liquid Neural Network's ability to model global relationships in addition to local dynamics.

[0049] After passing through the embedding layer, position encoder, encoder, decoder, and linear prediction layer of the Transformer module, the encoder and decoder are based on the multi-head self-attention mechanism to output the predicted probability distribution of each symbol in the fusion feature.

[0050] The Transformer module enhances the sequence through position encoding, and the position encoding formula is:

[0051] ;

[0052] ;

[0053] in, Indicates the second i Position encoding, that is, position encoding of even dimensions, Indicates the second i +1 positional encoding, i.e. positional encoding of odd dimensions, is the dimension of the model, is the position index; this position encoding adds position information to each element in the sequence, helping the model capture sequential relationships;

[0054] The encoder and decoder are based on the multi-head self-attention mechanism, and the calculation formula is:

[0055] ;

[0056] in, 、 、 are query, key, and value matrices respectively, is the dimension of the key vector; the multi-head attention mechanism enables the model to learn complex dependencies between data in multiple subspaces;

[0057] Based on the fused features, symbol-level prediction is achieved through a unified probabilistic modeling network. The fused features are first normalized and then input into the linear prediction head, which outputs the full symbol probability distribution (0 to 255) for each position. The predicted distribution is normalized by the logarithmic Softmax function, and its calculation formula is:

[0058] ;

[0059] in, It is i Real symbols The prediction score of It is j symbols The prediction score of It is a symbol The predicted probability distribution of

[0060] This formula converts scores into probabilities using the Softmax function, ensuring that the sum of the probabilities of all symbols is 1. This is used to provide conditional probability predictions for each symbol, ensuring efficient encoding during compression. This mechanism requires no structural adjustments for different data types, making it highly versatile. Symbol prediction accuracy directly impacts the compression performance of subsequent arithmetic coding. Therefore, through deep fusion and multi-scale modeling, high prediction accuracy can be maintained across a wide range of data types.

[0061] When model training is not yet complete, it will continue to receive input data for incremental training and continuously optimize model parameters. When the training data input is complete and reaches the preset convergence standard, the final symbol probability prediction model is generated.

[0062] This paper sequentially combines liquid neural networks with Transformer modules and dynamically integrates them across different data types through a modal control mechanism, achieving a balance between capturing local changes and understanding global structures. This hybrid structure demonstrates strong adaptability to different data modalities in compression tasks and is particularly suitable for compressing complex multimodal data that combines strong temporal and structural characteristics.

[0063] Step S3: Based on the predicted probability distribution of each symbol, an arithmetic coding method is used to dynamically allocate a coding interval according to the predicted probability distribution of each symbol, and the byte sequence of the original data is compressed;

[0064] Arithmetic coding achieves compression by continuously subdividing the current coding interval. The coding interval is calculated as follows:

[0065] ;

[0066] ;

[0067] in, 、 are the lower and upper bounds of the current encoding interval, 、 are the new lower and upper bounds of the current encoding interval, It is i Input symbols The cumulative probability distribution function of It is a symbol The cumulative probability distribution function of ; Since the multimodal feature fusion model can output accurate conditional probabilities, the final encoding length is close to the symbol entropy limit, thereby achieving a higher compression ratio on multiple data sources.

[0068] Step S4: Introduce a small-batch online training mechanism during the compression process, perform backpropagation and gradient update based on the negative log-likelihood loss (NLLLoss) between the predicted probability distribution of each symbol and the true symbol, complete the update of the multimodal feature fusion model, and compress various types of raw data.

[0069] The online training mechanism allows for fine-tuning and updating of multimodal feature fusion model parameters based on real-time input data, dynamically adapting to subtle changes in data distribution. On the decoding side, the same version of model parameters and arithmetic decoder as on the encoding side are loaded, and lossless reconstruction of the original data is completed based on the received encoded data stream and predicted probability distribution, ensuring data integrity and accuracy. The online adaptive mechanism not only enhances the stability and robustness of the model, but also improves the ability to continuously compress and transmit in dynamic environments.

[0070] The formula for negative log-likelihood loss is as follows:

[0071] ;

[0072] in, is the negative log-likelihood loss, It is i A real symbol, It is a real symbol The predicted probability distribution of .

[0073] By fine-tuning the model parameters after each batch of data is processed, it can dynamically adapt to statistical changes in the real-time data stream, improving long-term compression performance. On the decoding end, the target device loads the model and arithmetic decoder used during encoding, and gradually reconstructs the symbols based on the received data stream and predicted probabilities, thereby achieving lossless restoration of the original data, ensuring data integrity and real-time interactivity.

[0074] Figure 4 This graph shows the variation of the training loss (Train Loss) and validation loss (Val Loss) over the number of training rounds for the proposed method on the Children's Song Dataset (CSD). CSD is a children's song dataset consisting of 50 Korean and 50 English songs. Each audio data entry is accompanied by a MIDI transcript and grapheme- and phoneme-level lyric annotations. This dataset has a relatively balanced distribution across the four modalities of text, image, video, and audio, and was therefore selected as a training example for the proposed method to verify the model's ability to handle multimodal data. As shown in the graph, both the training loss and validation loss continue to decrease as training progresses, tending to converge after a certain number of rounds. The two curves follow a similar trajectory, with no significant overfitting observed. This demonstrates the proposed method's strong modeling capabilities for multi-source heterogeneous data and good generalization performance. The downward trend in the loss curve further validates the effectiveness of the proposed method's multi-scale feature fusion modeling strategy and online training mechanism, which enable rapid adaptation to complex and changing data environments while maintaining compression accuracy.

[0075] Table 1 shows the experimental results of bits per character (BPC) comparison between the present invention and mainstream compression methods on multiple mixed datasets (including CSD, TextVQA, BraVL, and MSRA):

[0076] Table 1. Comparative experimental results of bits per character (BPC)

[0077]

[0078] As can be seen from the table, compared with traditional general compression algorithms (such as 7-Zip, Zip, Bzip, Tar, BSC) and existing neural network compression methods (such as CMIX, Dzip), the method MFFC (Multi-modal Feature Fusion Compression) of the present invention achieves lower BPC values ​​on various data sets, with superior compression effects, especially when facing mixed and complex data sources. This further verifies the effectiveness of the multimodal feature fusion mechanism, liquid neural modeling strategy and online adaptive optimization mechanism proposed in the present invention. By stably achieving good compression effects on data sets of different types and sizes, the method of the present invention demonstrates strong versatility and high efficiency, and can meet the needs of real-time and efficient data compression and transmission in various practical application scenarios such as digital twin systems, intelligent manufacturing, and the Internet of Things.

[0079] In summary, this invention successfully achieves efficient lossless compression in a multi-source heterogeneous data environment through innovative multimodal feature fusion modeling, combining a liquid neural network with a Transformer hybrid architecture, and introducing an online adaptive training optimization mechanism. The overall system not only boasts an excellent compression ratio but also demonstrates significant advantages in compression speed, adaptability, and scalability. It can be widely used in digital twin systems and other intelligent application scenarios that require high-quality large-scale, diverse data processing and transmission.

[0080] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in this field without departing from the spirit of the present invention.

Claims

1. A general lossless data compression method based on multimodal feature fusion, characterized in that: Execute the following steps S1 to S4 to complete the fusion and compression of multiple types of data: Step S1: Collect various types of raw data, including text, audio, images, and sensor data, preprocess each type of raw data, and extract type features and content features for each type of preprocessed data. Type features describe the overall statistical characteristics of the data, while content features retain the structural information of the raw data. Step S2: Based on the cross-attention module, liquid neural network, and Transformer module, a multimodal feature fusion model is constructed and trained. The type feature and content feature are input into the multimodal feature fusion model for fusion to obtain the fusion feature of the type feature and content feature. Conditional probability prediction is performed for each symbol of the fusion feature, and the predicted probability distribution of each symbol is output; Step S3: Based on the predicted probability distribution of each symbol, an arithmetic coding method is used to dynamically allocate a coding interval according to the predicted probability distribution of each symbol, and the byte sequence of the original data is compressed; Step S4: Introduce a small batch online training mechanism in the compression process, perform back propagation and gradient update based on the negative log-likelihood loss between the predicted probability distribution of each symbol and the true symbol, complete the update of the multimodal feature fusion model, and compress various types of raw data.

2. A universal lossless data compression method based on multimodal feature fusion according to claim 1, characterized in that: The specific steps of step S1 are as follows: Step S1.1: Perform unified preprocessing on various types of raw data, standardize all types of raw data and convert them into byte sequences, and process the byte sequences through the embedding layer, normalization layer, and linear layer in sequence; Step S1.2: extracting content features and three types of features from the processed byte sequence; Step S1.3: Use the three types of features as query, key, and value respectively to perform self-attention calculation, output a unified fusion representation of the type features, and form a weighted representation of each type of feature under the current byte sequence.

3. The universal lossless data compression method based on multimodal feature fusion according to claim 1, characterized in that: The type features extracted in step S1 include data entropy, discrete cosine transform coefficients, and wavelet transform energy distribution; The calculation of data entropy is as follows: ； in, It is i input bytes The probability of occurrence in the data, represents data entropy; The calculation of discrete cosine transform coefficients is as follows: ； Where, For the k discrete cosine transform coefficients, k =0, 1, ..., N -1, is the discrete sequence n samples, N is the length of the discrete sequence; The calculation of wavelet transform energy distribution is as follows: ； in, z is the wavelet transform energy distribution, x Represents input data, is the mean of the data, is the standard deviation, is a constant to prevent division by zero.

4. The universal lossless data compression method based on multimodal feature fusion according to claim 1, characterized in that: The content features extracted from various types of raw data in step S1 are to truncate the byte sequence of the raw data to a preset maximum length. .

5. The universal lossless data compression method based on multimodal feature fusion according to claim 2, characterized in that: The specific steps of step S2 are as follows: Step S2.1: Map the fused representation of the type features to the same dimensional space as the content features and input them into the cross-attention module together with the content features. The cross-attention module uses the content features as queries and the type features as keys and values. It performs feature interaction between the type features and the content features through the cross-attention mechanism and outputs the fused features. Step S2.2: Use the gated fusion unit to dynamically weight the original content features and the fusion features, as shown in the following formula: ； in, is the gated fusion unit at the current time step t The weighted fusion features of the output, is the current time step t The fusion gating matrix, , is the cross attention module at the current time step t The output fusion features, is the current time step t The original content characteristics, Represents the Sigmoid activation function; W is a trainable weight parameter matrix; Step S2.3: Input the weighted fusion features output by the gated fusion unit into the liquid neural network. The liquid neural network uses recursive neural units to dynamically adjust the timing characteristics. Its state update formula is as follows: ； in, is the current time step t The hidden state of is the previous time step t A hidden state of -1, is the current time step t The update rate of the gated fusion unit control; Step S2.4: Introduce the modal control mechanism, as follows: ； in, Indicates that at the current time step t The hidden state after fusion control and normalization operation, is the current time step t The fusion proportional control gate, LayerNorm Representation layer normalization; Step S2.5: Input the output sequence of the liquid neural network into the Transformer module, and pass through the embedding layer, position encoder, encoder, decoder, and linear prediction layer of the Transformer module. The encoder and decoder are based on the multi-head self-attention mechanism and output the predicted probability distribution of each symbol in the fused feature.

6. The universal lossless data compression method based on multimodal feature fusion according to claim 1, characterized in that: The coding interval in step S4 is calculated as follows: ； ； in, 、 are the lower and upper bounds of the current encoding interval, 、 are the new lower and upper bounds of the current encoding interval, It is i Input symbols The cumulative probability distribution function of It is a symbol The cumulative probability distribution function of .

7. The universal lossless data compression method based on multimodal feature fusion according to claim 1, characterized in that: The formula for the negative log-likelihood loss in step S5 is as follows: ； in, is the negative log-likelihood loss, It is i A real symbol, It is a real symbol The predicted probability distribution of .

Images