Image compression method for multi-modal large model

Abstract

The invention provides an image compression method for a multi-modal large model, and the method comprises the steps: S1, obtaining any to-be-compressed image, carrying out the shallow feature extraction of the to-be-compressed image based on a pre-training visual model, and generating a semantic importance map; s2, encoding the to-be-compressed image, guiding bit rate distribution in the encoding process according to the semantic importance map, and generating a compressed bit stream at the same time; and S3, decoding the compressed bit stream to obtain a low-level reconstructed image, performing high-level semantic enhancement on potential features of the low-level reconstructed image based on a potential feature adapter to obtain enhanced potential features, and fusing the enhanced potential features with low-level features extracted from the low-level reconstructed image to obtain a low-level reconstructed image. A compressed reconstructed image is generated for use with the multi-modal large model. The method has the beneficial effects that the image processing performance of the multi-modal large model can be maintained to the greatest extent while efficient compression is realized.

Description

Technical Field

[0001] This invention relates to the technical field of image compression methods, and more specifically, to an image compression method for multimodal large models. Background Technology

[0002] In recent years, large multimodal models (MLLMs) have demonstrated remarkable capabilities in tasks such as image understanding and visual question answering. These models are typically deployed in the cloud and require processing massive amounts of image data transmitted from edge devices (such as smartphones and surveillance cameras). However, limited network bandwidth and storage resources have become key bottlenecks restricting their widespread application. Therefore, efficient compression of image data has become crucial.

[0003] Existing image compression methods are mainly divided into two categories, but neither can meet the unique requirements of multimodal large models: The first category consists of traditional codecs designed for the human visual system (HVS) (such as JPEG, HEIC, AV1, etc.). These methods aim to maintain the quality perceived subjectively by the human eye, typically preserving rich texture and color information while selectively discarding high-frequency details that are insensitive to the human eye. However, these discarded details (such as precise edges and character strokes) may be crucial information relied upon by multimodal large models when performing fine-grained tasks (such as optical character recognition and object counting). Conversely, smooth regions optimized for the human eye may not contain the semantic content relevant to multimodal large models. Therefore, directly using traditional codecs to compress images for multimodal large models leads to a mismatch between bit resource allocation and machine vision requirements, severely impairing the performance of multimodal large models at the same bitrate.

[0004] The second category is VCM-specific encoders and decoders. These methods are optimized for a specific or narrow range of computer vision tasks (such as image classification and object detection), guiding compression by minimizing task-specific losses (such as feature loss). While they may perform well on their specific tasks, their compressed representations are heavily dependent on the task model and dataset used during training, resulting in poor generalization ability. Multimodal large models, on the other hand, inherently handle open-world, unpredictable, and diverse tasks, potentially requiring both low-level details (for OCR) and high-level semantics (for scene understanding). VCM methods optimized for a single task cannot adapt to the general and variable task requirements of multimodal large models, exhibiting significant limitations in practical applications.

[0005] In summary, existing technologies lack a universal image compression method that can intelligently allocate bitrates based on the semantic importance of image content to multimodal large models, and collaboratively reconstruct a reconstructed image at the decoding end that is rich in both accurate low-level details and high-level semantic information. This would ensure efficient compression while preventing significant performance degradation of multimodal large models across various downstream tasks. This invention is proposed to overcome the shortcomings of the aforementioned existing technologies. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to maintain the performance of multimodal large model image processing to the greatest extent while achieving efficient compression. In order to overcome the defects of the above-mentioned prior art (or related art), this invention provides an image compression method for multimodal large models.

[0007] This invention provides an image compression method for multimodal large models, comprising the following steps: Step S1: Obtain any image to be compressed, perform shallow feature extraction on the image to be compressed based on a pre-trained visual model, and generate a semantic importance map; Step S2: Encode the image to be compressed and allocate the bit rate during the encoding process according to the semantic importance graph, while generating a compressed bit stream; Step S3: Decode the compressed bitstream to obtain a low-level reconstructed image; perform high-level semantic enhancement on the latent features of the low-level reconstructed image based on the latent feature adapter to obtain enhanced latent features; and fuse the enhanced latent features with the low-level features extracted from the low-level reconstructed image to generate a compressed reconstructed image for use in a multimodal large model.

[0008] The image compression method for multimodal large models proposed in this invention has the following advantages compared with existing technologies: In this invention, semantically guided intelligent bit allocation is achieved by generating a semantic importance map in step S1. This allows limited bit resources to be prioritized for protecting regions that are crucial for image content understanding in multimodal large models, fundamentally improving the compression rate and rationality. In step S3, the compressed and reconstructed image is obtained by fusing low-level features and enhanced latent features after latent feature adapter enhancement. This process ensures both low-level structural details and high-level semantic information of the image, effectively avoiding the drawbacks of existing compression methods that either sacrifice details or cause semantic distortion. This ensures the performance of multimodal large models on various downstream tasks, ultimately achieving efficient compression while maximizing the maintenance of the performance of multimodal large models in image processing.

[0009] In one possible implementation, the pre-trained visual model includes multiple shallow attention layers connected in sequence, and step S1 includes: Step S11: Obtain any of the images to be compressed, input the images to be compressed into the pre-trained visual model, and obtain the corresponding attention scores through analysis of each of the shallow attention layers; Step S12: According to the connection order of each shallow attention layer, extract the attention scores of the first N shallow attention layers, and average the extracted attention scores to obtain an initial importance map. Step S13: Quantize the initial importance graph to generate a discrete multi-level mask as the semantic importance graph.

[0010] Compared with existing technologies, the above-mentioned technical solution can provide an efficient, stable and interpretable importance judgment mechanism. It uses a pre-trained visual model to calculate the attention scores of the first N shallow attention layers, efficiently identifies potential regions of interest, and averages multiple attention scores to smooth out occasional attention fluctuations, resulting in a more stable and reliable initial importance map, avoiding noise or bias that may be caused by a single attention score.

[0011] In one possible implementation, in step S13, when quantizing the initial importance map, a three-level quantization strategy is used to convert each attention score into a three-level discrete mask. The three levels of the three-level discrete mask correspond to reducing the bit rate, maintaining the baseline bit rate, and increasing the bit rate, respectively.

[0012] Compared with existing technologies, the above technical solution can achieve precise and discrete control of bit rate allocation, and the three levels of instructions for reducing bit rate, maintaining the base bit rate, and increasing bit rate are clear and easy to integrate into existing coding frameworks.

[0013] In one possible implementation, the latent feature adapter is a lightweight network. In step S3, the latent features are extracted through the lightweight network and the enhanced latent features and the low-level features are fused by adding elements one by one.

[0014] Compared with existing technologies, the above technical solution uses a lightweight network as a latent feature adapter, which ensures low computational overhead in the semantic enhancement process and meets the compression efficiency requirements of compression methods. The simple, direct and efficient feature fusion method of element-wise addition can effectively combine low-level structural details and high-level semantic information, avoiding the introduction of complex network structures that would lead to model training difficulties or information loss.

[0015] In one possible implementation, the pre-trained visual model used in step S1 is trained using a multi-level fidelity loss function, which is obtained by weighted summation of low-level fidelity loss and high-level perceptual loss.

[0016] Compared with existing technologies, the above technical solution introduces a multi-level fidelity loss function for model training, providing a clear and correct guiding direction for the optimization of the entire compression method. By combining low-level fidelity loss and high-level perceptual loss, and constraining the pre-trained visual model to learn to reconstruct fine local details and maintain global semantic consistency, it ensures that the final output image can simultaneously meet the information needs of different levels of multimodal large models.

[0017] In one possible implementation, the image to be compressed and the compressed reconstructed image are respectively input into a pre-trained visual encoder to perform shallow feature analysis to obtain corresponding shallow image blocks, and the low-level fidelity loss is obtained based on the mean square error between each of the shallow image blocks.

[0018] Compared with existing technologies, the above-mentioned technical solution can directly and effectively constrain the pre-trained visual model to retain the low-level visual features that are crucial to the multimodal large model. By comparing the mean square error of the image to be compressed and the compressed and reconstructed image on the shallow features of the pre-trained visual model, the distortion in low-level structural details can be accurately measured and penalized.

[0019] In one possible implementation, the image to be compressed and the compressed reconstructed image are respectively input into a pre-trained visual encoder to perform deep feature analysis to obtain corresponding deep image blocks, and the deep-level fidelity loss is obtained based on the mean square error between each deep image block.

[0020] Compared with existing technologies, the above technical solution can ensure that the compressed and reconstructed image is consistent with the image to be compressed in terms of high-level semantic information, thus preventing semantic errors.

[0021] In one possible implementation, in step S1, when the image to be compressed is a high-resolution image, a hierarchical guidance mechanism is used to generate the semantic importance map, including: Step A1: Global downsampling is performed on the image to be compressed to obtain a global semantic importance map. Multiple local image patches are extracted from the image to be compressed, and feature extraction is performed on each local image patch to generate a corresponding local semantic importance map. Step A2: The global semantic importance graph and each of the local semantic importance graphs are fused to obtain the semantic importance graph.

[0022] Compared with existing technologies, the above-mentioned technical solution can solve the problem of insufficient accuracy of global importance map guidance when applying this method to high-resolution scenes. It provides the overall composition and key region distribution of the image through the global semantic importance map, avoiding the lack of global vision during local optimization. It provides more accurate and detailed bit rate allocation guidance by performing fine analysis on each local image block through the local semantic importance map. Attached Figure Description

[0023] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 This is a flowchart illustrating step S1 of the present invention. Figure 3 This is a schematic diagram illustrating the encoder-decoder principle consisting of a shallow CLIP-guided encoder and a multi-level fidelity decoder according to the present invention. Figure 3 (a) is the training phase. Figure 3 (b) is the reasoning stage. Figure 3 (c) is the expansion stage of high-resolution images. Detailed Implementation

[0024] First, those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] See Figure 1 This invention discloses an image compression method for multimodal large models, comprising: Step S1: Obtain any image to be compressed, perform shallow feature extraction on the image to be compressed based on the pre-trained visual model, and generate a semantic importance map; Step S2: Encode the image to be compressed and allocate the bit rate during the encoding process according to the semantic importance map, while generating a compressed bit stream; Step S3: Decode the compressed bitstream to obtain a low-level reconstructed image. Perform high-level semantic enhancement on the latent features of the low-level reconstructed image based on the latent feature adapter to obtain enhanced latent features. Then, fuse the enhanced latent features with the low-level features extracted from the low-level reconstructed image to generate a compressed reconstructed image for use in a multimodal large model.

[0027] In this embodiment of the invention, steps S1 and S2 constitute the encoding stage, and step S3 constitutes the decoding stage. Therefore, an encoder module is generated based on steps S1 and S2 to execute the content of steps S1 and S2, and a decoder module is generated based on step S3 to execute the content of step S3. Specifically, in the encoding stage, a semantic importance map is first generated using the output of a shallow network of a pre-trained visual model (such as CLIP). A shallow network is chosen because research shows that the shallow network of a pre-trained visual model is primarily responsible for preliminary, global filtering of the image, effectively identifying potential semantically critical regions. This semantic importance map is then used to guide the bitrate allocation of a basic image encoder (such as ELIC or DCAE), i.e., the semantic importance map. More bits are allocated to important regions marked on the topology, thus protecting the most important information for the multimodal large model at the compression source. In the decoding stage, an innovative multi-level fidelity decoding strategy is adopted. First, the image itself after decoding the compressed bitstream is used as a "reconstruction prior" to preserve low-level structural details (such as contours and textures) that are robust to standard compression and to mitigate the neighborhood offset problem. Second, a lightweight "latent feature adapter" is designed to operate directly on the decoded latent features, injecting high-level semantic information to enhance them. Finally, the low-level features from the reconstruction prior and the enhanced latent features generated by the latent feature adapter are fused to generate a compressed reconstructed image that is rich in high-quality low-level structural details and accurate high-level semantic information for use by the multimodal large model.

[0028] In this embodiment of the invention, through semantically guided bitrate allocation and multi-level fidelity reconstruction, the method of the present invention can significantly reduce the bitrate (saving transmission bandwidth and storage costs) while making the performance of multimodal large models in various benchmark tests (such as MME, MMBeen) essentially the same as when using the original image to be compressed, and significantly outperform existing state-of-the-art codecs. Furthermore, unlike ICM methods that are only for specific tasks, the method of the present invention can universally support various downstream tasks performed by multimodal large models by protecting multi-level visual features, whether it is a task that requires low-level structural details (such as OCR) or a task that requires high-level understanding (such as scene classification), especially for tasks that require a high degree of combination of both (such as counting).

[0029] In this embodiment of the invention, the pre-trained visual model includes multiple shallow attention layers connected in sequence, see [link to previous section]. Figure 2 Step S1 includes: Step S11: Obtain any image to be compressed, input the image to be compressed into the pre-trained visual model, and obtain the corresponding attention score through analysis of each shallow attention layer; Step S12: Based on the connection order of each shallow attention layer, extract the attention scores of the first N shallow attention layers, and average the extracted attention scores to obtain the initial importance map, where N is an integer greater than or equal to 1. Step S13: Quantize the initial importance graph to generate discrete multi-level masks as semantic importance graphs.

[0030] In this embodiment of the invention, in step S13, when quantizing the initial importance map, a three-level quantization strategy is used to convert each attention score into a three-level discrete mask. The three levels of the three-level discrete mask correspond to reducing the bit rate, maintaining the baseline bit rate, and increasing the bit rate, respectively.

[0031] In this embodiment of the invention, a frozen, pre-trained CLIP model's visual encoder is used to extract attention maps of the first three layers (Layer 0, 1, 2) of the CLS tokens. These attention maps reflect the CLIP model's attention to each region of the image in the early stages of processing. The attention maps of these three layers are averaged to obtain a smooth, continuous attention score map as the initial importance map. This initial importance map is downsampled (e.g., downsampled to 8x8) and quantized into a three-level discrete mask using a statistical method. These three levels represent "low importance" (reducing the bit rate), "medium importance" (maintaining the baseline bit rate), and "high importance" (increasing the bit rate), respectively. This final, small-sized (e.g., 8x8x2 bits, with minimal overhead) three-level discrete mask is used to patchwise adjust the quantization parameters within the pre-trained visual model, thereby achieving precise, semantically aware control of the bit rate.

[0032] In this embodiment of the invention, the latent feature adapter is a lightweight network. In step S3, latent features are extracted through the lightweight network and the enhanced latent features and low-level features are fused by adding elements one by one.

[0033] In this embodiment of the invention, the latent feature adapter is a neural network module composed of a lightweight network that directly operates on the latent code decoded from the compressed bitstream, rather than on the pixel domain. Its function is to learn and generate a "semantic enhancement feature", which is then fused with the low-level feature embedding extracted from the prior image to be detected by adding elements-wise. This strategy cleverly injects high-level semantic guidance directly into the feature domain without destroying the existing low-level information.

[0034] In this embodiment of the invention, the pre-trained visual model used in step S1 is trained using a multi-level fidelity loss function, which is obtained by weighted summation of low-level fidelity loss and high-level perceptual loss.

[0035] In this embodiment of the invention, the image to be compressed and the compressed and reconstructed image are respectively input into a pre-trained visual encoder to perform shallow feature analysis to obtain corresponding shallow image blocks, and the low-level fidelity loss is obtained based on the mean square error between each shallow image block.

[0036] In this embodiment of the invention, the image to be compressed and the compressed and reconstructed image are respectively input into a pre-trained visual encoder to perform deep feature analysis to obtain the corresponding deep image blocks, and the deep-level fidelity loss is obtained based on the mean square error between each deep image block.

[0037] In this embodiment of the invention, the multi-level fidelity loss function is obtained through the following calculation formula: ; in, Indicates multi-level fidelity loss. This indicates a loss of fidelity at lower levels. This indicates high-level perceived loss. and This represents the hyperparameter that balances the two losses.

[0038] In this embodiment of the invention, in step S1, when the image to be compressed is a high-resolution image, a hierarchical guidance mechanism is used to generate a semantic importance map, including: Step A1: Global downsampling is performed on the image to be compressed to obtain a global semantic importance map. Multiple local image patches are extracted from the image to be compressed, and feature extraction is performed on each local image patch to generate a corresponding local semantic importance map. Step A2: The global semantic importance graph and the semantic importance graphs of each local area are fused to obtain the semantic importance graph.

[0039] In this embodiment of the invention, when mainstream multimodal large models process high-resolution images, they typically employ a patch-based strategy. Directly applying the global low-resolution guidance map to each patch is inaccurate. Therefore, this invention proposes a hierarchical guidance mechanism: fusing (e.g., adding) the global semantic importance map generated from the globally downsampled image with the local semantic importance map independently generated for each high-resolution image patch. The resulting comprehensive guidance signal possesses both global awareness and local precision. Current mainstream video multimodal large models typically process video frames as a series of static images. Therefore, the image compression framework of this invention can be directly applied to frame-by-frame compression of video, providing semantic guidance for video stream compression.

[0040] See Figure 3 This invention also proposes a codec that mainly includes a shallow CLIP-guided encoder and a multi-level fidelity decoder, wherein... Figure 3 (a) is the training phase. Figure 3 (b) is the reasoning stage. Figure 3 (c) is the high-resolution image expansion stage. In the inference stage, the input image first passes through the encoder (Enc.). A semantic importance map is generated by the shallow CLIP guidance module inside the encoder (Enc.). The semantic importance map controls the quantization process of the basic encoder to achieve adaptive bit rate allocation and finally output a compressed bit stream. At the decoder (Dec.), the compressed bit stream is decoded. The decoded image is used as the reconstruction prior. Its image patch embeddings are added and fused with the enhanced latent features generated by the latent feature adapter formed by a lightweight network. The final feature sequence is fed into the multimodal large model.

[0041] In the description of this invention, the references to "one embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0042] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image compression method for multimodal large models, characterized in that, Includes the following steps: Step S1: Obtain any image to be compressed, perform shallow feature extraction on the image to be compressed based on a pre-trained visual model, and generate a semantic importance map; Step S2: Encode the image to be compressed and allocate the bit rate during the encoding process according to the semantic importance graph, while generating a compressed bit stream; Step S3: Decode the compressed bitstream to obtain a low-level reconstructed image; perform high-level semantic enhancement on the latent features of the low-level reconstructed image based on the latent feature adapter to obtain enhanced latent features; and fuse the enhanced latent features with the low-level features extracted from the low-level reconstructed image to generate a compressed reconstructed image for use in a multimodal large model.

2. The image compression method for multimodal large models according to claim 1, characterized in that, The pre-trained visual model includes multiple shallow attention layers connected in sequence, and step S1 includes: Step S11: Obtain any of the images to be compressed, input the images to be compressed into the pre-trained visual model, and obtain the corresponding attention scores through analysis of each of the shallow attention layers; Step S12: According to the connection order of each shallow attention layer, extract the attention scores of the first N shallow attention layers, and average the extracted attention scores to obtain an initial importance map. Step S13: Quantize the initial importance graph to generate a discrete multi-level mask as the semantic importance graph.

3. The image compression method for multimodal large models according to claim 2, characterized in that, In step S13, when quantizing the initial importance map, a three-level quantization strategy is used to convert each attention score into a three-level discrete mask. The three levels of the three-level discrete mask correspond to reducing the bit rate, maintaining the baseline bit rate, and increasing the bit rate, respectively.

4. The image compression method for multimodal large models according to claim 1, characterized in that, The latent feature adapter is a lightweight network. In step S3, the latent features are extracted through the lightweight network and the enhanced latent features and the low-level features are fused by adding elements one by one.

5. The image compression method for multimodal large models according to claim 1, characterized in that, The pre-trained visual model used in step S1 is trained using a multi-level fidelity loss function, which is obtained by weighted summation of low-level fidelity loss and high-level perceptual loss.

6. The image compression method for multimodal large models according to claim 5, characterized in that, The image to be compressed and the compressed and reconstructed image are respectively input into a pre-trained visual encoder to perform shallow feature analysis to obtain corresponding shallow image blocks, and the low-level fidelity loss is obtained based on the mean square error between each shallow image block.

7. The image compression method for multimodal large models according to claim 5, characterized in that, The image to be compressed and the compressed and reconstructed image are respectively input into a pre-trained visual encoder to perform deep feature analysis to obtain corresponding deep image blocks, and the deep-level fidelity loss is obtained based on the mean square error between each deep image block.

8. The image compression method for multimodal large models according to claim 1, characterized in that, In step S1, when the image to be compressed is a high-resolution image, a hierarchical guidance mechanism is used to generate the semantic importance map, including: Step A1: Global downsampling is performed on the image to be compressed to obtain a global semantic importance map. Multiple local image patches are extracted from the image to be compressed, and feature extraction is performed on each local image patch to generate a corresponding local semantic importance map. Step A2: The global semantic importance graph and each of the local semantic importance graphs are fused to obtain the semantic importance graph.

Images