A Semantically Interpretable Compression Method for Visual Understanding of Multimodal Large Language Models
By co-designing a dynamic object-aware adapter and a self-organizing evolutionary cross-modal shared semantic prototype library, the semantic opacity problem of image compression in multimodal large language models in visual question answering tasks is solved. This achieves high-fidelity reconstruction and semantic decoupling under high compression ratios, improving the accuracy and robustness of visual question answering.
Patent Information
- Application Number
- CN202611288230.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-24
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multimodal large language models face technical bottlenecks in visual question answering and image description tasks, particularly in maintaining high-fidelity image reconstruction capabilities under high compression ratios. They also lack semantic transparency and structural interpretability, resulting in insufficient accuracy and robustness in visual question answering.
By employing a dynamic object-aware adapter and a self-organizing evolutionary cross-modal shared semantic prototype library, and through zero-initialization gated cross-attention and tree-like soft quantization, a compact representation of hierarchical semantic weight distribution matrix and sparse prototype index is generated. Combined with a multi-loss function optimization model, high-fidelity image reconstruction and semantic decoupling are achieved.
It maintains high-fidelity image reconstruction capability under high compression ratio, provides semantically decoupled, conceptually readable and hierarchically structured visual knowledge representation for multimodal large language models, and improves the accuracy and out-of-distribution robustness of visual question answering.
Smart Images

Figure CN123118792A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent decision-making technology in supply chains, and in particular to a semantically interpretable compression method for visual understanding of multimodal large language models. Background Technology
[0002] Multimodal large language models have demonstrated powerful reasoning capabilities in cross-modal understanding tasks such as visual question answering and image description in recent years. Their core paradigm involves encoding visual signals into token embedding sequences, concatenating them with text instructions, and then feeding them into the large language model for joint reasoning. However, as multimodal applications extend to edge and real-time scenarios, the collaborative design between image compression and visual understanding has become a critical technical bottleneck that urgently needs to be overcome. Traditional image compression methods, such as neural image compression frameworks based on variational autoencoders, can reconstruct visually acceptable images at extremely low bitrates, but the resulting latent representations are typically continuous, dense feature tensors, lacking explicit semantic dimensions and structural interpretability. When such representations are fed into multimodal large language models, the models need to implicitly infer object relationships and semantic boundaries from unstructured continuous vectors, which not only increases the reasoning burden but also leads to a significant decrease in visual question answering accuracy in out-of-distribution scenarios.
[0003] To improve the semantic transparency of latent representations, some studies have attempted to introduce vector quantization or prototype learning mechanisms to map continuous features to discrete codebooks or a set of learnable semantic prototypes. However, existing prototype libraries are mostly statically constructed with fixed semantic meanings, making it difficult to cover the diverse visual concepts in the open world; the residuals generated during quantization are still retained in continuous numerical form, disrupting the interpretable closed loop of the overall representation. Furthermore, when adapting to the multi-level features of the visual encoder of a multimodal large language model, existing adaptation mechanisms typically use fixed cross-attention modules, lacking the ability to dynamically adjust the semantic focus range based on the input image content. In complex scenarios such as blurred semantic boundaries or overlapping objects, key details are easily lost during compression. Regarding cross-modal alignment, existing methods pre-set fixed language concept embeddings for each prototype, failing to consider the semantic shift of the same visual concept in different image contexts, leading to conceptual confusion during multiple rounds of understanding.
[0004] Therefore, there is an urgent need for a semantically interpretable compression method that comprehensively addresses the understanding needs of multimodal large language models, from encoding and quantization to decoupling. This method should maintain the high-fidelity reconstruction capability of images at high compression ratios, and provide downstream large language models with semantically decoupled, conceptually readable, and hierarchical topological structure-preserving structured visual knowledge representations, thereby improving the accuracy and robustness of visual understanding. Summary of the Invention
[0005] The purpose of this invention is to provide a semantically interpretable compression method for visual understanding of multimodal large language models, in order to solve at least some of the problems existing in the prior art.
[0006] The technical solution, a semantically interpretable compression method for visual understanding of multimodal large language models, includes the following steps:
[0007] During the encoding stage, a frozen multimodal large language model visual encoder is used to extract multi-level semantic feature maps of the input image. Through a dynamic object-aware adapter, fine-grained spatial location encoding and object-aware tokens adaptively generated according to the image content are gradually integrated into the multi-level semantic feature maps in a zero-initialization gated cross-attention manner to obtain the adapted feature map.
[0008] The adapted feature map is subjected to semantic-visual coupling projection to obtain a projected feature map. The projected feature map is then subjected to tree-like soft quantization using a self-organizing evolutionary cross-modal shared semantic prototype library to generate a compact representation consisting of a hierarchical semantic weight distribution matrix and a sparse prototype index. Each prototype vector in the prototype library is associated with a natural language concept phrase dynamically output by the language generator.
[0009] In the decoding stage, the compact representation is used for back projection and pixel decoding to reconstruct a high-fidelity reconstructed image;
[0010] During the training phase, the reconstruction loss, hierarchical semantic topology preservation loss, and context modulation cross-modal semantic decoupling loss between the high-fidelity reconstructed image and the real image are jointly calculated. The modules other than the frozen visual encoder are trained and optimized to obtain the image compression model.
[0011] In the inference application stage, the image to be compressed is input into the image compression model to obtain the compact representation; the compact representation is used as a visual semantic summary, concatenated with text instructions, and input into the embedding layer of the downstream multimodal large language model to support visual question answering and semantic understanding.
[0012] Beneficial effects: While maintaining high compression ratio and high-fidelity reconstruction, this invention provides interpretable visual representations for multimodal large language models that are semantically decoupled, conceptually readable, and have a clear topological structure, thereby improving the accuracy and out-of-distribution robustness of visual question answering. Attached Figure Description
[0013] Figure 1 This is a flowchart of the overall solution of the present invention. Detailed Implementation
[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0015] like Figure 1 As shown, a semantically interpretable compression method for visual understanding of multimodal large language models specifically includes:
[0016] S1: In the encoding stage, the frozen multimodal large language model visual encoder is used to extract multi-level semantic feature maps of the input image. Through a dynamic object-aware adapter, fine-grained spatial location encoding and object-aware tokens adaptively generated according to the image content are gradually integrated into the multi-level semantic feature maps in a zero-initialization gated cross-attention manner to obtain the adapted feature map.
[0017] In this embodiment, the encoding stage first performs multi-level semantic feature extraction and dynamic object-aware adaptation processing on the input image to generate an adapted feature map that integrates fine-grained spatial location information and object-level semantic context.
[0018] Specifically, the input image data to be compressed is first acquired and then fed into a multimodal large language model visual encoder with frozen parameters for forward propagation. This visual encoder can employ pre-trained architectures such as CLIP-ViT or BLIP-2 visual towers, with multiple intermediate layers outputting multi-level semantic feature map sets with different receptive fields and semantic abstraction levels. Lower-level feature maps tend to encode local details such as texture and edges, while higher-level feature maps contain more complete semantic representations of object parts or even the entire entity. These different levels of feature maps differ in channel dimension and spatial resolution, collectively forming a complete feature representation spectrum from low-level vision to high-level semantics.
[0019] After acquiring the multi-level semantic feature maps, they are fed into a dynamic object-aware adapter for processing. Unlike traditional fixed-adaptation mechanisms, the dynamic object-aware adapter in this application has the ability to adaptively adjust the semantic focus range according to the image content. Furthermore, the adapter internally maintains a set of learnable object query vectors, each of which can be considered an implicit candidate object detector. Cross-attention calculation is performed between the object query vector set and the semantic feature map of the current level, enabling the query vectors to actively aggregate region information with high semantic affinity from the feature map, generating a set of initial object-aware tokens. These tokens implicitly encode the approximate location and semantic attributes of possible visual entities in the image in vector form.
[0020] To enable object-aware tokens to more accurately focus on high semantic density regions in an image, this scheme introduces object-oriented prior guidance based on the frozen encoder's own attention mechanism. For example, a cross-modal attention map output from an intermediate layer of the frozen visual encoder is collected. This attention map records the distribution of attention intensity for different regions of the image learned by the visual encoder during image-text pair pre-training tasks. A region semantic saliency map is extracted from this attention map, specifically obtained by mean pooling or taking the maximum response value along the text query dimension. High response positions on this saliency map often correspond to image regions containing clear semantic concepts. This region semantic saliency map is used as object-oriented prior data to perform weighted guided updates on the initial object-aware tokens. In a preferred embodiment, the weighted guided update is achieved through element-wise multiplication: first, thresholding is performed on the region semantic saliency map to filter out noise responses and obtain a binarized mask; then, this mask is multiplied by the corresponding spatial component of the initial object-aware token, thereby strengthening the token response falling within high semantic regions and suppressing interference from background or semantically ambiguous regions. After weighted guided updates, object-aware tokens that focus on high semantic density regions are obtained.
[0021] Simultaneously, fine-grained spatial location encoding data needs to be generated to inject accurate spatial coordinate information into the adapter. According to a further improvement in this embodiment, the fine-grained spatial location encoding does not simply employ fixed-frequency sinusoidal location encoding, but rather combines fixed sinusoidal encoding with a set of learnable scaling factors. Specifically, for each spatial coordinate position, multi-frequency sinusoidal encoding values are calculated based on its horizontal and vertical coordinates, and then each frequency component is scaled and adjusted using learnable scaling factors, enabling the network to adaptively learn the spatial sensitivity best suited to the current feature layer level. The final generated fine-grained spatial location encoding data is dimensionally aligned with the feature map, preserving pixel-by-pixel absolute and relative positional information.
[0022] Subsequently, the three types of data prepared above are fed into a zero-initialization gated cross-attention module for fusion. In this module, the multi-level semantic feature map itself is used as the query, and the result of concatenating the fine-grained spatial location encoding data and the focused object-aware token along the channel or spatial dimension is used as the key and value, and cross-attention calculation is performed. The dot product operation between the query and the key generates a semantic affinity matrix, which measures the degree of attention of each position in the feature map to a specific location encoding or object token. The key is that this module sets up a lightweight gated network, which takes the semantic affinity matrix or directly the query features as input, and predicts an output channel-wise gated coefficient map with the same spatial size and number of channels as the feature map through several convolutional or fully connected layers. All parameters of this gated coefficient map are initialized to zero, ensuring that the adapter output is equivalent to the original feature map in the initial stage of training, thereby ensuring the stability of the training process. After the gated coefficient map is processed by the activation function, the output of cross-attention is weighted and modulated channel-wise, enabling the network to dynamically determine the strength of fusing location information and object context according to local semantic content. Finally, the modulated features are residually concatenated with the original input feature map, and layer normalization is performed to obtain the adapted feature map for that layer. The above process is performed progressively on multiple layers of feature maps, ultimately outputting a complete set of adapted feature maps that preserves the original semantic structure while embedding token-level details that are precisely aligned with the layout of image objects.
[0023] S2: Perform semantic-visual coupling projection on the adapted feature map to obtain a projected feature map, and use a self-organizing evolutionary cross-modal shared semantic prototype library to perform tree-like soft quantization on the projected feature map to generate a compact representation composed of a hierarchical semantic weight distribution matrix and sparse prototype indexes; each prototype vector in the prototype library is associated with a natural language concept phrase dynamically output by the language generator.
[0024] After completing multi-level feature adaptation processing, the encoding stage enters the crucial step of compressing the adapted feature maps into a compact and semantically interpretable representation. This step first requires mapping the adapted feature maps to an embedding space shared with the semantic prototype, i.e., performing semantic-visual coupled projection.
[0025] Specifically, the adapted feature map output from the previous stage is collected and input into a carefully designed bilinear decoupled projection network. The purpose of this projection network is to explicitly separate the mixed subject semantic information from the background context information in the feature map, thus avoiding interference from non-semantic factors such as background textures on subsequent semantic quantization. In one optional implementation, the bilinear decoupled projection network contains two parallel linear transformation branches: the first branch extracts the subject semantic components, and the second branch extracts the background context components. The adapted feature map undergoes linear mapping and activation by the first branch to obtain a semantic vector; after linear mapping and activation by the second branch, a context vector is obtained. These two vectors have the same size in the feature dimension. Subsequently, the semantic vector and the context vector are coupled and added, i.e., an element-wise summation operation is performed, so that they are reintegrated in a unified embedding space. The result of the coupled addition is the projected feature map, where the vector (denoted as z) at each spatial location lies in the same d-dimensional embedding space as the semantic prototype library. This projection process provides an unbiased and semantically pure feature representation for subsequent prototype matching.
[0026] After acquiring the projected feature map, a tree-like soft quantization process is performed on it using a self-organizing evolutionary cross-modal shared semantic prototype library to generate a compact representation. According to one aspect of this application, the self-organizing evolutionary cross-modal shared semantic prototype library consists of a set of learnable superclass semantic anchor vectors, denoted as set E. Unlike traditional static prototype libraries, each prototype vector in this library is dynamically associated with a natural language concept phrase describing the text output by a fine-tuned small language generator, such as "metallic wheel hub" or "semi-transparent glassware." This language generator is updated synchronously during training, allowing the concept represented by the prototype to adaptively adjust as the data distribution evolves, ensuring that the output phrase always maintains natural language readability and guarantees the openness of semantic interpretation.
[0027] Furthermore, the tree-like softmax process is divided into two levels. First, each local feature vector z of the projected feature map is traversed. In the first-level tree-like Gumbel-Softmax operation, z interacts with a learnable coarse-grained grouping matrix to calculate the probability distribution of z belonging to each preset coarse-grained prototype group (e.g., "texture class", "structure class", "entity class"). By introducing random perturbations to the Gumbel distribution and performing Softmax normalization, this process can output approximately discrete yet differentiable group assignment probabilities, obtaining coarse-grained group membership data. Based on the assignment results of the first level, the specific coarse-grained group to which z belongs is determined. Within this group, the second-level Gumbel-Softmax operation is performed to calculate the similarity score between z and each fine-grained prototype vector within the group, and the fine-grained assignment weights are also output via Gumbel-Softmax. Combining the two-level assignment results, a hierarchical semantic weight distribution vector can be generated for each local location on the feature map. Arranging the distribution vectors of all locations according to spatial structure forms the hierarchical semantic weight distribution matrix W. The matrix itself is a structured semantic parsing of visual content, with each element representing the confidence level that the corresponding local feature belongs to a specific readable concept.
[0028] However, relying solely on prototype weighted combination often fails to fully reconstruct all the information in the projected feature map, resulting in quantization residuals. In traditional methods, these residuals are directly preserved as unstructured numerical codes, compromising the overall interpretability of the representation. According to a further improvement in this embodiment, this scheme establishes a pre-defined second-level sub-concept prototype library. The difference between the projected feature map z and the feature vector reconstructed based on the weight distribution matrix W is calculated to obtain residual data. This residual data is mapped to a query vector through a shallow coding network and its similarity is calculated with all sub-concept prototypes in the second-level sub-concept prototype library. After another Gumbel-Softmax operation, sparsely assigned weights are output. The top K prototype indices with the largest weights and their corresponding weights are selected to form the sparse prototype index R. Thus, the residual information is also converted into a sparse form expressed by interpretable sub-concept prototype combinations. Finally, the data volume composed of the hierarchical semantic weight distribution matrix W and the sparse prototype index R is defined as a compact representation U. At this point, the image visual information is completely compressed into a latent code consisting of explicit, readable semantic prototypes and their combination relationships.
[0029] S3: Decoding stage, using the compact representation to perform back projection and pixel decoding processing to reconstruct a high-fidelity reconstructed image.
[0030] The task of the decoding stage is to recover a high-fidelity reconstructed image from the compact representation U. The quality of this reconstructed image directly serves as a supervisory basis for the information preservation capability of the encoding stage.
[0031] Specifically, a backprojection operation is first performed to reconstruct a continuous feature map. The hierarchical semantic weight distribution matrix W and the sparse prototype index R in the compact representation U are acquired. For the weight distribution matrix W, the prototype vectors in the self-organizing evolution cross-modal shared semantic prototype library are weighted and summed according to their recorded weight coefficients. This step is equivalent to approximating the main projection feature with a linear combination of semantic concepts. For the sparse prototype index R, the corresponding sub-concept prototype vector is found from the second-level sub-concept prototype library based on the index value, and weighted according to its corresponding sparse weights. The feature vectors obtained in the above two steps are linearly summed in the embedding space to fuse mainstream semantic information and detailed residual information, thus recovering a continuous feature map with dimensions completely consistent with the projected feature map.
[0032] Subsequently, the reconstructed continuous feature maps are input into a lightweight pixel decoder for upsampling and refinement. This lightweight pixel decoder mainly consists of a series of upsampling convolutional layers and adaptive group normalization layers stacked together. The upsampling convolutional layers are responsible for progressively increasing the spatial resolution of the feature maps. At each upsampling stage, the feature map size is enlarged through operations such as transposed convolution or subpixel convolution, while simultaneously compressing the number of channels. An adaptive group normalization layer is inserted after each convolutional layer to dynamically adjust the normalization parameters based on feature statistics, enhancing the decoder's adaptability to different feature distributions. After step-by-step decoding, a high-fidelity reconstructed image with the same spatial dimensions as the original input image is finally output. This reconstructed image should visually maintain a high degree of consistency with the original image, and its fidelity is quantified and measured by the loss function in subsequent training stages.
[0033] S4: During the training phase, the reconstruction loss, hierarchical semantic topology preservation loss, and context modulation cross-modal semantic decoupling loss between the high-fidelity reconstructed image and the real image are jointly calculated. The modules other than the frozen visual encoder are trained and optimized to obtain the image compression model.
[0034] The core objective of the training phase is to use multiple loss functions to jointly drive the parameter optimization of all trainable modules except the frozen visual encoder, and finally obtain a high-performance image compression model.
[0035] In this embodiment, the training process first calculates the reconstruction loss between the high-fidelity reconstructed image and the real image. Preferably, this reconstruction loss is not a single metric, but a composite loss function composed of weighted combinations of multiple loss terms. Specifically, it includes: pixel-level L1 loss, which directly constrains the absolute difference in pixel values between the reconstructed image and the original image; perceptual loss LPIPS, which uses a pre-trained deep network to extract high-level features and compares the differences between the two images in the perceptual feature space to enhance visual semantic fidelity; adversarial loss, which introduces a discriminator network to engage in adversarial game with the generator (i.e., the encoder-decoder structure in this scheme), prompting the distribution of the reconstructed image to approach the distribution of the real image and sharpening texture details; and CLIP-aligned language contrast loss, which uses a pre-trained CLIP model to extract the concept embeddings of the reconstructed image and the corresponding text description, maximizing the cosine similarity between the two, and ensuring that the reconstructed image remains consistent with the original image at the cross-modal semantic level.
[0036] In addition to reconstruction loss, this scheme also introduces two key semantic consistency regularization losses to enhance the semantic structural clarity and cross-modal parsing of compressed representations.
[0037] The first term is the hierarchical semantic topology preservation loss. According to one aspect of this application, the calculation of this loss is first based on the hierarchical semantic weight distribution matrix W, calculating the semantic prototype co-occurrence similarity between any two spatial locations i and j on the feature map. Specifically, the semantic weight distribution vectors of locations i and j can be regarded as probability distributions of the prototype set, and the similarity between these two distributions (such as the Bach coefficient or cosine similarity) is calculated to form a semantic prototype co-occurrence similarity matrix. This matrix reflects the association strength between any two points in the compressed domain in the semantic concept space. Simultaneously, the cross-modal attention map generated by the frozen visual encoder is collected, and a connection threshold is set according to the attention intensity. High attention feature points belonging to the same object part or region are connected to construct hyperedges. Multiple hyperedges together constitute a semantic hypergraph data, which reflects the object-level topological structure in the original high-dimensional feature space. To measure and maintain the consistency of the two structures, the normalized mutual information metric between the semantic prototype co-occurrence similarity matrix and the semantic hypergraph data is first calculated. The higher the value, the more consistent the global clustering structure. Furthermore, hierarchical clustering is performed on the semantic weight distribution matrix to form a semantic segmentation tree that reflects the gradual aggregation of features from fine-grained concepts to coarse-grained concepts. Similarly, the cross-modal attention map is also hierarchically segmented to obtain an attention segmentation tree. The Wasserstein distance between the two tree-like segmentations is calculated, which measures the minimum cost required to transfer the structure of one tree to another, thus accurately quantifying the degree of preservation of part-whole hierarchical relationships. Finally, the normalized mutual information metric and the Wasserstein distance are weighted and summed to obtain the hierarchical semantic topology preservation loss.
[0038] The second term is the context-modulated cross-modal semantic decoupling loss. First, the global semantic vector of the image is obtained after global pooling of the adapted feature map. This vector summarizes the scene context of the entire image. A static language concept description embedding is initialized for each semantic prototype in the prototype library. A lightweight transform network is used to map the global semantic vector of the image to a dynamic bias term with the same dimension as the language embedding. This dynamic bias term is added to the initial language embedding and layer normalization is performed to obtain the context-modulated dynamic language embedding. This means that the language vector used for cross-modal alignment of the same prototype concept will adaptively shift in different image contexts such as "sunny day outdoors" and "dim indoors". Next, the local feature vectors of the projected feature map are collected. For each visual feature, the prototype corresponding to the largest weight in its semantic weight distribution matrix W is taken as its belonging concept. The dynamic language embedding of this prototype after context modulation is used as a positive sample, and the dynamic language embeddings of all other prototypes are used as negative samples. Context-sensitive contrastive learning similarity calculation is performed to bring the visual feature closer to the positive embedding and push away the negative embedding. Furthermore, to prevent information entanglement between the visual and linguistic representations of each prototype, the similarity matrices of each pair of prototypes in visual space and linguistic space are calculated. Mutual information minimization orthogonal constraints are applied to these two matrices, forcing the visual and linguistic representations of different concepts to statistically tend towards independence. Combining the contrastive learning term and the orthogonal constraint term, a context-modulated cross-modal semantic decoupling loss is obtained.
[0039] Finally, the reconstruction loss, hierarchical semantic topology preservation loss, and context modulation cross-modal semantic decoupling loss are weighted and summed with preset weight coefficients to form the total loss function. Driven by this total loss function, end-to-end backpropagation and gradient updates are performed on all trainable modules except the frozen visual encoder, and the model is iteratively optimized until it converges, ultimately obtaining the image compression model.
[0040] S5: In the inference application stage, the image to be compressed is input into the image compression model to obtain the compact representation; the compact representation is used as a visual semantic summary, concatenated with text instructions, and input into the embedding layer of the downstream multimodal large language model to support visual question answering and semantic understanding.
[0041] After the model training is completed, the inference application stage will directly use the obtained image compression model to perform forward processing on any image to be compressed in order to extract its visual semantic summary and serve the downstream multimodal large language model understanding task.
[0042] Specifically, in the inference application phase, any input image to be compressed is acquired and fed into the trained image compression model. The model's frozen visual encoder, optimized dynamic object-aware adapter, projection network, and tree quantization module process the image sequentially, generating a compact representation U through rapid forward inference. This U contains a hierarchical semantic weight distribution matrix W and a sparse prototype index R. This compact representation is the visual semantic summary of the original image; it records, in a highly compressed form, the various visual concepts present in the image and their spatial distribution using interpretable semantic prototype combinations.
[0043] Furthermore, to enable interaction with a multimodal large language model, this visual semantic summary needs to be converted into a token embedding format acceptable to the large language model. According to one aspect of this application, the hierarchical semantic weight distribution matrix W and the sparse prototype index R are flattened into one-dimensional sequences, and then projected onto a space with the same word embedding dimension as the downstream large language model through a linear mapping layer, resulting in two sets of embedding token sequences. These two sets of embedding token sequences are then concatenated to form a complete visual embedding sequence. When performing visual question answering or semantic understanding tasks, the text instructions containing the task description are converted into text token embedding sequences through a tokenizer. The complete visual embedding sequence and the text token embedding sequence are then concatenated along the sequence dimension to form a multimodal joint input sequence. This joint input sequence is then fed into the embedding layer of the downstream multimodal large language model and subsequent Transformer processing layers. Since the visual information has been decoupled in advance into a structured knowledge representation with a clear semantic direction, the large language model does not need to re-infer object relationships from dense continuous pixel features. Its internal self-attention mechanism can directly locate and infer visual entities based on the prototype weight distribution, which significantly improves the accuracy of visual question answering and shows stronger semantic understanding robustness when facing out-of-distribution image samples.
[0044] The advantages of this invention lie in its integration of image compression and interpretable visual understanding using large language models, addressing the core pain points of traditional compression frameworks: opaque latent semantic representations that are difficult for large language models to directly parse. Specifically, at the encoding end, through the collaborative design of a dynamic object-aware adapter and a self-organizing evolutionary semantic prototype library, the compression process itself constitutes a structured semantic decomposition of visual content. The generated compact representation is entirely composed of prototype concepts and sparse sub-concepts that can be described by natural language, eliminating the uninterpretable continuous residuals in traditional methods and achieving transparent organization of information while maintaining a high compression ratio. Furthermore, hierarchical semantic topology preservation regularization ensures that the compression bottleneck does not disrupt the part-whole hierarchical relationship and spatial topological boundaries between objects, allowing the compact representation to faithfully preserve the high-level semantic structure. Context-modulated cross-modal semantic decoupling endows each semantic component with the dynamic cross-modal alignment capability to closely follow the image context. The same concept can adaptively adjust its semantic direction in different scenarios, enhancing the stability of multi-turn understanding. Ultimately, these mechanisms work together to enable multimodal large language models to perform accurate reasoning directly based on structured, interpretable semantic summaries, without inferring object relationships from dense pixel sequences. This improves both visual question answering accuracy and robustness to understanding out-of-distribution samples, balancing compression efficiency, reconstruction quality, and downstream understanding performance.
[0045] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.
Claims
1. A semantically interpretable compression method for visual understanding of multimodal large language models, characterized in that, include: During the encoding stage, a frozen multimodal large language model visual encoder is used to extract multi-level semantic feature maps of the input image. Through a dynamic object-aware adapter, fine-grained spatial location encoding and object-aware tokens adaptively generated according to the image content are gradually integrated into the multi-level semantic feature maps in a zero-initialization gated cross-attention manner to obtain the adapted feature map. The adapted feature map is subjected to semantic-visual coupling projection to obtain a projected feature map. The projected feature map is then subjected to tree-like soft quantization using a self-organizing evolutionary cross-modal shared semantic prototype library to generate a compact representation consisting of a hierarchical semantic weight distribution matrix and a sparse prototype index. Each prototype vector in the prototype library is associated with a natural language concept phrase dynamically output by the language generator. In the decoding stage, the compact representation is used for back projection and pixel decoding to reconstruct a high-fidelity reconstructed image; During the training phase, the reconstruction loss, hierarchical semantic topology preservation loss, and context modulation cross-modal semantic decoupling loss between the high-fidelity reconstructed image and the real image are jointly calculated. The modules other than the frozen visual encoder are trained and optimized to obtain the image compression model. In the inference application stage, the image to be compressed is input into the image compression model to obtain the compact representation; the compact representation is used as a visual semantic summary, concatenated with text instructions, and input into the embedding layer of the downstream multimodal large language model to support visual question answering and semantic understanding.
2. The method according to claim 1, characterized in that, The adapted feature map obtained in the encoding stage includes: Based on the multi-level semantic feature map, cross-attention calculation is performed between the learnable object query vector set and the feature map to generate an initial object-aware token. The cross-modal attention map output from the intermediate layer of the frozen visual encoder is collected, and the region semantic saliency map is extracted from it as object-oriented prior data. The object-oriented prior data is then used to perform weighted guided updates on the initial object-aware token to obtain an object-aware token focused on a high semantic density region. Generate fine-grained spatial location encoded data, which is a combination of fixed sinusoidal location codes and learnable scale factors; Using the multi-level semantic feature map as the query, and the spatial location encoded data and the object-aware token concatenated as the key and value, the query is input into the zero-initialization gated cross-attention module. Inside the module, the semantic affinity between the query and the key is calculated, and a channel-wise gated coefficient map is predicted via a lightweight gated network. The cross-attention output is weighted and modulated channel by channel using the gating coefficient map, and then residually connected and layer normalized with the original feature map to obtain the adapted feature map.
3. The method according to claim 2, characterized in that, The weighted guided update of the initial object-aware token using object-oriented prior data includes: The cross-modal attention map is thresholded to obtain a binary region saliency map. The region saliency map is then multiplied element-wise with the initial object-aware token to achieve weighted guidance.
4. The method according to claim 1, characterized in that, In the encoding stage, the adapted feature map is semantically-visually coupled and projected to obtain a projected feature map, including: The adapted feature map is input into a bilinear decoupled projection network to separate the main semantic components from the background context components, thereby obtaining semantic vectors and context vectors respectively. After performing linear transformations on the semantic vector and the context vector respectively, a coupled addition process is performed to obtain the projection feature map.
5. The method according to claim 1, characterized in that, The generation of a compact representation during the encoding stage includes: A set of learnable superclass semantic anchor vectors is initialized to form the self-organizing evolutionary cross-modal shared semantic prototype library; a fine-tuned small language generator is used to dynamically generate natural language concept phrase description text corresponding to the training state for each prototype vector. Collect each local feature vector of the projected feature map, and assign the local feature vector to a preset coarse-grained prototype group through the first layer tree-structured Gumbel-Softmax operation. Calculate the group assignment probability to obtain the coarse-grained group membership data. Within a defined coarse-grained prototype group, the local feature vectors are assigned to fine-grained prototypes within the group through a second-level Gumbel-Softmax operation, and the fine-grained assignment weights are calculated. Combining the two-level assignment results, a semantic weight distribution matrix with a hierarchical semantic structure is generated. Calculate the residual data between the projected feature map and the features reconstructed based on the semantic weight distribution matrix, and perform secondary Gumbel-Softmax soft quantization on the residual data using a preset second-level sub-concept prototype library to obtain the sparse prototype index. The semantic weight distribution matrix is combined with the sparse prototype index to form the compact representation.
6. The method according to claim 5, characterized in that, The secondary Gumbel-Softmax soft quantization process includes: The residual data is mapped into a query vector through a shallow coding network, and its similarity with the sub-concept prototypes in the second-level sub-concept prototype library is calculated. The Gumbel-Softmax outputs sparsely assigned weights, and the top K prototype indices with the largest weights and their corresponding weights are selected to form the sparse prototype index.
7. The method according to claim 1, characterized in that, The decoding stage utilizes compact representation to reconstruct a high-fidelity image, including: Based on the semantic weight distribution matrix and sparse prototype index in the compact representation, the prototype vector is weighted and summed according to the semantic weight distribution matrix, and the sub-concept prototype is looked up and weighted according to the sparse prototype index. The results of the two are linearly summed to obtain a continuous feature map. The continuous feature map is input into a lightweight pixel decoder, and then processed by an upsampling convolutional layer and an adaptive group normalization layer to obtain the high-fidelity reconstructed image.
8. The method according to claim 1, characterized in that, The hierarchical semantic topology preservation loss during the training phase is calculated through the following steps: Based on the hierarchical semantic weight distribution matrix, calculate the semantic prototype co-occurrence similarity matrix between each spatial location; Based on the cross-modal attention map generated by the frozen visual encoder, feature points belonging to the same object component or region are connected according to the attention intensity to construct semantic hypergraph data; Calculate the normalized mutual information metric between the semantic prototype co-occurrence similarity matrix and the semantic hypergraph data; Hierarchical clustering is performed on the semantic weight distribution matrix, and corresponding hierarchical segmentation is performed on the cross-modal attention map. The Wasserstein distance between the two tree-like segmentations is calculated. The hierarchical semantic topology preservation loss is obtained by weighted summation of the normalized mutual information metric and the Wasserstein distance.
9. The method according to claim 1, characterized in that, The context modulation cross-modal semantic decoupling loss during the training phase is calculated through the following steps: The global semantic vector of the image is obtained by global pooling of the adapted feature map; A language concept description embedding is initialized for each semantic prototype in the prototype library. The global semantic vector of the image is mapped to a dynamic bias term using a lightweight transform network. After being added to the initial language embedding, layer normalization is performed to obtain a context-modulated dynamic language embedding. Collect the local feature vectors of the projected feature map. For each visual feature, take the dynamic language embedding corresponding to the prototype with the largest semantic weight as a positive sample and take the dynamic language embedding of other prototypes as negative samples. Perform context-sensitive contrast learning similarity calculation. Calculate the similarity matrix of each pair of prototypes in the visual space and the linguistic space, and apply orthogonal constraints to minimize mutual information. By combining the similarity calculation results of contrastive learning with the orthogonal constraint processing results, the context modulation cross-modal semantic decoupling loss is obtained.
10. The method according to claim 1, characterized in that, The joint loss calculation during the training phase includes: The reconstruction loss is constructed as a weighted combination of pixel-level L1 loss, perceptual loss LPIPS, adversarial loss, and CLIP-aligned language contrast loss; The reconstruction loss, the hierarchical semantic topology preservation loss, and the context modulation cross-modal semantic decoupling loss are weighted and summed with preset weights to form a total loss function to optimize the image compression model. Furthermore, in the inference application stage, the compact representation is concatenated with the text instruction. Specifically, the semantic weight distribution matrix and sparse prototype index are flattened and linearly mapped to an embedded token sequence, which is then concatenated with the token embedding of the text instruction and input into the embedding layer of the downstream multimodal large language model.
