A system and method for compressed domain visual language large model inference

Abstract

The application discloses a system and method for compressed domain visual language large model reasoning, which comprises an image coding and decoding module, a perception enhancement module, a task-driven adaptation module and an anti-packet loss transmission module: the image coding and decoding module outputs compressed latent features with semantic preservation and compact structure; the perception enhancement module performs semantic enhancement and representation reorganization on the latent features to improve their adaptability to the large model; the task-driven adaptation module realizes precise docking and task-level mapping of the latent features and the visual language large model through a lightweight adapter; and the anti-packet loss transmission module is used to improve the robustness of the transmission of the latent features across networks. The application realizes low-bit-rate, low-delay compressed domain feature transmission and efficient large model reasoning adaptation while ensuring the integrity of semantic expression, is suitable for visual language large model cloud reasoning and other scenes, and has important application value.

Description

Technical Field

[0001] This invention relates to the field of image coding and visual language large model inference fusion, and particularly to a system and method for visual language large model inference in the compressed domain. Background Technology

[0002] Images have become a ubiquitous medium for recording and disseminating visual information, driven by ubiquitous cameras, smart devices, and connected sensors. According to the latest statistics, the number of photographs taken globally is projected to exceed 2.1 trillion by 2025, equivalent to an average of approximately 575 million images being taken worldwide every day. Beyond human consumption, images are also a primary data form in modern artificial intelligence processes, supporting human visual perception and communication, as well as core tasks such as visual-language question answering based on multimodal large language models. This explosive growth in data, coupled with the continuous improvement in image resolution, is placing an increasingly heavy burden on storage, transmission, and downstream computation of large visual-language models.

[0003] Neural network-based image compression methods have surpassed traditional hand-crafted codecs such as JPEG, BPG, and VVC / H.266 in recent years. Current mainstream learning-based image compression methods continuously improve image reconstruction fidelity and bit efficiency by stacking deeper neural networks and designing increasingly complex contextual entropy models. However, modern visual data is not only used by humans but also widely consumed by large visual language models. In large visual language model inference systems, maintaining task-relevant semantics and achieving high perceptual quality are equally crucial. Traditional processing workflows typically input reconstructed images directly into large visual language models, treating compression and inference as two separate stages. This design ignores the intrinsic connection between the codec and the large visual language model, and due to its mean squared error (MSE)-driven reconstruction objective, it often fails to retain the semantic information required by downstream large visual language models. This has prompted the development of a unified framework that can perform semantic inference directly in the compression domain and is adapted to large visual language models. Meanwhile, the number of parameters in advanced large visual language models is exploding: from early convolutional neural networks with tens of thousands of parameters to today's Transformer models with hundreds of millions or even billions of parameters. This trend makes on-device deployment increasingly impractical, driving many applications to move to cloud-based inference, transmitting compressed images to data centers for large-scale visual language model processing (such as visual question answering and image captioning). In such scenarios, the ability of a single compressed bitstream to serve large-scale visual language model inference with minimal decoding and model overhead is extremely valuable. Summary of the Invention

[0004] To address the technical problems in existing technologies, such as decoupling of image compression and large model inference processes, inability of latent features to be directly adapted to large models, low inference efficiency, and weak resistance to packet loss, the core innovation of this invention lies in proposing an integrated technical solution for directly integrating latent features from image compression into large models. Through a specially designed encoding module and a lightweight adaptation module, precise integration between compressed domain features and large models is achieved without pixel-level reconstruction.

[0005] This invention provides a system for large-scale visual language model inference in the compressed domain. It obtains high-dimensional latent features through image encoding, eliminating the need for image decoding to reconstruct the image. Instead, the high-dimensional latent features are directly input into a lightweight adapter unit. The output of the lightweight adapter unit can then be directly integrated into the intermediate hidden layers of the visual model for visual language model inference. This system can be used for tasks such as visual image understanding and question answering. Its main innovation lies in the fact that the latent features obtained from image compression can be directly integrated into a large model, enabling visual language task inference without the need for decoding-end reconstruction, thus significantly saving image decompression time.

[0006] It includes an image encoding / decoding module, a task-driven adaptation module, a perception enhancement module, and a packet loss-resistant transmission module;

[0007] The image encoding and decoding module is a lossy image encoding and decoding architecture, including an encoder unit, a priori encoder unit, an encoding-side quantization unit, and an arithmetic encoding unit that are connected in series to form the encoding end, and a priori decoder unit and an arithmetic decoding unit that are connected in series to form the decoding end.

[0008] The encoder unit is used to convert the input image into a high-dimensional latent feature representation;

[0009] The super prior encoder unit is used to perform convolution processing on high-dimensional latent features to generate super latent features;

[0010] The encoding-side quantization unit includes a latent quantization subunit and a super-prior quantization subunit. The quantization subunit and the super-prior latent quantization subunit are used to quantize latent features and super-prior latent features, respectively, and output quantized latent features that adapt to the semantic requirements of large models.

[0011] The arithmetic coding unit is used to encode the quantized latent features and the quantized prior latent features into a compact bit stream;

[0012] The super prior decoder unit is used to receive the transmitted bit stream and decode the quantized super latent features for convolution processing to obtain the parameters of the Gaussian entropy model, so as to support high-dimensional latent feature decoding.

[0013] The arithmetic decoding unit is used to decode the transmitted bit stream and the parameters of the Gaussian entropy model to obtain quantized high-dimensional latent features.

[0014] The task-driven adaptation module includes a lightweight adapter unit;

[0015] The lightweight adapter unit directly connects to the quantized high-dimensional latent features obtained by the arithmetic decoding unit at the decoding end. The lightweight adapter unit achieves accurate matching between the high-dimensional latent feature domain and the feature domain of the visual large model, and directly connects the high-dimensional latent features to the visual model and the downstream large model to complete inference without pixel-level reconstruction.

[0016] The perception enhancement module is used in the image encoding and decoding module training process to specifically optimize the semantic representation ability of the visual language large model adaptability of high-dimensional latent features and output semantically enhanced high-dimensional latent features.

[0017] The packet loss-resistant transmission module is embedded between the quantized high-dimensional latent variables output by the arithmetic decoding unit and the input of the lightweight adapter unit. The high-dimensional latent features are masked by the random mask fine-tuning unit to simulate the packet loss scenario during transmission. This is used to ensure the integrity of the high-dimensional latent features during transmission, improve the robustness of channel transmission, and ensure the stability of the visual language large model inference.

[0018] The lightweight adapter unit in the task-driven adaptation module can adapt to two or more different types of visual models without modifying the network structure or native parameters of the visual model. It serves as a feature connection link between the quantized high-dimensional latent features output by the arithmetic decoding unit and the visual model, realizing the conversion and access of the compressed domain latent features to the feature domain of the visual model.

[0019] The perception enhancement module is deployed during the training phase of the image encoding and decoding module. It is used to optimize the semantic representation adaptation capability of high-dimensional latent features to large visual language models and output semantically enhanced high-dimensional latent features.

[0020] The task-driven adaptation module is a dedicated adaptation structure for compressing domain latent features into a large visual language model, including a lightweight adapter unit.

[0021] The lightweight adapter unit is a lightweight convolutional network. The input of the lightweight adapter unit is directly connected to the quantized high-dimensional latent features output by the image encoding and decoding module. Through a configurable dimension transformation structure, the dimensions of the latent features and the input features of the large model are accurately matched. The output is directly connected to the feature input layer of the visual model and the downstream visual language large model.

[0022] The parameter scale of the lightweight adapter unit is only 1% to 5% of the parameter scale of the downstream large model, which reduces computational overhead and ensures inference efficiency while achieving accurate adaptation.

[0023] The packet loss-resistant transmission module includes a random mask fine-tuning unit, which generates a random binary mask to perform element-level masking on the quantized high-dimensional latent features to simulate the packet loss scenario. This drives the lightweight adapter unit to learn the global context completion capability based on incomplete latent features, ensuring that effective features adapted to large models can still be output even when some features are lost.

[0024] The arithmetic coding unit distributes the spatial tokens of the quantized high-dimensional potential features into more than N data packets through uniform grouping before outputting them, thus avoiding the loss of key semantic features of large models due to packet loss in consecutive blocks.

[0025] This invention also provides a method for image encoding and latent feature domain inference based on a large model of visual language using neural networks, implemented using the aforementioned system, and comprising the following steps:

[0026] Step 1, image encoding and compression, including:

[0027] Step 1.1: Read in the test image, normalize the color value range, and then input it into the system;

[0028] Step 1.2: The image is converted into a high-dimensional latent feature representation by the encoder unit of the image encoding and decoding module. The dimension of the high-dimensional latent feature is preset to be the basic dimension that adapts to the input of the subsequent large visual language model.

[0029] Step 1.3: Input the high-dimensional latent features into the super-prior encoder unit to obtain the super-prior features, quantize them into integers by the coding side quantization unit, and then obtain the super-prior feature bitstream through the arithmetic coding unit;

[0030] Step 1.4: Input the prior features into the prior decoder unit to obtain the mean and variance;

[0031] Step 1.5: The high-dimensional latent feature representation is quantized into an integer by the quantization unit on the encoding side. Combined with the mean and variance obtained in step 1.4, the high-dimensional latent feature code stream is obtained through the arithmetic coding unit. The high-dimensional latent feature code stream carries the core semantic features adapted to the large visual language model.

[0032] In step 1.5, before outputting the high-dimensional latent feature code stream, the spatial tokens of the quantized core latent features are distributed to N data packets through pseudo-random uniform grouping.

[0033] Step 2, Latent Feature Decoding and Visual Language Large Model Adaptation Inference, includes the following steps:

[0034] Step 2.1: Decode the super-prior feature bitstream using the arithmetic decoding unit to obtain the super-prior features, and input them into the super-prior decoder unit to obtain the mean and variance;

[0035] Step 2.2: The mean and variance obtained in step 2.1 are fed into the arithmetic decoding unit to decode the high-dimensional latent feature code stream and obtain the high-dimensional latent features that retain complete semantics.

[0036] Step 2.3: The high-dimensional latent features are directly fed into the lightweight adapter unit of the task-driven adaptation module, and the output feature representation compatible with the large visual language model is generated.

[0037] Step 2.4: The feature representation obtained in Step 2.3 is directly input into the visual model and the visual language large model. Combined with the text modality input, inference is performed to obtain the visual language interaction result. No pixel-level reconstruction step is required throughout the process.

[0038] In step 2.1, if there is data packet loss, the decoding process is directly initiated without retransmission. The lightweight adapter unit completes the adaptation based on the valid features received.

[0039] In step 2.4, the visual language interaction results include image descriptions or visual question-answering results. The visual language large model inference speed reduces latency by more than 30% compared to the traditional "reconstructing image-input large model" process.

[0040] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor performs the steps of the method, focusing on the adaptation and access steps of high-dimensional latent features and large visual language models.

[0041] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method.

[0042] The core innovation of this invention lies in obtaining high-dimensional latent features through image encoding, instead of obtaining a reconstructed image through image decoding. Instead, the high-dimensional latent features are directly input into a lightweight adapter unit. The output of the lightweight adapter unit can be directly used in the intermediate hidden layer of the visual model for inference of the visual language model. This system can be used for tasks such as visual image understanding and question answering, greatly saving image decompression time.

[0043] The core advantages of this invention lie in its construction of an integrated "compression, adaptation, and access" link, allowing latent features to be directly accessed in the large model, eliminating pixel-level reconstruction steps and reducing latency by more than 30%. It also features targeted optimization of the large model's adaptability to latent features, ensuring that inference accuracy is improved by 3% to 8% compared to traditional processes through semantic alignment, dimension matching, and distribution calibration. Furthermore, its lightweight adapter design minimizes adaptation overhead to only 1% to 5% of that of the large model, guaranteeing inference efficiency. Finally, its packet loss-resistant transmission design adapts to the transmission requirements of latent features, ensuring the stability of large model inference under complex channel conditions.

[0044] The beneficial effects of this invention include: breaking through the traditional inherent process of "compression, reconstruction, and inference," and pioneering an integrated link of "compressing latent features to directly access large models," eliminating the pixel reconstruction step and significantly reducing edge-cloud interaction latency; targeted optimization of large model adaptability across the entire link, from semantic preservation in the encoding stage to dimensional and distribution calibration in the adaptation stage, ensuring that latent features can directly drive efficient inference of large models; lightweight adapter design that balances adaptation accuracy and computational efficiency, reducing the cost of accessing large models; and packet loss-resistant transmission design that ensures inference stability under complex channels, expanding the practical application scenarios of the system. Attached Figure Description

[0045] Figure 1 This invention provides a schematic diagram of a latent feature domain inference structure for a large model of image encoding and visual language based on neural networks.

[0046] Figure 2 This is a schematic diagram of the encoder unit.

[0047] Figure 3 This is a schematic diagram of a hyper-prior encoder unit.

[0048] Figure 4 This is a schematic diagram of a hyper-prior decoder unit.

[0049] Figure 5 This is a schematic diagram of a lightweight adapter unit.

[0050] Figure 6 This is a schematic diagram of the feature extraction module. Detailed Implementation

[0051] This invention provides a system and method for large-scale visual language model inference in the compressed domain. The system includes an image encoding / decoding module, a perception enhancement module, a task-driven adaptation module, and a packet loss-resistant transmission module. The specific embodiments of this invention are described in detail below with reference to the accompanying drawings.

[0052] like Figure 1As shown, the system workflow of this invention includes four main stages: image encoding and compression, latent feature decoding, lightweight adaptation, and large model inference, without requiring pixel-level reconstruction steps throughout. In this embodiment, various multimodal datasets are used for visual language task testing, and the specific implementation steps are as follows:

[0053] Step 1: Image encoding and compression:

[0054] Step 1.1: Read in the test image, which has a resolution of 3×H×W, where H represents the height of the image and W represents the width of the image. Normalize the color values ​​to [0,1] and then input them into the system.

[0055] Step 1.2, as follows Figure 2 As shown, the image passes through the encoder unit of the image encoding / decoding module, and then continuously through the convolutional downsampling and feature extraction modules. The feature extraction module is as follows: Figure 6 As shown, the input image is progressively converted into high-dimensional latent features with 256 channels × H / 16 × W / 16.

[0056] Step 1.3, as follows Figure 3 As shown, the high-dimensional latent features are input into the super prior encoder unit to obtain super prior features, generating 128-channel ×H / 32×W / 32 super prior features. After being quantized into integers by the coding side quantization unit, the super prior feature bitstream is obtained through the arithmetic coding unit.

[0057] Step 1.4, as follows Figure 4 As shown, the super-prior features are input into the super-prior decoder unit to reconstruct the mean and variance of the Gaussian entropy model;

[0058] Step 1.5: The high-dimensional latent feature representation is quantized into an integer by the quantization unit on the encoding side. Combined with the mean and variance obtained in Step 1.4, the high-dimensional latent feature code stream is obtained by the arithmetic coding unit based on the Gaussian entropy model. The high-dimensional latent feature code stream carries the core semantic features adapted to the large visual language model. Before outputting the high-dimensional latent feature code stream, the spatial tokens of the quantized core latent features are distributed to 10 data packets through pseudo-random uniform grouping.

[0059] Step 2, Latent Feature Decoding and Visual Language Large Model Adaptation Inference:

[0060] Step 2.1: The arithmetic decoding unit decodes the prior feature code stream to obtain the prior features, and inputs them into the prior decoder unit to obtain the mean and variance. If there is data packet loss, the decoding process is directly entered without retransmission. The lightweight adapter unit completes the adaptation based on the received valid features.

[0061] Step 2.2: The mean and variance obtained in step 2.1 are sent into the arithmetic decoding unit to decode the high-dimensional latent feature code stream, and a high-dimensional latent feature with 256 channels × H / 16 × W / 16 retaining complete semantics is obtained.

[0062] Step 2.3, as follows Figure 5 As shown, the high-dimensional latent features are directly fed into the lightweight adapter unit of the task-driven adaptation module, and the output feature representation compatible with the large visual language model is output. Specifically, the lightweight adapter unit uses a lightweight convolutional network to convert the 256-channel ×H / 16 ×W / 16 high-dimensional latent features into 384-channel ×H / 4 ×W / 4 large model adaptation features, so as to achieve matching with the input feature dimension of the large visual language model.

[0063] Step 2.4: The feature representation obtained in Step 2.3 is directly input into the visual model and the visual language big model. Combined with the text modality input, inference is performed to obtain the visual language interaction result. No pixel-level reconstruction step is required throughout the process. In this embodiment, the visual language interaction result includes image description or visual question answering result. The inference speed of the visual language big model is reduced by more than 30% compared with the traditional "reconstruct image-input big model" process.

[0064] In the above process, the perceptual enhancement module is deployed during the training phase of the image encoding and decoding module, and includes a multi-scale feature extraction unit, a texture alignment loss unit, a semantic alignment loss unit, and an adversarial loss unit. The multi-scale feature extraction unit uses a pre-trained self-supervised learning visual Transformer network (DINO ViT) to extract feature maps of the input image and the reconstructed image at different levels; the texture alignment loss unit calculates the Euclidean distance metric of multi-scale texture features based on the learned perceptual patch similarity (LPIPS) index; the semantic alignment loss unit calculates the consistency of high-order semantic features based on DINO ViT features using cosine similarity; the adversarial loss unit contains a discriminator to distinguish the reconstructed image from the original image.

[0065] Table 1 shows the inference results for the visual language large model task, comparing the test results of this system with the inference results using the original image as input. Test metrics include the compression bitrate and the large model task score. Testing was conducted on six standard test sets. Higher scores indicate better performance. As shown in Table 2, compared to the traditional process of compressing and reconstructing images before inputting them into the visual language large model, latency is reduced by more than 30%.

[0066] Table 1 Comparison of Results for Visual Language Large Model Task

[0067]

[0068] Table 2 Comparison of Visual Model Inference Speed

[0069]

[0070] This invention also provides a computer program product, including computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the electronic device to perform the steps of the method, with a focus on achieving the adaptation and access of high-dimensional latent features and large visual language models.

[0071] This invention provides a system and method for reasoning in large-scale visual language models in compressed domains. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A system for reasoning in large-scale visual language models in compressed domains, characterized in that, It includes an image encoding / decoding module, a task-driven adaptation module, a perception enhancement module, and a packet loss-resistant transmission module; The image encoding and decoding module is a lossy image encoding and decoding architecture, including an encoder unit, a priori encoder unit, an encoding-side quantization unit, and an arithmetic encoding unit that are connected in series to form the encoding end, and a priori decoder unit and an arithmetic decoding unit that are connected in series to form the decoding end. The encoder unit is used to convert the input image into a high-dimensional latent feature representation; The super prior encoder unit is used to perform convolution processing on high-dimensional latent features to generate super latent features; The encoding-side quantization unit includes a latent quantization subunit and a super-prior quantization subunit. The quantization subunit and the super-prior latent quantization subunit are used to quantize latent features and super-prior latent features, respectively, and output quantized latent features that adapt to the semantic requirements of large models. The arithmetic coding unit is used to encode the quantized latent features and the quantized prior latent features into a compact bit stream; The super prior decoder unit is used to receive the transmitted bit stream and decode the quantized super latent features for convolution processing to obtain the parameters of the Gaussian entropy model, so as to support high-dimensional latent feature decoding. The arithmetic decoding unit is used to decode the transmitted bit stream and the parameters of the Gaussian entropy model to obtain quantized high-dimensional latent features. The task-driven adaptation module includes a lightweight adapter unit; The lightweight adapter unit directly connects to the quantized high-dimensional latent features obtained by the arithmetic decoding unit at the decoding end. The lightweight adapter unit achieves accurate matching between the high-dimensional latent feature domain and the feature domain of the visual large model, and directly connects the high-dimensional latent features to the visual model and the downstream large model to complete inference. The perception enhancement module is used in the image encoding and decoding module training process to specifically optimize the semantic representation ability of the visual language large model adaptability of high-dimensional latent features and output semantically enhanced high-dimensional latent features. The anti-packet loss transmission module is embedded between the quantized high-dimensional latent variables output by the arithmetic decoding unit and the input of the lightweight adapter unit. The high-dimensional latent features are masked by the random mask fine-tuning unit to simulate the transmission packet loss scenario for training. The lightweight adapter unit in the task-driven adaptation module can adapt to two or more different types of visual models without modifying the network structure or native parameters of the visual model, thereby realizing the conversion and access of the latent features of the compressed domain to the feature domain of the visual model. The perception enhancement module is deployed during the training phase of the image encoding and decoding module. It is used to optimize the semantic representation adaptation capability of high-dimensional latent features to large visual language models and output semantically enhanced high-dimensional latent features. The task-driven adaptation module is a dedicated adaptation structure for compressing domain latent features into a large visual language model, including a lightweight adapter unit. The lightweight adapter unit is a lightweight convolutional network. The input of the lightweight adapter unit is directly connected to the quantized high-dimensional latent features output by the image encoding and decoding module. The dimensionality matching between the latent features and the input features of the large model is achieved through a configurable dimension transformation structure. The output is directly connected to the feature input layer of the visual model and the downstream visual language large model. Through the output of the lightweight adapter unit, the high-dimensional latent features can be directly introduced into the intermediate hidden layer of the visual model. The packet loss-resistant transmission module includes a random mask fine-tuning unit, which generates a random binary mask to perform element-level masking on the quantized high-dimensional latent features to simulate the packet loss scenario and drive the lightweight adapter unit to learn the global context completion capability based on incomplete latent features. The arithmetic coding unit distributes the spatial tokens of the quantized high-dimensional latent features into more than N data packets through uniform grouping and then outputs them. The system is used to perform the following steps: Step 1, image encoding and compression, including: Step 1.1: Read in the test image, normalize the color value range, and then input it into the system; Step 1.2: The image is converted into a high-dimensional latent feature representation by the encoder unit of the image encoding and decoding module. The dimension of the high-dimensional latent feature is preset to be the basic dimension that adapts to the input of the subsequent large visual language model. Step 1.3: Input the high-dimensional latent features into the super-prior encoder unit to obtain the super-prior features, quantize them into integers by the coding side quantization unit, and then obtain the super-prior feature bitstream through the arithmetic coding unit; Step 1.4: Input the prior features into the prior decoder unit to obtain the mean and variance; Step 1.5: The high-dimensional latent feature representation is quantized into an integer by the quantization unit on the encoding side. Combined with the mean and variance obtained in step 1.4, the high-dimensional latent feature code stream is obtained through the arithmetic coding unit. The high-dimensional latent feature code stream carries the core semantic features adapted to the large visual language model. In step 1.5, before outputting the high-dimensional latent feature code stream, the spatial tokens of the quantized core latent features are distributed to N data packets through pseudo-random uniform grouping. Step 2, Latent Feature Decoding and Visual Language Large Model Adaptation Inference, includes the following steps: Step 2.1: Decode the super-prior feature bitstream using the arithmetic decoding unit to obtain the super-prior features, and input them into the super-prior decoder unit to obtain the mean and variance; Step 2.2: The mean and variance obtained in step 2.1 are fed into the arithmetic decoding unit to decode the high-dimensional latent feature code stream and obtain the high-dimensional latent features that retain complete semantics. Step 2.3: The high-dimensional latent features are directly fed into the lightweight adapter unit of the task-driven adaptation module to output a feature representation compatible with the large visual language model. Step 2.4: The feature representations obtained in Step 2.3 are directly input into the visual model and the large visual language model, and combined with the text modal input to perform inference and obtain the visual language interaction results; In step 2.1, if there is data packet loss, the decoding process is directly initiated without retransmission. The lightweight adapter unit completes the adaptation based on the valid features received. In step 2.4, the visual language interaction results include image descriptions or visual question-and-answer results.

2. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code, which, when executed by the processor, causes the processor to perform steps 1 to 2 as described in claim 1.

3. A storage medium, characterized in that, It stores a computer program or instructions that, when run on a computer, execute steps 1 to 2 as described in claim 1.

Images