A complexity-aware based visual information lightweight encoding method and system

Abstract

The application discloses a visual information lightweight coding method and system based on complexity perception, and belongs to the technical field of computer vision. The method comprises the following steps: identifying the complexity type of graphic text data, wherein the graphic text data comprises an original image and an original text; for graphic text data of a simple type, obtaining a coding result of the graphic text data by reducing the resolution of the original image; for graphic text data of an OCR enhancement type, obtaining a coding result of the graphic text data by reducing the resolution of the original image and supplementing the original text based on the original image; and for graphic text data of a difficult type, obtaining a coding result of the graphic text data by reducing the resolution of the original image and extracting a region enhancement image of the original image. The application can dynamically balance the calculation cost and visual understanding accuracy in the reasoning stage, and reduce the visual reasoning cost of a multi-modal large model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a lightweight encoding method and system for visual information based on complexity perception. Background Technology

[0002] With the rapid development of Multimodal Large Language Models (MLLMs), cross-modal joint modeling capabilities have been significantly improved. These models, by simultaneously encoding visual and textual information, can perform tasks such as question answering, web page content understanding, and image / text generation in complex network environments. Joint visual-text modeling frameworks, represented by BLIP-2 and the GPT series, have strong cross-modal understanding and generation capabilities supported by large-scale image-text alignment data, and have been widely applied in various web scenarios.

[0003] However, as model size continues to increase, computational costs rise sharply. To maintain the ability to identify fine-grained visual elements in images, existing models generally employ high-resolution visual feature encoding, which generates a large number of visual tokens (visual embedding units) for the input image. While this strategy can improve visual accuracy, it also leads to problems such as excessive GPU memory usage, significantly increased computational load, and longer inference latency during inference, thus limiting the application of models on resource-constrained devices and their deployment in large-scale real-time scenarios.

[0004] To alleviate these problems, researchers have proposed various lightweight visual encoding schemes. Early work mainly improved the Vision Transformer (ViT) structure by selectively preserving or pruning redundant image features to reduce the number of visual tokens. For example, DynamicViT reduces redundancy by dynamically filtering tokens between layers; ToMe (TokenMerging) achieves end-to-end token merging through feature aggregation; and BLIP-2 uses a Q-Former structure to compress image features, achieving efficient abstraction of visual information. Subsequently, schemes such as VisionZip and ATP-LLaVA introduced learnable visual token pruning mechanisms, obtaining the most information-dense key visual tokens through training, significantly reducing the number of visual tokens while maintaining essentially unchanged model performance.

[0005] Despite the progress made by existing technologies, the following limitations still exist: Most multimodal models use fixed-length visual tokens and rely on preset hyperparameters for uniform configuration. They cannot dynamically adjust the number of visual tokens according to the complexity of the input image or the type of task, resulting in wasted resources on simple images and insufficient information on complex images.

[0006] Although structures like ViT have global attention capabilities, they are still insufficient in capturing small targets, local details, and key semantic regions in real-world scenarios, lacking automatic zoom-in or attention enhancement mechanisms.

[0007] Many lightweight visual coding schemes are tightly bound to specific model architectures, making them difficult to directly migrate to other multimodal models or different network vision tasks. Their lack of versatility and scalability limits their practical application.

[0008] In summary, current technologies still lack a unified solution for lightweighting the visual encoding of large multimodal models. This solution should automatically adjust the visual encoding method based on the complexity of the input content, take into account fine-grained enhancement of key regions, and be universally transferable across different models. Therefore, it is necessary to propose a new technical architecture to address these issues, further reduce the visual inference cost of large multimodal models, and improve adaptability and scalability in complex Web scenarios. Summary of the Invention

[0009] To address the challenges of adaptive adjustment of visual tokens, insufficient capture of fine-grained information, and difficulty in transferring between different models in the visual encoding stage of existing multimodal large language models, this invention proposes a lightweight visual information encoding method and system based on complexity awareness. By routing different visual processing paths, it can dynamically balance computational cost and visual understanding accuracy during the inference stage, thereby reducing the visual inference cost of multimodal large models.

[0010] To achieve the above objectives, the technical solution of the present invention includes the following:

[0011] A complexity-aware lightweight encoding method for visual information, the method comprising: Identify the complexity type of the image and text data, which includes original images and original text; For image and text data of simple complexity type, the encoding result of the image and text data is obtained by reducing the resolution of the original image; For image and text data of OCR enhancement type, the encoding result of image and text data is obtained by reducing the resolution of the original image and semantically supplementing the original text based on the original image. For image-text data of the difficult complexity type, the encoding result of the image-text data is obtained by reducing the resolution of the original image and extracting regions from the original image to enhance the image.

[0012] Furthermore, the complexity type of the identified image and text data includes: The original image and original text are encoded separately to obtain image embedding vectors and text embedding vectors; The image embedding vector and the text embedding vector are normalized respectively to obtain the visual features and text features of the image and text data; Obtain the binary image corresponding to the original image and calculate the horizontal projection of the binary image; Density features of the text and image data are generated based on the binary image and its horizontal projection. Visual features, text features, and density features are combined, and the results are classified to obtain the complexity type of the image and text data.

[0013] Further, the binary image corresponding to the original image is obtained, and the horizontal projection of the binary image is calculated, including: The original image is converted to grayscale to obtain a grayscale image; Applying Sobel gradient, Otsu thresholding, and morphological closing operations to a grayscale image yields a binary image. The horizontal projection of the binary image is obtained by summing the pixel values ​​of each column.

[0014] Furthermore, based on the binary map and its horizontal projection, density features of the text-image data are generated, including: Calculate the text pixel ratio of image data based on binary images; The normalized number of text lines is calculated based on horizontal projection and a set projection threshold. The density characteristics of the image and text data are obtained based on the text pixel ratio and the normalized text line count.

[0015] Furthermore, by reducing the resolution of the original image, the encoded results of the image-text data are obtained, including: The original image is downsampled to obtain a low-resolution image; The low-resolution image and the original text are fed into a large language model to obtain the encoded image and text data.

[0016] Furthermore, by reducing the resolution of the original image and semantically supplementing the original text based on the original image, the encoding results of the image-text data are obtained, including: The original image is downsampled to obtain a low-resolution image; Extract the text content from the original image and concatenate the original text with the text content to obtain the text concatenation result; The low-resolution image and text concatenation result is fed into a large language model to obtain the encoded result of the image and text data.

[0017] Furthermore, by reducing the resolution of the original image and extracting regions from the original image to enhance the image, the encoding results of the graphic data are obtained, including: The original image is downsampled to obtain a low-resolution image; The original image and original text are encoded separately to obtain image embedding vectors and text embedding vectors. The image embedding vectors and text embedding vectors are then normalized to obtain the visual features and text features of the image and text data. Based on these visual and textual features, a text-based correlation heatmap is generated. The original image is divided into several windows, and a significance score is generated for each window based on the correlation heatmap. Based on the saliency score, at least one window is selected as the key region. Each key region is then cropped and scaled, and the combined regions are used to obtain the region-enhanced image. Generate region-enhanced image and text data based on the region-enhanced image and the original text; The image and text data and the region-enhanced image and text data are fed into a large language model to obtain the encoding result of the image and text data.

[0018] Furthermore, the method also includes: Based on the encoding results of the image and text data, the inference results of the image and text data are generated.

[0019] A complexity-aware lightweight encoding system for visual information, the system comprising: A type recognition module is used to identify the complexity type of graphic data, which includes original images and original text. The first encoding module is used to obtain the encoding result of image and text data with a simple complexity type by reducing the resolution of the original image. The second encoding module is used to obtain the encoding result of image and text data with OCR enhancement complexity by reducing the resolution of the original image and semantically supplementing the original text based on the original image. The third encoding module is used to obtain the encoding result of image and text data with a complexity type of hard by reducing the resolution of the original image and extracting regions of the original image to enhance the image.

[0020] A computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the complexity-aware lightweight encoding method for visual information as described above.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects.

[0022] 1) Significantly reduces the number of visual tokens and improves inference efficiency. This invention employs a differentiated visual encoding strategy for samples of varying complexity, which reduces a large number of redundant visual tokens while maintaining visual understanding capabilities. This reduces memory usage and computational load, thereby improving the operational efficiency of multimodal models during the inference phase.

[0023] 2) High efficiency while maintaining or improving performance. This invention demonstrates its effectiveness in cross-modal understanding tasks by reducing the number of visual tokens while maintaining or even improving the model's inference accuracy across multiple visual question answering tasks.

[0024] 3) It possesses good versatility and portability. Due to its pluggable structural design, this invention can be used with multimodal large language models of different architectures without modifying the original model structure, making it suitable for various practical application scenarios and possessing strong scalability. Attached Figure Description

[0025] Figure 1 A schematic diagram of a lightweight visual enhancement model based on complexity awareness.

[0026] Figure 2 Flowchart of a complexity-aware lightweight encoding method for visual information.

[0027] Figure 3 Block diagram of a complexity-aware lightweight encoding system for visual information.

[0028] Figure 4 A block diagram of computer equipment. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this does not constitute any limitation on the present invention.

[0030] This invention starts from the differences in input data complexity, arguing that different image and text samples differ significantly in visual detail density, text dependence, and inference difficulty. If a fixed visual encoding strategy is used in all input scenarios, it will result in wasted computational resources for simple samples or loss of detailed information for complex samples. Therefore, this invention proposes a lightweight visual enhancement model based on complexity awareness, using an overall mechanism of adaptive processing based on input complexity. Figure 1 As shown, adaptive visual encoding and visual token optimization are used to implement multimodal input.

[0031] To achieve quantifiable discrimination of input complexity, this invention constructs a multimodal complexity annotation dataset containing manually annotated image and text samples of different complexity levels. Based on this dataset, this invention trains a multimodal routing model that integrates CLIP and ALBERT to automatically identify the processing type of input samples and classify them into three categories: simple, difficult, and OCR-enhanced. Furthermore, this invention designs corresponding lightweight visual processing strategies for different types of samples to achieve adaptive visual information enhancement and dynamic optimization of visual tokens.

[0032] 1. Simple Sample Processing Strategy: For samples with simple structure, prominent visual key regions, and low text dependence, this invention reduces the number of visual tokens by lowering the input image resolution, thereby reducing memory usage and computational load, while maintaining basic visual understanding capabilities and achieving efficient reasoning.

[0033] 2. Difficult Sample Processing Strategy: For samples with dense visual details and multiple target or complex reasoning requirements, this invention adopts a "two-stage visual enhancement strategy": 1) First, the global image is downsampled to reduce the initial visual token; 2) Then, key regions in the image are extracted and magnified based on saliency detection and fused with the downsampled image to enhance local fine-grained visual cues, make up for the information loss caused by downsampling, and thus maintain the integrity of visual understanding.

[0034] 3. OCR Enhancement Sample Processing Strategy: For samples containing dense text regions or relying on text content in images for reasoning, this invention employs OCR text extraction technology, inputting the recognized text information and downsampled images into the model. By supplementing text semantics while reducing visual tokens, the understanding ability of tasks containing text in images is effectively improved.

[0035] Specifically, the complexity-aware lightweight encoding method for visual information of the present invention, such as... Figure 2 As shown, it includes the following steps.

[0036] Step S1: Identify the complexity type of the image and text data, which includes the original image and the original text.

[0037] For any input sample (I, T), where I is an image and T is text, the routing module first encodes the original image and the original text separately. The text encoder ALBERT and the image encoder CLIP respectively generate text embedding vectors. With image embedding vector : and carry out Normalization to eliminate scale differences: in, This represents the normalized text embedding vector. This represents the normalized image embedding vector. It is a small constant.

[0038] To characterize the possible text density in an image, this invention applies a Sobel gradient, an Otsu threshold, and a morphological closing operation to the grayscale image of the original image to obtain a binary image. ,in, Representing a binary graph of high, Representing a binary graph Width. Calculate the horizontal projection: And based on the projection threshold Estimate the number of lines of text. Among them, Indicates row index, Indicates column index.

[0039] Then define the text pixel ratio with normalized text line count : in, Indicates the domain of image coordinates. This indicates an indicator function, which is used to determine whether a certain condition is true or false.

[0040] Finally, the visual features, text features, and density features are concatenated as follows: And input a two-layer feedforward network to obtain the complexity of the class probabilities: ,in , , Indicates weight, , Indicates bias.

[0041] The model is trained using cross-entropy loss: in, Indicates the route category, Indicates the true label, Indicates that in a given original image and the original text Under the given conditions, predict which category the sample belongs to. The probability of.

[0042] Step S2: For image and text data of simple complexity type, the encoding result of the image and text data is obtained by reducing the resolution of the original image.

[0043] For samples with simple structures or low visual difficulty, this invention employs a unified low-resolution image encoding method. The input image is scaled down to... This reduces the number of visual tokens, thereby lowering memory and computational costs.

[0044] In one embodiment, the present invention first performs downsampling processing on the original image to obtain a low-resolution image; then, the low-resolution image and the original text are fed into a large language model to obtain the encoding result of the image and text data.

[0045] Step S3: For image-text data with OCR enhancement complexity, the encoding result of the image-text data is obtained by reducing the resolution of the original image and semantically supplementing the original text based on the original image.

[0046] For samples containing a large amount of text, an OCR model is used to extract the text content S, which is then concatenated with the original text query to form a new text. Meanwhile, the image is still scaled to... This method reduces visual tokens. It enhances the semantic integrity of the text while maintaining visual lightweightness.

[0047] In one embodiment, the present invention first downsamples the original image to obtain a low-resolution image; then extracts the text content of the original image and concatenates the original text with the text content to obtain a text concatenation result; finally, the low-resolution image and the text concatenation result are fed into a large language model to obtain the encoding result of the image and text data.

[0048] Step S4: For image and text data of the difficult complexity type, the encoding result of the image and text data is obtained by reducing the resolution of the original image and extracting regions of the original image to enhance the image.

[0049] For samples with complex visual elements or containing multiple areas of interest, this invention employs a dual-image input enhancement strategy.

[0050] First, the original image is globally downsampled to obtain... The overall structural features are preserved. Then, a text-based relevance heatmap is calculated: in For the normalized visual features of position (x,y), This is the normalized text representation.

[0051] Calculate the significance score for any window region using the integral image method: in, Indicates the window area.

[0052] Then, non-maximum suppression (NMS) is used to select the j key regions with the highest scores. Each region is cropped and scaled, and then combined to obtain a region-enhanced image: in For cropping and resampling operations, This is for splicing operations.

[0053] Finally, construct a dual-graph input: While keeping the total amount of visual tokens controllable, the model's ability to focus on key areas is improved.

[0054] By employing complexity classification, adaptive routing, and differentiated visual processing, this invention automatically selects the most suitable visual token generation method in different scenarios, effectively reducing redundant coding, improving inference efficiency, and maintaining a high level of cross-modal understanding.

[0055] In one embodiment, the present invention further includes step S5: generating a reasoning result for the graphic data based on the encoding result of the graphic data.

[0056] In summary, this invention constructs a visual enhancement module that can run independently of the main model, is applicable to mainstream multimodal large language models, and is a plug-and-play lightweight visual enhancement model. Specifically, through a technical approach of "complexity recognition—adaptive routing—typed visual enhancement," it achieves dynamic adjustment of the number of visual tokens, balancing model inference efficiency and visual understanding capabilities, while also possessing good transferability and being widely integrated into multimodal large models with different structures.

[0057] This invention trains a CLIP–ALBERT fusion router to automatically identify the complexity of input image and text samples and classifies the input into three types: simple, difficult, and OCR-enhanced. This allows for the dynamic selection of different enhancement paths during the visual encoding stage.

[0058] This invention designs resolution reduction strategies, saliency-guided region enhancement strategies, and OCR text enhancement strategies for samples of different complexities, and combines downsampling and local feature enhancement to achieve dynamic compression and compensation of visual tokens.

[0059] Based on the same concept, this invention also discloses a complexity-aware lightweight encoding system for visual information, such as... Figure 3 As shown, the system includes: A type recognition module is used to identify the complexity type of graphic data, which includes original images and original text. The first encoding module is used to obtain the encoding result of image and text data with a simple complexity type by reducing the resolution of the original image. The second encoding module is used to obtain the encoding result of image and text data with OCR enhancement complexity by reducing the resolution of the original image and semantically supplementing the original text based on the original image. The third encoding module is used to obtain the encoding result of image and text data with a complexity type of hard by reducing the resolution of the original image and extracting regions of the original image to enhance the image.

[0060] Based on the same concept, this invention also discloses a computer device, which may be a terminal, a laptop computer, a desktop computer, a server, a computer cluster, or other types of computer devices. For example... Figure 4 As shown, the computer device may include at least one processor and memory. The processor can execute instructions stored in the memory. The processor is communicatively connected to the memory via a data bus. In addition to the memory, the processor can also be communicatively connected to input devices, output devices, and communication devices via the data bus.

[0061] The processor can be any conventional processor. Processors may include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0062] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0063] In this embodiment of the invention, an executable instruction is stored in a memory. The processor can read the executable instruction from the memory and execute the instruction to implement all or part of the steps of the method of the invention.

[0064] Based on the same concept, the present invention also discloses a computer-readable storage medium including a computer program product or storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.

[0065] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0066] Computer-readable storage media can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.

[0067] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A lightweight encoding method for visual information based on complexity awareness, characterized in that, The method includes: Identify the complexity type of the image and text data, which includes original images and original text; For image and text data of simple complexity type, the encoding result of the image and text data is obtained by reducing the resolution of the original image; For image and text data of OCR enhancement type, the encoding result of image and text data is obtained by reducing the resolution of the original image and semantically supplementing the original text based on the original image. For image-text data of the difficult complexity type, the encoding result of the image-text data is obtained by reducing the resolution of the original image and extracting regions from the original image to enhance the image.

2. The method according to claim 1, characterized in that, The complexity types of the identified image and text data include: The original image and original text are encoded separately to obtain image embedding vectors and text embedding vectors; The image embedding vector and the text embedding vector are normalized respectively to obtain the visual features and text features of the image and text data; Obtain the binary image corresponding to the original image and calculate the horizontal projection of the binary image; Density features of the text and image data are generated based on the binary image and its horizontal projection. Visual features, text features, and density features are combined, and the results are classified to obtain the complexity type of the image and text data.

3. The method according to claim 2, characterized in that, Obtain the binary image corresponding to the original image, and calculate the horizontal projection of the binary image, including: The original image is converted to grayscale to obtain a grayscale image; Applying Sobel gradient, Otsu thresholding, and morphological closing operations to a grayscale image yields a binary image. The horizontal projection of the binary image is obtained by summing the pixel values ​​of each column.

4. The method according to claim 2, characterized in that, Based on the binary map and its horizontal projection, density features of the text-image data are generated, including: Calculate the text pixel ratio of image data based on binary images; The normalized number of text lines is calculated based on horizontal projection and a set projection threshold. The density characteristics of the image and text data are obtained based on the text pixel ratio and the normalized text line count.

5. The method according to claim 1, characterized in that, By reducing the resolution of the original image, the encoded image and text data is obtained, including: The original image is downsampled to obtain a low-resolution image; The low-resolution image and the original text are fed into a large language model to obtain the encoded image and text data.

6. The method according to claim 1, characterized in that, By reducing the resolution of the original image and semantically supplementing the original text based on the original image, the encoded results of the image-text data are obtained, including: The original image is downsampled to obtain a low-resolution image; Extract the text content from the original image and concatenate the original text with the text content to obtain the text concatenation result; The low-resolution image and text concatenation result is fed into a large language model to obtain the encoded image and text data.

7. The method according to claim 1, characterized in that, By reducing the resolution of the original image and extracting regions from the original image to enhance the image, the encoded results of the image-text data are obtained, including: The original image is downsampled to obtain a low-resolution image; The original image and original text are encoded separately to obtain image embedding vectors and text embedding vectors. The image embedding vectors and text embedding vectors are then normalized to obtain the visual features and text features of the image and text data. Based on these visual and textual features, a text-based correlation heatmap is generated. The original image is divided into several windows, and a significance score is generated for each window based on the correlation heatmap. Based on the saliency score, at least one window is selected as the key region. Each key region is then cropped and scaled, and the resulting images are combined to obtain the region-enhanced image. Generate region-enhanced image and text data based on the region-enhanced image and the original text; The image and text data and the region-enhanced image and text data are fed into a large language model to obtain the encoding result of the image and text data.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Based on the encoding results of the image and text data, the inference results of the image and text data are generated.

9. A lightweight encoding system for visual information based on complexity awareness, characterized in that, The system includes: A type recognition module is used to identify the complexity type of graphic data, which includes original images and original text. The first encoding module is used to obtain the encoding result of image and text data with a simple complexity type by reducing the resolution of the original image. The second encoding module is used to obtain the encoding result of image and text data with OCR enhancement complexity by reducing the resolution of the original image and semantically supplementing the original text based on the original image. The third encoding module is used to obtain the encoding result of image and text data with a complexity type of hard by reducing the resolution of the original image and extracting regions of the original image to enhance the image.

10. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the lightweight encoding method for visual information based on complexity perception as described in any one of claims 1-8.

Images