A method and device for processing images and texts of a marked compression frame

Abstract

The present invention discloses a graphic and text processing method and device for a marker compression framework, including: a visual feature extraction step; a visual marker screening and processing step; a text feature extraction step; a multi-modal fusion and model processing step. The beneficial effects of the present invention are as follows: The visual marker compression framework of the present invention significantly improves the inference efficiency of MLLMs without additional training; through the global and local information fusion of the DVTS module and the text-guided supplementation of the TGVC module, the number of visual markers is greatly reduced, while key visual information is retained and visual-text alignment is enhanced; experiments show that compared with existing methods, this framework significantly reduces the computational cost while maintaining or even improving the model performance in a variety of image and video benchmark tests, and has significant technical advantages and application potential.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and specifically provides a method and device for processing graphics and text in a marker compression framework. Background Art

[0002] 1. Multimodal large language models (MLLMs) integrate visual signals with language models to achieve the understanding and interaction of content such as images and videos. Its main structure includes a visual encoder, a visual projector, and a basic language model. By converting visual information into visual tokens and combining them with text tokens, the input is processed by the language model.

[0003] 2. When existing MLLMs process high-resolution images and videos, due to the excessive number of visual tokens, the computational complexity and inference cost increase significantly. For example, when processing high-resolution images, the number of visual tokens may reach thousands, which greatly increases the computational amount of the model and limits its deployment efficiency in practical applications. In addition, existing acceleration inference methods often only focus on a certain part of the model when reducing the number of visual tokens, such as only optimizing in the visual encoding stage or the language model decoding stage, lacking comprehensive consideration of the entire model process, resulting in performance degradation. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for processing graphics and text in a marker compression framework to solve the problems raised in the above background art.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a method for processing graphics and text in a marker compression framework, including:

[0007] Visual feature extraction step: Encode the input image through a visual encoder to generate a sequence of visual tokens;

[0008] Visual token screening and processing step:

[0009] Analyze the importance of the visual token sequence based on the CLS attention and local attention mechanisms, sort the visual tokens, and select the top K key tokens;

[0010] Use the remaining visual tokens to guide the clustering and merging of visual tokens with text information, and extract the feature information of the remaining tokens by repeating the processing N times;

[0011] Text feature extraction step: Encode the input text question through a text encoder to generate text features;

[0012] Multi-modal Fusion and Model Processing Steps: The first K key tokens, the remaining tokens after using text information to guide the clustering and merging of visual tokens, and text features are fused and input into a large language model for intermediate layer processing of the fused features to complete the calculation of the graphic and text tasks.

[0013] Preferably, analyzing the importance of the visual token sequence based on CLS attention and local attention mechanisms, sorting the visual tokens, and selecting the first K key tokens includes:

[0014] Input the special token CLS and the visual token sequence into the attention mechanism to calculate the attention weights of CLS for each visual token;

[0015] Apply the attention mechanism within the visual token sequence to calculate the local attention weights of the tokens and their surrounding neighboring tokens;

[0016] Fuse the CLS attention weights and local attention weights to generate the comprehensive importance scores for each visual token;

[0017] Sort the visual tokens according to the comprehensive scores, and select the top K tokens with the highest scores as key tokens to retain the most representative global semantic information and key local details in the image.

[0018] Preferably, using text information to guide the clustering and merging of the remaining visual tokens, and extracting the feature information of the remaining tokens through N repeated processes, includes:

[0019] Perform bidirectional similarity calculation on the remaining visual tokens and text tokens;

[0020] Based on the bidirectional similarity results, sort the remaining visual tokens and select the top R visual tokens as the clustering centers;

[0021] Assign the remaining visual tokens to the corresponding clustering centers according to the similarity and perform the clustering operation towards the centers;

[0022] For the visually marked clusters after merging, repeatedly execute the process of "similarity calculation → determine new centers → re-cluster and merge" again for N times.

[0023] Preferably, performing bidirectional similarity calculation on the remaining visual tokens and text tokens includes:

[0024] Calculate the similarity from visual tokens to text tokens to measure the matching degree between visual content and text semantics;

[0025] Calculate the similarity from text tokens to visual tokens to clarify the guiding emphasis direction of the text on visual tokens.

[0026] Preferably, for the visually marked clusters after merging, the process of "similarity calculation → determination of new centers → reclustering and merging" is repeatedly executed N times, including:

[0027] Each iteration further purifies the visual markers, gradually strengthening the core features guided by the text and eliminating irrelevant details;

[0028] Through multiple rounds of optimization, the visual markers are made to better conform to the semantic requirements of the text, and finally, visually marked features guided by the text and refined are output.

[0029] Preferably, the intermediate layer processing of the fusion features includes:

[0030] Screening key visual markers through global semantic importance and local spatial continuity;

[0031] Then, the text information is used to guide the clustering and merging of visual markers.

[0032] In a second aspect, an embodiment of the present invention provides a graphic and text processing device for a marker compression framework, including:

[0033] Visual feature extraction module: Encoding the input image through a visual encoder to generate a sequence of visual markers;

[0034] Visual marker screening and processing module:

[0035] Analyzing the importance of the sequence of visual markers based on CLS attention and local attention mechanisms, sorting the visual markers, and selecting the top K key markers;

[0036] Using the text information to guide the clustering and merging of the remaining visual markers, and extracting the feature information of the remaining markers through repeated processing N times;

[0037] Text feature extraction module: Encoding the input text question through a text encoder to generate text features;

[0038] Multi-modal fusion and model processing module: Fusing the top K key markers, the remaining markers processed by guiding the clustering and merging of visual markers using text information, and the text features, inputting them into a large language model, and performing intermediate layer processing on the fusion features to complete graphic and text task calculations.

[0039] In a third aspect, an embodiment of the present invention provides an electronic device, including at least one processor, and the processor is communicatively connected to at least one memory, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of the above embodiments.

[0040] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method described in any one of the above embodiments when executed.

[0041] Compared with the prior art, the beneficial effects of the present invention are: the visual marker compression framework of the present invention significantly improves the reasoning efficiency of MLLMs without the need for additional training; through the global and local information fusion of the DVTS module and the text-guided supplement of the TGVC module, the number of visual markers is greatly reduced while retaining key visual information and enhancing visual-text alignment; experiments show that in a variety of image and video benchmarks, compared with existing methods, this framework significantly reduces the computational cost while maintaining or even improving the model performance, and has significant technical advantages and application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0043] Figure 2 A framework diagram of a method according to an embodiment of the present invention;

[0044] Figure 3 A flow chart for selecting visually dominant markers of the present invention;

[0045] Figure 4 This is a module diagram of the device of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] See also Figure 1 The present invention provides a technical solution: a method for processing images and texts of a marked compression framework, comprising:

[0048] S100: Visual feature extraction step: Encode the input image through a visual encoder to generate a visual marker sequence.

[0049] S200: Visual marker screening process steps:

[0050] Analyze the importance of the visual marker sequence based on CLS attention and local attention mechanisms, sort the visual markers, and select the top K key markers;

[0051] Utilize the remaining visual tokens and use text information to guide the clustering and merging of visual tokens, and extract the feature information of the remaining tokens by repeating the processing N times.

[0052] S300: Text feature extraction step: Use a text encoder to encode the input text question to generate text features.

[0053] S400: Multimodal fusion and model processing step: Fuse the top K key tokens, the remaining tokens after using text information to guide the clustering and merging of visual tokens, and text features, input them into a large language model, and perform intermediate layer processing on the fused features to complete the graphic and text task calculation.

[0054] Please refer to Figure 2 , Figure 2 which shows the system framework corresponding to the graphic and text processing method of the token compression framework in the embodiments of the present invention, and details the process of processing graphic and text features by the above method.

[0055] First is the visual information processing. The visual encoder obtains the input image (such as an airplane picture) and generates a sequence of visual tokens. Analyze the importance of the tokens through CLS attention (blue association) and local attention (red association) to select visually dominant tokens. Sort the tokens and select the top K key tokens. The remaining tokens are subjected to feature extraction through the TGVC module (repeating the processing N times), and finally the visual features are integrated through a token mapper.

[0056] Secondly, text information processing. Text encoder: Process the text question (such as "What color is the airplane in the picture?"), and generate text features.

[0057] Thirdly, multimodal fusion and large language model processing. After fusing the visual tokens and text features, input them into a large language model. The model further processes the intermediate layer features through the DVTS and T6VC modules inside to integrate visual and language information, and finally completes the graphic and text question answering task.

[0058] The overall architecture reflects the graphic and text interaction logic of "visual token screening → multimodal feature fusion → large language model reasoning", aiming to solve the correlation analysis problem of image content understanding and text questions.

[0059] In an embodiment of the present invention, the framework mainly includes two plug-in modules: the Dominant Vision Token Selection (DVTS) module and the Text-Guided Vision Complement (TGVC) module. The DVTS module filters key visual tokens through global semantic importance and local spatial continuity to ensure the integrity of visual information and key details are retained. The TGVC module uses text information to guide the clustering and merging of visual tokens to complement visual information related to text instructions and enhance the alignment between visual and text representations. These two modules can be seamlessly inserted between any two layers of the visual encoder and the LLM and can function in both the visual encoding and LLM decoding stages.

[0060] Specifically, in the DVTS module, the local semantic importance is to extract the attention weights from the penultimate layer of the CLIP-based visual encoder and use the attention distribution of the CLS token to measure the global importance score of each visual token:

[0061] where H is the number of attention heads, is the attention score of the CLS token to the i-th visual token in the h-th attention head.

[0062] Local spatial continuity: The local spatial continuity is captured through a local token affinity measurement algorithm. This algorithm uses a dual kernel function to simultaneously consider feature similarity and position proximity and calculates the local importance score:

[0063] where, F xy and P xy are the feature vector and spatial coordinates of the token at position (x, y) respectively, σf and σp are the standard deviations of feature and position differences, w1, w2 and w3 are balance parameters, and (x, y) and (u, v) are two coordinates in the image feature space. and are the calculated intermediate values of feature similarity and spatial similarity, is the final local importance score obtained through intermediate value normalization.

[0064] Adaptive variance weighting: Combine the global and local importance scores and use an adaptive variance weighting mechanism to determine the final importance score S i :

[0065] where, where and They are the variances of the global and local importance scores respectively. represents the global importance score, represents the local importance score, and α is the weighting coefficient.

[0066] Specifically, in the TGVC module:

[0067] Determine the cluster centers: Calculate the similarity between the remaining visual tokens and the text features, and select the top R tokens with the highest similarity as the cluster centers:

[0068] Among them, V r is the remaining visual token, T is the text feature, d is the feature dimension.

[0069] Token assignment: For each remaining token, calculate its assignment score with each cluster center:

[0070] Where is the visual token similarity with the text, is the cluster center similarity with the text.

[0071] Cluster aggregation: For each cluster center, aggregate the assigned tokens through weighted average to obtain the final visual supplementary token:

[0072] Repeat the iteration T times to refine the clustering, where represents the j th final visual supplementary token obtained by weighted aggregation of the cluster center, representing the visual feature representation after fusing the relevant information within the cluster, cluster( j ) represents the and visual token set consisting of the j th cluster.

[0073] Please refer to Figure 3 , the processing process of TGVC is as follows:

[0074] 1. Similarity calculation: Build the visual and text association basis

[0075] Bidirectional similarity calculation between visual tokens and text tokens:

[0076] Calculate the similarity between the remaining visual markers and text markers, and select the top R visual markers as centers based on the similarity results. These central markers will become the core anchor points for subsequent clustering.

[0077] Meanwhile, calculate the similarity between text markers and visual markers to further clarify the guiding relationship of text information to visual markers, providing a basis for two-way association in clustering.

[0078] 2. Marker Clustering: Generate Supplementary Visual Markers

[0079] Centripetal Clustering Operation: Using the top R visual markers selected as centers, allocate the remaining visual markers to the vicinity of each center through a clustering algorithm (such as similarity-based aggregation). Each center and its clustered visual markers form a local set, and by means of "centripetal clustering", the scattered remaining visual markers are integrated into a structured supplementary marker group.

[0080] Generation of Supplementary Visual Markers: After clustering, generate supplementary visual markers with text guiding significance. These markers not only retain the original visual information but also incorporate the guiding direction of text semantics, providing more accurate visual feature supplementation for subsequent multimodal fusion and large language model calculations.

[0081] In an embodiment of the present invention, analyze the importance of the visual marker sequence based on the CLS attention and local attention mechanisms, sort the visual markers, and select the top K key markers, including:

[0082] Input the special marker CLS and the visual marker sequence into the attention mechanism to calculate the attention weights of CLS for each visual marker;

[0083] Apply the attention mechanism within the visual marker sequence to calculate the local attention weights of the marker and its surrounding adjacent markers;

[0084] Fuse the CLS attention weights and local attention weights to generate the comprehensive importance score for each visual marker;

[0085] Sort the visual markers according to the comprehensive score, and select the top K markers with the highest scores as key markers to retain the most representative global semantic information and key local details in the image.

[0086] In an embodiment of the present invention, use text information to guide the clustering and merging of the remaining visual markers, and extract the feature information of the remaining markers through N repeated processes, including:

[0087] Perform two-way similarity calculation between the remaining visual markers and text markers;

[0088] Based on the two-way similarity results, sort the remaining visual markers, and select the top R visual markers as the clustering centers;

[0089] Assign the remaining visual markers to the corresponding clustering centers according to the similarity, and perform the clustering operation towards the centers;

[0090] For the visual markers after clustering and merging, loop and execute the process of "similarity calculation → determine new centers → re-cluster and merge" again, repeating N times.

[0091] In the embodiments of the present invention, the parameter K is the number of key markers, N is the number of clustering iterations, and R is the number of clustering centers. K, N, and R can be set as: K = total number of image markers × 0.2, N = 3, R = number of remaining markers × 0.1.

[0092] In the embodiments of the present invention, the two-way similarity calculation between the remaining visual markers and the text markers includes:

[0093] Calculate the similarity from the visual markers to the text markers to measure the matching degree between the visual content and the text semantics;

[0094] Calculate the similarity from the text markers to the visual markers to clarify the guiding emphasis direction of the text on the visual markers.

[0095] In the embodiments of the present invention, for the visual markers after clustering and merging, loop and execute the process of "similarity calculation → determine new centers → re-cluster and merge" again, repeating N times, including:

[0096] Each iteration further purifies the visual markers, gradually strengthens the core features under the guidance of the text, and eliminates irrelevant details;

[0097] Through multiple rounds of optimization, the visual markers are made to better fit the text semantic requirements, and finally the visually guided and feature-condensed visual markers are output.

[0098] In the embodiments of the present invention, the intermediate layer processing of the fusion features includes:

[0099] Screen key visual markers through global semantic importance and local spatial continuity;

[0100] Then use the text information to guide the clustering and merging of the visual markers.

[0101] Please refer to Figure 4 , the embodiments of the present invention provide a graphic and text processing device for a marker compression framework, including:

[0102] Visual feature extraction module 100: Encode the input image through a visual encoder to generate a visual marker sequence;

[0103] Visual Mark Screening and Processing Module 200:

[0104] Analyze the importance of the visual mark sequence based on the CLS attention and local attention mechanisms, sort the visual marks, and select the top K key marks;

[0105] Use the remaining visual marks to guide the clustering and merging of visual marks with text information, and extract the feature information of the remaining marks by repeating the processing N times;

[0106] Text Feature Extraction Module 300: Encode the input text question using a text encoder to generate text features;

[0107] Multimodal Fusion and Model Processing Module 400: Fusion the top K key marks, the remaining marks after using text information to guide the clustering and merging of visual marks, and the text features, input them into a large language model, and perform intermediate layer processing on the fusion features to complete the calculation of the graphic and text tasks.

[0108] An embodiment of the present invention also provides an electronic device, including at least one processor, the processor is communicatively connected to at least one memory, wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of the above embodiments.

[0109] An embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to any one of the above embodiments when executed.

[0110] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for processing images and texts of a marked compression frame, characterized in that: Including: Visual feature extraction step: Encode the input image through a visual encoder to generate a sequence of visual tokens; Visual token screening and processing step: Analyze the importance of the visual token sequence based on CLS attention and local attention mechanisms, sort the visual tokens, and select the top K key tokens; Use text information to guide the clustering and merging of the remaining visual tokens, and extract the feature information of the remaining tokens through N repetitions of processing; Text feature extraction step: Encode the input text question using a text encoder to generate text features; Multi-modal fusion and model processing step: Fuse the top K key tokens, the remaining tokens after clustering and merging the visual tokens guided by text information, and the text features, input them into a large language model, and perform intermediate layer processing on the fused features to complete the calculation of the image-text task.

2. The graphic and text processing method of a marker compression framework according to claim 1, characterized in that: The analyzing the importance of the visual token sequence based on CLS attention and local attention mechanisms, sorting the visual tokens, and selecting the top K key tokens includes: Input the special token CLS and the visual token sequence into the attention mechanism to calculate the attention weights of CLS for each visual token; Apply the attention mechanism within the visual token sequence to calculate the local attention weights of the tokens and their surrounding neighboring tokens; Fuse the CLS attention weights and the local attention weights to generate the comprehensive importance scores for each visual token; Sort the visual tokens according to the comprehensive scores, and select the top K tokens with the highest scores as key tokens, retaining the most representative global semantic information and key local details in the image.

3. A graphic and text processing method for a marker compression framework according to claim 1, characterized in that: The using text information to guide the clustering and merging of the remaining visual tokens and extracting the feature information of the remaining tokens through N repetitions of processing includes: Perform bidirectional similarity calculation on the remaining visual tokens and text tokens; Based on the bidirectional similarity results, sort the remaining visual tokens and select the top R visual tokens as the clustering centers; Assign the remaining visual tokens to the corresponding clustering centers according to the similarity and perform the clustering operation towards the centers; For the visual tokens after clustering and merging, repeatedly execute the process of "similarity calculation → determine new centers → re-cluster and merge" in a loop for N times.

4. The method for processing images and texts in a marked compression framework according to claim 3, characterized in that: The performing bidirectional similarity calculation on the remaining visual tokens and text tokens includes: Calculate the similarity from visual tokens to text tokens to measure the matching degree between visual content and text semantics; Calculate the similarity from text tokens to visual tokens to clarify the guiding focus direction of the text on visual tokens.

5. A graphic and text processing method for a tag compression framework according to claim 1, characterized in that: The repeatedly executing the process of "similarity calculation → determine new centers → re-cluster and merge" in a loop for N times for the visual tokens after clustering and merging includes: Each iteration further purifies the visual tokens, gradually strengthens the core features under text guidance, and eliminates irrelevant details; Through multiple rounds of optimization, make the visual tokens more conform to the text semantic requirements, and finally output the text-guided and feature-condensed visual tokens.

6. A graphic and text processing method for a tag compression framework according to claim 1, characterized in that: The performing intermediate layer processing on the fused features includes: Screen key visual tokens through global semantic importance and local spatial continuity; Then use text information to guide the clustering and merging of visual tokens.

7. An image and text processing device for a tag compression framework, characterized in that, Including: Visual feature extraction module: encodes the input image through a visual encoder to generate a sequence of visual tokens; Visual token screening and processing module: Analyzes the importance of the visual token sequence based on CLS attention and local attention mechanisms, sorts the visual tokens, and selects the top K key tokens; Uses the remaining visual tokens to guide the clustering and merging of visual tokens with text information, and extracts the feature information of the remaining tokens by repeating the processing N times; Text feature extraction module: encodes the input text question through a text encoder to generate text features; Multi-modal fusion and model processing module: fuses the top K key tokens, the remaining tokens after using text information to guide the clustering and merging of visual tokens, and the text features, inputs them into a large language model, and performs intermediate layer processing on the fused features to complete the calculation of the text-image task.

8. An electronic device, characterized in that, Includes at least one processor, the processor is communicatively connected to at least one memory, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

Images