Visual language large model perception enhancement method and system based on visual prompt

Abstract

The invention discloses a visual language large model perception enhancement method and system based on visual prompt, and relates to the field of visual language large models. The problem that how to deploy a small-scale large-language model under the condition that resources are limited in the prior art becomes an urgent problem to be solved is solved. The method comprises the following steps of: segmenting an original image by adopting a segmentation component to generate a mask and an object segmentation list; respectively processing an original image and an image mask generated by a semantic segmentation device by using a visual encoder, so as to extract multi-level visual features which highlight the position and boundary of an object; layer normalization and MLP layer processing are carried out to form visual features; taking the generated mask and the segmentation result list of the object as a text instruction, and inputting the extracted multi-level visual features which highlight the position and the boundary of the object and the visual features into a visual language large model to perform autoregressive semantic generation; the method is also suitable for the technical field of improving the object perception and question-answering ability of the visual language large model without adding extra training parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of visual language large models, and in particular to a method and system for enhancing perception of visual language large models based on visual cues. Background Art

[0002] Large Vision-Language Models (LVLMs) further expand their tasks and capabilities by integrating various structures and components. These models typically combine Large Language Models (LLMs) with visual encoders to align image features with the LLM to enhance visual understanding. Although increasing model parameters and complexity generally enhances feature extraction and semantic generation capabilities, the ever-expanding size of models and datasets requires significant resources and processing time for training and inference.

[0003] Therefore, small-scale large language models that can be deployed under resource-constrained conditions have attracted attention. In order to make small-scale LVLMs perform similarly to traditional LVLMs in multimodal architectures, it is crucial to use specialized structures and training methods, where the key to success in multimodal tasks lies in effectively managing the interactions between different modalities.

[0004] In the LVLMs framework, visual cues are important tools to guide the model to process and understand visual information. Figure 1 As shown in Figure 2, these cues come in various forms, such as bounding boxes, labels, and pixel-level cues, which can significantly improve a model's ability to handle complex understanding tasks. However, bounding boxes often produce excessive whitespace when indicating object locations, labels fail to effectively outline object boundaries, and pixel-level cues typically focus only on individual objects. In contrast, visual cues for semantic segmentation effectively distinguish objects and provide optimal visual cues.

[0005] The effectiveness of visual cueing methods requires efficient training methods. The current common method is to use a multi-layer perceptron (MLP) to connect separately pre-trained single-modal base models, and align components from different modalities through pre-training (PT) and instruction tuning (IT). Summary of the Invention

[0006] The present invention addresses the problem that large visual language models in the existing technology have lightweight parameters. As the scale of large visual language models and data sets continues to expand, a large amount of resources and processing time are required for training and inference. Therefore, how to deploy small-scale large language models under resource-constrained conditions has become an urgent problem to be solved.

[0007] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0008] The present invention proposes a method for enhancing perception of a large visual language model based on visual cues, the method comprising the following steps:

[0009] Step 1: Use the Semantic-SAM segmentation component to segment the original image and generate a list of mask and object segmentation results;

[0010] Step 2: Input the mask generated in step 1 and the object segmentation result list into the visual encoder, and use the visual encoder to process the original image and the image mask generated by the semantic segmentor respectively to extract multi-level visual features that highlight the object location and boundary;

[0011] Step 3: Perform layer normalization and MLP layer processing on the multi-level visual features of the prominent object position and boundary extracted in step 2 to form visual features;

[0012] Step 4: Input the mask and object segmentation result list generated in step 1 as text instructions, the multi-level visual features of the highlighted object position and boundary extracted in step 2, and the visual features formed in step 3 into the visual language model for autoregressive semantic generation, and complete the perception enhancement of the visual language model based on visual cues.

[0013] Furthermore, a preferred embodiment is provided, in step 1, the method for segmenting the image using the Semantic-SAM segmentation component and generating a list of segmentation results of masks and objects is:

[0014] SAM output =(o1,...,o k )

[0015] Among them, each o i =(c i ,m i ) consists of a set of object category sequences c i and the corresponding image mask m i composition;

[0016] That is, select the object category corresponding to the mask area greater than 1 / 3 median threshold and collect it into c i This list is appended to the end of the input text after removing duplicate elements from the list.

[0017] Furthermore, a preferred embodiment is provided, in which step one also includes a step of independently processing the generated mask and object list using a MoF-MLP construction algorithm to form parallel multi-granularity cross-modal mixed features, and the MoF-MLP construction algorithm incorporates a memory enhancement mechanism.

[0018] Furthermore, a preferred embodiment is provided, wherein step four further includes the step of training the large visual language model, wherein the training includes a pre-training stage and a supervised fine-tuning stage;

[0019] The pre-training stage freezes some features of the large visual language model by updating the weights of the connectors, the features including the visual encoder, the image semantic segmenter, and the large language model;

[0020] The supervised fine-tuning phase uses pre-trained weights to initialize the connector, continues to freeze the image semantic segmenter and visual encoder, and updates the connector parameters and large language model in the visual language large model.

[0021] Furthermore, a preferred embodiment is provided, wherein the method for updating the weights of the connectors in the pre-training stage is:

[0022]

[0023] Among them, X input =(X q ,X v ), length is the answer Y a The text length is φ, and φ is a training parameter in the connector.

[0024] Furthermore, a preferred embodiment is provided, wherein the method for freezing some features of the large visual language model in the pre-training stage is:

[0025] F input =(Seg(X input ),X input )

[0026] Among them, F input is the input of the visual language model, X input For text instructions, Seg(X input ) is the data processed by the image semantic segmentor Segmentor.

[0027] Furthermore, a preferred embodiment is provided, in which the supervised fine-tuning stage uses pre-trained weights to initialize the connector, continues to freeze the image semantic segmenter and visual encoder, and updates the connector parameters and the large language model in the visual language model as follows:

[0028]

[0029] Solution 2: A visual language large model perception enhancement system based on visual cues, the system comprising:

[0030] The mask and object list generation module is used to segment the original image using the Semantic-SAM segmentation component and generate a list of segmentation results of masks and objects;

[0031] An input module is used to input the mask and object segmentation result list generated by the mask and object list generation module into the visual encoder, and use the visual encoder to process the original image and the image mask generated by the semantic segmentor respectively to extract multi-level visual features that highlight the object location and boundary;

[0032] The preprocessing module is used to perform layer normalization and MLP layer processing on the multi-level visual features of the prominent object position and boundary extracted by the input module to form visual features;

[0033] The generation module is used to input the generated masks and object segmentation result lists as text instructions, the multi-level visual features of the prominent object positions and boundaries extracted in step 2, and the visual features formed in step 3 into the visual language model for autoregressive semantic generation, thereby completing the perception enhancement of the visual language model based on visual cues.

[0034] Solution 3: A computer device includes a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes any one of the methods described in Solution 1.

[0035] Solution 4: A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of Solution 1.

[0036] The present invention is beneficial in that:

[0037] The present invention describes a method and system for enhancing perception of a large visual language model based on visual cues. It proposes a method for interleaving and fusing the mask generated by semantic segmentation with the original image and inputting it into the large visual language model. This method can make the performance of a small-scale model comparable to that of a large-scale model. This will serve as a new idea to solve the problem of lightweight parameters of large visual language models.

[0038] The present invention is also applicable to the technical field of improving the object perception and question-answering capabilities of large visual language models without adding additional training parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a modeling block diagram of the visual cue method in the visual language large model perception enhancement method based on visual cues described in Implementation Method 1.

[0040] Among them, (a) is a schematic diagram of using bounding boxes for prompting, (b) is a schematic diagram of using special tags for prompting, (c) is a schematic diagram of pixel-level prompting, and (d) is a schematic diagram of using image semantic segmentation for prompting.

[0041] Figure 2 This is an architectural diagram of a visual language large model perception enhancement method based on visual cues as described in Implementation Method 1.

[0042] Figure 3 This is a schematic diagram of the principle of the visual cue method in the visual language large model perception enhancement method based on visual cues described in embodiment eleven.

[0043] Figure 4 This is a training flowchart of a visual language large model perception enhancement method based on visual cues as described in embodiment 11.

[0044] Among them, (a) is a schematic diagram of the pre-training stage, and (b) is a schematic diagram of the supervised fine-tuning stage.

[0045] Figure 5 This is a schematic diagram of a comparison of a large-scale visual language model perception enhancement method based on visual cues described in embodiment 11 using multiple evaluation indicators and models with different parameter scales.

[0046] Figure 6 This is a schematic diagram of the attention comparison between the method adopted in the present application and the prior art in a visual language large model perception enhancement method based on visual cues as described in Implementation 11.

[0047] Among them, (a) is a schematic diagram of the LLaVA-1.5 model, (b) is a schematic diagram of the attention generated by the existing LLaVA architecture, and (c) is a schematic diagram of the large visual language model trained by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the implementation methods of this application clearer, the technical solutions in the implementation methods of this application will be clearly and completely described below in combination with the drawings in the implementation methods of this application. Obviously, the described implementation methods are only part of the implementation methods of this application, not all of the implementation methods.

[0049] Implementation 1: This implementation provides a method for enhancing perception of a large visual language model based on visual cues. The method comprises the following steps:

[0050] Step 1: Use the Semantic-SAM segmentation component to segment the original image and generate a list of mask and object segmentation results;

[0051] Step 2: Input the mask and object segmentation result list generated in step 1 into the visual encoder, and use the visual encoder to process the original image and the image mask generated by the semantic segmentor respectively to extract multi-level visual features that highlight the object location and boundary;

[0052] Step 3: Perform layer normalization and MLP layer processing on the multi-level visual features of the prominent object position and boundary extracted in step 2 to form visual features;

[0053] Step 4: Input the mask and object segmentation result list generated in step 1 as text instructions, the multi-level visual features of the highlighted object position and boundary extracted in step 2, and the visual features formed in step 3 into the visual language model for autoregressive semantic generation, and complete the perception enhancement of the visual language model based on visual cues.

[0054] Implementation 2: This implementation further limits the visual language large model perception enhancement method based on visual cues described in Implementation 1. In step 1, the Semantic-SAM segmentation component is used to segment the image, and the method for generating a list of segmentation results of masks and objects is as follows:

[0055] SAM output =(o1,...,o k )

[0056] Each of them consists of a set of object category sequences and corresponding image masks. The present invention selects the object categories corresponding to the mask area larger than 1 / 3 of the median threshold and collects them into a list. After removing duplicate elements, this list is appended to the end of the input text.

[0057] Implementation method three: This implementation method further limits the visual language large model perception enhancement method based on visual cues described in implementation method two. Step one also includes the step of using the MoF-MLP construction algorithm to independently process the generated mask and object list to form parallel multi-granularity cross-modal mixed features. The MoF-MLP construction algorithm incorporates a memory enhancement mechanism.

[0058] Implementation 4: This implementation further limits the method for enhancing perception of a large visual language model based on visual cues described in Implementation 1. Step 4 also includes a step of training the large visual language model, wherein the training includes a pre-training stage and a supervised fine-tuning stage.

[0059] The pre-training stage freezes some features of the large visual language model by updating the weights of the connectors, the features including the visual encoder, the image semantic segmenter, and the large language model;

[0060] The supervised fine-tuning phase uses pre-trained weights to initialize the connector, continues to freeze the image semantic segmenter and visual encoder, and updates the connector parameters and large language model in the visual language large model.

[0061] Implementation 5: This implementation further limits the method for enhancing perception of a large visual language model based on visual cues described in Implementation 4. The method for updating the weights of the connectors in the pre-training phase is:

[0062]

[0063] Among them, X input =(X q ,X v ), length is the answer Y a The text length is φ, and φ is a training parameter in the connector.

[0064] Implementation 6: This implementation further limits the method for enhancing perception of a large visual language model based on visual cues described in Implementation 4. The method for freezing some features of the large visual language model in the pre-training stage is:

[0065] F input =(Seg(X input ),X input )

[0066] Among them, F input is the input of the visual language model, X input For text instructions, Seg(X input ) is the data processed by the image semantic segmentor Segmentor.

[0067] Implementation 7. This implementation further limits the visual language large model perception enhancement method based on visual cues described in Implementation 4. In the supervised fine-tuning stage, the connector is initialized using pre-trained weights, the image semantic segmenter and visual encoder are frozen, and the connector parameters and large language model in the visual language large model are updated as follows:

[0068]

[0069] Embodiment 8: This embodiment proposes a visual language large model perception enhancement system based on visual cues, the system comprising:

[0070] The mask and object list generation module is used to segment the original image using the Semantic-SAM segmentation component and generate a list of segmentation results of masks and objects;

[0071] An input module is used to input the mask and object segmentation result list generated by the mask and object list generation module into the visual encoder, and use the visual encoder to process the original image and the image mask generated by the semantic segmentor respectively to extract multi-level visual features that highlight the object location and boundary;

[0072] The preprocessing module is used to perform layer normalization and MLP layer processing on the multi-level visual features of the prominent object position and boundary extracted by the input module to form visual features;

[0073] The generation module is used to input the generated masks and object segmentation result lists as text instructions, the multi-level visual features of the prominent object positions and boundaries extracted in step 2, and the visual features formed in step 3 into the visual language model for autoregressive semantic generation, thereby completing the perception enhancement of the visual language model based on visual cues.

[0074] Embodiment 9: A computer device includes a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the method described in any one of embodiments 1 to 7.

[0075] Embodiment 10: A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of embodiments 1 to 7 are implemented.

[0076] Implementation 11: The examples provided in this implementation are used to explain the above implementations 1 to 10, and specifically include the following contents:

[0077] See also Figures 1 to 6 This embodiment describes the general structure of the existing Large Image and Text Model (LLaVA). This invention incorporates the segmentation mask of the semantic segmentation component and specifically integrates visual features to improve performance. The semantic segmenter can accurately perceive and distinguish the number and location of objects in an image, which is crucial for improving the LVLM's ability in visual understanding tasks.

[0078] like Figure 2 As shown, the overall architecture of the present invention mainly includes three parts: a visual feature input part, a text feature input part and a large language model processing part.

[0079] Specifically, the Semantic-SAM segmentation component is used to segment the image and generate masks and object lists.

[0080] The mask and image are fed into a visual encoder to extract multi-level visual features that highlight object locations and boundaries.

[0081] These image block features are processed by layer normalization (LayerNorm) and MLP layer to form visual features.

[0082] Subsequently, visual features, text instructions, and detected object information are input into large language models (LLMs) for autoregressive semantic generation.

[0083] Step 1: Visual Cue Engineering

[0084] like Figure 3 As shown, the present invention uses a visual encoder to process the original image and the image mask generated by the semantic segmenter respectively. The input form is Where C is the number of channels and (H, W) is the resolution of the original input image.

[0085] The projection output of the penultimate layer of the visual encoder is called the image patch embedding. The resolution of a single image patch is (P, P), and the number of image patches generated per image is N = HW / P. 2 , extract hidden embeddings As the corresponding image feature sequences. The two sets of image sequences have the same format and size. The image blocks shown in the figure are not actual images, but visual hidden features generated by processing.

[0086] In the present invention, the connector is required to learn the correspondence between visual cues and original images, so as to improve the semantic segmentation mask prompt effect.

[0087] To ensure that the model understands that the mask cues are aligned with the content in the image, the model uses a Mix of Features (MoF)-MLP-GELU multi-layer perceptron structure. Unlike the commonly used multi-layer MLP, it incorporates a memory enhancement mechanism. The complete construction process is summarized in Algorithm 1.

[0088] This memory mechanism is crucial for improving the model's ability to learn long-term dependencies and capture complex patterns, enabling it to handle more complex tasks. MoF-MLP employs a dual-branch mechanism, with each branch independently processing input data to generate features. These features alternately fuse information from the two branches to form a parallel, multi-granular, cross-modal hybrid feature set. This enhances the perception of the large backend language model, thereby achieving better performance.

[0089] The MoF-MLP construction algorithm is:

[0090] Input: x v ,x m : Input image and mask features

[0091] Output: merged_features: merged image and mask features

[0092] Function MoFMLP(x v ,x m ):

[0093] / / Extract features from original image and mask

[0094] Extractimage_features_raw,image_features_mask←x v ,x m

[0095] bs ← batch size x

[0096] total_len←total length of the feature

[0097] dim←feature dimension

[0098] merged_features ← empty tensor of size (bs, total_len)

[0099] for i from 0 to 2*bl do

[0100] merged_features 2i ←image_features_raw i ;

[0101] merged features 2i+1 +image_features_mask;

[0102] end

[0103] return merged_features.

[0104] Step 2: Target segmentation semantic text auxiliary prompt

[0105] When using the Semantic-SAM method for image segmentation, a segmentation result list will be generated for each image, as shown in the following formula: i =(c i ,m i ) contain object category c i and the corresponding image mask m i We select the image mask m i The c corresponding to the effective correlation area i , and save it to the object prompt list. After removing duplicate elements, the object list is used as a text prompt and appended to the input instruction of the large model. This method uses the image semantic segmentation layer to let the large model perceive the objects in the image at the text feature level.

[0106] SAM output =(o1,...,o k )

[0107] Step 3: Pre-training of the model

[0108] The model training strategy of the present invention includes two stages: pre-training stage and supervised fine-tuning stage. Figure 4 The training data is organized in a similar way to the LLaVA instruction following data. Specifically, the text sequence can be in the form of a multi-round dialogue. The present invention uses an image-semantic style data format (X q ,X v ), where X q , X a Represents questions and answers respectively, X v Represents an image. During training, the model learns to predict the answer based on the question and the corresponding image, so only the "Assistant" part is used to calculate the loss:

[0109] Human:X q1 <\n><X v >

[0110] Assistant:X a1

[0111] Human:X q2

[0112] Assistant:X a2 ...:...

[0114] At this stage, the training goal is to strengthen the feature alignment between image and text information. To achieve this goal, we only update the weights of the connector while freezing the rest of the model (visual encoder, image semantic segmenter, large language model). Training the multimodal large language model involves using autoregressive methods to predict responses based on multimodal inputs. The goal is to maximize the difference between the generated and the true answer Y a (Image description) Probability of matching tokens.

[0115]

[0116] Among them, X input =(X q ,X v ), length represents the answer Y a The text length is φ, and φ represents the trainable parameters in the connector. The large language model predicts F θ and visual encoder Vψ The parameters θ and ψ in are frozen.

[0117] F input =(Seg(X input ),X input )

[0118] Input F of the large model input Contains the original image and instruction X input And Seg(X input ).

[0119] Step 4: Supervised fine-tuning of the model

[0120] This process focuses on training models using a variety of datasets tailored for different downstream tasks, aiming to enhance multimodal semantics. This further improves model generalization and robustness while deepening understanding. In supervised fine-tuning, the present invention initializes the connector with pre-trained weights, continues to freeze the segmenter and visual encoder, and updates the rest of the model. The parameters of both the large language model and the connector are trained;

[0121]

[0122] Among them, as in pre-training, length represents the answer Y a The text length is , but only the parameters ψ in the visual encoder are frozen, and the large language model θ and the connector parameters φ are trained.

[0123] It is worth noting that the parameters of the semantic segmenter remain unchanged during both pre-training and fine-tuning. However, the significant increase in overall training parameters leads to higher resource consumption compared to pre-training. Therefore, a fast attention mechanism optimization algorithm and an efficient fine-tuning method (Low Rank Adaptation (LoRA)) are adopted to reduce memory usage and speed up training.

[0124] Example 1. The method proposed in the present invention can be tested on standard large-scale visual question answering datasets with pictures and texts, and can be optimized in a plug-and-play manner on some existing algorithms. In order to ensure a fair comparison and prove the superiority of the proposed algorithm, the present invention directly uses the publicly available visual encoders and large language models in existing literature. The Clip-VL-14-336 visual encoder, the Phi-3.5 large language model, and the semantic segmenter Semantic-sam are used as benchmark components, and the method proposed in the present invention is used for training, and named Seg-LLaVA. Our model is compared with various state-of-the-art (SOTA) models in multiple benchmark tests. As Figure 5As shown in Figure 5, our small-scale model outperforms existing small-scale models on VQAv2, GQA, SQA, MMB, and POPE tasks, and is comparable to or surpasses some large-scale models on specific tasks.

[0125] We then analyzed the enhancing effect of the hinting method on the attention mechanism. Figure 6 Two sets of object attention heatmaps are shown: (a) from the LLaVA-1.5 model, and (c) from a model trained using our method. (b) shows the attention map generated by the traditional LLaVA architecture, where the object edges appear blurred. The right column shows a significant improvement in the attention distribution after applying our method. Our method enhances the focus on these two objects, strengthening the model's ability to detect and focus on various objects in the image, making their outlines clearly visible and improving its performance in visual question answering tasks.

[0126] At the same time, the present invention presents two visual question-answering examples to demonstrate the visual understanding ability of our model. A picture of a green vegetable soup and a plate of spinach and nuts were input into the models of LLaVA-1.5 and the present invention at the same time. When asked to identify food with a similar color to the soup, the LLaVA-1.5 model chose nuts, while the model of the present invention accurately identified vegetables. This highlights the improvement of the present invention in distinguishing color patterns. At the same time, the computing and reasoning performance of the model was tested using a top-down view of a lane. There are three lanes in the picture, and the number of vehicles on each lane is different. Compared with LLaVA-1.5, the model of the present invention not only accurately detects tiny objects in the image, but also gets the correct answer that there are the fewest vehicles on the left road.

[0127] In summary, the present invention proposes a method that uses the mask generated by semantic segmentation and the original image to interleave and fuse the input into a large visual language model, which can make the performance of a small-scale model comparable to that of a large-scale model. This will serve as a new idea to solve the problem of lightweight parameters of large visual language models.

[0128] Those skilled in the art will understand that the above description is only a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of the present disclosure may be combined or coupled in various ways, even if such a combination or coupling is not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0129] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A method for enhancing perception of large visual language models based on visual cues, characterized in that: The method for enhancing perception of a large visual language model comprises the following steps: Step 1: Use the Semantic-SAM segmentation component to segment the original image and generate a list of mask and object segmentation results; Step 2: Input the mask generated in step 1 and the object segmentation result list into the visual encoder, and use the visual encoder to process the original image and the image mask generated by the semantic segmentor respectively to extract multi-level visual features that highlight the object location and boundary; Step 3: Perform layer normalization and MLP layer processing on the multi-level visual features of the prominent object position and boundary extracted in step 2 to form visual features; Step 4: Input the mask and object segmentation result list generated in step 1 as text instructions, the multi-level visual features of the highlighted object position and boundary extracted in step 2, and the visual features formed in step 3 into the visual language model for autoregressive semantic generation, and complete the perception enhancement of the visual language model based on visual cues.

2. The method for enhancing perception of a large visual language model based on visual cues according to claim 1, characterized in that: In step 1, the Semantic-SAM segmentation component is used to segment the image. The method to generate the segmentation result list of masks and objects is as follows: ALONE output =(o1,...,o k ) Among them, each o i,i∈[1,k] =(c i ,m i ) consists of a set of object category sequences c i and the corresponding image mask m i composition; That is, select mask m i The object categories corresponding to the area greater than 1 / 3 of the median threshold are collected into c i This list is appended to the end of the input text after removing duplicate elements from the list.

3. The method for enhancing perception of a large visual language model based on visual cues according to claim 1, characterized in that: Step one also includes the step of independently processing the generated mask and object list using the MoF-MLP construction algorithm to form a parallel multi-granularity cross-modal mixed feature, wherein the MoF-MLP construction algorithm incorporates a memory enhancement mechanism.

4. The method for enhancing perception of a large visual language model based on visual cues according to claim 1, characterized in that: Step 4 also includes the step of training the large visual language model, which includes a pre-training stage and a supervised fine-tuning stage; The pre-training stage freezes some features of the large visual language model by updating the weights of the connectors, the features including the visual encoder, the image semantic segmenter, and the large language model; The supervised fine-tuning phase uses pre-trained weights to initialize the connector, continues to freeze the image semantic segmenter and visual encoder, and updates the connector parameters and large language model in the visual language large model.

5. The method for enhancing perception of a large visual language model based on visual cues according to claim 4, characterized in that: The method for updating the weights of the connectors in the pre-training phase is: Among them, X input =(X q ,X v ), length is the answer Y a The text length is φ, and φ is a training parameter in the connector.

6. The method for enhancing perception of a large visual language model based on visual cues according to claim 4, characterized in that: The method for freezing some features of the large visual language model in the pre-training stage is: F input =(Seg(X input ),X input ) Among them, F input is the input of the visual language model, X input For text instructions, Seg(X input ) is the data processed by the image semantic segmentor Segmentor.

7. The method for enhancing perception of a large visual language model based on visual cues according to claim 4, characterized in that: The supervised fine-tuning phase uses pre-trained weights to initialize the connector, continues to freeze the image semantic segmenter and visual encoder, and updates the connector parameters and the large language model in the visual language model as follows:

8. A visual language large model perception enhancement system based on visual cues, characterized by: The system comprises: The mask and object list generation module is used to segment the original image using the Semantic-SAM segmentation component and generate a list of segmentation results of masks and objects; An input module is used to input the mask and object segmentation result list generated by the mask and object list generation module into the visual encoder, and use the visual encoder to process the original image and the image mask generated by the semantic segmentor respectively to extract multi-level visual features that highlight the object location and boundary; The preprocessing module is used to perform layer normalization and MLP layer processing on the multi-level visual features of the prominent object position and boundary extracted by the input module to form visual features; The generation module is used to input the generated masks and object segmentation result lists as text instructions, the multi-level visual features of the prominent object positions and boundaries extracted in step 2, and the visual features formed in step 3 into the visual language model for autoregressive semantic generation, thereby completing the perception enhancement of the visual language model based on visual cues.

9. A computer device comprising a memory and a processor, characterized in that A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images