Efficient visual localization under multi-modal large model dialog capability preservation

Abstract

The application relates to an efficient visual positioning method under multi-modal large model dialogue capability, characterized in that the positioning network D-LMM comprises a multi-modal large model LMM image encoder CLIP and a multi-modal large model LMM text encoder, a frozen LLM, an instance-level text feature extractor, a dynamic instance feature modulation module DIFM and a fusion segmentation mask head, for a given image and text, the positioning process of D-LMM comprises the following steps: inputting the image into the image encoder CLIP to obtain multi-layer features; splicing the input visual features and the input text features together; inputting the output text features of the LLM into the instance-aware text feature extractor ITE to obtain instance-aware text features; inputting the average text features of each instance and the multi-level visual features obtained from the image encoder into the dynamic instance feature modulation module DIFM, and converting the multi-level visual features into instance-aware multi-level feature maps.

Description

Technical Field

[0001] This invention relates to localization methods in multimodal understanding and dialogue, based on frozen large models and related segmentation techniques. Background Technology

[0002] Visual segmentation aims to assign pixels with semantic labels. We divide visual segmentation into traditional segmentation and reference segmentation. Traditional segmentation, such as semantic segmentation[1] and instance segmentation[2], is usually performed by giving a set of semantic categories during training or inference. In contrast, reference segmentation, such as reference expression segmentation (RES)[3] and panoramic narrative localization (PNG)[4], aims to segment instances based on linguistic descriptions. Compared with traditional segmentation, reference segmentation is more challenging as it does not restrict categories. Specifically, RES aims to segment different instances corresponding to different linguistic descriptions, while PNG generates a mask for all objects in a given linguistic description. Related reference segmentation methods[5] can be divided into two-stage and one-stage methods. Two-stage methods first extract candidate instances and then match the instances with linguistic descriptions. Compared with two-stage methods, one-stage methods can directly assign corresponding pixels to linguistic descriptions. In recent years, researchers[6] have mainly explored how to use the Transformer structure for one-stage reference segmentation. In addition, some methods have explored a unified framework for various segmentation tasks[7][8].

[0003] Large multimodal models (LMMs) [9]

[10]

[11] have been used to process and understand visual and linguistic data, achieving remarkable success in multimodal understanding and dialogue. However, most LMMs only provide text responses and cannot provide pixel localization of instances within an image. To achieve this goal, researchers have explored how to use LMMs for pixel localization in recent years. Existing methods

[12]

[13]

[14] can be mainly divided into two categories: methods based on fine-tuning LMMs and methods based on freezing LMMs.

[0004] The method based on fine-tuning LMM

[12]

[13] adapts the LMM to pixel localization by fine-tuning a pre-trained model on segmented data. For example, two representative works, LISA

[12] and GLaMM

[13] , train the LMM to predict one or more localization tokens. <seg>The learned tokens are then fed into an additional heavy segmenter SAM

[15] for pixel localization. While these methods have achieved superior performance in pixel localization, they cause LMMs to overfit the segmentation training data to the point of losing general instruction following and dialogue capabilities. In contrast, frozen LMM-based methods aim to perform pixel localization directly based on frozen LMMs, which preserves the original multimodal capabilities. The representative method F-LMM

[14] first uses attention maps belonging to instances to generate a coarse segmentation map, and then refines it by feeding the coarse segmentation map into a fine-tuned SAM to obtain the final segmentation mask. LocalHead

[16] selects several representative attention heads to identify instances. F-LMM and LocalHead achieve superior pixel localization performance and preserve the original multimodal capabilities of LMMs, such as visual question answering (VQA). Nevertheless, they still fail to fully utilize the inherent features of LMMs.

[0005] References:

[0006] [1] J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in IEEE / CVF Conference on ComputerVision and Pattern Recognition, 2015, pp. 3431–3440.

[0007] [2] B. Cheng, A. Schwing, and A. Kirillov, "Per-pixel classification is not all you need for semantic segmentation," in Advances in NeuralInformation Processing Systems, 2021, pp. 17 864–17 875.

[0008] [3] R. Hu, M. Rohrbach, and T. Darrell, "Segmentation from natural language expressions," in European Conference on Computer Vision, 2016, pp.108–124.

[0009] [4] C. Gonz´alez, N. Ayobi, I. Hern´andez, J. Hern´andez, J. Pont-Tuset, and P. Arbel´aez, "Panoptic narrative grounding,” in IEEE / CVFInternational Conference on Computer Vision, 2021, pp. 1364–1373.

[0010] [5] H. Ding, S. Tang, S. He, C. Liu, Z. Wu, and Y.-G. Jiang,"Multimodal referring segmentation: A survey,” arXiv preprint arXiv:2508.00265, 2025.

[0011] [6] H. Ding, C. Liu, S. Wang, and X. Jiang, "Vision-languageTransformer and query generation for referring segmentation,” in IEEE / CVFInternational Conference on Computer Vision, 2021, pp. 16 321–16 330.

[0012] [7] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L.Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar´, and R.Girshick, "Segment anything,” in IEEE / CVF International Conference onComputer Vision, 2023, pp. 4015–4026.

[0013] [8] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J.Gao, and Y. J. Lee, "Segment everything everywhere all at once,” in Advancesin Neural Information Processing Systems, 2023, pp. 19 769–19 782.

[0014] [9] H. Liu, C. Li, Q. Wu, and Y. J. Lee, "Visual instruction tuning,”in Advances in Neural Information Processing Systems, 2023, pp. 34 892–34916.

[0015]

[10] H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T.Ren, Z. Li, H. Yang, Y. Sun, C. Deng, H. Xu, Z. Xie, and C. Ruan, "Deepseek-vl: Towards real-world vision-language understanding,” arXiv preprint arXiv:2403.05525, 2024.

[0016]

[11] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C.Zhou, and J. Zhou, "Qwen-vl: A frontier large vision-language model withversatile abilities,” arXiv preprint arXiv:2308.12966, 2023.

[0017]

[12] X. Lai, Z. Tian, Y. Chen, Y. Li, Y. Yuan, S. Liu, and J. Jia,"Lisa: Reasoning segmentation via large language model,” in IEEE / CVFConference on Computer Vision and Pattern Recognition, 2024, pp. 9579–9589.

[0018]

[13] H. Rasheed, M. Maaz, S. Shaji, A. Shaker, S. Khan, H. Cholakkal,R. M. Anwer, E. Xing, M.-H. Yang, and F. S. Khan, "Glamm: Pixel groundinglarge multimodal model,” IEEE / CVF Conference on Computer Vision and PatternRecognition, pp. 13 009–1 013 018, 2024.

[0019]

[14] S. Wu, S. Jin, W. Zhang, L. Xu, W. Liu, W. Li, and C. C. Loy,"F-lmm: Grounding frozen large multimodal models,” IEEE / CVF Conference onComputer Vision and Pattern Recognition, pp. 24 710–24 721, 2025.

[0020]

[15] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L.Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar´, and R.Girshick, "Segment anything,” in IEEE / CVF International Conference onComputer Vision, 2023, pp. 4015–4026.

[0021]

[16] model only needs a few attention heads for visual grounding,” IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 9339–9350, 2025. Summary of the Invention

[0022] The main technical problem addressed by this invention is how to efficiently perform pixel localization by fully utilizing visual and textual features in a frozen Level Memory Model (LMM) while maintaining the multimodal large-scale model's dialogue capabilities. To address this problem, this invention proposes a Dynamic Instance Feature Modulation (DIFM) module. By introducing this module, accurate segmentation can be achieved with a lightweight approach, while preserving the multimodal dialogue capabilities of the LMM. The technical solution is as follows:

[0023] An efficient visual localization method while preserving the dialogue capability of a multimodal large model is characterized by the use of a localization network, D-LMM, comprising a multimodal large model LMM image encoder CLIP and a multimodal large model LMM text encoder, a frozen LMM, an instance-level text feature extractor, a dynamic instance feature modulation module DIFM, and a fusion segmentation mask. For a given image and text, the localization process using D-LMM includes:

[0024] 1) Input the image into the image encoder CLIP to obtain multi-layer features; use the last layer as the input visual features of the large model LLM. Simultaneously, several layers are selected from the multi-layer features, and the layers are grouped in pairs and concatenated along the channel dimension to obtain multi-level visual features; the text input encoder extracts the input text features. ;

[0025] 2) Input visual features and input text features The features are concatenated together and transformed into the same feature space through a projection layer to obtain the total input features of the LLM. The total input features are fed into the LLM to obtain the corresponding answer output and the multi-layer text features corresponding to the answer. The multi-layer text features of the answer are passed through a linear network to obtain the weight coefficients of each layer of text features. The weighted sum is then obtained to obtain the final output text features.

[0026] 3) Input the output text features of LLM into the instance-aware text feature extractor (ITE) to obtain instance-aware text features. ITE extracts the text features of the corresponding instance from the output text features based on the word index of the instance. Specifically, it averages the output text features of all words belonging to each instance to generate the average text features of each instance. ;

[0027] 4) Average text features for each instance The system inputs multi-level visual features obtained from the image encoder into the Dynamic Instance Feature Modulation (DIFM) module. Utilizing textual features belonging to the instance, the system transforms the multi-level visual features into instance-aware multi-level feature maps through dynamic convolution. The method is as follows: the multi-level visual features extracted by the image encoder CLIP are fed into a convolutional block containing one 1×1 convolution and two 3×3 convolutions for dimensionality reduction to obtain preliminary features; the preliminary features are then processed using dynamic convolution operations to generate instance-aware multi-level feature maps, ultimately resulting in a multi-level feature map for each instance.

[0028] 5) An FPN structure is adopted to sample multi-level feature maps at different resolutions and gradually fuse them to generate high-resolution feature maps for mask prediction of each instance.

[0029] Furthermore, the specific method for selecting several layers from the multi-layer features, grouping them pairwise, and concatenating them along the channel dimension to obtain multi-level visual features is as follows: The LMM image encoder CLIP outputs 24 feature layers. Eight layers are selected from these: {3, 6, 9, 12, 15, 18, 20, 23}. These layers are then grouped pairwise into {3, 6}, {9, 12}, {15, 18}, and {20, 23}, and concatenated along the channel dimension to obtain multi-level visual features. }

[0030] Furthermore, in the dynamic convolution operation, a linear layer is used to generate convolution weights W corresponding to the instance from the instance-aware text features; the convolution weights corresponding to different instances are then applied to multi-level visual features. Above this, generate multi-level feature maps for each instance. For different feature levels, convolution weights are shared in convolutional blocks and dynamic convolution operations to reduce the number of learnable parameters.

[0031] Furthermore, the training process of the localization network D-LMM includes: initializing the network parameters of each part of the localization network D-LMM and the hyperparameters required for the training process; using fixed frozen weights for the large model and keeping them frozen and not updated during training to avoid losing the dialogue ability of the multimodal large model during training; setting a loss function and using the backpropagation algorithm to update the weights of the localization network; using cross-entropy loss and Dice coefficient loss to jointly guide mask learning; obtaining the final localization network after a set number of training rounds; and the weights learned at this time are the final weights of the localization network.

[0032] Furthermore, the testing process for the localization network D-LMM includes: preparing the test image and the corresponding question input; inputting the image into the image encoder CLIP to obtain multi-layer features; and using the last layer as the input visual features for the large model LLM. Simultaneously, several layers are selected from the multi-layer features, and the layers are grouped in pairs and concatenated along the channel dimension to obtain multi-level visual features; the text input encoder extracts the input text features. The input visual features and input text features are concatenated and transformed into the same feature space through a projection layer to obtain the total input features of the LLM. These total input features are then fed into the LLM to obtain the corresponding answer output and the corresponding multi-layer text features. The multi-layer text features of the answer are then passed through a linear network to obtain the weight coefficients of each layer, and a weighted sum is obtained to obtain the final output text features. The output text features of the LLM are then input into the instance-aware text feature extractor (ITE) to obtain instance-aware text features. The ITE extracts the average text features of each instance from the output text features based on the instance's word index. ;Average text features for each instance And the Multi-Level Visual Feature Input Dynamic Instance Feature Modulation (DIFM) module, which obtains multi-level visual feature input from the image encoder, utilizes text features belonging to the instance. The system transforms multi-level visual features into instance-aware multi-level feature maps through dynamic convolution; it employs an FPN structure to sample the multi-level feature maps at different resolutions and then gradually fuses them to generate high-resolution feature maps for mask prediction of different instances. Attached Figure Description

[0033] Figure 1 Dynamic Instance Feature Modulation Module (DIFM)

[0034] Figure 2 D-LMM, a multimodal large model localization network based on DIFM.

[0035] Figure 3 Specific implementation of the proposed method

[0036] Figure 4 Example of the effect of implementing this method. Detailed Implementation

[0037] The present invention will now be described in conjunction with the accompanying drawings and embodiments.

[0038] From the perspective of brain-inspired intelligence, the strategy of freezing the large model is similar to the stability of "long-term memory" in the brain—that is, the core cognitive architecture and knowledge representation remain unchanged to ensure that the model's basic comprehension, general dialogue, and reasoning abilities are not impaired. Adaptive learning for specific downstream tasks (such as pixel localization) can be achieved by introducing small-scale, task-specific "plasticity" modules, similar to the local adjustment of synaptic strength in the brain, i.e., "short-term plasticity" or "experience-dependent plasticity." This brain-inspired "freezing the backbone + local fine-tuning" model not only effectively maintains the dialogue ability of the multimodal large model and significantly reduces the risk of overfitting and catastrophic forgetting, but also improves the model's adaptability to new tasks. Therefore, this invention introduces a small module to freeze the large model, improving its localization ability without sacrificing its general dialogue capabilities.

[0039] First, we introduce the DIFM module: Dynamic Instance Feature Modulation. DIFM aims to utilize textual features belonging to instances to transform the multi-level visual features generated by the LMM image encoder into instance-aware feature maps through dynamic convolution. Then, we explain how to apply the proposed DIFM module to multimodal large model localization.

[0040] (1) Dynamic instance feature modulation module

[0041] Figure 1 A basic schematic diagram of the Dynamic Instance Feature Modulation (DIFM) module is presented. In pixel localization tasks, it is necessary to segment instances corresponding to different text descriptions. Therefore, extracting instance-aware feature maps to perform mask prediction for different instances is crucial. To achieve this goal, the Dynamic Instance Feature Modulation (DIFM) module is introduced. DIFM uses the text features of instances to generate instance-aware multi-level feature maps through dynamic convolution for segmentation of different instances.

[0042] like Figure 1 As shown, an instance-aware text feature extractor (ITE) is first used to extract the text features of instances from the output of the LLM. For the multi-layer transformer output of LLM (24 layers for the deepseek_vl_1.3b model), we first use a small linear layer to distribute the output weights of each layer, and then obtain the final text features through weighted summation between layers. We then extract the text features of each instance based on the word index of the located instance. For pixel localization tasks, multiple words (e.g., hot air balloon) often correspond to a single instance. Therefore, in ITE, the output embeddings of all words belonging to an instance are averaged to generate the average text features for each instance. .

[0043] To extract various information from the LMM image encoder, multi-level visual features are selected from the image encoder CLIP. Instance-aware multi-level feature generation is performed. First, 8 layers are selected from the 24 layers of features output by CLIP, {3, 6, 9, 12, 15, 18, 20, 23}, and then they are paired into {3, 6}, {9, 12}, {15, 18}, and {20, 23}, respectively. Simultaneously, these layers are concatenated along the channel dimension to obtain the final multi-level visual features. (In this invention, there are 4 levels), each instance corresponds to a multi-level visual feature. Then, the multi-level visual features of each instance are fed into a convolutional block containing one 1×1 convolution and two 3×3 convolutions for dimensionality reduction. Then, dynamic convolution operations are used to generate instance-aware feature maps. In dynamic convolution, firstly, linear layers are used to generate convolution weights from the instance-aware text features. This invention uses two layers of dynamic convolution, so the final generated convolution weights should consider the input and output dimensions of both layers. Then, the convolution weights corresponding to different instances are applied to the multi-level visual features respectively, generating a multi-level feature map for each instance. For different feature levels, convolution weights are shared in convolutional blocks and dynamic convolutions to reduce the number of learnable parameters.

[0044] Based on DIFM, this invention can fully extract diverse and instance-aware information from the multi-level features of the image encoder while maintaining the multimodal large model dialogue capability, so as to accurately segment instances.

[0045] (2) Multimodal large model localization based on DIFM

[0046] The following describes how to apply the proposed DIFM module to the localization task. Figure 2 A schematic diagram of an efficient multimodal large model localization structure based on DIFM is presented. D-LMM includes a multimodal large model (containing an image encoder and a text encoder), ITE (instance-level text feature extractor), DIFM (Dynamic Instance Feature Modulation module), and a fusion segmentation masking module. For a given image and text, the D-LMM localization process includes: extracting visual features using the image encoder. Simultaneously, multi-level visual features are extracted from the image encoder. Then, a text encoder is used to extract features from the input text. Visual features output by the image encoder and input text features The data is fed into the LLM to obtain the corresponding answer and multi-layer features of the output text. Then, an instance-aware text feature extractor (ITE) is used to process the multi-layer features of the output text, extracting the text features corresponding to the instance based on the instance's word index. Among them, ITE. Then, a Dynamic Instance Feature Modulation (DIFM) module is used to generate instance-aware multi-level feature maps. Then, using an FPN structure, the multi-level feature maps are sampled at different resolutions, and finally they are gradually fused to generate high-resolution feature maps for mask prediction of different instances.

[0047] Applying the constructed D-LMM framework to multimodal large model localization mainly involves two distinct steps: a training phase and a testing phase. The training phase is used to learn the D-LMM model parameters, while the testing phase utilizes the trained D-LMM to localize the input image, answer the input question, and segment the corresponding answer mask.

[0048] First, the training process is as follows:

[0049] Step 1: Prepare the training dataset (e.g., RefCOCO, RefCOCO+, RefCOCOg, PNG). The data needs to contain a certain number of training images and corresponding object annotations, which are text and the corresponding masks in the images.

[0050] Step 2: Construct the D-LMM network, including a multimodal large model (containing an image encoder and a text encoder), ITE (instance-level text feature extractor), DIFM (Dynamic Instance Feature Modulation module), and a fusion segmentation mask. Set the training loss function for the output mask.

[0051] Step 3: Initialize the network parameters of each part of the localization network and the hyperparameters required for the training process. Network parameters can be initialized randomly. Hyperparameters for the training process include the number of iterations, learning rate, batch size, etc.

[0052] Step 4: Update the weights of the localization network using the backpropagation algorithm. After a set number of training iterations, the final localization network is obtained.

[0053] The testing process is as follows:

[0054] Prepare the test image and corresponding question input. Use an image encoder to extract visual features, and simultaneously extract multi-level visual features from the image encoder. Then, use a text encoder to extract the input text features. Feed the visual features output from the image encoder and the input text features together into the LLM to obtain the corresponding answer and the multi-level features corresponding to the output text. Next, use an instance-aware text feature extractor (ITE) to process the multi-level features of the output text, extracting the text features corresponding to the instance. The ITE extracts the text features of the instance based on the instance's word index. Finally, use a Dynamic Instance Feature Modulation (DIFM) module to generate an instance-aware multi-level feature map. Then, using an FPN structure, the multi-level feature maps are sampled at different resolutions, and finally they are gradually fused to generate high-resolution feature maps for mask prediction of different instances.

[0055] The method proposed in this invention is applicable to general multimodal large model localization tasks. Furthermore, the proposed module, D-LMM, can be used with different large models (such as deepseek-1.3b and llava-1.5-7b) for localization. Figure 3 A brief description of the specific implementation of the method proposed in this invention is given, and the specific steps are as follows:

[0056] Step 1: Select a large model localization dataset based on the application scenario, containing several images along with corresponding text and annotation information. Pixel localization tasks include Reference Expression Segmentation (RES) and Panoramic Narrative Localization (PNG). In RES, object segmentation is required based on reference expressions; therefore, three representative datasets, RefCOCO, RefCOCO+, and RefCOCOg, are used. RefCOCO and RefCOCO+ contain over 19,000 images, and RefCOCOg contains over 26,000 images. In PNG, segmentation is required for all objects corresponding to a given image narrative. The PNG dataset contains 133,103 training images and 8,380 test images. This invention integrates the proposed D-LMM into three open-source multimodal large models, including DeepSeekVL-1.3B, DeepSeekVL-7B, and LLaVA-1.57B. Training samples from the RefCOCO+, RefCOCOg, and PNG datasets are mixed for model training. The text input for the LMM is formatted as "User: Describe this image. Model: Description", where the description is provided by the RES and PNG datasets.

[0057] Step 2: Construct the D-LMM network, which includes a multimodal large model (containing an image encoder and a text encoder), ITE (instance-level text feature extractor), DIFM (dynamic instance feature modulation module), fusion segmentation mask, and other components.

[0058] Step 3: Initialize the network parameters for each part of the localization network and the hyperparameters required for training. Large models use fixed frozen weights, which remain frozen and unupdated during training. The batch size is 4, and the number of training epochs is 4. The AdamW optimizer is used with a learning rate of 0.0001 and weight decay of 0.01.

[0059] Step 4: Define the loss function and update the weights of the localization network using the backpropagation algorithm. During this process, cross-entropy loss and Dice coefficient loss are used to jointly guide mask learning. After the set number of training epochs, the final localization network is obtained. The learned weights at this point are the final weights of the localization network.

[0060] Step 5: Prepare the test images and corresponding question inputs. Use an image encoder to extract visual features. Simultaneously, multi-level visual features are extracted from the image encoder. Then, a text encoder is used to extract features from the input text. Visual features output by the image encoder and input text features The data is fed into the LLM to obtain the corresponding answer and multi-layer features of the output text. Then, an instance-aware text feature extractor (ITE) is used to process the multi-layer features of the output text, extracting the text features corresponding to the instance based on the instance's word index. Then, a Dynamic Instance Feature Modulation (DIFM) module is used to generate instance-aware multi-level feature maps. Then, using an FPN structure, the multi-level feature maps are sampled at different resolutions, and finally they are gradually fused to generate high-resolution feature maps for mask prediction of different instances.< / seg>

Claims

1. An efficient visual localization method that preserves multimodal large-model dialogue capabilities, characterized in that, The localization network used, D-LMM, includes a multimodal large model LMM image encoder CLIP and a multimodal large model LMM text encoder, a frozen LLM, an instance-level text feature extractor, a dynamic instance feature modulation module (DIFM), and a fusion segmentation mask. For a given image and text, the localization process using D-LMM includes: 1) Input the image into the image encoder CLIP to obtain multi-layer features; use the last layer as the input visual features of the large model LLM. Simultaneously, several layers are selected from the multi-layer features, and the layers are grouped in pairs and concatenated along the channel dimension to obtain multi-level visual features; the text input encoder extracts the input text features. ; 2) Input visual features and input text features The features are concatenated together and transformed into the same feature space through a projection layer to obtain the total input features of the LLM. The total input features are fed into the LLM to obtain the corresponding answer output and the multi-layer text features corresponding to the answer. The multi-layer text features of the answer are passed through a linear network to obtain the weight coefficients of each layer of text features. The weighted sum is then obtained to obtain the final output text features. 3) Input the output text features of LLM into the instance-aware text feature extractor (ITE) to obtain instance-aware text features. ITE extracts the text features of the corresponding instance from the output text features based on the word index of the instance. Specifically, it averages the output text features of all words belonging to each instance to generate the average text features of each instance. ; 4) Average text features for each instance The system inputs multi-level visual features obtained from the image encoder into the Dynamic Instance Feature Modulation (DIFM) module. Utilizing textual features belonging to the instance, the system transforms the multi-level visual features into instance-aware multi-level feature maps through dynamic convolution. The method is as follows: the multi-level visual features extracted by the image encoder CLIP are fed into a convolutional block containing one 1×1 convolution and two 3×3 convolutions for dimensionality reduction to obtain preliminary features; the preliminary features are then processed using dynamic convolution operations to generate instance-aware multi-level feature maps, ultimately resulting in a multi-level feature map for each instance. 5) An FPN structure is adopted to sample multi-level feature maps at different resolutions and gradually fuse them to generate high-resolution feature maps for mask prediction of each instance.

2. The efficient visual localization method for preserving multimodal large-model dialogue capability according to claim 1, characterized in that, The specific method for selecting several layers from multiple feature sets, grouping them pairwise, and concatenating them along the channel dimension to obtain multi-level visual features is as follows: The LMM image encoder CLIP outputs 24 feature layers. Eight layers are selected from these: {3, 6, 9, 12, 15, 18, 20, 23}. These layers are then grouped pairwise into {3, 6}, {9, 12}, {15, 18}, and {20, 23}, and concatenated along the channel dimension to obtain multi-level visual features. } 3. The efficient visual localization method for preserving multimodal large-model dialogue capability according to claim 2, characterized in that, In dynamic convolution operations, linear layers are used to generate convolution weights W corresponding to the instance from instance-aware text features; the convolution weights corresponding to different instances are then applied to multi-level visual features. Above this, generate multi-level feature maps for each instance. For different feature levels, convolution weights are shared in convolutional blocks and dynamic convolution operations to reduce the number of learnable parameters.

4. The efficient visual localization method for preserving multimodal large-model dialogue capability according to claim 1, characterized in that, The training process of the localization network D-LMM includes: initializing the network parameters of each part of the localization network D-LMM and the hyperparameters required for training; using fixed frozen weights for the large model and keeping them frozen during training to avoid losing the dialogue ability of the multimodal large model during training; setting a loss function and using the backpropagation algorithm to update the weights of the localization network; using cross-entropy loss and Dice coefficient loss to jointly guide mask learning; obtaining the final localization network after a set number of training rounds; and the weights learned at this point are the final weights of the localization network.

5. The efficient visual localization method for preserving multimodal large-model dialogue capability according to claim 1, characterized in that, The testing process for the localization network D-LMM includes: preparing the test image and corresponding question input; inputting the image into the image encoder CLIP to obtain multi-layer features; and using the last layer as the input visual features for the large model LLM. Select several layers from the multi-layer features, group them in pairs, and concatenate them along the channel dimension to obtain multi-level visual features; extract the input text features from the text input text encoder. The input visual features and input text features are concatenated and transformed into the same feature space through a projection layer to obtain the total input features of the LLM. These total input features are then fed into the LLM to obtain the corresponding answer output and the corresponding multi-layer text features. The multi-layer text features of the answer are then passed through a linear network to obtain the weight coefficients of each layer, and a weighted sum is obtained to obtain the final output text features. The output text features of the LLM are then input into the instance-aware text feature extractor (ITE) to obtain instance-aware text features. The ITE extracts the average text features of each instance from the output text features based on the instance's word index. ; Average text features for each instance And the Multi-Level Visual Feature Input Dynamic Instance Feature Modulation (DIFM) module, which obtains multi-level visual feature input from the image encoder, utilizes text features belonging to the instance. The system transforms multi-level visual features into instance-aware multi-level feature maps through dynamic convolution; it employs an FPN structure to sample the multi-level feature maps at different resolutions and then gradually fuses them to generate high-resolution feature maps for mask prediction of different instances.

Images