A visual question answering and target positioning method based on multi-modal large model reasoning

Abstract

The application discloses a kind of visual question answering and target positioning method based on multimodal big model inference, belong to the technical field of image recognition;Including the following steps: step S1: obtaining image information and text information;Step S2: setting question and answer processing model, process image information and text information, step S3: corresponding production mask, mask is decoded to form mask image, and form question and answer in combination with text;Step S4: question and answer processing model is trained until question and answer processing model can generate the inference answer that meets scene actual to image and question.The application adopts the above method, and fuses picture and language, reduces the positioning error caused by inference and segmentation disjunction, semantic alignment deficiency, provides verifiable pixel-level visual evidence for scene inference conclusion, realizes the semantic alignment of scene inference and target positioning by joint optimization, forms complete "inference-fusion-positioning-optimization" closed loop.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a visual question answering and target localization method based on multimodal large model reasoning. Background Technology

[0002] With the development of multimodal scene understanding technology, research focus has gradually shifted from early single-modal reasoning and independent segmentation to multimodal fusion-based scene perception methods. However, in image scene understanding and target localization, multimodal fusion often faces semantic alignment problems. Specifically, the differences between multimodal large language models (MLLM) and segmentation models (such as SAM) in terms of task objectives and feature representations, coupled with the lack of dedicated fusion and alignment mechanisms, often lead to a disconnect between inference conclusions and segmentation results, thus affecting the reliability of the perception system. Therefore, how to effectively integrate multimodal reasoning and visual segmentation to solve the semantic alignment problem has become a key challenge in improving the accuracy and robustness of autonomous driving scene understanding and target localization.

[0003] To address the above issues, most existing methods attempt to simply concatenate multimodal reasoning with segmentation models, but lack dedicated feature fusion and semantic alignment mechanisms, failing to bridge the gap between high-level reasoning semantics and low-level pixel localization. Some methods learn intermodal mappings through cross-attention mechanisms, but neglect targeted extraction of reasoning semantics, leading to irrelevant semantic noise interfering with segmentation accuracy. Even in complex road conditions, these methods still cannot meet the high-precision, high-reliability requirements of autonomous driving for scene understanding and target localization. These problems mainly manifest in two aspects. First, MLLM is used to process in-vehicle images and scene queries, achieving scene reasoning and outputting textual conclusions, but it cannot provide pixel-level visual evidence, resulting in a lack of verifiability of reasoning conclusions and difficulty in supporting the reliability of autonomous driving decisions. Second, segmentation models rely solely on visual features for pixel-level segmentation, lacking guidance from scene reasoning logic, making them prone to segmentation bias, especially in complex road conditions such as rainy nights and heavy fog, exhibiting poor segmentation robustness and a disconnect from reasoning conclusions.

[0004] Therefore, a visual question answering and target localization method based on multimodal large model reasoning is provided. Summary of the Invention

[0005] The purpose of this invention is to provide a visual question answering and target localization method based on multimodal large model reasoning to reduce localization errors caused by the disconnect between reasoning and segmentation and insufficient semantic alignment, improve the accuracy of target localization in complex images, and provide verifiable pixel-level visual evidence for scene reasoning conclusions.

[0006] To achieve the above objectives, this invention provides a visual question answering and target localization method based on multimodal large model reasoning, comprising the following steps: Step S1: Obtain the image information I to be processed and the text information corresponding to the scene text query Q. The text information includes target localization related semantics. Preprocess the image information and text information. Step S2: Set up the question-answering processing model, including a multimodal large language model (MLLM), an anchored inference pooling module, an image encoder, a fusion module, and a mask decoder. The working process of the question-answering processing model is as described in steps S3 to S6. Step S3: Use a multimodal large language model to process the preprocessed image and text information in step S1, generate an autoregressive reasoning answer A that conforms to the actual scene, embed the evidence label [SEG] into the target localization related semantics of the multimodal large language model, select the semantic information related to target localization in the hidden state, and obtain the hidden state H with the evidence label [SEG]. Step S4: Input the hidden state H with evidence label [SEG] into the anchored inference pooling module. Using the target localization semantics labeled [SEG] as the query vector, perform attention-weighted aggregation on the hidden state. Extract the compact semantic features most relevant to the target to be localized from the complete inference context, and filter out irrelevant linguistic descriptions to obtain the inference representation. ; Step S5: Use an image encoder to obtain multi-scale visual feature maps F from the image information in step S1, and process the visual feature maps to obtain a global visual context vector. , global visual context vector Compared with the reasoning representation in step S4 The semantically aware conditional vector is obtained by concatenating the data through the fusion module. ; Step S6: Use the semantically aware conditional vector from step S5 Given the visual feature map F, a pixel-level target mask with the same size as the input image is generated by a mask decoder, ensuring that the mask matches the inferred answer. semantic consistency; Step S7: Train the question-answering model using end-to-end joint training until the question-answering model satisfies the loss function. The question-answering model generates an autoregressive reasoning answer A and a target mask image that conform to the actual scene based on the image and text information to be identified.

[0007] Preferably, in step S1, the preprocessing steps are as follows: preprocessing the acquired RGB image, including size normalization and normalization operations, to ensure the consistency of the input image; the scene text query Q adopts a natural language format, covering the core query requirements for images.

[0008] Preferably, in step S4, the operation of the anchored inference pooling module is as follows: Perform [SEG] anchor point localization: Construct a binary position mask Select the feature vector in the hidden state H that corresponds to the [SEG] label, where This indicates that the i-th sequence position corresponds to the [SEG] tag semantics. This position indicates that the semantics are irrelevant; the segmentation anchor point is calculated using masked average pooling. The calculation formula is as follows: ; In the above formula, It is a very small constant to prevent division by zero errors. This represents the i-th feature vector of the hidden state; Attention pooling is used to extract inference features: segmentation anchor points Projected onto the query vector through a linear projection layer. , , among them As the key dimension, the hidden state H is mapped to a key matrix through another linear projection layer. Sum matrix V The key matrix Value matrix in, Indicates the sequence length. This represents the hidden dimension features. Each word token in the inference answer A is a reasoning label. Attention weights are calculated. The relevance of each inference marker to the [SEG] anchor is measured; the higher the relevance, the greater the weight. The calculation formula is as follows: ; In the above formula, This is a scaling factor for the key dimension, used to avoid attention distribution bias caused by excessively high dimensionality, through attention weights. Log-value matrix V The weighted summation is calculated using the following formula: ; In the above formula, This is a representation of reasoning.

[0009] Preferably, step S5 is implemented as follows: Image information I is processed using an image encoder to extract multi-scale visual feature maps. Where C is the number of feature channels, The height of the feature map, Represents the width of the feature map; achieved through global average pooling. Folding the spatial dimension transforms multi-scale visual feature maps into global visual feature vectors, which are then linearly projected. Processing with the nonlinear activation function σ generates a global visual context vector. The calculation formula is as follows: ; In the above formula This represents a linear projection operation. This indicates a global average pooling operation; Then, multimodal deep fusion is performed: the inference representation is... With global visual context vector Concatenating along the channel dimension forms a cross-modal feature tensor with dimension . ,in, Indicates batch size, Indicates the hidden feature dimension. express The feature dimension of the cross-modal feature tensor integrates scene reasoning semantics and image visual priors; the cross-modal feature tensor is input into two layers of parameters. The multilayer perceptron (MLP) performs feature fusion and mapping. The first layer of the MLP fuses cross-modal features to obtain fused features, and the second layer of the MLP maps the fused features to a dimension adapted to the mask decoder to generate semantically aware conditional vectors. The formula is as follows: ; In the above formula, This represents the channel concatenation operation, where the semantically aware conditional vector is used. It includes both scene reasoning logic and spatial location information.

[0010] Preferably, step S6 is implemented as follows: The mask decoder employs a SAM decoder. After receiving the semantically aware conditional vector c and the visual feature map F, it gradually recovers the feature map size through transposed convolution. Combined with an attention mechanism, it focuses on the target region, ultimately generating a pixel-level target mask with the same size as the input image. The formula is expressed as: ; In the above formula, This represents the mask decoding function of the S-mask decoder, which converts the semantically aware conditional vector... With visual features The process involves fusing the data and outputting a pixel-level target mask. This target mask is then processed by RGB to form a target mask image.

[0011] Preferably, in step S7, the training process is as follows: Training employs end-to-end joint optimization, jointly training text generation and target localization tasks within a unified framework. The total loss function consists of two parts: a cross-entropy loss to evaluate the accuracy of the text answer, and a loss to evaluate the localization accuracy. The formula for calculating the total loss function L is as follows: ; In the above formula, The weights representing the autoregressive cross-entropy loss are... Autoregressive cross-entropy loss is used to optimize the text generation accuracy of scene-based visual question answering. For the inference text of the scene predicted by the model, Inference text for real-world scenarios; To divide the overall weight of the loss, The weights represent the calculation weights for the pixel-by-pixel binary cross-entropy loss. Ŵ is the pixel-wise binary cross-entropy loss, used to optimize the pixel-level classification accuracy of the target mask, where Ŵ is the target mask predicted by the model and M is the true target mask. The weights used to calculate the Dice loss are represented. The Dice loss is used to address the class imbalance problem in target segmentation for autonomous driving.

[0012] Therefore, the present invention employs the above-mentioned visual question answering and target localization method based on multimodal large model reasoning, which has the following advantages: In this invention, the reasoning feature extraction module uses [SEG] markers as anchors to accurately extract reasoning semantics related to localization and filter out irrelevant noise; the cross-modal fusion module achieves deep fusion of reasoning semantics and visual features to bridge the cross-modal semantic gap; and end-to-end joint training achieves semantic alignment between reasoning and segmentation to ensure that the target mask is always pixel-level evidence of the reasoning conclusion. In terms of interpretability, by providing accurate visual localization results for each semantic answer, the multimodal large language model is transformed from a "black box" into an interpretable and verifiable tool for understanding the autonomous driving environment, greatly enhancing the system's transparency regarding decision-making basis.

[0013] Regarding the consistency between reasoning and localization, the anchored reasoning pooling and reasoning-to-visual fusion mechanism ensure that the localization results are strictly guided by semantic reasoning, fundamentally solving the problem of mismatch between the answer and visual evidence, and achieving true alignment between language and vision.

[0014] In terms of performance, by jointly optimizing text generation and target localization tasks within a unified framework, the model not only maintains the accuracy of visual question answering, but also surpasses existing methods in localization accuracy, achieving an overall performance improvement.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a visual question answering and target localization method based on multimodal large model reasoning according to the present invention. Figure 2 This is a schematic diagram of the question processing model in a visual question answering and target localization method based on multimodal large model reasoning according to the present invention; Figure 3 This is a schematic diagram of the workflow of the anchored inference pooling module of the present invention; Figure 4 This is a schematic diagram illustrating the workflow of the inference to the visual fusion module of the present invention; Figure 5 The image shows the effect of the visual question answering and target localization method based on multimodal large model reasoning of the present invention compared with other methods. In the image, (a) is the input image, (b) is the ground truth mask, (c) is the mask of the baseline method, and (d) is the mask of the method of the present invention. Figure 6 This is a diagram illustrating the implementation effect of a visual question answering and target localization method based on multimodal large model reasoning according to the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Specific model specifications need to be selected and determined according to the actual specifications of the device, etc. The specific selection calculation method adopts existing technology in the art, and therefore will not be described in detail.

[0018] Example like Figure 1 As shown, this invention provides a visual question answering and target localization method based on multimodal large model reasoning, comprising the following steps: Step S1: Obtain the image information I to be processed and the text information corresponding to the scene text query Q. The text information includes target localization related semantics. Preprocess the image information and text information. The preprocessing steps are as follows: The acquired RGB images are preprocessed, including size normalization and normalization operations, to ensure the consistency of the input images; the scene text query Q adopts a natural language format, covering the core query requirements for images, including but not limited to target recognition, location annotation, scene judgment, etc.

[0019] Step S2: Set up the question-answering processing model, including a multimodal large language model (MLLM), an anchored inference pooling module, an image encoder, a fusion module, and a mask decoder. The specific model architecture is as follows: Figure 2 As shown, the overall execution framework of the method of this invention is presented. Based on a multimodal large language model, it sequentially realizes the generation of inference anchor points and feature extraction, anchored inference pooling, inference from fusion module to visual fusion, target localization generation, and finally achieves semantic alignment between scene inference and target localization through end-to-end joint optimization, forming a complete "inference-fusion-localization-optimization" closed loop. The logical connection between each step is intuitively shown, and the transmission path of core intermediate products such as inference anchor points, compact inference features, and semantically aware conditional vectors is clarified. The working process of the question-answering processing model is as shown in steps S3 to S6. Step S3: The preprocessed image and text information from Step S1 is processed using a multimodal large language model. An autoregressive reasoning answer A, consistent with the actual scenario, is generated. The multimodal large language model embeds the evidence marker [SEG] into the target localization-related semantics, selects semantic information related to target localization in the hidden state, and obtains the hidden state H with the evidence marker [SEG]. This marker is activated when the model needs to locate an entity in the query. Subsequently, this invention extracts the representation of the [SEG] marker in the hidden layer of the multimodal large language model (MLLM). Step S4: Input the hidden state H with evidence label [SEG] into the anchored inference pooling module. Using the target localization semantics labeled [SEG] as the query vector, perform attention-weighted aggregation on the hidden state. Extract the compact semantic features most relevant to the target to be localized from the complete inference context, and filter out irrelevant linguistic descriptions to obtain the inference representation. ; like Figure 3 The diagram illustrates the complete execution process of anchored inference pooling. This process centers on embedded inference anchors ([SEG] markers). First, a binary position mask is constructed to precisely isolate semantic information related to target localization in the hidden state, filtering out irrelevant semantic noise. Then, a stable anchor representation is calculated through mask average pooling, improving robustness to inference sequence shifts. Finally, an attention pooling mechanism is used, with the anchor representation as the query vector, to weighted aggregate the hidden states of the entire inference sequence, extracting the most relevant compact inference features to the target to be located, providing high-quality semantic input for subsequent inference-to-visual fusion. This process effectively solves the problems of messy inference semantics and irrelevant noise interference, ensuring the relevance and effectiveness of inference features. The working process of the anchored inference pooling module is as follows: Perform [SEG] anchor point localization: Construct a binary position mask Select the feature vector in the hidden state H that corresponds to the [SEG] label, where This indicates that the i-th sequence position corresponds to the [SEG] tag semantics. This position indicates that the semantics are irrelevant; the segmentation anchor point is calculated using masked average pooling. The calculation formula is as follows: ; In the above formula, It is a very small constant to prevent division by zero errors. This represents the i-th feature vector of the hidden state; Attention pooling is used to extract inference features: segmentation anchor points Projected onto the query vector through a linear projection layer. , , among them As the key dimension, the hidden state H is mapped to a key matrix through another linear projection layer. Sum matrix V The key matrix Value matrix in, Indicates the sequence length. This represents the hidden dimension features. Each word token in the inference answer A is a reasoning label. Attention weights are calculated. The relevance of each inference marker to the [SEG] anchor is measured; the higher the relevance, the greater the weight. The calculation formula is as follows: ; In the above formula, This is a scaling factor for the key dimension, used to avoid attention distribution bias caused by excessively high dimensionality, through attention weights. Log-value matrix V The weighted summation is calculated using the following formula: ; In the above formula, This is a representation of reasoning.

[0020] Step S5: Use an image encoder to obtain multi-scale visual feature maps F from the image information in step S1, and process the visual feature maps to obtain a global visual context vector. , global visual context vector Compared with the reasoning representation in step S4 The semantically aware conditional vector is obtained by concatenating the data through the fusion module. ; like Figure 4The diagram illustrates the deep fusion process of inference features and visual features. First, multi-scale visual feature maps are extracted from the visual encoder. A global average pooling operation is then used to obtain a global visual context vector, capturing prior information about the road scene in the image. Next, this global visual vector is concatenated with the compact inference features obtained in step 2 along the channel dimension, forming a cross-modal feature tensor that integrates semantic inference and visual priors. Finally, a multilayer perceptron performs deep fusion and dimensional mapping on this tensor to generate a semantic-aware conditional vector. This vector contains both scene inference logic and spatial location information, effectively bridging the gap between abstract inference semantics and spatial visual features, providing reliable semantic guidance for subsequent target localization. The specific implementation process of the image encoder and fusion module is as follows: Image information I is processed using an image encoder to extract multi-scale visual feature maps. Where C is the number of feature channels, The height of the feature map, Represents the width of the feature map; achieved through global average pooling. Folding the spatial dimension transforms multi-scale visual feature maps into global visual feature vectors, which are then linearly projected. Processing with the nonlinear activation function σ generates a global visual context vector. The calculation formula is as follows: ; In the above formula This represents a linear projection operation. This indicates a global average pooling operation; Then, multimodal deep fusion is performed: the inference representation is... With global visual context vector Concatenating along the channel dimension forms a cross-modal feature tensor with dimension . ,in, Indicates batch size, Indicates the hidden feature dimension. express The feature dimension of the cross-modal feature tensor integrates scene reasoning semantics and image visual priors; the cross-modal feature tensor is input into two layers of parameters. The multilayer perceptron (MLP) performs feature fusion and mapping. The first layer of the MLP fuses cross-modal features to obtain fused features, and the second layer of the MLP maps the fused features to a dimension adapted to the mask decoder to generate semantically aware conditional vectors. The formula is as follows: ; In the above formula, This represents the channel concatenation operation, where the semantically aware conditional vector is used. It includes both scene reasoning logic and spatial location information.

[0021] Step S6: Use the semantically aware conditional vector from step S5 Given the visual feature map F, a pixel-level target mask with the same size as the input image is generated by a mask decoder, ensuring that the mask matches the inferred answer. semantic consistency; In step S6, the implementation process is as follows: The mask decoder employs a SAM decoder. After receiving the semantically aware conditional vector c and the visual feature map F, it gradually recovers the feature map size through transposed convolution. Combined with an attention mechanism, it focuses on the target region, ultimately generating a pixel-level target mask with the same size as the input image. The formula is expressed as: ; In the above formula, This represents the mask decoding function of the S-mask decoder, which converts the semantically aware conditional vector... With visual features The process involves fusing the data and outputting a pixel-level target mask. This target mask is then processed by RGB to form a target mask image.

[0022] Step S7: Train the question-answering model using end-to-end joint training until the model satisfies the loss function. The question-answering model uses autoregression to generate an inference answer A and a target mask image that match the actual scene from the image and text information to be recognized. The training process is as follows: Training employs end-to-end joint optimization, jointly training text generation and target localization tasks within a unified framework. The total loss function consists of two parts: a cross-entropy loss to evaluate the accuracy of the text answer, and a loss to evaluate the localization accuracy. The formula for calculating the total loss function L is as follows: ; In the above formula, The weights representing the autoregressive cross-entropy loss are... Autoregressive cross-entropy loss is used to optimize the text generation accuracy of scene-based visual question answering. For the inference text of the scene predicted by the model, Inference text for real-world scenarios; To divide the overall weight of the loss, The weights represent the calculation weights for the pixel-by-pixel binary cross-entropy loss. Ŵ is the pixel-wise binary cross-entropy loss, used to optimize the pixel-level classification accuracy of the target mask, where Ŵ is the target mask predicted by the model and M is the true target mask. The weights used to calculate the Dice loss are represented. The Dice loss is used to address the class imbalance problem in target segmentation for autonomous driving.

[0023] In the embodiments of this application, the technical solution of the present invention can be applied to autonomous driving to solve the problems of disconnect between scene reasoning and target segmentation and insufficient semantic alignment in the prior art, improve the accuracy of autonomous driving scene understanding and target localization, use the images acquired by the vehicle camera as image information, and use the driver's questions as text information.

[0024] To verify the effectiveness and rationality of the method of this invention, relevant simulation experiments were conducted on the Cityscapes dataset to verify its performance on autonomous driving scene understanding and target localization tasks. Specifically, the method of this invention is implemented based on the PyTorch framework, using Qwen3-VL-4B-Instruct as a multimodal large language model and SAM as a segmentation model, where SAM adopts its original image encoder and mask decoder structure. Data augmentation methods such as random scaling, flipping, and brightness adjustment are applied to the image data, and the attention pooling key dimension of the inference feature extraction module is set to 256, with ε set to a value of [value missing]. AdamW was used as the optimizer, and the learning rate was set to All experiments were performed on two NVIDIA 48GA6000 GPUs. The Cityscapes dataset contains a large number of urban road scene images with precise pixel-level annotations, covering core targets such as pedestrians, vehicles, and lane lines, which can fully verify the performance of the method of this invention.

[0025] For the Cityscapes dataset, the method of this invention uses global intersection-union ratio (gIoU) and class intersection-union ratio (cIoU) as the target localization accuracy indicators, and BLEU-1~4 and ROUGE-1 / 2 / L as the scene inference text generation quality indicators. The definitions of each indicator are as follows: gIoU (Global Intersection over Union): Measures the degree of global overlap between the predicted mask and the true mask. The formula is as follows: in To predict the intersection-union ratio (IoU) of the mask and the real mask, The area of ​​the smallest bounding rectangle containing the predicted mask and the real mask is denoted by gIoU, and U is the area of ​​the union of the predicted mask and the real mask. The higher the gIoU value, the higher the segmentation accuracy.

[0026] cIoU (Category Intersection over Union): The intersection over union ratio is calculated separately for each target category, and the average value of all categories is taken. It is used to measure the segmentation accuracy of different target categories. The calculation formula is: in For the number of target categories, For the first Intersection over union (IoU) ratio of target classes.

[0027] BLEU (Bilingual Assessment Alternate): Used to measure the similarity between inference text and real text, with a value ranging from 0 to 100. A higher value indicates better text generation accuracy. BLEU-1: Calculates similarity based on unary grammar (single word) to measure the overall accuracy of text; BLEU-2: Calculates similarity based on 2-grams (two consecutive words) to measure the phrase coherence of text; BLEU-3: Calculates similarity based on ternary syntax to measure the sentence fluency of text; BLEU-4: Calculates similarity based on 4-gram syntax to measure the semantic integrity of text.

[0028] ROUGE (Recall-Oriented Evaluation Substitute): Used to measure the coverage of the inferred text to the real text, with a value ranging from 0 to 100. The higher the value, the more complete the text generation. ROUGE-1: Calculates recall based on unary syntax, measuring the coverage of individual words; ROUGE-2: Calculates recall based on 2-gram syntax to measure phrase coverage; ROUGE-L: Calculates recall based on the longest common subsequence, measuring the overall semantic coherence of the text.

[0029] Table 1 Comparison of the effectiveness of the method of the present invention with other methods in autonomous driving scenario understanding and target localization tasks.

[0030] This invention compares the proposed method with other methods, and the simulation results are shown in Table 1. It demonstrates outstanding performance in core localization metrics (gIoU, cIoU) and text generation metrics (BLEU, ROUGE series). Specifically, gIoU shows a 12.75% improvement compared to DeepLab, a 0.91% improvement compared to PSPNet, and a 0.74% improvement compared to HRNet; cIoU shows a 9.33% improvement compared to DeepLab, a 4.22% improvement compared to PSPNet, and a 13.68% improvement compared to LISA. Regarding text generation metrics, compared to LISA, the only method with text generation capabilities, BLEU shows a 54.56% improvement, ROUGE-1 a 28.89% improvement, and ROUGE-L a 63.41% improvement. This fully demonstrates the superiority of this invention in target localization accuracy, scene inference text generation quality, and inference-mask semantic alignment. A comparison of this application with existing methods shows the following results: Figure 5As shown, (a) columns represent the input images, (b) columns represent the ground truth mask, (c) columns represent the mask from the baseline method, and (d) is the mask from the method of this invention. This demonstrates a pixel-level mask visualization comparison between the method of this invention and the baseline method in a Cityscapes urban road scene. The visualization clearly shows that, compared to the baseline method, the target mask generated by the method of this invention has a higher degree of fit with the ground truth mask, accurately covering the target boundary and effectively avoiding missed or incorrect segmentation. This improvement is particularly evident in the detection and segmentation of small targets in pedestrian and cyclist categories, fully demonstrating the semantic alignment advantages of the method of this invention.

[0031] like Figure 6 The diagram shows the interactive effect of this invention. Compared with existing fusion methods and traditional segmentation methods, this invention's method accurately extracts reasoning semantics related to localization through [SEG] anchor embedding and reasoning feature extraction modules, filtering out irrelevant noise and solving the segmentation deviation problem caused by messy reasoning semantics. This also makes this method significantly improve the localization metrics such as gIoU and cIoU compared with traditional segmentation methods such as DeepLab and PSPNet. Through the cross-modal fusion module, it achieves deep fusion of reasoning semantics and visual features, bridging the cross-modal semantic gap, replacing traditional geometric prompts, and improving the segmentation robustness under complex road conditions. Through end-to-end joint training, it achieves semantic alignment between reasoning and segmentation, ensuring that the target mask is always pixel-level evidence of the reasoning conclusion, effectively solving the core pain point of "disconnect between reasoning and segmentation". This is also the key reason why this method achieves a leapfrog improvement in text generation quality and localization accuracy compared with the LISA method, which only has basic text generation capabilities.

[0032] Therefore, this invention employs a visual question answering and target localization method based on multimodal large model reasoning. The reasoning feature extraction module uses [SEG] markers as anchors to accurately extract reasoning semantics related to localization and filter out irrelevant noise. A cross-modal fusion module achieves deep fusion of reasoning semantics and visual features, bridging the cross-modal semantic gap. End-to-end joint training achieves semantic alignment between reasoning and segmentation, ensuring that the target mask always serves as pixel-level evidence of the reasoning conclusion. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A visual question answering and target localization method based on multimodal large model reasoning, characterized in that: Includes the following steps: Step S1: Obtain the image information I to be processed and the text information corresponding to the scene text query Q. The text information includes target localization related semantics. Preprocess the image information and text information. Step S2: Set up the question-answering processing model, including a multimodal large language model (MLLM), an anchored inference pooling module, an image encoder, a fusion module, and a mask decoder. The working process of the question-answering processing model is as described in steps S3 to S6. Step S3: Use a multimodal large language model to process the preprocessed image and text information in step S1, generate an autoregressive reasoning answer A that conforms to the actual scene, embed the evidence label [SEG] into the target localization related semantics of the multimodal large language model, select the semantic information related to target localization in the hidden state, and obtain the hidden state H with the evidence label [SEG]. Step S4: Input the hidden state H with evidence label [SEG] into the anchored inference pooling module. Using the target localization semantics labeled [SEG] as the query vector, perform attention-weighted aggregation on the hidden state. Extract the compact semantic features most relevant to the target to be localized from the complete inference context, and filter out irrelevant linguistic descriptions to obtain the inference representation. ; Step S5: Use an image encoder to obtain multi-scale visual feature maps F from the image information in step S1, and process the visual feature maps to obtain a global visual context vector. , global visual context vector Compared with the reasoning representation in step S4 The semantically aware conditional vector is obtained by concatenating the data through the fusion module. ; Step S6: Use the semantically aware conditional vector from step S5 Using the visual feature map F, a pixel-level target mask with the same size as the input image is generated by a mask decoder, ensuring that the mask matches the inferred answer. semantic consistency; Step S7: Train the question-answering model using end-to-end joint training until the question-answering model satisfies the loss function. The question-answering model generates an autoregressive reasoning answer A and a target mask image that conform to the actual scene based on the image and text information to be identified.

2. The visual question answering and target localization method based on multimodal large model reasoning according to claim 1, characterized in that: In step S1, the preprocessing steps are as follows: the acquired RGB image is preprocessed, including size normalization and normalization operations, to ensure the consistency of the input image; the scene text query Q adopts a natural language format, covering the core query requirements for images.

3. The visual question answering and target localization method based on multimodal large model reasoning according to claim 1, characterized in that: In step S4, the operation of the anchored inference pooling module is as follows: Perform [SEG] anchor point localization: Construct a binary position mask Select the feature vector in the hidden state H that corresponds to the [SEG] label, where This indicates that the i-th sequence position corresponds to the [SEG] tag semantics. This position is considered semantically irrelevant; the segmentation anchor point is calculated using masked average pooling. The calculation formula is as follows: ； In the above formula, It is a very small constant to prevent division by zero errors. This represents the i-th feature vector of the hidden state; Attention pooling is used to extract inference features: segmentation anchor points Projected onto the query vector through a linear projection layer. , , among them As the key dimension, the hidden state H is mapped to a key matrix through another linear projection layer. Sum matrix V The key matrix Value matrix in, Indicates the sequence length. This represents the hidden dimension features. Each word token in the inference answer A is a reasoning label. Attention weights are calculated. The relevance of each inference token to the [SEG] anchor is measured; the higher the relevance, the greater the weight. The calculation formula is as follows: ； In the above formula, This is a scaling factor for the key dimension, used to avoid attention distribution bias caused by excessively high dimensionality, through attention weights. Log-value matrix V The weighted summation is calculated using the following formula: ； In the above formula, This is a representation of reasoning.

4. The visual question answering and target localization method based on multimodal large model reasoning according to claim 1, characterized in that: The specific implementation process of step S5 is as follows: Image information I is processed using an image encoder to extract multi-scale visual feature maps. Where C is the number of feature channels, The height of the feature map, Represents the width of the feature map; achieved through global average pooling. Folding the spatial dimension transforms multi-scale visual feature maps into global visual feature vectors, which are then linearly projected. Processing with the nonlinear activation function σ generates a global visual context vector. The calculation formula is as follows: ； In the above formula This represents a linear projection operation. This indicates a global average pooling operation; Then, multimodal deep fusion is performed: the inference representation is... With global visual context vector Concatenating along the channel dimension forms a cross-modal feature tensor with dimension . ,in, Indicates batch size, Indicates the hidden feature dimension. express The feature dimension of the cross-modal feature tensor integrates scene reasoning semantics and image visual priors; the cross-modal feature tensor is input into two layers of parameters. The multilayer perceptron (MLP) performs feature fusion and mapping. The first layer of the MLP fuses cross-modal features to obtain fused features, and the second layer of the MLP maps the fused features to a dimension adapted to the mask decoder to generate semantically aware conditional vectors. The formula is as follows: ； In the above formula, This represents the channel concatenation operation, where the semantically aware conditional vector is used. It includes both scene reasoning logic and spatial location information.

5. The visual question answering and target localization method based on multimodal large model reasoning according to claim 4, characterized in that: In step S6, the implementation process is as follows: The mask decoder employs a SAM decoder. After receiving the semantically aware conditional vector c and the visual feature map F, it gradually recovers the feature map size through transposed convolution. Combined with an attention mechanism, it focuses on the target region, ultimately generating a pixel-level target mask with the same size as the input image. The formula is expressed as: ； In the above formula, This represents the mask decoding function of the S-mask decoder, which converts the semantically aware conditional vector... With visual features The process involves fusing the data and outputting a pixel-level target mask. This target mask is then processed by RGB to form a target mask image.

6. The visual question answering and target localization method based on multimodal large model reasoning according to claim 1, characterized in that: In step S7, the training process is as follows: Training employs end-to-end joint optimization, jointly training text generation and target localization tasks within a unified framework. The total loss function consists of two parts: a cross-entropy loss to evaluate the accuracy of the text answer, and a loss to evaluate the localization accuracy. The formula for calculating the total loss function L is as follows: ； In the above formula, The weights representing the autoregressive cross-entropy loss are... Autoregressive cross-entropy loss is used to optimize the text generation accuracy of scene-based visual question answering. For the inference text of the scene predicted by the model, Inference text for real-world scenarios; To divide the overall weight of the loss, The weights represent the calculation weights for the pixel-by-pixel binary cross-entropy loss. Ŵ is the pixel-wise binary cross-entropy loss, used to optimize the pixel-level classification accuracy of the target mask, where Ŵ is the target mask predicted by the model and M is the true target mask. The weights used to calculate the Dice loss are represented. Dice loss is used to address the class imbalance problem in target segmentation for autonomous driving.

Images