Target detection system and method based on multi-modal large language model

Abstract

The invention discloses a target detection system and method based on a multi-modal large language model, and the method employs a DETR to generate a proposal box as an object candidate, and achieves the precise detection of complex targets, such as long-tail classes, combined objects, and the like, through the powerful semantic reasoning capability of an MLLM. The method comprises the following steps: constructing a dual-encoder architecture in which a main visual encoder and an auxiliary visual encoder cooperate, enabling the main visual encoder to continue MLLM global semantic comprehension ability, enabling the auxiliary visual encoder to specially extract local features of an object, and achieving deep complementation of semantic and perceptual information on a dimension level through multi-layer and multi-scale feature splicing; and an object index and coordinate prediction dual-output mode is innovatively designed to form a retrieval-generation dual-guarantee system, so that the target detection precision and robustness are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal large models, and in particular to a target detection system and method based on a multimodal large language model, which is applicable to cross-modal tasks combining visual perception and language understanding. Background Technology

[0002] Existing multimodal large language models (MLLMs) demonstrate some capability in visual understanding tasks, but they suffer from significant limitations in refined object detection scenarios. Traditional methods quantize bounding box coordinates into text tokens for autoregressive prediction, which suffers from problems such as error propagation, prediction order ambiguity, and limitations in quantization range, resulting in low recall rates in complex scenes (e.g., only 43.9% on the COCO dataset). Furthermore, a single visual encoder struggles to simultaneously capture global semantics and local object details, limiting the model's ability to detect multi-granularity objects. Summary of the Invention

[0003] This invention primarily addresses the technical problem of low recall rates in complex scenarios in existing technologies. It provides a target detection system and method based on a multimodal large language model, solving the problems of missed detections and false detections in traditional target detection (OD) in complex scenarios, while retaining the model's original cross-modal understanding capabilities and improving the model's detection accuracy in complex scenarios.

[0004] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a target detection system based on a multimodal large language model, comprising: DETR model: Extracts the coordinate bounding boxes of all foreground objects in the image to be detected and inputs them into the object feature fusion module. The coordinate bounding boxes are the proposal boxes. DETR (Detection Transformer) is an end-to-end object detection model based on the Transformer architecture. In this scheme, a DETR model is pre-trained, which can automatically extract the coordinate bounding boxes of all foreground objects in the image. These bounding boxes serve as proposal boxes for subsequent processing, providing input for object feature fusion and MLLM.

[0005] The main visual encoder takes the image to be detected as input and outputs the last layer as the full-image feature vector, which is then input to the image adapter. The last four hidden states of the main visual encoder are used as multi-layer features of the main visual encoder and input to the object feature fusion module. The main visual encoder employs a pre-trained visual encoder built into a multimodal large language model (such as Qwen ViT), achieving innovation in feature extraction. On one hand, the output of the last layer of the model is used as a full-image feature vector, which is mapped to the input token of the LLM (Large Language Model) through an image adapter (typically composed of multi-layered MLPs). This operation preserves the fundamental capabilities of traditional MLLM models in image semantic understanding, ensuring the model's performance in tasks such as image description and question answering. On the other hand, it innovatively extracts the hidden states of the last four layers of the model as multi-layered feature Fp of the main visual encoder. Compared to the single-mode approach of traditional MLLM models that rely solely on full-image feature vectors, the newly added multi-layered features provide richer semantic information for subsequent object feature extraction, effectively enhancing the model's ability to recognize and locate complex objects in images.

[0006] Image adapter: Composed of several layers of MLP, it maps the feature vectors of the whole image into image tokens that are input into the large language model.

[0007] The auxiliary visual encoder is a high-resolution convolutional structure. The image to be detected is input into the auxiliary visual encoder to construct multi-scale features through several layers of feature maps of different resolutions and then input into the object feature fusion module. Unlike traditional MLLM models with a single primary visual encoder architecture, this solution innovatively introduces an auxiliary visual encoder for object feature pre-extraction. This encoder, based on a high-resolution convolutional structure (such as ConvNext), constructs multi-scale features Fa by stacking feature maps of different resolutions (i.e., the hidden states output between multiple network layers) to form a list. The auxiliary visual encoder captures local object details, providing multi-granularity feature representations from pixel-level to region-level, thus offering rich semantic information for subsequent object feature extraction.

[0008] Object feature fusion module: Based on proposal boxes, multi-layer features and multi-scale feature extraction, fusion is obtained to obtain fused features, and the fused features are input into the object feature adapter.

[0009] Object Feature Adapter: Maps fused features to object tokens that are input into a large language model; This scheme uses a multilayer perceptron (MLP) as an object feature adapter to fuse features V. concat Mapping to the input dimension of the LLM. Assume the LLM input dimension is D. llm Then through V final =MLP(V concat The adapted feature vector V is obtained. final This ensures that visual features are compatible with the input dimensions of the language model, providing a foundation for subsequent cross-modal interaction.

[0010] Text segmenter: Converts text into text tokens that are input into a large language model.

[0011] Large Language Model: Combine object tokens with the index of the proposal box, then combine image tokens and text tokens, input them into the large language model, and output the object detection results.

[0012] This solution is an improvement on the original multimodal large language model, in which both the main visual encoder and the large language model are derived from the original pre-trained multimodal large language model.

[0013] Preferably, the input and output weight matrices of the large language model are concatenated to form a matrix with dimensions [i,Dim], where i is the number of proposal boxes and Dim is the length of the feature dimension.

[0014] This scheme introduces *i* special target tokens (denoted as *obj_1*, *obj_2*, ..., *obj_i*) into the input and output layers of the LLM, corresponding to the indices of the *i* proposal boxes. The addition of these special target tokens is achieved by expanding the weight matrices of the input and output layers. Specifically, the original dimensions of the weight matrices in the input and output layers should be [N, Dim], where N represents the vocabulary size of the model and Dim represents the length of the feature dimension. This scheme concatenates these two weight matrices into a new matrix of [i, Dim] (the values ​​of this weight matrix are randomly initialized), making the dimensions of the weight matrices in the input and output layers [N+i, Dim], with the added *i* weights corresponding to the newly introduced *i* feature target tokens. When inputting into the model, the *i* object tokens extracted from the *i* candidate boxes in the previous steps are interleaved with the *i* special target tokens added here, allowing the model to understand the coordinates of each index and the corresponding object feature information. For example, the specific input should be:<obj_1> <object token_1><obj_2> <object token_2>,...,<obj_i> <object token_i>.

[0015] The model construction process is as follows: Model building steps 1. Pre-trained model initialization: Load the pre-trained multimodal large language model (such as QwenVL), retain its main visual encoder (ViT structure) and language model parameters, and use it as the basic backbone network.

[0016] 2. Accessing the auxiliary visual encoder: A high-resolution convolutional encoder such as ConvNext is accessed in the parallel path of the main visual encoder. Typically, the weights of the visual model pre-trained on a large number of images are used as initialization.

[0017] 3. DETR Proposal Box Generation: The DETR structure is used to generate object candidate boxes, which are then aligned with the real labels using the Hungarian matching algorithm, supporting open set object detection.

[0018] 4. Feature Fusion Layer Construction: For the multi-layer features of the main and auxiliary encoders, region features are extracted by RoI Align, spliced ​​along the dimension after the position embedding, and the dimension is adjusted by the object feature adapter before inputting into LLM.

[0019] 5. Dual-mode output layer design: Update the input and output layers of the MLLM in the MLLM and add i special object tokens, where i is usually 100 or 300.

[0020] Preferably, during training, the pre-trained multimodal large language model weights are loaded and training continues. The training process is divided into two stages: Phase 1: Partial parameter training: Freeze the parameters of other parts of the system and train only the object feature adapter, the input layer of the large language model, and the output layer of the large language model; this phase focuses on adapting the key modules to the task requirements; the training iterations are usually one epoch, that is, the training stops after traversing the data once; Phase 2: Full model training: Unfreeze the parameters and train the entire system to achieve collaborative parameter optimization.

[0021] During training, the loss function is calculated using the cross-entropy loss function, consistent with traditional Multimodal Large Language Models (MLLM) and Large Language Models (LLM). By minimizing the cross-entropy between the predicted output and the true label, the training direction is ensured to align with the optimization objective of the Multimodal Large Language Model, thereby effectively improving the model's ability to understand and process multimodal data.

[0022] A target detection method based on a multimodal large language model, based on the aforementioned target detection system, includes the following steps: S1: Input the image to be detected into the DETR model to extract the proposal boxes of all foreground objects in the image; S2: Input the image to be detected into the main visual encoder. The output of the last layer of the main visual encoder is used as the full image feature vector and input into the image adapter. The image adapter maps the full image feature vector into image tokens that are input into the large language model. The hidden states of the last four layers of the main visual encoder are used as multi-layer features and input into the object feature fusion module. S3: Input the image to be detected into the auxiliary visual encoder, and the auxiliary visual encoder outputs multi-scale features to the object feature fusion module; S4: The object feature fusion module extracts and fuses features based on proposal boxes, multi-layer features and multi-scale features to obtain fused features, and inputs the fused features into the object feature adapter; S5: The object feature adapter maps fused features into object tokens that are input into the large language model; S6: Transform the text input to the text segmenter into text tokens that are input to the large language model; S7: Combine object tokens with the index of the proposal box, then combine image tokens and text tokens, input them into the large language model, and output the object detection results.

[0023] Preferably, step S4 is as follows: S401: Based on the proposal box, candidate region features of the object are extracted from multi-layer features Fp and multi-scale features Fa respectively; specifically, RoI Align technology is used. C j =RoIAlign(F,B j ); Among them B j The j-th proposal box generated for the DETR model, where the total number of j is typically 100 or 300; S402: For each proposal box B j The coordinates are used to perform sine and cosine position encoding to obtain the position embedding PE(B) j The enhanced feature V is obtained by adding the object candidate region features to the corresponding location embedding. j =C j +PE(B j ); S403: Concatenate the enhanced features corresponding to multi-layer features and the enhanced features corresponding to multi-scale features to obtain the fused feature: V concat =Concat(V P V a ); V concat This represents the feature vector obtained after concatenation and fusion, which is the result of feature fusion between the main visual encoder and the auxiliary visual encoder; Concat is the function for concatenation operation, which means concatenating the two feature vectors V within the parentheses. p and V a Segmenting along a certain dimension; V p V represents the enhanced object feature vector obtained by embedding the superimposed position of the object candidate region features extracted by the main visual encoder; a The enhanced object feature vector is obtained by embedding the superimposed position of the object candidate region features extracted by the auxiliary visual encoder.

[0024] As a preferred approach, when the large language model outputs data, if the set of proposal boxes covers the target object, the model outputs the index of the corresponding proposal box. This is the target token output mode. This mode utilizes the prior position of pre-generated proposal boxes and achieves rapid localization through token mapping, significantly improving detection efficiency. If the set of proposal boxes does not cover the target object or there is a localization error, the bounding box coordinates are directly output through the regression head. This is the coordinate regression output mode. This mode is based on autoregressive text generation or numerical prediction mechanisms, effectively compensating for insufficient proposal boxes and enhancing the model's robustness to complex scenes. Whether the target object is covered is determined by the model itself during inference. During training, the training data includes detected targets covered by proposal boxes and targets that are not covered and need to be supplemented by outputting bounding box coordinates, thereby training the model's ability to determine which output mode to use.

[0025] The special target tokens newly introduced in the input and output layers of LLM are not proposal boxes themselves, but rather "indexes" or "codes" pointing to these proposal boxes.

[0026] First, the DETR model analyzes the image and generates a series of (e.g., i=100 or 300) candidate object bounding boxes (proposal boxes).

[0027] These bounding boxes contain specific coordinate data, for example: Box 1: [10, 20, 80, 90], Box 2: [100, 150, 200, 250]... Box i: [x1, y1, x2, y2]. These are the sets of candidate answers.

[0028] To enable the Large Language Model (LLM) to understand and reference these candidate boxes, this scheme adds i special, unique tokens to the LLM vocabulary, such as...<obj_1> ,<obj_2> ,...,<obj_i> Each specific target token corresponds one-to-one with a proposal box generated by DETR. For example: <obj0>↔Box 0; <obj1>↔Box 1; and so on. Therefore, this special target token is an index; it doesn't contain coordinate information, but the system internally knows which bounding box it represents. When inputting into the model, the i object tokens extracted from the i candidate boxes in the previous steps are interleaved with the i special target tokens added here in a corresponding order, allowing the model to know which bounding box each index represents and the corresponding object feature information. For example, the specific input should be:<obj_1> <object token_1><obj_2> <object token_2>,...,<obj_i> <object token_i>. In this way, this solution transforms a complex "coordinate regression" problem into a simpler "classification / selection" problem.

[0029] In the traditional approach, LLM needs to directly generate a string of numbers representing coordinates, such as "car [0.0, 10.0, 50.0, 80.0]". This approach is prone to problems such as error propagation and uncertain order. LLM is not good at accurately generating a long string of continuous numbers.

[0030] The new target token output mode in this scheme is: DETR provides a pre-defined "menu" of 100 candidate boxes. When a user asks, "Where is the car in the picture?", the task of LLM is no longer to describe the car's coordinates. Its task is to select the candidate box that most accurately covers the "car" from these 100 candidate boxes. If LLM determines that the m-th candidate box is the correct answer, it only needs to output the corresponding special token.<obj_m> That's it. This is a single, discrete output, and LLMs are very good at handling these kinds of choices, thus locating the correct option faster and more accurately.

[0031] This dual "retrieval-generation" guarantee mechanism is the key innovation of this solution in improving robustness and accuracy.

[0032] The substantial effects of this invention are: 1. A groundbreaking cross-modal detection innovation through deep integration of DETR and MLLM: This invention innovatively integrates the DETR structure with a multimodal large language model. DETR generates proposal boxes as object candidates, while MLLM leverages its powerful semantic reasoning capabilities to achieve accurate detection of complex targets such as long-tail categories and combined objects. Compared to traditional object detection models, this invention breaks the limitations of fixed category labels and, with the advantage of large-scale model contextual understanding, can efficiently handle complex queries with natural language descriptions such as "a person wearing a green hat," opening a new paradigm for cross-modal object detection. Furthermore, the introduction of object features through an object feature fusion module further enhances the model's understanding and detection capabilities.

[0033] 2. Dual-Visual Encoder Feature Fusion Optimization Strategy: A dual-encoder architecture is constructed, consisting of a primary visual encoder and an auxiliary visual encoder. The primary visual encoder continues the global semantic understanding capability of MLLM, while the auxiliary visual encoder specializes in extracting local object features. Through multi-layer, multi-scale feature concatenation, deep complementarity of semantic and perceptual information is achieved at the dimensional level. Furthermore, an object feature extraction and adaptation mechanism is introduced. RoI Align technology is used to extract candidate region features of objects, which are then combined with location embedding enhancement and dimensional concatenation fusion. Finally, an object feature adapter is used for dimensional mapping, allowing visual features to better adapt to LLM input. Validated on the COCO dataset, this fusion strategy improves target localization accuracy (mAP) by over 50%, significantly outperforming single-encoder solutions.

[0034] 3. Robustness Enhancement Mechanism of Dual-Mode Output: An innovative dual-output mode of object indexing and coordinate prediction is designed to form a "retrieval-generation" dual-guarantee system. When the proposal box is accurate, rapid positioning is achieved through indexing; in extreme cases where the proposal box is missing or incorrect, coordinate generation is used to effectively compensate, successfully avoiding the cascading errors caused by quantization errors in traditional methods, and demonstrating excellent robustness, especially in small target detection scenarios. The introduction of object features also provides richer and more accurate information support for the dual-mode output.

[0035] 4. Lightweight Incremental Training and Feature Expansion Solution: Incremental training is performed based on existing pre-trained MLLMs (such as QwenVL), eliminating the need to train a large language model from scratch and significantly reducing computational resource consumption. By loading pre-trained MLLM model weights for further training, the model retains its original image description and question-answering functions while adding object detection capabilities in a "plug-and-play" manner, significantly improving the model's practicality and scalability. The adaptation process for object features is also achieved through lightweight adjustments, further ensuring the efficiency of model feature expansion. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the target detection system structure according to the present invention; In the diagram: 1-DETR model; 2-Main visual encoder; 3-Image adapter; 4-Auxiliary visual encoder; 5-Object feature fusion module; 6-Object feature adapter; 7-Text segmenter; 8-Large language model. Detailed Implementation

[0037] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0038] Example: A target detection system based on a multimodal large language model, such as Figure 1 As shown, it includes: DETR model: Extracts the coordinate bounding boxes of all foreground objects in the image to be detected and inputs them into the object feature fusion module. The coordinate bounding boxes are the proposal boxes. DETR (Detection Transformer) is an end-to-end object detection model based on the Transformer architecture. In this scheme, a DETR model is pre-trained, which can automatically extract the coordinate bounding boxes of all foreground objects in the image. These bounding boxes serve as proposal boxes for subsequent processing, providing input for object feature fusion and MLLM.

[0039] The main visual encoder takes the image to be detected as input and outputs the last layer as the full-image feature vector, which is then input to the image adapter. The last four hidden states of the main visual encoder are used as multi-layer features of the main visual encoder and input to the object feature fusion module. The main visual encoder employs a pre-trained visual encoder built into a multimodal large language model (such as Qwen ViT), achieving innovation in feature extraction. On one hand, the output of the last layer of the model is used as a full-image feature vector, which is mapped to the input token of the LLM (Large Language Model) through an image adapter (typically composed of multi-layered MLPs). This operation preserves the fundamental capabilities of traditional MLLM models in image semantic understanding, ensuring the model's performance in tasks such as image description and question answering. On the other hand, it innovatively extracts the hidden states of the last four layers of the model as multi-layered feature Fp of the main visual encoder. Compared to the single-mode approach of traditional MLLM models that rely solely on full-image feature vectors, the newly added multi-layered features provide richer semantic information for subsequent object feature extraction, effectively enhancing the model's ability to recognize and locate complex objects in images.

[0040] Image adapter: Composed of several layers of MLP, it maps the feature vectors of the whole image into image tokens that are input into the large language model.

[0041] The auxiliary visual encoder is a high-resolution convolutional structure. The image to be detected is input into the auxiliary visual encoder to construct multi-scale features through several layers of feature maps of different resolutions and then input into the object feature fusion module. Unlike traditional MLLM models with a single primary visual encoder architecture, this solution innovatively introduces an auxiliary visual encoder for object feature pre-extraction. This encoder, based on a high-resolution convolutional structure (such as ConvNext), constructs multi-scale features Fa by stacking feature maps of different resolutions (i.e., the hidden states output between multiple network layers) to form a list. The auxiliary visual encoder captures local object details, providing multi-granularity feature representations from pixel-level to region-level, thus offering rich semantic information for subsequent object feature extraction.

[0042] Object feature fusion module: Based on proposal boxes, multi-layer features and multi-scale feature extraction, fusion is obtained to obtain fused features, and the fused features are input into the object feature adapter.

[0043] Object Feature Adapter: Maps fused features to object tokens that are input into a large language model; This scheme uses a multilayer perceptron (MLP) as an object feature adapter to fuse features V. concat Mapping to the input dimension of the LLM. Assume the LLM input dimension is D. llm Then through V final =MLP(V concat The adapted feature vector V is obtained. final This ensures that visual features are compatible with the input dimensions of the language model, providing a foundation for subsequent cross-modal interaction.

[0044] Text segmenter: Converts text into text tokens that are input into a large language model.

[0045] Large Language Model: Combine object tokens with the index of the proposal box, then combine image tokens and text tokens, input them into the large language model, and output the object detection results.

[0046] This solution is an improvement on the original multimodal large language model, in which the main visual encoder, large language model and text segmenter are all derived from the original pre-trained multimodal large language model.

[0047] The input and output weight matrices of a large language model are concatenated to form a matrix of dimension [i, Dim], where i is the number of proposal boxes and Dim is the length of the feature dimension.

[0048] This scheme introduces *i* special target tokens (denoted as *obj_1*, *obj_2*, ..., *obj_i*) into the input and output layers of the LLM, corresponding to the indices of the *i* proposal boxes. The addition of these special target tokens is achieved by expanding the weight matrices of the input and output layers. Specifically, the original dimensions of the weight matrices in the input and output layers should be [N, Dim], where N represents the vocabulary size of the model and Dim represents the length of the feature dimension. This scheme concatenates these two weight matrices into a new matrix of [i, Dim] (the values ​​of this weight matrix are randomly initialized), making the dimensions of the weight matrices in the input and output layers [N+i, Dim], with the added *i* weights corresponding to the newly introduced *i* feature target tokens. When inputting into the model, the *i* object tokens extracted from the *i* candidate boxes in the previous steps are interleaved with the *i* special target tokens added here, allowing the model to understand the coordinates of each index and the corresponding object feature information. For example, the specific input should be:<obj_1> <object token_1><obj_2> <object token_2>,...,<obj_i> <object token_i>.

[0049] The model construction process is as follows: Model building steps 1. Pre-trained model initialization: Load the pre-trained multimodal large language model (such as QwenVL), retain its main visual encoder (ViT structure) and language model parameters, and use it as the basic backbone network.

[0050] 2. Accessing the auxiliary visual encoder: A high-resolution convolutional encoder such as ConvNext is accessed in the parallel path of the main visual encoder. Typically, the weights of the visual model pre-trained on a large number of images are used as initialization.

[0051] 3. DETR Proposal Box Generation: The DETR structure is used to generate object candidate boxes, which are then aligned with the real labels using the Hungarian matching algorithm, supporting open set object detection.

[0052] 4. Feature Fusion Layer Construction: For the multi-layer features of the main and auxiliary encoders, region features are extracted by RoI Align, spliced ​​along the dimension after the position embedding, and the dimension is adjusted by the object feature adapter before inputting into LLM.

[0053] 5. Dual-mode output layer design: Update the input and output layers of the MLLM in the MLLM and add i special object tokens, where i is usually 100 or 300.

[0054] During training, the pre-trained multimodal large language model weights are loaded and training continues. The training process is divided into two stages: Phase 1: Partial Parameter Training: Freeze the parameters of other parts of the system and train only the object feature adapter, the input layer of the large language model, and the output layer of the large language model; this phase focuses on adapting key modules to task requirements. Phase 2: Full model training: Unfreeze the parameters and train the entire system to achieve collaborative parameter optimization.

[0055] During training, the loss function is calculated using the cross-entropy loss function, consistent with traditional Multimodal Large Language Models (MLLM) and Large Language Models (LLM). By minimizing the cross-entropy between the predicted output and the true label, the training direction is ensured to align with the optimization objective of the Multimodal Large Language Model, thereby effectively improving the model's ability to understand and process multimodal data.

[0056] A target detection method based on a multimodal large language model, based on the aforementioned target detection system, includes the following steps: S1: Input the image to be detected into the DETR model to extract the proposal boxes of all foreground objects in the image; S2: Input the image to be detected into the main visual encoder. The output of the last layer of the main visual encoder is used as the full image feature vector and input into the image adapter. The image adapter maps the full image feature vector into image tokens that are input into the large language model. The hidden states of the last four layers of the main visual encoder are used as multi-layer features and input into the object feature fusion module. S3: Input the image to be detected into the auxiliary visual encoder, and the auxiliary visual encoder outputs multi-scale features to the object feature fusion module; S4: The object feature fusion module extracts and fuses features based on proposal boxes, multi-layer features and multi-scale features to obtain fused features, and inputs the fused features into the object feature adapter; S5: The object feature adapter maps fused features into object tokens that are input into the large language model; S6: Transform the text input to the text segmenter into text tokens that are input to the large language model; S7: Combine object tokens with the index of the proposal box, then combine image tokens and text tokens, input them into the large language model, and output the object detection results.

[0057] Step S4 is as follows: S401: Based on the proposal box, candidate region features of the object are extracted from multi-layer features Fp and multi-scale features Fa respectively; specifically, RoI Align technology is used. C j =RoIAlign(F,B j ); Among them B j The j-th proposal box generated for the DETR model, where the total number of j is typically 100 or 300; S402: For each proposal box B j The coordinates are used to perform sine and cosine position encoding to obtain the position embedding PE(B) j The enhanced feature V is obtained by adding the object candidate region features to the corresponding location embedding. j =C j +PE(B j ); S403: Concatenate the enhanced features corresponding to multi-layer features and the enhanced features corresponding to multi-scale features to obtain the fused feature: V concat =Concat(V P V a ); V concat This represents the feature vector obtained after concatenation and fusion, which is the result of feature fusion between the main visual encoder and the auxiliary visual encoder; Concat is the function for concatenation operation, which means concatenating the two feature vectors V within the parentheses. p and V a Segmenting along a certain dimension; V p V represents the enhanced object feature vector obtained by embedding the superimposed position of the object candidate region features extracted by the main visual encoder; a The enhanced object feature vector is obtained by embedding the superimposed position of the object candidate region features extracted by the auxiliary visual encoder.

[0058] When the large language model outputs its data, if the set of proposal boxes covers the target object, it outputs the index of the corresponding proposal box. This is the target token output mode. This mode utilizes the prior position of pre-generated proposal boxes and achieves fast localization through token mapping, significantly improving detection efficiency. If the set of proposal boxes does not cover the target object or there is a localization error, it directly outputs the bounding box coordinates through the regression head. This is the coordinate regression output mode. This mode is based on autoregressive text generation or numerical prediction mechanisms, effectively compensating for insufficient proposal boxes and enhancing the model's robustness to complex scenes. Whether the target object is covered is determined by the model itself during inference. During training, the training data includes detected targets covered by proposal boxes and targets that are not covered and need to be supplemented by outputting bounding box coordinates, thereby training the model's ability to determine which output mode to use.

[0059] The proposed dual-visual encoder fusion architecture combines the semantic understanding advantages of pre-trained MLLM with the object feature extraction capabilities of the auxiliary encoder. Through feature fusion and dual-mode output mechanisms, it effectively improves the accuracy and robustness of object detection without affecting the original model's functionality.

[0060] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0061] Although this paper frequently uses terms such as primary visual encoder, secondary visual encoder, and object feature fusion module, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A target detection system based on a multimodal large language model, characterized in that, include: DETR model: Extracts the coordinate bounding boxes of all foreground objects in the image to be detected and inputs them into the object feature fusion module. The coordinate bounding boxes are the proposal boxes. The main visual encoder takes the image to be detected as input and outputs the last layer as the full-image feature vector, which is then input to the image adapter. The last four hidden states of the main visual encoder are used as multi-layer features of the main visual encoder and input to the object feature fusion module. Image adapter: Consists of several layers of MLP, which maps the feature vectors of the whole image into image tokens input to the large language model; The auxiliary visual encoder is a high-resolution convolutional structure. The image to be detected is input into the auxiliary visual encoder to construct multi-scale features through several layers of feature maps of different resolutions and then input into the object feature fusion module. The object feature fusion module extracts and fuses features based on proposal boxes, multi-layer features, and multi-scale features to obtain fused features, and then inputs the fused features into the object feature adapter. Object Feature Adapter: Maps fused features to object tokens that are input into a large language model; Text segmenter: Converts text into text tokens that are input into a large language model; Large Language Model: Combine object tokens with the index of the proposal box, then combine image tokens and text tokens, input them into the large language model, and output the object detection results.

2. The target detection system based on a multimodal large language model according to claim 1, characterized in that, The input and output weight matrices of the large language model are concatenated to form a matrix of dimension [i, Dim], where i is the number of proposal boxes and Dim is the length of the feature dimension.

3. A target detection system based on a multimodal large language model according to claim 1 or 2, characterized in that, During training, the pre-trained multimodal large language model weights are loaded and training continues. The training process is divided into two stages: Phase 1: Partial parameter training: Freeze the parameters of other parts of the system and train only the object feature adapter, the input layer of the large language model, and the output layer of the large language model; Phase 2: Full model training: Unfreeze the parameters and train the entire system.

4. A target detection method based on a multimodal large language model, based on the target detection system of claim 1, characterized in that, Includes the following steps: S1: Input the image to be detected into the DETR model to extract the proposal boxes of all foreground objects in the image; S2: Input the image to be detected into the main visual encoder. The output of the last layer of the main visual encoder is used as the full image feature vector and input into the image adapter. The image adapter maps the full image feature vector into image tokens that are input into the large language model. The hidden states of the last four layers of the main visual encoder are used as multi-layer features and input into the object feature fusion module. S3: Input the image to be detected into the auxiliary visual encoder, and the auxiliary visual encoder outputs multi-scale features to the object feature fusion module; S4: The object feature fusion module extracts and fuses features based on proposal boxes, multi-layer features and multi-scale features to obtain fused features, and inputs the fused features into the object feature adapter; S5: The object feature adapter maps fused features into object tokens that are input into the large language model; S6: Transform the text input to the text segmenter into text tokens that are input to the large language model; S7: Combine object tokens with the index of the proposal box, then combine image tokens and text tokens, input them into the large language model, and output the object detection results.

5. The target detection method based on a multimodal large language model according to claim 4, characterized in that, The specific steps of S4 shown are as follows: S401: Extract candidate region features of objects based on multi-layer and multi-scale features respectively, using proposal boxes; S402: Perform sine and cosine position encoding on the coordinates of each proposal box to obtain the position embedding, and add the object candidate region features to the corresponding position embedding to obtain the enhanced features; S403: The enhanced features corresponding to the multi-layer features and the enhanced features corresponding to the multi-scale features are concatenated to obtain the fused features.

6. A target detection method based on a multimodal large language model according to claim 4 or 5, characterized in that, When the large language model outputs, if the set of proposal boxes covers the target object, the large language model outputs the index of the corresponding proposal box; if the set of proposal boxes does not cover the target object or there is a positioning error, the bounding box coordinates are directly output through the regression head.

Images