Meta-learning based multi-modal large model task-driven target detection method and system

Abstract

The application provides a multi-modal large model task-driven target detection method and system based on meta learning, relates to the technical field of target detection, and comprises the following steps: obtaining multi-modal input data and performing preprocessing to obtain unified semantic representation; a general candidate generation network and a large language model are used to generate a candidate region set and a visual attribute word group set; region feature extraction is performed on each candidate target frame in the candidate region set to obtain a candidate visual embedding sequence; the visual attribute word group set is subjected to vectorization processing to obtain an attribute text embedding sequence; the candidate visual embedding sequence and the attribute text embedding sequence are jointly calibrated to obtain a calibrated candidate visual embedding sequence and a calibrated attribute text embedding sequence; a task-related score is calculated through a trainable scoring function, and all candidate frames in the candidate region set are screened to obtain a target detection result. The application realizes target detection guided by task description and has the ability of cross-task rapid adaptation.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a multimodal large model task-driven target detection method and system based on meta-learning. Background Technology

[0002] Object detection is one of the core tasks in computer vision, aiming to identify and locate objects of interest in a given image. Traditional object detection methods typically rely on supervised training with a large number of labeled samples. The detection categories of the model are fixed during the training phase, making it difficult to flexibly adapt to the dynamically changing task requirements in open scenes. With the development of deep learning, end-to-end detection frameworks, represented by Faster R-CNN and the YOLO series, have achieved good performance on closed category sets. However, their generalization ability is still limited by the category distribution of the training data. When facing new categories or complex semantic tasks, it is often necessary to collect data again and retrain the model, which is costly in engineering.

[0003] In recent years, the rise of multimodal pre-trained models has brought new ideas to object detection tasks. Visual language pre-trained models, represented by CLIP, have demonstrated strong generalization ability in semantic alignment between images and text through large-scale image-text contrastive learning, promoting the development of research directions such as open-vocabulary object detection. At the same time, the capabilities of large language models in semantic understanding and text generation have been gradually introduced into visual tasks to provide rich semantic priors. However, how to effectively combine multimodal semantic representations with region-level visual features in object detection tasks and maintain good adaptability across cross-task scenarios remains one of the main challenges facing current research.

[0004] Chinese invention patent application CN119538179A, entitled "Fine-Grained Multimodal Cue Learning Method Based on Visual Language Pre-trained Model," discloses a method for cross-modal alignment using learnable visual cues and class-aware text cues. This method designs a global visual cue branch and a fine-grained visual cue branch, and leverages a shared semantic space to achieve collaborative optimization of visual and textual features, achieving good generalization results in multiple downstream image classification tasks. However, it does not consider how to quickly adapt and transfer the key parameters of the detection model across tasks under dynamically changing task descriptions, making it difficult to directly apply to complex detection scenarios requiring target localization based on task semantics. Summary of the Invention

[0005] In view of this, the present invention provides a multimodal large model task-driven object detection method and system based on meta-learning, which can solve the problems of existing multimodal object detection methods being unable to accurately correlate task semantics and region-level visual features under dynamic task changes, and the difficulty in adapting model parameters. It realizes object detection guided by task description and has the ability to quickly adapt across tasks.

[0006] The technical solution of this invention is implemented as follows: On the one hand, this invention provides a multimodal large model task-driven object detection method based on meta-learning, including: S1. Obtain multimodal input data containing images and text, perform feature extraction and cross-modal alignment to obtain aligned multimodal features, and input the aligned multimodal features into a multimodal cross-attention fusion encoder to obtain a unified semantic representation; S2. A general candidate generation network is used to scan the images in the multimodal input data to generate a set of candidate regions. The task description is then input into a large language model to generate a set of visual attribute phrases. S3. Extract region features from each candidate bounding box in the candidate region set to obtain a candidate visual embedding sequence; vectorize the visual attribute phrase set to obtain an attribute text embedding sequence; input the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner, and perform joint calibration through bidirectional cross-attention from vision to text and from text to vision to obtain the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence. S4. Using the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence as input, the task-related score of each candidate target box is calculated through a trainable scoring function, and the candidate region set is filtered based on the task-related score to obtain the target detection result.

[0007] Based on the above technical solutions, preferably, the multimodal input data includes image data and text data, and step S1 specifically includes: The image data is divided into fixed-size image blocks and converted into an image sequence. The text data is then masked by covering the words according to a preset mask ratio to form a masked text sequence. The image sequence and the masked text sequence are respectively input into the visual encoder and the text encoder for feature extraction, resulting in the visual token sequence embedding sequence. and text token sequence embedded sequence ,in Special classification morphemes Visual representation, For the first Visual embeddings corresponding to image patches For embedded dimensions, The total number of image patches, Special classification morphemes Textual representation, Indicates the total number of lexical units; Based on the visual token sequence embedding sequence and the text token sequence embedding sequence, each of the following... Characterization, selecting the most similar mode from the opposing modes based on cosine similarity. 10 samples constitute the image candidate set With text candidate set ; A contrastive learning strategy is used for the image candidate set. With text candidate set Cross-modal alignment is performed to obtain aligned multimodal features, which are then input into a multimodal cross-attention fusion encoder to obtain a unified semantic representation.

[0008] Based on the above technical solutions, preferably, the cross-modal alignment specifically includes: Based on the linear similarity function, the image candidate sets are respectively... With text candidate set Calculate the Softmax normalized similarity score: ; ; ; in, Representing an image With text candidate set The Middle text Image-to-text normalized similarity scores between the two texts. This represents the natural exponential function. The linear similarity function representing image-text pairs, This indicates the currently queried image sample. Represents the text candidate set The m-th text in the middle, This indicates a learnable temperature parameter. Indicates the candidate sample index. Representing text With image candidate set The Middle One image Text-to-image normalized similarity scores between the two. This represents the current query text sample. Represents the candidate set of images The m-th image sample, The linear similarity score scalar representing the image-text pair. Represents the visual linear transformation function. Represents a text linear transformation function; Aligning features from different modalities within the same sample to a shared low-dimensional space yields aligned multimodal features.

[0009] Based on the above technical solutions, preferably, the step of inputting the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner specifically includes: Region feature extraction is performed on each candidate bounding box in the candidate region set to obtain a candidate visual embedding sequence. ; The set of visual attribute phrases is vectorized by a text encoder to obtain the attribute text embedding sequence. ; embedding candidate visual sequences With attribute text embedding sequence A common input embedding recalibrator is used for bidirectional joint calibration to obtain the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence.

[0010] Based on the above technical solutions, preferably, the embedded recalibration aligner adopts a lightweight Transformer cross-attention architecture, including a cross-attention submodule for the visual-to-text modality and a cross-attention submodule for the text-to-visual modality, wherein each cross-attention submodule is calculated using a scaled dot product attention method. For the cross-attention submodule of the visual-to-text modality, with candidate visual embedding sequences For query source, embed sequence with attribute text Using the key-value source, perform the following update to obtain the calibrated candidate visual embedding. : ; ; ; ; ; in, Represents the query matrix. Represents a value matrix, A trainable projection matrix representing a visual query. Represents the key matrix, This indicates an attribute text embedding sequence. The trainable projection matrix representing the text keys. Indicates a sequence of embedded attribute text. through The transformed value matrix, A trainable projection matrix representing text values. This represents the output of visual-to-text cross-attention. This represents the scaling dot product attention calculation function. For layer normalization operators, It is a feedforward network. This represents the calibrated candidate visual embedding sequence; For the text-to-visual modality cross-attention submodule, the attribute text embedding sequence is used. For query source, candidate visual embedding sequence Using the key-value source, the cross-attention submodule of the visual-to-text modality is updated to obtain the calibrated attribute text embedding sequence. : ; ; ; ; ; in, Represents a text query matrix. The trainable projection matrix representing a text query. Represents the key matrix, The trainable projection matrix representing the visual key. Represents a sequence of visual embeddings via projection matrix The transformed value matrix, The trainable projection matrix representing the visual value. The output represents the cross-attention from text to visual modality. This represents the calibrated attribute text embedding sequence.

[0011] Based on the above technical solutions, preferably, step S4 specifically includes: Calculate the calibrated attribute text embedding sequence The semantic similarity between attribute phrases is determined, and the calibrated attribute text is embedded into the sequence based on the semantic similarity. It is divided into several semantic subsets, and each subset corresponds to a task semantic dimension; Within each semantic subset, attention-weighted aggregation is performed to obtain the corresponding semantic sub-vectors; The semantic sub-vectors are fused to obtain the task attribute vector that represents the overall semantics of the current task. ; Using task attribute vectors With calibrated candidate visual embedding sequences The visual embeddings of each candidate bounding box are used as input, and a task-related score for each candidate bounding box is calculated using a trainable scoring function. The task-related score Indicates the probability that the candidate bounding box fits the current task: ; ; ; in, This represents the task-related score of the j-th candidate bounding box. This represents the Sigmoid function. This represents the transpose of the trainable score weight vector. Represents the calibrated candidate visual embedding sequence The visual embedding corresponding to the j-th candidate target box in the image. Represents a task attribute vector. Represents element-wise product. This represents the confidence level of the j-th candidate bounding box. Indicates the index of the attribute text embedding. Indicates the total number of attribute phrases. This represents the attention weight embedded in the i-th attribute. Represents the calibrated attribute text embedding sequence The text embedding corresponding to the i-th attribute phrase in the text. This represents the transpose of the trainable attribute aggregation query vector. This represents the normalized summation index of the attribute embedding. Represents the calibrated attribute text embedding sequence The text embedding corresponding to the k-th attribute phrase; Based on task-related scores For candidate region set Perform threshold filtering sequentially and retain the set. , The preset threshold; set The task-related scores were sorted in descending order, and the Top-M candidate boxes were selected. Non-maximum suppression is applied to the Top-M candidate boxes according to the preset IoU threshold, and the target detection results are output.

[0012] Based on the above technical solutions, preferably, the loss function for object detection includes contrastive learning loss, scoring function supervision loss, and object detection loss: ; ; ; ; in, Represents the joint loss function for multiple objectives. Indicates contrastive learning loss The weighting coefficients, Represents the supervised loss of the scoring function. The weighting coefficients, Indicates target detection loss The weighting coefficients, Indicates the contrast learning loss. This represents the supervised loss of the scoring function. Indicates the target detection loss. This represents all image-text pairs in the multimodal data distribution D. Seeking expectations, Represents the cross-entropy loss function. Representing an image In the text candidate set The one-hot distribution of the true matching samples. Representing an image With text candidate set Image-to-text normalized similarity scores for each text. Representing text In the image candidate set The one-hot distribution of the true matching samples. Representing text With image candidate set Text-to-image normalized similarity scores for each image. Indicates the index of the candidate target box. This indicates the total number of candidate bounding boxes. The task target supervision marker represents the j-th candidate target bounding box. This represents the task-related score of the j-th candidate bounding box.

[0013] Furthermore, the parameters of the embedded recalibrated aligner and the trainable scoring function are obtained through meta-learning. Specifically, a task set containing multiple tasks is constructed, and a Reptile-based meta-learning algorithm is used to iteratively update the parameters of the embedded recalibrated aligner and the trainable scoring function across tasks. The outer loop step size is adaptively adjusted based on the task representation vector obtained by aggregating the calibrated attribute text embedding sequences of each task. After the iteration converges, the obtained meta-initialization parameters are used as the starting point for fast parameter adaptation of the embedded recalibrated aligner and the trainable scoring function on the new task.

[0014] Based on the above technical solutions, preferably, the meta-learning process specifically includes: Step 1: Construct a task set containing multiple tasks For each task Sampling support set With query set ; Step 2: Denote the parameters of the embedded recalibration aligner and the parameters of the trainable scoring function as the meta-initialization parameters. ; Step 3: For any task , set the initialization parameters The copy serves as the starting point for the parameters of this task, within the support set. Execute preset Step-by-step gradient iteration, with each step updating in the following form: ; in, Indicates the first The parameters after one iteration Indicates task The parameters after the gradient iteration at step t, For the inner loop learning rate, Indicates the parameter Operator for finding gradient Indicates task loss; Step 4: Aggregate the calibrated attribute text embeddings T′ of the current task τ to obtain the task representation vector. ,Will The input scalar mapping function is used to calculate the adaptive outer loop step size and update the meta-initialization parameters: ; ; in Based on the outer loop step size, It is a scalar mapping function; Step 5: Repeat steps 1 through 4 until convergence, and obtain the meta-initialization parameters. This serves as the starting point for the embedded recalibration aligner and the trainable scoring function to perform fast parameter adaptation on the new task.

[0015] In addition, the present invention also provides a multimodal large model task-driven target detection system based on meta-learning to implement the above-mentioned method, comprising: The data acquisition module is used to acquire multimodal input data containing images and text, perform feature extraction and cross-modal alignment to obtain aligned multimodal features, and input the aligned multimodal features into the multimodal cross-attention fusion encoder to obtain a unified semantic representation; The data processing module is used to scan the images in the multimodal input data using a general candidate generation network, generate a set of candidate regions, and input the task description into a large language model to generate a set of visual attribute phrases. The feature correction module is used to extract region features from each candidate target box in the candidate region set to obtain a candidate visual embedding sequence; to vectorize the visual attribute phrase set to obtain an attribute text embedding sequence; and to input the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner, and to perform joint calibration through bidirectional cross attention from vision to text and text to vision to obtain the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence. The target detection module takes the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence as input, calculates the task-related score of each candidate target box through a trainable scoring function, and filters the candidate region set based on the task-related score to obtain the target detection result.

[0016] The present invention has the following advantages over the prior art: This invention forms a unified semantic representation by fusing multimodal inputs through cross-modal alignment and cross-attention. Visual attribute phrases generated by a large language model serve as a structured expression of task semantics, driving the scoring and selection of candidate regions. This allows the overall method to balance the accuracy of semantic understanding and the flexibility of cross-task adaptation in task-driven object detection scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the multimodal large model task-driven target detection method based on meta-learning of the present invention; Figure 2 This is a framework diagram of the multimodal large model task-driven target detection method based on meta-learning of the present invention; Figure 3 This is a flowchart illustrating the multimodal large model task-driven target detection method based on meta-learning of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 and Figure 2 As shown, this invention provides a multimodal large model task-driven target detection method based on meta-learning, comprising: S1. Obtain multimodal input data containing images and text, perform feature extraction and cross-modal alignment to obtain aligned multimodal features, and input the aligned multimodal features into a multimodal cross-attention fusion encoder to obtain a unified semantic representation; Specifically, step S1 includes: The image data is divided into fixed-size image blocks and converted into an image sequence. The text data is then masked by covering the words according to a preset mask ratio to form a masked text sequence. The image sequence and the masked text sequence are respectively input into the visual encoder and the text encoder for feature extraction, resulting in the visual token sequence embedding sequence. and text token sequence embedded sequence ,in Special classification morphemes Visual representation, For the first Visual embeddings corresponding to image patches For embedded dimensions, The total number of image patches, Special classification morphemes Textual representation, Indicates the total number of lexical units; Based on the visual token sequence embedding sequence and the text token sequence embedding sequence, each of the following... Characterization, selecting the most similar mode from the opposing modes based on cosine similarity. 10 samples constitute the image candidate set With text candidate set ; A contrastive learning strategy is used for the image candidate set. With text candidate set Cross-modal alignment is performed to obtain aligned multimodal features, which are then input into a multimodal cross-attention fusion encoder to obtain a unified semantic representation.

[0021] The image modality feature extraction process employs a multi-layer Transformer encoder to process the image sequence. Each encoder layer includes a multi-head self-attention mechanism, a feedforward neural network, and residual connections. After processing by the encoder, the sequence is subjected to global average pooling and then mapped and output through a fully connected layer.

[0022] In one embodiment of the present invention, text modality feature extraction is processed based on the BERT architecture, with masked language modeling as the pre-training objective, and the loss function is: in, This represents the pre-training loss for masked language modeling. This represents all text sequences containing masks in the text data distribution D. Seeking expectations, Represents the distribution of text data. Represents the cross-entropy loss function. This represents the one-hot distribution of the actual words corresponding to the mask positions. This represents the probability distribution of the BERT architecture's predictions for words at mask positions. This represents a text sequence containing a mask.

[0023] Furthermore, the cross-modal alignment specifically includes: Based on the linear similarity function, the image candidate sets are respectively... With text candidate set Calculate the Softmax normalized similarity score: in, Representing an image With text candidate set The Middle text Image-to-text normalized similarity scores between the two texts. This represents the natural exponential function. This represents a linear similarity function between image-text pairs. This indicates the currently queried image sample. Represents the text candidate set The m-th text in the middle, This indicates a learnable temperature parameter. Indicates the candidate sample index; accordingly, Representing text With image candidate set The Middle One image Text-to-image normalized similarity scores between the two. This represents the current query text sample. Represents the candidate set of images The m-th image sample, The linear similarity score scalar representing the image-text pair. Represents the visual linear transformation function. These represent text linear transformation functions that map image and text features to a normalized low-dimensional representation. Aligning features from different modalities within the same sample to a shared low-dimensional space yields aligned multimodal features.

[0024] Understandably, the embedding recalibrator is used to recalibrate the local visual features of the candidate target and the text representation composed of task attribute phrases, so that the candidate visual embedding sequence and the attribute text embedding sequence are more matched in the joint space, thereby reducing misselection and omission caused by the inconsistency between "object image and task attribute".

[0025] S2. A general candidate generation network is used to scan the image in the multimodal input data to generate a set of candidate regions. ,in For the first Candidate target boxes, The confidence level is determined, and the task description is input into a large language model to generate a set of visual attribute phrases. ,in This represents the i-th visual attribute phrase. Indicates the visual attribute phrase index. This represents the total number of visual attribute phrases, which are structured text descriptions of the visual features of the task target. S3. Extract region features from each candidate bounding box in the candidate region set to obtain a candidate visual embedding sequence; vectorize the visual attribute phrase set to obtain an attribute text embedding sequence; input the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner, and perform joint calibration through bidirectional cross-attention from vision to text and from text to vision to obtain the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence. Specifically, the step of inputting the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner includes: For candidate region set Candidate target boxes in Perform region feature extraction to obtain candidate visual embedding sequences. ,in For the first The visual embeddings corresponding to each candidate target bounding box This indicates the total number of candidate bounding boxes. Indicates the embedding dimension; Set of visual attribute phrases After vectorization by the text encoder, the attribute text embedding sequence is obtained. ,in For the first Text embedding corresponding to each attribute phrase Indicates the total number of attribute phrases; embedding candidate visual sequences With attribute text embedding sequence A bidirectional joint calibration is performed using a common input embedding recalibration aligner to obtain the calibrated candidate visual embedding sequence. With calibrated attribute text embedding sequence .

[0026] In one embodiment of the present invention, the embedded recalibration aligner adopts a lightweight Transformer cross-attention architecture, including a cross-attention submodule for the visual-to-text modality and a cross-attention submodule for the text-to-visual modality, wherein each cross-attention submodule is calculated using a scaled dot product attention form: in, This represents the scaling dot product attention calculation function. For querying the matrix, The key matrix, For value matrices, The dimension of the key vector; For the cross-attention submodule of the visual-to-text modality, with candidate visual embedding sequences For query source, embed sequence with attribute text Using the key value source, perform the following update to obtain the calibrated candidate visual embedding sequence. : in, Represents the query matrix. Represents a value matrix, A trainable projection matrix representing a visual query. Represents the key matrix, This indicates an attribute text embedding sequence. The trainable projection matrix representing the text keys. Indicates a sequence of embedded attribute text. through The transformed value matrix, A trainable projection matrix representing text values. This represents the output of visual-to-text cross-attention. This represents the scaling dot product attention calculation function. For layer normalization operators, It is a feedforward network. This represents the calibrated candidate visual embedding sequence; For the text-to-visual modality cross-attention submodule, the attribute text embedding sequence is used. For query source, candidate visual embedding sequence Using the key-value source, the cross-attention submodule of the visual-to-text modality is updated to obtain the calibrated attribute text embedding sequence. : in, Represents a text query matrix. Represents a value matrix, The trainable projection matrix representing a text query. Represents the key matrix, This indicates an attribute text embedding sequence. The trainable projection matrix representing the visual key. Represents a sequence of visual embeddings via projection matrix The transformed value matrix, The trainable projection matrix representing the visual value. The output represents the cross-attention from text to visual modality. This represents the scaling dot product attention calculation function. For layer normalization operators, It is a feedforward network. This represents the calibrated attribute text embedding sequence; This invention uses an embedding recalibration aligner to perform bidirectional joint calibration of candidate visual embedding sequences and attribute text embedding sequences, thereby improving the discriminative ability of subsequent scoring.

[0027] S4. Using the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence as input, the task-related score of each candidate target box is calculated through a trainable scoring function, and the candidate region set is filtered based on the task-related score to obtain the target detection result.

[0028] Specifically, step S4 includes: Calculate the calibrated attribute text embedding sequence The semantic similarity between attribute phrases is determined, and the calibrated attribute text is embedded into the sequence based on the semantic similarity. It is divided into several semantic subsets, and each subset corresponds to a task semantic dimension; Within each semantic subset, attention-weighted aggregation is performed to obtain the corresponding semantic sub-vectors; The semantic sub-vectors are fused to obtain the task attribute vector that represents the overall semantics of the current task. ; Using the task attribute vector a and the calibrated candidate visual embedding sequence The task-related score of each candidate bounding box is calculated using a trainable scoring function, with the visual embedding sequence of each candidate bounding box as input. The task-related score Indicates the probability that the candidate bounding box fits the current task: in, This represents the task-related score of the j-th candidate bounding box. This represents the Sigmoid function. This represents the transpose of the trainable score weight vector. Represents the calibrated candidate visual embedding sequence The visual embedding corresponding to the j-th candidate target box in the image. Represents a task attribute vector. Represents element-wise product. This represents the confidence level of the j-th candidate bounding box. Indicates the index of the attribute text embedding. Indicates the total number of attribute phrases. This represents the attention weight embedded in the i-th attribute. Represents the calibrated attribute text embedding sequence The text embedding corresponding to the i-th attribute phrase in the text. This represents the transpose of the trainable attribute aggregation query vector. This represents the normalized summation index of the attribute embedding. Represents the calibrated attribute text embedding sequence The text embedding corresponding to the k-th attribute phrase; Based on task-related scores For candidate region set Perform threshold filtering sequentially and retain the set. , The preset threshold; set The task-related scores were sorted in descending order, and the Top-M candidate boxes were selected. Non-maximum suppression (NMS) is performed on the Top-M candidate boxes according to the preset IoU threshold, and the object detection results are output.

[0029] Understandably, the preset IoU threshold can be set according to actual usage needs.

[0030] This invention enables the training of a scoring function that simultaneously captures the direct similarity and interactive differences between visual features and task semantics, avoiding information loss that may result from relying solely on a single matching metric and improving the accuracy of the screening results.

[0031] In one embodiment of the present invention, the loss function for object detection includes a contrastive learning loss, a scoring function supervision loss, and an object detection loss: in, Represents the joint loss function for multiple objectives. Indicates contrastive learning loss The weighting coefficients, Represents the supervised loss of the scoring function. The weighting coefficients, Indicates target detection loss The weighting coefficients, Indicates the contrast learning loss. This represents the supervised loss of the scoring function. This represents the target detection loss, calculated using the mean squared error loss to determine the difference between the predicted and ground truth bounding boxes. This represents all image-text pairs in the multimodal data distribution D. Seeking expectations, Represents the cross-entropy loss function. Representing an image In the text candidate set The one-hot distribution of the true matching samples. Representing an image With text candidate set Image-to-text normalized similarity scores for each text. Representing text In the image candidate set The one-hot distribution of the true matching samples means that the probability of a positive sample pair is 1, and the probability of a negative sample pair is 0. Representing text With image candidate set Text-to-image normalized similarity scores for each image. Indicates the index of the candidate target box. This indicates the total number of candidate bounding boxes. This represents the task target supervision tag for the j-th candidate target box.

[0032] like Figure 3 As shown, in one embodiment of the present invention, the parameters of the embedded recalibration aligner and the parameters of the trainable scoring function are obtained through meta-learning. Specifically, a task set containing multiple tasks is constructed, and a meta-learning algorithm based on Reptile is used to iteratively update the parameters of the embedded recalibration aligner and the parameters of the trainable scoring function across tasks. The outer loop step size is adaptively adjusted according to the task representation vector obtained by aggregating the calibrated attribute text embedding sequences of each task. After the iteration converges, the obtained meta-initialization parameters are used as the starting point for fast parameter adaptation of the embedded recalibration aligner and the trainable scoring function on the new task.

[0033] Specifically, the meta-learning process is as follows: Step 1: Construct a task set containing multiple tasks For each task Sampling support set With query set ; Step 2: Denote the parameters of the embedded recalibration aligner and the parameters of the trainable scoring function as the meta-initialization parameters. ; Step 3: For any task , set the initialization parameters The copy serves as the starting point for the parameters of this task, within the support set. Execute preset Step-by-step gradient iteration, with each step updating in the following form: ; in, Indicates the first The parameters after one iteration Indicates task The parameters after the gradient iteration at step t, For the inner loop learning rate, Indicates the parameter Operator for finding gradient Indicates task loss; Step 4: Aggregate the calibrated attribute text embedding sequence T′ of the current task τ to obtain the task representation vector. ,Will The input scalar mapping function is used to calculate the adaptive outer loop step size and update the meta-initialization parameters: ; ; in Based on the outer loop step size, It is a scalar mapping function; Step 5: Repeat steps 1 through 4 until convergence, and obtain the meta-initialization parameters. This serves as the starting point for the embedded recalibration aligner and the trainable scoring function to perform fast parameter adaptation on the new task.

[0034] Understandably, this invention constructs a model structure through a cross-modal feature extraction mechanism, an embedded recalibration aligner, and a trainable scoring function. It then optimizes the meta-initialization parameters using a meta-learning-based cross-task iterative training mechanism, resulting in a task-driven detection model capable of supporting both object detection and task understanding in complex scenarios. When faced with a completely new task, this model requires only a few gradient update steps to complete parameter adaptation. Therefore, it is not limited to a dedicated detector optimized for a specific task, but rather is a general task-driven detection model capable of simultaneously achieving "task understanding and object localization."

[0035] This invention uses the difference between the task-adapted parameters and the meta-initialization parameters to perform weighted updates on the meta-initialization parameters. Through iterative updates across multiple tasks, the parameters of the embedded recalibration aligner and trainable scoring function are obtained to achieve appropriate meta-initialization. This enables the two to achieve parameter adaptation with fewer gradient update steps when facing new tasks, thereby improving the initialization quality and parameter convergence efficiency of the model on new tasks.

[0036] Understandably, the standard Reptile algorithm typically uses a fixed or preset outer loop learning rate. Initialization parameters Updates are performed without distinguishing between different tasks based on semantic complexity or attribute combinations; the update magnitude of the meta-initialization parameters remains consistent in each outer loop. To enhance the adaptability of meta-learning in situations with uneven task distribution, this invention introduces a task attribute-driven adaptive step-size mechanism based on the Reptile outer loop update framework: utilizing the calibrated attribute text embedding sequence. After semantic grouping, the task representation vector is obtained through attention-weighted aggregation. Then through learnable scalar mapping functions Will The adaptive outer loop step size is mapped to an adaptive outer loop step size adjustment factor, thus obtaining the adaptive outer loop step size. in This mechanism allows the outer loop update step size to be dynamically adjusted based on the semantic complexity of the task attributes, thus enabling the meta-initialization parameters to maintain greater sensitivity to tasks with high difficulty in aligning "attribute phrases" with "candidate target local regions" during cross-task iterative updates. Under conditions of uneven task distribution, this mechanism helps the model form meta-initializations with better transfer quality, allowing the embedding recalibration aligner and trainable scoring function to complete parameter adaptation with fewer gradient update steps when facing new tasks.

[0037] The output of the embedded recalibration aligner of this invention It not only serves as input to a trainable scoring function to compute the task attribute vector 'a' and drive the scoring of candidate boxes, but also as the task representation vector in improved meta-learning algorithms. The computational source of this information allows the calculation of the adaptive outer loop step size to better reflect the true adaptation difficulty of the current task; simultaneously, the parameters embedded in the recalibration aligner are themselves used as meta-initialization parameters. The components of the algorithm are continuously updated by the improved Reptile algorithm during the cross-task meta-training phase, which optimizes its initial cross-modal calibration capability on new tasks at the meta level. In other words, meta-learning not only optimizes the initial parameters of the scoring function, but also optimizes the initial parameters of the embedded recalibration aligner, so that both can quickly adjust to a state that matches the current task with a small adaptation cost when facing new tasks.

[0038] In one embodiment of this invention, the COCO Tasks dataset, a task-driven object detection benchmark dataset built based on the MS-COCO image library, is used to compare and evaluate the technical solution of this invention. The evaluation metric is AP@0.5 (meaning that in the object detection task, when the IoU value between the rectangle output by the model and the manually labeled rectangle is greater than 0.5, the rectangle output by the model is considered correct). This dataset includes 14 everyday life operation tasks, as shown in Table 1. Each task has independent training and test set annotations (stored in JSON format). The object detection candidate boxes are obtained from the detection results of the pre-trained Faster R-CNN network. Comparison methods include GGNN (Gated Graph Neural Network), TOIST (Task-Oriented Instance Segmentation Transformer Model), and TOIST_DISTILL (Instance Segmentation Distillation Model Based on Transformer Architecture). The multimodal task-driven object detection method based on meta-learning proposed in this invention is also compared with the above methods, and the experimental results are shown in Table 2.

[0039] Table 1 Task Content Table 2 Experimental Results As shown in Tables 1 and 2, the method of this invention outperforms the comparative methods on most tasks, and its overall performance surpasses GGNN, TOIST, and TOIST_DISTILL. Especially in scenarios with complex semantic relationships requiring deeper semantic matching between attribute phrases and candidate regions, such as Tasks 4 and 6, the method of this invention achieves improvements of 12.7 and 7.5 percentage points respectively compared to TOIST_DISTILL, demonstrating stronger task understanding and cross-modal alignment capabilities. In scenarios with a more direct relationship between the task and the target (such as Task 7), the performance of the method of this invention is basically close to that of TOIST_DISTILL, indicating that the proposed meta-learning mechanism, while maintaining the performance of existing methods, can further improve the model's cross-task generalization ability in complex task scenarios.

[0040] This invention also provides a multimodal large model task-driven target detection system based on meta-learning, characterized in that the system is used to implement the method described above, including: The data acquisition module is used to acquire multimodal input data containing images and text, perform feature extraction and cross-modal alignment to obtain aligned multimodal features, and input the aligned multimodal features into the multimodal cross-attention fusion encoder to obtain a unified semantic representation; The data processing module is used to scan the images in the multimodal input data using a general candidate generation network, generate a set of candidate regions, and input the task description into a large language model to generate a set of visual attribute phrases. The feature correction module is used to extract region features from each candidate target box in the candidate region set to obtain a candidate visual embedding sequence; to vectorize the visual attribute phrase set to obtain an attribute text embedding sequence; and to input the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner, and to perform joint calibration through bidirectional cross attention from vision to text and text to vision to obtain the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence. The target detection module takes the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence as input, calculates the task-related score of each candidate target box through a trainable scoring function, and filters the candidate region set based on the task-related score to obtain the target detection result.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal large-model task-driven target detection method based on meta-learning, characterized in that, include: S1. Obtain multimodal input data containing images and text, perform feature extraction and cross-modal alignment to obtain aligned multimodal features, and input the aligned multimodal features into a multimodal cross-attention fusion encoder to obtain a unified semantic representation; S2. A general candidate generation network is used to scan the images in the multimodal input data to generate a set of candidate regions. The task description is then input into a large language model to generate a set of visual attribute phrases. S3. Extract region features from each candidate bounding box in the candidate region set to obtain a candidate visual embedding sequence; vectorize the visual attribute phrase set to obtain an attribute text embedding sequence; input the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner, and perform joint calibration through bidirectional cross-attention from vision to text and from text to vision to obtain the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence. S4. Using the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence as input, the task-related score of each candidate target box is calculated through a trainable scoring function, and the candidate region set is filtered based on the task-related score to obtain the target detection result.

2. The multimodal large model task-driven target detection method based on meta-learning as described in claim 1, characterized in that: The multimodal input data includes image data and text data. Step S1 specifically includes: The image data is divided into fixed-size image blocks and converted into an image sequence. The text data is then masked by covering the words according to a preset mask ratio to form a masked text sequence. The image sequence and the masked text sequence are respectively input into the visual encoder and the text encoder for feature extraction, resulting in the visual token sequence embedding sequence. and text token sequence embedded sequence ,in Special classification morphemes Visual representation, For the first Visual embeddings corresponding to image patches For embedded dimensions, The total number of image patches, Special classification morphemes Textual representation, Indicates the total number of lexical units; Based on the visual token sequence embedding sequence and the text token sequence embedding sequence, each of the following... Characterization, selecting the most similar mode from the opposing modes based on cosine similarity. 10 samples constitute the image candidate set With text candidate set ; A contrastive learning strategy is used for the image candidate set. With text candidate set Cross-modal alignment is performed to obtain aligned multimodal features, which are then input into a multimodal cross-attention fusion encoder to obtain a unified semantic representation.

3. The multimodal large model task-driven target detection method based on meta-learning as described in claim 2, characterized in that: The cross-modal alignment specifically includes: Based on the linear similarity function, the image candidate sets are respectively... With text candidate set Calculate the Softmax normalized similarity score: ； ； ； in, Representing an image With text candidate set The Middle text Image-to-text normalized similarity scores between the two texts. This represents the natural exponential function. The linear similarity function representing image-text pairs, This indicates the currently queried image sample. Represents the text candidate set The m-th text in the middle, This indicates a learnable temperature parameter. Indicates the candidate sample index. Representing text With image candidate set The Middle One image Text-to-image normalized similarity scores between the two. This represents the current query text sample. Represents the candidate set of images The m-th image sample, Represents the visual linear transformation function. Represents a text linear transformation function; Aligning features from different modalities within the same sample to a shared low-dimensional space yields aligned multimodal features.

4. The multimodal large model task-driven target detection method based on meta-learning as described in claim 1, characterized in that: The step of inputting the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner specifically includes: Region feature extraction is performed on each candidate bounding box in the candidate region set to obtain a candidate visual embedding sequence. ; The set of visual attribute phrases is vectorized by a text encoder to obtain the attribute text embedding sequence. ; embedding candidate visual sequences With attribute text embedding sequence A common input embedding recalibrator is used for bidirectional joint calibration to obtain the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence.

5. The multimodal large model task-driven target detection method based on meta-learning as described in claim 4, characterized in that: The embedded recalibration aligner employs a lightweight Transformer cross-attention architecture, comprising a cross-attention submodule for the visual-to-text modality and a cross-attention submodule for the text-to-visual modality. Each cross-attention submodule is computed using a scaled dot product attention approach. For the cross-attention submodule of the visual-to-text modality, with candidate visual embedding sequences For query source, embed sequence with attribute text Using the key value source, perform the following update to obtain the calibrated candidate visual embedding sequence. : ； ； ； ； ； in, Represents the query matrix. Represents a value matrix, A trainable projection matrix representing a visual query. Represents the key matrix, This indicates an attribute text embedding sequence. The trainable projection matrix representing the text keys. Indicates a sequence of embedded attribute text. through The transformed value matrix, A trainable projection matrix representing text values. This represents the output of visual-to-text cross-attention. This represents the scaling dot product attention calculation function. For layer normalization operators, It is a feedforward network. This represents the calibrated candidate visual embedding sequence; For the text-to-visual modality cross-attention submodule, the attribute text embedding sequence is used. For query source, candidate visual embedding sequence Using the key-value source, the cross-attention submodule of the visual-to-text modality is updated to obtain the calibrated attribute text embedding sequence. : ； ； ； ； ； in, Represents a text query matrix. The trainable projection matrix representing a text query. Represents the key matrix, The trainable projection matrix representing the visual key. Represents a sequence of visual embeddings via projection matrix The transformed value matrix, The trainable projection matrix representing the visual value. The output represents the cross-attention from text to visual modality. This represents the calibrated attribute text embedding sequence.

6. The multimodal large model task-driven target detection method based on meta-learning as described in claim 3, characterized in that: Step S4 specifically includes: Calculate the calibrated attribute text embedding sequence The semantic similarity between attribute phrases is determined, and the calibrated attribute text is embedded into the sequence based on the semantic similarity. It is divided into several semantic subsets, and each subset corresponds to a task semantic dimension; Within each semantic subset, attention-weighted aggregation is performed to obtain the corresponding semantic sub-vectors; The semantic sub-vectors are fused to obtain the task attribute vector that represents the overall semantics of the current task. ; Using task attribute vectors With calibrated candidate visual embedding sequences The visual embeddings of each candidate bounding box are used as input, and a task-related score for each candidate bounding box is calculated using a trainable scoring function. The task-related score Indicates the probability that the candidate bounding box fits the current task: ； ； ； in, This represents the task-related score of the j-th candidate bounding box. This represents the Sigmoid function. This represents the transpose of the trainable score weight vector. Represents the calibrated candidate visual embedding sequence The visual embedding corresponding to the j-th candidate target box in the image. Represents a task attribute vector. Represents element-wise product. This represents the confidence level of the j-th candidate bounding box. Indicates the index of the attribute text embedding. Indicates the total number of attribute phrases. This represents the attention weight embedded in the i-th attribute. Represents the calibrated attribute text embedding sequence The text embedding corresponding to the i-th attribute phrase in the text. This represents the transpose of the trainable attribute aggregation query vector. This represents the normalized summation index of the attribute embedding. Represents the calibrated attribute text embedding sequence The text embedding corresponding to the k-th attribute phrase; Based on task-related scores For candidate region set Perform threshold filtering sequentially and retain the set. , The preset threshold; set The task-related scores were sorted in descending order, and the Top-M candidate boxes were selected. Non-maximum suppression is applied to the Top-M candidate boxes according to the preset IoU threshold, and the target detection results are output.

7. The multimodal large model task-driven target detection method based on meta-learning as described in claim 1, characterized in that: The loss function for object detection includes contrastive learning loss, scoring function supervision loss, and object detection loss: ； ； ； ； in, Represents the joint loss function for multiple objectives. Indicates contrastive learning loss The weighting coefficients, Represents the supervised loss of the scoring function. The weighting coefficients, Indicates target detection loss The weighting coefficients, Indicates the contrast learning loss. This represents the supervised loss of the scoring function. Indicates the target detection loss. This represents all image-text pairs in the multimodal data distribution D. Seeking expectations, Represents the cross-entropy loss function. Representing an image In the text candidate set The one-hot distribution of the true matching samples. Representing an image With text candidate set Image-to-text normalized similarity scores for each text. Representing text In the image candidate set The one-hot distribution of the true matching samples. Representing text With image candidate set Text-to-image normalized similarity scores for each image. Indicates the index of the candidate target box. This indicates the total number of candidate bounding boxes. The task target supervision marker represents the j-th candidate target bounding box. This represents the task-related score of the j-th candidate bounding box.

8. The multimodal large model task-driven target detection method based on meta-learning as described in claim 1, characterized in that: The parameters of the embedded recalibrated aligner and the trainable scoring function are obtained through meta-learning. Specifically, a task set containing multiple tasks is constructed, and a Reptile-based meta-learning algorithm is used to iteratively update the parameters of the embedded recalibrated aligner and the trainable scoring function across tasks. The outer loop step size is adaptively adjusted based on the task representation vector obtained by aggregating the calibrated attribute text embedding sequences of each task. After the iteration converges, the obtained meta-initialization parameters are used as the starting point for fast parameter adaptation of the embedded recalibrated aligner and the trainable scoring function on the new task.

9. The multimodal large model task-driven target detection method based on meta-learning as described in claim 8, characterized in that: The meta-learning process is specifically as follows: Step 1: Construct a task set containing multiple tasks For each task Sampling support set With query set ; Step 2: Denote the parameters of the embedded recalibration aligner and the parameters of the trainable scoring function as the meta-initialization parameters. ; Step 3: For any task , set the initialization parameters The copy serves as the starting point for the parameters of this task, within the support set. Execute preset Step-by-step gradient iteration, with each step updating in the following form: ； in, Indicates the first The parameters after one iteration Indicates task The parameters after the gradient iteration at step t, For the inner loop learning rate, Indicates the parameter Operator for finding gradient Indicates task loss; Step 4: Aggregate the calibrated attribute text embedding sequence T′ of the current task τ to obtain the task representation vector. ,Will The input scalar mapping function is used to calculate the adaptive outer loop step size and update the meta-initialization parameters: ； ； in Based on the outer loop step size, It is a scalar mapping function; Step 5: Repeat steps 1 through 4 until convergence, and obtain the meta-initialization parameters. This serves as the starting point for the embedded recalibration aligner and the trainable scoring function to perform fast parameter adaptation on the new task.

10. A multimodal large-model task-driven target detection system based on meta-learning, characterized in that, The system is used to implement the method as described in any one of claims 1-9, comprising: The data acquisition module is used to acquire multimodal input data containing images and text, perform feature extraction and cross-modal alignment to obtain aligned multimodal features, and input the aligned multimodal features into the multimodal cross-attention fusion encoder to obtain a unified semantic representation. The data processing module is used to scan the images in the multimodal input data using a general candidate generation network, generate a set of candidate regions, and input the task description into a large language model to generate a set of visual attribute phrases. The feature correction module is used to extract region features from each candidate target box in the candidate region set to obtain a candidate visual embedding sequence; to vectorize the visual attribute phrase set to obtain an attribute text embedding sequence; and to input the candidate visual embedding sequence and the attribute text embedding sequence into the embedding recalibration aligner, and to perform joint calibration through bidirectional cross attention from vision to text and text to vision to obtain the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence. The target detection module takes the calibrated candidate visual embedding sequence and the calibrated attribute text embedding sequence as input, calculates the task-related score of each candidate target box through a trainable scoring function, and filters the candidate region set based on the task-related score to obtain the target detection result.

Images