Few-shot object detection method, device and equipment based on multi-modal model

Abstract

The embodiment of the application provides a kind of based on multimodal model's few sample target detection method, device and storage medium, can be used in few sample learning technical field.The method is by obtaining the target to be identified input by user and analysis draws corresponding target identification feature, while determining support identification target and support identification target corresponding support feature, then based on target identification feature and support feature determines visual feature, and obtains text feature, finally according to visual feature and text feature after multimodal fusion produces fusion feature, determines the final target detection result, by introducing multimodal model replaces traditional detection model, the method enhances the comprehensive understanding ability of model to input data, and also improve the accuracy and robustness of model.

Description

Technical Field

[0001] This application relates to the field of few-shot learning technology, and in particular to a few-shot target detection method, apparatus, device and storage medium based on a multimodal model. Background Technology

[0002] With the rapid development of computer vision technology, object detection, as an important branch, has made significant progress. However, traditional object detection methods typically rely on large amounts of labeled data to train deep learning models, which is not only time-consuming and labor-intensive, but also, in some specialized fields, high-quality labeled data is often very scarce. Therefore, how to achieve efficient and accurate object detection with a limited number of labeled samples has become an urgent problem to be solved.

[0003] In existing technologies, while visual-based object detection models have been widely used in numerous publications, they are generally considered inferior to multimodal object detection models in terms of generalization ability. Currently, most few-shot learning methods are still built upon these unimodal base models, resulting in performance limitations. In contrast, while multimodal models possess stronger expressive power and generalization potential, they are prone to overfitting when fine-tuning all parameters using a small number of samples, affecting the overall performance of the model. Even when fine-tuning by fixing some parameters to reduce overfitting, although recall can be improved to some extent, the false detection rate will also increase. Furthermore, multimodal models based on support sets perform well in standard object detection tasks, but their performance improvement is relatively limited for detecting complex or rare samples such as pits. These issues indicate that to further optimize the performance of few-shot object detection, more effective model architectures and training strategies need to be explored.

[0004] Therefore, this invention provides a few-sample target detection scheme based on a multimodal model. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for few-sample target detection based on a multimodal model, in order to improve the existing technology, which usually relies on a large amount of labeled data to train deep learning models, which is time-consuming and labor-intensive, and in some professional fields, high-quality labeled data is often very scarce.

[0006] In a first aspect, embodiments of this application provide a few-sample target detection method based on a multimodal model, the method comprising:

[0007] The system obtains the target to be identified by the user input, analyzes and processes the target to be identified, and obtains the target identification features corresponding to the target to be identified.

[0008] Based on the target to be identified, a supporting target is determined, and based on the supporting target, the supporting features corresponding to the supporting target are determined;

[0009] Based on the target recognition features and the supporting features, a fusion feature is determined, and based on the fusion feature, a target detection result is determined.

[0010] Optionally, the target to be identified includes: a first target and a second target; the target identification features include: a first feature and a second feature; the step of obtaining the target to be identified input by the user, analyzing and processing the target to be identified to obtain the target identification features corresponding to the target to be identified includes:

[0011] The first identification target and the second identification target are analyzed and processed to obtain the first feature corresponding to the first identification target and the second feature corresponding to the second identification target.

[0012] Optionally, before determining supporting targets based on the target to be identified, and determining supporting features corresponding to the supporting targets based on the supporting targets, the method further includes:

[0013] The target recognition features corresponding to multiple targets to be identified are obtained, and the target recognition features are filtered by the first encoder according to the first preset conditions to obtain the supporting features corresponding to each target feature.

[0014] The supporting features corresponding to each feature to be identified are comprehensively processed to obtain a set of supporting identification targets.

[0015] Optionally, determining supporting targets based on the target to be identified, and determining supporting features corresponding to the supporting targets based on the supporting targets, includes:

[0016] Based on the target to be identified, determine the supporting targets that are associated with the target to be identified from the set of supporting targets;

[0017] Based on the supported identification target, the first encoder performs analysis and processing to determine the supporting features corresponding to the supported identification target.

[0018] Optionally, determining the fusion feature based on the target recognition feature and the supporting feature, and determining the target detection result based on the fusion feature, includes:

[0019] The first feature and the supporting feature are processed to obtain the third feature;

[0020] Based on the second feature and the third feature, the fusion feature is determined;

[0021] The fused features are regressed and classified using a preset module to obtain the target detection result.

[0022] Optionally, the step of processing the first feature and the supporting feature to obtain the third feature includes:

[0023] The supporting features are normalized and then input into the multi-head attention module, while the first feature is also input into the multi-head attention module.

[0024] According to the second preset condition, the first feature and the supporting feature are processed by the multi-head attention module to obtain the third feature;

[0025] The third feature is input into the single-head attention module and enhanced by the single-head attention module to obtain the processed third feature.

[0026] Optionally, determining the fusion feature based on the second feature and the third feature includes:

[0027] The second feature and the third feature are dimensionality reduced, and the dimensionality-reduced second feature and the third feature are then dimensionality-increased to obtain a fused feature, wherein the dimension of the fused feature is the same as the dimension of the second feature and the third feature.

[0028] Secondly, embodiments of this application provide a few-sample target detection device based on a multimodal model, the device comprising:

[0029] The acquisition module is used to acquire the target to be identified input by the user, analyze and process the target to be identified, and obtain the target recognition features corresponding to the target to be identified;

[0030] The determination module is used to determine a supporting target based on the target to be identified, and to determine the supporting features corresponding to the supporting target based on the supporting target; to determine the fusion features based on the target identification features and the supporting features, and to determine the target detection result based on the fusion features.

[0031] Optionally, the device further includes: a processing module;

[0032] The processing module is used to analyze and process the first identification target and the second identification target to obtain a first feature corresponding to the first identification target and a second feature corresponding to the second identification target.

[0033] Optionally, the acquisition module is further configured to acquire target recognition features corresponding to multiple targets to be identified, and to filter the target recognition features through a first encoder according to a first preset condition to obtain supporting features corresponding to each target to be identified.

[0034] The processing module is further configured to perform comprehensive processing on the supporting features corresponding to each feature to be identified, to obtain a set of supporting identification targets.

[0035] Optionally, the determining module is further configured to determine, based on the target to be identified, a supporting identification target that is associated with the target to be identified in the set of supporting identification targets; and, based on the supporting identification target, perform analysis and processing through the first encoder to determine the supporting features corresponding to the supporting identification target.

[0036] Optionally, the processing module is further configured to perform data processing on the first feature and the supporting feature to obtain a third feature;

[0037] The determining module is further configured to determine a fusion feature based on the second feature and the third feature;

[0038] The processing module is also used to perform regression and classification processing on the fused features through a preset module to obtain target detection results.

[0039] Optionally, the processing module is further configured to normalize the supporting features and input the processed supporting features into the multi-head attention module, while simultaneously inputting the first feature into the multi-head attention module; according to a second preset condition, the first feature and the supporting features are processed by the multi-head attention module to obtain a third feature; the third feature is input into the single-head attention module and enhanced by the single-head attention module to obtain the processed third feature.

[0040] Optionally, the processing module is further configured to perform dimensionality reduction processing on the second feature and the third feature, and perform dimensionality increase processing on the dimensionality-reduced second feature and the third feature to obtain a fused feature, wherein the dimension of the fused feature is the same as the dimension of the second feature and the third feature.

[0041] Thirdly, embodiments of this application provide a few-sample target detection device based on a multimodal model, including: a memory and a processor;

[0042] The memory stores computer-executed instructions;

[0043] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the multimodal model-based few-sample target detection method described above.

[0044] Fourthly, this application provides a computer storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the multimodal model-based few-sample target detection method as described in the first aspect and various possible implementations of the first aspect.

[0045] The few-shot target detection method based on a multimodal model provided in this application obtains the target to be identified by the user input and analyzes it to obtain the corresponding target recognition features. At the same time, it determines the supporting target and the supporting features corresponding to the supporting target. Then, it determines the visual features based on the target recognition features and the supporting features, and obtains the text features. Finally, it generates fused features by multimodal fusion based on the visual features and the text features, and determines the final target detection result. This method improves the model capability by introducing a multimodal model to replace the traditional detection model. At the same time, it introduces a low-rank adaptation module to solve the overfitting problem that occurs in the placement model, and uses an attention module to solve the problems of hard sample learning and background false detection. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 A flowchart illustrating a few-shot target detection method based on a multimodal model provided in this application. Figure 1 ;

[0048] Figure 2 A flowchart illustrating a few-shot target detection method based on a multimodal model provided in this application. Figure 2 ;

[0049] Figure 3 A schematic diagram of a few-sample target detection device based on a multimodal model provided in this application;

[0050] Figure 4 This is a schematic diagram of a few-sample target detection device based on a multimodal model, which is provided for this application.

[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0054] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] First, let's explain the terms used in this application:

[0056] Support set: This is a concept in machine learning, especially in the fields of meta-learning and few-shot learning. It typically refers to a limited set of examples used to train a model, containing a small number of examples from some classes, but only a few samples from each class. The main purpose of the support set is to allow the model to quickly adapt and make accurate predictions when exposed to small amounts of new data.

[0057] Multimodal object detection model: This is a machine learning or deep learning model that can process and fuse input data from multiple different sources or types, i.e., multimodal data, to improve the accuracy and robustness of object detection.

[0058] Overfitting refers to a machine learning model performing too well on the training data, to the point that it learns not only useful patterns in the data but also noise and outliers. In this case, the model's performance on the training set, such as accuracy, is very high, but its performance on unseen test data or new data in real-world applications drops significantly. In other words, an overfitted model "memorizes" too much of the training data and fails to generalize well to new, unseen data.

[0059] Overfitting typically occurs when a model is too complex, for example, with too many parameters, or when the amount of training data is relatively small. Overfitting is a common problem in machine learning because it causes the model to perform poorly in real-world applications and fail to accurately predict new data.

[0060] True Positive Rate (TPR) is an important metric for evaluating classification models, especially when the positive class is of greater interest than the negative class. High recall means the model can identify the majority of actual positive samples, even if there are some false positives—that is, misidentifying some negative samples as positive. In certain applications, such as medical diagnosis or fraud detection, high recall is crucial because missing a single true positive can have serious consequences.

[0061] With the rapid development of computer vision technology, object detection, as an important branch, has made significant progress. However, traditional object detection methods typically rely on large amounts of labeled data to train deep learning models, which is not only time-consuming and labor-intensive, but also, in some specialized fields, high-quality labeled data is often very scarce. Therefore, how to achieve efficient and accurate object detection with a limited number of labeled samples has become an urgent problem to be solved.

[0062] In existing technologies, while visual-based object detection models have been widely used in numerous publications, they are generally considered to be inferior to multimodal object detection models in terms of generalization ability. Currently, most few-shot learning methods are still built upon these unimodal base models, resulting in limitations in their performance.

[0063] In contrast, while multimodal models possess stronger expressive power and generalization potential, they are prone to overfitting when fine-tuning all parameters using a small number of samples, impacting overall model performance. Even when fine-tuning by fixing some parameters to reduce overfitting, while recall can be improved to some extent, the false positive rate also increases. Furthermore, multimodal models based on support sets perform well in standard object detection tasks, but their performance improvement is relatively limited for detecting complex or rare samples such as pits. These issues indicate that to further optimize the performance of object detection with few samples, more effective model architectures and training strategies need to be explored.

[0064] To address the aforementioned issues, this application proposes a few-sample target detection method based on a multimodal model.

[0065] This method acquires the target input by the user and analyzes it to obtain the corresponding target recognition features. At the same time, it determines the supporting target and its corresponding supporting features. Then, based on the target recognition features and supporting features, it determines the visual features and obtains the text features. Finally, it generates fused features by multimodal fusion based on the visual features and text features, and determines the final target detection result. This method improves the model capability by introducing a multimodal model to replace the traditional detection model. It also introduces a low-rank adaptation module to solve the overfitting problem that occurs in the placement model, and uses an attention module to solve the problems of hard sample learning and background false detection.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0067] Figure 1 A flowchart illustrating a few-shot target detection method based on a multimodal model provided in this application. Figure 1 ,like Figure 1 As shown, the few-shot target detection method based on a multimodal model provided in this embodiment includes:

[0068] S101: Obtain the target to be identified input by the user, analyze and process the target to be identified, and obtain the target identification features corresponding to the target to be identified.

[0069] The target to be identified can be, for example, a set of images to be identified and a set of text descriptions. The target recognition features refer to the features of the set of images to be identified extracted by the visual encoder and the text features of the set of text descriptions extracted by the text encoder.

[0070] The purpose of this step, which involves acquiring the set of images and text descriptions input by the user and further analyzing them to derive their corresponding features, is to provide structured and easily understood input information for subsequent recognition, classification, or prediction tasks, thereby improving the accuracy and efficiency of recognition.

[0071] In one possible implementation, a visual encoder can be used. Each image in the image set to be recognized is sequentially input into a pre-trained deep learning model, such as a convolutional neural network. After processing through multiple layers of convolution, pooling, and activation functions, a high-dimensional feature vector is generated at one of the intermediate or final layers of the encoder. This feature vector is a feature representation of the image, capturing key visual information such as shape, texture, and color, and can be used for subsequent similarity comparison or classification tasks. Alternatively, a text encoder can be used. The input text description is parsed word by word and converted into numerical embedding vectors. These vectors are then processed through multiple layers of neural networks, such as Transformer, LSTM, or GRU, to capture semantic information and contextual relationships within the text. Ultimately, the encoder generates a high-dimensional feature vector that not only contains the meaning of individual words but also reflects the dependencies between words and the overall context, thus accurately representing the meaning of the entire text and providing a powerful feature representation for subsequent tasks.

[0072] S102: Based on the target to be identified, determine the supporting target, and based on the supporting target, determine the supporting feature corresponding to the supporting target.

[0073] The supporting recognition target can be, for example, a supporting image set, and the supporting features refer to the support vector features extracted from the supporting image set by the visual encoder.

[0074] The purpose of determining the support image set based on the user-input set of images to be identified is to provide a reliable category reference for each image, ensuring fast and accurate classification or matching during the inference phase. By selecting a support image set relevant to the set of images to be identified, the model can perform efficient category inference with a small number of samples, making it particularly suitable for few-shot learning and meta-learning tasks. Each image in the support image set has its high-dimensional features pre-extracted by a visual encoder and aggregated into support vectors, which represent typical features of each category. Thus, during inference, the category of the image to be identified can be determined by comparing the similarity between the feature vectors of the image to be identified and the support vectors, thereby improving the accuracy and efficiency of classification.

[0075] In one possible implementation, the support image set is a representative sample selected from a large number of pre-labeled images. Features are extracted from these images by a visual encoder and aggregated into support vectors, representing typical features of each category. The support image set is determined based on the image set to be identified by analyzing the context of the images to be identified, task requirements, or category distribution, and selecting the most relevant support images.

[0076] Understandably, support vector sets can be pre-constructed during the training phase. This is achieved by inputting a large number of labeled support images into the visual encoder, extracting high-dimensional feature vectors for each image, and aggregating feature vectors of the same category, such as by calculating the mean or generating prototypes, thus obtaining the support vectors for each category. These support vectors represent the typical features of each category. During the inference phase, after the input image to be recognized is processed by the visual encoder to obtain query vector features, the category of the image to be recognized can be quickly determined by comparing the similarity between the query vector and the support vectors. The support vector set provides an efficient category representation, enabling the model to perform accurate classification with a small number of samples.

[0077] S103: Determine the fusion features based on the target recognition features and the supporting features, and determine the target detection result based on the fusion features.

[0078] Among them, fusion features refer to combining feature vectors from different modalities, such as visual, text, and audio, in a certain way to form a comprehensive feature representation.

[0079] This step determines the fusion features based on target recognition features and supporting features, and then determines the target detection result based on the fusion features. The aim is to combine the advantages of multiple modalities, capture richer information, and improve the model's performance in multimodal tasks. By using target recognition features, supporting features, and text features, the model can simultaneously utilize visual cues in images and semantic information in text, thereby better understanding complex scenes and contextual relationships. The fusion feature vector not only enhances the model's comprehensive understanding of input data but also improves its accuracy and robustness in tasks such as classification, retrieval, and generation.

[0080] In one possible implementation, the target recognition features and support features can be processed by the attention module to obtain visual features, which are then processed by the multimodal fusion module with the text features extracted by the text encoder to obtain fused features. Finally, the fused features are input into a classifier or regression model for prediction, and the final detection result is generated through post-processing, such as nonmaximum suppression and threshold filtering.

[0081] This application provides a few-shot target detection method based on a multimodal model. It acquires user-inputted target data and analyzes it to derive corresponding target recognition features. Simultaneously, it determines supporting targets and their corresponding supporting features. Based on the target recognition features and supporting features, it determines visual features and acquires text features. Finally, it generates fused features through multimodal fusion of visual and text features to determine the final target detection result. By introducing a multimodal model to replace the traditional detection model, it enhances the model's comprehensive understanding of input data and improves its accuracy and robustness.

[0082] Figure 2 A flowchart illustrating a few-shot target detection method based on a multimodal model provided in this application. Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, a possible implementation of a few-shot target detection method based on a multimodal model is described in detail. The method includes:

[0083] S201: Analyze and process the first identification target and the second identification target to obtain the first feature corresponding to the first identification target and the second feature corresponding to the second identification target.

[0084] The target to be identified includes: a first target and a second target. The target identification features include: a first feature and a second feature. The first target can be, for example, a set of images to be identified, and the second target can be, for example, a set of text descriptions. The first feature can be, for example, a feature of the set of images to be identified extracted by a visual encoder, and the second feature can be, for example, a text feature of the set of text descriptions extracted by a text encoder.

[0085] This step analyzes and processes the image set and text description set to be identified separately, deriving corresponding image and text features. The aim is to fully utilize multimodal information, capturing visual cues in images and semantic information in text, thereby enhancing the model's understanding of the input data and improving task accuracy and robustness. By combining visual and text features, the model can more comprehensively understand complex scenes and contextual relationships, especially in tasks requiring cross-modal reasoning, such as visual question answering, image captioning generation, and cross-modal retrieval. This multimodal feature fusion can significantly improve the model's performance.

[0086] In one possible implementation, the image to be recognized can be input into a pre-trained visual encoder. After processing through multiple layers of convolution, pooling, and activation functions, a high-dimensional feature vector is output at one of the intermediate or final layers of the encoder. This feature vector captures key visual information in the image, such as shape, texture, and color, and represents it as a numerical vector. Alternatively, text descriptions can be input into a pre-trained text encoder, parsed word by word, and converted into numerical embedding vectors. These vectors are then processed through multiple layers of neural networks to capture semantic information and contextual relationships within the text, ultimately generating a high-dimensional feature vector.

[0087] S202: Based on the target to be identified, determine the supporting targets that are associated with the target to be identified from the set of supporting targets.

[0088] Among them, the target to be recognized can be, for example, a set of images.

[0089] In one possible implementation, a visual encoder, such as a convolutional neural network (CNN), can extract corresponding image features from multiple images to be identified, capturing key visual information for each target. Then, based on preset conditions, such as category relevance and feature similarity, the visual encoder further filters these target recognition features, selecting the features most relevant to the images to be identified. After this filtering process, one or more supporting features are matched to the features of each image to be identified. These supporting features typically come from a pre-built set of supporting images, which have already been extracted and aggregated into high-dimensional feature vectors by the visual encoder, representing typical features of each category.

[0090] S203: Based on the supported identification target, the first encoder is used for analysis and processing to determine the supporting features corresponding to the supported identification target.

[0091] Step S203 is similar to step S102 described above, and will not be repeated here.

[0092] S204: Normalize the supporting features and input the processed supporting features into the multi-head attention module, while simultaneously inputting the first feature into the multi-head attention module.

[0093] The first feature can be, for example, a feature of the set of images to be identified extracted by a visual encoder.

[0094] This step normalizes the supporting features and inputs the processed features into the multi-head attention module. Simultaneously, the features corresponding to the image to be recognized are also input into this module. The ultimate goal of generating visual features is to enhance the model's understanding of the target and improve the accuracy and robustness of detection or classification tasks through the fusion of multimodal information. Normalization ensures that features from different sources are within the same numerical range, allowing the attention mechanism to compare and process these features more fairly. The multi-head attention module captures the complex relationships between supporting features and the features of the image to be recognized through multiple parallel attention heads, thereby generating richer visual feature representations.

[0095] In one possible implementation, to process the constructed support features, the support features can first be normalized through a normalization layer to eliminate dimensional differences between features, ensuring that the subsequent attention mechanism can more fairly compare and process these features. The normalized support features are then input into the multi-head attention module as Query (Q), Key (K), and Value (V), respectively. Simultaneously, the image features corresponding to the image to be recognized are also input into the multi-head attention module as Query (Q), Key (K), and Value (V), respectively.

[0096] S205: According to the second preset condition, the first feature and the supporting feature are processed by the multi-head attention module to obtain the third feature.

[0097] The second preset condition can be, for example, a weight condition, which can map the first feature and supporting features to different feature spaces based on different weights. The third feature can be a visual feature.

[0098] In one possible implementation, a multi-head attention module can capture the complex relationships between support vectors using multiple parallel attention heads. The features output after support feature processing are then used as K and V and input into a single-head attention module. Simultaneously, the image features of the image to be recognized are also processed by a multi-head attention module, and the generated features are used as Q and input into the single-head attention module. In the single-head attention module, Q interacts with K and V from the multi-head attention module, and by calculating attention weights, a fused third feature—the fused visual feature—is generated.

[0099] Understandably, to enhance the information interaction between the image features and support vectors of the image to be identified, these two features can be mapped to a new feature space through a self-attention mechanism. In one possible implementation, the mapped features can be input into a single-head attention module, where Q (Query), K (Key), and V (Value) are mapped to another feature space through linear layers. Then, Q and K undergo matrix multiplication, and the similarity score is adjusted by scaling. A mask matrix is ​​then applied to eliminate invalid attention positions. After a softmax operation, the attention weights are calculated, and finally, these weights are multiplied by V to generate a cross-feature that integrates the two feature information—the processed third feature.

[0100] S206: Input the third feature into the single-head attention module, and perform enhancement processing through the single-head attention module to obtain the processed third feature.

[0101] Among them, the single-head attention module can capture the complex relationships between features through multiple parallel attention heads, generating features that fuse multiple perspectives.

[0102] In one possible implementation, the processed third feature can be output to a feedforward network module. In this process, the feature is first expanded to a higher dimension through a linear layer to increase the model's capacity. Then, a non-linear activation function introduces non-linear characteristics, enabling the model to capture more complex feature patterns. Finally, another linear layer maps the feature back to its original dimension. In this way, the FFN module not only enhances the expressive power of the features but also provides richer and more refined feature representations for subsequent tasks.

[0103] S207: Perform dimensionality reduction processing on the second feature and the third feature, and perform dimensionality increase processing on the dimensionality-reduced second feature and the third feature to obtain the fused feature.

[0104] The second feature can be, for example, a text feature, and the third feature can be, for example, a visual feature. The dimensions of the fused feature are the same as those of the second and third features.

[0105] When performing feature mapping in QKV, a bypass can be introduced. This bypass first performs feature dimensionality reduction and then feature dimensionality increase to restore the original dimensionality. The purpose is to preserve the original information of the input features while enhancing the expressive power of the features through linear layers. This design incorporates the idea of ​​residual connections, ensuring that the model's performance does not degrade due to information loss as the network depth increases. Through dimensionality reduction and increase operations, the bypass can introduce additional nonlinear transformations while maintaining the feature dimensionality, further enriching the feature representation and improving the model's robustness and expressive power.

[0106] In one possible implementation, visual and textual features are each encoded by their respective encoders to extract high-dimensional feature vectors. These feature vectors are then typically mapped to Q (Query), K (Key), and V (Value) through linear layers, such as fully connected layers or 1x1 convolutions. The Q, K, and V features are then fed into the linear layers for feature mapping to enhance their expressive power. Simultaneously, a bypass is introduced. This bypass first reduces the feature dimension through a dimensionality reduction operation, such as a 1x1 convolution or linear layer, reducing computational complexity and capturing a more compact feature representation. Then, the bypass performs a dimensionality increase operation to restore the feature dimension to its original size. Finally, the outputs of the linear layers and the bypass are added together to obtain the final Q, K, and V features, i.e., the fused features.

[0107] S208: The fused features are regressed and classified using a preset module to obtain the target detection result.

[0108] Among them, the preset modules can be, for example, regression modules and classification modules.

[0109] This step uses a pre-defined module to perform regression and classification processing on the fused features to obtain the target detection results. The aim is to fully utilize the rich information in the multimodal fusion features to achieve precise target location and accurate category identification. This design can simultaneously capture complementary information from visual and textual features, improving the model's detection accuracy and robustness in complex scenes. By combining regression and classification tasks within a unified framework, the model can more comprehensively understand the target's attributes, thereby generating more reliable target detection results.

[0110] In one possible implementation, the fused features obtained after multimodal fusion can be input into a pre-defined module, which typically contains two main branches: a regression branch and a classification branch. The regression branch is responsible for predicting the bounding box coordinates of the target, such as the center point, width, and height, to achieve accurate target localization. The classification branch is responsible for predicting the category label of the target and its confidence score. These two branches share the same feature extraction layer but independently optimize the loss functions for the regression and classification tasks. Finally, the model is jointly trained to simultaneously optimize the regression and classification tasks, ensuring synergy between the two and generating accurate target detection results.

[0111] Understandably, model training is required before object detection, which consists of two phases: pre-training and few-shot learning. During pre-training, the multimodal fusion module in the network can use the original open-set detector's multimodal fusion module, loading the pre-trained parameters of the open-set detector. Simultaneously, the attention module's parameters are deregulated, while other module parameters are fixed. Furthermore, open-source datasets such as the Object365 or Cocoa2017 datasets can be used. For each category, multiple samples can be selected and processed by the visual encoder to extract visual features. Then, based on the annotation results, visual features of specific regions are extracted to form a support vector set. Simultaneously, random sampling is performed in the background target region to construct a background support vector set. In the few-shot learning phase, the multimodal fusion module can be replaced with a low-rank adaptive multimodal fusion module. Secondly, the attention module parameters and the low-rank adaptive multimodal fusion module are deregulated. Additionally, industry-specific samples, manually annotated target coordinates, and target categories can be used. Following the support vector set construction method of the pre-training phase, the data is replaced with industry-specific data to construct the support vector set. The loss function is constructed in accordance with the loss function used in the pre-training phase.

[0112] The few-shot target detection method based on a multimodal model provided in this application analyzes the image to be identified and the text description set to obtain corresponding image features and text features. Then, it determines the corresponding support image set based on the image to be identified, and simultaneously parses the support image set to obtain support features through a visual encoder. The support features are then normalized, and the processed results, along with the image features, are input into a multi-head attention module for further processing to obtain visual features. Finally, low-rank adaptive multimodal fusion processing is performed based on the visual features and text features to obtain fused features, ultimately yielding the target detection result. This method improves the model's capabilities by introducing a multimodal model to replace the traditional detection model. It also introduces a low-rank adaptation module to address the overfitting problem that occurs in traditional detection models and uses an attention module to solve the problems of hard sample learning and background false detection.

[0113] Figure 3 A schematic diagram of a few-sample target detection device based on a multimodal model provided in this application is shown below. Figure 3 As shown, the multimodal model-based few-sample target detection device 300 provided in this embodiment includes:

[0114] The acquisition module 301 is used to acquire the target to be identified input by the user, analyze and process the target to be identified, and obtain the target recognition features corresponding to the target to be identified.

[0115] The determining module 302 is used to determine a supporting target based on the target to be identified, and to determine the supporting features corresponding to the supporting target based on the supporting target; to determine the fusion features based on the target identification features and the supporting features, and to determine the target detection result based on the fusion features.

[0116] Optionally, the device further includes: a processing module 303;

[0117] The processing module 303 is used to analyze and process the first identification target and the second identification target to obtain a first feature corresponding to the first identification target and a second feature corresponding to the second identification target.

[0118] Optionally, the acquisition module 301 is further configured to acquire target recognition features corresponding to multiple targets to be identified, and to filter the target recognition features through a first encoder according to a first preset condition to obtain supporting features corresponding to each target to be identified.

[0119] The processing module 303 is further configured to perform comprehensive processing on the supporting features corresponding to each feature to be identified, to obtain a set of supporting identification targets.

[0120] Optionally, the determining module 302 is further configured to determine, based on the target to be identified, a supporting identification target that is associated with the target to be identified in the set of supporting identification targets; and, based on the supporting identification target, perform analysis and processing through the first encoder to determine the supporting features corresponding to the supporting identification target.

[0121] Optionally, the processing module 303 is further configured to perform data processing on the first feature and the supporting feature to obtain a third feature;

[0122] The determining module 302 is further configured to determine a fusion feature based on the second feature and the third feature;

[0123] The processing module 303 is further configured to perform regression and classification processing on the fused features through a preset module to obtain target detection results.

[0124] Optionally, the processing module 303 is further configured to normalize the supporting features and input the processed supporting features into the multi-head attention module, while simultaneously inputting the first feature into the multi-head attention module; according to a second preset condition, the first feature and the supporting features are processed by the multi-head attention module to obtain a third feature; the third feature is input into the single-head attention module and enhanced by the single-head attention module to obtain the processed third feature.

[0125] Optionally, the processing module 303 is further configured to perform dimensionality reduction processing on the second feature and the third feature, and perform dimensionality increase processing on the dimensionality-reduced second feature and the third feature to obtain a fused feature, wherein the dimension of the fused feature is the same as the dimension of the second feature and the third feature.

[0126] This embodiment provides a few-sample target detection device based on a multimodal model, which can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0127] Figure 4 This is a schematic diagram of a few-sample target detection device based on a multimodal model, provided in this application. Figure 4 As shown, this application provides a few-sample target detection device 400 based on a multimodal model, including: a receiver 401, a transmitter 402, a processor 403, and a memory 404.

[0128] Receiver 401 is used to receive instructions and data;

[0129] Transmitter 402 is used to send commands and data;

[0130] Memory 404 is used to store instructions executed by the computer;

[0131] The processor 403 is used to execute the computer execution instructions stored in the memory 404 to implement the various steps of the few-shot target detection method based on the multimodal model in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the few-shot target detection method based on the multimodal model.

[0132] Alternatively, the memory 404 can be either standalone or integrated with the processor 403.

[0133] When the memory 404 is set up independently, the electronic device also includes a bus for connecting the memory 404 and the processor 403.

[0134] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the multimodal model-based few-sample target detection method performed by the aforementioned multimodal model-based few-sample target detection device.

[0135] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0136] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0137] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0140] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0142] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for detecting targets with few samples based on a multimodal model, characterized in that, include: The system acquires the target input by the user, analyzes and processes the target to obtain the target recognition features corresponding to the target; Based on the target to be identified, a supporting target is determined, and based on the supporting target, the supporting features corresponding to the supporting target are determined; Based on the target recognition features and the supporting features, a fusion feature is determined, and based on the fusion feature, a target detection result is determined.

2. The method according to claim 1, characterized in that, The target to be identified includes: a first target and a second target; the target identification features include: a first feature and a second feature; the process of obtaining the target to be identified input by the user, analyzing and processing the target to be identified, and obtaining the target identification features corresponding to the target to be identified includes: The first identification target and the second identification target are analyzed and processed to obtain the first feature corresponding to the first identification target and the second feature corresponding to the second identification target.

3. The method according to claim 1, characterized in that, Before determining supporting targets based on the target to be identified, and determining supporting features corresponding to the supporting targets based on the supporting targets, the method further includes: The target recognition features corresponding to multiple targets to be identified are obtained, and the target recognition features are filtered by the first encoder according to the first preset conditions to obtain the supporting features corresponding to each target feature. The supporting features corresponding to each feature to be identified are comprehensively processed to obtain a set of supporting identification targets.

4. The method according to claim 3, characterized in that, The step of determining supporting targets based on the target to be identified, and determining supporting features corresponding to the supporting targets based on the supporting targets, includes: Based on the target to be identified, determine the supporting targets that are associated with the target to be identified from the set of supporting targets; Based on the supported identification target, the first encoder performs analysis and processing to determine the supporting features corresponding to the supported identification target.

5. The method according to claim 1, characterized in that, The step of determining fusion features based on the target recognition features and the supporting features, and determining the target detection result based on the fusion features, includes: The first feature and the supporting feature are processed to obtain the third feature; Based on the second feature and the third feature, the fusion feature is determined; The fused features are regressed and classified using a preset module to obtain the target detection result.

6. The method according to claim 5, characterized in that, The step of processing the first feature and the supporting features to obtain the third feature includes: The supporting features are normalized and then input into the multi-head attention module. At the same time, the first feature is input into the multi-head attention module. According to the second preset condition, the first feature and the supporting feature are processed by the multi-head attention module to obtain the third feature; The third feature is input into the single-head attention module and enhanced by the single-head attention module to obtain the processed third feature.

7. The method according to claim 6, characterized in that, The step of determining the fusion feature based on the second feature and the third feature includes: The second feature and the third feature are dimensionality reduced, and the dimensionality-reduced second feature and the third feature are then dimensionality-increased to obtain a fused feature, wherein the dimension of the fused feature is the same as the dimension of the second feature and the third feature.

8. A few-sample target detection device based on a multimodal model, characterized in that, The device includes: The acquisition module is used to acquire the target to be identified input by the user, analyze and process the target to be identified, and obtain the target recognition features corresponding to the target to be identified; The determination module is used to determine a supporting target based on the target to be identified, and to determine the supporting features corresponding to the supporting target based on the supporting target; to determine the fusion features based on the target identification features and the supporting features, and to determine the target detection result based on the fusion features.

9. A few-sample target detection device based on a multimodal model, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform any one of the few-sample target detection methods based on a multimodal model as described in claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement any one of the multimodal model-based few-sample target detection methods as described in claims 1-7.

Images