A few-shot object detection method based on meta-learning and text prompt

Abstract

The application discloses a few-shot target detection method based on meta-learning and text prompt, and belongs to the technical field of image processing.The application comprises the following steps: pre-processing and labeling acquired image data to obtain a base class data set, and then randomly dividing the base class data set into a support set and a query set according to a predetermined proportion; collecting a certain amount of image data containing detection targets, pre-processing and labeling the image data to obtain a new class data set, and then randomly dividing the new class data set into a support set and a query set according to a predetermined proportion; performing feature extraction on the base class and the new class based on a pre-trained feature extraction module; constructing a text prompt-based candidate box generation module and a prediction head; training the constructed network in the form of a meta task using the base class data set, and then fine-tuning the trained module in the form of a meta task using the new class data set, so as to obtain a target detection model used for detecting new class objects.The application improves the speed of adapting a model to a new task and reduces the calculation amount of the model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology based on machine learning, and in particular to a few-shot target detection method based on meta-learning and text prompts. Background Technology

[0002] With the continuous updating and development of machine learning technologies and theories, computer vision has developed rapidly in recent years. Object detection, a crucial technology in computer vision, has been widely applied across various industries, including security monitoring systems and autonomous driving. Currently, mainstream object detection models are still based on deep learning frameworks, relying on a large amount of manually labeled sample data for supervised training. As the application scope of object detection expands and large models emerge, the demand for data continues to increase. The cost and time spent on manually labeled data are too high, leading to insufficient sample data, which has become an unavoidable problem for current object detection models. Therefore, the problem of few-shot object detection has become a popular and promising direction in the field. Meta-learning, by quickly learning the commonalities between different tasks, obtains prior knowledge for similar tasks and adapts rapidly to new tasks, exhibiting strong generalization ability. Based on the characteristics of meta-learning, network models can learn prior knowledge from existing labeled data and then apply it to new object detection tasks, achieving rapid and accurate detection of target objects even with insufficient labeled samples, demonstrating high research and application value.

[0003] Currently, few-shot object detection algorithms mainly focus on data augmentation, transfer learning, and meta-learning. Data augmentation involves processing training images by flipping, rotating, scaling, cropping, shifting, and adding Gaussian noise to obtain new training images, thereby expanding the dataset. Transfer learning utilizes source domain data to assist in task learning in the target domain.

[0004] However, when encountering new object classes, data augmentation requires reprocessing the images for the new classes and retraining the model. Furthermore, transfer learning relies on the sharing of feature spaces between the source and target domains, thus placing significant demands on the selection and quality of the source domain data. Therefore, neither of the two common methods mentioned above can quickly and conveniently adapt the model to new object classes. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this invention provides a few-shot target detection method based on meta-learning and text prompts, aiming to solve the problem that existing few-shot target detection methods cannot quickly adapt to new tasks.

[0006] The technical solution adopted in this invention is as follows:

[0007] A few-shot object detection method based on meta-learning and text prompting, comprising the following steps:

[0008] Step 1: Preprocess and label the acquired image data, labeling the location and category of the detected objects in the image to obtain the base class dataset; then randomly divide the base class dataset into the support set and query set of the base class according to a predetermined ratio;

[0009] A certain amount of image data containing the detection target is collected, and the data is preprocessed and labeled to label the location and category of the detection target in the image to obtain a new class dataset. The new class dataset is then randomly divided into a support set and a query set for the new class according to a predetermined ratio.

[0010] Step 2: Based on the pre-trained feature extraction module, perform feature extraction on the images in the base class dataset and the new class dataset obtained in Step 1 to obtain the feature map X of each image. f ;

[0011] Step 3: Construct a candidate box generation module based on text prompts. The input of this module is the feature map X of the image. f Output feature map X f The corresponding candidate box;

[0012] Step 4: Construct a prediction head to calculate the candidate boxes generated in Step 3, output the class probability vector, and obtain the offset of each candidate box based on bounding box regression to predict the target detection box.

[0013] Step 5, Training Phase: Construct a base class meta-task using the base class dataset. Use the base class meta-task to train the constructed text-hint-based candidate box generation module and prediction head to optimize the network parameters of the text-hint-based candidate box generation module and prediction head.

[0014] Step 6: Fine-tuning phase. A new meta-task is constructed using the new class dataset. The network parameters of the text-hint-based candidate box generation module and prediction head trained in Step 5 are fine-tuned using the new meta-task, so that it can quickly adapt to the detection of new class objects with few sample data, thereby obtaining a target detection model for detecting new class objects.

[0015] Furthermore, in step 1, both the support set and query set of the base class include N types of detection objects, and each type of detection object has K images, where N and K are preset values.

[0016] Furthermore, in step 1, the preprocessing specifically includes:

[0017] The image size is normalized to a uniform size, preferably 800*600; and for the support set of the base class or new class, the images in it are horizontally flipped with a specified probability.

[0018] Each image in the dataset is labeled with text. The corresponding text prompts include: the location and category of each detected object or target, as well as the interaction information between objects or targets. For example: A man is on the left side of the photo, riding a horse;

[0019] Furthermore, in step 2, the feature extraction module adopts a ResNet network structure, which includes, in sequence: one convolutional layer, two pooling layers, several residual blocks, and one fully connected layer. Preferably, a ResNet50 network can be used.

[0020] Furthermore, step 3 specifically includes:

[0021] Step 3.1, for feature map X f For each pixel, several anchor boxes are generated with that pixel as the center, and q is defined as the number of anchor boxes;

[0022] Step 3.2, extract the text prompt data T = (t1,...,t) corresponding to the images in the dataset. s The input is fed into a Transformer-based text encoder (USE) to extract the corresponding text features T' = (t1',...,t...). s '), where s represents the total number of cue texts in the image, and the number of cue texts is consistent with the number of detection objects or targets in each image, t k ,t' k (k = 1, ..., s) represent the kth text prompt data and its corresponding text features, respectively;

[0023] Step 3.3: For each anchor box, calculate its intersection-union ratio (IU) with the bounding boxes of all real detected objects or targets, and determine the real label for the foreground and background binary classification of each anchor box based on the set IU threshold.

[0024] Step 3.4: Perform dimension mapping on the text feature T' and the feature maps corresponding to the q anchor boxes based on the linear layer to obtain the mapped feature map N of the anchor boxes. fb =(n fb ,1,...,n fb,q ) and text features T fb =(t fb ,1,...,t fb,s ), where n fb,i This represents the feature map of the i-th anchor box after mapping, where i = 1, ..., q, t fb,k Let k represent the k-th text feature after mapping, where k = 1, ..., s, and s is the number of prompt texts in each image;

[0025] For each anchor frame, calculate its feature map n. fb,i With each text feature t fb,k The cosine similarity between the two values ​​is taken as the maximum similarity metric, denoted as S. i ;

[0026] Step 3.5, for the similarity metric S i The Sigmoid function is used to calculate the binary classification probability p of the foreground and background classes for each anchor box. i ;

[0027] Step 3.6, predict the probability p based on the foreground and background categories. i Take the anchor boxes corresponding to the first N' maximum values ​​as candidate boxes, and perform regression calculations on the candidate boxes to calculate the offset of each candidate box relative to the true bounding box of the detected object or target, including: the horizontal and vertical coordinate offset of the candidate box center, and the width and length offset of the candidate box.

[0028] Furthermore, in step 5, the loss function expression for the candidate box generation module based on text prompts is:

[0029]

[0030]

[0031] in, p represents the foreground / background classification loss. l With y l Let be the predicted foreground and background probabilities and the true label of the l-th anchor box, respectively; γ represent the influence factor; and q represent the number of anchor boxes. This represents the anchor box regression loss, where N' represents the number of candidate boxes. and represents the bounding box parameters predicted for the l-th candidate box and the ground truth bounding box parameters, respectively; Smooth_L1() represents the smoothing L1 loss function;

[0032] The joint loss function for the text-based candidate box generation module is: Where α is the preset first balance loss parameter.

[0033] Furthermore, the prediction head constructed in step 4 is specifically as follows:

[0034] The RoIPooling layer is used to partition and pool the feature maps corresponding to the candidate boxes, resulting in a feature region vector X of uniform size. f ';

[0035] The feature region vector X is processed using a fully connected layer and a softmax function.f 'Perform target classification, and classify the feature region vector X' f 'Mapped to the target category prediction probability distribution, and the category with the highest probability in the target category prediction probability distribution is taken as the predicted classification result;

[0036] The feature region vector X is processed through a fully connected layer. f Perform bounding box regression to predict the coordinate offset values ​​of the target box and obtain the final target bounding box. The coordinate offset values ​​of the target box include: the horizontal and vertical coordinate offsets of the target box center, and the width and length offsets of the candidate box.

[0037] Furthermore, in step 5, the loss function expression for the prediction head is:

[0038]

[0039]

[0040] in, Let p represent the classification loss and regression loss generated by the prediction head, respectively. n ,y n These represent the predicted target category and the true target category label, respectively. and These represent the predicted bounding box parameters and the ground truth bounding box parameters of the nth candidate box, respectively, where n is the candidate box number and N' is the number of candidate boxes; Smooth_L1() represents the smoothing L1 loss function.

[0041] The joint loss function for the prediction heads is: α′ is a preset second balance loss parameter.

[0042] Preferably, step 5 specifically includes the following steps:

[0043] Step 5.1: Randomly select support sets and query sets from the base class dataset to form multiple base class meta-tasks. Each base class meta-task includes multiple support sets and one query set.

[0044] Step 5.2: Load the pre-trained weights, freeze the feature extraction module, and use the base class meta-task to train the text-based candidate box generation module and the prediction head.

[0045] Furthermore, step 5.2 includes:

[0046] Inner loop: For each new base class meta-task Ti, perform batch gradient updates based on a certain number of samples to obtain model parameters θ adapted to that base class meta-task. i '; where the number of samples used does not exceed a preset value, that is, mini-batch gradient updates are achieved based on a small number of samples;

[0047] Outer loop: The inner loop is executed multiple times, and θ is obtained from the base class metatask of the single inner loop. i 'To update the initial model parameters θ i This allows it to achieve a better initial state when facing new tasks;

[0048] The specific steps include:

[0049] Step 5.2.1: Sample a base class meta-task m, construct a temporary model identical to the original model (including the pre-feature extraction module (whose network parameters are frozen), the candidate box generation module, and the prediction head), and assign the original model parameters θ to the temporary model, setting the parameters to θ. m ;

[0050] Step 5.2.2: Apply temporary model parameters θ based on the support set in the base class meta-task m. m After multiple optimizations, the updated and optimized model parameters θ are obtained. m ;

[0051] Step 5.2.3: Based on the model parameters θ obtained in Step 5.2.2 m Using the query set from the base class meta-task m, calculate the loss of the base class meta-task m, and calculate this loss for the model parameters θ. m The gradient g; where the loss includes: foreground / background classification loss. Anchor frame regression loss The classification loss generated by the prediction head and regression loss

[0052] Step 5.2.4: If the current loss is less than the loss of the previous base class task, then use the gradient g multiplied by the learning rate. Update the original model parameters θ; otherwise, leave the original model parameters θ unchanged.

[0053] Step 5.2.5: Repeat steps 5.2.1 to 5.2.4 until the loss converges, thereby obtaining the optimal initialization parameters of the original model.

[0054] Preferably, the fine-tuning stage in step 6 specifically refers to: training the model (pre-feature extraction module (with its network parameters frozen), candidate box generation module, and prediction head) obtained after training in step 5 using a new class dataset. The training steps are the same as in step 5. After undergoing a certain number (not exceeding a set value) of outer loop training, the model can quickly adapt to the new task and complete the accurate detection of objects with few samples.

[0055] The technical solution provided by this invention brings at least the following beneficial effects:

[0056] This invention utilizes text prompts to construct a novel candidate box generation module and uses a meta-learning paradigm to train the model. This allows the model to learn from base class data samples and, combined with text prompts, summarize prior knowledge and obtain optimal initialization parameters even with scarce data samples. This enables the model to converge quickly after gradient descent using only a small number of new class data samples, achieving accurate detection of new objects, improving the model's speed of adapting to new tasks, and reducing the computational load of the model. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram of the meta-training stage of a few-shot target detection method based on meta-learning and text prompting, provided in an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of a meta-task in an embodiment of the present invention;

[0060] Figure 3 This is a structural diagram of the candidate box generation module in an embodiment of the present invention; Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0062] Reference Figure 1 and Figure 2 As one possible implementation, this invention provides a few-shot target detection method based on meta-learning and text prompts, comprising the following steps:

[0063] Step 1: Dataset creation.

[0064] 1) Collect a large amount of image data of common real-world objects and a small amount of image data of objects that need to be detected. Label the image data, including the bounding boxes, category names, and corresponding text prompts for the objects in the images. T = (t1,...,t...)s );

[0065] 2) Data partitioning. The processed image data is randomly partitioned according to a predetermined support set to query set ratio of 1:1 to obtain a base class support set, a query set, and a new class support set and query set. Each support set and query set is a small N-way K-shot dataset, where N-way means that the dataset has N classes of objects, and K-shot means that there are K images for each class of objects.

[0066] 3) Data preprocessing. Crops the images in the dataset and resizes them to 800*600; horizontally flips the images in the support set with probability p (preset value), while leaving the images in the query set unchanged.

[0067] Step 2: Based on the ResNet network structure, construct a feature extraction module. In this embodiment of the invention, a ResNet50 network is used. Feature extraction is performed on the image data obtained in Step 1 to obtain the image feature map X. f .

[0068] Step 3: Build a candidate box generation module based on text prompts.

[0069] 3-1) Generate corresponding candidate boxes using the feature maps generated in step 2; the specific structure of the candidate box generation module is as follows: Figure 3 As shown; the extracted feature map X f The data is fed into the anchor generator, which generates 9 anchor boxes for each pixel, denoted as N = (N1, ..., N2). q The anchor frame is divided into 3 different proportions, with the aspect ratio Width:Height∈{1:1,1:2,2:1}. The largest anchor frame with a aspect ratio of 2:1 is 736*384, and the largest anchor frame with a aspect ratio of 1:2 is 352*704.

[0070] 3-2) Next, a general text encoder based on Transformer is used to encode the text data T to obtain the text features T′. The text encoder consists of 6 stacked Transformer layers, each with a self-attention module, followed by a feedforward network.

[0071] 3-3) A linear layer is used to unify the dimension of the text feature T′ and all anchor boxes. In this embodiment, the linear layer is a single fully connected layer, and the mapped result is T′. fb =(t fb ,1,...,t fb,s ) and N fb =(n fb ,1,...,n fb,q ), calculate each n fb,i(i = 1, ..., q) and each t fb,k The cosine similarity of (k=1,...,s) is taken as the maximum value and denoted as S. i (i = 1, ..., q); the specific expression is as follows:

[0072]

[0073] Where s represents the number of prompts in the image.

[0074] 3-4) Based on S i The Sigmoid function is used to calculate the binary classification probability p for both the foreground and background categories. i The specific expression is as follows:

[0075] p i =Sigmoid(S i )

[0076] 3-5) Based on feature map X f Take the anchor boxes with the top N′ predicted probabilities for the foreground and background categories, perform regression calculations, and calculate the offset Δ(Δx) of each anchor box relative to the object's true bounding box. center ,Δy center (Δwidth, Δheight). In this embodiment, two convolutional layers are used to implement regression calculation. The first layer is a 3*3 convolutional kernel, and the second layer is a 1*1 convolutional kernel. The specific expressions are as follows:

[0077] Δ=w2ReLu(w1X fi +b1)+b2

[0078] Where w1, w2, b1, b2 are the weights and biases of the convolutional layer, and X fi This represents the feature map corresponding to the i-th anchor box;

[0079] 3-6) Output candidate boxes proposals based on the anchor box N' and the predicted box offset Δ;

[0080] 3-7) For each generated anchor box, calculate its Intersection over Union (IoU) with all ground truth bounding boxes of objects, set a threshold, and use a step function to obtain a binary sequence, i.e., the ground truth class label (foreground / background) y of the anchor box. i This is used to calculate the foreground / background classification loss during training.

[0081] 3-8) Design the loss function for the candidate box generation module. The specific expression is as follows:

[0082]

[0083]

[0084] In the formula, p represents the foreground / background classification loss. l With y l γ represents the predicted probability of the foreground and background and the true label of the l-th anchor box, respectively. γ represents the influence factor, which is set to 1 in this embodiment. q represents the number of anchor boxes. Indicates regression loss, and represents the predicted bounding box parameters and the ground truth bounding box parameters of the l-th candidate box, respectively; Smooth_L1() represents the smoothing L1 loss function, the specific expression of which is as follows:

[0085]

[0086] The joint loss function is as follows:

[0087]

[0088] In the formula, α represents the balance loss parameter.

[0089] Step 4: Construct the prediction head.

[0090] 4-1) In this embodiment, based on the RoIPooling layer in the Faster R-CNN model, the feature map corresponding to each candidate box is divided and pooled to obtain a feature region vector X of uniform size. f ′;

[0091] 4-2) Design a fully connected layer with an output dimension of N+1 (total number of object categories N + background category) as the classifier, which will output X... f After mapping to the category space, the Softmax function is used to obtain the object category probability p, and the object category with the highest probability in p is taken as the object classification result. The specific expression is as follows:

[0092] p = Softmax(Classifier(X) f ′))

[0093] 4-3) Design a fully connected layer with an output dimension of 4N as a regressor to map the feature vector X. f Perform bounding box regression to predict the coordinate offset (Δx) of the target box. center ,Δy center The final object bounding box is obtained by calculating Δwidth and Δheight.

[0094] 4-4) Using the cross-entropy loss function Calculate the classification loss, and then use the smoothed L1 loss function to calculate the regression loss. The specific expression is as follows:

[0095]

[0096]

[0097] In the formula, p represents the classification loss. n ,y n These represent the predicted target category and the actual target category label, respectively. Indicates regression loss, and These represent the predicted bounding box parameters and the ground truth bounding box parameters of the nth candidate box proposal, respectively, where n is the candidate box number and N' is the number of candidate boxes; Smooth_L1() represents the smoothing L1 loss function.

[0098] The joint loss function is as follows:

[0099]

[0100] In the formula, α′ represents the balance loss parameter.

[0101] Step 5: Randomly select support sets and query sets from the base class dataset to construct multiple meta-tasks, see [link to relevant documentation]. Figure 2 Each meta-task includes multiple support sets and one query set; pre-trained weights are loaded, the feature extraction network is frozen, and the meta-task is used to train the candidate box generation module and the prediction head in an inner loop.

[0102] 5-1) Sample a meta-task m, construct a temporary model identical to the original model, and assign the original model parameters θ to the temporary model, setting the parameters to θ. m ;

[0103] 5-2) Using the support set data from meta-task m, adjust the temporary model parameters θ. m Perform multiple optimizations and update θ m ;

[0104] 5-3) Based on the optimized θ m Using the query set data from metatask m, calculate the loss of metatask m, and calculate the loss with respect to θ. m The gradient g;

[0105] 5-4) If the above loss is less than the loss of the previous task, then use the gradient g, multiplied by the learning rate. Update the original model parameters θ; otherwise, leave the original model parameters unchanged.

[0106] Step 6: Use the meta-task to train the candidate box generation module and the prediction head in the outer loop, that is, repeat step 5 multiple times until the model obtains the best initialization parameters.

[0107] Step 7: Train the model using the new class dataset. The training method is still the same as the training process in Step 6. After a small amount of outer loop training, the model can quickly adapt to the new task and complete the accurate detection of objects with few samples.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0109] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A few-shot target detection method based on meta-learning and text prompts, characterized in that, Includes the following steps: Step 1: Preprocess and label the acquired image data, labeling the location and category of the detected objects in the image to obtain the base class dataset; then randomly divide the base class dataset into the support set and query set of the base class according to a predetermined ratio; A certain amount of image data containing the detection target is collected, and the data is preprocessed and labeled to label the location and category of the detection target in the image to obtain a new class dataset. The new class dataset is then randomly divided into a support set and a query set for the new class according to a predetermined ratio. Step 2: Based on the pre-trained feature extraction module, perform feature extraction on the images in the base class dataset and the new class dataset obtained in Step 1 to obtain the feature map X of each image. f ; Step 3: Construct a candidate box generation module based on text prompts. The input of this module is the feature map X of the image. f Output feature map X f The corresponding candidate box; Step 4: Construct a prediction head to calculate the candidate boxes generated in Step 3, output the class probability vector, and obtain the offset of each candidate box based on bounding box regression to predict the target detection box. Step 5, Training Phase: Construct a base class meta-task using the base class dataset. Use the base class meta-task to train the constructed text-hint-based candidate box generation module and prediction head to optimize the network parameters of the text-hint-based candidate box generation module and prediction head. Step 6: Fine-tuning phase. A new meta-task is constructed using the new class dataset. The network parameters of the text-hint-based candidate box generation module and prediction head trained in Step 5 are fine-tuned using the new meta-task to obtain the target detection model for detecting new types of objects.

2. The method as described in claim 1, characterized in that, In step 1, both the support set and query set of the base class include N types of detection objects, and each type of detection object has K images, where N and K are preset values.

3. The method as described in claim 1, characterized in that, In step 1, the preprocessing specifically includes: The image size is normalized to a uniform size, and for the support set of the base class or new class, the images in it are also horizontally flipped with a specified probability; Each image in the dataset is labeled with text. The corresponding text prompts include the location and category of each detected object or target, as well as the interaction information between the objects or targets.

4. The method as described in claim 1, characterized in that, In step 2, the feature extraction module adopts the ResNet network structure, which includes: 1 convolutional layer, 2 pooling layers, several residual blocks and 1 fully connected layer.

5. The method according to any one of claims 1 to 4, characterized in that, Step 3 specifically includes: Step 3.1, for feature map X f For each pixel, several anchor boxes are generated with that pixel as the center, and q is defined as the number of anchor boxes; Step 3.2, extract the text prompt data T = (t1,...,t) corresponding to the images in the dataset. s The input is fed into a Transformer-based text encoder to extract the corresponding text features T' = (t1',...,t...). s '), where s represents the total number of prompts in the image, t k ,t' k (k = 1, ..., s) represent the kth text prompt data and its corresponding text features, respectively; Step 3.3: For each anchor box, calculate its intersection-union ratio (IU) with the bounding boxes of all real detected objects or targets, and determine the real label for the foreground and background binary classification of each anchor box based on the set IU threshold. Step 3.4: Perform dimension mapping on the text feature T' and the feature maps corresponding to the q anchor boxes based on the linear layer to obtain the mapped feature map N of the anchor boxes. fb =(n fb,1 ,...,n fb,q ) and text features T fb =(t fb,1 ,...,t fb,s ), where n fb,i This represents the feature map of the i-th anchor box after mapping, where i = 1, ..., q, t fb,k Let k represent the k-th text feature after mapping, where k = 1, ..., s; For each anchor frame, calculate its feature map n. fb,i With each text feature t fb,k The cosine similarity between the two values ​​is taken as the maximum similarity metric, denoted as S. i ; Step 3.5, for the similarity measure S i The Sigmoid function is used to calculate the binary classification probability p of the foreground and background classes for each anchor box. i ; Step 3.6, predict the probability p based on the foreground and background categories. i Take the anchor boxes corresponding to the first N' maximum values ​​as candidate boxes, and perform regression calculations on the candidate boxes to calculate the offset of each candidate box relative to the true bounding box of the detected object or target, including: the horizontal and vertical coordinate offset of the candidate box center, and the width and length offset of the candidate box.

6. The method as described in claim 5, characterized in that, In step 5, the loss function expression for the candidate box generation module based on text prompts is: in, p represents the foreground / background classification loss. ll With y l Let be the predicted foreground and background probabilities and the true label of the l-th anchor box, respectively; γ represent the influence factor; and q represent the number of anchor boxes. This represents the anchor box regression loss, where N' represents the number of candidate boxes. and These represent the predicted bounding box parameters and the ground truth bounding box parameters for the l-th candidate box, respectively; Smooth_L1() represents the smoothing L1 loss function. The joint loss function for the text-based candidate box generation module is: Where α is the preset first balance loss parameter.

7. The method as described in claim 1, characterized in that, The prediction head constructed in step 4 is specifically as follows: The RoIPooling layer is used to partition and pool the feature maps corresponding to the candidate boxes, resulting in a feature region vector X of uniform size. f '; The feature region vector X is processed using a fully connected layer and a softmax function. f 'Perform target classification, and classify the feature region vector X' f 'Mapped to the target category prediction probability distribution, and the category with the highest probability in the target category prediction probability distribution is taken as the predicted classification result; The feature region vector X is processed through a fully connected layer. f Perform bounding box regression to predict the coordinate offset values ​​of the target box and obtain the final target bounding box. The coordinate offset values ​​of the target box include: the horizontal and vertical coordinate offsets of the target box center, and the width and length offsets of the candidate box.

8. The method as described in claim 7, characterized in that, In step 5, the loss function expression for the prediction head is: in, Let p represent the classification loss and regression loss generated by the prediction head, respectively. n ,y n These represent the predicted target category and the true target category label, respectively. and These represent the bounding box parameters predicted for the nth candidate box and the ground truth bounding box parameters, respectively. N' is the number of candidate boxes, and Smooth_L1() represents the smooth L1 loss function. The joint loss function for the prediction heads is: α′ is a preset second balance loss parameter.

9. The method as described in claim 1, characterized in that, Step 5 specifically includes: Step 5.1: Randomly select support sets and query sets from the base class dataset to form multiple base class meta-tasks. Each base class meta-task includes multiple support sets and one query set. Step 5.2: Load the pre-trained weights, freeze the feature extraction module, and use the base class meta-task to train the text-based candidate box generation module and the prediction head.

10. The method as described in claim 9, characterized in that, Step 5.2 includes: Step 5.2.1: Sample a base class meta-task m, construct a temporary model identical to the original model, and assign the original model parameters θ to the temporary model, setting the parameters to θ. m The original model includes a pre-feature extraction module, a candidate box generation module, and a prediction head. During training, the network parameters of the pre-feature extraction module of both the original model and the temporary model are frozen. Step 5.2.2: Apply temporary model parameters θ based on the support set in the base class meta-task m. m After multiple optimizations, the updated and optimized model parameters θ are obtained. m ; Step 5.2.3: Based on the model parameters θ obtained in Step 5.2.2 m Using the query set from the base class meta-task m, calculate the loss of the base class meta-task m, and calculate this loss for the model parameters θ. m The gradient g; where the loss includes: foreground / background classification loss. Anchor frame regression loss The classification loss generated by the prediction head and regression loss Step 5.2.4: If the current loss is less than the loss of the previous base class task, then multiply the gradient g by the preset learning rate. Update the original model parameters θ; otherwise, leave the original model parameters θ unchanged. Step 5.2.5: Repeat steps 5.2.1 to 5.2.4 until the loss converges.

Images