An open vocabulary object detection method, system, medium and program product based on multi-object classification

Abstract

The application discloses a multi-target classification-based open vocabulary target detection method, system, medium and program product, and belongs to the technical field of computer vision. The method detects new classes of objects that do not exist in a training set by using context information and comprises the following steps: a backbone feature extraction network is used to perform feature extraction on an image to obtain a feature map; a region proposal network generates a group of candidate frames in the feature map; a candidate frame expansion module is applied to all the candidate frames to obtain expanded frames; a distillation module is used to learn knowledge from a CLIP image encoder to obtain a distillation loss; features extracted from the candidate frames and the expanded frames and text features are input into a multi-target classification module to obtain a classification loss; the model is trained in combination with the distillation loss and the classification loss; finally, a picture to be detected is input into the trained model to generate a predicted target region, corresponding class names and corresponding confidence, so that more accurate open vocabulary target detection is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to open vocabulary object detection methods, systems, media and program products based on multi-object classification. Background Technology

[0002] In recent years, with the improvement of computer performance and the development of big data, visual information data has increased rapidly. Multimedia data, including still images, moving images, video files, and audio files, are spreading quickly on various social media platforms. Object detection, as one of the most fundamental problems in computer vision, is widely used in object tracking, behavior understanding, human-computer interaction, facial recognition, and many other fields, attracting widespread attention and research from scholars since the early 20th century. Humans primarily receive external information through vision, so visual information-based application technologies will be a forward-looking research area in artificial intelligence. Among these, facial recognition, video surveillance, object detection, internet image content review, and biometric recognition technologies have become current research hotspots. These technologies are also widely used in fields such as healthcare, elderly care, transportation, urban operations, and security, for example: medical image diagnosis, pose estimation, station security checks, autonomous driving, vehicle speed detection, and video surveillance behavior analysis.

[0003] The goal of traditional object detection is to identify and label all objects of known categories in an image. However, to detect a wider range of categories in real-world scenarios, the model must add more categories and their corresponding fine-grained data to the training set, which incurs high costs. Furthermore, linearly adding new categories exacerbates the long-tailed distribution problem of categories.

[0004] Open-vocabulary object detection has been proposed as a solution in recent years. It aims to detect novel categories never seen during training. In contrast to zero-shot learning, which strictly limits information about novel categories, open-vocabulary object detection allows access to semantic information associated with the novel category, derived from image-text pairs with image-level labels or pre-trained visual-language models. This facilitates the learning of a cross-modal visual-language representation space, linking image regions to word descriptions.

[0005] Existing open-vocabulary object detection methods primarily transfer new category knowledge from pre-trained visual-language models to the object detector through knowledge distillation, thereby gaining the ability to identify new categories.

[0006] OVR-CNN (Open Vocabulary Region-Convolutional Neural Network) pioneered the use of contrastive learning to expand the vocabulary across a large number of image-text pairs, while simultaneously fine-tuning on detection data with a limited vocabulary. Subsequent models have primarily improved performance through modifications in the following areas: knowledge distillation, region-text pre-training, cue modeling, region-text alignment, and transfer learning. For example, DK-DETR (Distilling Knowledge Determination Transformer) models object relationships by aligning feature relation matrices between student and teacher models. OADP (Object Aware Distillation Pyramid) enhances the student model by aligning image patch embeddings. HierarchicalKD (Hierarchical Knowledge Distillation) extracts multi-scale global features from images to align descriptions. GKC (Gated Contextual Kernel) avoids overfitting by introducing additional category synonyms. ZeroSeg (Zero-shot Segmentation) extracts visual concepts by processing images into multi-scale views.

[0007] However, these methods introduce noise when acquiring knowledge of new categories and lack sufficient attention to context during training, leading to overfitting of the model to the base category and limitations in performance when detecting new categories. The context in these images contains the inherent structure of multiple semantic concepts, which may include objects of new categories, the targets we need to detect.

[0008] The existing background cue learning method in open vocabulary object detection (CN202410651398) uses an RPN network to directly generate candidate boxes that may contain new object classes. However, since the RPN network is only trained on the base class, the candidate boxes generated in this way have significant room for improvement. Summary of the Invention

[0009] The purpose of this invention is to provide an open vocabulary object detection method, system, medium, and program product based on multi-target classification, which can utilize contextual information to detect new categories of objects that have not appeared in the training set, thereby achieving more accurate open vocabulary object detection.

[0010] To achieve the above objectives, the solution of the present invention is:

[0011] An open-lexicon object detection method based on multi-object classification utilizes contextual information to detect new categories of objects that have not appeared in the training set, including the following steps:

[0012] Step S1: Construct an open vocabulary object detection model based on multi-object classification; the model includes a backbone feature extraction network ResNet50, a region proposal network RPN, a candidate box expansion module CBM, a distillation module KD, and a multi-object classification module MSC;

[0013] Step S2, train the model, which includes the following steps:

[0014] Step S21, given a training image sample set;

[0015] Step S22: Randomly select an image from the training image sample set, use the backbone feature extraction network ResNet50 to extract features from the image to obtain a feature map, and input the feature map into the region proposal network RPN;

[0016] Step S23: The Region Proposal Network (RPN) generates a set of candidate boxes in the feature map, and then applies the Candidate Box Extension Module (CBM) to all candidate boxes to obtain extended boxes located near the candidate boxes for adding contextual information.

[0017] Step S24: Learn knowledge from the CLIP image encoder using the distillation module KD, and extract the features F of the extended boxes and candidate boxes using the backbone feature extraction network ResNet50 of the model. E The candidate bounding box and the expanded bounding box are processed using the CLIP image encoder to obtain their features F. C Then, InfoNCE Loss is applied to calculate the two corresponding features F. E and F C Distillation loss L between KD ;

[0018] Step S25: Obtain text features F by processing the base category names of the training image sample set using the CLIP text encoder. T , feature F E and text features F T The candidate boxes are classified in the multi-objective classification module MSC, and the total classification loss L is obtained. MSC ;

[0019] Step S26, combined with distillation loss L KD Total classification loss L MSC The model is trained to obtain a multi-object classification open vocabulary object detection model;

[0020] Step S3: Input the image to be detected into the trained model, use the trained model to generate the predicted target region and the corresponding category name on the image, and give the confidence score of this prediction. This confidence score comes from the classification probability score of the target region by the multi-object classification module MSC.

[0021] Furthermore, in step S22, an image is arbitrarily selected from the training image sample set as the image I∈R used for training. H×W×C Where H, W, and C represent the image height, image width, and image dimension, respectively. The feature map is obtained by using the ResNet50 backbone feature extraction network to extract features from the image I.

[0022] Step S23: Generate a set of candidate boxes P = {p0, p1, ..., p...} from the feature map using the Region Proposal Network (RPN). n}, where p i Let represent the i-th candidate box, containing four values ​​that record its relative position in the image, where n is the number of candidate boxes. Furthermore, to enhance contextual information, a candidate box expansion module (CBM) is applied to all candidate boxes P, resulting in expanded boxes B = {B0, B1, ..., B...}. n’}, Expanded box B i Adopt and p i The same expression, B i Let represent the i-th expanded box, which also contains four values ​​used to record the relative position of the i-th expanded box in the image, and n' be the number of expanded boxes. Simultaneously, a regression module is applied to optimize the candidate boxes P and obtain the regression loss L. reg .

[0023] Furthermore, in step S23, candidate boxes p are first selected. i Expand each candidate box by using eight nearby regions to obtain P ie ={p i ,y i1 ,…,y i8}, where p i ∈P, where p i This represents the i-th candidate box selected earlier, while r i1 ,…,r i8 Representative candidate box p i The eight surrounding areas;

[0024] Then filter P ie To remove regions that extend beyond the image boundaries, P is obtained. ir ={p i ,r i1 ,…,r ik}, where k represents the number of remaining regions, and k≤8;

[0025] In order to find the existence of new types of objects, based on p i Similarity to P ir Sort the data in ascending order, while simultaneously filtering out regions with low objectivity scores, to obtain P. i c ={p i ,r i1 ,…,r ij}, j≤k;

[0026] Finally, from P i c Choose three with p i Find the boxes with the lowest similarity and match them with p. ii This forms an expanded frame. Further, the final step in step S23 is represented as follows:

[0027] B i ={p i ,p im c ,p i(m+1) c ,p i(m+2) c}

[0028] Where m represents selecting a similarity region most likely to contain the new class of objects, and p im c ,p i(m+1) c ,p i(m+2) c For the three selected surrounding regions, the final expanded bounding box B is obtained based on these three regions and the original region. i ;

[0029] Expressed as a formula, it is denoted as:

[0030] p i c =Mask(Sort(Φ(p) i ·r i )));

[0031] Where Φ(·) represents the cosine similarity, Sort represents the process of calculating and sorting similarities, and Mask represents the selection of surrounding regions based on the similarity sorting results. i ∈P ir And r i ≠p i .

[0032] Furthermore, in step S24, the ResNet50 backbone feature extraction network of the model extracts the features of the expanded boxes and the features F of the candidate boxes.E :F E ={F E 1,F E 2,…,F E n” ,F E (n”+1) ,…,F E 2n”}, where the first n” features are the features of the expanded box, and the last n” features are the features of the candidate box;

[0033] The CLIP image encoder is then used to process the candidate bounding boxes and expanded boxes to obtain their features F. C :F C ={F C 1,F C 2,…,F C N” ,F C (N”+1) ,…,F C 2N”}, where the first N” features are the features of the expanded boxes, and the last N” features are the features of the candidate boxes;

[0034] Next, the InfoNCE loss is applied to calculate the distillation loss L between the two corresponding features. KD By optimizing distillation loss L KD The feature F generated by this model E Features F generated by CLIP image encoder C This allows for the learning of knowledge from the CLIP image encoder.

[0035] Furthermore, in step S25, the category name is processed using the CLIP text encoder to obtain text features F. T ={F T 1,F T 2,…,F T q}, where the meaning of q changes depending on the model's state. During training, since the model can only use information from the basic categories of the training image sample set, q represents the number of basic categories in the training image sample set. During testing, since the model detects both basic and new categories of objects simultaneously, q represents the total number of categories in the test image sample set. Then, the multi-objective classification module MSC processes the features F... E and text features F T The classification loss L is obtained by performing classification. MSC ;

[0036] First, the candidate boxes are classified: the features F extracted by the ResNet50 backbone feature extraction network of this invention are classified. E={F E 1,F E 2,…,FE n”+1 ,F E n”+2 ,…,F E 2n” In the multi-objective classification module MSC, for each pair of extended boxes B and their corresponding candidate boxes p, the true class G = {g1, g2, ..., g} is generated by calculating the IoU between p and the fact box. n},g1,g2,…,g n Corresponding to the basic target categories in the training image sample set;

[0037] Then, by calculating F E i and F T gi， The cosine similarity of (gi∈G) generates the classification score, where F E i and F T gi It refers to F E and F T For each candidate box, a classification loss L is calculated based on a single category, using one of the features in the dataset. cls For each expanded bounding box, calculate the classification loss, which is divided into L... B and L N Two parts, of which L B L represents the loss of the basic category of objects. N The loss is the loss for potential new categories of objects, defined as:

[0038]

[0039] Where Φ(·) represents cosine similarity, Softmax represents normalization of the numerical values, and log represents taking the logarithm.

[0040] Select the top k categories with the highest classification scores, G′={g1′,g2′,…,g…} k As a category of alignment, the loss is defined as:

[0041]

[0042] Where Φ(·) represents the cosine similarity, softmax represents the normalization of the numerical values, and log represents taking the logarithm.

[0043] Total classification loss L MSC Defined as:

[0044] L MSC =L cls +L B+λ4L N

[0045] Where λ4 represents the weight of the loss for potential new object categories.

[0046] The present invention also provides an open vocabulary target detection method system based on multi-target classification, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method.

[0047] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method described herein.

[0048] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described herein.

[0049] By adopting the above technical solution, this invention mines potential new target classes from contextual information and designs detailed steps to filter candidate boxes, thereby obtaining richer information on new target classes and detecting new categories of objects not present in the training set. Simultaneously, a multi-object classification method is used to utilize new class information, which also improves the model's understanding of new class information. This invention uses the OV-COCO and OV-LVIS datasets, trains the model according to the above steps, and tests it using a test set. This invention uses a validation set to evaluate performance on the OV-COCO and OV-LVIS datasets. Evaluation Metrics This represents the average accuracy (mAP) of the new class at an IoU threshold of 0.5; the evaluation metric AP r , where mAP represents the average accuracy (mAP) for rare categories (new categories). Our method achieves a recognition performance of 38.5% (ResNet50-FPN) for new categories on the OV-COCO dataset and 24.1% (detection task) and 23.4% (segmentation task) for new categories on the OV-LVIS dataset, demonstrating higher accuracy compared to other methods and proving the superior performance of this invention in new category target recognition. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the network structure of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0052] like Figure 1 This invention provides an open vocabulary target detection method based on multi-target classification. The detection method of this invention includes the following steps:

[0053] Step S1: Construct an open vocabulary object detection model (CANet) based on multi-object classification. This model includes a backbone feature extraction network (ResNet50), a region proposal network (RPN), a candidate box expansion module (CBM), a knowledge distillation module (KD), and a multi-object classification module (MSC).

[0054] The ResNet50 backbone feature extraction network is used to extract features from any image in the training image sample set to obtain a feature map; the Region Proposal Network (RPN) is used to generate a set of candidate boxes from the feature map extracted by the ResNet50 backbone feature extraction network; the Candidate Box Expansion Module (CBM) is used to expand the candidate boxes, and can also rank them according to the similarity between the expanded parts and the candidate boxes, and retain the best candidate boxes; the Distillation Module (KD) is used to merge the image features generated by the ResNet50 backbone feature extraction network and the CLIP (Contrastive Language-Image Pre-Training) image encoder; the Multi-Object Classification Module (MSC) is used to classify the candidate boxes and the expanded boxes simultaneously.

[0055] Step S2: Train the model: Randomly select an image from the training image sample set, use the ResNet50 backbone feature extraction network to extract features from the image to obtain a feature map, and use the Region Proposal Network (RPN) to generate a set of candidate boxes from the feature map; the candidate box expansion module (CBM) expands all candidate boxes to obtain expanded boxes located near the candidate boxes to add contextual information; the distillation module (KD) learns knowledge from the CLIP image encoder and extracts the features F of the expanded boxes and candidate boxes. E The candidate bounding box and the expanded bounding box are processed using the CLIP image encoder to obtain their features F. C Then, InfoNCE Loss is applied to calculate the two corresponding features F. E and F C Distillation loss L between KD Text features F are obtained by processing the base category names of the training image sample set using the CLIP text encoder. T , feature F E and text features FT The data is input into the multi-object classification module MSC for classification, and the classification loss L is obtained. MSC Combined with distillation loss L KD And classification loss L MSC The model is trained to obtain a multi-object classification open vocabulary object detection model;

[0056] Step S3: Input the image to be detected into the trained model, use the trained model to generate the predicted target region and the corresponding category name on the image, and give the confidence score of this prediction. This confidence score comes from the classification probability score of the target region by the multi-object classification module MSC.

[0057] In step S2, the model training method specifically includes the following steps:

[0058] Step S21: Given a dataset set with image-level labels, divide the set into a training image sample set and a test image sample set;

[0059] Step S22: Randomly select one image from the training image sample set as the image I∈R used for training. H ×W×C Where H, W, and C represent the image height, image width, and image dimension, respectively. The feature map is obtained by using the ResNet50 backbone feature extraction network to extract features from the image I.

[0060] Step S23: Generate a set of candidate boxes P = {p0, p1, ..., p...} from the feature map using the Region Proposal Network (RPN). n}, where p i Let $p$ represent the $i$-th candidate box, containing four values ​​that record its relative position in the image. $n$ is the number of candidate boxes. Furthermore, to add contextual information, the Candidate Box Extension Module (CBM) is applied to all candidate boxes $P$, resulting in extended boxes $B = {B0, B1, ..., B$. n’}, Expanded box B i Adopt and p i The same expression, B i This represents the i-th expanded bounding box, which also contains four values ​​used to record the relative position of the i-th expanded bounding box in the image, and n' is the number of expanded bounding boxes. Simultaneously, this invention applies a regression module to optimize the candidate bounding box P and obtains the regression loss L. reg That is, by optimizing the regression loss, the candidate box P extracted by the model can be closer to the actual location of the object.

[0061] Specifically, first, the candidate box p is retrieved. i Expand each candidate box by using eight nearby regions to obtain Pie ={p i ,r i1 ,…,r i8}, where p i ∈P, where p i This represents the i-th candidate box selected earlier, while r i1 ,…,r i8 Representative candidate box p i The eight surrounding areas;

[0062] Then filter P ie To remove regions that extend beyond the image boundaries, P is obtained. ir ={p i ,r i1 ,…,r ik}, where k represents the number of remaining regions, and k≤8;

[0063] Furthermore, this invention aims to discover the existence of new types of objects. Therefore, this invention is based on p i Similarity to P ir Sort the data in ascending order, while simultaneously filtering out regions with low objectivity scores, to obtain P. i c ={p i ,r i1 ,…,r ij}, j≤k;

[0064] Finally, the present invention starts from P i c Choose three with p i Find the boxes with the lowest similarity and match them with p. i Combined, they form an expanded frame.

[0065] This process can be represented as:

[0066] B i ={p i ,p im c ,p i(m+1) c ,p i(m+2) c}

[0067] Where m represents selecting a similarity region most likely to contain the new class of objects, and p im c ,p i(m+1) c ,p i(m+2) c For the three selected surrounding regions, the final expanded bounding box B is obtained based on these three regions and the original region. i .

[0068] This can be expressed as a formula:

[0069] p i c =Mask(Sort(Φ(p) i ·r i )));

[0070] Where Φ(·) represents the cosine similarity, Sort represents the process of calculating and sorting similarities, and Mask represents the selection of surrounding regions based on the similarity sorting results. i ∈P i r and r i ≠p i .

[0071] Step S24: The present invention uses the distillation module KD to learn knowledge from the CLIP image encoder:

[0072] The ResNet50 backbone feature extraction network of the model extracts features from the expanded bounding boxes and the candidate bounding boxes. E :F E ={F E 1,F E 2,…,F E n” ,F E (n”+1) ,…,F E 2n”}, where the first n” features are the features of the expanded bounding box, and the last n” features are the features of the candidate bounding box. Simultaneously, this invention also uses a CLIP image encoder to process the candidate bounding box and the expanded bounding box to obtain their features F. C :F C＝ {F C 1,F C 2,…,F C N” ,F C (N”+1) ,…,F C 2N”}, where the first N” features are the features of the expanded bounding box, and the last N” features are the features of the candidate bounding box. Here, this invention uses InfoNCE loss (information contrast loss) to calculate the distillation loss L between the two corresponding features. KD By optimizing this distillation loss L KD This makes the features generated by the present invention similar to those generated by the CLIP image encoder, thereby enabling the learning of knowledge from the CLIP image encoder.

[0073] Step S25: Use the CLIP text encoder to process the category names to obtain text features F. T ={F T1,F T 2,…,F T q}, where the meaning of q changes depending on the model's state. During training, since the model can only use information from the basic categories of the training image sample set, q represents the number of basic categories in the training image sample set. During testing, since the model detects both basic and new categories of objects simultaneously, q represents the total number of categories in the test image sample set. Then, the multi-objective classification module MSC processes the features F... E and text features F T The classification loss L is obtained by performing classification. MSC ;

[0074] First, classify the candidate boxes:

[0075] The features F extracted by the ResNet50 backbone feature extraction network of this invention are... E ={F E 1,F E 2,…,F E n”+1 ,F E n”+2 ,…,F E 2n” In the multi-objective classification module MSC, for each pair of extended boxes B and its corresponding candidate box p, the true category G = {g1, g2, ..., g} is generated by calculating the IoU (intersection over Union) between p and the fact box. n},g1,g2,…,g n The corresponding basic target categories in the training image sample set.

[0076] Then, by calculating F E i and F T gi The cosine similarity of (gi∈G) generates the classification score, where F E i and F T gi It refers to F E and F T For each candidate box, a classification loss L is calculated based on a single category, using one of the features in the dataset. cls For each expanded bounding box, calculate the classification loss, which can be divided into L... B and L N Two parts, of which L B L represents the loss of the basic category of objects. N The loss is the loss for potential new categories of objects, defined as:

[0077]

[0078] Where Φ(·) represents cosine similarity, Softmax represents normalization of the numerical values, and log represents taking the logarithm.

[0079] Select the top k categories with the highest classification scores, G′={g1′,g2′,…,g…} k As a category of alignment, the loss is defined as:

[0080]

[0081] Where Φ(·) represents cosine similarity, softmax represents normalization of the numerical values, and log represents taking the logarithm.

[0082] Total classification loss L MSC Defined as:

[0083] L MSC =L cls +L B +λ4L N

[0084] Where λ4 represents the weight of the loss for potential new object categories.

[0085] Step S26, combined with distillation loss L KD Total classification loss L MSC The model was trained to obtain a multi-object classification open vocabulary object detection model.

[0086] Extensive experiments conducted on the OV-COCO and OV-LVIS datasets demonstrate that the proposed CANet achieves significant and consistent performance improvements compared to other competing methods.

[0087] This invention was developed on the Ubuntu platform, and the deep learning framework developed is based on PyTorch. The primary language used in this invention is Python.

[0088] Take the OV-COCO and OV-LVIS datasets, train the model according to the steps above, and test it using the test image sample set.

[0089] Tables 1 and 2 show the test results of our invention and other methods on two datasets, respectively. It can be seen that our invention performs best compared to other methods. CANet(Ours) represents the results of our invention. We use a validation set to evaluate the performance of our method on the OV-COCO and OV-LVIS datasets. Evaluation Metrics This represents the average accuracy (mAP) of the new class at an IoU threshold of 0.5; the evaluation metric APr , where mAP represents the average accuracy (mAP) for rare categories (new categories). Our method achieves a recognition performance of 38.5% (ResNet50-FPN) for new categories on the OV-COCO dataset and 24.1% (detection task) and 23.4% (segmentation task) for new categories on the OV-LVIS dataset, demonstrating higher accuracy compared to other methods and proving the superior performance of this invention in new category target recognition.

[0090]

[0091] Table 1 compares the model performance on OV-COCO with state-of-the-art techniques.

[0092]

[0093] Table 2 compares the model performance on OV-LVIS with state-of-the-art techniques.

[0094] The present invention also provides an open vocabulary target detection method system based on multi-target classification, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method.

[0095] The computer program product includes a computer program / instructions that, when executed by a processor, implement the method described.

[0096] The present invention also provides a computer-readable storage medium storing the above-described computer program, which, when executed by a processor, implements the method described above.

Claims

1. An open vocabulary target detection method based on multi-target classification, characterized in that, Using contextual information, new categories of objects not present in the training set are detected, including the following steps: Step S1: Construct an open vocabulary object detection model based on multi-object classification; the model includes a backbone feature extraction network ResNet50, a region proposal network RPN, a candidate box expansion module CBM, a distillation module KD, and a multi-object classification module MSC; Step S2, train the model, which includes the following steps: Step S21, given a training image sample set; Step S22: Randomly select an image from the training image sample set, use the backbone feature extraction network ResNet50 to extract features from the image to obtain a feature map, and input the feature map into the region proposal network RPN; Step S23: The Region Proposal Network (RPN) generates a set of candidate boxes in the feature map, and then applies the Candidate Box Extension Module (CBM) to all candidate boxes to obtain extended boxes located near the candidate boxes for adding contextual information. Step S24: Learn knowledge from the CLIP image encoder using the distillation module KD, and extract the features F of the extended boxes and candidate boxes using the backbone feature extraction network ResNet50 of the model. E The candidate bounding box and the expanded bounding box are processed using the CLIP image encoder to obtain their features F. C Then, InfoNCE Loss is applied to calculate the two corresponding features F. E and F C Distillation loss L between KD ; Step S25: Obtain text features F by processing the base category names of the training image sample set using the CLIP text encoder. T , feature F E and text features F T The candidate boxes are classified in the multi-objective classification module MSC, and the total classification loss L is obtained. MSC ; Step S26, combined with distillation loss L KD Total classification loss L MSC The model is trained to obtain a multi-object classification open vocabulary object detection model; Step S3: Input the image to be detected into the trained model, use the trained model to generate the predicted target region and the corresponding category name on the image, and give the confidence score of this prediction. This confidence score comes from the classification probability score of the target region by the multi-object classification module MSC.

2. The open vocabulary target detection method based on multi-target classification according to claim 1, characterized in that: Step S22: Randomly select one image from the training image sample set as the image I∈R used for training. H×W×C Where H, W, and C represent the image height, image width, and image dimension, respectively. The feature map is obtained by using the ResNet50 backbone feature extraction network to extract features from the image I. Step S23: Generate a set of candidate boxes P = {p0, p1, ..., p...} from the feature map using the Region Proposal Network (RPN). n }, where p i Let represent the i-th candidate box, containing four values ​​that record its relative position in the image, where n is the number of candidate boxes. Furthermore, to enhance contextual information, a candidate box expansion module (CBM) is applied to all candidate boxes P, resulting in expanded boxes B = {B0, B1, ..., B...}. n’ }, Expanded box B i Adopt and p i The same expression, B i Let represent the i-th expanded box, which also contains four values. These values ​​record the relative position of the i-th expanded box in the image, and n' is the number of expanded boxes. Simultaneously, a regression module is applied to optimize the candidate boxes P and obtain the regression loss L. reg .

3. The open vocabulary target detection method based on multi-target classification according to claim 2, characterized in that: In step S23, candidate boxes p are first selected. i Expand each candidate box by using eight nearby regions to obtain P ie ={p i ,r i1 ,…,r i8 }, where p i ∈P, where p i This represents the i-th candidate box selected earlier, while r i1 ,…,r i8 Representative candidate box p i The eight surrounding areas; Then filter P ie To remove regions that extend beyond the image boundaries, P is obtained. ir ={p i ,r i1 ,…,r ik }, where k represents the number of remaining regions, and k≤8; In order to find the existence of new types of objects, based on p i Similarity to P ir Sort the data in ascending order, while simultaneously filtering out regions with low objectivity scores, to obtain P. i c ={p i ,r i1 ,…,r ij }, j≤k; Finally, from P i c Choose three with p i Find the boxes with the lowest similarity and match them with p. i Combined, they form an expanded frame.

4. The open vocabulary target detection method based on multi-target classification according to claim 3, characterized in that: The process in the last step of step S23 is represented as follows: B i ={p i ,p im c ,p i(m+1) c ,p i(m+2) c } Where m represents selecting a similarity region most likely to contain the new class of objects, and p im c p i(m+1) c p i(m+2) c For the three selected surrounding regions, the final expanded bounding box B is obtained based on these three regions and the original region. i ; Expressed as a formula, it is denoted as: p i c =Mask(Sort(Φ(p i ·r i ))); Where Φ(·) represents the cosine similarity, Sort represents the process of calculating and sorting similarities, and Mask represents the selection of surrounding regions based on the similarity sorting results. i ∈P ir And r i ≠p i .

5. The open vocabulary target detection method based on multi-target classification according to claim 1, characterized in that: In step S24, the ResNet50 backbone feature extraction network of the model extracts the features of the expanded boxes and the candidate boxes F. E :F E ={F E 1,F E 2, …,F E n” ,F E (n”+1) ,…,F E 2n” }, where the first n” features are the features of the expanded box, and the last n” features are the features of the candidate box; The CLIP image encoder is then used to process the candidate bounding boxes and expanded boxes to obtain their features f. C :F C ={F C 1,F C 2, …,F C N” ,F C (N”+1) ,…,F C 2N” }, where the first N” features are the features of the expanded boxes, and the last N” features are the features of the candidate boxes; Next, the InfoNCE loss is applied to calculate the distillation loss L between the two corresponding features. KD By optimizing distillation loss L KD The feature F generated by this model E Features F generated by CLIP image encoder C This allows for the learning of knowledge from the CLIP image encoder.

6. The open vocabulary target detection method based on multi-target classification according to claim 5, characterized in that: In step S25, the category name is processed using the CLIP text encoder to obtain text features F. T ={F T 1,F T 2,…,F T q }, where the meaning of q changes depending on the state of the model. When the model is being trained, since it can only use the information of the basic categories of the training image sample set for training, q represents the number of basic categories in the training image sample set. During the testing process, since the model detects objects of both basic and new categories at the same time, q represents the total number of categories in the test image sample set. Then, the feature F is processed by the multi-objective classification module MSC. E and text features F T The classification loss L is obtained by performing classification. MSC ; First, the candidate boxes are classified: the features F extracted by the backbone feature extraction network ResNet50 are classified. E ={F E 1,F E 2,…,F E n”+1 ,F E n”+2 ,…,F E 2n” In the multi-objective classification module MSC, for each pair of extended boxes B and their corresponding candidate boxes p, the true class G = {g1, g2, ..., g} is generated by calculating the IoU between p and the fact box. n },g1,g2,…,g n The corresponding basic target categories in the training image sample set; Then, by calculating F E i and F T gi， The cosine similarity of (gi∈G) generates the classification score, where F E i and F T gi It refers to F E and F T For each candidate box, based on a single category, calculate the classification loss L using a specific feature. cls For each expanded bounding box, calculate the classification loss, which is divided into L... B and L N Two parts, of which L B L represents the loss of the basic category of objects. N The loss is the loss for potential new categories of objects, defined as: Where Φ(·) represents the cosine similarity, Softmax represents the normalization of the numerical values, and log represents taking the logarithm. Select the top k categories with the highest classification scores, G′={g1′,g2′,…,g…} k As a category of alignment, the loss is defined as: Where Φ(·) represents the cosine similarity, softmax represents the normalization of the numerical values, and log represents taking the logarithm. Total classification loss L MSC Defined as: L MSC =L cls +L B +λ4L N Where λ4 represents the weight of the loss for potential new object categories.

7. An open vocabulary target detection method system based on multi-target classification, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as claimed in any one of claims 1 to 6.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images