A method and system for detecting targets of unknown category

Abstract

This invention provides a method and system for detecting unknown category objects, relating to the field of object detection. The method mainly includes: generating pseudo-label information for unknown category objects in each image sample of the training sample set based on a constructed semantic image segmentation model; training a category-related object detection head and a category-independent object detection head respectively; acquiring the target image to be detected, and extracting and fusing all category-related features and all category-independent features in the target image respectively; inputting the fused category-related features into the trained category-related object detection head to obtain the detection result of known category objects, and inputting the fused category-independent features into the trained category-independent object detection head to obtain the detection result of foreground objects; and obtaining the detection result of unknown category objects in the target image based on the foreground object detection result and the detection result of known category objects. This method can effectively improve the detection accuracy of unknown category objects.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method and system for detecting unknown categories of targets. Background Technology

[0002] Object detection is a fundamental and challenging task in computer vision, with the core objective of identifying predefined object categories. This task has wide applications in various fields, including but not limited to facial recognition, autonomous driving, and industrial quality inspection. In practical applications such as robotics, autonomous vehicles, healthcare, and surveillance, ensuring system safety is paramount; therefore, detection systems need the ability to identify unknown object categories. However, during model training, we cannot know in advance which unknown object categories will be involved in the testing phase. Therefore, how to effectively detect unknown object categories has always been an important research topic in the field of object detection.

[0003] Currently, energy distribution or probability distribution is commonly used to distinguish between known and unknown categories of objects. For energy distribution, additional datasets of unknown categories are often needed to generate the energy distribution of these objects, and the unknown category is detected by separating the energy distributions of known and unknown categories. For probability distribution, localization quality scores such as centrality and intersection-union ratio (IUU) are typically used to assist in detecting unknown categories. However, neither energy distribution-based nor probability distribution-based detection methods currently accurately identify unknown categories of targets, generally suffering from low detection accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for detecting unknown categories of targets, so as to improve the detection accuracy of unknown categories of targets and more accurately and reliably identify known category objects and unknown category objects from any image.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] On the one hand, this invention proposes a method for detecting targets of unknown categories, including:

[0007] Obtain a training sample set, which includes several image samples. Each image sample includes several objects of unknown category and several objects of known category. The objects of known category are labeled with label information indicating the object category.

[0008] Based on the constructed semantic image segmentation model, pseudo-label information is generated for unknown category objects in each image sample in the training sample set; the semantic image segmentation model includes a semantic segmentation model and a general image segmentation model.

[0009] A class-related object detection head and a class-independent object detection head are used. Based on the pseudo-label information of unknown class objects and the label information of known class objects in each image sample in the training sample set, the class-related object detection head and the class-independent object detection head are trained respectively to obtain trained class-related object detection heads and trained class-independent object detection heads.

[0010] A target image to be detected is acquired, and all class-related features and all class-independent features in the target image are extracted respectively. The class-related features and all class-independent features are then fused to obtain fused class-related features and fused class-independent features.

[0011] The fused category-related features are input into the trained category-related object detection head to obtain the detection results of objects of known categories. The fused category-independent features are input into the trained category-independent object detection head to obtain the detection results of foreground objects. The foreground objects include objects of known categories and objects of unknown categories.

[0012] Based on the foreground object detection results and the known category object detection results, the detection results of unknown category objects in the target image are obtained.

[0013] Optionally, generating pseudo-label information for unknown category objects in each image sample of the training sample set based on the constructed semantic image segmentation model specifically includes:

[0014] The image sample is input into the semantic image segmentation model, so that the semantic segmentation model generates a background mask for the image sample, and the general image segmentation model generates all binary masks for the image sample.

[0015] Based on the background mask and all the binary masks, determine and delete the binary masks belonging to the background to obtain the remaining binary masks.

[0016] Generate the bounding box of the remaining binary mask.

[0017] The bounding boxes whose intersection-union ratio (IU) with the label information of known category objects is less than a first IU threshold are retained and used as pseudo-label information for the unknown category objects.

[0018] Optionally, the semantic segmentation model is the Mask2former model.

[0019] Optionally, the general image segmentation model is the SAM model.

[0020] Optionally, based on the pseudo-label information of unknown category objects and the label information of known category objects for each image sample in the training sample set, the category-related target detection head and the category-independent target detection head are trained respectively to obtain trained category-related target detection heads and trained category-independent target detection heads, specifically including:

[0021] The ATSS method is used to generate the training objective for the classification task of the category-related object detection head based on the label information of the known categories of objects. Training objectives for regression tasks After iterative training reaches the maximum number of iterations, a well-trained class-related object detection head is obtained.

[0022] The ATSS method is used to generate the training objective for the regression task of the class-independent object detection head based on the label information of the known object categories. The training objective for the classification task of the category-independent object detection head is generated based on the label information of the known category objects and the pseudo-label information of the unknown category objects. After iterative training reaches the maximum number of iterations, a well-trained class-independent object detection head is obtained.

[0023] Optionally, all class-related features and all class-independent features are extracted from the target image to be detected, and feature fusion is performed on all class-related features and all class-independent features to obtain fused class-related features and fused class-independent features, specifically including:

[0024] The target image to be detected is input into the feature extractors of the category-related detection branch and the category-independent detection branch, respectively, for feature extraction. This yields all category-related features and all category-independent features of the target image, expressed by the following formula:

[0025] f cs =Extractor CS (Image);

[0026] f ca =Extractor CA (Image);

[0027] Among them, f cs f represents category-related features. ca Extractor represents category-independent features. CS The feature extractor representing the category-related detection branch. CA The feature extractor represents the category-independent detection branch, and Image represents the target image to be detected.

[0028] The Fusion module is used to concatenate all class-related features and all class-independent features separately to obtain the concatenated features, expressed by the formula:

[0029] feature = Concat(f cs f ca );

[0030] Here, feature represents the concatenated feature, and Concat(·) represents the concatenation operation.

[0031] The global and spatial information of the splicing features are calculated separately, and the formula is expressed as follows:

[0032]

[0033]

[0034]

[0035]

[0036] in, This represents global information about the category-related feature portion of the concatenated features. This represents the spatial information of the category-related feature portion in the spliced ​​features. This represents global information about the category-independent feature portion of the concatenated features. This represents the spatial information of the category-independent feature portion in the concatenated features. GAP represents global average pooling, FC represents a fully connected layer, ReLU and Sigmoid are both activation functions, and Conv represents a convolutional layer.

[0037] Based on the global information and the spatial information, the fused category-related features and the fused category-independent features are obtained, expressed by the following formula:

[0038]

[0039] out cs =conv(f′) cs );

[0040]

[0041] out ca =conv(f′) ca );

[0042] Where, f′ cs f′ represents the intermediate category-related features. ca Indicates intermediate class-independent features, out csRepresents the fused category-related features, out ca This represents the category-independent features after fusion.

[0043] Optionally, the feature extractors for both the category-related detection branch and the category-independent detection branch are ResNet50 backbone network models.

[0044] Optionally, based on the foreground object detection results and the known category object detection results, the detection results of unknown category objects in the target image are obtained, specifically including:

[0045] Calculate the cross-union ratio (CUP) between each foreground bounding box in the foreground object detection results and each known category bounding box in the known category object detection results.

[0046] When the cross-union ratio is less than the second cross-union ratio threshold, the foreground object corresponding to the foreground detection box is determined to be an unknown category object; otherwise, the foreground object corresponding to the foreground detection box is a known category object.

[0047] Optionally, the second crossover ratio threshold is 0.6.

[0048] On the other hand, the present invention also proposes an unknown category target detection system, comprising:

[0049] The training sample set acquisition module is used to acquire a training sample set, which includes several image samples. Each image sample includes several unknown category objects and several known category objects. The known category objects are labeled with label information indicating the object category.

[0050] The pseudo-label information generation module is used to generate pseudo-label information for unknown category objects in each image sample in the training sample set based on the constructed semantic image segmentation model; the semantic image segmentation model includes a semantic segmentation model and a general image segmentation model.

[0051] The detection head training module is used to train the class-related target detection head and the class-independent target detection head respectively, based on the pseudo-label information of unknown class objects and the label information of known class objects in each image sample in the training sample set, to obtain the trained class-related target detection head and the trained class-independent target detection head.

[0052] The feature extraction and fusion module is used to acquire the target image to be detected, extract all class-related features and all class-independent features from the target image to be detected, and perform feature fusion on all class-related features and all class-independent features to obtain fused class-related features and fused class-independent features.

[0053] The module for obtaining detection results of known category objects and foreground objects is used to input the fused category-related features into the trained category-related target detection head to obtain the detection results of known category objects, and to input the fused category-independent features into the trained category-independent target detection head to obtain the detection results of foreground objects; the foreground objects include known category objects and unknown category objects.

[0054] The unknown category object detection result acquisition module is used to obtain the detection result of unknown category objects in the target image based on the foreground object detection result and the known category object detection result.

[0055] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0056] This invention provides a method and system for detecting objects of unknown categories. First, a training sample set is acquired, comprising several image samples. Each image sample includes several objects of unknown categories and several objects of known categories, with the known category objects labeled with tags indicating their category. Then, based on a constructed semantic image segmentation model, pseudo-label information is generated for the unknown category objects in each image sample of the training sample set. The semantic image segmentation model includes a semantic segmentation model and a general image segmentation model. Next, a category-related object detection head and a category-independent object detection head are used. Based on the pseudo-label information of the unknown category objects and the label information of the known category objects in each image sample of the training sample set, the category-related object detection head and the category-independent object detection head are trained respectively to obtain trained categories. The system employs a relevant object detection head and a trained class-independent object detection head. It then acquires the target image to be detected and extracts all class-related and class-independent features from the target image. These features are then fused to obtain fused class-related and class-independent features. The fused class-related features are then input into the trained class-related object detection head to obtain the detection results for objects of known categories. The fused class-independent features are then input into the trained class-independent object detection head to obtain the detection results for foreground objects, which include both known and unknown categories. Finally, based on the foreground and known category detection results, the detection results for unknown categories of objects in the target image are obtained.

[0057] As can be seen, this invention combines a semantic segmentation model with a general image segmentation model. It not only leverages the semantic segmentation model's ability to identify common backgrounds, compensating for the lack of effective background information in object detection, but also utilizes the general image segmentation model to generate pseudo-labels for potential unlabeled objects in the data, training a more powerful category-independent object detector. This enhances the ability to detect and identify objects of unknown categories, thereby improving the detection accuracy of unknown category objects. Furthermore, this invention uses category-related object detection heads and category-independent object detection heads separately to obtain detection results for known category objects and foreground objects. Since foreground objects only include known category objects and unknown category objects, this invention can accurately and reliably detect unknown category objects by utilizing the difference between category-related and category-independent object detection, effectively achieving precise detection of unknown category objects. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of an unknown category target detection method provided in Embodiment 1 of the present invention;

[0060] Figure 2 This is a schematic diagram showing that there are unlabeled objects in each sample image of the training sample set in the method provided in Embodiment 1 of the present invention.

[0061] Figure 3 This is a schematic diagram of the unknown category target detection framework based on image segmentation and foreground detection in the method provided in Embodiment 1 of the present invention;

[0062] Figure 4 This is a schematic diagram of the Fusion module in the method provided in Embodiment 1 of the present invention;

[0063] Figure 5 This is a schematic diagram illustrating the principle of identifying objects of unknown category based on the difference between category-related target detection and category-independent target detection in the method provided in Embodiment 1 of the present invention.

[0064] Figure 6 This is a structural block diagram of an unknown category target detection system provided in Embodiment 2 of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Currently, object detection methods can be categorized into single-stage object detectors based on convolutional neural networks, two-stage object detectors, and object detectors based on attention networks. Each type of method divides the object detection task into classification and regression subtasks. However, existing object detectors are trained on predefined categories and can only detect objects of those categories during the inference phase. They cannot handle objects of unknown categories, which limits their applicability in the real world.

[0067] Therefore, open-world object detection has been proposed to address the challenge of detecting unknown objects in object detection. Existing open-world object detection methods can be divided into two categories: one uses an energy function to distinguish between known and unknown categories, but requires additional data on unknown categories to calculate the energy distribution and to differentiate the energy distributions of known and unknown categories during training. The other category employs an attention-based pseudo-labeling strategy, utilizing localization quality scores such as intersection-over-union ratio (IoU) and centrality to detect unknown objects. However, despite these methods, current open-world object detection methods still cannot effectively detect objects of unknown categories, and their detection accuracy is not high.

[0068] Furthermore, image segmentation is a fundamental and crucial task in computer vision. Its main goal is to subdivide an image into different regions or objects to identify the boundaries and locations of objects within the image. This task has vital applications in multiple fields, including medical image processing, autonomous driving, satellite image analysis, and image editing. Traditional segmentation methods typically employ techniques such as edge detection, thresholding, and region growing to divide an image into different regions based on features like color, brightness, and texture. Modern deep learning-based segmentation methods, however, achieve more precise segmentation by learning feature representations from the pixel level to the object level. Semantic segmentation, in particular, can not only distinguish different objects in an image but also assign a corresponding category label to each pixel.

[0069] Existing open-world object detection methods employ contrastive clustering to separate categories in the feature space. These methods aim to ensure that instances of the same category are close to each other in the feature space, while instances of different categories are kept far apart. During training, these methods achieve category separation in the feature space by minimizing contrastive loss, thereby effectively distinguishing different categories. For example, for each category, a fixed-size feature queue is typically maintained to store the latest feature vectors, and then prototype vectors for each category are calculated as the distinguishing criterion.

[0070] Furthermore, current open-world object detection methods also employ energy distributions or probability distributions to distinguish between known and unknown categories. Regarding energy distributions, additional datasets of unknown categories are often needed to generate energy distributions for these categories, and the unknown categories are detected by separating the energy distributions of known and unknown categories. For probability distributions, localization quality scores such as centrality and intersection-union ratio (IUU) are typically used to assist in detecting unknown categories. Additionally, since datasets often contain unlabeled objects, some open-world object detection methods attempt to generate pseudo-labels for unlabeled objects using selective search or candidate region networks during training, and incorporate these pseudo-labels as part of the unknown category into the training process.

[0071] However, energy-based unknown category recognizers rely on additional unknown category datasets to establish the energy distribution of unknown categories. This means that information about unknown categories is introduced during the training phase, potentially leading to data leakage. On the other hand, the feature prototypes required for contrastive clustering necessitate maintaining a feature queue. As the number of categories increases, the space occupied by the feature queue also increases significantly, making it difficult to deploy on embedded devices. Furthermore, during the training phase, using candidate region networks to generate pseudo-labels for unknown category objects introduces noise. In the early stages of training, the predicted bounding boxes of the candidate region network may not be accurate enough. If candidate regions with the highest scores but not overlapping with the original annotations are directly selected as pseudo-labels for unknown targets, these inaccurate pseudo-labels may harm the model's regression performance. This not only reduces the accurate detection capability for unknown categories but also affects the detection performance of known categories. The training phase does not supervise the localization quality scores such as centrality and intersection-over-union ratio corresponding to the background, resulting in the localization quality scores being unable to effectively distinguish between foreground and background. Therefore, methods that use localization quality scores to assist in detecting unknown category objects are also ineffective in detecting unknown category objects.

[0072] In summary, the purpose of this invention is to provide a method and system for detecting unknown category objects. It combines semantic segmentation technology with general image segmentation technology to form a semantic image segmentation model for generating pseudo-label information for unknown category objects. Simultaneously, it utilizes the difference between category-related object detection and category-independent object detection to detect unknown category objects. In terms of model structure, the category-related object detection head and the category-independent object detection head are decoupled. While maintaining high-quality detection capabilities for known category objects, it effectively detects objects of unknown category. Better results are achieved by decoupling category-related and category-independent object detection. Finally, based on this, a framework and detection process for detecting unknown category objects combining image segmentation technology (including semantic segmentation and general image segmentation technology) and foreground detection technology are proposed. This effectively improves the detection accuracy of unknown category objects, enabling more accurate and reliable identification of known and unknown category objects from any image.

[0073] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] Example 1:

[0075] This embodiment provides a method for detecting targets of unknown categories, such as Figure 1 As shown, the method includes the following steps:

[0076] Step S1: Obtain a training sample set, which includes several image samples. Each image sample includes several unknown category objects and several known category objects. The known category objects are labeled with label information indicating the object category.

[0077] Step S2: Based on the constructed semantic image segmentation model, generate pseudo-label information for unknown category objects in each image sample in the training sample set; the semantic image segmentation model includes a semantic segmentation model and a general image segmentation model.

[0078] Step S3: Using a category-related object detection head and a category-independent object detection head, the category-related object detection head and the category-independent object detection head are trained respectively based on the pseudo-label information of unknown category objects and the label information of known category objects in each image sample in the training sample set, to obtain trained category-related object detection heads and trained category-independent object detection heads.

[0079] Step S4: Obtain the target image to be detected, and extract all category-related features and all category-independent features from the target image to be detected, and perform feature fusion on all category-related features and all category-independent features to obtain fused category-related features and fused category-independent features.

[0080] Step S5: Input the fused category-related features into the trained category-related target detection head to obtain the detection results of known category objects; input the fused category-independent features into the trained category-independent target detection head to obtain the detection results of foreground objects; the foreground objects include known category objects and unknown category objects.

[0081] Step S6: Based on the foreground object detection results and the known category object detection results, obtain the detection results of unknown category objects in the target image.

[0082] In this embodiment, the training sample set obtained in step S1 is the image dataset used to train the class-related object detection head and the class-independent object detection head, such as... Figure 2 As shown, each image sample (foreground object) includes several objects of unknown category and several objects of known category. Objects of known category are labeled with information indicating their category; that is, the category of the object is known. For example, Figure 2 Within the detection box, the "airplane" label indicates that the object is an airplane, the "motorcycle" label indicates that the object is a motorcycle, and the "person" label indicates that the object is a person. The focus of this invention is on the detection of objects of unknown categories, that is, detecting and identifying objects in an image that are not labeled and whose category is unknown, such as… Figure 2 The "unlabeled" label inside the detection box indicates that the object has no label information, therefore, the object belongs to the unknown category.

[0083] Currently, with the continuous evolution of neural network architectures and the expansion of training data scale, image segmentation techniques using large models have demonstrated superior performance in segmentation tasks. Semantic segmentation datasets (such as ADE20K, COCO stuff, etc.) typically contain annotations of common background categories, such as sky, roads, mountains, floors, buildings, etc. Semantic segmentation can solve the problem of lacking common background information in object detection tasks. Recently, through continuous annotation and iterative training of large-scale data, a new model called the "Segment Everything Model" (SAM model) has been developed. This model can generate high-precision binary masks, but the SAM model performs general image segmentation and does not involve category prediction.

[0084] Therefore, this invention combines semantic segmentation with general image segmentation, that is, it combines the semantic segmentation model with the general image segmentation model to obtain a semantic image segmentation model. This not only utilizes the ability of the semantic segmentation model to identify common backgrounds, thus compensating for the lack of effective background information in object detection, but also uses the general image segmentation model to generate pseudo-labels for potential unlabeled objects in the data, thereby training a more powerful category-independent object detector, thereby enhancing the ability to detect and identify objects of unknown categories, and thus improving the detection accuracy of objects of unknown categories.

[0085] In this embodiment, the semantic segmentation model is preferably the Mask2former model, and the general image segmentation model is preferably the SAM model.

[0086] In this embodiment, step S2 generates pseudo-label information for unknown category objects in each image sample of the training sample set based on the constructed semantic image segmentation model, specifically including:

[0087] Step S21: Input the image sample into the semantic image segmentation model so that the semantic segmentation model generates the background mask of the image sample, and so that the general image segmentation model generates all binary masks of the image sample.

[0088] Assuming the input image sample is "Image", and the label information of the known categories of labeled objects is "GTs", such as... Figure 3 As shown, the semantic segmentation model is Model sem_seg Generated background mask bg General Image Segmentation Model gen_seg All generated binary masks (Bin_Masks). Here, the semantic segmentation model (Model) is used. sem_seg The Mask2former (Masked-attention Mask Transformer) model is used, which is a semantic segmentation model and was pre-trained on the ADE20K dataset. The general image segmentation model... gen_seg The model used is SAM (Segment Anything Model), which is a general image segmentation model that was pre-trained on the SA-1B dataset.

[0089] Step S22: Based on the background mask and all the binary masks, determine and delete the binary masks belonging to the background to obtain the remaining binary masks, expressed by the formula:

[0090] Masks = {mask|mask&mask} bg =mask,mask∈Bin_Masks} (1);

[0091] Among them, mask bg Bin represents the background mask, Bin_Masks represents the set of binary masks, mask represents a binary mask in the set of binary masks, and Masks represents the set of binary masks remaining after removing the binary masks that belong to the background.

[0092] Step S23: Generate the bounding box of the remaining binary mask, represented as:

[0093] Boxes={bounding_box(mask)|mask∈Masks} (2);

[0094] Where Boxes represents the set of bounding boxes of the remaining binary mask, and bounding_box(.) represents the function that calculates the bounding boxes of the regions segmented from the binary mask, expressed as:

[0095] bounding_box(mask)=[xmin(mask), ymin(mask), xmax(mask), ymax(mask)](3);

[0096] Where xmin(mask), ymin(mask), xmax(mask), and ymax(mask) represent the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the positions with a value of 1 in the binary mask, respectively.

[0097] Step S24: Retain the bounding boxes whose intersection-union ratio (IU) with the label information of known category objects is less than the first IU threshold, and use them as pseudo-label information for the unknown category objects. The formula is expressed as:

[0098] Pseudo_Boxes={box|max{IoU(box,gt)|gt∈GTs}<δ, box∈Boxes} (4);

[0099] Wherein, Pseudo_Boxes represents the pseudo-label information of objects of unknown category, box represents one of the bounding boxes in the set of bounding boxes of the remaining binary mask, δ represents the first intersection-union ratio threshold, and this embodiment found that the effect is best when the intersection-union ratio is less than 0.3 after multiple experiments, so δ = 0.3 is set, and gt represents the label of one of the objects of known category.

[0100] In this embodiment, step S3 trains the category-related target detection head (also called a detector, hereinafter collectively referred to as the detection head) and the category-independent target detection head respectively based on the pseudo-label information of unknown category objects and the label information of known category objects for each image sample in the training sample set, to obtain the trained category-related target detection head and the trained category-independent target detection head, specifically including:

[0101] Step S31: Using the ATSS method, generate the training objective for the classification task of the category-related target detection head based on the label information of the known category objects. Training objectives for regression tasks After iterative training reaches the maximum number of iterations, a well-trained class-related object detection head is obtained.

[0102] During the training phase, the class-related object detection head CSDH is trained using the known object label information (GTs). The CSDH head has the same structure as the TOOD (Task-aligned One-stage Object Detection) head, both using six convolutional layers to extract features. Then, the features from these six convolutional layers are combined, and a separate convolutional layer is used to obtain classification scores. cs and bounding box prediction results cs During the training phase, the ATSS (Adaptive Training Sample Selection) algorithm is used to generate training objectives for classification and regression tasks based on the known object label information (GTs) of objects of known categories. Training objectives for regression tasks The loss function of the CSDH class-related object detection head is Loss. CS Loss CS It consists of the classification loss function focal loss and the regression loss function gIoU loss, and the formula is expressed as:

[0103]

[0104] Step S32: Using the ATSS method, generate the training objective for the regression task of the category-independent object detection head based on the label information of the known category objects. The training objective for the classification task of the category-independent object detection head is generated based on the label information of the known category objects and the pseudo-label information of the unknown category objects. After iterative training reaches the maximum number of iterations, a well-trained class-independent object detection head is obtained.

[0105] Simultaneously with step S31, the category-independent object detection head CADH is trained using the known category object label information GTs and the unlabeled object pseudo-label information Pseudo_Boxes generated in step S2. The structure of the category-independent object detection head CADH is the same as that of the TOOD detection head and the category-related object detection head CSDH, both using 6 convolutional layers to extract features. Then, the features from these 6 convolutional layers are combined, and a classification score is obtained using a separate convolutional layer for each feature. ca and bounding box prediction results ca Because regression tasks are more susceptible to inaccurate pseudo-labels, leading to inaccurate detection boxes, the ATSS algorithm is used during the training phase to generate the training objectives for the regression task based solely on the known category label information (GTs) of objects. Because classification tasks are less affected by the quality of pseudo-labels, ATSS is used during the training phase to generate training objectives for the classification task based on GTs∪Pseudo_Boxes. Loss function of class-independent object detection head CADH CA and Loss CS Similarly, it also consists of the classification loss function focal loss and the regression loss function gIoU loss, expressed by the following formula:

[0106]

[0107] In this embodiment, step S4 extracts all class-related features and all class-independent features from the target image to be detected, and performs feature fusion on all class-related features and all class-independent features to obtain fused class-related features and fused class-independent features, specifically including:

[0108] Step S41: Input the target image to be detected into the feature extractor of the category-related detection branch and the feature extractor of the category-independent detection branch respectively for feature extraction, to obtain all category-related features and all category-independent features of the target image to be detected, expressed by the formula:

[0109] f cs =Extractor CS (Image) (7);

[0110] f ca =Extractor CA (Image) (8);

[0111] Among them, f cs f represents category-related features. caExtractor represents category-independent features. CS The feature extractor representing the category-related detection branch. CA The feature extractor represents the category-independent detection branch, and Image represents the target image to be detected.

[0112] In this embodiment, the feature extractors for both the category-related detection branch and the category-independent detection branch are preferably backbone networks in deep learning. Different backbone networks can be used as needed, or the same backbone network can be used. For example, the feature extractors for both the category-related detection branch and the category-independent detection branch are ResNet50 (Deep residual network 50) backbone network models. The ResNet50 model is an image classification model and is commonly used as a feature extractor for object detection models.

[0113] Step S42: Use the Fusion fusion module to concatenate all class-related features and all class-independent features to obtain the concatenated features, expressed by the formula:

[0114] feature = Concat(f cs f ca (9);

[0115] Where feature represents the concatenated feature, and category-related feature f cs Category-independent features f ca The shapes of all are (B, C, H, W), therefore, the shape of the splicing feature is (B, 2C, H, W), and Concat(·) represents the splicing operation.

[0116] This embodiment can use any feature fusion module for feature fusion, but the Fusion fusion module is preferred. The structure of the Fusion fusion module is as follows: Figure 4 As shown, it should be noted that since the Fusion fusion module is a commonly used fusion module in this field, and the Fusion fusion module is not the focus of this invention, its structure will not be described in detail here.

[0117] Step S43: Calculate the global information and spatial information of the splicing feature respectively, expressed by the formula:

[0118]

[0119]

[0120]

[0121]

[0122] in, This represents global information about the category-related feature portion of the concatenated features. This represents the spatial information of the category-related feature portion in the spliced ​​features. This represents global information about the category-independent feature portion of the concatenated features. This represents the spatial information of the class-independent feature portion in the concatenated features. GAP (Global Average Pooling) represents global average pooling, FC represents a fully connected layer, ReLU and Sigmoid are both activation functions, and Conv represents a convolutional layer.

[0123] Step S44: Based on the global information and the spatial information, obtain the fused category-related features and the fused category-independent features, expressed by the formula:

[0124]

[0125] out cs =conv(f′) cs (15);

[0126]

[0127] out ca =conv(f′) ca (17);

[0128] Where, f′ cs f′ represents the intermediate category-related features. ca Indicates intermediate class-independent features, out cs Represents the fused category-related features, out ca This represents the category-independent features after fusion.

[0129] Due to the newly formed category-related feature f′ cs Category-independent feature f′ ca The shapes are all (B, 2C, H, W), and another convolutional layer is needed to combine the intermediate class-related features f′. cs Category-independent feature f′ ca The fused category-related features are transformed into shapes (B, C, H, W) respectively. cs out features unrelated to the category after fusion ca .

[0130] Figure 5 This diagram illustrates the principle of identifying objects of unknown categories based on the differences between category-related target detection and category-independent target detection in Embodiment 1. In step S4, the fused category-related features (out) are obtained.cs out features unrelated to the category after fusion ca Next, step S5 outputs the fused category-related features. cs The data is input into the class-related object detection head CSDH trained in step S3. After non-maximum suppression (NMS), the detection results of objects of known classes can be obtained, as expressed by the formula:

[0131] cs_results = NMS(CSDH(out cs )) (18);

[0132] Here, cs_results represents the detection results of objects of known categories, and NMS(·) represents the non-maximum suppression operation. Non-maximum suppression is abbreviated as NMS, and its function is to remove redundant detection boxes. The redundancy removal method is to remove detection box results that overlap with maxima.

[0133] At the same time, the fused class-independent features are output. ca The data is input into the class-independent object detection head CADH trained in step S3 to obtain the foreground object detection results, expressed by the formula:

[0134] ca_results = NMS(CADH(out ca )) (19);

[0135] Here, ca_results represents the foreground object detection results, which is essentially a set of foreground detection boxes. In other words, the foreground object detection results actually include the detection boxes of multiple foreground objects. The detection boxes enclose all foreground objects in the image. These foreground objects include not only objects of unknown categories but also objects of known categories.

[0136] Since the foreground objects in the detection results include objects of known categories and objects of unknown categories, step 6 is essentially to delete a portion of the foreground objects that have a high degree of overlap with the known category objects, and the remaining portion of objects can be considered as objects of unknown categories.

[0137] Therefore, in this embodiment, step S6, based on the foreground object detection result and the known category object detection result, obtains the detection result of the unknown category object in the target image, specifically including:

[0138] Step S61: Calculate the intersection-union ratio (IUU) between each foreground detection box in the foreground object detection result and each known category detection box in the known category object detection result.

[0139] Step S62: When the cross-union ratio is less than the second cross-union ratio threshold, determine that the foreground object corresponding to the foreground detection box is an unknown category object; otherwise, the foreground object corresponding to the foreground detection box is a known category object, i.e., step S63.

[0140] Step S63: When the cross-union ratio is greater than or equal to the second cross-union ratio threshold, the foreground object corresponding to the foreground detection box is determined to be a known category object.

[0141] like Figure 5 As shown, objects with a higher intersection-union (UC) between the foreground object detection results ca_results and the known category object detection results cs_results are considered as known category objects. The remaining objects are considered as unknown category objects. The formula is as follows:

[0142] unknown_results={ca_box|max{IoU(ca_box, cs_box)|cs_box∈cs_results}<σ, ca_box∈ca_results} (20);

[0143] Wherein, unknown_results represents the detection results of objects of unknown category, ca_box represents a foreground detection box in the foreground object detection box set of the foreground object detection results ca_results, cs_box represents a known category detection box in the known category detection box set of the known category object detection results cs_results, max{IoU(ca_box, cs_box)|cs_box∈cs_results} represents the maximum value of the IoU between the foreground detection box ca_box and all known category detection boxes in the known category object detection results cs_results, and σ represents the second intersection-union threshold. After multiple experiments, it was found that the second intersection-union threshold of 0.6 has the best effect. Therefore, in this embodiment, σ = 0.6 is set.

[0144] This embodiment utilizes a category-related object detection head and a category-independent object detection head to perform detection, obtaining detection results for known category objects and foreground objects. Since foreground objects only include known category objects and unknown category objects, this embodiment uses the principle of detecting unknown category objects based on the difference between category-related object detection and category-independent object detection. That is, the detection results for known category objects obtained by category-related object detection and the detection results for foreground objects obtained by category-independent object detection differ only in the detection results for unknown category objects. By combining the method of comparing the cross-union ratio (CUI) and the CUI threshold between detection boxes, it is determined whether the foreground object corresponding to the foreground detection box is a known category object or an unknown category object. This allows for the rapid, accurate, and reliable detection and identification of unknown category objects, effectively achieving accurate detection of unknown category objects.

[0145] Example 2

[0146] This embodiment proposes an unknown category target detection system, which corresponds to the unknown category target detection method in Embodiment 1, such as... Figure 6 As shown, the system includes:

[0147] Training sample set acquisition module 1 is used to acquire a training sample set, which includes several image samples. Each image sample includes several unknown category objects and several known category objects. The known category objects are labeled with label information indicating the object category.

[0148] The pseudo-label information generation module 2 is used to generate pseudo-label information for unknown category objects in each image sample in the training sample set according to the constructed semantic image segmentation model; the semantic image segmentation model includes a semantic segmentation model and a general image segmentation model;

[0149] The detection head training module 3 is used to train the class-related target detection head and the class-independent target detection head respectively, based on the pseudo-label information of unknown class objects and the label information of known class objects in each image sample in the training sample set, to obtain the trained class-related target detection head and the trained class-independent target detection head.

[0150] The feature extraction and fusion module 4 is used to acquire the target image to be detected, extract all class-related features and all class-independent features from the target image to be detected, and perform feature fusion on all class-related features and all class-independent features to obtain fused class-related features and fused class-independent features.

[0151] The module 5 for obtaining detection results of known category objects and foreground objects is used to input the fused category-related features into the trained category-related target detection head to obtain the detection results of known category objects, and to input the fused category-independent features into the trained category-independent target detection head to obtain the detection results of foreground objects; the foreground objects include known category objects and unknown category objects;

[0152] The unknown category object detection result acquisition module 6 is used to obtain the detection result of unknown category objects in the target image based on the foreground object detection result and the known category object detection result.

[0153] Specific examples are used in this article, but the above description is only to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. Those skilled in the art should understand that the various modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, and thus, they can be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any specific combination of hardware and software.

[0154] Furthermore, those skilled in the art will recognize that, based on the principles of this invention, there will be variations in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as limiting the invention.

Claims

1. A method for detecting targets of unknown category, characterized in that, include: Obtain a training sample set, which includes several image samples. Each image sample includes several objects of unknown category and several objects of known category. The objects of known category are labeled with label information indicating the object category. Based on the constructed semantic image segmentation model, pseudo-label information is generated for unknown category objects in each image sample in the training sample set; the semantic image segmentation model includes a semantic segmentation model and a general image segmentation model; A class-related object detection head and a class-independent object detection head are used. Based on the pseudo-label information of unknown class objects and the label information of known class objects in each image sample in the training sample set, the class-related object detection head and the class-independent object detection head are trained respectively to obtain trained class-related object detection heads and trained class-independent object detection heads. Acquire the target image to be detected, and extract all class-related features and all class-independent features from the target image to be detected. Then, perform feature fusion on all class-related features and all class-independent features to obtain fused class-related features and fused class-independent features. The fused category-related features are input into the trained category-related object detection head to obtain the detection results of objects of known categories. The fused category-independent features are input into the trained category-independent object detection head to obtain the detection results of foreground objects. Foreground objects include objects of known categories and objects of unknown categories; Based on the foreground object detection results and the known category object detection results, the detection results of unknown category objects in the target image are obtained.

2. The method for detecting unknown category targets according to claim 1, characterized in that, The step of generating pseudo-label information for unknown category objects in each image sample in the training sample set based on the constructed semantic image segmentation model specifically includes: The image sample is input into the semantic image segmentation model, so that the semantic segmentation model generates the background mask of the image sample, and the general image segmentation model generates all binary masks of the image sample. Based on the background mask and all the binary masks, determine and delete the binary masks belonging to the background to obtain the remaining binary masks; Generate the bounding box of the remaining binary mask; The bounding boxes whose intersection-union ratio (IU) with the label information of known category objects is less than a first IU threshold are retained and used as pseudo-label information for the unknown category objects.

3. A method for detecting unknown category targets according to claim 1 or 2, characterized in that, The semantic segmentation model is the Mask2former model.

4. A method for detecting unknown category targets according to claim 1 or 2, characterized in that, The general image segmentation model is the SAM model.

5. The method for detecting unknown category targets according to claim 1, characterized in that, Based on the pseudo-label information of unknown category objects and the label information of known category objects for each image sample in the training sample set, the category-related object detection head and the category-independent object detection head are trained respectively to obtain trained category-related object detection heads and trained category-independent object detection heads, specifically including: The ATSS method is used to generate the training objective for the classification task of the category-related object detection head based on the label information of the known categories of objects. Training objectives for regression tasks After iterative training reaches the maximum number of iterations, a well-trained class-related object detection head is obtained. The ATSS method is used to generate the training objective for the regression task of the class-independent object detection head based on the label information of the known object categories. The training objective for the classification task of the category-independent object detection head is generated based on the label information of the known category objects and the pseudo-label information of the unknown category objects. After iterative training reaches the maximum number of iterations, a well-trained class-independent object detection head is obtained.

6. The method for detecting unknown category targets according to claim 1, characterized in that, All class-related features and all class-independent features are extracted from the target image to be detected, and feature fusion is performed on all class-related features and all class-independent features to obtain fused class-related features and fused class-independent features, specifically including: The target image to be detected is input into the feature extractors of the category-related detection branch and the category-independent detection branch, respectively, for feature extraction. This yields all category-related features and all category-independent features of the target image, expressed by the following formula: f cs =Extractor CS (Image); f ca =Extractor CA (Image); Among them, f cs f represents category-related features. ca Extractor represents category-independent features. CS The feature extractor representing the category-related detection branch. CA The feature extractor represents the category-independent detection branch, and Image represents the target image to be detected. The Fusion module is used to concatenate all class-related features and all class-independent features separately to obtain the concatenated features, expressed by the formula: feature=Concat(f cs ,f ca ); Where feature represents the concatenated feature, and Concat(·) represents the concatenation operation; The global and spatial information of the splicing features are calculated separately, and the formula is expressed as follows: in, This represents global information about the category-related feature portion of the concatenated features. This represents the spatial information of the category-related feature portion in the spliced ​​features. This represents global information about the category-independent feature portion of the concatenated features. This represents the spatial information of the class-independent feature part in the concatenated feature set, GAP represents global average pooling, FC represents a fully connected layer, ReLU and Sigmoid are both activation functions, and Conv represents a convolutional layer. Based on the global information and the spatial information, the fused category-related features and the fused category-independent features are obtained, expressed by the following formula: out cs =conv(f′ cs ); out ca =conv(f′ ca ); Where, f′ cs f′ represents the intermediate category-related features. ca Indicates intermediate class-independent features, out cs Represents the fused category-related features, out ca This represents the category-independent features after fusion.

7. The method for detecting unknown category targets according to claim 6, characterized in that, The feature extractors for both the category-related detection branch and the category-independent detection branch are ResNet50 backbone network models.

8. The method for detecting unknown category targets according to claim 1, characterized in that, Based on the foreground object detection results and the known category object detection results, the detection results of unknown category objects in the target image are obtained, specifically including: Calculate the intersection-union ratio (IUU) between each foreground bounding box in the foreground object detection results and each known category bounding box in the known category object detection results; When the cross-union ratio is less than the second cross-union ratio threshold, the foreground object corresponding to the foreground detection box is determined to be an unknown category object; otherwise, the foreground object corresponding to the foreground detection box is a known category object.

9. The method for detecting unknown category targets according to claim 8, characterized in that, The second crossover ratio threshold is 0.

6.

10. A target detection system for an unknown category, characterized in that, include: The training sample set acquisition module is used to acquire a training sample set, which includes several image samples. Each image sample includes several unknown category objects and several known category objects. The known category objects are labeled with label information indicating the object category. The pseudo-label information generation module is used to generate pseudo-label information for unknown category objects in each image sample in the training sample set based on the constructed semantic image segmentation model; the semantic image segmentation model includes a semantic segmentation model and a general image segmentation model; The detection head training module is used to train the class-related target detection head and the class-independent target detection head respectively, based on the pseudo-label information of unknown class objects and the label information of known class objects in each image sample in the training sample set, to obtain the trained class-related target detection head and the trained class-independent target detection head. The feature extraction and fusion module is used to acquire the target image to be detected, extract all class-related features and all class-independent features from the target image to be detected, and perform feature fusion on all class-related features and all class-independent features to obtain fused class-related features and fused class-independent features. The module for obtaining detection results of known category objects and foreground objects is used to input the fused category-related features into the trained category-related target detection head to obtain the detection results of known category objects, and to input the fused category-independent features into the trained category-independent target detection head to obtain the detection results of foreground objects; Foreground objects include objects of known categories and objects of unknown categories; The unknown category object detection result acquisition module is used to obtain the detection result of unknown category objects in the target image based on the foreground object detection result and the known category object detection result.

Images