Weakly supervised open vocabulary object detection method, system and device

Abstract

The application discloses a weakly supervised open vocabulary object detection method, system and device, and belongs to the field of computer vision detection, comprising: adaptively shielding local discriminative features by using a learnable feature mask module to suppress overfitting; constructing a visual prototype alignment multi-instance learning module, calculating instance-level classification scores based on visual prototypes generated by a diffusion model, and realizing pure visual semantic alignment; filtering noise and merging fragments by using an instance completion module to generate high-quality pseudo labels of complete targets; finally, constructing a cascade refinement branch of a target detection model for iterative optimization training, and using the trained and optimized target detection model for open vocabulary visual target detection. Through feature masking, visual prototype alignment and instance completion, the application solves the problems of local overfitting and pseudo label fragmentation, and significantly improves the positioning integrity and classification accuracy of visual target detection.

Description

Technical Field

[0001] This invention relates to a weakly supervised open vocabulary object detection method, system, and device, belonging to the field of computer vision and deep learning technology. Background Technology

[0002] Artificial intelligence (AI) technology is rapidly permeating various industries, showing great promise, particularly in sectors such as industrial manufacturing, smart transportation, healthcare, and education. As a crucial component of AI, computer vision, especially advanced learning and cognitive technologies such as visual object detection, has become a key driver of industrial intelligent upgrading.

[0003] While deep learning-based visual object detection models have made significant progress in accuracy, their heavy reliance on large-scale manually labeled data remains a major bottleneck. This labeling process is time-consuming, labor-intensive, and costly, directly limiting the efficient use of large-scale data and severely restricting the potential for expansion and application of related technologies in open scenarios.

[0004] To reduce reliance on costly manual annotation, researchers have proposed weakly supervised visual object detection techniques. However, existing weakly supervised and open-vocabulary detection methods still suffer from the following problems:

[0005] (1) Chinese patent application with publication number CN114743137A discloses an online exam cheating identification method based on multi-instance learning. Such multi-instance learning methods tend to focus on the most identifiable local area of ​​the target, such as only detecting the "head" and ignoring the "body", resulting in the generated detection box not being able to completely cover the target object and lacking effective use of contextual information.

[0006] (2) In open vocabulary scenarios, existing methods often directly utilize the text embedding of pre-trained models (such as CLIP) to match image regions, such as the domain generalization method and system based on semantic information extraction and target alignment of the CLIP model disclosed in Chinese Patent Publication No. CN119091214A. However, there is a significant distributional difference between the high abstractness of text features and the specific diversity of image region features. This intermodal misalignment directly leads to inherent local bias and cross-modal misalignment problems.

[0007] (3) In addition, in the early stages of iterative training, the pseudo-labels generated by the model are often accompanied by a lot of background noise, and the targets are fragmented. The lack of an effective pseudo-label correction and completion mechanism causes erroneous supervision signals to accumulate continuously during training, which limits the upper limit of the detector's performance. Summary of the Invention

[0008] The technical problem solved by this invention is to address the issues of incomplete target localization, large open-vocabulary classification bias, and low pseudo-label quality in existing weakly supervised visual target detection technologies. This invention provides a weakly supervised open-vocabulary object detection method, system, and device, aiming to generate high-quality, complete target pseudo-labels through adaptive masking at the feature level, visual prototype alignment, and instance completion correction at the label level, thereby significantly improving the localization completeness and classification accuracy of weakly supervised open-vocabulary target detection methods.

[0009] This invention is achieved using the following technical solution:

[0010] This invention first discloses a weakly supervised open-vocabulary object detection method, comprising the following steps:

[0011] S1. Obtain the input image and simultaneously extract candidate region proposals from the input image using a region proposal network and a pre-trained image segmentation model.

[0012] S2. Obtain the initial feature vector corresponding to each candidate region proposal through region of interest pooling operation, and suppress overfitting in the local discriminative region of the initial feature vector to obtain the overfitted feature vector of the candidate region proposal.

[0013] S3. Based on the pre-trained diffusion model, a category-specific visual prototype is generated for the target category to be detected, and the overfitted candidate region proposal feature vector is mapped to the unified feature space where the visual prototype is located, so that the feature vector is aligned with the visual prototype information, and a semantic alignment score is obtained. Based on the semantic alignment score, the instance-level classification score of each candidate region proposal for each target category is obtained.

[0014] S4. Obtain initial pseudo-labels based on the instance-level classification scores of each proposal and the corresponding target category. Filter the initial pseudo-labels corresponding to fragmented regional proposals in the candidate regional proposals, fuse and correct them, and generate a high-quality pseudo-label set.

[0015] S5. Construct a target detection model with cascaded refinement branches, perform iterative training and optimization using a high-quality pseudo-label set, and use the trained and optimized target detection model to perform visual target detection for open vocabulary.

[0016] In a weakly supervised open vocabulary object detection method of the present invention, further, in step S1, the multi-scale feature map of the input image is input into a region proposal network to extract and generate a first region proposal, the input image is input into a pre-trained image segmentation model to extract and generate a second region proposal, and the first region proposal and the second region proposal are merged to form a candidate region proposal set.

[0017] In a weakly supervised open vocabulary object detection method of the present invention, further, in step S2, a spatial probability map is generated based on the initial feature vector of the candidate region proposal, wherein the value of each spatial location represents the probability of occluding the feature at that location. A binary spatial mask is generated by differentiable sampling of the spatial probability map, and the binary spatial mask is applied to the initial feature vector of the corresponding candidate region proposal to occlude the local discriminative region features identified as overactive, thereby obtaining the feature vector of the candidate region proposal after suppressing overfitting.

[0018] In a weakly supervised open vocabulary object detection method of the present invention, the differentiable sampling is further implemented by the Gumbel-Sigmoid function, and a differentiable approximation of the binary space mask is generated based on the spatial probability map.

[0019] In a weakly supervised open vocabulary object detection method of the present invention, step S3 further includes the following sub-steps:

[0020] S31. For each detected target category, a representative image is generated based on a pre-trained text-to-image diffusion model, and a cross-attention mechanism is used to extract class-level visual prototypes focusing on the foreground target category from the generated representative images; S32. A semantic alignment score is obtained by using the semantic similarity between the feature vector after suppressing overfitting through candidate region proposals and the visual prototypes of each target category, and the detection score of the corresponding candidate region proposal is predicted through the detection branch of the multilayer perceptron; S33. For each candidate region proposal and each target category, the corresponding semantic alignment score is combined with the detection score to obtain an instance-level classification score indicating that the corresponding region proposal belongs to that target category; S34. Based on the instance-level classification scores of all region proposals, an image-level target category prediction is obtained by pooling the scores, and the target detection model is trained using image-level target category labels and multi-instance learning loss.

[0021] In a weakly supervised open-vocabulary object detection method of the present invention, further, in step S31, multiple representative images are generated for each target category using a pre-trained text-to-image diffusion model; for each generated representative image, a heatmap for locating the target category is generated by extracting the cross-attention information associated with the target category text generated by the text-to-image diffusion model during the denoising process; a segmentation mask of the target foreground object in the generated representative image is obtained based on the heatmap; using the segmentation mask, the feature representation of the foreground object is extracted from the depth features corresponding to the representative image as an object-level visual prototype; multiple object-level visual prototypes belonging to the same target category are aggregated to obtain the class-level visual prototype of the target category;

[0022] In a weakly supervised open vocabulary object detection method of the present invention, step S4 further includes the following sub-steps:

[0023] S41. Assign category labels and confidence scores to the corresponding candidate region proposals as an initial set of pseudo-labels based on the instance-level classification scores of each candidate region proposal.

[0024] S42. Set a confidence threshold, traverse the initial pseudo-label set, and remove candidate region proposals with confidence scores lower than the confidence threshold to obtain a set of retained region proposals.

[0025] S43. For fragmented region proposals that cover different parts of the same object in the reserved region proposal set, process them according to the target category. Cluster the reserved region proposals belonging to the same target category according to the degree of spatial overlap. If the intersection-union ratio of two reserved region proposals is greater than the preset overlap threshold, then the two reserved region proposals are assigned to the same cluster.

[0026] S44. Obtain the smallest bounding rectangle that can tightly enclose all reserved region proposals within each cluster, thereby merging the fragmented region proposals within that cluster into a complete target bounding box.

[0027] S45. The merged complete target bounding box and its corresponding target category average confidence score are used as high-quality pseudo-labels.

[0028] In a weakly supervised open-vocabulary object detection method of the present invention, further, in step S5, the first refinement branch of the object detection model receives the instance-level classification score obtained in step S3 as input, and each subsequent refinement branch receives the output of the previous refinement branch as input; at the beginning of each iteration of the object detection model, the high-quality pseudo-label set generated by the previous iteration of the object detection model is used as a supervision signal; the classification loss and regression loss of each refinement branch are obtained using the supervision signal, and the object detection model parameters are updated through backpropagation;

[0029] After each iteration of the target detection model, the high-confidence detection results output by the last refinement branch are selected as a new set of high-quality pseudo-labels for supervision of the next iteration of the target detection model. The above supervised training and high-quality pseudo-label dynamic update steps are repeated until the preset maximum number of iterations or the preset training termination condition is reached.

[0030] This invention also discloses a weakly supervised open-vocabulary object detection system, which includes the following components within the framework of an object detection model:

[0031] The region proposal extraction module uses a region proposal network and a pre-trained image segmentation model to simultaneously extract candidate region proposals from the input image.

[0032] The learnable feature mask module obtains the initial feature vectors corresponding to each candidate region proposal through region of interest pooling operation, and performs overfit suppression on the local discriminative regions of the initial feature vectors to obtain the overfit-suppressed feature vectors of the candidate region proposals.

[0033] The visual prototype alignment multi-instance learning module maps the overfitted feature vectors of candidate region proposals to a unified visual prototype space to generate category-specific visual prototypes, thereby aligning the feature vectors with the visual prototype information and obtaining a semantic alignment score. Based on the semantic alignment score, the instance-level classification score of each candidate region proposal for each target category is obtained.

[0034] The instance completion module obtains initial pseudo-labels based on the instance-level classification scores of each proposal and category. It then filters the initial pseudo-labels corresponding to fragmented region proposals in the candidate region proposals, merges and corrects them, and generates a high-quality pseudo-label set. The high-quality pseudo-label set is used for iterative training and optimization of the object detection model, and the trained and optimized object detection model is used for visual object detection of open vocabulary.

[0035] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the weakly supervised open vocabulary object detection method described above.

[0036] In summary, this invention provides a weakly supervised open-vocabulary object detection method, system, and device based on learnable feature masking and visual prototypes. First, a learnable feature masking module adaptively suppresses local overfitting during image feature extraction, forcing the model to extract complete target context information. Then, a visual prototype generated based on a text-to-image diffusion model is introduced. A purely visual semantic alignment mechanism is constructed through a visual prototype alignment multi-instance learning module, effectively avoiding the cross-modal semantic gap problem in traditional methods. Finally, an instance completion module denoises and spatially fuses fragmented pseudo-labels, and iteratively optimizes the object detection model using cascading branches. This invention works synergistically from three dimensions—feature enhancement, prototype guidance, and label completion—significantly solving the core problems of incomplete target localization, large open-vocabulary classification bias, and low-quality pseudo-labels in weakly supervised detection. It can generate high-quality detection boxes under weak supervision, greatly improving the accuracy and robustness of the object detection model, and has broad application prospects in industrial inspection, smart security, and other fields.

[0037] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0039] Figure 1 This is a flowchart illustrating the weakly supervised open vocabulary object detection method and system in an embodiment of the present invention.

[0040] Figure 2 This is an overall block diagram of the weakly supervised open vocabulary object detection method in an embodiment of the present invention.

[0041] Figure 3 This diagram illustrates the processing of the learnable feature mask module in a weakly supervised open-vocabulary object detection method.

[0042] Figure 4 This diagram illustrates the processing of the visual prototype alignment multi-instance learning module in a weakly supervised open-vocabulary object detection method.

[0043] Figure 5 This diagram illustrates the instance completion module processing in a weakly supervised open-vocabulary object detection method. Detailed Implementation

[0044] The following detailed description of the invention is provided with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings, not the entire structure.

[0045] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0046] Example

[0047] like Figure 1 and Figure 2 As shown, the weakly supervised open vocabulary object detection system of the present invention sets up a region proposal extraction module, a learnable feature mask module, a visual prototype alignment multi-instance learning module, and an instance completion module under the target detection model framework, and implements a weakly supervised open vocabulary object detection method with the following steps. This method is a specific implementation scheme of the weakly supervised open vocabulary object detection method of the present invention.

[0048] S1. Obtain the input image and simultaneously extract candidate region proposals from the input image using a region proposal network and a pre-trained image segmentation model. Specifically, the region proposal network extracts and generates a first region proposal from the multi-scale feature map of the input image, and the pre-trained image segmentation model extracts and generates a second region proposal from the input image. The first and second region proposals are then merged to form a candidate region proposal set.

[0049] This step is implemented by the region proposal extraction module, which is integrated into the object detection model framework. Internally, it integrates a Region Proposal Network (RPN) and a pre-trained image segmentation model (SAM). The specific method for generating region proposals is as follows: The input image is fed into the backbone network of the object detection model framework to extract multi-scale feature maps; the multi-scale feature maps are processed by the Region Proposal Network (RPN) to generate a first region proposal; the input image is fed into the pre-trained image segmentation model (SAM) to generate a second region proposal; the first and second region proposals are merged to form a candidate region proposal set. .

[0050] In this step, a standard Region Proposal Network (RPN) is used to generate the first region proposal. The RPN slides on the multi-scale feature map output by the backbone network of the object detection model framework. Based on multiple preset anchor boxes of different scales and aspect ratios, it predicts whether an object exists at each location and the coarse offset of the object's bounding box, thereby filtering out the set of first region proposals.

[0051] To capture regions with greater object integrity, this step also introduces a pre-trained image segmentation model (SAM) as an auxiliary proposal generator. Specifically, the SAM is used to perform full-image automatic segmentation pattern inference on the input image, generating a series of high-quality segmentation masks. Subsequently, the bounding box of each segmentation mask is calculated and used as a set of second region proposals. Because the SAM is trained on massive amounts of data, its generated proposals tend to better fit object boundaries, effectively supplementing object parts that the Region Proposal Network (RPN) might miss.

[0052] S2. Obtain the initial feature vector corresponding to each candidate region proposal through region of interest pooling operation, and suppress overfitting in the local discriminative region of the initial feature vector to obtain the feature vector of the candidate region proposal after overfitting suppression.

[0053] This step first involves the candidate region proposal set. Each regional proposal Perform Region of Interest (RoI Pooling) operation to extract the corresponding fixed-size initial feature vector from the multi-scale feature map output by the backbone network of the object detection model framework. Where C is the number of channels, and h and w are the height and width of the feature map, respectively.

[0054] This embodiment is implemented by a learnable feature mask module. The learnable feature mask module is inserted at the output of the pooling operation. Based on the initial feature vector of the candidate region proposal, a spatial probability map is learned and generated. The value of each spatial location represents the probability of masking the feature at that location. A binary spatial mask is generated by differentiable sampling of the spatial probability map. The binary spatial mask is applied to the initial feature vector of the corresponding candidate region proposal to mask the local discriminative region features identified as overactive, thereby weakening the over-enhancement of local discriminative region features. This forces the model to aggregate information from the rest of the target to obtain the feature vector of the candidate region proposal after suppressing overfitting.

[0055] Specifically, such as Figure 3 As shown, the learnable feature masking module utilizes a lightweight residual network module to generate the initial feature vector of the candidate region proposal. The data is processed to form a single-channel spatial probability map. , .in, This represents the Sigmoid activation function, used to map output values ​​to the [0,1] interval. Each value in this probability plot represents the probability that the corresponding spatial location feature is discarded.

[0056] It should be noted that, to achieve end-to-end training based on backpropagation, binary masks cannot be directly sampled from the Bernoulli distribution because this operation is non-differentiable. This step employs the Gumbel-Softmax relaxation technique to generate a differentiable approximation of the binary spatial mask, implemented using the Gumbel-Sigmoid function. The differentiable approximation of the binary spatial mask is generated based on the spatial probability map. The value of each spatial location in the binary spatial mask is obtained by adding uniformly distributed random noise to the log-probability value of the corresponding location in the spatial probability map, and then calculating under the control of the Sigmoid function and temperature parameters. To more effectively suppress continuous feature regions rather than isolated pixels, this invention expands the seed mask using a max-pooling operation, generating a block mask as the final binary spatial mask. This operation expands the originally isolated mask seed points into block regions covering the surrounding neighborhood. Applying the generated block mask to the original feature f yields a feature vector mask representation after overfitting suppression. .

[0057] Specifically, the value of the seed mask at spatial location (i,j) Where U is random noise sampled from a uniform distribution, U ~ Uniform(0,1); τ is a temperature parameter that controls the sharpness of the approximate distribution. When τ→0, the output approaches a discrete binary distribution. In a preferred setting of this embodiment, the temperature parameter τ is set to 0.3.

[0058] Seed mask is processed using max pooling operation. Expand to generate the final block mask. Where k represents the block size. In a preferred setting of this embodiment, k=3. This operation expands the originally isolated mask seed point into a block-shaped region covering the surrounding neighborhood.

[0059] The generated block mask is applied to the original feature f to obtain the feature vector mask representation after overfitting is suppressed. Where ⊙ represents element-wise multiplication broadcast along the channel dimension.

[0060] Through the above mechanism, the learnable feature mask module can randomly mask the most discriminative feature region based on the learned probability m during the training process, preventing the classifier from over-relying on a single local feature (such as a human head), thereby prompting the detection head to discover and aggregate the remaining parts of the target object (such as a human torso), learn a more robust and complete feature representation, and maintain consistency with the overall visual prototype generated in subsequent stages.

[0061] S3. Based on the pre-trained diffusion model, a category-specific visual prototype is generated for the target category to be detected. The feature vector of the candidate region proposal after suppressing overfitting is mapped to the unified feature space where the visual prototype is located, so that the feature vector and the visual prototype information are aligned and a semantic alignment score is obtained. Based on the semantic alignment score, the instance-level classification score of each candidate region proposal for each target category is obtained.

[0062] This step is implemented by the visual prototype alignment multi-instance learning module. The overfit-suppressed feature vector is input into the visual prototype alignment multi-instance learning module, which performs the following sub-steps:

[0063] S31. For each detected target category, a representative image is generated based on the pre-trained text-to-image diffusion model (StableDiffusion). The cross-attention mechanism of the text-to-image diffusion model U-Net is then used to extract the class-level visual prototypes focusing on the foreground target category from the generated representative images. The foreground target category is the target category remaining after removing background noise from the representative images generated by the diffusion model for each detected target category.

[0064] For each generated representative image, a heatmap for locating the target category is generated by extracting the cross-attention information associated with the target category text generated during the denoising process of the text-to-image diffusion model. Based on the heatmap, a segmentation mask for the target foreground object in the generated representative image is obtained. Using the segmentation mask, the feature representation of the foreground object is extracted from the depth features corresponding to the representative image as an object-level visual prototype. Multiple object-level visual prototypes belonging to the same target category are aggregated to obtain the class-level visual prototype of the target category.

[0065] S32. Obtain the semantic alignment score by using the semantic similarity between the feature vector after suppressing overfitting through candidate region proposals and the visual prototype of each target category. Predict the detection score of the corresponding region proposal through the detection branch of a multilayer perceptron. The prediction head of the multilayer perceptron directly receives the features of the candidate region and outputs a detection score matrix, which is used to determine whether the region is "accurately bound (localized)".

[0066] S33. For each candidate region proposal and each target category, combine the corresponding semantic alignment score with the detection score to obtain the instance-level classification score of the corresponding region proposal belonging to the target category.

[0067] S34. Based on the instance-level classification scores of all region proposals, image-level target category predictions are aggregated, and the target detection model is trained using image-level target category labels and multi-instance learning loss. Based on the image-level target category prediction probabilities and the given true image-level target category labels, the multi-instance learning loss is calculated, and then backpropagation is used to train and optimize the network parameters of the target detection model (including parameters of the backbone network, feature mapping network, and detection branches). The network parameters of the pre-trained text-to-image diffusion model are frozen during the multi-instance joint training of the target detection model.

[0068] This step aims to address the inconsistency between text feature and image feature distributions by utilizing the complete object visual features generated by the diffusion model as a classification benchmark. Specifically, as follows... Figure 4 As shown, this step specifically includes two stages: "generation of visual prototype" and "instance score calculation based on dual-branch architecture".

[0069] In the "Generation of Visual Prototypes" step, in order to obtain a visual representation for each target category, this step does not rely on existing image datasets, but instead uses a pre-trained text-to-image diffusion model (Stable Diffusion) to generate a representative set of images.

[0070] For each detected target category, a descriptive cue text is constructed. This cue text is then input into a pre-trained text-to-image diffusion model (Stable Diffusion) to generate a batch of N representative images. The preferred resolution for each image is 512×512.

[0071] To avoid background noise interfering with the visual prototype quality, a cross-attention interception mechanism is utilized in the U-Net backbone network of the text-to-image diffusion model. During the denoising process of the diffusion model, all attention heads and cross-attention scores from multiple layers are captured. Multi-layer scores are spatially aligned using bicubic interpolation and aggregated along the time step to form a heatmap corresponding to the target category c. Subsequently, a threshold is set to convert the heatmap into a binary segmentation mask; in this embodiment, the threshold is set to 0.5, and the mask is rescaled to a 1×h×w dimension.

[0072] This paper utilizes the multi-scale internal features of the text-to-image diffusion model U-Net backbone network to extract object-level representations for foreground focusing. Specifically, it intercepts the query vectors of the cross-attention layers of downsampled, intermediate, and upsampled blocks in U-Net, spatially rescales and concatenates these multi-scale features to form a representation corresponding to a representative image. High-dimensional feature maps .

[0073] Next, by calculating the average value of the feature embeddings within the masked region, the i-th representative image corresponding to target category c is obtained. object-level prototype :

[0074] .

[0075] Where h and w are the height and width of the feature map, respectively. For binary segmentation masks in spatial coordinates If the response intensity of the heatmap at the corresponding location is greater than or equal to the threshold, then hic(u,v) takes the value 1, indicating that the location belongs to the foreground target category region; otherwise, hic(u,v) takes the value 0, indicating that the location belongs to the background region. This represents a broadcasting operation, which aims to extend the binary segmentation mask of a single channel (1×h×w) by copying the data along the channel dimension to match the dimension of the multi-channel feature map. For the corresponding representative image I C High-dimensional feature maps.

[0076] Aggregate object-level visual prototypes belonging to the same target category. These object-level visual prototypes are generated from representative images, with a quantity of N. Calculate the average value to obtain the class-level visual prototype for that target category.

[0077] .

[0078] Finally, a visual prototype matrix containing all C target categories is obtained. .

[0079] After obtaining the visual prototype matrix P, this step assigns a category label and confidence score to each region proposal in the "Instance Score Calculation Based on Dual-Branch Architecture" stage. Specifically, it includes the following two parallel branches:

[0080] The prototype similarity branch is used to identify the category of each candidate region proposal. The feature vector of the candidate region proposals is then adjusted to suppress overfitting. After two projection layers, the feature matrix is ​​obtained. , Where M is the number of candidate region proposals. The feature vectors representing the M candidate region proposals after overfitting suppression are mapped through two projection layers, so that the dimension of the feature vector is consistent with the dimension of the class-level visual prototype.

[0081] Subsequently, the cosine similarity between the feature matrix F and the visual prototype matrix P of the candidate region proposals is calculated, and the Softmax function is applied to obtain the normalized similarity matrix. :

[0082] .

[0083] in, This indicates that the L2 norm is calculated for each column of the matrix. This represents the Softmax operation performed along the category axis, where T is the temperature coefficient. This step achieves semantic feature alignment between the feature matrix F of the candidate region proposal and the visual prototype matrix P.

[0084] The detection branch focuses on locating regions of a specific category. Region proposal features are input into a multilayer perceptron (MLP) head to predict the detection score matrix. The score corresponding to the i-th candidate region proposal is... .in, This indicates a Softmax operation performed along the proposal axis. This operation causes the model to automatically select the best matching region for the target class c and suppress background proposals.

[0085] Element-wise multiplication of the two score matrices yields the final instance-level classification score matrix. :

[0086] .

[0087] The normalized similarity matrix, This is the detection score matrix. The design ensures that a candidate region proposal is only assigned a high score if it is semantically highly aligned with the target class prototype and is confidently located as belonging to that category by the detected branch. This enhances the reliability of subsequent pseudo-label generation.

[0088] During the training phase of the object detection model, weakly supervised learning is performed using image-level category labels. The predicted image-level probabilities are obtained by summing the scores of all proposals for each category. , Subsequently, the Multiple Instance Learning (MIL) loss function was utilized. To monitor the network. Among these, These are real image-level labels.

[0089] S4. Obtain initial pseudo-labels based on the instance-level classification scores of each candidate region proposal and its corresponding target category. Filter the initial pseudo-labels corresponding to fragmented region proposals in the candidate region proposals, merge and correct them, and generate a high-quality pseudo-label set.

[0090] This step is implemented by the instance completion module. It filters the initial pseudo-labels corresponding to fragmented region proposals in the initial region proposal set and inputs them into the instance completion module, such as... Figure 5 As shown, the specific steps include the following:

[0091] S41. Assign category labels and confidence scores to the corresponding candidate region proposals as an initial set of pseudo-labels based on the instance-level classification scores of each candidate region proposal.

[0092] S42. Set a confidence threshold, traverse the initial pseudo-label set, and remove candidate region proposals with confidence scores lower than the confidence threshold to obtain a set of retained region proposals.

[0093] S43. For fragmented region proposals that cover different parts of the same object in the reserved region proposal set, process them separately according to the target category. Cluster the reserved region proposals belonging to the same target category according to the degree of spatial overlap. If the intersection-union ratio (IoU) of two reserved region proposals is greater than the preset overlap threshold, then the two reserved region proposals are assigned to the same cluster.

[0094] S44. Obtain the smallest bounding rectangle that can tightly enclose all reserved region proposals within each cluster, thereby merging the fragmented region proposals within that cluster into a complete target bounding box.

[0095] S45. The merged complete target bounding box and its corresponding target category average confidence score are used as high-quality pseudo-labels to provide supervision signals for the score output of subsequent model training.

[0096] Although the multi-instance learning module for visual prototype alignment described above exhibits stronger robustness compared to cross-modal alignment, the resulting pseudo-labels still face two key issues: first, noisy matching, where background region proposals are incorrectly assigned high similarity scores; and second, fragmentation, where different parts of the same object (such as a person's head and torso) are covered by multiple independent region proposals. The similarity score generated during the prototype matching stage can serve as a natural confidence metric for the category to which each region proposal belongs. Experimental observations have revealed that region proposals corresponding to cluttered backgrounds or highly blurred regions often have lower similarity to the visual prototype. To address these issues, this step employs an instance completion module to optimize the pseudo-labels through a two-stage refinement process, eliminating these unreliable candidate boxes.

[0097] Specifically, firstly, based on the output of the multi-instance learning module aligned with the aforementioned visual prototype, a category label is assigned to the i-th proposal. and confidence score :

[0098] ,

[0099] .

[0100] in It is the instance-level classification score matrix output by the VPA-MIL module.

[0101] Next, for all candidate region proposal sets For each proposal, a confidence threshold t is set; in this embodiment, t = 0.4. If the candidate region proposal set... If the confidence score of a candidate box is lower than the threshold, it is considered a low-quality candidate box and is removed. Formally, the high-confidence proposals are retained. Constitutes a set of proposals for the preservation area , This pruning step removes the most significant errors, ensuring that only proposals with strong semantic consistency with the class prototype are retained for subsequent refinement.

[0102] After filtering, the retained pseudo-labels, while having high confidence, may exhibit a fragmented distribution. For example, a pedestrian might be framed by two separate proposals, one for the "head" and the other for the "torso," both of which correctly match the "pedestrian" prototype. To construct a single pseudo-label for the complete object, a fragment clustering and merging operation is performed.

[0103] Let the set of pseudo-labels for high-confidence reserved region proposals after threshold filtering be . ,in This represents the i-th high-confidence proposal. bounding box, The x and y coordinates of the top-left vertex of the bounding box of the i-th high-confidence proposal are located in the image coordinate system. This represents the width of the bounding box. This represents the height of the bounding box. For the i-th high-confidence proposal Assigned target category labels, For the i-th high-confidence proposal The confidence score. The specific clustering and merging process is as follows:

[0104] For each target category present in the image, extract a subset of proposals belonging to that category. .

[0105] Analyze the spatial relationships among proposals in the proposal subset, and cluster and merge proposals with significant intersection-over-union (IoU) overlap. Specifically, Divided into several disjoint clusters , where k is the index of the cluster, representing "the k-th cluster"; K represents the set of all possible values ​​for k. The condition is that any two bounding boxes within the same cluster satisfy:

[0106] .

[0107] in, In this embodiment, the preset intersection-union ratio (IU) overlap threshold is used. =0.5, These are any two bounding boxes within a cluster. The clustering process can be implemented using either the Greedy Agglomerative Merging algorithm or the Disjoint-Set Union (DSU) algorithm.

[0108] For each identified cluster Calculate the smallest bounding rectangle that tightly encloses all bounding boxes within the cluster, and use it as the complete target bounding box after merging. The calculation formula is as follows:

[0109] .

[0110] Represents cluster The width of the bounding box b of each candidate region proposal within the area. Represents cluster The height of the bounding box b of each candidate region proposal within.

[0111] Calculate each cluster The average confidence score of all proposals is used as the confidence score of the merged complete target bounding box. The final high-quality pseudo-label set for this image. Where ⋃ represents the union operation of sets; C is the set of target categories existing in the image. This formula means: traversing all target category labels c existing in the image and traversing all clusters belonging to the target category c, the generated bounding box containing the complete target will be obtained. The corresponding target category c and the average confidence score The combined items are summarized to form a high-quality set of pseudo-labels for supervising the object detection network.

[0112] Through the sequential processing of noise filtering and fragment clustering merging described above, the instance completion module successfully transforms the raw output of the multi-instance learning stage into a high-quality set of bounding boxes. These high-quality bounding boxes are then used as clearer and more accurate cues, generating compact segmentation masks, or directly as supervision signals for training the final object detection model.

[0113] S5. Construct a target detection model with cascaded refinement branches, perform iterative training and optimization using a high-quality pseudo-label set, and use the trained and optimized target detection model to perform visual target detection for open vocabulary.

[0114] Specifically, the training of the object detection model is coupled and iterative: the learningable feature mask module, the visual prototype alignment multi-instance learning module, and the instance completion module generate high-quality pseudo-labels. The localization score output by the detection head of the object detection model is combined with the prototype semantic alignment score to jointly determine the pseudo-labels for the next round, and then the network is trained in turn to realize the collaborative feedback mechanism of the entire closed-loop system.

[0115] The iterative training process of the object detection model is as follows: The first refinement branch of the object detection model receives the instance-level classification score obtained in step S3 as input, and each subsequent refinement branch receives the output of the previous refinement branch as input. At the beginning of each iteration, the object detection model uses the high-quality pseudo-label set generated by the previous iteration as a supervision signal. The classification loss and regression loss of each refinement branch are obtained using the supervision signal, and the object detection model parameters are updated through backpropagation. After each iteration, the high-confidence detection result output by the last refinement branch is selected as a new set of high-quality pseudo-labels for supervision in the next iteration. The above supervised training and high-quality pseudo-label dynamic update steps are repeated until the preset maximum number of iterations or the preset training termination condition is reached.

[0116] See you again Figure 1 The high-quality pseudo-tag set generated in step S4 As the initial supervisory signal, a multi-stage online instance classifier refinement strategy is used to gradually improve the localization accuracy and classification accuracy of the object detection model. This embodiment constructs an object detection network containing K cascaded refinement branches (classifiers), with K=3 being the preferred value in this embodiment. The specific iterative training process is as follows:

[0117] Following the visual prototype alignment multi-instance learning module, K structurally identical refinement branches (classifiers) are sequentially connected. Each refinement branch (classifier) ​​typically consists of fully connected layers, outputting a classification score vector and bounding box regression offset for each region proposal. The input features of the first refinement branch come from shared backbone network features. While subsequent refinement branches share features, their supervision signals come from the prediction results of the previous branch.

[0118] For the first refinement branch (K=1), the high-quality pseudo-label set output in step S4 is used directly. As a monitoring signal. Specifically, it calculates all regional proposals and a set of high-quality pseudo-labels. The intersection-over-union (IoU) ratio of the pseudo-label bounding boxes is determined by the threshold value. If the IoU is greater than the preset IoU overlap threshold, then... If the proposal in that region is positive, its target category is the category of the pseudo-label; otherwise, it is a negative background sample. For subsequent refinement branches (K>1), the supervision signal is generated online. That is, using the predicted confidence score of all proposals from the K-1 branch and the bounding box coordinates after regression, high-confidence bounding boxes are selected and used as the dynamic pseudo-labels for the K-th branch.

[0119] As the number of training iterations increases, the predictive power of the refinement branch gradually improves. After each training epoch or iteration, the high-confidence detection results output by the last refinement branch (the Kth branch) are selected, processed by non-maximum suppression (NMS), and updated into a new pseudo-label set. This updated set will be used to supervise the first-level branch in the next training epoch, thus forming a virtuous cycle of "prediction-correction-retraining". The above supervised training and pseudo-label dynamic update steps are repeated until the preset maximum number of iterations is reached or the model's performance on the validation set no longer improves.

[0120] This paper describes a visual object detection task using a trained and optimized object detection model. The input is a test image and a user-specified set of open-vocabulary object categories. The output includes the bounding box coordinates of the object in the image, the corresponding open-vocabulary category label, and the prediction confidence. Only the output score of the last refinement branch of the object detection model is used as the final detection result, and the coordinates of the corresponding proposal plus the predicted regression bounding box offset are used as the final detection box.

[0121] This embodiment performs actual visual target detection based on a computer device. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the weakly supervised open vocabulary object detection method described above in this embodiment.

[0122] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such a step or method.

[0123] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A weakly supervised open-vocabulary object detection method, characterized in that, Includes the following steps: S1. Obtain the input image and simultaneously extract candidate region proposals from the input image using a region proposal network and a pre-trained image segmentation model. S2. Obtain the initial feature vector corresponding to each candidate region proposal through region of interest pooling operation, and suppress overfitting in the local discriminative region of the initial feature vector to obtain the overfitted feature vector of the candidate region proposal. S3. Based on the pre-trained diffusion model, a category-specific visual prototype is generated for the target category to be detected, and the overfitted candidate region proposal feature vector is mapped to the unified feature space where the visual prototype is located, so that the feature vector is aligned with the visual prototype information, and a semantic alignment score is obtained. Based on the semantic alignment score, the instance-level classification score of each candidate region proposal for each target category is obtained. S4. Obtain initial pseudo-labels based on the instance-level classification scores of each proposal and the corresponding target category. Filter the initial pseudo-labels corresponding to fragmented regional proposals in the candidate regional proposals, fuse and correct them, and generate a high-quality pseudo-label set. S5. Construct a target detection model with cascaded refinement branches, perform iterative training and optimization using a high-quality pseudo-label set, and use the trained and optimized target detection model to perform visual target detection for open vocabulary.

2. The weakly supervised open vocabulary object detection method according to claim 1, characterized in that: In step S1, the multi-scale feature map of the input image is input into the region proposal network to extract and generate a first region proposal, the input image is input into the pre-trained image segmentation model to extract and generate a second region proposal, and the first region proposal and the second region proposal are merged to form a candidate region proposal set.

3. The weakly supervised open vocabulary object detection method according to claim 1, characterized in that: In step S2, a spatial probability map is generated based on the initial feature vector of the candidate region proposal, where the value of each spatial location represents the probability of occluding the feature at that location. A binary spatial mask is generated by differentiable sampling of the spatial probability map. The binary spatial mask is applied to the initial feature vector of the corresponding candidate region proposal to obscure the local discriminative region features identified as overactive, thereby obtaining the feature vector of the candidate region proposal after suppressing overfitting.

4. The weakly supervised open vocabulary object detection method according to claim 3, characterized in that: The differentiable sampling is implemented using the Gumbel-Sigmoid function, which generates a differentiable approximation of the binary space mask based on the spatial probability map.

5. The weakly supervised open vocabulary object detection method according to claim 1, characterized in that: Step S3 includes the following sub-steps: S31. For each detected target category, a representative image is generated based on a pre-trained text-to-image diffusion model, and a cross-attention mechanism is used to extract class-level visual prototypes focusing on the foreground target category from the generated representative images; S32. A semantic alignment score is obtained by using the semantic similarity between the feature vector after suppressing overfitting through candidate region proposals and the visual prototypes of each target category, and the detection score of the corresponding candidate region proposal is predicted through the detection branch of the multilayer perceptron; S33. For each candidate region proposal and each target category, the corresponding semantic alignment score is combined with the detection score to obtain the instance-level classification score of the corresponding region proposal belonging to the target category; S34. Based on the instance-level classification scores of all region proposals, the image-level target category prediction is obtained by aggregation, and the target detection model is trained using the image-level target category label and multi-instance learning loss.

6. The weakly supervised open vocabulary object detection method according to claim 5, characterized in that: In step S31, multiple representative images are generated for each target category using a pre-trained text-to-image diffusion model; for each generated representative image, a heatmap for locating the target category is generated by extracting the cross-attention information associated with the target category text generated by the text-to-image diffusion model during the denoising process. Based on the heatmap, a segmentation mask for the target foreground object in the generated representative image is obtained; using the segmentation mask, the feature representation of the foreground object is extracted from the depth features corresponding to the representative image as an object-level visual prototype. Aggregate multiple object-level visual prototypes belonging to the same target category to obtain the class-level visual prototype of that target category.

7. The weakly supervised open vocabulary object detection method according to claim 5, characterized in that: Step S4 includes the following sub-steps: S41. Assign category labels and confidence scores to the corresponding candidate region proposals as an initial set of pseudo-labels based on the instance-level classification scores of each candidate region proposal. S42. Set a confidence threshold, traverse the initial pseudo-label set, and remove candidate region proposals with confidence scores lower than the confidence threshold to obtain a set of retained region proposals. S43. For fragmented region proposals that cover different parts of the same object in the reserved region proposal set, process them according to the target category. Cluster the reserved region proposals belonging to the same target category according to the degree of spatial overlap. If the intersection-union ratio of two reserved region proposals is greater than the preset overlap threshold, then the two reserved region proposals are assigned to the same cluster. S44. Obtain the smallest bounding rectangle that can tightly enclose all reserved region proposals within each cluster, thereby merging the fragmented region proposals within that cluster into a complete target bounding box. S45. The merged complete target bounding box and its corresponding target category average confidence score are used as high-quality pseudo-labels.

8. The weakly supervised open vocabulary object detection method according to claim 5, characterized in that: In step S5, the first refinement branch in the target detection model receives the instance-level classification score obtained in step S3 as input, and each subsequent refinement branch receives the output of the previous refinement branch as input. At the start of each iteration of the object detection model, the high-quality pseudo-label set generated in the previous iteration is used as a supervision signal; the classification loss and regression loss of each refinement branch are obtained using the supervision signal, and the object detection model parameters are updated through backpropagation. After each iteration of the target detection model, the high-confidence detection results output by the last refinement branch are selected as a new set of high-quality pseudo-labels for supervision of the next iteration of the target detection model. The above supervised training and high-quality pseudo-label dynamic update steps are repeated until the preset maximum number of iterations or the preset training termination condition is reached.

9. A weakly supervised open-vocabulary object detection system, characterized in that: The object detection model framework includes: The region proposal extraction module uses a region proposal network and a pre-trained image segmentation model to simultaneously extract candidate region proposals from the input image. The learnable feature mask module obtains the initial feature vectors corresponding to each candidate region proposal through region of interest pooling operation, and performs overfit suppression on the local discriminative regions of the initial feature vectors to obtain the overfit-suppressed feature vectors of the candidate region proposals. The visual prototype alignment multi-instance learning module maps the overfitted feature vectors of candidate region proposals to a unified visual prototype space to generate category-specific visual prototypes, thereby aligning the feature vectors with the visual prototype information and obtaining a semantic alignment score. Based on the semantic alignment score, the instance-level classification score of each candidate region proposal for each target category is obtained. The instance completion module obtains initial pseudo-labels based on the instance-level classification scores of each proposal and category. It then filters the initial pseudo-labels corresponding to fragmented region proposals in the candidate region proposals, merges and corrects them, and generates a high-quality pseudo-label set. The high-quality pseudo-label set is used for iterative training and optimization of the object detection model, and the trained and optimized object detection model is used for visual object detection of open vocabulary.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the weakly supervised open vocabulary object detection method as described in any one of claims 1-8.

Images