Weak supervision target detection method guided by cross-modal pseudo tag

Abstract

The invention relates to the technical field of computer vision and multi-modal learning, in particular to a weak supervision target detection method guided by cross-modal pseudo labels. According to the method, a labeled source domain data set is constructed to train an image classification teacher model, and a teacher-student network structure is constructed; clustering the regional features of the target domain image, allocating pseudo tags to each cluster by optimizing the allocation cost between the source domain category and the target domain cluster, and constructing a pseudo tag pool; and training a student model on the pseudo label pool for region feature detection of the target domain image. According to the method, a cross-modal attention mechanism is introduced, so that more accurate semantic alignment between a source category label and a target domain feature is realized; the stability of label distribution is improved by a structure keeping regular term; the generalization ability of the model is further enhanced by multiple rounds of pseudo-label confidence learning. The method can be widely applied to tasks such as target detection, cross-domain transfer learning and open world recognition, and efficient and accurate weak supervision target detection is realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal learning technology, and in particular to a weakly supervised target detection method guided by cross-modal pseudo-labels. Background Technology

[0002] In recent years, weakly supervised object detection (WSOD) technology has been widely used in medical imaging, remote sensing, autonomous driving, and other scenarios. Compared with fully supervised methods, WSOD can significantly reduce annotation costs; however, the quality of pseudo-labels has a significant impact on the final detection performance.

[0003] In weakly supervised object detection tasks, traditional methods typically rely on image-level labels, pseudo-label generation, or pre-trained models for training to reduce reliance on manually labeled bounding box data. However, existing technologies still have several significant drawbacks in practical applications: 1. Coarse pseudo-label assignment and inaccurate category alignment: Most existing methods use maximum similarity matching or heuristic rules to assign pseudo-labels to image regions, ignoring the global optimal matching relationship between the source domain category distribution and the target domain semantic structure, which easily leads to pseudo-label offset and category confusion.

[0004] 2. The reliability of pseudo-labels is unstable and the difficulty level is not considered: Traditional methods often treat all pseudo-labels the same, lacking the assessment and stratification of sample difficulty, making them susceptible to interference from low-quality labels and affecting the robustness of the detector.

[0005] 3. Lack of confidence multi-granularity modeling and collaborative supervision mechanism: Existing pseudo-label optimization techniques mainly focus on a single scale (such as category or image level), ignoring the multi-level collaborative effects of inter-category uncertainty, regional feature residuals and global semantic consistency of the image, making it difficult to effectively control the quality of pseudo-labels.

[0006] 4. Lack of closed-loop optimization mechanism between pseudo-labels and model training: Most methods adopt a one-time pseudo-label generation strategy or a static pseudo-supervision strategy, which lacks dynamic update and collaborative feedback mechanism, making it difficult to adapt to the fluctuation of label quality and the evolution of model structure during the training process. Summary of the Invention

[0007] To overcome the shortcomings of existing weakly supervised object detection methods in terms of pseudo-label accuracy control, confidence modeling, label alignment strategies, and feedback mechanisms during training, this invention proposes a cross-modal pseudo-label-guided weakly supervised object detection method to improve object detection performance.

[0008] This invention proposes a weakly supervised target detection method guided by cross-modal pseudo-labels, comprising the following steps: A labeled source domain dataset is constructed, an image classification teacher model is trained, and a teacher-student network structure is built. Cluster the regional features of the target domain image; assign pseudo-labels to each cluster by optimizing the optimal cross-modal assignment relationship between the source domain category and the target domain cluster, forming a pseudo-label pool; initialize a high-confidence pseudo-label sample set and a low-confidence pseudo-label sample set; Train the student model on the pseudo-label pool, and use the converged student model to update the pseudo-label pool. Update the high-confidence pseudo-label sample set with pseudo-label samples in the pseudo-label pool that meet the constraints, and update the low-confidence pseudo-label samples with pseudo-label samples in the pseudo-label pool that do not meet the constraints. The constraints are: the standard deviation of the predicted probability distribution and the variance of the historical category probability sequence are both lower than the set threshold; the historical category probability sequence is the sequence of pseudo-label sample category probabilities at each pseudo-label pool update; Repeatedly train the student model until the pseudo-label pool overlaps with the high-confidence pseudo-label sample set, then output the student model to detect the region category of the target domain image.

[0009] Preferably, before training the student model on the pseudo-label pool for the first time, the following steps are performed to optimize the pseudo-label pool: The confidence level of pseudo-label samples is evaluated, and they are divided into a high-confidence pseudo-label sample set and a low-confidence pseudo-label sample set. An auxiliary network is trained on the high-confidence pseudo-label sample set to correct the low-confidence pseudo-label samples and update the pseudo-label pool.

[0010] Preferably, the optimization objective of the optimal cross-modal assignment relationship between the source domain category and the target domain cluster in step S2 is: ; Transmission matrix T= C represents the number of source domain categories, and K represents the number of target domain clusters. To assign weights from source domain category j to target domain cluster k, for each target domain cluster k, select... The source domain category j corresponding to the maximum value is used as the pseudo-label, and a pseudo-label pool is constructed. ; ; in, The cost of the source domain category j and the target domain cluster k; Let be the text modal label prototype vector of source domain category j. Let k be the cluster center of the target domain cluster cluster k; The attention cost weight; The semantic deviation term is the label prototype vector of the source domain category j to the cluster center of the target domain cluster k. The allocation entropy for cluster k in the target domain; is the entropy regularization weight; the label prototype vector is the mean of the image and text embedding representations of all labeled regions under the same label.

[0011] Preferably, the auxiliary network is used on a high-confidence pseudo-label sample set. The loss function for training is: ; in: Samples for auxiliary network prediction The maximum probability of the category distribution; The set positive and negative sample balance factor; The difficulty adjustment factor is set. The confidence weighting function is set. , To set a constant, and .

[0012] Preferably, the loss function during the training process of the student model on the pseudo-label pool. It includes one or more of the following: source domain supervision loss, focus loss, transport alignment loss, cross-round stability regularization term and multi-granularity confidence alignment total loss. The cross-round stability regularization term is represented by the KL divergence of the predicted probability distribution of the corresponding converged student model after the two most recent iterations of the pseudo-label pool.

[0013] Preferably, the total alignment loss includes one or more of the following: category information entropy, instance classification loss, feature residual penalty loss, image-level KL alignment loss, and image entropy regularization term.

[0014] Preferably, the total alignment loss also includes the KL alignment loss of the structural region. The source dataset consists of {images with labeled borders, border labels; labeled region masks, label prototype vectors}. ; in, For the first image in the source domain The salience weight of each labeled region in the structural mask Let i be the region feature i of the target domain image, predicted by the teacher model. Let i be the maximum probability predicted by the student model for the region feature i of the target domain image.

[0015] Preferably, the method for obtaining the labeled region mask is as follows: use a pre-trained SAM model to perform foreground segmentation on the image, and extract the binary mask of the labeled region as the labeled region mask.

[0016] Preferably, the total alignment loss It is a weighted sum of category information entropy, instance classification loss, feature residual penalty loss, image-level KL alignment loss, image entropy regularization term and structural region KL alignment loss, and the weights of each sub-item loss are dynamically adjusted.

[0017] Preferred, characteristic residual penalty loss and instance classification loss The calculation formula is as follows: ; ; in, For instance feature residuals, The image features of sample i from the pseudo-label pool are output by the visual encoder. The image-text embedding representation is obtained by modal fusion of the image features of the pseudo-label pool sample i after passing through the visual encoder and the text features of the diverse semantic description text after passing through the text encoder; Represents the L2 norm; ; ; ; in, The weights of sample i in the pseudo-label pool. For cross-entropy loss, The class probability of pseudo-label pool sample i predicted by the student model. Let i be the label category of sample i in the pseudo-label pool; For activation functions; The set boundary sensitivity adjustment parameters; This represents the confidence boundary difference between the predicted probability and the second-highest probability of a sample in the pseudo-label pool. The pseudo-label category of sample i in the pseudo-label pool after passing through the student model. The corresponding probability, The probability of sample i in the pseudo-label pool being the second highest class probability after passing through the student model.

[0018] The advantages of this invention are: (1) Improve the accuracy and global consistency of pseudo-label alignment: By introducing the allocation between the source domain category and the target domain cluster, a probability allocation alignment is achieved between the source domain category and the target domain feature cluster, which effectively avoids pseudo-label category mismatch, improves the coverage of low resource categories, and enhances the robustness of overall pseudo-supervised training.

[0019] (2) Enhance the confidence control of pseudo-labels and the stability of training effect: introduce pseudo-label confidence to dynamically update the pseudo-label pool, reduce training interference caused by low confidence pseudo-labels, and significantly improve the detector's ability to identify complex samples; further introduce the design of difficulty-aware loss function, dynamically weight the sample weights according to confidence, and focus on learning to distinguish regions with blurred boundaries or difficult model prediction.

[0020] (3) Form a closed-loop mechanism for pseudo-label generation, optimization and training: adopt a teacher-student collaborative training structure to realize the dynamic updating of pseudo-labels and the synchronization of feature module memory, construct a feedback learning process, so that the detector can gradually converge and continuously optimize its pseudo-label quality and detection performance.

[0021] (4) Implement a multi-granularity confidence alignment mechanism at the category, instance and image levels: This invention introduces strategies such as temperature control, feature residuals and full-image KL consistency at different granularity levels to build a multi-dimensional pseudo-label quality control framework from fine granularity to global space, thereby enhancing the model's ability to model the structural information of pseudo-supervised samples.

[0022] (5) High generalization and adaptability: This method is compatible with mainstream detection frameworks and can be adapted to different types of visual-language models and structural masking mechanisms. It is suitable for detection tasks with high annotation costs or incomplete labels in practical applications.

[0023] (6) Significantly improves detection performance and training efficiency: In experimental verification, the present invention has shown high target detection accuracy and training stability on multiple public datasets (such as PASCAL VOC and COCO), especially in low-supervision or bounding box labeling environments, showing significant advantages.

[0024] (7) In summary, this invention provides a complete, efficient, and theoretically innovative weakly supervised object detection solution from pseudo-label generation, credibility modeling, structure guidance, multi-granularity alignment to closed-loop training optimization. This invention can stably generate high-quality pseudo-labels and continuously optimize the detection model without the need for precise bounding box annotation, thus achieving efficient and accurate weakly supervised object detection. This invention can be widely applied to object detection scenarios such as medical imaging, remote sensing, and autonomous driving. Attached Figure Description

[0025] Figure 1 This is a flowchart of a cross-modal pseudo-label-guided weakly supervised target detection method proposed in this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1 Reference Figure 1 The present embodiment proposes a cross-modal pseudo-label-guided weakly supervised target detection method, which includes the following steps S1-S6.

[0028] S1. Construct a labeled source domain dataset and train a teacher model for image classification; construct a teacher-student dual network structure based on the teacher model.

[0029] In this embodiment, the source domain dataset is denoted as {image with labeled borders, border labels; labeled region mask, label prototype vector}; the labeled region prototype vector is the image-text embedding representation of the labeled region, and the label prototype vector is the mean of all labeled region prototype vectors under the same label.

[0030] The source dataset can specifically use images from open datasets (such as PASCAL VOC, COCO, OpenImages, etc.); the bounding boxes can be labeled using object detection algorithms to identify the target region and the category of the target object.

[0031] The labeled region mask is obtained by processing the image using the SAM model. Specifically, a pre-trained SAM model is used to segment the foreground of the image, and the binary mask of the labeled region is extracted. This mask is then used for saliency guidance in subsequent modality alignment.

[0032] The method for obtaining the prototype vector of the label is as follows: First, obtain the diverse semantic description text and text semantic features of each labeled region on the image. Then, align the image feature encoding and text semantic features of the labeled region modally to generate the image-text embedding representation of each labeled region. Take the average of the image-text embedding representations of the same type of labeled region as the prototype vector of that label category.

[0033] In practice, block coding can be used to obtain the image feature codes of each labeled region on the image. First, a visual encoder (e.g., ViT-B / 16, ResNet50) is used to encode the labeled regions of the image in blocks. Then, the ROIAlign operation is performed on each labeled region to extract spatial feature vectors as image feature encodings. .

[0034] The method for obtaining diverse semantic description text and text semantic features of the labeled regions is as follows: Class name descriptions are constructed for the border labels, and diverse semantic description texts of the class names are generated using VLMs (such as BLIP and CLIP); then, text encoders (such as BERT, T5, and GPT2) are used to embed the diverse semantic description texts to obtain the text semantic features. .

[0035] In this embodiment, the image features of each labeled region are encoded using the Cross-Modal Transformer module. and text semantic features t r Modal alignment is performed to generate image-text embeddings. The Cross-Modal Transformer module employs a contrastive learning strategy or distance constraints (such as InfoNCE or Triplet Loss) to bring image-text pairs with the same semantics closer together in the embedding space, while keeping samples with different semantics spaced apart.

[0036] The teacher model is trained as follows: First, a source domain training set {images with labeled borders, border labels; labeled region masks, label prototype vectors} is extracted from the source dataset {images with labeled borders, prototype vectors of each labeled region}. Then, the teacher model is trained on the source domain training set until convergence. The teacher model is used to label the target regions of the image and generate the probability distribution of the prototype vectors of each target region.

[0037] The teacher model can specifically use the Faster R-CNN detector; the Faster R-CNN detector outputs multi-scale feature maps, target regions, and label prototype vectors for each target region based on the image with labeled bounding boxes.

[0038] Specifically, the Faster R-CNN detector extracts multi-scale feature maps from the source domain image and identifies the target region. Then, it encodes the features of each target region, calculates and normalizes the similarity between the encoded vector and each label prototype vector, and generates the label prototype vector probability distribution of the target region based on the similarity normalization value.

[0039] In practical implementation, the formula for calculating the probability distribution of the tag prototype vector of the target area is: P(Xs∈c)= ; in, For source domain samples The similarity between the encoded vector generated by the Faster R-CNN detector (i.e., the source domain image annotation region) and the label prototype vector of the source domain category c can be obtained by using a correlation algorithm or Euclidean distance; C is the number of source domain categories.

[0040] During the teacher model training process, classification loss can be used. Border regression loss Alignment loss with prototype The total loss function consists of three parts. In practice, losses are categorized. Cross-entropy loss and bounding box regression loss can be used. Specifically, the Smooth L1 function and prototype alignment loss can be used. Specifically, it can be in the form of InfoNCE.

[0041] S2. Cluster the regional features of the target domain image to obtain cluster centers; generate a cross-modal semantic attention-guided transfer matrix T by optimizing the optimal cross-modal allocation relationship between the source domain category and the target domain clusters, and assign pseudo-labels to each cluster accordingly, thereby constructing a pseudo-label pool {regional features of the target domain image, pseudo-labels}. The transfer matrix T adopts the CAM-Trans matrix.

[0042] Regional features of the target domain image can be extracted using pre-trained image recognition algorithms. In practice, when the difference between the source and target domains is small, a Faster R-CNN network can be used to identify the target region and extract its features. When the difference between the source and target domains is large, a segmentation model (SAM) can be used to extract salient mask regions and generate regional features. Alternatively, other image recognition algorithms can be used to extract regional features, as specified by the administrator.

[0043] Step S2 can be further divided into the following sub-steps S21-S24.

[0044] S21. Cluster the regional features of the target domain image and obtain the cluster center of each cluster. That is, the cluster center is the average feature vector of the regional features of all target domain images in the cluster.

[0045] In practical implementation, Mini-Batch KMeans can be used to cluster the regional features of the target domain image, generating K clusters, with the set of cluster centers as follows: ; Let K denote the cluster center of cluster k in the target domain, where 1 ≤ k ≤ K; and These represent the cluster centers of the 1st and Kth clusters in the target domain, respectively.

[0046] S22. Construct the cost matrix between the source domain categories and the target domain clusters. Furthermore, a cross-modal attention mechanism is introduced to construct a transfer matrix that enables cross-modal semantic attention guidance. That is, T= The transmission matrix T can specifically be a CAM-Trans matrix. C represents the number of source domain categories, and K represents the number of target domain clusters. The weights assigned from source domain category j to target domain cluster k, >0, and =1.

[0047] ; ; in, Let t be the text modality label prototype vector of source domain class j, that is, the text features t of all samples under source domain class j. r The average; Let k be the cluster center of the target domain cluster cluster k; The cost of the source domain category j and the target domain cluster k; This is the attention cost weight, and its value ranges from [0.1, 1). In subsequent embodiments, it is specifically set to 0.5. The semantic bias term from source domain category j to target domain cluster k is calculated by the cross-modal attention mechanism. It is obtained by projecting the label prototype vector of source domain category j and the cluster center of target domain cluster k onto a unified semantic space, calculating the similarity between the two and normalizing it with softmax to obtain the attention weight. The difference measure (such as Euclidean distance or cosine distance) between the label prototype vector of source domain category j and the cluster center of target domain cluster k is weighted by this attention weight and used as the semantic bias term.

[0048] The optimization objective of the transfer matrix T is: ; ; in, The allocation entropy for cluster k in the target domain; This is the entropy regularization weight, with a typical value range of [0.01, 0.1]. The dimension of the transfer matrix T is . and all elements All are greater than 0.

[0049] S23. Based on the pseudo-label categories assigned to clusters by the transfer matrix T, construct a pseudo-label pool {regional features of the target domain image, pseudo-labels}; Specifically, for each target domain cluster k, select The source domain category j corresponding to the maximum value is used as the pseudo-label, thus constructing a pseudo-label pool {regional features of the target domain image, pseudo-label}; the formula is expressed as: ; The pseudo-labels are clusters k in the target domain; this pseudo-label pool will be used for subsequent confidence prediction, auxiliary correction, and student model training.

[0050] S3. Use a pre-trained confidence prediction model to predict the confidence level of pseudo-label samples in the pseudo-label pool, and divide the pseudo-label samples into a high-confidence pseudo-label sample set. And low-confidence pseudo-labeled samples. The confidence prediction model can specifically use the pre-trained CLIP model, or other known models.

[0051] In practice, this section compares the model prediction confidence with a set confidence threshold, and classifies pseudo-label samples in the pseudo-label pool with a confidence level greater than or equal to the confidence threshold into a high-confidence pseudo-label sample set. Pseudo-labeled samples with a confidence level lower than the confidence threshold are classified as low-confidence pseudo-labeled samples. The confidence threshold can specifically take values ​​in the range [0.6, 0.9], for example, it is set to 0.8 in the following embodiment.

[0052] S4. In the high-confidence pseudo-label sample set The auxiliary network is trained to classify the regional features of the target domain image. The converged auxiliary network is used to correct low-confidence pseudo-label samples and update the pseudo-label pool.

[0053] The auxiliary network can be any known image classification network. In subsequent embodiments, a CLIP-Adapter module can be used to align with the category text prototype, combined with a quality-aware head for sample confidence estimation, and a text feature fusion of OCR / ASR / metadata can be achieved through a text evidence gating module, thereby forming a combined auxiliary network.

[0054] The auxiliary network is used to classify samples in the target domain, and the updated pseudo-label pool is equivalent to a high-confidence pseudo-label sample set. The pseudo-label pool is a set of low-confidence pseudo-label samples that have been corrected by an auxiliary network. Specifically, for low-confidence pseudo-label samples in the pseudo-label pool {regional features of the target domain image, pseudo-labels}, the original labels are replaced with pseudo-labels corrected by the auxiliary network, thus updating the pseudo-label pool. The low-confidence pseudo-label samples are essentially the pseudo-label pool {regional features of the target domain image, pseudo-labels} minus the set of high-confidence pseudo-label samples. The remaining samples.

[0055] The auxiliary network on the high-confidence pseudo-label sample set The loss function used in training is focus loss. ; ; in: Samples for auxiliary network prediction The maximum probability of the category distribution, i.e. ,sample From high-confidence pseudo-label sample set ; The set positive and negative sample balance factor, ; The difficulty adjustment factor is set to focus on difficult samples. ; This is a confidence weighting function used to introduce a confidence-difficulty mapping relationship; Specifically, confidence level is inversely proportional to weight; that is, a high-confidence pseudo-label sample set... In the image sample The higher the confidence level, the better the image sample weight The smaller; specifically, it can be set as: , To set a constant, and In subsequent embodiments, Set to 2.0.

[0056] S5. Train the student model on the pseudo-label pool until convergence; use the converged student model to predict the predicted probability of each sample in the pseudo-label pool and update the pseudo-label pool, then add the high-confidence pseudo-label sample set. The pseudo-label samples are updated to those that meet the constraints in the pseudo-label pool, while the low-confidence pseudo-label samples are updated to those that do not meet the constraints in the pseudo-label pool.

[0057] The process of training the student model on the pseudo-label pool in step S5 is as follows: S51. Randomly select training samples from the pseudo-label pool to iterate the student model. When the number of iterations reaches a set value A, the student model is used to traverse the pseudo-label pool, calculate the loss function, and update the student model through the direction of the loss function. S52. Repeat step S51 until the student model converges, then update the pseudo-label pool using the student model.

[0058] Specifically, the set value A can be dynamically set. Thus, by increasing A, the model loss converges quickly in the early iterations, improving convergence efficiency; in later iterations, the update interval is stabilized, avoiding overfitting; finally, the student model is fixed and the pseudo-label pool is updated when the pseudo-label change rate is very small. In a subsequent specific embodiment, A=2 can be set in the first 10 iterations; and A=5 in subsequent iterations.

[0059] Specifically, the convergence condition for the student model can be set as follows: if the number of iterations of the student model reaches a set value, such as 100, then the student model in the current pseudo-label pool is considered to have converged.

[0060] During the student model iteration process, the class probability of pseudo-label sample i is calculated for each iteration of the pseudo-label pool, i.e., the maximum predicted probability output by the student model. This constitutes the historical class probability sequence of pseudo-label sample i, and the variance of this sequence is denoted as Var(p). i ); The constraints used for the updated pseudo-label pooling filter after the student model converges are as follows: and ; in, The temperature of pseudo-labeled sample i is represented by the standard deviation of the predicted probability distribution of the student model after convergence. and For setting value, The value range is [0.05, 0.2]. The value range is [0.005, 0.05].

[0061] Loss function of student model during training on pseudo-label pool for: ; in, For source domain monitoring loss; Loss of focus; The optimal transmission alignment loss is obtained using the Sinkhorn algorithm; For cross-round stability regularization, KL consistency can be used. Total loss for multi-granularity confidence alignment; , , , , As the weight, it can be set to .

[0062] ; in, This represents the number of samples in the pseudo-label pool. This represents the predicted probability distribution of sample i in the pseudo-label pool when the student model converges in this round. represents the predicted probability distribution of sample i in the pseudo-label pool when the student model converged in the previous round; KL is the divergence function.

[0063] ; in, To align the total loss, For category information entropy, For instance classification loss, For characteristic residual penalty loss, For image-level KL alignment loss, This is an image entropy regularization term that restricts overly chaotic or scattered prediction distributions. The KL alignment loss for the structural region; , , , , , The loss weights for each sub-item can be set as hyperparameters.

[0064] In practice, , , , , , A phased, dynamic adjustment mechanism is adopted, with the following strategy: In the initial phase, rounds 1-10, the loss weights are set as follows to strengthen instance supervision and weaken other regularizations: =0.1、 =1.0、 =0.0、 =0.1、 =0.0、 =0.1; In the intermediate stage, from rounds 11 to 70, the loss weights are set as follows: feature residual constraints and image entropy regularization are gradually introduced to maintain instance supervision stability. =0.5、 =1.0、 =0.2、 =0.7、 =0.3、 =0.7; In the later stages, from rounds 71 ​​to 100, the loss weights are set as follows: structural consistency convergence is strengthened, instance supervision is weakened, and residuals and regularization are appropriately retained. =0.7、 =0.5、 =0.2、 =0.7、 =0.2、 =1.0.

[0065] In practice, all weights can be kept balanced or further optimized through adaptive adjustment (such as gradient regression or attention mechanisms). This dynamic strategy can effectively address issues such as fluctuations in pseudo-label quality and sample imbalance, improving overall training stability and detection accuracy.

[0066] ; Where K is the set number of target domain categories. This represents the number of pseudo-labeled samples belonging to target domain category k in the current training round. The predicted probability of the nth pseudo-label sample of the kth class predicted by the student model; ; ; ; in, For the first The weights of each sample, For cross-entropy loss, The class probability of pseudo-label pool sample i predicted by the student model. Let i be the label category of sample i in the pseudo-label pool; For activation functions; This represents the confidence boundary difference between the predicted probability and the second-highest probability of a sample in the pseudo-label pool. The label category of sample i in the pseudo-label pool after passing through the student model The corresponding probability, The second highest class probability of sample i in the pseudo-label pool after passing through the student model; The set boundary sensitivity adjustment parameter can specifically take values ​​in the range [5, 10]; in a specific embodiment, the boundary sensitivity adjustment parameter... The optimal setting is 8, but it can also be set to 5 in the early iterations to enhance sample coverage, and then gradually adjusted to 10 in the later iterations to improve the strictness of pseudo-label filtering and convergence stability.

[0067] ; ; in, For instance feature residuals, The image features of sample i from the pseudo-label pool are output by the visual encoder. The image-text embedding representation of pseudo-label pool sample i is obtained by modal fusion of image features from a visual encoder and text features from a text encoder, representing diverse semantic descriptions of the sample. In practice, the visual encoder and image-text fusion method used in the source domain dataset can be directly adopted to obtain the image-text embedding representation of pseudo-label pool sample i. and .

[0068] ; in, For the teacher model, pseudo-label samples The output pseudo-label probability distribution; For student model pairs The output pseudo-label probability distribution; KL represents the KL divergence function; I i This is pseudo-label sample i in the pseudo-label pool.

[0069] ; ; in, The overall prediction entropy for pseudo-labeled samples is also called the average entropy of all categories. For the category of pseudo-label sample i ; The pseudo-label category output by the student model for all samples in the pseudo-label pool. The average predicted probability, i.e. , This represents the probability that the pseudo-label sample i, predicted by the student model, belongs to the pseudo-label category h.

[0070] ; in, For the source domain dataset, the first The saliency weight of the labeled region in the structural mask, i.e. the weight of the labeled region. Each labeled region is output by the SAM model; The predicted probability distribution of pseudo-labeled sample i using the teacher model. The predicted probability distribution of pseudo-labeled sample i by the student model; This represents the number of samples in the pseudo-label pool.

[0071] S6. Repeat steps S4-S5 until the pseudo-label pool converges, meaning all samples in the pseudo-label pool satisfy the constraints.

[0072] In practical applications, for the target domain image to be detected, the target region and its features are first extracted using an image recognition algorithm. Then, the student model generates pseudo-labels for each target region based on these features. Specifically, the image recognition algorithm used in the actual application and the pseudo-label pool construction should be consistent.

[0073] Example 2 Compared to Example 1, in step S1 of this embodiment, the source domain dataset is {image with labeled borders, border labels; label prototype vectors}; In step S5 of this embodiment, the multi-granularity confidence-aligned total loss is used during student model training. That is, the KL alignment loss of the structural region was removed. .

[0074] Compared to Example 1, this implementation removes the mask model SAM and the corresponding loss during the source domain dataset processing.

[0075] Example 3 Compared to Example 1, the total loss of multi-granularity confidence alignment in this embodiment is... The sub-item weights are fixed values, which are fixed at 1. =0.1、 =1.0、 =0.0、 =0.1、 =0.0、 =0.1.

[0076] Example 4 In this implementation, the prototype vectors of the source domain dataset are encoded using visual image encoding, abandoning image-text fusion. That is, the prototype vector of the labeled region is the image encoding vector obtained by a pre-trained visual image encoder of the labeled region image, and the prototype vector of the label is the average of the prototype vectors of all labeled regions under the same label.

[0077] Example 5 Compared to Example 1, in step S5 of this embodiment, after updating the pseudo-label pool using the converged student model, the confidence prediction model is still used to segment the high-confidence pseudo-label sample set. The sample includes low-confidence pseudo-labeled samples, and then returns to step S4. That is, compared with Example 1, this implementation removes the constraints, which is equivalent to removing the stability constraints.

[0078] Example 6 Compared to Example 1, the loss function used by the teacher model in step S1 of this embodiment during the training process of the source domain training set includes classification loss. and border regression loss Removed prototype alignment loss .

[0079] The invention will be verified by specific experiments in the following sections.

[0080] In this experiment, the source dataset used is the public dataset COCO, and the images all have multi-class bounding box annotation information, which can provide sufficient and diverse sample support for the construction of class prototype vectors. There are two target domain datasets, OpenImages and Cityscapes, both of which are unlabeled images used to simulate real-world cross-domain unsupervised classification scenarios and to test the adaptability of the method on different target domains.

[0081] Specifically, the target domain dataset was divided into a target domain training set and a target domain test set in a 2:8 ratio. During the experiment, the model was trained using the target domain training set, and after the model was trained, its performance was tested on the target domain test set.

[0082] The teacher and student models use the Faster R-CNN network.

[0083] The experimental setup is as follows: Hardware environment: NVIDIA GPU (A100), CUDA 11.6; Optimizer and parameters: AdamW, learning rate 1e-4, batch size 32 (i.e., 32 random pseudo-label samples are drawn each time during the training of the student model), training epoch 100 (i.e., the student model is considered to have converged after 100 iterations after each pseudo-label pool update).

[0084] This experiment is divided into a comparative experiment and an ablation experiment. In the comparative experiment, the student model trained in Example 1 (hereinafter referred to as the model of this invention) is compared with two comparative models.

[0085] The two comparison models are as follows: Comparison Model 1: The model trained using only source domain supervision; specifically, the Faster R-CNN network is trained 100 times on the source domain dataset, and then directly transferred to the target domain test set for image classification. Comparison Model 2: Conventional unsupervised domain adaptive classification method, which directly trains the Faster R-CNN network 100 times on the target domain training set using an unsupervised method, and then uses it for classification on the target domain test set.

[0086] The evaluation metrics for the comparative experiment included mAP, classification accuracy, and F1 score. All three models used the class with the highest probability in the output predicted probability distribution as the target class. The evaluation metrics are shown in Table 1 below.

[0087] It is worth noting that the evaluation metric in the comparative experiment was the average value obtained from the two target domain datasets.

[0088] On the cross-domain task COCO→OpenImages, the teacher model (Faster R-CNN network) trained on the source domain dataset is used to extract the target region and distinguishing features of the target domain image; In the cross-domain task COCO→Cityscapes, the SAM model is used to extract salient regions as target regions, and the mask features of the salient regions are used as region features.

[0089] Table 1: Statistical Analysis of Comparative Experiment Results ; As can be seen, compared with source domain supervised training methods, the model of this invention improves the performance of the three metrics by 14.6%, 13.5%, and 0.14, respectively, which is a significant improvement. Compared with conventional unsupervised domain adaptation methods, the model of this invention improves the performance of the three metrics by 8.3%, 6.7%, and 0.08, respectively. Therefore, compared with conventional unsupervised domain adaptation, the method of this invention improves the quality of pseudo-labels and has unique advantages in cross-modal alignment and stability constraints, thus exhibiting better transferability and robustness in different task environments.

[0090] To further verify the universality of the method of this invention, comparative experiments were conducted on two cross-domain tasks: COCO→OpenImages and COCO→Cityscapes. The evaluation index was mAP@0.5, and the experimental results are shown in Table 2 below.

[0091] Table 2: Cross-domain comparison experiment ; Note: Although the absolute mAP values ​​of the model were generally low on the two cross-domain tasks COCO→OpenImages and COCO→Cityscapes, this was due to a combination of factors, including incomplete alignment of class spaces, significant differences in data distribution, differences in task definitions, and long-tail distribution issues. Even under such stringent cross-domain conditions, the method of this invention still achieved a significant relative improvement over the comparison methods, fully demonstrating the advantages of this method in pseudo-label optimization, cross-modal semantic alignment, and robustness enhancement.

[0092] Compared to the comparative model 1, the model of this invention improves the mAP by 13.9 in OpenImages and by 12.0 in Cityscapes, verifying the significant advantage of the model of this invention in cross-domain image classification.

[0093] Compared to Comparative Model 2, the model of this invention further integrates pseudo-label pooling, cross-modal alignment, and a teacher-student dual network, achieving optimal performance in both target domains. This experiment fully demonstrates the leading-edge nature and universality of the proposed solution in unsupervised adaptive scenarios.

[0094] This experiment further compares the various models in Examples 1-6 on the evaluation metric mAP, and the results are shown in Table 3; the evaluation metric in this comparative experiment is the average value of the two target domain datasets.

[0095] Table 3: Statistical Analysis of Comparative Experiment Results ; As shown in Table 3, the model of the present invention, namely Example 1, has the best performance. The models in Examples 2-6 are slightly worse than Example 1, but are still much better than the comparative model.

[0096] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0097] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0098] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A weakly supervised target detection method guided by cross-modal pseudo-labels, characterized in that, Includes the following steps: A labeled source domain dataset is constructed, an image classification teacher model is trained, and a teacher-student network structure is built. Clustering of regional features in the target domain image; By optimizing the optimal cross-modal assignment relationship between source domain categories and target domain clusters, pseudo-labels are assigned to each cluster to form a pseudo-label pool; a high-confidence pseudo-label sample set and a low-confidence pseudo-label sample set are initialized. Train the student model on the pseudo-label pool, and use the converged student model to update the pseudo-label pool. Update the high-confidence pseudo-label sample set with pseudo-label samples in the pseudo-label pool that meet the constraints, and update the low-confidence pseudo-label samples with pseudo-label samples in the pseudo-label pool that do not meet the constraints. The constraints are: the standard deviation of the predicted probability distribution and the variance of the historical category probability sequence are both lower than the set threshold; the historical category probability sequence is the sequence of pseudo-label sample category probabilities at each pseudo-label pool update; Repeatedly train the student model until the pseudo-label pool overlaps with the high-confidence pseudo-label sample set, then output the student model to detect the region category of the target domain image.

2. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 1, characterized in that, Before training the student model on the pseudo-label pool for the first time, the following steps were taken to optimize the pseudo-label pool: The confidence level of pseudo-label samples is evaluated, and they are divided into a high-confidence pseudo-label sample set and a low-confidence pseudo-label sample set. An auxiliary network is trained on the high-confidence pseudo-label sample set to correct the low-confidence pseudo-label samples and update the pseudo-label pool.

3. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 1 or 2, characterized in that, The optimization objective of the optimal cross-modal assignment relationship between the source domain categories and the target domain clusters in step S2 is: ； Transmission matrix T= C represents the number of source domain categories, and K represents the number of target domain clusters. To assign weights from source domain category j to target domain cluster k, for each target domain cluster k, select... The source domain category j corresponding to the maximum value is used as the pseudo-label, and a pseudo-label pool is constructed. ； ； in, The cost of the source domain category j and the target domain cluster k; Let be the text modal label prototype vector of source domain category j. The cluster center of cluster k in the target domain; The attention cost weight; The semantic deviation term is the label prototype vector of the source domain category j to the cluster center of the target domain cluster k. The allocation entropy for cluster k in the target domain; is the entropy regularization weight; the label prototype vector is the mean of the image and text embedding representations of all labeled regions under the same label.

4. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 1 or 2, characterized in that, The auxiliary network on the high-confidence pseudo-label sample set The loss function for training is: ； in: Samples for auxiliary network prediction The maximum probability of the category distribution; The positive and negative sample balance factor is set. The difficulty adjustment factor is set. The confidence weighting function is set. , To set a constant, and .

5. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 4, characterized in that, Loss function of student model during training on pseudo-label pool It includes one or more of the following: source domain supervision loss, focus loss, transport alignment loss, cross-round stability regularization term and multi-granularity confidence alignment total loss. The cross-round stability regularization term is represented by the KL divergence of the predicted probability distribution of the corresponding converged student model after the two most recent iterations of the pseudo-label pool.

6. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 5, characterized in that, The total alignment loss includes one or more of the following: category information entropy, instance classification loss, feature residual penalty loss, image-level KL alignment loss, and image entropy regularization term.

7. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 6, characterized in that, The total alignment loss also includes the KL alignment loss of the structural regions. The source dataset consists of {images with labeled borders, border labels; labeled region masks, label prototype vectors}. ； in, For the first image in the source domain The salience weight of each labeled region in the structural mask Let i be the region feature i of the target domain image, predicted by the teacher model. Let i be the maximum probability predicted by the student model for the region feature i of the target domain image.

8. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 7, characterized in that, The method for obtaining the labeled region mask is as follows: use a pre-trained SAM model to perform foreground segmentation on the image, and extract the binary mask of the labeled region as the labeled region mask.

9. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 7, characterized in that, Total alignment loss It is a weighted sum of category information entropy, instance classification loss, feature residual penalty loss, image-level KL alignment loss, image entropy regularization term and structural region KL alignment loss, and the weights of each sub-item loss are dynamically adjusted.

10. The cross-modal pseudo-label-guided weakly supervised target detection method as described in claim 6, characterized in that, Feature residual penalty loss and instance classification loss The calculation formula is as follows: ； ； in, For instance feature residuals, The image features of pseudo-label pool sample i after passing through the visual encoder. The image-text embedding representation is obtained by modal fusion of the image features of the pseudo-label pool sample i after passing through the visual encoder and the text features of the diverse semantic description text after passing through the text encoder; Represents the L2 norm; ； ； ； in, The weights of sample i in the pseudo-label pool. For cross-entropy loss, The class probability of pseudo-label pool sample i predicted by the student model. Let i be the label category of sample i in the pseudo-label pool; For activation functions; The set boundary sensitivity adjustment parameters; This represents the confidence boundary difference between the predicted probability and the second-highest probability of a sample in the pseudo-label pool. The pseudo-label category of sample i in the pseudo-label pool after passing through the student model. The corresponding probability, The probability of sample i in the pseudo-label pool being the second highest class probability after passing through the student model.

Images