Cross-modal label pooling and classification methods for weakly supervised training tasks
By constructing a cross-modal label pool and a pseudo-label pool, and optimizing the allocation and confidence of pseudo-labels, the problem of poor detection accuracy in cross-domain classification is solved, and the robustness and stability of weakly supervised target detection are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DATA SPACE RES INST
- Filing Date
- 2025-10-16
- Publication Date
- 2026-05-26
AI Technical Summary
Existing weakly supervised object detection methods face challenges such as inconsistent data distribution, insufficient samples, and domain differences in cross-domain classification, resulting in poor detection accuracy.
By constructing a cross-modal label pool and combining the domain alignment idea of cross-domain classification, the pseudo-label pool is optimized. The transfer matrix T is used to allocate weights between the source domain category and the target domain cluster. The pseudo-label pool is constructed and updated by the confidence prediction model, thus optimizing the training process of the target domain diagnostic model.
It effectively reduces pseudo-label category mismatch, enhances the robustness of pseudo-supervised training, and improves the training stability and detection performance of the target domain diagnostic model.
Smart Images
Figure CN121415045B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and multimodal learning technology, and in particular to a method for constructing and classifying cross-modal label pools for weakly supervised training tasks. Background Technology
[0002] Object detection is a core task in computer vision, aiming to identify the location and category of objects in an image. Traditional object detection methods rely on large amounts of precisely labeled data, which is often expensive and time-consuming in practical applications. Weakly supervised object detection (WSOD) provides a solution that allows for effective object detection using small amounts of or inaccurate labeled data.
[0003] In weakly supervised object detection tasks, traditional methods usually rely on image-level labels, pseudo-label generation, or pre-trained models to assist in training, in order to reduce the dependence on manually labeled bounding box data, but the labeling accuracy is difficult to guarantee.
[0004] Cross-domain classification refers to classification tasks performed across different domains or data distributions, typically involving differences in feature distributions between the source and target domains. Due to the inconsistency in distributions between domains, cross-domain classification faces numerous challenges, such as data distribution shifts, insufficient samples, and domain differences. Summary of the Invention
[0005] To overcome the shortcomings of poor accuracy in weakly supervised target detection tasks in the existing technologies, this invention proposes a cross-modal label pooling method for weakly supervised training tasks. It optimizes the pseudo-label pool by combining the domain alignment idea of cross-domain classification, thus laying a sample foundation for weakly supervised training.
[0006] This invention proposes a method for constructing a cross-modal label pool for weakly supervised training tasks, comprising the following steps:
[0007] S1. Construct a labeled source domain dataset {images with labeled borders, border labels; label prototype vectors};
[0008] S2. Cluster the regional features of the target domain image and obtain the cluster centers;
[0009] S3. Optimize the transfer matrix T according to the optimization objective. ;
[0010] The optimization objective is: ;
[0011] in, The weights assigned from source domain category j to target domain cluster k; C is the number of source categories; K is the number of target domain clusters; The cost of the source domain category j and the target domain cluster k; The allocation entropy for cluster k in the target domain; Entropy regularization weights;
[0012] S4. Assign pseudo-labels to each cluster based on the transfer matrix T, and construct a pseudo-label pool {regional features of the target domain image, pseudo-labels}.
[0013] Preferred; ;
[0014] ;
[0015] in, Let be the prototype vector of the text modal label of the source domain category j; Let k be the cluster center of the target domain cluster cluster k; This is the semantic bias term from source domain category j to target domain cluster k, calculated by the cross-modal attention mechanism; This represents the attention cost weight.
[0016] Preferred semantic bias term The method for obtaining the similarity is as follows: project the label prototype vector of the source domain category j and the cluster center of the target domain cluster k onto a unified semantic space to calculate the similarity, and then normalize it to obtain the attention weight. The difference between the two is then weighted by the attention weight.
[0017] Preferably, after step S4, the method further includes: using a pre-trained confidence prediction model to predict the confidence level of pseudo-label samples in the pseudo-label pool, and segmenting the high-confidence pseudo-label sample set. And low-confidence pseudo-labeled samples, and then high-confidence pseudo-labeled sample set The auxiliary network is trained to update the pseudo-labels of low-confidence pseudo-label samples in the pseudo-label pool.
[0018] The present invention proposes a cross-modal classification method, characterized by comprising the following steps:
[0019] First, a pseudo-label pool is constructed using the cross-modal label pooling method described above for weakly supervised training tasks, and the target domain diagnostic model is trained on the pseudo-label pool.
[0020] Then, a target detection algorithm is used to label the regional features of the target domain image to be diagnosed, and the regional features are input into the target domain diagnostic model to obtain pseudo-labels.
[0021] The present invention proposes a cross-modal classification method, comprising the following steps:
[0022] Train the target domain diagnostic model on the pseudo-label pool {regional features of the target domain image, pseudo-labels} until convergence; during the model iteration, calculate the variance of the historical category probability sequence for each sample in the pseudo-label pool.
[0023] The standard deviation of the predicted probability distribution of each sample in the pseudo-label pool is used as the pseudo-label sample temperature by the converged target domain diagnostic model.
[0024] The constraints for constructing pseudo-label samples are: the variance of the historical category probability sequence and the temperature of the pseudo-label samples are both less than the corresponding set thresholds;
[0025] The target domain diagnostic model is used to update the pseudo-labels of samples in the pseudo-label pool that do not meet the constraints.
[0026] Repeat the above steps until all samples in the pseudo-label pool meet the constraints. Then output the target domain diagnostic model, which is used for the differential feature classification of the target domain image.
[0027] Preferably, when performing fault diagnosis on a target domain image, a target detection algorithm is first used to label the regional features of the target domain image to be diagnosed, and the regional features are input into the target domain diagnosis model to obtain pseudo-labels.
[0028] The present invention proposes a cross-modal classification system, including a memory and a processor. The memory stores a computer program, and the processor is connected to the memory. The processor is used to execute the computer program to implement the cross-modal classification method.
[0029] The present invention proposes a storage medium storing a computer program, which, when executed, is used to implement the cross-modal classification method.
[0030] The advantages of this invention are:
[0031] (1) By introducing the allocation between the source domain category and the target domain cluster, the present invention achieves probability allocation alignment between the source domain category and the target domain feature cluster, effectively avoiding pseudo-label category mismatch, improving the coverage of low resource categories, and enhancing the robustness of overall pseudo-supervised training.
[0032] (2) The present invention introduces pseudo-label confidence to update the pseudo-label pool, which reduces training interference caused by low-confidence pseudo-labels and helps to enhance the confidence control of pseudo-labels, thereby increasing the stability of the training effect of the target domain diagnostic model.
[0033] (3) In the training process of the target domain diagnostic model given in this invention, the model training is linked with the pseudo-label pool to construct a feedback learning process, so that the model can gradually converge and continuously optimize its pseudo-label quality and detection performance. Attached Figure Description
[0034] Figure 1 Here is a flowchart of a pseudo-label pool construction method for weakly supervised training tasks proposed in Example 1;
[0035] Figure 2 Here is a flowchart of another pseudo-label pool construction method for weakly supervised training tasks proposed in Example 2;
[0036] Figure 3 This is a flowchart of a cross-modal target domain classification method proposed in Example 1. Detailed Implementation
[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0038] Example 1: A method for constructing a pseudo-label pool for weakly supervised training tasks
[0039] Reference Figure 1 This method includes the following steps:
[0040] S1. Construct a labeled source domain dataset {images with labeled borders, border labels; label prototype vectors}; the labeled region prototype vector is the image-text embedding representation vector of the labeled region, and the label prototype vector is the mean of all labeled region prototype vectors under the same label.
[0041] The source dataset can specifically use images from open datasets (such as PASCAL VOC, COCO, OpenImages, etc.); the bounding boxes can be labeled using object detection algorithms to identify the target region and the category of the target object.
[0042] The method for obtaining the prototype vector of the label is as follows: First, obtain the diverse semantic description text and text semantic features of each labeled region on the image. Then, align the image feature encoding and text semantic features of the labeled region modally to generate the image-text embedding representation of each labeled region. Take the average of the image-text embedding representations of the same type of labeled region as the prototype vector of that label category.
[0043] In practice, block coding can be used to obtain the image feature codes of each labeled region on the image. First, a visual encoder (e.g., ViT-B / 16, ResNet50) is used to encode the labeled regions of the image in blocks. Then, the ROIAlign operation is performed on each labeled region to extract spatial feature vectors as image feature encodings. .
[0044] The method for obtaining diverse semantic description text and text semantic features of the labeled regions is as follows: Class name descriptions are constructed for the border labels, and diverse semantic description texts of the class names are generated using VLMs (such as BLIP and CLIP); then, text encoders (such as BERT, T5, and GPT2) are used to embed the diverse semantic description texts to obtain the text semantic features. ;
[0045] In this embodiment, the image features of each labeled region are encoded using the Cross-Modal Transformer module. and text semantic features r Modal alignment is performed to generate image-text embeddings. The Cross-Modal Transformer module employs a contrastive learning strategy or distance constraints (such as InfoNCE or Triplet Loss) to bring image-text pairs with the same semantics closer together in the embedding space, while keeping samples with different semantics spaced apart.
[0046] S2. Cluster the regional features of the target domain image and obtain the cluster centers;
[0047] The cluster center is the average feature vector of the regional features of all target domain images in the cluster.
[0048] In practical implementation, Mini-Batch KMeans can be used to cluster the regional features of the target domain image, generating K clusters, with the set of cluster centers as follows: ; Let K denote the cluster center of cluster k in the target domain, where 1 ≤ k ≤ K; and These represent the cluster centers of the 1st and Kth clusters in the target domain, respectively.
[0049] Regional features of the target domain image can be extracted using pre-trained image recognition algorithms. In practice, when the difference between the source and target domains is small, a Faster R-CNN network can be used to identify the target region and extract its features. When the difference between the source and target domains is large, a segmentation model (SAM) can be used to extract salient mask regions and generate regional features. Alternatively, other image recognition algorithms can be used to extract regional features, as specified by the administrator.
[0050] S3. Optimize the optimal cross-modal allocation relationship between the source domain category and the target domain cluster, i.e., the transfer matrix.
[0051] The optimization objective of the transfer matrix T is:
[0052] in, The weights assigned from source domain category j to target domain cluster k, >0, and =1; T= C represents the number of originating categories; K represents the number of clusters in the target domain. The cost of the source domain category j and the target domain cluster k; The allocation entropy for cluster k in the target domain; This is the entropy regularization weight, with a typical value range of [0.01, 0.1]. The dimension of the transfer matrix T is . and all elements All are greater than 0.
[0053]
[0054] in, Let t be the text modality label prototype vector of source domain class j, that is, the text features t of all samples under source domain class j. r The average; Let k be the cluster center of the target domain cluster cluster k; This is the attention cost weight, with a value range of [0.1, 1), and is specifically set to 0.5 in subsequent embodiments;
[0055] The semantic bias term is calculated by the cross-modal attention mechanism from the source domain category j to the target domain cluster k. It is obtained by projecting the label prototype vector of the source domain category j and the cluster center of the target domain cluster k onto a unified semantic space, calculating the similarity between the two and normalizing it with softmax to obtain the attention weight. The difference measure (such as Euclidean distance or cosine distance) between the label prototype vector of the source domain category j and the cluster center of the target domain cluster k is weighted by this attention weight and used as the semantic bias term.
[0056]
[0057] Where C represents the number of source domain categories.
[0058] S4. Assign pseudo-labels to each cluster according to the transfer matrix T, thereby constructing a pseudo-label pool {regional features of the target domain image, pseudo-labels}.
[0059] Specifically, for each target domain cluster k, select The source domain category j corresponding to the maximum value is used as the pseudo-label, thus constructing a pseudo-label pool {regional features of the target domain image, pseudo-label}; the formula is expressed as:
[0060]
[0061] The pseudo-labels are clusters k in the target domain; this pseudo-label pool will be used for subsequent confidence prediction, auxiliary correction, and target domain diagnostic model training.
[0062] Example 2: A method for constructing a pseudo-label pool for weakly supervised training tasks
[0063] Reference Figure 2 Based on Example 1, this example uses a pre-trained confidence prediction model to predict the confidence level of pseudo-label samples in the pseudo-label pool, thus dividing the pseudo-label samples into a high-confidence pseudo-label sample set. And low-confidence pseudo-labeled samples; then in the high-confidence pseudo-labeled sample set The auxiliary network is trained to update the pseudo-labels of low-confidence pseudo-label samples in the pseudo-label pool in order to obtain a high-confidence pseudo-label pool.
[0064] The confidence prediction model can be a pre-trained CLIP model or other known models.
[0065] In practice, this section compares the model prediction confidence with a set confidence threshold, and classifies pseudo-label samples in the pseudo-label pool with a confidence level greater than or equal to the confidence threshold into a high-confidence pseudo-label sample set. Pseudo-labeled samples with a confidence level lower than the confidence threshold are classified as low-confidence pseudo-labeled samples. The confidence threshold can specifically take values in the range [0.6, 0.9], for example, it is set to 0.8 in the following embodiment.
[0066] Example 3: The First Cross-Modal Target Domain Classification Method
[0067] Includes the following steps:
[0068] First, a pseudo-label pool {regional features of the target domain image, pseudo-labels} is constructed using the pseudo-label pool construction method for weakly supervised training tasks given in Example 1. The target domain diagnostic model is then trained on the pseudo-label pool to diagnose the regional features of the target domain image.
[0069] During diagnosis, the image to be diagnosed is first acquired in the target domain. Then, the target region is labeled using a target detection algorithm, and the image features of the target domain are extracted and input into the target domain diagnostic model to obtain pseudo-labels for each target region.
[0070] Example 4: A second cross-modal target domain classification method
[0071] First, the pseudo-label pool {regional features of the target domain image, pseudo-labels} is constructed using the pseudo-label pool construction method for weakly supervised training tasks given in Example 2. The target domain diagnostic model is then trained on the pseudo-label pool to diagnose the regional features of the target domain image.
[0072] During diagnosis, the image to be diagnosed is first acquired in the target domain. Then, the target region is labeled using a target detection algorithm, and the image features of the target domain are extracted and input into the target domain diagnostic model to obtain pseudo-labels for each target region.
[0073] Example 5: The Third Cross-Modal Target Domain Classification Method
[0074] Reference Figure 3 This method includes the following steps:
[0075] St1. Train the target domain diagnostic model on the pseudo-label pool {regional features of the target domain image, pseudo-labels} until convergence;
[0076] In practice, in this step, training samples are randomly selected from the pseudo-label pool to iterate the target domain diagnostic model. Then, the target domain diagnostic model is used to traverse the pseudo-label pool, calculate the loss function, and update the target domain diagnostic model through the direction of the loss function. This process is repeated for a set number of rounds, and then the convergence of the target domain diagnostic model is determined.
[0077] The loss function used by the label domain diagnostic model during training on the pseudo-label pool is the cross-entropy loss between predicted pseudo-labels and known pseudo-labels.
[0078] During the iterative process of the target domain diagnostic model, the class probability of pseudo-label sample i is statistically analyzed each time the pseudo-label pool is traversed. This is the maximum predicted probability output by the target domain diagnostic model, forming a historical class probability sequence of pseudo-label sample i. The variance of this sequence is denoted as Var(p i ).
[0079] St2. Use the converged target domain diagnostic model to predict the predicted probability of each sample in the pseudo-label pool, and determine whether there are pseudo-label samples in the pseudo-label pool that do not meet the following constraints.
[0080] The constraints are:
[0081] and
[0082] in, The temperature of pseudo-labeled sample i is represented by the standard deviation of the predicted probability distribution of the target domain diagnostic model after convergence. and For setting value, The value range is [0.05, 0.2]. The value range is [0.005, 0.05].
[0083] If yes, then for pseudo-labeled samples that do not meet the constraints, use the target domain diagnostic model to update their pseudo-labels, update the pseudo-label pool, and then return to step St1;
[0084] No, the fixed target domain diagnostic model is used for fault diagnosis of regional features of the target domain image; at the same time, the pseudo-label pool can also be fixed as a backup.
[0085] In practical applications of the target domain diagnostic model, for the target domain image to be detected, the target region and its features are first extracted using an image recognition algorithm. Then, the target domain diagnostic model generates pseudo-labels for each target region based on these features. Specifically, the image recognition algorithm used in the actual application and the pseudo-label pool construction should be consistent.
[0086] The invention will be verified by specific experiments in the following sections.
[0087] In this experiment, the source dataset used is the public dataset COCO, and the images all have multi-class bounding box annotation information, which can provide sufficient and diverse sample support for the construction of class prototype vectors.
[0088] There are two target domain datasets, OpenImages and Cityscapes, both of which are unlabeled images used to simulate real-world cross-domain unsupervised classification scenarios and to test the adaptability of the method on different target domains.
[0089] Specifically, the target domain dataset was divided into a target domain training set and a target domain test set in a 2:8 ratio. During the experiment, the model was trained using the target domain training set, and after the model was trained, its performance was tested on the target domain test set.
[0090] The models in Examples 3-5 use the Faster R-CNN network.
[0091] The experimental setup is as follows:
[0092] Hardware environment: NVIDIA GPU (A100), CUDA 11.6;
[0093] Optimizer and parameters: AdamW, learning rate 1e-4, batch size 32 (i.e., 32 random pseudo-label samples are drawn each time during model training), training epoch 100 (i.e., after each pseudo-label pool update, the target domain diagnostic model is considered to have converged after 100 iterations).
[0094] In the comparative experiment, the target domain diagnostic models trained in Examples 3-5 were compared with two comparative models. The target domain diagnostic model obtained in Example 3 is denoted as Model 1 of this invention, the target domain diagnostic model obtained in Example 4 is denoted as Model 2 of this invention, and the target domain diagnostic model obtained in Example 5 is denoted as Model 3 of this invention. In this experiment, comparative model 3 was trained on the pseudo-label pool obtained in Example 2 using the method of Example 5.
[0095] It is worth noting that in each round of pseudo-label pooling in Examples 3-4 and Example 5, during the training of the target domain diagnostic model, 5 training samples are extracted each time, and the number of rounds is set to 100. That is, during the training of the target domain diagnostic model, 5 training samples are extracted in each batch to update the model; then the pseudo-label pool is traversed to calculate the loss function, and the target domain diagnostic model is updated through the direction of the loss function; this is repeated 100 times, and then the target domain diagnostic model is considered to have converged. The loss function adopts the cross-entropy loss between the predicted pseudo-label and the known pseudo-label.
[0096] The two comparison models are as follows:
[0097] Comparison Model 1: The model trained using only source domain supervision; specifically, the Faster R-CNN network is trained 100 times on the source domain dataset, and then directly transferred to the target domain test set for image classification.
[0098] Comparison Model 2: Conventional unsupervised domain adaptive classification method, which directly trains the Faster R-CNN network 100 times on the target domain training set using an unsupervised method, and then uses it for classification on the target domain test set.
[0099] Both the comparative model and the experimental model used the category with the highest probability in the output predicted probability distribution as the target category. The evaluation metric was mAP@0.5. The experimental results are shown in Table 2 below.
[0100] On the cross-domain task COCO→OpenImages, the source domain diagnostic model (Faster R-CNN network) trained on the source domain dataset is used to extract the target region and distinguishing features of the target domain image;
[0101] In the cross-domain task COCO→Cityscapes, the trained SAM model is used to extract salient regions as target regions, and the mask features of the salient regions are used as region features.
[0102] Table 2: Cross-domain comparison experiment
[0103]
[0104] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0105] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
[0106] The technologies, shapes, and structures not described in detail in this invention are all known technologies.
Claims
1. A method for constructing a cross-modal label pool for weakly supervised training tasks, characterized in that, Includes the following steps: S1. Construct a labeled source domain dataset {images with labeled borders, border labels; label prototype vectors}; S2. Cluster the regional features of the target domain image and obtain the cluster centers; S3. Optimize the transfer matrix T according to the optimization objective. ; The optimization objective is: ; in, The weights assigned from source domain category j to target domain cluster k; C is the number of source categories; K is the number of target domain clusters; The cost of the source domain category j and the target domain cluster k; The allocation entropy for cluster k in the target domain; Entropy regularization weights; S4. Assign pseudo-labels to each cluster based on the transfer matrix T, and construct a pseudo-label pool {regional features of the target domain image, pseudo-labels}.
2. The method for constructing a cross-modal label pool for weakly supervised training tasks as described in claim 1, characterized in that; ； ； in, Let be the prototype vector of the text modal label of the source domain category j; The cluster center of cluster k in the target domain; This is the semantic bias term from source domain category j to target domain cluster k, calculated by the cross-modal attention mechanism; This represents the attention cost weight.
3. The method for constructing a cross-modal label pool for weakly supervised training tasks as described in claim 2, characterized in that, Semantic bias term The method for obtaining the similarity is as follows: project the label prototype vector of the source domain category j and the cluster center of the target domain cluster k onto a unified semantic space to calculate the similarity, and then normalize it to obtain the attention weight. The difference between the two is then weighted by the attention weight.
4. The method for constructing a cross-modal label pool for weakly supervised training tasks as described in claim 1, characterized in that, Step S4 is followed by: using a pre-trained confidence prediction model to predict the confidence of pseudo-label samples in the pseudo-label pool, and segmenting the high-confidence pseudo-label sample set. And low-confidence pseudo-labeled samples, and then high-confidence pseudo-labeled sample set The auxiliary network is trained to update the pseudo-labels of low-confidence pseudo-label samples in the pseudo-label pool.
5. A cross-modal classification method employing the cross-modal label pooling construction method for weakly supervised training tasks as described in any one of claims 1-4, characterized in that, Includes the following steps: First, a pseudo-label pool is constructed using the cross-modal label pooling construction method for weakly supervised training tasks as described in any one of claims 1-4, and the target domain diagnostic model is trained on the pseudo-label pool. Then, a target detection algorithm is used to label the regional features of the target domain image to be diagnosed, and the regional features are input into the target domain diagnostic model to obtain pseudo-labels.
6. A cross-modal classification method employing the cross-modal label pooling construction method for weakly supervised training tasks as described in any one of claims 1-4, characterized in that, Includes the following steps: Train the target domain diagnostic model on the pseudo-label pool {regional features of the target domain image, pseudo-labels} until convergence; during the model iteration, calculate the variance of the historical category probability sequence for each sample in the pseudo-label pool. The standard deviation of the predicted probability distribution of each sample in the pseudo-label pool is used as the pseudo-label sample temperature by the converged target domain diagnostic model. The constraints for constructing pseudo-label samples are: the variance of the historical category probability sequence and the temperature of the pseudo-label samples are both less than the corresponding set thresholds; The target domain diagnostic model is used to update the pseudo-labels of samples in the pseudo-label pool that do not meet the constraints. Repeat the above steps until all samples in the pseudo-label pool meet the constraints. Then output the target domain diagnostic model, which is used for the differential feature classification of the target domain image.
7. The cross-modal classification method as described in claim 6, characterized in that, When performing fault diagnosis on a target domain image, the first step is to use a target detection algorithm to label the regional features of the target domain image to be diagnosed, input the regional features into the target domain diagnosis model, and obtain pseudo-labels.
8. A cross-modal classification system, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, the processor is connected to the memory, and the processor is used to execute the computer program to implement the cross-modal classification method as described in claim 5, 6, or 7.
9. A storage medium, characterized in that, The system contains a computer program that, when executed, is used to implement the cross-modal classification method as described in claim 5, 6, or 7.