Weakly supervised multi-label out-of-distribution sample classification detection method

Abstract

The application relates to a weakly supervised multi-label out-of-distribution sample classification detection method. In the training stage, all unknown labels are assumed to be negative labels, noise labels are introduced, mixed data enhancement preprocessing is carried out on the image, noise label loss effect is used to identify noise labels causing large loss values, the noise labels are adjusted for model training, and label sparsity is combined to enable the method to adaptively identify noise label loss values in different training stages. In the detection stage, all in-distribution samples and out-of-distribution samples are given full negative labels, noise labels of in-distribution samples are identified by combining noise label effect and label sparsity, and the detection of out-of-distribution samples is realized.

Description

Technical Field

[0001] This invention relates to the field of image classification, and in particular to a weakly supervised multi-label distribution out-of-sample classification and detection method. Background Technology

[0002] There are three typical types of weak supervision information. The first is incomplete or semi-supervised supervision. In this case, only a very small portion of the training data is fully labeled, while the rest is unlabeled. A common application scenario is to obtain a small portion of fully supervised information through manual annotation, then acquire a large amount of unlabeled data from the internet, and finally use both simultaneously for model training. The second is coarse-grained or imprecise supervision. This refers to all training data having coarse-grained supervision information. For example, it might provide image-level classification supervision information only for object localization or segmentation tasks, rather than object-level or pixel-level supervision information. The third is inaccurate or partial supervision. Each training example in the training data has supervision information, but this information is incomplete or contains noise. For example, in a multi-label image classification task, each multi-label image is only given labeling information for one category, or the given labeling information contains some errors. Figure 1 It shows different levels of supervision information for a multi-labeled image.

[0003] For the third type of weakly supervised information, there are two common approaches: the first is to directly ignore all unknown label information and only use existing labels to guide the model's learning; the second approach assumes all unknown labels are negative, which can obtain a large amount of effective supervision information, but also introduces noisy labels, which may impair the model's training performance. Since the former loses a large amount of potential supervision information, the latter approach is preferred for handling unknown label information. Furthermore, if noisy labels can be effectively identified and processed, the negative impact of noisy labels on model training can be significantly reduced. Arpit et al. discovered an interesting phenomenon regarding noisy labels: deep learning networks can learn not only real data features but also memorize noisy data. However, the network tends to prioritize learning clean data before memorizing noisy data; the loss value corresponding to clean labels decreases first, followed by the loss value corresponding to noisy labels. Kim et al. found that this loss effect also exists in multi-label scenarios. In the field of single-label image classification, this characteristic has been fully explored and widely applied, achieving remarkable results.

[0004] Existing multi-label out-of-sample detection methods require training data to have complete fully supervised information. However, such training data requires a high amount of labeling, especially in fields with high technical requirements. Labeling is slow and difficult, and the high cost of labeling becomes prohibitive. Summary of the Invention

[0005] To address the issue that existing multi-label out-of-distribution sample detection methods require fully supervised training data, this invention proposes an out-of-distribution sample detection method based on the noise label loss effect. During the training phase, this method utilizes the noise label loss effect under weak supervision to guide the model's parameter learning, while simultaneously leveraging the inherent sparsity of the multi-label data label space to adaptively adjust for the abnormal loss caused by noise labels. Finally, in the inference phase, the trained model can achieve multi-label out-of-distribution sample detection under weak supervision.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a weakly supervised multi-label distribution out-of-sample classification detection method.

[0007] The testing method includes the following steps:

[0008] S1: Preprocessing of original image labels, for multi-labeled images x i Its original tag There is only one of them. Set all its unknown annotation information to negative labels, x i After the raw weakly supervised information is preprocessed using the negative label hypothesis, x i The label becomes

[0009]

[0010] S2: Training the CNN model:

[0011] The image after S1 label preprocessing and its label constitute the unenhanced image. Hybrid data augmentation preprocessing is performed on all unenhanced images to generate enhanced images. Then enhance the image The labels of the corresponding unenhanced images are used as training samples. All training samples constitute the training set;

[0012] The samples in the training set are output to the CNN in batches for training. The training ends when the value of the loss function no longer decreases, and the trained model is obtained.

[0013] S3: Use the trained model to detect the target image.

[0014] The target image is input into the trained model, which then classifies and predicts it, providing the prediction result y. i Then use the prediction result y i With artificially assigned full negative labels Calculate the multi-label classification loss:

[0015]

[0016] Where L is the number of categories in the label space. This indicates a manually assigned all-negative label, with all values ​​being 0;

[0017] The fully negative label loss value is calculated based on equation (1.2). Subsequently, by identifying excessive loss values ​​caused by noise labels, the detection score for out-of-distribution samples is calculated:

[0018]

[0019] in

[0020] If score < λ, then the target image is an out-of-distribution sample; otherwise, it is an in-distribution sample. i Indicates the penalty for loss;

[0021] Preferably, the hybrid data augmentation preprocessing in S2 is as follows:

[0022] from Another unenhanced image is randomly sampled from the training batch. Then randomly generate a cropping box {z} h ,z v ,w,h}, where z h With z v These represent the cropping box in image x. A and image x B The x and y coordinates of the top-left starting point in the image are used to mark the starting position of the cropping box, while w and h represent the width and height of the cropping box, respectively, used to mark its size. After obtaining the cropping box, it is first extracted from the image x... B crop out the image region reg B The starting position and size of the image region are consistent with the cropping frame, and then the image region is reg B Randomly perform existing data augmentation operations to obtain image regions reg B Finally, based on the previously obtained cropping box, in image x... A Mark the image region to be masked (reg) A Then use image region reg B Replace the image region to be masked (reg) A Generate the final enhanced image. The above process can be formally represented as:

[0023]

[0024] Where M∈{0,1}w×h It is a binary matrix where a value of 0 indicates that the pixel value at the corresponding position in the image is discarded, a value of 1 indicates that the pixel value at the corresponding position is retained, and 1 is a matrix with all elements equal to 1.

[0025] Preferably, the loss function used in S2 is As shown in equation (1.5):

[0026]

[0027] Where λ i This is a loss penalty term, whose value is in the interval [0,1]. At that time, α i =1, at this time the corresponding label is an accurate positive label and no correction is needed; conversely, when And corresponding to l i If the value is greater than the loss threshold δ, it means that the corresponding label is more likely to be a noise label. In this case, let α... i =0 indicates that the loss value is discarded;

[0028] The possible values ​​for the loss penalty term are as follows:

[0029]

[0030] Where δ is the loss threshold.

[0031] Preferably, the loss threshold adjustment method is as follows: select the largest (∈×100)% loss value in the current training batch.

[0032] The minimum value in the range is used as the value of the loss threshold δ:

[0033] δ=min(top ∈ (l i )),#(1.7)

[0034] Where ∈ is the sparsity control parameter, a hyperparameter used to measure the sparsity of the labels in the dataset.

[0035] Compared with the prior art, the present invention has at least the following advantages:

[0036] During the training phase, all unknown labels are initially assumed to be negative, introducing noise labels. Then, mixed data augmentation preprocessing is performed on the images to significantly suppress excessively high confidence levels in out-of-distribution samples. Next, based on the noise label loss effect, large loss values ​​caused by noise labels are identified and adjusted for model training. This, combined with label sparsity, allows the algorithm to adaptively identify noise label loss values ​​at different training stages, reducing the constraints of training data annotation information. In the inference phase, both in-distribution and out-of-distribution samples are assigned entirely negative labels. By combining the noise label effect and label sparsity, in-distribution samples with noise labels are identified, thus enabling the detection of out-of-distribution samples. Attached Figure Description

[0037] Figure 1 Given multi-labeled images under different supervision intensities, where √ indicates the presence of a corresponding category in the image, × indicates the absence of a corresponding category, and “?” indicates that the existence of a corresponding category is unknown. (a) Full supervision information given all positive and negative labels; (b) Weak supervision information given some positive and some negative labels; (c) Weak supervision information with one and only one positive label.

[0038] Figure 2 This is an overview diagram of the WSML-OOD method of the present invention.

[0039] Figure 3 Enhance the processing of mixed data.

[0040] Figure 4 The graphs show the true positive rate (TPR) of in-distribution samples as a function of λ on the PascalVOC2012 dataset (a), the MicrosoftCOCO dataset (b), and the NUS-WIDE dataset (c). When the TPR reaches 95%, the corresponding λ is used as the decision threshold for the weakly supervised multi-label out-of-distribution sample detector. Detailed Implementation

[0041] The present invention will now be described in further detail.

[0042] The method of this invention is abbreviated as WSML-OOD. In addition, the terms "in-distribution data" and "out-of-distribution data" are explained as follows: Assuming that the dataset has a total of Q categories, the data consisting of all images belonging to these Q categories is in-distribution data, and the data consisting of the remaining images that do not belong to any of the Q categories is out-of-distribution data.

[0043] Given image data Its corresponding tag space is Where L is the number of categories in the label space. Let w represent the set of real numbers, h represent the image width, c represent the image height, and u represent the unknown image label. This represents the tag space composed of positive tags. The tag space represents the unknown tags. Let j represent the j-th label; Represents the label space consisting of negative labels. Under weak supervision with a single positive label, for any multi-label image x i Its original tag There is only one of them. This means that there is a target object of category k in this example (k can be any real integer in the range of 1 to L), and the rest... All of these are unknown annotation information.

[0044] To rationally process and effectively utilize this type of label information, WSML-OOD employs a label preprocessing step of "Assume Negative (AN)". By assuming all unknown labels are negative, it obtains supervision information that simultaneously includes the true label, the true negative label, and the false positive label. Simultaneously, to mitigate overfitting and overconfidence in the model and reduce the negative impact of prediction results on out-of-distribution sample detection, WSML-OOD proposes a "Mixed Augmentation (MixAug)" image preprocessing step to augment the multi-label image data. During the training phase, based on the noise label loss effect and label sparsity, the large loss value caused by noise labels is iteratively adjusted to guide the training to obtain a multi-label image classification model. Finally, in the out-of-distribution sample detection stage, noise labels are artificially introduced by assigning all unknown samples full negative labels. Then, combining the noise label loss effect and label sparsity, out-of-distribution sample scores are obtained, achieving weakly supervised multi-label out-of-distribution sample detection.

[0045] See Figure 2 and Figure 3 A weakly supervised multi-label out-of-distribution sample classification and detection method includes the following steps:

[0046] S1: To maximize the use of existing supervisory information, we assume all unknown labels are negative, resulting in a subset of true negative labels and a subset of false negative labels. These unknown labels can be assumed to be either positive or negative because the sparsity of the label space is a natural property of multi-label data; the number of target classes appearing simultaneously in a multi-label image sample is always much smaller than the total number of classes in the label space. Therefore, assuming the unknown labels are negative ensures that the proportion of true labels is greater than the proportion of noisy labels.

[0047] Original image label preprocessing, for multi-label images x i Its original tag There is only one of them. Set all its unknown annotation information to negative labels, x i After the raw weakly supervised information is preprocessed using the negative label hypothesis, x i The label becomes

[0048]

[0049] By assuming the original unknown labeled information as negative labels, a large number of true negative labels and a small number of false negative labels are introduced. The true negative labels provide effective supervision information for model training, while the false negative labels, as noise labels, allow the model to make certain identifications based on the noise label loss effect and label sparsity, and process them from the perspective of loss values ​​for model training.

[0050] S2: Training the CNN model:

[0051] The image after S1 label preprocessing and its label constitute the unenhanced image. Hybrid data augmentation preprocessing is performed on all unenhanced images to generate enhanced images. Then enhance the image The labels of the corresponding unenhanced images are used as training samples. All training samples constitute the training set. Hybrid data augmentation is introduced as a data preprocessing technique to enhance the training data during training, aiming to suppress the classification model from overconfidently generating excessive confidence in unknown categories and reduce interference with the detection of out-of-distribution samples.

[0052] The samples in the training set are output to the CNN in batches for training. The training ends when the value of the loss function no longer decreases, and the trained model is obtained.

[0053] S3: Use the trained model to detect the target image:

[0054] During the detection phase, the input target image faced by the model can be either an in-distribution sample or an out-of-distribution sample. For in-distribution samples, the image contains target objects of several categories, and its true label has some positive labels. However, out-of-distribution sample images do not contain any target objects, therefore their true labels are all negative. Therefore, to detect out-of-distribution samples, the first step is to assign all negative labels to all inferred samples. This results in in-distribution samples containing some noise labels, while out-of-distribution samples contain no noise labels. Combined with the noise label loss effect and label sparsity, in-distribution and out-of-distribution samples can be effectively distinguished from a loss perspective.

[0055] The target image is input into the trained model, which then classifies and predicts it, providing the prediction result y. i Then use the prediction result y i With artificially assigned full negative labels Calculate the multi-label classification loss:

[0056]

[0057] According to the noise label loss effect, in-distribution samples with noise labels generate more and larger loss values ​​than out-of-distribution samples without any target category. At the same time, thanks to the suppression of model overconfidence by data augmentation, the difference between in-distribution samples and out-of-distribution samples in terms of loss value becomes more prominent.

[0058] Where L is the number of categories in the label space. This indicates a manually assigned all-negative label, with all values ​​being 0.

[0059] The fully negative label loss value is calculated based on equation (1.2). Subsequently, by identifying excessive loss values ​​caused by noise labels, the detection score for out-of-distribution samples is calculated:

[0060]

[0061] in

[0062] If score < λ, then the target image is an out-of-distribution sample; otherwise, it is an in-distribution sample. i This represents the loss penalty term. Unlike the training phase where excessively large loss values ​​are discarded when adjusting the loss for noise labels, the out-of-distribution sample detection step requires retaining large loss values ​​while discarding small loss values ​​to calculate the out-of-distribution sample detection score. This is because a large loss value indicates that the corresponding label is a false negative noise label, meaning the corresponding sample is an in-distribution sample and should therefore receive a higher out-of-distribution sample score.

[0063] Finally, the out-of-distribution sample detection method WSML-OOD based on the noise label loss effect proposed in this chapter can be formally expressed as:

[0064]

[0065] Where λ is the decision threshold for detecting out-of-distribution samples, which must ensure that the positive sample rate is greater than or equal to 95%.

[0066] Specifically, the hybrid data augmentation preprocessing process in S2 is as follows:

[0067] from Another unenhanced image is randomly sampled from the training batch. Then randomly generate a cropping box {z} h ,z v ,w,h}, where z h With z v These represent the cropping box in image x. A and image x B The x and y coordinates of the top-left starting point in the image are used to mark the starting position of the cropping box, while w and h represent the width and height of the cropping box, respectively, used to mark its size. After obtaining the cropping box, it is first extracted from the image x... B crop out the image region reg B The starting position and size of the image region are consistent with the cropping frame, and then the image region is reg B Randomly perform existing data augmentation operations [including horizontal flipping, vertical flipping, rotation, shifting, and Gaussian noise, etc.] to obtain the image region reg. B Finally, based on the previously obtained cropping box, in image x... A Mark the image region to be masked (reg) A Then use image region reg B Replace the image region to be masked (reg) A Generate the final enhanced image. The above process can be formally represented as:

[0068]

[0069] Where M∈{0,1} w×h It is a binary matrix where a value of 0 indicates that the pixel value at the corresponding position in the image is discarded, a value of 1 indicates that the pixel value at the corresponding position is retained, and 1 is a matrix with all elements equal to 1.

[0070] Specifically, during the training phase, by assuming all unknown labels in the weak supervision information of a single positive label are negative labels, the supervision information of a multi-label image becomes a combination of labels consisting of one true label, several true negative labels, and several false negative labels. Given the sparsity of the label space in multi-label data, the proportion of true negative labels is greater than that of false negative labels. The true and true negative labels in the label combination can correctly guide the multi-label image classification model in parameter learning; therefore, identifying and utilizing the noise labels—the false negative labels—is crucial.

[0071] During model training, noisy labels can cause excessively large loss values. By identifying such abnormal loss values, false negative labels can be easily identified. Furthermore, by processing such abnormal losses, the model can be guided to train correctly even when the relevant category label information is unknown.

[0072] Therefore, in order to correct the model training process containing noisy loss, the loss function in S2 As shown in equation (1.5):

[0073]

[0074] Where λ i This is a loss penalty term, whose value lies in the interval [0,1], when i∈y pos At that time, α i =1, at this time the corresponding label is an accurate positive label and no correction is needed; conversely, when i∈y un And corresponding to l i If the value is greater than the loss threshold δ, it means that the corresponding label is more likely to be a noise label. In this case, let α... i =0 indicates that the loss value is discarded to reduce the impact of noise labels on model training;

[0075] The possible values ​​for the loss penalty term are as follows:

[0076]

[0077] Where δ is the loss threshold.

[0078] Specifically, during the iterative training of the model, the loss values ​​obtained from noise labels and non-noise labels are constantly changing, and as the number of training rounds increases, the corresponding loss values ​​of both converge. Therefore, the key to adjusting the abnormal loss due to noise labels is how to adaptively select the correct loss threshold δ during training to reasonably define the "excessive" loss value caused by noise labels. Under weak supervision with a single positive label, there is only one accurate positive label given; therefore, the number of false negative noise labels should be... in The number of real labels in a multi-label image is such that the proportion of noise labels to the total number of labels should be close to the sparsity of the label space.

[0079] The loss threshold adjustment method is as follows: the minimum value among the largest (∈×100)% loss values ​​in the current training batch is selected as the value of the loss threshold δ.

[0080] δ=min(top ∈ (l i )),#(1.7)

[0081] Where ∈ is the sparsity control parameter, a hyperparameter used to measure the sparsity of the labels in the dataset.

[0082] Experimental Results and Analysis

[0083] 1. Experimental Environment

[0084] Table 1 Experimental environment of WSML-OOD method

[0085]

[0086]

[0087] 2. Experimental Setup

[0088] To simulate the weakly supervised scenario with a single positive label, this chapter adopts the existing document Multi-Label Learning from Single Positive Labels; that is, based on the settings in multi-label learning with a single positive label, on the PascalVOC2012, MicrosoftCOCO and NUS-WIDE fully supervised multi-label datasets, by randomly retaining one positive label and treating the remaining labels as unknown labels, a single positive label dataset is obtained.

[0089] This experiment selected a ResNet-50 deep learning network model pre-trained on the ImageNet-1K dataset as the multi-label image classification model, and trained it on the training set of the in-distribution dataset. The training samples of the in-distribution dataset were first subjected to data augmentation processing, and then uniformly adjusted to a resolution of 256×256 before being input into the multi-label image classification model. During the training phase, the multi-label image classification model used an adaptive moment estimation optimizer to optimize the network model parameters, with a learning rate of 0.01, for a total of 30 training epochs.

[0090] 3 Experimental Results

[0091] To verify the feasibility and effectiveness of the WSML-OOD method in weakly supervised multi-label out-of-distribution sample detection, experiments were conducted on three combinations of in-distribution / out-of-distribution datasets under weak supervision with single positive labels: PascalVOC2012—ImageNet-for-VOC-COCO, MicrosoftCOCO—ImageNet-for-VOC-COCO, and NUS-WIDE—ImageNet-for-NUS-WIDE. The experimental results were compared with some high-performance out-of-distribution sample detection methods. Except for the WSML-OOD method, the other methods are all fully supervised multi-label or single-label out-of-distribution sample detection methods. The JointEnergy method is a fully supervised multi-label out-of-distribution sample detection method, while the others are fully supervised single-label out-of-distribution sample detection methods.

[0092] The IsolationForest method is a tree-based approach that detects out-of-distribution samples based on the path length from the root node to the terminal node. The LOF method uses the K-Nearest Neighbors (KNN) method for local density estimation, distinguishing between in-distribution and out-of-distribution samples based on density differences. The MSP, MaxLogit, ODIN, and Mahalanobis methods primarily design an out-of-distribution sample detector based on confidence or distance, deriving out-of-distribution sample scores by statistically analyzing the maximum values ​​across all classes, thus performing out-of-distribution sample detection. All of the above methods rely on the same trained convolutional neural network as their backbone.

[0093] Tables 2, 3, and 4 record the experimental results of WSML-OOD under three combinations of in-distribution and out-of-distribution datasets, and its performance comparison with other out-of-distribution sample detection methods. Each method was tested with and without mixed data augmentation (+MixAug). As shown in Tables 2, 3, and 4, WSML-OOD achieved near-optimal results in FPR@95, AUROC, and AUPR metrics under the three combinations of in-distribution and out-of-distribution datasets, only achieving a suboptimal AUPR result under the PascalVOC2012—ImageNet-for-VOC-COCO combination. Under the PascalVOC2012—ImageNet-for-VOC-COCO combination, compared to the suboptimal multi-label out-of-distribution sample detection method JointEnergy, the WSML-OOD method achieved improvements of 1.05%, 0.49%, and 0.79% in FPR@95, AUROC, and AUPR metrics, respectively. Similarly, with the Microsoft COCO—ImageNet-for-VOC-COCO combination, the WSML-OOD method improved the three metrics by 0.97%, 0.42%, and 0.34% respectively, compared to the JointEnergy method. With the NUS-WIDE—ImageNet-for-NUS-WIDE combination, the three metrics increased by 0.35%, 0.24%, and 0.21% respectively. These experimental results demonstrate that applying WSML-OOD to weakly supervised multi-label out-of-distribution sample detection is effective.

[0094] On the other hand, even without using mixed data augmentation for model training, the WSML-OOD method still achieves the best performance, strongly demonstrating the feasibility and effectiveness of introducing noisy labels into in-distribution samples by artificially assigning all negative labels to unknown samples, and then distinguishing between in-distribution and out-of-distribution samples based on the loss value dimension, according to the noise label loss effect and label sparsity. Meanwhile, by applying mixed data augmentation techniques, almost all out-of-distribution sample detection methods have achieved significant performance improvements. For example, in the PascalVOC2012—ImageNet-for-VOC-COCO combination, the JointEnergy method using mixed data augmentation achieves performance improvements of 2.71%, 0.47%, and 0.29% on FPR@95, AUROC, and AUPR respectively compared to the JointEnergy method without mixed data augmentation. Similarly, the WSML-OOD method using mixed data augmentation achieves improvements of 2.65%, 0.47%, and 0.23% on the same three metrics compared to the WSML-OOD method without mixed data augmentation. The experimental results above demonstrate that hybrid data augmentation techniques can, to some extent, suppress excessively high confidence levels in the model, thereby improving the detection performance of out-of-sample samples in weakly supervised multi-label distributions.

[0095] Furthermore, compared to single-label out-of-distribution detection methods, both the WSML-OOD method and the fully supervised multi-label out-of-distribution detection method JointEnergy demonstrate superior performance in multi-label out-of-distribution detection under weak supervision. For example, under three combinations of in-distribution / out-of-distribution datasets, compared to the best-performing fully supervised single-label out-of-distribution detection method, the JointEnergy method improves the FPR@95 metric by 1.06%, 5.65%, and 1.41%, respectively, while the WSML-OOD method further improves by 2.11%, 6.62%, and 1.76%. These experimental results indicate that multi-label out-of-distribution detection tasks, especially weakly supervised multi-label out-of-distribution detection tasks, are more realistic and challenging than single-label out-of-distribution detection tasks, warranting further in-depth research and exploration.

[0096] Table 2 shows the performance (%) of different methods in detecting multi-label out-of-distribution samples on the PascalVOC2012—ImageNet-for-VOC-COCO dataset combination (inside / outside distribution) under weak supervision. "↓" indicates a smaller value, better performance; "↑" indicates a larger value, better performance. "+MixAug" indicates that mixed data augmentation was used; otherwise, it indicates that mixed data augmentation was not used. Bold text indicates the best experimental results under this in-distribution / outside dataset combination, and underlined text indicates the second-best experimental results.

[0097] Table 2

[0098]

[0099] Table 3 shows the performance (%) of different methods in detecting multi-label out-of-distribution samples on the combination of Microsoft COCO-ImageNet-for-VOC-COCO in-distribution and out-of-distribution datasets under weak supervision.

[0100] Table 3

[0101]

[0102] Table 4 shows the performance (%) of different methods in detecting multi-label out-of-distribution samples on the combination of NUS-WIDE—ImageNet-for-NUS-WIDE in-distribution and out-of-distribution datasets under weak supervision.

[0103] Table 4

[0104]

[0105] To further verify the impact of method hyperparameters on method performance, this section presents ablation experiments under the aforementioned three different combinations of in-distribution and out-of-distribution datasets.

[0106] 4. Ablation test

[0107] ① The impact of different values ​​of ∈ on method performance

[0108] This section examines the performance of the WSML-OOD method under three combinations of in-distribution and out-of-distribution datasets, with different values ​​of ∈ (i.e., different label sparsity assumptions). Out-of-distribution sample detection experiments were conducted with ∈ = 0.05, ∈ = 0.10, and ∈ = 0.30, respectively. The value of ∈ = 0.05 is used to measure the out-of-distribution sample detection. + 0.10 + and 0.30 + The value indicates that mixed data augmentation was used; otherwise, it indicates that mixed data augmentation was not used. The experimental results are shown in Table 5.

[0109] Table 5 shows the experimental results, indicating that the method achieves the best performance when ∈ = 0.10 under the three combinations of in-distribution / out-distribution datasets. Increasing or decreasing sparsity leads to varying degrees of performance degradation. For example, under the PascalVOC2012—ImageNet-for-VOC-COCO in-distribution / out-distribution dataset combination, when ∈ = 0.10, the WSML-OOD method improves the FPR@95 metric by 35.93% and 0.79% respectively compared to the best results when ∈ = 0.05 and ∈ = 0.30. A smaller ∈ indicates a lower assumption of a smaller number of labels in the dataset. Especially in the PascalVOC2012 in-distribution dataset, the sparsity assumption of ∈ = 0.05 results in an average of only 1 label per sample, which is clearly inconsistent with reality. Conversely, an excessively large ∈ indicates an excessive number of labels in the dataset, which does not conform to the natural sparsity of multi-label data, making it difficult for the method to correctly identify excessive loss values ​​caused by noisy labels. In summary, setting ∈ = 0.10 is a good assumption for dataset sparsity and can achieve better performance.

[0110] Meanwhile, in almost all settings, the use of hybrid data augmentation improved the method's performance to varying degrees. For example, with three combinations of in-distribution / out-of-distribution datasets, when ∈ = 0.10, the WSML-OOD method using hybrid data augmentation achieved improvements of 2.65%, 13.63%, and 2.76% respectively on the FPR@95 metric compared to the WSML-OOD method without hybrid data augmentation. This further demonstrates the effectiveness of using hybrid data augmentation to suppress excessively high model confidence and improve the detection performance of weakly supervised multi-label out-of-distribution samples.

[0111] Table 5 Performance of the WSML-OOD method under different ∈ values

[0112]

[0113] ② Selection of decision threshold λ for out-of-distribution sample detection

[0114] The threshold λ is the decision threshold for the weakly supervised multi-label out-of-distribution sample detector. Its value must be such that the true positive rate (TPR) of in-distribution samples is greater than or equal to 95%. When the out-of-distribution sample detection score of a sample is greater than or equal to λ, the sample is considered to belong to the in-distribution data; otherwise, it belongs to the out-of-distribution data. Figure 4 The graph shows the relationship between λ and the true positive rate of samples within the distribution. As can be seen from the graph, when the true positive rate is 95%, the values ​​of λ on the PascalVOC2012, MicrosoftCOCO, and NUS-WIDE distribution datasets are around 2, 9, and 2, respectively.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A weakly supervised multi-label out-of-distribution sample classification detection method, characterized in that, Comprising the following steps: S1: original image label preprocessing, for multi-label image x i , its original label There is and only one item All unknown annotation information is set to negative label, x i After the original weak supervision information is preprocessed by the negative label assumption, x i The label becomes wherein, represents a label space constituted by positive labels, represents a label space constituted by unknown labels; S2: training the CNN model: The image and its label preprocessed by S1 label preprocessing constitute an unenhanced image All the unenhanced images are subjected to mixed data enhancement preprocessing to generate an enhanced image Then the enhanced image and the label of the corresponding unenhanced image thereof are taken as training samples All the training samples constitute a training set; The samples in the training set are output in batches for training in the CNN, and the training is ended when the value of the loss function no longer decreases, and a trained model is obtained; S3: detecting the target image using the trained model The target image is input into the trained model, the trained model makes a classification prediction thereon and gives a prediction result y i Then the prediction result y i And the all-negative label artificially given Calculate the multi-label classification loss: wherein L is the number of classes in the label space, denotes the all-negative label artificially assigned, whose value is all 0; The full negative label loss value is calculated based on formula (1.2) Afterwards, by identifying the excessively large loss value caused by the noise label, the out-of-distribution sample detection score is calculated: wherein λ i = a i denotes a loss penalty term, and δ is a loss threshold value; If score < λ, the target image is an out-of-distribution sample, otherwise an in-distribution sample, α i represents the loss penalty term. 2.The weakly supervised multi-label out-of-distribution sample classification detection method of claim 1, wherein: The process of the mixed data enhancement preprocessing in S2 is as follows: From randomly sample another un-augmented image from the same training batch Then randomly generate a crop box {z h ,z v ,w,h} where z h and z v represent the horizontal and vertical coordinate of the top-left starting point of the crop box in image x A and image x B respectively, w and h represent the width and height of the crop box respectively; after obtaining the crop box, crop the image region reg B from image x B with the same starting position and size as the crop box, then randomly apply existing data augmentation operations to the image region reg B to obtain the image region reg B '; finally, mark the image region reg A to be masked in image x A according to the crop box obtained above, then replace the image region reg B to be masked with the image region reg A , and generate the final augmented image x The above process can be formalized as: where M e {0, 1} w×h is a binary matrix, where a value of 0 indicates discarding the pixel value at the corresponding position of the image, and a value of 1 indicates retaining the pixel value at the corresponding position, and 1 is a matrix with all element values being 1. 3.The weakly supervised multi-label distribution outlier sample classification detection method of claim 2, wherein: The loss function employed in S2 As shown in equation (1.5): where λ i is a loss penalty term, whose value is in the interval [0, 1], when α i = 1, which means that the corresponding label is accurate positive label and no correction is needed, otherwise, when and the corresponding l i is greater than the loss threshold δ, which means that the corresponding label is more likely to be a noisy label, then α i = 0, which means that the loss value is discarded. The value of the loss penalty term is as follows: Wherein δ is the loss threshold. 4.The weakly supervised multi-label distribution outlier sample classification detection method of claim 3, wherein: The loss threshold adjustment method is: that is, the minimum value in the loss value of the front (∈×100) % of the current training batch is selected as the value of the loss threshold δ: delta = min(top ∈ (l i )), (1.7) where e is a sparsity control parameter, and is a hyperparameter that measures the label sparsity of the dataset.

Images