Class-adaptive label discovery and noise rejection method and device for multi-label recognition

Abstract

The application relates to the technical field of image classification, in particular to a class adaptive label discovery and noise rejection method and device for multi-label recognition, which comprises the following steps: S10, associating each sample image in a sample set with a label vector; S20, extracting global features of the sample; S30, obtaining semantic representations of each class in the sample based on the global features of the sample and the label vector of the sample; S40, calculating semantic similarity between samples of each class, and generating a pseudo label vector based on the semantic similarity; S50, calculating a class weight of each sample based on the semantic similarity between samples of each class, and adaptively discarding low-weight samples as noise labels according to the class weight; S60, constructing a target function based on the label vector, the pseudo label vector and the sample class weight; and S70, training the sample set through the target function to obtain a multi-label recognition model; the method has the beneficial effect of high performance.

Description

Technical Field

[0001] This invention relates to the technical field of image classification, and more specifically to a method and apparatus for category-adaptive label discovery and noise rejection in multi-label recognition. Background Technology

[0002] Currently, multi-label image recognition (MLR) algorithms mainly rely on collecting large-scale, clean, and complete simulated label datasets. As the number of categories and images in the dataset increases, this becomes not only time-consuming but also expensive.

[0003] To address this, some scholars have conducted research on weakly supervised multi-label image recognition (WSMLR). However, in different settings of WSMLR, only a few positive labels are labeled, and each image lacks other labels, i.e., multi-label recognition with partial positive labels (MLR-PPL). The MLR-PPL task suffers from the problems of losing a large number of supervision signals and the dilemma of lacking any negative labels, which causes the model to always predict positive labels.

[0004] To address the above problems, the mainstream solutions include:

[0005] (1) Use algorithms or visual attention mechanisms to capture distinctive local regions for feature enhancement; or capture label dependencies to standardize the training of multi-label recognition models.

[0006] This approach introduces structured graphs to explicitly model label relevance to facilitate multi-label image recognition; however, such solutions always rely on collecting large-scale, clean, and complete multi-label datasets to learn discriminative feature representations, which is time-consuming and expensive, especially as the number of categories and images in the dataset increases.

[0007] (2) Set the training method to learn a multi-label recognition model with some positive labels, that is, only label some positive labels.

[0008] One approach proposes training the model using the hypothesis of negativity (AN), where all unlabeled samples are treated as negative. A second approach proposes correcting mislabeled samples by exploring label dependencies. Besides these two methods, other suggestions include adjusting the weight of each label based on the prediction score or correcting mislabeled samples; specifically, training a partially labeled model and employing a course learning method to use the model's predictions to label some unlabeled simple samples.

[0009] However, the aforementioned techniques still rely on a large number of positive labels for training images or similar models, which leads to poor performance when the proportion of known positive labels drops to a low level. Summary of the Invention

[0010] In view of this, and to address the problems that need to be addressed, this application provides a method and apparatus for category-adaptive label discovery and noise rejection in multi-label recognition.

[0011] In a first aspect, this disclosure provides a category-adaptive label discovery and noise rejection method for multi-label recognition, including:

[0012] S10, associate each sample image in the sample set with a label vector;

[0013] S20, extract global features of the sample;

[0014] S30: Based on the global features of the sample and the label vector of the sample, obtain the semantic representation of each category in the sample;

[0015] S40, calculate the semantic similarity between samples of each category, and generate pseudo-label vectors based on the semantic similarity;

[0016] S50: Based on the semantic similarity between samples of each category, calculate the weight of each category of the sample; and adaptively discard low-weight samples that are used as noise labels according to the category weights.

[0017] S60, construct an objective function based on label vectors, pseudo-label vectors, and sample class weights;

[0018] S70 uses an objective function to train the sample set to obtain a multi-label recognition model.

[0019] Secondly, this disclosure provides an electronic device, including:

[0020] Memory;

[0021] Processor; and

[0022] Computer programs;

[0023] The computer program is stored in the memory and configured to be executed by the processor to implement the method described above.

[0024] Thirdly, this disclosure provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method described above.

[0025] It should be understood that the descriptions in this section are not intended to identify key or essential features of the embodiments of this disclosure, nor are they intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0026] The advantages of the category-adaptive label discovery and noise rejection method and device for multi-label recognition provided in this disclosure compared with the prior art are as follows:

[0027] 1. This disclosure can obtain the semantic representation of each category in the sample based on the global features and category labels of the sample; generate pseudo-label vectors by calculating the semantic similarity between samples of each category; calculate the category weight of each sample based on the semantic similarity between samples of each category; and adaptively discard low-weight samples that are used as noise labels according to the category weights.

[0028] Based on this, this disclosure constructs an objective function based on partial positive label vectors, pseudo label vectors, and sample weights; and trains the sample set using the objective function to obtain a multi-label recognition model.

[0029] This disclosure improves the performance of multi-label recognition models by supplementing unknown labels and discarding noisy labels through cross-image semantic association of categories, and is highly practical.

[0030] 2. In this disclosure, a category-adaptive threshold update is set to adaptively adjust the threshold of each category, avoiding the extremely time-consuming and laborious manual adjustment. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0032] Figure 1 A flowchart illustrating the category-adaptive label discovery and noise rejection method for multi-label recognition provided in this application embodiment;

[0033] Figure 2 A schematic diagram illustrating the implementation process of the category-adaptive label discovery and noise rejection method for multi-label recognition provided in this application embodiment;

[0034] Figure 3 A schematic diagram illustrating the implementation process of the Category Adaptive Label (CALD) module provided in this application embodiment;

[0035] Figure 4 This is a schematic diagram illustrating the implementation process of the Category Adaptive Noise Rejection (CANR) module provided in an embodiment of this application. Detailed Implementation

[0036] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0037] The following describes in detail, with reference to the accompanying drawings, the category-adaptive tag discovery and noise rejection method and apparatus for tag recognition mentioned in the embodiments of this disclosure.

[0038] See Figure 1 , Figure 2 Multi-label recognition class-adaptive label discovery and noise rejection methods include:

[0039] S10, associate each sample image in the sample set with a label vector;

[0040] S20, extract global features of the sample;

[0041] S30: Based on the global features of the sample and the label vector of the sample, obtain the semantic representation of each category in the sample;

[0042] S40, calculate the semantic similarity between samples of each category, and generate pseudo-label vectors based on the semantic similarity;

[0043] S50: Based on the semantic similarity between samples of each category, calculate the weight of each category of the sample; and adaptively discard low-weight samples that are used as noise labels according to the category weights.

[0044] S60, construct an objective function based on label vectors, pseudo-label vectors, and sample class weights;

[0045] S70 uses an objective function to train the sample set to obtain a multi-label recognition model.

[0046] For example, in S10, the expression associated with each sample in the sample set and its category label is:

[0047]

[0048] Where N represents the total number of samples; y n Represents a label vector;

[0049] Label vector y n The expression is:

[0050] For sample I n If label c exists, then Assign a value of 1 if it does not exist, or assign a value of 0 if it does not exist.

[0051] It is important to understand that in multi-label recognition, each image I n With label vector Related, i.e., y n It is a C-dimensional vector.

[0052] In partial positive multi-label learning, each sample (image) is labeled with only a few positive labels, while other labels are left unlabeled; in this disclosure, y n The value space is set to {0, 1}. If the corresponding label is unknown, the value is assigned to 0.

[0053] In this partially positive label setting, sample I n The loss function based on label vectors is the "negative hypothesis" (AN) loss function. In this disclosure, a multi-label recognition model is trained under supervision by assuming that unobserved labels are negative, and the loss function is described. The expression is:

[0054]

[0055] Where 1[·] represents the index function, which has a value of 1 if the parameter is positive and 0 otherwise;

[0056] Indicates sample I n The predicted score for category c;

[0057] Sample I n The set of predicted scores for all categories is:

[0058] It should be understood that this disclosure uses ResNet-101 as the backbone network to extract the global feature map f. n Furthermore, the semantic representation of each category is obtained through the Category-Specific Semantic Learning Module (CSSL).

[0059] For example, in S30, the semantic representation of each category in the sample is expressed as follows:

[0060]

[0061] In equation (1), Indicates sample I n The semantic representation of category c, f n Indicates sample I n global features, u c φ represents the semantic embedding of category c. cssl (·) denotes a category-specific semantic learning function; c∈(1,2,3…,C);

[0062] Sample I n The set of semantic representations for all categories is:

[0063] For example, this disclosure may use a linear classifier and a sigmoid function to compute predicted scores for all categories.

[0064] See Figure 3 It should be understood that in S40, the Category Adaptive Label (CALD) module can be used to calculate the semantic similarity between samples of each category and generate pseudo-label vectors based on the semantic similarity. In specific implementation, the CALD module first measures the semantic similarity between positive samples and then adaptively supplements unknown labels with high similarity.

[0065] For example, for each category of image I n and I m Semantic similarity can be calculated using the cosine distance between semantic representations;

[0066] S40 includes:

[0067] S401, Calculate the sample I for each category. n Sample I m The semantic similarity between them; the calculation expression is:

[0068]

[0069] in, Indicates sample I n Sample I m The semantic similarity between categories c is given by <·,·>, which represents the inner product of two vectors, and ||·||2 represents the L2 norm of the vectors.

[0070] It should be noted that, due to the scarcity of positive samples in the MLR-PPL task and the complex variations during data collection (e.g., pose, lighting), generating pseudo-labels solely based on a single semantic similarity is extremely unstable. To address this issue, this disclosure constructs a positive sample queue. This ensures that each image in the positive sample queue has a positive label.

[0071] S402, Calculate the average positive semantic similarity between samples; including:

[0072] Constructing a positive sample queue and To ensure that each sample has a positive label, the expression for the average positive semantic similarity between samples is:

[0073]

[0074] in, Indicates sample I n The average positive semantic similarity between category c of sample c and category c of other samples;

[0075] It should be noted that if a positive label for a certain sample category c is missing, the semantic similarity between that sample and another positive sample can be used to complete this unknown label and generate a pseudo label.

[0076] S403, Calculate sample I n Corresponding pseudo-label vector include:

[0077] Sample I n pseudo-tags of category c The expression is:

[0078] Tag vector The expression is:

[0079] in, This represents the adjustment threshold for category c;

[0080] 1[·] represents the index function. If the parameter of the index function is positive, its value is 1, otherwise it is 0.

[0081] Sample I n The expression for the loss function based on pseudo-label vectors is:

[0082]

[0083] See Figure 4 It should be understood that in S50, a Class Adaptive Noise Rejection (CANR) module can be used to calculate the class weight of each sample; and based on the class weight, low-weight samples that are used as noise labels are adaptively discarded. In specific implementation, the CANR module first calculates the sample weight based on the semantic similarity from different samples, and then adaptively discards noise labels with low weights.

[0084] For example, in S50, the expression for calculating the weight of each category of the sample is:

[0085]

[0086] in, Indicates sample I n The weight corresponding to category c; X represents the adjustment threshold used to control the accuracy of noise labels in the projection category, rejecting noise labels; X is a random variable that follows a uniform distribution.

[0087] Sample I n The expression for the loss function based on label vectors and weights is:

[0088]

[0089] Where, λ n Indicates sample I n The weighting terms

[0090] It should be noted that this disclosure introduces a loss function with partial positive labels, pseudo labels, and sample weights. Therefore, the objective function can be defined as the sum of the above losses.

[0091] For example, in S60, the expression for the objective function is:

[0092]

[0093] in, For sample I n Loss function based on label vectors, For sample I n Loss function based on pseudo-label vectors, For sample I n A loss function based on label vectors and weights.

[0094] It should be noted that due to the scarcity of positive samples and the complex variations in the data collection process, the semantic representation of each category is not compact enough, especially when the proportion of known positive labels is small. To guide the model in learning more compact semantic representations, this disclosure also introduces a cross-image semantic learning loss for training, expressed as:

[0095]

[0096]

[0097] Therefore, the final objective function can be defined as a summary of the total loss function and the cross-image semantic learning loss function; the expression of the objective function is:

[0098]

[0099] Here, α is a balance parameter that ensures the two losses have comparable magnitudes; in the experiment, α can be set to 0.05.

[0100] It should be noted that adjusting the threshold and It is an important parameter for controlling the accuracy and recall of generating pseudo-labels and rejecting noisy labels of category c;

[0101] On the one hand, the optimal thresholds differ significantly between different categories; this is because the training samples for different categories are significantly different, and the difficulty of learning compact representations for different categories also varies. For example, compared to people (category), toothbrushes (category) have significantly different optimal thresholds. The value should be lower because there are fewer positive samples for the toothbrush (category), and the learned visual representation of toothbrushes is more easily dispersed. On the other hand, the optimal threshold varies significantly across different datasets and with different proportions of known labels. Therefore, thoroughly searching for the optimal threshold is crucial for achieving best performance.

[0102] However, the traditional manual adjustment process is extremely time-consuming and labor-intensive. This disclosure proposes a novel Category Adaptive Threshold Update (CATU) mechanism that adaptively adjusts the threshold for each category, avoiding the extremely time-consuming and labor-intensive manual adjustment on different datasets and with different known positive label ratios.

[0103] Exemplary embodiments of this disclosure also include: adjusting the threshold. and Adaptive updates are performed; specifically:

[0104] Calculate the average positive similarity probability and average negative similarity probability for each category;

[0105] Calculate the average similarity momentum based on the average positive similarity probability and average negative similarity probability of the previous period and the current period;

[0106] Based on the similarity momentum average, the threshold is adjusted. and Perform adaptive updates.

[0107] For example, the expressions for the average positive similarity probability and the average negative similarity probability are respectively:

[0108]

[0109]

[0110] in, Let c be the average positive similarity probability. Let be the average negative similarity probability of category c; There exists a subset of images of category c. It is a subset of images that does not belong to category c;

[0111] The expression for the average similarity momentum is:

[0112]

[0113]

[0114] Where b represents the index of the current batch in the current period, and B represents the total number of batches in each period;

[0115] and It is the average similarity in the current period. and It is the average similarity of the previous period;

[0116] Adjusting the threshold and The expression for adaptive updating is:

[0117]

[0118]

[0119] Where θ represents the preset threshold.

[0120] This disclosure also provides an electronic device, including:

[0121] Memory; processor; and computer programs;

[0122] The computer program is stored in the memory and configured to be executed by the processor to implement the method described above.

[0123] This disclosure also provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method described above.

[0124] In this embodiment of the disclosure, the processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform the desired functions.

[0125] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the model training method, image recognition method, and / or other desired functions of the various embodiments of this application mentioned above. Various contents, such as samples of image sequences to be recognized, may also be stored in the computer-readable storage medium.

[0126] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to the various embodiments of this application described above.

[0127] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0128] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0129] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A category-adaptive label discovery and noise rejection method for multi-label recognition, characterized in that, include: S10, associate each sample image in the sample set with a label vector; S20, extract global features of the sample; S30: Based on the global features of the sample and the label vector of the sample, obtain the semantic representation of each category in the sample; S40, calculate the semantic similarity between samples of each category, and generate pseudo-label vectors based on the semantic similarity; S50, calculates the weight of each class of a sample based on the semantic similarity between samples of each class; And based on the category weights, low-weight samples that are used as noise labels are adaptively discarded; S60, construct an objective function based on label vectors, pseudo-label vectors, and sample class weights; S70 uses an objective function to train the sample set to obtain a multi-label recognition model.

2. The category-adaptive label discovery and noise rejection method for multi-label recognition according to claim 1, characterized in that, In step S10, the expression associated with each sample image and the label vector is: Where N represents the total number of samples; y n Represents a label vector; Label vector y n The expression is: C represents the total number of categories; For sample I n If category c exists, then Assign a value of 1 if it does not exist, or assign a value of 0 if it does not exist.

3. The category-adaptive label discovery and noise rejection method for multi-label recognition according to claim 2, characterized in that, In step S30, the semantic representation of each category in the sample is expressed as follows: In equation (1), Indicates sample I n The semantic representation of category c, f n Indicates sample I n global features, u c φ represents the semantic embedding of category c. cssl (·) denotes a category-specific semantic learning function; c∈(1,2,3…,C); Sample I n The set of semantic representations for all categories is:

4. The category-adaptive label discovery and noise rejection method for multi-label recognition according to claim 3, characterized in that, S40 includes: S401, calculate sample I for each category. n Sample I m The semantic similarity between them; the calculation expression is: in, Indicates sample I n Sample I m The semantic similarity between categories c is given by <·,·>, which represents the inner product of two vectors, and ||·|2 represents the L2 norm of the vectors. S402, Calculate the average positive semantic similarity between samples; including: Constructing a positive sample queue and This ensures that each sample has a positive label; where m represents sample I. m The m-th sample in; Indicates As a condition, from sample I m Samples that meet the criteria are extracted from the data to form a positive sample queue. Then, the expression for the average positive semantic similarity between samples is: in, Indicates sample I n The average positive semantic similarity between category c of sample c and category c of other samples; Represents the positive sample queue The number of samples included in it; S403, Calculate sample I n Corresponding pseudo-label vector include: Sample I n pseudo-tags of category c The expression is: Tag vector The expression is: in, This represents the adjustment threshold for category c; 1[·] represents an index function. For an index function 1[·], if the condition inside its square brackets is true, the value of the index function 1[·] is 1; if the condition inside its square brackets is false, the value of the index function 1[·] is 0.

5. The category-adaptive label discovery and noise rejection method for multi-label recognition according to claim 4, characterized in that, In S50, the formula for calculating the weight of each category of the sample is: in, Indicates sample I n The weight corresponding to category c; This represents the adjustment threshold used to control the accuracy of noise tags for projection categories; X is a random variable that follows a uniform distribution.

6. The category-adaptive label discovery and noise rejection method for multi-label recognition according to claim 5, characterized in that, Also includes: Adjusting the threshold and Adaptive updates are performed; specifically: Calculate the average positive similarity probability and average negative similarity probability for each category; The average similarity momentum is calculated based on the average positive similarity probability and the average negative similarity probability of the previous period and the current period. Based on the similarity momentum average, the adjusted threshold is... and Perform adaptive updates.

7. The category-adaptive label discovery and noise rejection method for multi-label recognition according to claim 6, characterized in that, The expressions for the average positive similarity probability and the average negative similarity probability are as follows: in, Let c be the average positive similarity probability. Let be the average negative similarity probability of category c; There exists a subset of images of category c. It is a subset of images that does not belong to category c; express The number of elements contained therein express The number of elements contained therein; The expression for the average similarity momentum is: Where b represents the index of the current batch in the current period, and B represents the total number of batches in each period; and It is the average similarity in the current period. and It is the average similarity of the previous period; Adjusting the threshold and The expression for adaptive updating is: Where θ represents the preset threshold.

8. The category-adaptive label discovery and noise rejection method for multi-label recognition according to claim 6, characterized in that, In step S60, the expression for the objective function is: in: Let α represent the total loss function, and let α represent the balancing parameter. Represents the loss function for cross-image semantic learning; As an intermediate variable; l AN (y n ,p n ) represents sample I n Loss function based on label vectors, Indicates sample I n The loss function based on pseudo-label vectors, l AN (y n ,p n ,λ n ) represents sample I n Loss function based on label vectors and weights; The l AN (y n ,p n The expression for ) is: The The expression is: The l AN (y n ,p n ,λ n The expression for ) is: Where 1[·] represents the index function. For the index function 1[·], if the condition inside its square brackets is true, the value of the index function 1[·] is 1; if the condition inside its square brackets is false, the value of the index function 1[·] is 0. Indicates sample I n The predicted score for category c; Sample I n The set of predicted scores for all categories is: λ n Indicates sample I n The weighting terms 9. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon; the computer program is executed by a processor to implement the method as described in any one of claims 1 to 8.

Images