Multi-label image classification method based on comparison sample generation learning

By comparing sample generation learning and Gaussian mixed models, the problem of overfitting and insufficient comparison learning samples of multi-label image classification model is solved, and the image label classification accuracy and model classification ability are improved.

CN119091205AActive Publication Date: 2024-12-06SOUTH CHINA AGRICULTURAL UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411151635.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-06
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

The existing multi-label image classification model based on semantic correlation is prone to overfitting, resulting in reduced image label classification accuracy, and insufficient number of comparison samples and low quality in comparison learning, which affects the model classification ability.

Method used

Using a method based on comparison sample generation learning, the target semantic features are extracted through feature extraction networks, and the comparison samples are generated using Gaussian mixed model, and the comparison learning is carried out to constrain the discrimination of target semantic features to improve overfitting problems and improve the comparison learning efficiency. .

Benefits of technology

It effectively improves the classification accuracy of image tags, avoids model overfitting, enhances the efficiency and effectiveness of comparison learning, and improves the classification ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091205A_ABST
    Figure CN119091205A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-label image classification method based on comparative sample generation learning, and compared with the prior art, the method has the following beneficial effects: (1) the distance between different types of target semantic features is constrained by comparative learning, that is, the different types of target semantic features are constrained by comparative learning to maintain discrimination, and the classification efficiency is improved; therefore, the problem that an existing multi-label image classification method is prone to overfitting is solved, and the image label classification precision is improved. And (2) in addition, aiming at the problem that comparative learning requires a large number of comparative samples, a Gaussian mixture model is utilized to generate the comparative samples of the target semantic features, and the problems of insufficient number and low quality of the comparative samples in the comparative learning can be solved, so that the efficiency and effectiveness of the comparative learning are improved, and finally the classification capability of the model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image classification in computer vision, and in particular to a multi-label image classification method based on comparative sample generation learning. Background Art

[0002] Multi-label image classification is a basic and important task in the field of computer vision. Its target output is a set of labels (objects or attributes) associated with the input image. It has a wide range of applications, such as attribute recognition, weakly supervised semantic segmentation, automatic image tagging, etc. Different from single-label image classification tasks, multi-label image classification tasks face more complex and realistic images. The number, position and size of objects in the image vary, which brings additional difficulty to the extraction of semantic features of each object in the image.

[0003] Most of the existing multi-label image classification methods based on deep neural networks are optimized based on single-label image classification methods. These methods are straightforward and simple to implement, but due to the diversity and complexity of multi-label images, these methods cannot effectively extract the semantic features of the target, resulting in a reduction in the final classification effect. In view of the defects of the above methods, many researchers have noticed that there is a very strong semantic correlation between the targets in multi-label images, that is, some targets in the image tend to appear at the same time, such as keyboard and mouse, knife and fork, etc., also known as label co-occurrence. They added a semantic correlation module to the multi-label classification model and used the prior information of semantic correlation to improve the classification effect of the model.

[0004] However, there are still problems in the multi-label image classification model based on semantic relevance: deep networks are prone to overfitting in the semantic relevance they define, that is, the model ignores the image content and relies entirely on semantic relevance to classify image targets. For example, if the model recognizes that there is a "mouse" in the image, then the model ignores the image content and directly judges that there is a "keyboard" in the image based on semantic relevance. In other words, the above multi-label image classification model is prone to overfitting problems, which affects the accuracy of image label classification. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a multi-label image classification method based on comparative sample generation learning, which can solve the above technical problems.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention provides the following technical solution: a multi-label image classification method based on comparative sample generation learning, comprising the following steps:

[0009] S1: Extract the target semantic features of the image using the feature extraction network;

[0010] S2: Generate comparison samples of target semantic features using Gaussian mixture model;

[0011] S3: Use contrast samples and target semantic features extracted by feature extraction network for contrast learning;

[0012] S4: Use the target semantic features after contrastive learning to classify image labels.

[0013] Preferably, the feature extraction network includes an image feature extraction module, a semantic feature extraction module and a semantic feature optimization module; step S1 includes the following sub-steps:

[0014] S11: extracting features from the image using an image feature extraction module to obtain image features;

[0015] S12: Decoupling the image features using a semantic feature extraction module to obtain target semantic features;

[0016] S13: Optimize the target semantic features obtained in sub-step S12 using a semantic feature optimization module.

[0017] Preferably, the image feature extraction module is a convolutional neural network, and sub-step S11 specifically utilizes the residual network of the convolutional neural network to extract features from the image.

[0018] Preferably, in sub-step S11, before feature extraction, the method further includes: performing interpolation processing on the image.

[0019] Preferably, the semantic feature extraction module is a spatial attention network.

[0020] Preferably, the semantic feature optimization module is a graph neural network, and sub-step S13 is to use the graph neural network to perform semantic interaction to optimize the target semantic features obtained in sub-step S12.

[0021] Preferably, step S2 includes the following sub-steps:

[0022] S21: collect target semantic features;

[0023] S22: updating and fitting multiple Gaussian distributions in combination with the target semantic features collected in sub-step S21;

[0024] S23: Randomly sample various target semantic features from multiple Gaussian distributions after updating and fitting as comparison samples.

[0025] Preferably, step S4 is to classify the target semantic features after contrast learning into image labels through a linear layer.

[0026] Preferably, while performing image label classification in step S4, a cross entropy loss function is used for constraint.

[0027] (III) Beneficial effects

[0028] Compared with the prior art, the present invention provides a multi-label image classification method based on contrast sample generation learning, which has the following beneficial effects: (1) The present invention uses contrast learning to constrain the distance between different types of target semantic features, that is, through contrast learning, the semantic features of each type of target are constrained to maintain the discriminability between each other, thereby improving the problem of easy overfitting of the existing multi-label image classification method, so as to improve the image label classification accuracy; (2) In addition, the present invention uses a Gaussian mixture model to generate contrast samples of target semantic features, which can solve the problems of insufficient number and low quality of contrast samples in contrast learning, thereby improving the efficiency and effectiveness of contrast learning, and ultimately improving the model classification ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of the steps of the multi-label image classification method based on comparative sample generation and learning of the present invention;

[0030] Figure 2 A flow chart of sub-steps for extracting target semantic features of an image using a feature extraction network in the present invention;

[0031] Figure 3 This is a sub-step flowchart of the present invention for generating comparison samples of target semantic features using a Gaussian mixture model. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The present invention provides a multi-label image classification method based on comparative sample generation learning, comprising the following steps:

[0034] S1: Use the feature extraction network to extract the target semantic features of the image.

[0035] Preferably, the feature extraction network includes an image feature extraction module, a semantic feature extraction module, and a semantic feature optimization module. Furthermore, the image feature extraction module is a convolutional neural network (CNN), the semantic feature extraction module is a spatial attention network, and the semantic feature optimization module is a graph neural network (GNN); in this step S1, the residual network (resnet) of the convolutional neural network is specifically used to extract image features and the attention mechanism is used to decouple the target semantic features therefrom, and the graph neural network is used to model the semantic correlation between the targets, thereby obtaining high-quality target semantic features. The step S1 is described in detail below.

[0036] The step S1 includes the following sub-steps:

[0037] S11: Using an image feature extraction module to extract features from the image to obtain image features.

[0038] Preferably, in sub-step S11, before feature extraction, the image is also pre-processed: the image is interpolated, specifically, the natural image in 8-bit RGB format is interpolated to unify its image length and width and normalize it, as shown in the following formula (1):

[0039]

[0040] In the above formula (1), I i is the image, bicubic represents the bilinear interpolation function, 0≤i<N, N is the number of images in the training set.

[0041] Preferably, sub-step S11 specifically uses the residual network of the convolutional neural network to process the preprocessed image Perform feature extraction as shown in the following formula (2):

[0042]

[0043] In the above formula (2), resnet101 is a residual network with 101 layers, and F i Represents image features.

[0044] S12: Decouple the image features using a semantic feature extraction module to obtain target semantic features.

[0045] The semantic feature extraction module corresponds to the semantic feature extraction stage, and aims to decouple the image features containing all image information to obtain the target features associated with the task (i.e., the multi-label image classification task), i.e., the semantic features (target semantic features). This embodiment uses the attention mechanism of the spatial attention network to decouple the features of the entire image to obtain the target semantic features required for the task: first, the target word embedding P associated with the task is obtained through the pre-trained word embedding model BERT, which represents the required target information associated with the task; then the spatial attention network is constructed to compare the target word embedding P with the image feature F i , and finally obtain the target semantic features S in the image features i The specific process is shown in the following equations (3), (4), (5), and (6), where linear is the linear layer; tanh is the inverse tangent triangle activation function; sum is the summation function, which sums in the channel dimension; softmax is the normalized activation function, which performs normalization in the length and width dimensions; equation (5) obtains the spatial attention weight attn of each target in the image; equation (6) represents the weighted sum of the attention weight of each target and its information to obtain the semantic feature S of each target i .

[0046]

[0047] S i =sum(attn×F i ) (6)

[0048] Among them, for the above-mentioned pre-trained word embedding model BERT, it is a pre-trained language model (PLM), the full name of BERT is Bidirectional Encoder Representations from Transformer (bidirectional encoder representation from Transformer), BERT is an encoder based on the Transformer model, BERT and the above-mentioned target word embedding P are existing technologies, and no further details are given here.

[0049] S13: Optimize the target semantic features obtained in sub-step S12 using a semantic feature optimization module.

[0050] Preferably, sub-step S13 is to use graph neural network for semantic interaction to optimize the target semantic features obtained in sub-step S12; in the semantic feature optimization stage, the image information is divided into K scenes, and the image label (target) information in different scenes is counted to obtain the semantic relationships between the targets associated with the tasks in different scenes, and then a graph neural network is constructed based on this; that is, the target semantic features are optimized by inputting the image scene information and the target semantic features into the graph neural network for semantic interaction.

[0051] Further, for this sub-step S13, first, the two-dimensional image feature F is average pooled. i It is converted into a one-dimensional vector representation, as shown in equation (7); then the scene to which it belongs is determined through a linear layer, as shown in equation (8); and the scene judgment is made normal by minimizing the entropy loss function and the regularized entropy loss function, as shown in equations (9), (10), and (11), where B is the batch size, is the image scene judgment output, L1 is the minimized entropy loss function, and L2 is the regularized entropy loss function.

[0052] G i =average pooling(F i ) (7)

[0053]

[0054]

[0055] Further, for the sub-step S13, the label co-occurrence matrix (semantic relationship) M of each scene is calculated by combining the image label information. k As shown in the following formula (12), where N k is the number of images of the kth scene, y k The image label for this scene.

[0056]

[0057] For this sub-step S13, a graph neural network is further constructed based on image scene judgment for symbiotic learning. The information propagation process of the graph neural network is shown in the following equations (13)-(17), where [] is a vector concatenation operator; W z , U z , W r , U r , W h , U s is a learnable parameter, and the entire information propagation process is repeated t times. is the target semantic feature (original semantic feature) S of the above sub-step S12 i The optimized target semantic features (optimized semantic features, i.e., the target semantic features finally extracted by the feature extraction network), where

[0058]

[0059] S2: Generate contrast samples of target semantic features using Gaussian mixture model.

[0060] In the semantic feature extraction optimization based on the attention mechanism and graph neural network in the above step S1, the graph neural network based on scene judgment is used to optimize the features. However, with continuous iterative training, the model is prone to overfitting problems, resulting in its judgment of image labels completely relying on the relationship between image scene information and targets without considering the specific target information present in the image. Therefore, the present invention uses a semantic feature optimization method based on contrastive learning to constrain various target semantic features to maintain discriminability with each other, thereby avoiding the overfitting problem of the model; at the same time, in order to solve the shortcomings of the number and quality of contrast samples in the existing contrastive learning methods, the target semantic feature distribution is set to multiple Gaussian distributions, and a Gaussian mixture model is used to fit the target semantic feature space, and then various target semantic features are sampled from the Gaussian mixture model as contrast samples, thereby improving the efficiency and effectiveness of contrastive learning, thereby optimizing the target semantic features and improving the classification accuracy of the model.

[0061] Contrastive Learning is a discriminative representation learning framework method based on contrastive thinking. Through contrastive learning, the representation distances of semantically similar examples are closer in the representation space, while the representation distances of semantically dissimilar examples are farther. Contrastive learning is divided into two stages: semantic feature collection and fitting, and semantic feature generation and comparison.

[0062] Preferably, step S2 includes the following sub-steps:

[0063] S21: Collect target semantic features.

[0064] In the semantic feature collection and fitting stage, the feature extraction network of the above step S1 is defined as Φ, and the parameter is W; when the model is trained, the semantic features of the training set images are extracted and collected using the model parameters of the previous round, as shown in the following formula (18); of course, the target semantic features of other training rounds can also be collected.

[0065]

[0066] S22: Update and fit multiple Gaussian distributions in combination with the target semantic features collected in sub-step S21.

[0067] In this sub-step S22, define C Gaussian mixture models (Gaussian distribution) Ψ with mean μ, mean square error ∑, weight ω, and component M, and combine the semantic feature set S collected by the above formula (18) t-1 The EM algorithm is used to update the parameters μ, ∑ and W as shown in the following equation (19), that is, multiple Gaussian distributions are updated and fitted.

[0068] μ t ,∑ t ,ω t =EM(μ t-1 ,∑ t-1 ,ω t-1 , S t-1 ) (19)

[0069] It can be understood that the Gaussian Mixture Model (GMM) uses the Gaussian probability density function (normal distribution curve) to accurately quantify things. It is a model that decomposes things into several Gaussian probability density functions (normal distribution curves). The Gaussian mixture model is a statistical model that assumes that all data points are generated by several Gaussian distributions. Each Gaussian distribution represents a "class", and the probability that a data point belongs to a class is determined by the Gaussian distribution of the class. The Gaussian mixture model estimates the parameters (mean and covariance matrix) of each Gaussian distribution by maximizing the likelihood function of all data points, thereby realizing data clustering.

[0070] The above-mentioned EM (Expectation-Maximization) algorithm is an iterative optimization technique used for maximum likelihood estimation of parameters of probability models containing latent variables. In the Gaussian mixture model, the EM algorithm is used to estimate the parameters of each Gaussian distribution and the probability that each data point belongs to each class (i.e., latent variables). As a method for GMM parameter estimation, the EM algorithm finds the optimal Gaussian distribution parameters and the category affiliation of each data point through an iterative optimization process.

[0071] S23: Randomly sample various target semantic features from multiple Gaussian distributions after updating and fitting as comparison samples.

[0072] In the semantic feature generation stage, the semantic features of each type of target are randomly sampled from the multiple Gaussian distributions after the above update fitting as comparison samples FS t As shown in the following formula (20), SN is the number of samples and C is the number of target categories to be classified.

[0073]

[0074] S3: Use contrast samples and target semantic features extracted by the feature extraction network for contrastive learning.

[0075] It can be understood that step S3 corresponds to the semantic feature comparison stage. t Compared with the real sample S t (According to the above formula (18) and the current model parameters of the feature extraction network, S t That is, the target semantic features extracted by the corresponding feature extraction network ) Calculate the contrast loss L cl As shown in the following formula (21), where FS tpa For FS t The same feature set as a in FS tna For FS t The feature set in a is heterogeneous with a. Through contrastive learning, the distances between relevant features (positive samples) are constrained to be close to each other, and the distances between irrelevant features (negative samples) are constrained to be far away from each other. That is, through contrastive learning, various target semantic features are constrained to maintain discriminability with each other, and the target semantic features are optimized.

[0076]

[0077] S4: Use the target semantic features after contrastive learning to classify image labels.

[0078] Preferably, step S4 is to classify the target semantic features after contrast learning by a linear layer: the target semantic features (original semantic features) of the above sub-step S12 are combined by a linear layer. i Compared with the above, the target semantic features after learning (optimized semantic features) Calculate image label classification results As shown in the following equations (22) and (23). In addition, while performing image label classification in step S4, the cross entropy loss function is used to constrain and optimize the entire network, as shown in the following equation (24), where y i is the true value of the image label.

[0079]

[0080] In addition, the following Table 1 compares the classification effects of the method proposed in the present invention (Ours) and the prior art method on the public dataset MSCOCO2014.

[0081] Table 1 Comparison of classification effects between the proposed method and existing methods on MSCOCO2014

[0082] Method Backbone Resolution mAP↑ CP↑ CR↑ CF1↑ OP↑ OR↑ OF1↑ Q2L ResNet101 448×448 84.9 84.8 74.5 79.3 86.6 76.9 81.5 MulCon ResNet101 448×448 84.9 84.0 74.8 79.2 85.6 78.0 81.6 SADCL ResNet101 448×448 85.6 84.6 76.0 79.8 86.0 78.5 82.1 SALGL ResNet101 448×448 85.75 87.2 74.5 80.4 87.8 77.6 82.4 Ours ResNet101 448×448 85.84 70.4 85.0 76.7 69.9 87.6 77.8

[0083] It can be seen from Table 1 above that the classification effect of the method proposed in the present invention on the public dataset MSCOCO2014 is better than that of the existing methods, and it takes the lead in the main classification performance indicators such as mAP.

[0084] Compared with the prior art, the present invention provides a multi-label image classification method based on contrast sample generation learning, which has the following beneficial effects: (1) The present invention uses contrast learning to constrain the distance between different types of target semantic features, that is, through contrast learning, the semantic features of each type of target are constrained to maintain the discriminability between each other, thereby improving the problem of easy overfitting of the existing multi-label image classification method, so as to improve the image label classification accuracy; (2) In addition, in view of the problem that contrast learning requires a large number of contrast samples, the present invention uses a Gaussian mixture model to generate contrast samples of target semantic features, which can solve the problem of insufficient number and low quality of contrast samples in contrast learning, thereby improving the efficiency and effectiveness of contrast learning, and ultimately improving the model classification ability.

[0085] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0086] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-label image classification method based on contrast sample generation learning, characterized in that: The following steps are involved: S1: Extract the target semantic features of the image using the feature extraction network; S2: Generate comparison samples of target semantic features using Gaussian mixture model; S3: performing comparative learning using the comparison sample and the target semantic features extracted by the feature extraction network; S4: classifying the target semantic features after the contrastive learning into image labels.

2. The multi-label image classification method based on contrast sample generation learning according to claim 1 is characterized in that: The feature extraction network includes an image feature extraction module, a semantic feature extraction module and a semantic feature optimization module; the step S1 includes the following sub-steps: S11: extracting features from the image using the image feature extraction module to obtain image features; S12: Decoupling the image features using the semantic feature extraction module to obtain target semantic features; S13: Utilize the semantic feature optimization module to optimize the target semantic features obtained in the sub-step S12.

3. The multi-label image classification method based on contrast sample generation learning according to claim 2 is characterized in that: The image feature extraction module is a convolutional neural network, and the sub-step S11 specifically utilizes the residual network of the convolutional neural network to extract features from the image.

4. The multi-label image classification method based on contrast sample generation learning according to claim 3 is characterized in that: In the sub-step S11, before performing feature extraction, it also includes: performing interpolation processing on the image.

5. The multi-label image classification method based on contrast sample generation learning according to claim 2 is characterized in that: The semantic feature extraction module is a spatial attention network.

6. The multi-label image classification method based on contrast sample generation learning according to claim 2 is characterized in that: The semantic feature optimization module is a graph neural network, and the sub-step S13 is to use the graph neural network to perform semantic interaction to optimize the target semantic features obtained in the sub-step S12.

7. The multi-label image classification method based on contrast sample generation learning according to claim 1, characterized in that: The step S2 comprises the following sub-steps: S21: collect target semantic features; S22: updating and fitting multiple Gaussian distributions in combination with the target semantic features collected in the sub-step S21; S23: Randomly sampling various target semantic features from the multiple Gaussian distributions after updating and fitting as the comparison samples.

8. The multi-label image classification method based on contrast sample generation learning according to claim 1, characterized in that: The step S4 is to classify the target semantic features after contrast learning into image labels through a linear layer.

9. The multi-label image classification method based on contrast sample generation learning according to claim 8, characterized in that: While performing image label classification in step S4, a cross entropy loss function is used for constraints.

Citation Information

Patent Citations

  • GCN-based multi-label image classification method, model construction method and device

    CN114612681A

  • Multi-target classification model training method, multi-target identification method and electronic equipment

    CN117689938A

  • Multi-label image classification method based on attention mechanism and double contrast learning

    CN118097243A

  • Multi-label image classification method and apparatus, and multi-label image classification model training method and apparatus

    WO2024124770A1