Multi-label image classification model training method, application method and related system

Abstract

The invention discloses a multi-label image classification model training method, an application method and a related system, and relates to the technical field of image classification processing, and the training method comprises the steps: obtaining a to-be-classified sample image; performing feature extraction on the sample image to obtain image features; calculating Euclidean distance-based feature neighborhoods and Hamming distance-based label neighborhoods of the sample images on each category representation so as to respectively determine feature maps and label maps of the sample images on each category representation; respectively determining an aggregation feature represented by each category on the feature graph and an aggregation feature represented by each category on the tag graph; inputting into a classifier of a multi-label image classification model for classification; a knowledge distillation technology is adopted, prediction response of a label graph is extracted from classification loss to serve as supervision knowledge which is used for guiding learning of a feature graph, and a trained model is obtained after training optimization. According to the invention, the accuracy of multi-label image classification can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of image classification processing, and in particular to a multi-label image classification model training method, application method and related system. Background Art

[0002] In the field of image classification, multi-label image classification tasks are extremely challenging due to their complexity and diversity. In actual scenarios, a sample may have multiple labels to reveal its attributes. Therefore, the model needs to predict these labels at the same time. At present, multi-label image classification methods usually use the global features of the image to predict multiple categories. This approach lacks attention to each category, making it difficult to classify some insignificant categories of visual features in the input image. Therefore, the classification accuracy of this method is not high. The current method combined with GNN (Graph Neural Network) first extracts the representation of each category from the global features of the image sample, and then learns the graph representation of all categories in the sample. However, most categories do not exist in an image. Using all category representations in an image as nodes to form a graph will inevitably introduce noise information of negative classes, which may cause the model to make incorrect classifications. Therefore, the classification accuracy of this method is also not high. Summary of the invention

[0003] The purpose of this application is to provide a multi-label image classification model training method, application method and related system, which can improve the accuracy of multi-label image classification.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a multi-label image classification model training method, the multi-label image classification model training method comprising:

[0006] A sample image to be classified is obtained; the sample image includes a plurality of labels, and each label corresponds to a category.

[0007] Feature extraction is performed on the sample image to obtain image features.

[0008] According to the image features, the feature neighbors based on Euclidean distance and the label neighbors based on Hamming distance of the sample image in each category representation are calculated.

[0009] A feature map and a label map of the sample image on each category representation are determined respectively according to the feature neighbors and the label neighbors.

[0010] According to the feature map and the label map, aggregate features of each category representation on the feature map and aggregate features of each category representation on the label map are determined respectively.

[0011] The aggregated features representing each category on the feature map and the aggregated features representing each category on the label map are respectively input into the classifier of the multi-label image classification model for classification to determine the classification loss; the multi-label image classification model is a model that uses the aggregated features as input and the predicted label distribution result as output to classify multi-label images.

[0012] The knowledge distillation technology is used to extract the predicted response of the label map from the classification loss as supervisory knowledge to guide the learning of the feature map, and the multi-label image classification model is trained and optimized to obtain a trained multi-label image classification model.

[0013] In a second aspect, the present application provides a multi-label image classification model application method, the multi-label image classification model application method comprising:

[0014] A target image to be classified is obtained; the target image includes a plurality of labels, and each label corresponds to a category.

[0015] Feature extraction is performed on the target image to obtain image features.

[0016] According to the image features, feature neighbors of the target image based on Euclidean distance in each category representation are calculated.

[0017] A feature map of the target image on each category representation is determined according to the feature neighbors.

[0018] According to the feature map, aggregate features represented by each category on the feature map are determined.

[0019] The aggregated features represented by each category on the feature map are input into the classifier of the multi-label image classification model for classification, and the label distribution result corresponding to the target image is predicted; the multi-label image classification model is a model trained according to the multi-label image classification model training method described in the first aspect.

[0020] In a third aspect, the present application provides a multi-label image classification model training system, the multi-label image classification model training system comprising:

[0021] The sample image acquisition module is used to acquire a sample image to be classified; the sample image includes a plurality of labels, and each label corresponds to a category.

[0022] The sample image feature extraction module is used to extract features from the sample image to obtain image features.

[0023] The feature neighbor and label neighbor calculation module is used to calculate the feature neighbor based on Euclidean distance and the label neighbor based on Hamming distance of the sample image in each category representation according to the image features.

[0024] The feature map and label map determination module is used to determine the feature map and label map of the sample image in each category representation according to the feature neighbors and the label neighbors.

[0025] The feature map and label map aggregate feature determination module is used to determine the aggregate features represented by each category on the feature map and the aggregate features represented by each category on the label map according to the feature map and the label map.

[0026] The classification loss determination module is used to input the aggregated features represented by each category on the feature map and the aggregated features represented by each category on the label map into the classifier of the multi-label image classification model for classification, and determine the classification loss; the multi-label image classification model is a model that uses the aggregated features as input and the predicted label distribution results as output to classify multi-label images.

[0027] The model training module is used to adopt the knowledge distillation technology to extract the predicted response of the label map from the classification loss as the supervision knowledge, which is used to guide the learning of the feature map, train and optimize the multi-label image classification model, and obtain a trained multi-label image classification model.

[0028] In a fourth aspect, the present application provides a multi-label image classification model application system, the multi-label image classification model application system comprising:

[0029] The target image acquisition module is used to acquire a target image to be classified; the target image includes a plurality of labels, and each label corresponds to a category.

[0030] The target image feature extraction module is used to extract features from the target image to obtain image features.

[0031] The feature nearest neighbor calculation module is used to calculate the feature nearest neighbor of the target image based on the Euclidean distance in each category representation according to the image features.

[0032] The feature map determination module is used to determine the feature map of the target image in each category representation according to the feature neighbors.

[0033] The feature graph aggregate feature determination module is used to determine the aggregate features represented by each category on the feature graph based on the feature graph.

[0034] A classification module is used to input the aggregated features represented by each category on the feature map into the classifier of the multi-label image classification model for classification, and predict the label distribution result corresponding to the target image; the multi-label image classification model is a model trained according to the multi-label image classification model training method described in the first aspect.

[0035] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0036] The present application provides a multi-label image classification model training method, application method and related system, which calculates the feature neighbors based on Euclidean distance and label neighbors based on Hamming distance of each category according to the image features, and then further constructs the feature map and label map of the sample image on each category representation according to the feature neighbors and label neighbors. During model training, the aggregated features on the feature map and the aggregated features on the label map are input into the model, and the knowledge distillation technology is used to extract the predicted response of the label map as supervisory knowledge to guide the learning of the feature map, so that the multi-label image classification model can fully learn the image features according to the actual labels, and realizes the combination of label map classification loss, feature map classification loss and knowledge distillation loss, and jointly optimizes the model parameters. Therefore, the multi-label image classification model trained based on the above training method has high precision and strong classification performance, and improves the accuracy of the model for multi-label image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A flowchart of a multi-label image classification model training method provided in Example 1 of the present application.

[0039] Figure 2 Schematic diagram of the multi-label image classification model training process provided in Example 1 of the present application.

[0040] Figure 3 This is a schematic diagram of the inference process of the multi-label image classification model provided in Example 1 of the present application.

[0041] Figure 4A flowchart of a multi-label image classification model application method provided in Example 2 of the present application.

[0042] Figure 5 A structural schematic diagram of a multi-label image classification model training system provided in Example 3 of the present application.

[0043] Figure 6 A structural diagram of a multi-label image classification model application system provided in Example 4 of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] Example 1

[0046] like Figure 1 and Figure 2 As shown, this embodiment proposes a multi-label image classification model training method, which is applied to a multi-label image classification model and is a method for training a multi-label image classification model. The multi-label image classification model training method includes the following steps:

[0047] Step S101, obtaining a sample image to be classified. The sample image includes multiple labels, and each label corresponds to a category.

[0048] This embodiment uses the VOC (Visual Object Classes) 2007 dataset when acquiring sample images to be classified. The VOC2007 dataset contains images from 20 categories, with a total of 9963 images, of which 5011 are used as training images and 4952 are used as test images.

[0049] Step S102: extract features from the sample image to obtain image features. Specifically, the following steps are included:

[0050] Step S1021 , performing scaling processing, cropping processing, and random flipping processing on the sample image in sequence to obtain a pre-processed sample image.

[0051] Step S1022: input the preprocessed sample image into the TResNet pre-training model to extract image features.

[0052] After obtaining the sample image to be classified, this embodiment performs preprocessing operations on it, including scaling, cropping, random flipping, etc., and obtains its real label vector, which is sent to the feature extraction network to extract image features. Specifically, the input sample image to be classified is scaled to a resolution of 448×448, randomly cropped, randomly flipped, etc., and input into the feature extraction network.

[0053] This embodiment uses the TResNet model as the feature extraction network. The specific processing process of the TResNet model is as follows: first, the pre-trained TResNet model extracts image features; then the extracted image features are flattened and position embeddings are added; then the features and position embeddings are input into the Transformer encoder for feature encoding; then the query embedding and position embedding are used to guide the Transformer decoder to generate category representations.

[0054] In this embodiment, extracting image features specifically includes the following contents:

[0055] (1) The input image data is denoted as I, and its size is (H, W, C), where H and W represent the height and width of the image, respectively, and C represents the number of channels. The input image is scaled to a resolution of 448 × 448, cropped, randomly flipped, etc., and then input into the feature extraction network.

[0056] (2) Use the TResNet model to extract image features. The image features output by the convolutional neural network are denoted as F∈R H '×W'×D , where H' and W' are the height and width of the feature map, respectively, and D is the feature dimension. The process is expressed as:

[0057] F = TResNet(I) (1).

[0058] The input image is processed through several convolutional layers, pooling layers, and residual blocks of the TResNet model, and finally the image feature F is obtained at the output layer of the network. This feature can be regarded as a representation of the image in a high-dimensional space, which contains information such as the texture and shape of the image.

[0059] (3) Input the image features into the Transformer encoder for feature encoding. First, flatten the feature map into a sequence:

[0060] F flat =reshape(F,(H'W',D)) (2).

[0061] Among them, F flat represents the flattened features, F(H',W',D) represents the image features with height H', width W', and dimension D; reshape(·) represents the reshape function.

[0062] Then, we add the position embedding P∈R H'W'×D , expressed as:

[0063] F pos =F flat +P (3).

[0064] Among them, F pos It means that the position embedded feature is added to the flattened feature, and P represents the position embedding.

[0065] The calculation method of position embedding can be expressed as:

[0066]

[0067]

[0068] Among them, pos represents the position, P (pos,2i) represents the position embedding at the even-numbered dimension index of the pos position, P (pos,2i+1) Represents the position embedding at the odd dimension index of the pos position, i represents the dimension index, and d represents the embedding dimension.

[0069] Then, the input encoder layer is expressed as:

[0070] E = TransformerEncoder(F pos ) (6).

[0071] Where E represents the output of the encoder, E∈R H'×W'×D , TransformerEncoder(·) represents the Transformer encoder.

[0072] (4) Use query embedding and position embedding to guide the Transformer decoder to generate category representation:

[0073] Define query embedding Q∈R C×D , where C is the number of categories. The query input of the decoder can be initialized as a tensor with a value of 0, that is, Q = 0 ∈ R C×D The decoder forward propagation process is as follows:

[0074] D=TransformerDecoder(query=Q, key=E, value=E) (7).

[0075] Where D represents the output of the decoder, D∈R C×D , TransformerDecoder(·) represents a Transformer decoder. In this embodiment, the number of categories C=20, and the feature dimension D=2432.

[0076] Step S103: Calculate the feature neighbors based on Euclidean distance and the label neighbors based on Hamming distance of the sample image in each category representation according to the image features.

[0077] After extracting the image features of the sample image, this embodiment calculates the first k feature neighbors of the sample image in each category representation based on Euclidean distance and the first k label neighbors of each category of the sample image based on Hamming distance. Specifically, the following steps are included:

[0078] Step S1031: a sample image set F composed of all the sample images c ={x 1c ,x 2c …,x ic ,…x Nc}, respectively calculating the Euclidean distance between each sample image in the sample image set and the remaining sample images; wherein, x ic ∈R d is the cth category representation vector of the ith sample image, c represents the category, d represents the dimension, and N represents the total number of sample images.

[0079] Step S1032: According to the Euclidean distance, for each of the sample images, the first k sample images with the closest distance are selected as the feature neighbors of the sample image in category c, and the feature neighbors based on the Euclidean distance of the sample image in each category representation are obtained. This step is performed for C categories to obtain the feature neighbors of each sample image in all categories. In this embodiment, k is set to 30, and the first 30 closest samples are selected for each category to add to the feature neighbors, where C represents the number of categories, that is, the total number of all categories; c represents the category, which refers to a category among all categories, such as the cth category.

[0080] Step S1033: extract the label vector corresponding to each sample image to form a label set Y = {y1, y2…, y i ,…y N}, where y i ={y i1 ,y i2 ,...,y iC} represents the label vector of the i-th sample image. When the sample image contains category c, then y ic =1, otherwise y ic =0.

[0081] Step S1034: Based on the label set, calculate the Hamming distance between the label vector corresponding to each of the sample images in the label set and the label vectors corresponding to the remaining sample images.

[0082] Step S1035: According to the Hamming distance, for each sample image, select the first k label neighbors with the closest distance as the label neighbors of the sample image in category c, and obtain the label neighbors based on the Hamming distance of the sample image in each category representation. In this embodiment, k is set to 30, and the first 30 closest samples are selected for each category to add to the label neighbors.

[0083] Step S104: Determine a feature map and a label map of the sample image in each category representation according to the feature neighbors and the label neighbors.

[0084] This embodiment constructs two graphs for each class of input samples based on the obtained feature neighbors and label neighbors, namely, feature graph G f and label graph G l The specific steps include:

[0085] Step S1041: For feature graph G f , the feature representation f of the sample image in category c input in each training batch c ={x 1c ,x 2c …,x ic ,…x nc} as the k-th layer node, using H k Represents, and at the same time constitutes the original node in the neighbor graph; where x ic ∈R d is the c-th class representation vector of the ith sample image in the training batch, and n represents the number of the sample images input into a training batch.

[0086] Step S1042: For H k For each node in , query the feature neighbor of the node on the category, and add the category representation of the feature neighbor as a node to the neighbor graph; for an edge, if node j is the feature neighbor of node i, then there is an edge connecting the i-th node and the j-th node in the feature graph.

[0087] Step S1043: Use the adjacent nodes obtained by query to form the k-1th layer of nodes, and use H k-1 Indicates that, according to the set search depth K, the operation of "querying the feature neighbors of the node in the category, and adding the category representation of the feature neighbors as nodes to the neighbor graph" is performed layer by layer, and the operation is repeated K times until the node H0 of the 0th layer is obtained, and the feature graph of the sample image in each category representation is obtained. In this embodiment, K is set to 2, and a total of two layers of neighbor queries are performed. Thus, the feature graph G is constructed. f And the corresponding node information of each layer.

[0088] Then use a similar method to get the label graph G l The corresponding node information of each layer is constructed in a similar way to the feature map. For each layer of nodes, its label neighbors are queried, and the feature representation of the neighbor in the category is added to the label map. This operation is repeated K times. The specific steps include:

[0089] Step S1044: For label graph G l , the feature representation f of the sample image in category c input in each training batch c ={x 1c ,x 2c …,x ic ,…x nc} as the k-th layer node, using H k Indicates, and at the same time constitutes the original node in the label graph; where x ic ∈R d is the c-th class representation vector of the ith sample image in the training batch, and n represents the number of the sample images input into a training batch.

[0090] Step S1045: For H k For each node in , query the label neighbor of the node on the category, and add the category representation of the label neighbor as a node to the neighbor graph; for an edge, if node j is the label neighbor of node i, then there is an edge connecting the i-th node and the j-th node in the label graph.

[0091] Step S1046: Use the adjacent nodes obtained by the query to form the k-1th layer of nodes, and use H k-1 Indicates that, according to the set search depth K, the operation of "querying the label neighbors of the node in the category and adding the category representation of the label neighbors as nodes to the neighbor graph" is performed layer by layer, and the operation is repeated K times until the node H0 of the 0th layer is obtained, and the label graph of the sample image in each category representation is obtained. In this embodiment, K is set to 2, and a total of two layers of neighbor queries are performed. Thus, the label graph G is constructed. l And the corresponding node information of each layer.

[0092] Step S1047: Perform the above operations for each category c to obtain the feature map of the input sample in category c. and label map

[0093] Step S105: Determine, according to the feature map and the label map, respectively, aggregate features represented by each category on the feature map and aggregate features represented by each category on the label map.

[0094] After obtaining the feature map and the label map, this embodiment performs sampling and aggregation operations on each feature map and the corresponding label map, aggregates the features of neighbor nodes layer by layer, and obtains various aggregate features represented on the two maps. Specifically, the following steps are included:

[0095] Step S1051: Set an aggregator for each layer k , used for H k The feature representations of neighbor nodes of the layer are aggregated.

[0096] Step S1052: Before the k-th layer aggregation operation, for H k+1 The node v in the layer is uniformly sampled from its neighbor nodes to obtain the neighbor set N(v) after sampling. The characteristics of each node u in N(v) They are all processed independently by an Edge module, which fuses the features of neighbor nodes with the features of node v through low-rank bilinear pooling.

[0097] And from this, an attention coefficient is calculated as the weight of the edge between v and its neighbors. This weight is learnable. The above steps can be expressed as:

[0098]

[0099] Among them, x v,u is the fusion feature of node v and u, P, U, V, b are learnable parameters, N(v) is the neighbor set of node v obtained by uniform sampling. v,u is the attention coefficient, i.e., the learnable edge weight, and Φ(·) is an attention function implemented by a fully connected network. In this embodiment, d1 and d2 are both kept equal to d k Same dimensions.

[0100] Step S1053: Set the edge weight α v,u And the corresponding neighbor node features Multiply to get the node features after attention weighting It is expressed as:

[0101]

[0102] in, It is the node feature after attention weighting.

[0103] Step S1054: Send it to the aggregator to aggregate the weighted neighbor features. The aggregator of each layer k This is done by the max pooling operation. The aggregator can be expressed as:

[0104]

[0105] Among them, aggregator k represents the aggregator, max(·) is the maximum pooling operator, and the maximum value on each column is selected as the aggregation result. The aggregator at each layer is shared by C feature maps and label maps.

[0106] Step S1055: and After the splicing operation, the updated The entire aggregation operation can be expressed as:

[0107]

[0108] Among them, σ is a nonlinear function, is a learnable transformation matrix, which transforms the node features Updated to where d k is the feature dimension before the k-th layer node aggregation, d k+1 is the dimension after node aggregation update, concat(·) is the concat function, Node feature Updated node features, Represents the aggregation result.

[0109] Step S106: Input the aggregated features of each category represented on the feature map and the aggregated features of each category represented on the label map into the classifier of the multi-label image classification model for classification, and determine the classification loss. The multi-label image classification model is a model that uses the aggregated features as input and the predicted label distribution result as output to classify multi-label images.

[0110] After obtaining the aggregated features of the feature graph and the aggregated features of the label graph, this embodiment sends each category representation after the feature graph and the label graph are aggregated, that is, the aggregated features of each category representation on the feature graph and the aggregated features of each category representation on the label graph, to the same classifier for classification. After performing feature graph learning and label graph learning on the representation of each category, the corresponding enhanced representation can be obtained, that is, the feature graph representation set D F =[D f1 ,D f2 ,...,D fC ] T ∈R C×d and label graph representation set D L =D l1,D l2 ,...,D lC ] T ∈R C×d , where C is the total number of categories and d is the final feature dimension of each category.

[0111] In order to perform multi-label classification tasks, this embodiment regards the prediction of each label as a binary classification task, projects the features of each category to logit (logical) values ​​through linear projection, and classifies the feature map representation and label map representation respectively. This step can be expressed as:

[0112]

[0113] Among them, W c , b c is a learnable parameter, W c ∈R d , b c ∈R;D fc represents the feature map representation, p fc represents the predicted response of the feature map, p F represents the set of predicted responses of each category after feature map learning, p F =[p f1 ,…,p fC ] T ∈R C ; accordingly, D lc represents the label graph representation, p lc represents the predicted response of the label map, p L represents the set of predicted responses for each category after label map learning, p L =[p l1 ,…,p lC ] T ∈R C , T represents transpose.

[0114] Step S107: Using the knowledge distillation (KD) technology, extract the predicted response of the label map from the classification loss as supervisory knowledge to guide the learning of the feature map, train and optimize the multi-label image classification model, and obtain a trained multi-label image classification model.

[0115] This embodiment uses the traditional multi-label classification loss function, namely the binary cross entropy loss function, to calculate the classification loss of the feature map representation and the label map representation respectively. and The binary cross entropy loss function is expressed as:

[0116]

[0117] in, represents the loss function, y c is the true label of the sample image in category c, σ(p c ) is the predicted label, and σ(·) is the sigmoid function.

[0118] Knowledge distillation is a classic model compression method. Its core idea is to improve the performance of the student model without changing its structure by guiding the lightweight student model to "imitate" the teacher model with better performance and more complex structure. This embodiment takes into account that the label map fully utilizes the real label information of the sample image. The quality of the label map representation is extremely high, which can guide the learning of the multi-label image classification model in terms of feature maps. Therefore, this embodiment uses the idea of ​​knowledge distillation and uses the predicted response represented by the label map as high-quality supervisory knowledge to guide the learning of the feature map, which is expressed as:

[0119]

[0120] in, represents the knowledge distillation loss, p fc represents the predicted response of the feature map, p lc represents the predicted response of the label map, and σ(·) is the sigmoid function.

[0121] This embodiment combines the classification loss (including feature map classification loss and label map classification loss) with the knowledge distillation loss to jointly optimize the model, and its loss function is expressed as:

[0122]

[0123] in, represents the loss function, represents the feature map classification loss, Represents the label graph classification loss

[0124] lose, represents the knowledge distillation loss, and λ represents the weighting coefficient.

[0125] In this embodiment, λ is set to 0.5. Based on the loss function of the combination of feature map classification loss, label map classification loss and knowledge distillation loss, a stochastic gradient descent (SGD) optimizer is used, with the momentum set to 0.9 and the weight decay to 0.0001. The back propagation (BP) algorithm is used to learn and update the parameters of the model, and finally a multi-label image classification model with training convergence is obtained. The multi-label image classification model is actually a feature map model. In practical applications, accurate classification of multi-label images can be achieved based on feature maps. The aggregated features of each category represented on the feature map are input into the trained multi-label image classification model to predict the corresponding multi-label distribution results.

[0126] In the feature extraction stage of this embodiment, the cross-attention module built into the Transformer decoder provides an effective way to use label embedding as a query to detect and aggregate class-related features from the feature map calculated by the visual backbone for subsequent classification tasks. Then, based on the extracted reliable feature representation, a label map and a feature map representing each category are constructed. Among them, the label map makes full use of the real label information of the sample, does not introduce any information that does not exist in the sample, and the learned representation of each category can be not limited to a single sample and truly reflects the feature knowledge of the category. Since the real label of the sample cannot be known in the reasoning stage of the actual application of the model. Therefore, this embodiment adopts the method of feature map learning, and uses the high-quality representation ability of label map learning to guide the model to better learn the feature map. The model finally obtained by the present invention can realize feature extraction of input images with unknown labels, calculate feature neighbors in the training data set, construct feature maps and perform graph learning, predict real labels, and realize multi-label image classification tasks.

[0127] In order to verify the effectiveness of the multi-label image classification model for classification tasks, this embodiment selects some of the current mainstream models for comparative verification, and takes mAP (meanAverage Precision) as an indicator. The verification results are shown in Table 1. According to Table 1, it can be seen that the model trained by the multi-label image classification model training method has the highest mAP score, high classification accuracy and strong performance, and can accurately complete the multi-label image classification task.

[0128] Table 1 Comparison of model classification results

[0129]

[0130]

[0131] At present, the multi-label image classification method based on graph learning will inevitably introduce the noise representation of negative classes, which may cause the learned image representation to contain a lot of negative information, further affecting the decision-making performance of the model. In this embodiment, the image feature extraction network based on the TResNet model extracts each class representation of the input sample, and on the basis of the feature representation of the specific class of the sample, respectively constructs a graph structure based on the label and feature (i.e., the label graph and the feature graph), uses the attention mechanism to obtain the learnable edge weight, and simultaneously performs graph learning on the label graph and the feature graph to enhance the representation ability of each class, thereby improving the classification performance of the model and improving the classification accuracy of the model. Both the label graph and the feature graph use the representation of each class as the graph node. Among them, the label graph only exists in the training stage (the training stage of the multi-label image classification model), and the real label of the sample is used to determine the connection state of the edge, which can accurately aggregate the same category representation. Using knowledge distillation technology, the high-quality representation ability of label graph learning is transferred to feature graph learning. In the inference stage (the actual application stage of the trained multi-label image classification model), this embodiment only uses feature maps that do not require label information to complete an effective multi-label image category prediction process and predicts a more accurate and reliable label distribution result for the target image.

[0132] The multi-label image classification model training method proposed in this embodiment constructs a feature map and a label map for each category representation. The graph nodes come from all training samples, and the information of multiple training samples can be aggregated to obtain each category of graph representation with better generalization ability. In addition, this embodiment implements a classification method for dual-graph representation learning based on feature maps and label maps. By simultaneously constructing a label map based on label relationships and a feature map based on feature relationships, and using the high-quality representation ability of the label map that considers the real label information to guide the model to perform better feature map learning, the trained multi-label image classification model has ideal classification accuracy and classification performance.

[0133] Example 2

[0134] like Figure 3 and Figure 4 As shown, this embodiment proposes a multi-label image classification model application method, which is actually an application method for performing actual reasoning using the trained multi-label image classification model after training the multi-label image classification model using the multi-label image classification model training method of embodiment 1. The multi-label image classification model application method includes the following steps:

[0135] Step S201: Obtain a target image to be classified. The target image includes multiple labels, and each label corresponds to a category.

[0136] Step S202: extract features from the target image to obtain image features.

[0137] Step S203: Calculate the feature neighbors of the target image based on the Euclidean distance in each category representation according to the image features.

[0138] Step S204: Determine a feature map of the target image in each category representation according to the feature neighbors.

[0139] Step S205: Determine, according to the feature graph, aggregate features represented by each category on the feature graph.

[0140] Step S206: Input the aggregated features of each category represented on the feature map into the classifier of the multi-label image classification model for classification, and predict the label distribution result corresponding to the target image. The multi-label image classification model is a model trained according to the multi-label image classification model training method described in Example 1. The steps in this embodiment are the same as the processing process of the feature map part of the sample image in Example 1.

[0141] Example 3

[0142] like Figure 5 As shown, this embodiment proposes a multi-label image classification model training system, which is actually a system corresponding to the multi-label image classification model training method in Embodiment 1. The multi-label image classification model training system includes the following functional modules:

[0143] The sample image acquisition module M1 is used to acquire a sample image to be classified; the sample image includes a plurality of labels, and each label corresponds to a category.

[0144] The sample image feature extraction module M2 is used to extract features from the sample image to obtain image features.

[0145] The feature neighbor and label neighbor calculation module M3 is used to calculate the feature neighbors based on Euclidean distance and the label neighbors based on Hamming distance of the sample image in each category representation according to the image features.

[0146] The feature map and label map determination module M4 is used to determine the feature map and label map of the sample image in each category representation according to the feature neighbors and the label neighbors.

[0147] The feature map and label map aggregate feature determination module M5 is used to determine the aggregate features represented by each category on the feature map and the aggregate features represented by each category on the label map according to the feature map and the label map.

[0148] The classification loss determination module M6 is used to input the aggregated features represented by each category on the feature map and the aggregated features represented by each category on the label map into the classifier of the multi-label image classification model for classification, and determine the classification loss; the multi-label image classification model is a model that uses the aggregated features as input and the predicted label distribution results as output, and is used to classify multi-label images.

[0149] The model training module M7 is used to adopt the knowledge distillation technology to extract the predicted response of the label map from the classification loss as the supervisory knowledge, which is used to guide the learning of the feature map, train and optimize the multi-label image classification model, and obtain a trained multi-label image classification model.

[0150] Example 4

[0151] like Figure 6 As shown, this embodiment provides a multi-label image classification model application system, which is actually a system corresponding to the multi-label image classification model application method in Embodiment 2. The multi-label image classification model application system includes the following functional modules:

[0152] The target image acquisition module N1 is used to acquire a target image to be classified. The target image includes multiple labels, and each label corresponds to a category.

[0153] The target image feature extraction module N2 is used to extract features from the target image to obtain image features.

[0154] The feature nearest neighbor calculation module N3 is used to calculate the feature nearest neighbor of the target image based on the Euclidean distance in each category representation according to the image features.

[0155] The feature map determination module N4 is used to determine the feature map of the target image in each category representation according to the feature neighbors.

[0156] The feature graph aggregate feature determination module N5 is used to determine the aggregate features represented by each category on the feature graph according to the feature graph.

[0157] The classification module N6 is used to input the aggregated features represented by each category on the feature map into the classifier of the multi-label image classification model for classification, and predict the label distribution result corresponding to the target image; the multi-label image classification model is a model trained according to the multi-label image classification model training method described in Example 1.

[0158] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-label image classification model training method, characterized in that: The multi-label image classification model training method comprises: Obtain a sample image to be classified; the sample image includes multiple labels, and each label corresponds to a category; Extracting features from the sample image to obtain image features; According to the image features, calculating the feature neighbors based on Euclidean distance and the label neighbors based on Hamming distance of the sample image in each category representation; Determine a feature map and a label map of the sample image on each category representation according to the feature neighbors and the label neighbors respectively; According to the feature map and the label map, respectively determine the aggregated features represented by each category on the feature map and the aggregated features represented by each category on the label map; The aggregated features represented by each category on the feature map and the aggregated features represented by each category on the label map are respectively input into the classifier of the multi-label image classification model for classification, and the classification loss is determined; the multi-label image classification model is a model for classifying multi-label images using the aggregated features as input and the predicted label distribution result as output; The knowledge distillation technology is used to extract the predicted response of the label map from the classification loss as supervisory knowledge to guide the learning of the feature map, and the multi-label image classification model is trained and optimized to obtain a trained multi-label image classification model.

2. The multi-label image classification model training method according to claim 1, characterized in that: Extracting features from the sample image to obtain image features specifically includes: The sample image is sequentially scaled, cropped and randomly flipped to obtain a preprocessed sample image; The preprocessed sample image is input into the TResNet pre-training model to extract image features.

3. The multi-label image classification model training method according to claim 1, characterized in that: According to the image features, calculating the feature neighbors based on Euclidean distance and the label neighbors based on Hamming distance of the sample image in each category representation, specifically including: Based on the sample image set F composed of all the sample images c ={x 1c ,x 2c …,x ic ,…x Nc }, respectively calculating the Euclidean distance between each sample image in the sample image set and the remaining sample images; wherein, x ic ∈R d is the c-th category representation vector of the i-th sample image, where c represents the category, d represents the dimension, and N represents the total number of sample images; According to the Euclidean distance, for each of the sample images, the first k sample images with the closest distance are selected as the feature neighbors of the sample image in category c, so as to obtain the feature neighbors of the sample image based on the Euclidean distance in each category representation; Extract the label vector corresponding to each sample image to form a label set Y = {y1, y2…, y i ,…y N }, where y i ={y i1 ,y i2 ,...,y iC } represents the label vector of the i-th sample image; Based on the label set, calculating the Hamming distance between the label vector corresponding to each of the sample images in the label set and the label vectors corresponding to the remaining sample images; According to the Hamming distance, the first k label neighbors with the closest distance are selected for each of the sample images as the label neighbors of the sample image in category c, so as to obtain the label neighbors of the sample image based on the Hamming distance in each category representation.

4. The multi-label image classification model training method according to claim 1, characterized in that: Determining a feature map and a label map of the sample image on each category representation according to the feature neighbors and the label neighbors respectively, specifically comprising: For the feature map G f , the feature representation f of the sample image in category c input in each training batch c ={x 1c ,x 2c …,x ic ,…x nc } as the k-th layer node, using H k Represents, and at the same time constitutes the original node in the neighbor graph; where x ic ∈R d is the c-th class representation vector of the ith sample image in the training batch, and n represents the number of sample images input into a training batch; For H k For each node in , query the feature neighbor of the node on the category, and add the category representation of the feature neighbor as a node to the neighbor graph; for an edge, if node j is the feature neighbor of node i, then there is an edge connecting the i-th node and the j-th node in the feature graph; The adjacent nodes obtained by query are used to form the k-1th layer nodes, and H k-1 It means that, according to the set search depth K, the operation of "querying the feature neighbors of the node in the category, and adding the category representation of the feature neighbors as nodes to the neighbor graph" is performed layer by layer, and the operation is repeated K times until the node H0 of the 0th layer is obtained, and the feature graph of the sample image in each category representation is obtained; For the label graph G l , the feature representation f of the sample image in category c input in each training batch c ={x 1c ,x 2c …,x ic ,…x nc } as the k-th layer node, using H k Indicates, and at the same time constitutes the original node in the label graph; where x ic ∈R d is the c-th class representation vector of the ith sample image in the training batch, and n represents the number of sample images input into a training batch; For H k For each node in , query the label neighbor of the node on the category, and add the category representation of the label neighbor as a node to the neighbor graph; for an edge, if node j is the label neighbor of node i, then there is an edge connecting the i-th node and the j-th node in the label graph; The adjacent nodes obtained by query are used to form the k-1th layer nodes, and H k-1 It means that according to the set search depth K, the operation of "querying the label neighbors of the node in the category, and adding the category representation of the label neighbors as nodes to the neighbor graph" is performed layer by layer, and the operation is repeated K times until the node H0 of the 0th layer is obtained, and the label graph of the sample image in each category representation is obtained.

5. The multi-label image classification model training method according to claim 4, characterized in that: The following formula is used to calculate the aggregated features of each category represented on the feature map and the aggregated features of each category represented on the label map: Among them, x v,u is the fusion feature of nodes v and u; P, U, V, b are learnable parameters; N(v) represents the neighbor set of node v obtained by uniform sampling; α v,u is the attention coefficient, Φ(·) is the attention function; σ is the nonlinear function; W k is the learnable transformation matrix, and is the node feature; is the attention-weighted node feature, Node feature Updated node features, d k is the feature dimension before the k-th layer node aggregation, d k+1 Aggregator is the dimension after node aggregation update; k represents an aggregator; represents the aggregation result; concat(·) is the concat function; max(·) is the maximum pooling operator.

6. The multi-label image classification model training method according to claim 1, characterized in that: The predicted response of the label map is calculated using the following formula: p L =[p l1 ,…,p lC ] T ; Among them, p lc represents the predicted response of the label map; p L represents the set of predicted responses for each category after label map learning; W c 、b c is a learnable parameter; D lc Represents label graph representation; The predicted response of the feature map is calculated using the following formula: p F =[p f1 ,…,p fC ] T ; Among them, p fc represents the predicted response of the feature map; p F Represents the predicted response set of each category after feature map learning; D fc Represents feature map representation.

7. The multi-label image classification model training method according to claim 1, characterized in that: The loss function of the multi-label image classification model is expressed as follows: in, represents the loss function, represents the feature map classification loss, represents the label map classification loss, represents the knowledge distillation loss, and λ represents the weighting coefficient.

8. A multi-label image classification model application method, characterized in that: The multi-label image classification model application method includes: Acquire a target image to be classified; the target image includes multiple labels, and each label corresponds to a category; Extracting features from the target image to obtain image features; According to the image features, calculating the feature neighbors of the target image based on the Euclidean distance on each category representation; Determine a feature map of the target image on each category representation according to the feature neighbors; According to the feature map, determining the aggregated features represented by each category on the feature map; The aggregated features represented by each category on the feature map are input into the classifier of the multi-label image classification model for classification, and the label distribution result corresponding to the target image is predicted; the multi-label image classification model is a model trained by the multi-label image classification model training method according to claim 1.

9. A multi-label image classification model training system, characterized in that: The multi-label image classification model training system comprises: A sample image acquisition module, used to acquire a sample image to be classified; the sample image includes a plurality of labels, and each label corresponds to a category; A sample image feature extraction module, used to extract features from the sample image to obtain image features; A feature neighbor and label neighbor calculation module, used to calculate the feature neighbor based on Euclidean distance and the label neighbor based on Hamming distance of the sample image in each category representation according to the image features; A feature map and label map determination module, used to determine the feature map and label map of the sample image on each category representation according to the feature neighbors and the label neighbors respectively; A feature map and label map aggregate feature determination module, used to determine, based on the feature map and the label map, the aggregate features of each category represented on the feature map and the aggregate features of each category represented on the label map; A classification loss determination module, used to input the aggregated features represented by each category on the feature map and the aggregated features represented by each category on the label map into a classifier of a multi-label image classification model for classification, and determine the classification loss; the multi-label image classification model is a model for classifying multi-label images using the aggregated features as input and the predicted label distribution result as output; The model training module is used to adopt the knowledge distillation technology to extract the predicted response of the label map from the classification loss as the supervision knowledge, which is used to guide the learning of the feature map, train and optimize the multi-label image classification model, and obtain a trained multi-label image classification model.

10. A multi-label image classification model application system, characterized in that: The multi-label image classification model application system includes: A target image acquisition module, used to acquire a target image to be classified; the target image includes a plurality of labels, and each label corresponds to a category; A target image feature extraction module is used to extract features from the target image to obtain image features; A feature neighbor calculation module, used for calculating the feature neighbor of the target image based on the Euclidean distance in each category representation according to the image features; A feature map determination module, used to determine a feature map of the target image on each category representation according to the feature neighbors; A feature graph aggregate feature determination module, used to determine, based on the feature graph, aggregate features represented by each category on the feature graph; A classification module is used to input the aggregated features represented by each category on the feature map into the classifier of a multi-label image classification model for classification, and predict the label distribution result corresponding to the target image; the multi-label image classification model is a model trained by the multi-label image classification model training method according to claim 1.

Images