Multi-label image classification method based on selective prototype loss function

Abstract

The invention discloses a multi-label image classification method based on a selective prototype loss function, and the method comprises the following steps: S1, obtaining an original image data set, carrying out the preprocessing of the original image data set to form a training set, and enabling all images in the original image data set to have M types and each image to have at least one label; s2, establishing a convolutional neural network model and performing training by adopting the training set, wherein the convolutional neural network model uses a selective prototype loss function as a penalty function; and S3, classifying the to-be-detected image by using the trained convolutional neural network model to obtain a prediction probability of a corresponding category, namely a multi-label image classification result. The method has the advantages of being more accurate in recognition and higher in generalization ability.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a multi-label image classification method based on a selective prototype loss function. Background Art

[0002] With the development of deep learning, the role of multi-label learning has been explored and realized. Multi-label learning refers to tasks where each image can simultaneously belong to multiple categories. Unlike traditional single-label classification, multi-label learning allows a data point to have more than one associated category label. It is valued for its ability to more realistically reflect the complexity of objects and events in the real world.

[0003] Multi-labeling also means that training objects require more labels. However, obtaining comprehensive and accurate label information typically requires specialized knowledge and significant human and material resources. As the number of labels increases, the difficulty of labeling also increases, further exacerbating the cost of data preparation. Therefore, existing technologies typically use partial labeling methods. Partial labeling methods involve only partially knowing the categories of certain training images, while the remaining true labels are unlabeled or unknown.

[0004] However, actual datasets often have an uneven distribution of categories, that is, some categories have more training images, while other categories have relatively fewer. This image imbalance will cause the model to tend to predict the majority class, but have poor recognition ability for the minority class. Furthermore, in many practical applications, only partial label information can be obtained. In this case of partial annotation, it is difficult for existing models to effectively use known labels to infer unknown labels, which may lead to insufficient learning of unlabeled features by the model. Finally, traditional loss functions often fail to fully consider the correlation between labels and the differences in the importance of different labels when dealing with multi-label learning tasks. These problems will affect their generalization ability and prediction accuracy. Summary of the Invention

[0005] The purpose of this invention is to address the above problems and propose a multi-label image classification method based on a selective prototype loss function, which has the characteristics of more accurate recognition and stronger generalization ability.

[0006] To achieve the above object, the technical solution adopted by the present invention is: The present invention proposes a multi-label image classification method based on a selective prototype loss function, comprising the following steps: S1. Obtain the original image dataset and preprocess it to form a training set. All images in the original image dataset have M categories and each image has at least one label; S2. Establish a convolutional neural network model and train it using the training set. The convolutional neural network model uses the selective prototype loss function as a penalty function. S3. Use the trained convolutional neural network model to classify the image to be tested, and obtain the predicted probability of the corresponding category, which is the multi-label image classification result.

[0007] Preferably, the selective prototype loss function is obtained as follows: S21, dynamically determine the number of prototypes for each category, then j Number of prototypes per category The formula is as follows: ; in, Indicates that it contains j The total number of images in each category, Indicates the maximum limit of the preset number of prototypes. ; S22. Use the KMeans algorithm to cluster all semantic features of each category and select the cluster center as the prototype to obtain the prototype set of all categories. , prototype set The formula is as follows: ; in, ; ; in, Indicates the j All semantic features of categories, Indicates the i Image No. j The semantic features of the categories, Indicates the j Category k A prototype, , , N Indicates the total number of images, Indicates that it contains j The number of images in each category, Represents the KMeans algorithm; S23, calculate the semantic features and prototype sets of all images corresponding to the categories The cosine similarity between and the probability confidence of each category; S24, semantic features and prototype sets corresponding to categories of all images The cosine similarity between and the probability confidence of each category is used to calculate the selective prototype loss function , the formula is as follows: ; in, ; in, Represents the discrimination criterion, Indicates the increase multiple. represents the cosine similarity of the i-th image and the j-th category, represents the probability confidence of the j-th category, s represents the preset hyperparameter, Represents the label of the jth category of the i-th image, the first conditional function If is equal to -1, then the value of the first conditional function is , otherwise 0; the second conditional function If is equal to 1, then the value of the second conditional function is , otherwise 0; the third conditional function If is equal to 0, then the value of the third conditional function is , otherwise 0; Represents the probability confidence threshold, and if the label does not exist, the probability If the probability confidence threshold is less than the probability confidence threshold, the corresponding image is considered to contain the label of the jth category of the i-th image Otherwise, it is considered that the corresponding image does not contain the label of the jth category of the i-th image .

[0008] Preferably, the semantic features and prototype sets of all images corresponding to the categories are calculated separately The cosine similarity between and the probability confidence of each category is as follows: ; ; in, represents the predicted probability of the jth category of the i-th image output by the convolutional neural network model, Indicates the module length.

[0009] Preferably, the convolutional neural network model includes a backbone network, a semantic parsing module, a category relationship building module, and a category probability prediction module connected in sequence, wherein: The semantic parsing module performs the following operations: The initial feature map output by the backbone network is transformed into the spatial dimension using the torch.transpose function, and the three channels of the initial feature map after spatial dimension transformation are flattened into one dimension respectively. The three channels are the batch size, length and width. A pre-training model is established and pre-trained using the original image dataset. The pre-trained weights are saved as pre-training features, and the pre-training features are aligned with the flattened initial feature map in the spatial dimension. The pre-training model is a ResNet50 network model. Multiply the flattened initial feature map and the pre-trained feature after feature alignment element by element to form a fused feature; The fusion features are sequentially passed through the tanh activation function and the fully connected layer to obtain the weight coefficient; Perform weighted summation of the fusion features and weight coefficients to output all semantic features of each category; The category relationship building module performs the following operations: Treat each category as a node, and initialize all semantic features of each category output by the semantic parsing module as the state of the corresponding node; Count the original image data set a Category and b The frequency of common occurrence of categories is used to initialize the in-degree matrix and out-degree matrix. The in-degree matrix and out-degree matrix are M × M The two-dimensional adjacency matrix of a OK b The elements of the column represent the a Category and b The relevance of the categories, ; Carry out the t Iterate to update the node state, specifically: add the in-degree matrix to the t -1 iteration, the states of all nodes are multiplied element by element to obtain the in-degree association matrix. t -1 iterations, the states of all nodes are multiplied element by element to obtain the out-degree association matrix, and the in-degree association matrix and the out-degree association matrix are concatenated into neighborhood information; then the t The states of all nodes in the -1 iteration are added element by element with the neighborhood information, and then the first sigmoid function outputs the updated gate value and the second sigmoid function outputs the reset gate value. The updated gate value is used to represent the retained t -1 The proportion of the state of the node in the iteration, the reset gate value is used to represent the t -1 iteration of the state of all nodes in the generation t The participation of all nodes in the state of the first iteration; t The state of all nodes in the -1 iteration is multiplied element by element with the reset gate value and then fused with the neighborhood information through the tanh function to generate a temporary feature. t-1 iteration of the state of all nodes: temporary feature = R: (1-R) ​​element by element addition to complete feature fusion, as the first t The state of all nodes in the iteration, R is the update gate value; Place t = t +1, loop iterates until the number of iterations is completed, and the final state of all nodes is obtained as the gated graph feature, where , Indicates the number of iterations; The category probability prediction module performs the following operations: All semantic features and gated graph features of each category output by the semantic parsing module are aligned and concatenated in three channel dimensions, and then the prediction probability of the corresponding category is output through the tanh activation function.

[0010] Preferably, feature alignment uses the view function and concatenation uses the torch.cat function.

[0011] Preferably, the backbone network is a ResNet101 network model.

[0012] Preferably, the pre-processing is as follows: S11, mask some labels in the original image dataset; S12, performing data enhancement processing on all images in the original image dataset; S13. The images after data augmentation processing and the labels in the masked original image dataset are combined into a training set.

[0013] Preferably, some labels in the original image dataset are masked as follows: Establish the label matrix of the original image dataset. The size of the label matrix of the original image dataset is N OK M Column, and n OK m The column value is -1 or 1, where -1 means n images do not contain m Category; 1 means n The image contains m categories, n =1~ N , m =1~ M , N Indicates the total number of images; Generate an initial matrix of the same size as the label matrix of the original image dataset and calculate the value of α Convert the initial matrix into a mask matrix, that is, if the first n OKm The element of the column is smaller than the preset value α , then the position is reset to 1, otherwise the position is reset to 0. Each element in the initial matrix is ​​a random floating point number in [0.0, 1.0). α =[0.1, 0.2, 0.3, ..., 0.9]; The label matrix of the original image dataset is multiplied element by element by the mask matrix to obtain the mask matrix, and the labels with elements 1 in the mask matrix are regarded as the labels in the masked original image dataset.

[0014] Preferably, each element in the initial matrix is ​​assigned a value using the numpy.random.random function.

[0015] Preferably, the data enhancement process includes cropping, random horizontal flipping and normalization in sequence.

[0016] Compared with the prior art, the present invention has the following beneficial effects: In order to address the limitations of existing technologies in insufficient learning when processing partially labeled data and to cope with the degradation of classification performance caused by image imbalance, the present invention proposes a multi-label image classification method based on a selective prototype loss function. By constructing an image imbalance dataset (training set), the imbalanced characteristics of the image are distinguished and utilized. The known and unknown labels of the image are then trained using a convolutional neural network model. The convolutional neural network model adopts an uncertainty-aware selective prototype loss function. Specifically, by enhancing the prototype representation of known labels and adaptively mining potential positive images, the problem of missing supervision signals for some labels is alleviated. At the same time, the statistical prior of the training set is used to generate a probabilistic confidence for each category. Finally, the loss weight is increased for unknown labels that may be positive in unlabeled images, while images that may be negative are ignored to optimize the learning performance of the model, ensuring that the model can more accurately process uncertain images. Finally, the precise representation of known labels and potential positive images are effectively mined based on the trained convolutional neural network model, aiming to significantly improve the performance of image classification and have the characteristics of more accurate recognition and stronger generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart of the multi-label image classification method based on the selective prototype loss function of the present invention; Figure 2 Flowchart of the convolutional neural network model of the present invention; Figure 3 This is a flowchart of the semantic parsing module of the present invention; Figure 4 A flow chart of a category relationship building module for the present invention; Figure 5 Flowchart of the category probability prediction module of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that when a component is referred to as being "connected" to another component, it may be directly connected to the other component or there may be an intermediate component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of this application. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0020] like Figures 1-4 As shown in FIG, a multi-label image classification method based on a selective prototype loss function includes the following steps: S1. Obtain the original image dataset and preprocess it to form a training set. All images in the original image dataset have M categories and each image has at least one label.

[0021] In one embodiment, the preprocessing is as follows: S11, mask some labels in the original image dataset; S12, performing data enhancement processing on all images in the original image dataset; S13. The images after data augmentation processing and the labels in the masked original image dataset are combined into a training set.

[0022] In one embodiment, some labels in the original image dataset are masked as follows: Establish the label matrix of the original image dataset. The size of the label matrix of the original image dataset is N OK M Column, and n OK m The column value is -1 or 1, where -1 means n images do not contain m Category; 1 means n The image contains m categories, n =1~ N , m =1~ M , N Indicates the total number of images; Generate an initial matrix of the same size as the label matrix of the original image dataset and calculate the value of α Convert the initial matrix into a mask matrix, that is, if the first n OK m The element of the column is smaller than the preset value α , then the position is reset to 1, otherwise the position is reset to 0. Each element in the initial matrix is ​​a random floating point number in [0.0, 1.0). α =[0.1, 0.2, 0.3, ..., 0.9]; The label matrix of the original image dataset is multiplied element by element by the mask matrix to obtain the mask matrix, and the labels with elements 1 in the mask matrix are regarded as the labels in the masked original image dataset.

[0023] In one embodiment, each element in the initial matrix is ​​assigned a value using the numpy.random.random function.

[0024] In one embodiment, the data augmentation process includes cropping, random horizontal flipping, and normalization in sequence.

[0025] Among them, based on the preset value α Mask out some labels as the starting labels of the training set. Use the random label unknown setting to simulate the partial labeling of the training set.

[0026] Specifically, the preset value α The range is 10%-90%, with an increase of 10%. In this embodiment, each preset value is tested once, that is, a single data set is tested 9 times in total. When masking some labels in the original image data set, an initial matrix with the same size as the label matrix of the original image data set is first generated, and the initial matrix is ​​masked according to the preset value. α Convert the initial matrix into a mask matrix, that is, if the first n OK m The element of the column is smaller than the preset value α , then the position is reset to 1, otherwise the position is reset to 0. Each element in the initial matrix is ​​a random floating point number [0.0, 1.0) assigned by the numpy.random.random function. α =[0.1, 0.2, 0.3, …, 0.9]. Specifically, if the label matrix of the original image dataset is multiplied by 1 in the mask matrix, the original label information is retained; if it is multiplied by 0 in the mask matrix, the original label information is masked, because no matter whether the original label is 1 or -1, its information is set to 0. By inputting a fixed ratio of 10% to 90% with an increase of 10% (the preset value α) Randomly determine that the label is unknown to simulate the partial labeling situation, and randomly perform data augmentation on the image. Then, the data augmented image and the simulated label data are combined to construct multiple training sets with different images. The constructed multiple training sets are shown below: Training set A: Covering 20 categories, with approximately 10,000 images, 5,011 images are used as training images, and the remaining 4,952 images are used as test images. Data augmentation is performed on all images in the original image dataset. Assuming that the images are resized to 512×512, a value is randomly selected from the set {512, 448, 384, 320, 256} as the width and height of the cropped images. The images are then resized to 448×448, followed by random horizontal flipping and image normalization. For labels, a mask matrix Mask20 is generated with the same size as the label matrix of the original image dataset. This is applied to the mask matrix by masking some of the labels. The data augmented images and the masked labels (labels with 1 elements in the mask matrix) are combined to form training set A.

[0027] Training set B: Covering 80 categories, it contains 120k images, split into 80k for training and 40k for testing. Data augmentation is performed on all images in the original image dataset. Assuming the images are resized to 512×512, a value is randomly selected from the set {512, 448, 384, 320, 256} to be used as the width and height for cropping the images. The images are then resized to 448×448, followed by random horizontal flipping and image normalization. For labels, a mask matrix Mask80 is generated that has the same size as the label matrix of the original image dataset. This is applied to the mask matrix to mask some of the labels. The data augmented images and masked labels (labels with 1 elements in the mask matrix) are combined to form training set B.

[0028] Training set C: Covering 200 categories, 10,000 images are selected for training and 98,249 images for testing. Data augmentation is performed on all images in the original image dataset. Assuming that the images are resized to 512×512, a value is randomly selected from the set {512, 448, 384, 320, 256} as the width and height of the cropped images. The images are then resized to 448×448, followed by random horizontal flipping and image normalization. For labels, a mask matrix Mask200 is generated with the same size as the label matrix of the original image dataset. This is applied to the mask matrix to mask some of the labels in the original image dataset. The data augmented images and the masked labels (labels with 1 in the mask matrix) are combined to form training set C.

[0029] As mentioned above, Dataset A is widely used in general object detection and classification tasks and is suitable for common object recognition scenarios, typically including categories such as people, cars, and bicycles. Its label design focuses on basic object category coverage and is suitable for verifying the model's robustness to common objects. Dataset B, for example, covers everyday objects, human body parts (such as "hands" and "feet"), and small targets in complex scenes. Its label design emphasizes fine-grained differentiation capabilities. Dataset C, for example, uses a high label density (each image is annotated with an average of more than 30 objects), covering not only specific objects but also their corresponding attributes, such as ("cup" - "white"). Its label types and application scenarios reflect the high complexity and fine-grained requirements of multi-label classification tasks.

[0030] S2. Establish a convolutional neural network model and use the training set for training. The convolutional neural network model uses the selective prototype loss function as the penalty function.

[0031] In one embodiment, the selective prototype loss function is obtained as follows: S21, dynamically determine the number of prototypes for each category, then j Number of prototypes per category The formula is as follows: ; in, Indicates that it contains j The total number of images in each category, Indicates the maximum limit of the preset number of prototypes. ; S22. Use the KMeans algorithm to cluster all semantic features of each category and select the cluster center as the prototype to obtain the prototype set of all categories. , prototype set The formula is as follows: ; in, ; ; in, Indicates the j All semantic features of categories, Indicates the i Image No. j The semantic features of the categories, Indicates the j Category k A prototype, , , N Indicates the total number of images, Indicates that it contains j The number of images in each category, Represents the KMeans algorithm; S23, calculate the semantic features and prototype sets of all images corresponding to the categories The cosine similarity between and the probability confidence of each category; S24, semantic features and prototype sets corresponding to categories of all images The cosine similarity between and the probability confidence of each category is used to calculate the selective prototype loss function , the formula is as follows: ; in, ; in, Represents the discrimination criterion, Indicates the increase multiple. represents the cosine similarity of the i-th image and the j-th category, represents the probability confidence of the j-th category, s represents the preset hyperparameter, Represents the label of the jth category of the i-th image, the first conditional function If is equal to -1, then the value of the first conditional function is , otherwise 0; the second conditional function If is equal to 1, then the value of the second conditional function is , otherwise 0; the third conditional function If is equal to 0, then the value of the third conditional function is , otherwise 0; Represents the probability confidence threshold, and if the label does not exist, the probability If the probability confidence threshold is less than the probability confidence threshold, the corresponding image is considered to contain the label of the jth category of the i-th image Otherwise, it is considered that the corresponding image does not contain the label of the jth category of the i-th image .

[0032] Preferably, the semantic features and prototype sets of all images corresponding to the categories are calculated separately The cosine similarity between and the probability confidence of each category is as follows: ; ; in, represents the predicted probability of the jth category of the i-th image output by the convolutional neural network model, Indicates the module length.

[0033] The number of prototypes for each category is dynamically determined based on the number of images. The KMeans algorithm is then applied to cluster the semantic features of the images and the cluster center is selected as the prototype. Contrastive learning is used to calculate the cosine similarity between the semantic features of the category and the prototype set of the category. After the category probability prediction module obtains the predicted probability, it performs average statistics to obtain the predicted probability of each category. The probability confidence of each category is calculated. Similarity with cosine Put them into the discrimination criteria for comparison, select the true label of the unknown label according to the result, and make corresponding loss measures. The specific design is: Indicates the similarity between the semantic features of the image and the prototype, with a value range of [-1, 1]. The closer to 1, the more similar it is. The predicted probability cannot be a negative number, and directly comparing it with the similarity will produce a lot of errors, so the similarity needs to be converted into a non-negative number, that is, ;Will The result is regarded as a measure of the possibility that the label does not exist, and its value range is [0, 2]. The higher, The smaller it is, the less likely the label is to not exist; Represents the probability confidence threshold obtained during model training. If the probability measure of the label's true non-existence is less than the probability confidence threshold, the image is considered to contain the label and its loss is amplified. Otherwise, the image is considered not to contain the label and the loss is suppressed.

[0034] In one embodiment, the convolutional neural network model includes a backbone network, a semantic parsing module, a category relationship building module, and a category probability prediction module connected in sequence, wherein: The semantic parsing module performs the following operations: The initial feature map output by the backbone network is transformed into the spatial dimension using the torch.transpose function, and the three channels of the initial feature map after spatial dimension transformation are flattened into one dimension respectively. The three channels are the batch size, length and width. A pre-training model is established and pre-trained using the original image dataset. The pre-trained weights are saved as pre-training features, and the pre-training features are aligned with the flattened initial feature map in the spatial dimension. The pre-training model is a ResNet50 network model. Multiply the flattened initial feature map and the pre-trained feature after feature alignment element by element to form a fused feature; The fusion features are sequentially passed through the tanh activation function and the fully connected layer to obtain the weight coefficient; Perform weighted summation of the fusion features and weight coefficients to output all semantic features of each category; The category relationship building module performs the following operations: Treat each category as a node, and initialize all semantic features of each category output by the semantic parsing module as the state of the corresponding node; Count the original image data set a Category and b The frequency of common occurrence of categories is used to initialize the in-degree matrix and out-degree matrix. The in-degree matrix and out-degree matrix are M × M The two-dimensional adjacency matrix of a OK b The elements of the column represent the a Category and b The relevance of the categories, ; Carry out the t Iterate to update the node state, specifically: add the in-degree matrix to the t -1 iteration, the states of all nodes are multiplied element by element to obtain the in-degree association matrix. t -1 iterations, the states of all nodes are multiplied element by element to obtain the out-degree association matrix, and the in-degree association matrix and the out-degree association matrix are concatenated into neighborhood information; then the t The states of all nodes in the -1 iteration are added element by element with the neighborhood information, and then the first sigmoid function outputs the updated gate value and the second sigmoid function outputs the reset gate value. The updated gate value is used to represent the retained t -1 The proportion of the state of the node in the iteration, the reset gate value is used to represent the t -1 iteration of the state of all nodes in the generation tThe participation of all nodes in the state of the first iteration; t The state of all nodes in the -1 iteration is multiplied element by element with the reset gate value and then fused with the neighborhood information through the tanh function to generate a temporary feature. t -1 iteration of the state of all nodes: temporary feature = R: (1-R) ​​element by element addition to complete feature fusion, as the first t The state of all nodes in the iteration, R is the update gate value; Place t = t +1, loop iterates until the number of iterations is completed, and the final state of all nodes is obtained as the gated graph feature, where , Indicates the number of iterations; The category probability prediction module performs the following operations: All semantic features and gated graph features of each category output by the semantic parsing module are aligned and concatenated in three channel dimensions, and then the prediction probability of the corresponding category is output through the tanh activation function.

[0035] In one embodiment, feature alignment uses the view function, and concatenation uses the torch.cat function.

[0036] In one embodiment, the backbone network is a ResNet101 network model.

[0037] Specifically, this convolutional neural network model uses the ResNet101 network model as the backbone network, followed by three modules (semantic parsing module, category relationship building module, and category probability prediction module). The backbone network input is a 448×448, three-channel color RGB image. The BatchSize is a preset fixed value, set to 32 in experimental training. After passing through the backbone network, the output is an initial feature map with a size of (BatchSize, 3, 448, 448).

[0038] In the semantic parsing module, the initial feature map is converted to a single dimension using the torch.transpose function and flattened to a single dimension across the three channels. A ResNet50 network model is pre-trained using the original image dataset. The pre-trained weights are saved as pre-trained features. The pre-trained features are spatially aligned using the view function and then interacted with the initial feature map. Each element at the same position is multiplied to generate a fused feature. The fused feature is activated using the tanh function to generate weights, which are then normalized using a fully connected layer to obtain weight coefficients. The fused feature is then weighted and summed with the weight coefficients to output the complete semantic features for each category.

[0039] In the category relationship building module, each category is regarded as a node, and all the semantic features of each category output by the semantic parsing module are initialized to the state of the corresponding node, that is, the state of the node is the semantic feature; the number of the first a Category and b The frequencies of common occurrence of categories are used as element values ​​to initialize the in-degree matrix and out-degree matrix. Both the in-degree matrix and the out-degree matrix are M × M The two-dimensional adjacency matrix of , all elements are initialized to the frequency value of the corresponding statistics. a OK b Column representation a Category and b The strength of the association between categories, a OK b The value of the column represents the a Category and b The degree of correlation between the categories is the node value. Then K Iterations (such as K =5), calculate the adjacency information through the in-degree matrix and out-degree matrix, and then t The states of all nodes in the -1 iteration are added element by element with the neighborhood information, and then the first sigmoid function outputs the updated gate value and the second sigmoid function outputs the reset gate value. The updated gate value is used to represent the retained t -1 iteration of the state of the node ratio (for example, 0.7 means retaining 70% of the first t -1 iteration), reset the gate value to represent the state of the node t -1 iteration of the state of all nodes in the generation t The participation of all nodes in the state of the iteration (for example, 0.3 means generating the t The states of all nodes in the iteration are filtered out by 30% of the t -1 iteration of all nodes participate); t The state of all nodes in the -1 iteration is multiplied element by element with the reset gate value and then fused with the neighborhood information through the tanh function to generate a temporary feature. t -1 iteration of the state of all nodes: temporary feature = R: (1-R) ​​element by element addition to complete feature fusion (such as the update gate value is 0.7, indicating that 70% of the t -1 iteration node state + 30% temporary features are added element by element to complete feature fusion), as the first t The states of all nodes in the iteration are updated to adjust the corresponding semantic features, so that the more related the categories, the more similar the semantic feature representations will be, and finally the gated graph features are obtained.

[0040] In the category probability prediction module, the model recognizes and learns which information needs to be retained and which useless information should be discarded, and uses the tanh activation function to calculate the predicted probability of each category. The backbone network, semantic parsing module, category relationship construction module and category probability prediction module are connected in sequence, and the obtained semantic features and predicted probabilities are utilized and put into the discrimination criteria to make countermeasures to obtain the selective prototype loss function value. , then the Adam optimization algorithm is used to iteratively update the model parameters to minimize the loss function to obtain the trained convolutional neural network model.

[0041] S3. Use the trained convolutional neural network model to classify the image to be tested, and obtain the predicted probability of the corresponding category, which is the multi-label image classification result.

[0042] The following is demonstrated by specific experimental data.

[0043] Tables 1 to 3 are comparison tables of classification result evaluation indicators on different datasets (the aforementioned datasets A, B, and C). The tables show the comparative experimental results of the HST model (HST for short, from the reference: Chen T, Pu T, Liu L, et al. Heterogeneous semantic transfer for multi-label recognition with partiallabels[J]. International Journal of Computer Vision, 2024, 132(12): 6091-6106.). There are three indicators, including: 1) Average precision: usually used in ranking tasks to measure the accuracy of correct identification of any category. It considers the relationship between recall and precision. Recall refers to the ratio of the number of positive images predicted by the model to the number of all true positive images. Precision refers to the ratio of the actual correct positive images in the model's prediction results. The higher the result value, the better the recognition performance of the model. 2) Macro F1 Score: This involves calculating the F1 score for each class individually and then averaging it. The F1 score here is the harmonic mean of precision and recall. 3) Micro F1 Score: This involves globally counting the true positives, false positives, and false negatives for all classes. The overall precision and recall are then calculated based on these totals to create the macro average.

[0044] Table 1 Comparison of evaluation indicators of classification results on dataset A

[0045] Table 2 Comparison of evaluation indicators of classification results on dataset B

[0046] Table 3 Comparison of evaluation indicators of classification results on dataset C

[0047] In terms of the Average Precision metric on Dataset A, this method achieved a 2.7% improvement over the HST model at α = 10%, and maintained a stable 1% gain at α = 90%, demonstrating its robustness to varying annotation densities. Furthermore, improvements in both the macro-average F1 score and the micro-average F1 score are primarily reflected in the improved predictive power of long-tail categories (for example, the micro-average F1 score on Dataset C improved by 5.9% at α = 10%), directly reflecting the model's effectiveness in alleviating class imbalance. By combining the collaborative optimization mechanism of dynamic prototype number generation and probability confidence thresholds in the experiment, this method achieves more efficient utilization of partially annotated data by enhancing representation learning of known labels and mining potential positive examples.

[0048] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] The above-described embodiments merely represent specific and detailed examples of the present application and should not be construed as limiting the scope of the present application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A multi-label image classification method based on a selective prototype loss function, characterized by: The multi-label image classification method based on the selective prototype loss function comprises the following steps: S1, obtain the original image data set and preprocess it to form a training set, all images in the original image data set have M categories and each image has at least one label; S2. Establishing a convolutional neural network model and training it using the training set, wherein the convolutional neural network model uses a selective prototype loss function as a penalty function; S3. Use the trained convolutional neural network model to classify the image to be tested, and obtain the predicted probability of the corresponding category, which is the multi-label image classification result.

2. The multi-label image classification method based on the selective prototype loss function according to claim 1, wherein: The selective prototype loss function is obtained as follows: S21, dynamically determine the number of prototypes for each category, then j Number of prototypes per category The formula is as follows: ； in, Indicates that it contains j The total number of images in each category, Indicates the maximum limit of the preset number of prototypes. ; S22. Use the KMeans algorithm to cluster all semantic features of each category and select the cluster center as the prototype to obtain the prototype set of all categories. , the prototype set The formula is as follows: ； in, ； ； in, Indicates the j All semantic features of categories, Indicates the i Image No. j The semantic features of the categories, Indicates the j Category k A prototype, , , N Indicates the total number of images, Indicates that it contains j The number of images in each category, Represents the KMeans algorithm; S23, calculate the semantic features and prototype sets of all images corresponding to the categories The cosine similarity between and the probability confidence of each category; S24, semantic features and prototype sets corresponding to categories of all images The cosine similarity between and the probability confidence of each category is used to calculate the selective prototype loss function , the formula is as follows: ； in, ； in, Represents the discrimination criterion, Indicates the increase multiple. represents the cosine similarity of the i-th image and the j-th category, represents the probability confidence of the j-th category, s represents the preset hyperparameter, Represents the label of the jth category of the i-th image, the first conditional function If is equal to -1, then the value of the first conditional function is , otherwise 0; the second conditional function If is equal to 1, then the value of the second conditional function is , otherwise 0; the third conditional function If is equal to 0, then the value of the third conditional function is , otherwise 0; Represents the probability confidence threshold, and if the label does not exist, the probability If the probability confidence threshold is less than the probability confidence threshold, the corresponding image is considered to contain the label of the jth category of the i-th image Otherwise, it is considered that the corresponding image does not contain the label of the jth category of the i-th image .

3. The multi-label image classification method based on the selective prototype loss function according to claim 2, wherein: The semantic features and prototype sets of the corresponding categories of all images are calculated separately The cosine similarity between and the probability confidence of each category is as follows: ； ； in, represents the predicted probability of the jth category of the i-th image output by the convolutional neural network model, Indicates the module length.

4. The multi-label image classification method based on the selective prototype loss function according to claim 1, wherein: The convolutional neural network model includes a backbone network, a semantic parsing module, a category relationship building module, and a category probability prediction module connected in sequence, wherein: The semantic parsing module performs the following operations: The initial feature map output by the backbone network is converted into a feature dimension. The feature dimension is converted into the spatial dimension of the initial feature map using the torch.transpose function, and the three channels of the initial feature map after the spatial dimension conversion are flattened into one dimension respectively. The three channels are batch size, length and width; Establish a pre-training model and use the original image dataset for pre-training, save the pre-trained weights as pre-training features, and align the pre-training features with the flattened initial feature map in the spatial dimension. The pre-training model is a ResNet50 network model; Multiply the flattened initial feature map and the pre-trained feature after feature alignment element by element to form a fused feature; The fusion features are sequentially passed through the tanh activation function and the fully connected layer to obtain the weight coefficient; Perform weighted summation of the fusion features and weight coefficients to output all semantic features of each category; The category relationship building module performs the following operations: Treat each category as a node, and initialize all semantic features of each category output by the semantic parsing module as the state of the corresponding node; Count the original image data set a Category and b The frequencies of common occurrence of categories are used to initialize the in-degree matrix and out-degree matrix, both of which are M × M The two-dimensional adjacency matrix of a OK b The elements of the column represent the a Category and b The relevance of the categories, ; Carry out the t Iterate to update the node state, specifically: add the in-degree matrix to the t -1 iteration, the states of all nodes are multiplied element by element to obtain the in-degree association matrix. t -1 iterations, the states of all nodes are multiplied element by element to obtain the out-degree association matrix, and the in-degree association matrix and the out-degree association matrix are concatenated into neighborhood information; then the t The states of all nodes in the -1 iteration are added element by element with the neighborhood information, and then the first sigmoid function is used to output the updated gate value and the second sigmoid function is used to output the reset gate value. The updated gate value is used to represent the retained t -1 iteration of the state of the node, the reset gate value is used to represent the t -1 iteration of the state of all nodes in the generation t The participation of all nodes in the state of the first iteration; t The state of all nodes in the -1 iteration is multiplied element by element with the reset gate value and then fused with the neighborhood information through the tanh function to generate a temporary feature. t -1 iteration of the state of all nodes: temporary feature = R: (1-R) ​​element by element addition to complete feature fusion, as the first t The state of all nodes in the iteration, R is the update gate value; Place t = t +1, loop iterates until the number of iterations is completed, and the final state of all nodes is obtained as the gated graph feature, where , Indicates the number of iterations; The category probability prediction module performs the following operations: All semantic features and gated graph features of each category output by the semantic parsing module are aligned and concatenated in three channel dimensions, and then the prediction probability of the corresponding category is output through the tanh activation function.

5. The multi-label image classification method based on the selective prototype loss function according to claim 4, wherein: The feature alignment uses the view function, and the splicing uses the torch.cat function.

6. The multi-label image classification method based on the selective prototype loss function according to claim 4, wherein: The backbone network is the ResNet101 network model.

7. The multi-label image classification method based on the selective prototype loss function according to claim 1, wherein: The preprocessing is as follows: S11, mask some labels in the original image dataset; S12, performing data enhancement processing on all images in the original image dataset; S13. The images after data augmentation processing and the labels in the masked original image dataset are combined into a training set.

8. The multi-label image classification method based on the selective prototype loss function according to claim 7, wherein: The method of masking some labels in the original image dataset is as follows: Establish a label matrix of the original image dataset, the size of the label matrix of the original image dataset is N OK M Column, and n OK m The column value is -1 or 1, where -1 means n images do not contain m Category; 1 means n The image contains m categories, n =1~ N , m =1~ M , N Indicates the total number of images; Generate an initial matrix of the same size as the label matrix of the original image dataset and calculate the value of α Convert the initial matrix into a mask matrix, that is, if the first n OK m The element of the column is smaller than the preset value α , then the position is reset to 1, otherwise the position is reset to 0, and each element in the initial matrix is ​​a random floating point number [0.0, 1.0). α =[0.1, 0.2, 0.3, ..., 0.9]; The label matrix of the original image dataset is multiplied element by element by the mask matrix to obtain the mask matrix, and the labels with elements 1 in the mask matrix are regarded as the labels in the masked original image dataset.

9. The multi-label image classification method based on the selective prototype loss function according to claim 8, wherein: Each element in the initial matrix is ​​assigned a value using the numpy.random.random function.

10. The multi-label image classification method based on the selective prototype loss function according to claim 7, wherein: The data enhancement process includes cropping, random horizontal flipping and normalization in sequence.

Images