A multi-label image classification method based on a selective prototype loss function

Abstract

This invention discloses a multi-label image classification method based on a selective prototype loss function, comprising the following steps: S1, obtaining the original image dataset and preprocessing it to form a training set, wherein all images in the original image dataset have M The method involves: S2, establishing a convolutional neural network (CNN) model and training it using a training set, with the CNN model employing a selective prototype loss function as a penalty function; and S3, classifying the test images using the trained CNN model to obtain the predicted probabilities of the corresponding categories, which are the multi-label image classification results. This method is characterized by more accurate recognition and stronger generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a multi-label image classification method based on a selective prototype loss function. Background Technology

[0002] With the development of deep learning, the role of multi-label learning has been explored and realized. Multi-label learning refers to the task of allowing each image to belong to multiple categories simultaneously. Unlike traditional single-label classification, multi-label learning allows a data point to have more than one associated category label, and it is valued for its ability to more realistically reflect the complexity of objects and events in the real world.

[0003] Multi-labeling also indicates that training objects require more labels, and obtaining comprehensive and accurate label information usually requires professional knowledge and a significant investment of human and material resources. As the number of labels increases, the labeling difficulty also increases, which further exacerbates the cost problem in the data preparation stage. Therefore, existing technologies usually adopt partial labeling methods, which means that for some training images, it is only known which categories they belong to, while the remaining true labels are unlabeled or unknown.

[0004] However, real-world datasets often exhibit uneven class distribution, meaning some classes have a larger number of training images while others have relatively fewer. This image imbalance causes models to tend to predict the majority class, resulting in poor recognition of the minority class. Furthermore, in many practical applications, only partial label information is available. In such partially labeled cases, existing models struggle to effectively use known labels to infer unknown labels, potentially leading to insufficient learning of unlabeled features. Finally, traditional loss functions, when handling multi-label learning tasks, often fail to adequately consider the correlation between labels and the differences in importance between different labels. These issues all affect their generalization ability and prediction accuracy. Summary of the Invention

[0005] The purpose of this invention is to address the above-mentioned problems by proposing a multi-label image classification method based on a selective prototype loss function, which has the characteristics of more accurate recognition and stronger generalization ability.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention proposes a multi-label image classification method based on a selective prototype loss function, comprising the following steps:

[0008] S1. Obtain the original image dataset and preprocess it to form a training set. All images in the original image dataset have... M Each image has at least one label and is categorized into several categories.

[0009] S2. Establish a convolutional neural network model and train it using a training set. The convolutional neural network model uses a selective prototype loss function as a penalty function.

[0010] S3. Use the trained convolutional neural network model to classify the image to be tested, and obtain the predicted probability of the corresponding category, which is the multi-label image classification result.

[0011] Preferably, the selective prototype loss function is obtained as follows:

[0012] S21. Dynamically determine the number of prototypes for each category, then the number of prototypes for the first category... j Number of prototypes in each category The formula is as follows:

[0013] ;

[0014] in, Indicates containing the first j The total number of images in each category This indicates the maximum limit for the preset number of prototypes. ;

[0015] S22. Using the KMeans algorithm, cluster all semantic features of each category and select the cluster centers as prototypes to obtain the prototype set of all categories. Prototype set The formula is as follows:

[0016] ;

[0017] in,

[0018] ;

[0019] ;

[0020] in, Indicates the first j All semantic features of each category Indicates the first i The first image j Semantic features of each category Indicates the first j The first category k A prototype, , , N This indicates the total number of images. Indicates containing the first j Number of images in each category This represents the KMeans algorithm;

[0021] S23. Calculate the semantic features and prototype sets of all images corresponding to their respective categories. The cosine similarity between them and the probability confidence of each category;

[0022] S24. Based on the semantic features and prototype set of all images corresponding to their respective categories. The selective prototype loss function is calculated using the cosine similarity between the groups and the probability confidence of each category. The formula is as follows:

[0023] ;

[0024] in,

[0025] ;

[0026] in, Indicate the discrimination criteria, Indicates the multiplier. Let represent the cosine similarity of the i-th image to the j-th category. Let represent the probability confidence level of the j-th category, and s represent the preset hyperparameters. The first conditional function represents the label of the i-th image in the j-th category. Indicates if If the value is -1, then the value of the first conditional function is... Otherwise, it is 0; second condition function Indicates if If the value equals 1, then the value of the second conditional function is 1. Otherwise, it is 0; third condition function Indicates if If the value is 0, then the value of the third conditional function is... Otherwise, it is 0; This represents the probability confidence threshold, and the probability that the label truly does not exist. If the probability confidence threshold is less than 1, then the corresponding image is considered to contain the label of the j-th category of the i-th image. Otherwise, it is considered that the corresponding image does not contain the label of the j-th category of the i-th image. .

[0027] Preferably, the semantic features and prototype sets of all images corresponding to their respective categories are calculated respectively. The cosine similarity between the pairs and the probability confidence of each category are calculated using the following formula:

[0028] ;

[0029] ;

[0030] in, This represents the predicted probability of the i-th image belonging to the j-th class, as output by the convolutional neural network model. Indicates the modulus length.

[0031] Preferably, the convolutional neural network model includes a backbone network, a semantic parsing module, a category relationship construction module, and a category probability prediction module connected in sequence, wherein:

[0032] The semantic parsing module performs the following operations:

[0033] The initial feature map output by the backbone network is transformed in terms of feature dimension. The feature dimension is converted into the spatial dimension of the initial feature map using the torch.transpose function. The three channels of the initial feature map after spatial dimension transformation are flattened into one dimension, namely batch size, length and width.

[0034] A pre-trained model was established and pre-trained using the original image dataset. The pre-trained weights were saved as pre-trained features, and the pre-trained features were aligned with the flattened initial feature map in the spatial dimension. The pre-trained model was a ResNet50 network model.

[0035] The flattened initial feature map is multiplied element-wise with the pre-trained features after feature alignment to form a fused feature;

[0036] The fused features are passed sequentially through a tanh activation function and a fully connected layer to obtain the weight coefficients.

[0037] The fused features and weight coefficients are weighted and summed to output all semantic features for each category;

[0038] The category relationship construction module performs the following operations:

[0039] Each category is treated as a node, and all semantic features of each category output by the semantic parsing module are initialized to the state of the corresponding node.

[0040] Statistical analysis of the first image in the original image dataset a The category and the first b The frequencies of common occurrence of each category are used to initialize the in-degree and out-degree matrices, both of which are... M × M A two-dimensional adjacency matrix, and located at the th a OK b The element of the column represents the first... a The category and the first b The degree of relevance of each category ;

[0041] Conduct the first t The iteration updates the state of the nodes, specifically by: interpolating the in-degree matrix with the first iteration.t The in-degree correlation matrix is ​​obtained by element-wise multiplying the states of all nodes in the -1 iteration, and the out-degree matrix is ​​then compared with the state of the -1th iteration. t The out-degree correlation matrix is ​​obtained by element-wise multiplication of the states of all nodes in the -1st iteration, and the in-degree correlation matrix and the out-degree correlation matrix are concatenated to form neighborhood information; then the -1st iteration... t After summing the states and neighborhood information of all nodes in the -1 iteration element by element, the values ​​are then processed by the first sigmoid function to output an update gate and by the second sigmoid function to output a reset gate. The update gate is used to represent the value retained in the -1 iteration. t The proportion of the node's state in the -1st iteration, with the reset threshold used to represent the proportion of the node's state in the -1st iteration. t The states of all nodes in the -1st iteration are generated in the... t The participation of all nodes in the state of the next iteration; the participation of the first... t The states of all nodes in the -1st iteration are multiplied element-wise by the reset threshold, then fused with the neighborhood information using the tanh function to generate a temporary feature. The th iteration is then selected. t The state of all nodes in the -1st iteration: Temporary feature = R: (1-R) ​​is added element-wise to complete feature fusion, which is used as the first iteration. t The state of all nodes in the next iteration, R is the update gate value;

[0042] Place t = t +1, iterate until the required number of iterations is reached, and the final state of all nodes is the feature of the gated graph. , Indicates the number of iterations;

[0043] The category probability prediction module performs the following operations:

[0044] The semantic features and gating graph features of each category output by the semantic parsing module are aligned in three channel dimensions and then concatenated. The tanh activation function is then used to output the predicted probability of the corresponding category.

[0045] Preferably, feature alignment is performed using the view function, and concatenation is performed using the torch.cat function.

[0046] Preferably, the backbone network is a ResNet101 network model.

[0047] Preferably, the preprocessing is as follows:

[0048] S11, Masking some labels in the original image dataset;

[0049] S12. Perform data augmentation processing on all images in the original image dataset;

[0050] S13. Combine the data augmented images and the labels from the original masked image dataset to form a training set.

[0051] Preferably, some labels in the original image dataset are masked, specifically as follows:

[0052] Construct a label matrix for the original image dataset. The size of the label matrix for the original image dataset is... N OK M Column, and the first n OK m The value of a column is either -1 or 1, where -1 indicates the first column. n The image does not contain the first one. m The first category; 1 indicates the first... n The image contains the first m Categories n =1~ N , m =1~ M , N Indicates the total number of images;

[0053] Generate an initial matrix of the same size as the label matrix of the original image dataset, and based on preset values. α Transform the initial matrix into a mask matrix, that is, if the initial matrix's first... n OK m The elements in the column are less than the preset value α If the value is true, the position is reset to 1; otherwise, the position is reset to 0. Each element in the initial matrix is ​​a random floating-point number in the range [0.0, 1.0). α =[0.1, 0.2, 0.3, ..., 0.9];

[0054] The masking matrix is ​​obtained by multiplying the label matrix and the masking matrix element by element in the original image dataset. The labels with an element of 1 in the masking matrix are then regarded as the labels in the original image dataset after masking.

[0055] Preferably, each element in the initial matrix is ​​assigned a value using the numpy.random.random function.

[0056] Preferably, the data augmentation process includes sequentially performing cropping, random horizontal flipping, and normalization.

[0057] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0058] To address the limitations of existing technologies in processing partially labeled data and the performance degradation caused by image imbalance, this invention proposes a multi-label image classification method based on a selective prototype loss function. This method constructs an imbalanced image dataset (training set), distinguishes between known and unknown labels, and trains the method using a convolutional neural network (CNN). The CNN employs an uncertainty-aware selective prototype loss function, specifically by enhancing the prototype representation of known labels and adaptively mining potential positive images to alleviate the problem of missing supervision signals for some labels. Simultaneously, it utilizes statistical priors from the training set to generate probability confidence scores for each category. Finally, it adds loss weights to potentially positive unknown labels in unlabeled images while ignoring potentially negative images to optimize the model's learning performance, ensuring more accurate handling of uncertain images. Finally, the trained CNN effectively mines the accurate representation of known labels and potential positive images, aiming to significantly improve image classification performance, exhibiting more accurate recognition and stronger generalization capabilities. Attached Figure Description

[0059] Figure 1 This is a flowchart of the multi-label image classification method based on the selective prototype loss function of the present invention;

[0060] Figure 2 This is a flowchart of the convolutional neural network model of the present invention;

[0061] Figure 3 This is a flowchart of the semantic parsing module of the present invention;

[0062] Figure 4 This is a flowchart of the category relationship construction module of the present invention;

[0063] Figure 5 This is a flowchart of the category probability prediction module of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0065] It should be noted that when a component is referred to as being "connected" to another component, it can be directly connected to the other component or there may be an intervening component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application.

[0066] like Figures 1-4 As shown, a multi-label image classification method based on a selective prototype loss function includes the following steps:

[0067] S1. Obtain the original image dataset and preprocess it to form a training set. All images in the original image dataset have... M There are 10 categories and each image has at least one label.

[0068] In one embodiment, the preprocessing is as follows:

[0069] S11, Masking some labels in the original image dataset;

[0070] S12. Perform data augmentation processing on all images in the original image dataset;

[0071] S13. Combine the data augmented images and the labels from the original masked image dataset to form a training set.

[0072] In one embodiment, some labels in the original image dataset are masked, specifically as follows:

[0073] Construct a label matrix for the original image dataset. The size of the label matrix for the original image dataset is... N OK M Column, and the first n OK m The value of a column is either -1 or 1, where -1 indicates the first column. n The image does not contain the first one. m The first category; 1 indicates the first... n The image contains the first m Categories n =1~ N , m =1~ M , N Indicates the total number of images;

[0074] Generate an initial matrix of the same size as the label matrix of the original image dataset, and based on preset values. α Transform the initial matrix into a mask matrix, that is, if the initial matrix's first... n OK m The elements in the column are less than the preset value αIf the value is true, the position is reset to 1; otherwise, the position is reset to 0. Each element in the initial matrix is ​​a random floating-point number in the range [0.0, 1.0). α =[0.1, 0.2, 0.3, ..., 0.9];

[0075] The masking matrix is ​​obtained by multiplying the label matrix and the masking matrix element by element in the original image dataset. The labels with an element of 1 in the masking matrix are then regarded as the labels in the original image dataset after masking.

[0076] In one embodiment, each element in the initial matrix is ​​assigned a value using the numpy.random.random function.

[0077] In one embodiment, the data augmentation process includes sequentially performing cropping, random horizontal flipping, and normalization.

[0078] Among them, based on preset values α Partially masking the labels is used as the starting labels for the training set. Random, unknown labels are used to simulate partial labeling in the training set.

[0079] Specifically, preset value α The range is 10%-90%, with an increase of 10%. In this embodiment, each preset value is tested once, meaning a single dataset undergoes nine experiments. When masking some labels in the original image dataset, an initial matrix of the same size as the label matrix of the original image dataset is first generated, and then based on the preset values... α Transform the initial matrix into a mask matrix, that is, if the initial matrix's first... n OK m The elements in the column are less than the preset value α If the value is true, the position is reset to 1; otherwise, it is reset to 0. Each element in the initial matrix is ​​a random floating-point number in the range [0.0, 1.0) assigned using the numpy.random.random function. α =[0.1, 0.2, 0.3, ..., 0.9]. Specifically, if the label matrix of the original image dataset is multiplied by 1 in the mask matrix, it means that the original label information is preserved; if it is multiplied by 0 in the mask matrix, it means that the original label information is masked, because regardless of whether the original label is 1 or -1, its information is set to 0. By inputting a fixed ratio of 10% to 90% with an increase of 10% (preset value)... α The labels are randomly assigned to simulate partial labeling. Data augmentation is then applied to the images randomly. The augmented images and the simulated label data are combined to construct multiple training sets with different images, as shown below:

[0080] Training set A: Covers 20 categories, with approximately 10,011 images. 5,011 images are used as training images, and the remaining 4,952 images are used as test images. Data augmentation is performed on all images in the original image dataset. Assuming the images are resized to 512×512, a value can be randomly selected from the set {512, 448, 384, 320, 256} to be used as the width and height for cropping the image. The image size is then resized to 448×448, followed by random horizontal flipping and image normalization. For the labels, a mask matrix Mask20 of the same size as the label matrix of the original image dataset is generated. This mask matrix is ​​applied to partially cover the labels in the original image dataset. The augmented images and the masked labels (labels with elements of 1 in the mask matrix) are then combined to form training set A.

[0081] Training set B: Covering 80 categories, containing 120k images, divided into 80k and 40k pairs for training and testing respectively. Data augmentation is performed on all images in the original image dataset. Assuming the images are resized to 512×512, a value can be randomly selected from the set {512, 448, 384, 320, 256} as the width and height for cropping the image. The image size is then adjusted to 448×448, followed by random horizontal flipping and image normalization. For the labels, a mask matrix Mask80 of the same size as the label matrix of the original image dataset is generated. This mask matrix is ​​applied to partially cover the labels in the original image dataset. The augmented images and the masked labels (labels with elements of 1 in the mask matrix) are then combined to form training set B.

[0082] Training set C: Covers 200 categories, with 10,000 images selected for training and 98,249 images for testing. Data augmentation is performed on all images in the original image dataset. Assuming the images are resized to 512×512, a value can be randomly selected from the set {512, 448, 384, 320, 256} as the width and height for cropping the image. The image size is then resized to 448×448, followed by random horizontal flipping and image normalization. For the labels, a mask matrix Mask200 of the same size as the label matrix of the original image dataset is generated. This mask matrix is ​​applied to partially cover the labels in the original image dataset. The augmented images and the masked labels (labels with elements of 1 in the mask matrix) are then combined to form training set C.

[0083] The datasets described above are widely used in general object detection and classification tasks, suitable for recognizing common objects, typically including categories such as people, cars, and bicycles. The label design emphasizes coverage of basic object categories, making it suitable for verifying the model's robustness to common objects. Dataset B, on the other hand, covers everyday items, body parts (such as "hands" and "feet"), and small targets in complex scenes. Its label design emphasizes fine-grained discriminative ability. Dataset C, employing high label density (an average of over 30 objects labeled per image), includes not only specific objects but also their corresponding attributes, such as ("cup" - "white"). Its label types and application scenarios reflect the high complexity and fine-grained requirements of multi-label classification tasks.

[0084] S2. Establish a convolutional neural network model and train it using a training set. The convolutional neural network model uses a selective prototype loss function as a penalty function.

[0085] In one embodiment, the selective prototype loss function is obtained as follows:

[0086] S21. Dynamically determine the number of prototypes for each category, then the number of prototypes for the first category... j Number of prototypes in each category The formula is as follows:

[0087] ;

[0088] in, Indicates containing the first j The total number of images in each category This indicates the maximum limit for the preset number of prototypes. ;

[0089] S22. Using the KMeans algorithm, cluster all semantic features of each category and select the cluster centers as prototypes to obtain the prototype set of all categories. Prototype set The formula is as follows:

[0090] ;

[0091] in,

[0092] ;

[0093] ;

[0094] in, Indicates the first j All semantic features of each category Indicates the first i The first image j Semantic features of each category Indicates the firstj The first category k A prototype, , , N This indicates the total number of images. Indicates containing the first j Number of images in each category This represents the KMeans algorithm;

[0095] S23. Calculate the semantic features and prototype sets of all images corresponding to their respective categories. The cosine similarity between them and the probability confidence of each category;

[0096] S24. Based on the semantic features and prototype set of all images corresponding to their respective categories. The selective prototype loss function is calculated using the cosine similarity between the groups and the probability confidence of each category. The formula is as follows:

[0097] ;

[0098] in,

[0099] ;

[0100] in, Indicate the discrimination criteria, Indicates the multiplier. Let represent the cosine similarity of the i-th image to the j-th category. Let represent the probability confidence level of the j-th category, and s represent the preset hyperparameters. The first conditional function represents the label of the i-th image in the j-th category. Indicates if If the value is -1, then the value of the first conditional function is... Otherwise, it is 0; second condition function Indicates if If the value equals 1, then the value of the second conditional function is 1. Otherwise, it is 0; third condition function Indicates if If the value is 0, then the value of the third conditional function is... Otherwise, it is 0; This represents the probability confidence threshold, and the probability that the label truly does not exist. If the probability confidence threshold is less than 1, then the corresponding image is considered to contain the label of the j-th category of the i-th image. Otherwise, it is considered that the corresponding image does not contain the label of the j-th category of the i-th image. .

[0101] Preferably, the semantic features and prototype sets of all images corresponding to their respective categories are calculated respectively. The cosine similarity between the pairs and the probability confidence of each category are calculated using the following formula:

[0102] ;

[0103] ;

[0104] in, This represents the predicted probability of the i-th image belonging to the j-th class, as output by the convolutional neural network model. Indicates the modulus length.

[0105] In this process, the number of prototypes for each category is dynamically determined based on the number of images. Next, the KMeans algorithm is applied to cluster the semantic features of the images, and the cluster centers are selected as prototypes. Contrastive learning is used to calculate the cosine similarity between the semantic features of a category and its prototype set. After obtaining the predicted probabilities, the category probability prediction module performs average statistics to obtain the predicted probability for each category. The confidence score for each category is then calculated. Similarity to cosine They are compared together using the discrimination criteria. Based on the results, the original true label of the unknown label is selected, and corresponding loss measures are taken. The specific design is as follows: Because... This represents the similarity between the semantic features of an image and its prototype, with a value ranging from [-1, 1]. The closer the value is to 1, the greater the similarity. Since the prediction probability cannot be negative, directly comparing it to the similarity score would introduce significant errors. Therefore, the similarity score needs to be converted to a non-negative value. ;Will The result is considered a measure of the probability that the label does not actually exist, with a value range of [0, 2]. The similarity score... The higher, The smaller the value, the less likely the label is to actually not exist; This represents the probability confidence threshold obtained during model training. If the probability measure of the label not actually existing is less than the probability confidence threshold, the image is considered to contain the label, and its loss is amplified. Conversely, if it is greater than the threshold, the image is considered not to contain the label, and its loss is suppressed.

[0106] In one embodiment, the convolutional neural network model includes a backbone network, a semantic parsing module, a category relationship construction module, and a category probability prediction module connected in sequence, wherein:

[0107] The semantic parsing module performs the following operations:

[0108] The initial feature map output by the backbone network is transformed in terms of feature dimension. The feature dimension is converted into the spatial dimension of the initial feature map using the torch.transpose function. The three channels of the initial feature map after spatial dimension transformation are flattened into one dimension, namely batch size, length and width.

[0109] A pre-trained model was established and pre-trained using the original image dataset. The pre-trained weights were saved as pre-trained features, and the pre-trained features were aligned with the flattened initial feature map in the spatial dimension. The pre-trained model was a ResNet50 network model.

[0110] The flattened initial feature map is multiplied element-wise with the pre-trained features after feature alignment to form a fused feature;

[0111] The fused features are passed sequentially through a tanh activation function and a fully connected layer to obtain the weight coefficients.

[0112] The fused features and weight coefficients are weighted and summed to output all semantic features for each category;

[0113] The category relationship construction module performs the following operations:

[0114] Each category is treated as a node, and all semantic features of each category output by the semantic parsing module are initialized to the state of the corresponding node.

[0115] Statistical analysis of the first image in the original image dataset a The category and the first b The frequencies of common occurrence of each category are used to initialize the in-degree and out-degree matrices, both of which are... M × M A two-dimensional adjacency matrix, and located at the th a OK b The element of the column represents the first... a The category and the first b The degree of relevance of each category ;

[0116] Conduct the first t The iteration updates the state of the nodes, specifically by: interpolating the in-degree matrix with the first iteration. t The in-degree correlation matrix is ​​obtained by element-wise multiplying the states of all nodes in the -1 iteration, and the out-degree matrix is ​​then compared with the state of the -1th iteration. t The out-degree correlation matrix is ​​obtained by element-wise multiplication of the states of all nodes in the -1st iteration, and the in-degree correlation matrix and the out-degree correlation matrix are concatenated to form neighborhood information; then the -1st iteration... tAfter summing the states and neighborhood information of all nodes in the -1 iteration element by element, the values ​​are then processed by the first sigmoid function to output an update gate and by the second sigmoid function to output a reset gate. The update gate is used to represent the value retained in the -1 iteration. t The proportion of the node's state in the -1st iteration, with the reset threshold used to represent the proportion of the node's state in the -1st iteration. t The states of all nodes in the -1st iteration are generated in the... t The participation of all nodes in the state of the next iteration; the participation of the first... t The states of all nodes in the -1st iteration are multiplied element-wise by the reset threshold, then fused with the neighborhood information using the tanh function to generate a temporary feature. The th iteration is then selected. t The state of all nodes in the -1st iteration: Temporary feature = R: (1-R) ​​is added element-wise to complete feature fusion, which is used as the first iteration. t The state of all nodes in the next iteration, R is the update gate value;

[0117] Place t = t +1, iterate until the required number of iterations is reached, and the final state of all nodes is the feature of the gated graph. , Indicates the number of iterations;

[0118] The category probability prediction module performs the following operations:

[0119] The semantic features and gating graph features of each category output by the semantic parsing module are aligned in three channel dimensions and then concatenated. The tanh activation function is then used to output the predicted probability of the corresponding category.

[0120] In one embodiment, feature alignment is performed using the `view` function, and concatenation is performed using the `torch.cat` function.

[0121] In one embodiment, the backbone network is a ResNet101 network model.

[0122] Specifically, this convolutional neural network model uses a ResNet101 network as its backbone, which is then connected to three modules in sequence: a semantic parsing module, a category relationship construction module, and a category probability prediction module. The input to the backbone network is a 448×448 three-channel color RGB image, with a preset fixed value for BatchSize, which was set to 32 during experimental training. After passing through the backbone network, the output is an initial feature map with dimensions (BatchSize, 3, 448, 448).

[0123] In the semantic parsing module, the initial feature map is transformed in spatial dimension using the `torch.transpose` function, flattening it into one dimension across the three channels. A ResNet50 network model is built as a pre-trained model and pre-trained using the original image dataset. The pre-trained weights are saved as pre-trained features. The pre-trained features are aligned in spatial dimension using the `view` function and interacted with the initial feature map. Elements at each corresponding position are multiplied to obtain fused features. The fused features are then processed using the tanh activation function to generate weights, which are then normalized using a fully connected layer to obtain weight coefficients. The fused features and weight coefficients are weighted and summed to output all semantic features for each category.

[0124] In the category relationship construction module, each category is treated as a node, and all semantic features of each category output by the semantic parsing module are initialized as the state of the corresponding node; that is, the state of the node is the semantic feature. The statistics of the first [node name] in the original image dataset are then analyzed. a The category and the first b The frequencies of common occurrence of each category are used as element values ​​to initialize the in-degree and out-degree matrices, both of which are... M × M A two-dimensional adjacency matrix, where all elements are initialized to the frequency values ​​of the corresponding statistics. a OK b Column representation a Category and b There is a correlation strength between categories, the first a OK b The value of the column represents the first a The category and the first b The relevance of each category, i.e., the node value. Then, after... K The next iteration (such as) K =5), calculate the adjacency information using the in-degree matrix and the out-degree matrix, and then... t After summing the states and neighborhood information of all nodes in the -1 iteration element by element, the values ​​are then processed by the first sigmoid function to output an update gate and by the second sigmoid function to output a reset gate. The update gate is used to represent the value retained in the -1 iteration. t The proportion of the state of the node in the -1st iteration (e.g., 0.7 means retaining 70% of the state in the -1st iteration). t (The state of the node in the -1st iteration), the reset threshold is used to represent the state of the node in the -1st iteration. t The states of all nodes in the -1st iteration are generated in the... t The participation of all nodes in the state of the next iteration (e.g., 0.3 indicates the generation of the first iteration) t When considering the states of all nodes in the next iteration, select 30% of the nodes in the first iteration. t (Involving the states of all nodes in the -1st iteration); ... tThe states of all nodes in the -1st iteration are multiplied element-wise by the reset threshold, then fused with the neighborhood information using the tanh function to generate a temporary feature. The th iteration is then selected. t The state of all nodes in the -1st iteration: Temporary feature = R:(1-R) ​​is summed element by element to complete feature fusion (e.g., updating the gate value to 0.7 indicates that 70% of the nodes in the -1st iteration are in the state of the temporary feature = R:(1-R). t -1 iteration's node state + 30% temporary features are added element-wise to complete feature fusion), as the first iteration's... t The state of all nodes in the next iteration is updated to adjust the corresponding semantic features, so that the more related the categories are, the more similar the semantic feature representations will be, and finally the gated graph features are obtained.

[0125] In the category probability prediction module, the model recognizes and learns which information needs to be retained and which useless information should be discarded, and uses the tanh activation function to calculate the predicted probability for each category. The backbone network, semantic parsing module, category relationship construction module, and category probability prediction module are sequentially connected. The obtained semantic features and predicted probabilities are utilized and fed into the discrimination criteria to formulate a strategy, resulting in the selective prototype loss function value. Next, the model parameters are iteratively updated using the Adam optimization algorithm to minimize the loss function and obtain the trained convolutional neural network model.

[0126] S3. Use the trained convolutional neural network model to classify the image to be tested, and obtain the predicted probability of the corresponding category, which is the multi-label image classification result.

[0127] The following demonstration is based on specific experimental data.

[0128] Tables 1-3 compare the classification results evaluation metrics on different datasets (datasets A, B, and C mentioned above). The tables show the comparative experimental results of the HST model (HST for short, sourced from reference: Chen T, Pu T, Liu L, et al. Heterogeneous semantic transfer for multi-label recognition with partial labels[J]. International Journal of Computer Vision, 2024, 132(12): 6091-6106.), with three metrics: 1) Average precision: commonly used for ranking tasks to measure the accuracy of correctly identifying any class. It considers the relationship between recall and precision. Recall refers to the proportion of positive class images predicted by the model out of all true positive class images. Precision refers to the proportion of actually correct positive class images in the model's predictions. A higher result value indicates better model recognition performance. 2) Macro F1 Score: This refers to calculating the F1 score for each category separately and then averaging them. The F1 score here is the harmonic mean of precision and recall. 3) Micro F1 Score: This involves first globally counting the true positives, false positives, and false negatives for all categories, and then calculating the overall precision and recall based on these totals to obtain the macro average.

[0129] Table 1 Comparison of classification result evaluation metrics on dataset A

[0130]

[0131] Table 2 Comparison of classification result evaluation metrics on dataset B

[0132]

[0133] Table 3 Comparison of classification result evaluation metrics on dataset C

[0134]

[0135] On dataset A, our method achieves a 2.7% improvement in average precision compared to the HST model at α=10%, and maintains a stable gain of 1% at α=90%, indicating good robustness to different label densities. Furthermore, the improvements in macro and micro F1 scores are primarily reflected in the optimization of prediction capabilities for long-tailed categories (e.g., a 5.9% improvement in micro F1 score at α=10% for dataset C), directly reflecting the model's effectiveness in mitigating class imbalance. By combining the collaborative optimization mechanism of dynamic prototype generation and probability confidence thresholds in the experiments, our method achieves more efficient utilization of partially labeled data by enhancing the representation learning of known labels and mining potential positive samples.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The embodiments described above are merely specific and detailed examples of the embodiments described in this application, and should not be construed as limiting the scope of the application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A multi-label image classification method based on selective prototype loss function, characterized in that: The multi-label image classification method based on the selective prototype loss function includes the following steps: S1. Obtain the original image dataset and preprocess it to form a training set. All images in the original image dataset have... M Each image has at least one label and is categorized into several categories. S2. Establish a convolutional neural network model and train it using a training set. The convolutional neural network model includes a backbone network, a semantic parsing module, a category relationship construction module, and a category probability prediction module connected in sequence. The convolutional neural network model uses a selective prototype loss function as a penalty function. The selective prototype loss function is obtained as follows: S21. Dynamically determine the number of prototypes for each category, then the number of prototypes for the first category... j Number of prototypes in each category The formula is as follows: ； in, Indicates containing the first j The total number of images in each category This indicates the maximum limit for the preset number of prototypes. ; S22. Using the KMeans algorithm, cluster all semantic features of each category and select the cluster centers as prototypes to obtain the prototype set of all categories. The prototype set The formula is as follows: ； in, ； ； in, Indicates the first j All semantic features of each category Indicates the first i The first image j Semantic features of each category Indicates the first j The first category k A prototype, , , N This indicates the total number of images. Indicates containing the first j Number of images in each category This represents the KMeans algorithm; S23. Calculate the semantic features and prototype sets of all images corresponding to their respective categories. The cosine similarity between them and the probability confidence of each category; S24. Based on the semantic features and prototype set of all images corresponding to their respective categories. The cosine similarity between the pairs and the probability confidence of each category are used to calculate the selective prototype loss function. The formula is as follows: ； in, ； in, Indicate the discrimination criteria, Indicates the multiplier. Let represent the cosine similarity of the i-th image to the j-th category. Let represent the probability confidence level of the j-th category, and s represent the preset hyperparameters. The first conditional function represents the label of the i-th image in the j-th category. Indicates if If the value is -1, then the value of the first conditional function is... Otherwise, it is 0; second condition function Indicates if If the value equals 1, then the value of the second conditional function is 1. Otherwise, it is 0; third condition function Indicates if If the value is 0, then the value of the third conditional function is... Otherwise, it is 0; This represents the probability confidence threshold, and the probability that the label truly does not exist. If the probability confidence threshold is less than 1, then the corresponding image is considered to contain the label of the j-th category of the i-th image. Otherwise, it is considered that the corresponding image does not contain the label of the j-th category of the i-th image. ; S3. Use the trained convolutional neural network model to classify the image to be tested, and obtain the predicted probability of the corresponding category, which is the multi-label image classification result.

2. The multi-label image classification method based on selective prototype loss function as described in claim 1, characterized in that: The semantic features and prototype sets corresponding to all image categories are calculated respectively. The cosine similarity between the pairs and the probability confidence of each category are calculated using the following formula: ； ； in, This represents the predicted probability of the i-th image belonging to the j-th class, as output by the convolutional neural network model. Indicates the modulus length.

3. The multi-label image classification method based on selective prototype loss function as described in claim 1, characterized in that: The semantic parsing module performs the following operations: The initial feature map output by the backbone network is transformed in terms of feature dimension. The feature dimension is converted into the spatial dimension of the initial feature map using the torch.transpose function. The three channels of the initial feature map after spatial dimension transformation are flattened into one dimension, namely the batch size, length and width. A pre-trained model is established and pre-trained using the original image dataset. The pre-trained weights are saved as pre-trained features, and the pre-trained features are aligned with the flattened initial feature map in the spatial dimension. The pre-trained model is a ResNet50 network model. The flattened initial feature map is multiplied element-wise with the pre-trained features after feature alignment to form a fused feature; The fused features are passed sequentially through a tanh activation function and a fully connected layer to obtain the weight coefficients. The fused features and weight coefficients are weighted and summed to output all semantic features for each category; The category relationship construction module performs the following operations: Each category is treated as a node, and all semantic features of each category output by the semantic parsing module are initialized to the state of the corresponding node. Statistical analysis of the first image in the original image dataset a The category and the first b The frequencies of common occurrence of each category are used to initialize the in-degree and out-degree matrices, which are both... M × M A two-dimensional adjacency matrix, and located at the th a OK b The element of the column represents the first... a The category and the first b The degree of relevance of each category ; Proceed to the first t The iteration updates the state of the nodes, specifically by: interpolating the in-degree matrix with the first iteration. t The in-degree correlation matrix is ​​obtained by element-wise multiplying the states of all nodes in the -1 iteration, and the out-degree matrix is ​​then compared with the state of the -1th iteration. t The out-degree correlation matrix is ​​obtained by element-wise multiplication of the states of all nodes in the -1st iteration, and the in-degree correlation matrix and the out-degree correlation matrix are concatenated to form neighborhood information; then the -1st iteration... t After summing the states of all nodes and their neighborhood information element by element in the -1 iteration, the results are processed by the first sigmoid function to output an update gate and by the second sigmoid function to output a reset gate. The update gate is used to represent the value retained in the -1 iteration. t The proportion of the node's state in the -1st iteration, where the reset threshold is used to represent the proportion of the node's state in the -1st iteration. t The states of all nodes in the -1st iteration are generated in the... t The participation of all nodes in the state of the next iteration; the participation of the first... t The states of all nodes in the -1st iteration are multiplied element-wise by the reset threshold, then fused with the neighborhood information using the tanh function to generate a temporary feature. The th iteration is then selected. t The state of all nodes in the -1st iteration: Temporary feature = R: (1-R) ​​is added element-wise to complete feature fusion, which is used as the first iteration. t The state of all nodes in the next iteration, R is the update gate value; Place t = t +1, iterate until the required number of iterations is reached, and the final state of all nodes is the feature of the gated graph. , Indicates the number of iterations; The category probability prediction module performs the following operations: The semantic features and gating graph features of each category output by the semantic parsing module are aligned in three channel dimensions and then concatenated. The result is then processed by the tanh activation function to output the predicted probability of the corresponding category.

4. The multi-label image classification method based on selective prototype loss function as described in claim 3, characterized in that: The feature alignment uses the view function, and the splicing uses the torch.cat function.

5. The multi-label image classification method based on selective prototype loss function as described in claim 3, characterized in that: The backbone network is a ResNet101 network model.

6. The multi-label image classification method based on selective prototype loss function as described in claim 1, characterized in that: The preprocessing is as follows: S11, Masking some labels in the original image dataset; S12. Perform data augmentation processing on all images in the original image dataset; S13. Combine the data augmented images and the labels from the original masked image dataset to form a training set.

7. The multi-label image classification method based on selective prototype loss function as described in claim 6, characterized in that: The specific details of masking some labels in the original image dataset are as follows: Establish a label matrix for the original image dataset, the size of which is... N OK M Column, and the first n OK m The value of a column is either -1 or 1, where -1 indicates the first column. n The image does not contain the first one. m The first category; 1 indicates the first... n The image contains the first m Categories n =1~ N , m =1~ M , N Indicates the total number of images; Generate an initial matrix of the same size as the label matrix of the original image dataset, and based on preset values. α Transform the initial matrix into a mask matrix, that is, if the initial matrix's first... n OK m The elements in the column are less than the preset value α If the value is 1, the position is reset to 1; otherwise, the position is reset to 0. Each element in the initial matrix is ​​a random floating-point number in the range [0.0, 1.0). α =[0.1, 0.2, 0.3, ..., 0.9]; The masking matrix is ​​obtained by multiplying the label matrix and the masking matrix element by element in the original image dataset. The labels with an element of 1 in the masking matrix are then regarded as the labels in the original image dataset after masking.

8. The multi-label image classification method based on selective prototype loss function as described in claim 7, characterized in that: Each element in the initial matrix is ​​assigned a value using the numpy.random.random function.

9. The multi-label image classification method based on selective prototype loss function as described in claim 6, characterized in that: The data augmentation process includes cropping, random horizontal flipping, and normalization in sequence.

Images