A method, apparatus, electronic device, and storage medium for label classification

Abstract

An embodiment of the present invention provides a method, device, electronic device, and storage medium for label classification. The method includes: obtaining a sample image data set, and iteratively executing the following steps: sequentially inputting a plurality of sample images into a multi-label classification model to obtain a sample prediction probability vector and a sample feature map; determining labels for each category according to the sample prediction probability vector; after each iteration is completed, calculating the feature clustering diameter of the category; when the number of iterations meets a preset number, obtaining the feature clustering diameters of the previous N times to adjust the weight parameter in the loss function, so as to adjust the labels of the categories in the sample image. The present invention sets a weight parameter in the loss function, and adjusts the weight parameter by obtaining the feature clustering diameters obtained in the previous N iterations. Through the adjusted weight parameter and the loss function, the labels of the categories in the sample image are adjusted according to the training effect, so as to realize the excavation of potential label information as much as possible.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method for label classification, a device for label classification, an electronic device, and a computer-readable storage medium. Background Art

[0002] In recent years, a trend in multi-label learning is to start paying attention to how to construct a better learning model under limited supervision. Related multi-label learning problems include two categories: multi-label learning with missing labels (MLML) and multi-label learning with partial annotations (PML). Among them, the former focuses on unlabeled data, and the latter focuses on difficult-to-label data.

[0003] In the related art, there is a problem that different instance images of the same category in the feature correlation between images also contain other categories, resulting in the inability to accurately measure the feature correlation and label correlation between images. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a method for label classification, a device for label classification, an electronic device, and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0005] To solve the above problems, in a first aspect, embodiments of the present invention disclose a method for label classification, the method including:

[0006] Obtain a sample image data set and a label data set; the sample image data set includes a plurality of sample images;

[0007] Iteratively execute the following steps:

[0008] Input the label data set into a multi-label classification model, and sequentially input a plurality of sample images in the sample image data set into the multi-label classification model to obtain a sample prediction probability vector and a sample feature map corresponding to each of the plurality of sample images one by one;

[0009] Determine the labels of each category in the sample image according to the sample prediction probability vector; the labels include positive labels, negative labels, and missing labels;

[0010] After each iteration is completed, calculate the feature clustering diameter of each category according to the sample feature map;

[0011] When the number of iterations meets a preset number, obtain the feature clustering diameters obtained in the previous N iterations;

[0012] Adjust the weight parameter for the missing label according to the feature clustering diameters obtained in the previous N iterations, where the weight parameter is used to measure the credibility of the missing label;

[0013] Determine the classification loss value of the category in the sample image according to the weight parameter and the loss function, so as to adjust the label of the category in the sample image.

[0014] Optionally, calculating the feature clustering diameter of each category according to the sample feature map includes:

[0015] Determine the feature vectors of the positive label and the missing label of the category in the sample image according to the sample feature map;

[0016] Pass the feature vectors through the K-means clustering algorithm to calculate the feature clustering diameter of each category.

[0017] Optionally, the multi-label classification model includes a convolutional hidden layer and a fully connected hidden layer. Inputting the label data set into the multi-label classification model, and sequentially inputting multiple sample images in the sample image data set into the multi-label classification model to obtain a sample prediction probability vector and a sample feature map corresponding to each of the multiple sample images, includes:

[0018] Input the label data set into the convolutional hidden layer of the multi-label classification model, and sequentially input multiple sample images in the sample image data set into the convolutional hidden layer of the multi-label classification model. Generate the sample feature map corresponding to each sample image through the convolutional hidden layer of the multi-label classification model, and input the sample feature map into the fully connected hidden layer of the multi-label classification model to output the sample prediction probability vector corresponding to each sample image, where the sample prediction probability vector includes the classification prediction probability values of each category.

[0019] Optionally, determining the labels of each category in the sample image according to the sample prediction probability vector includes:

[0020] If the classification prediction probability value is greater than the preset probability threshold and the label value of the category is 0, determine that the label of the category in the sample image is a missing label.

[0021] Optionally, when N is 2, the feature clustering diameters obtained in the previous N iterations include a first feature clustering diameter and a second feature clustering diameter; the value of the weight parameter is 0 or 1. Adjusting the weight parameter for the missing label according to the feature clustering diameters obtained in the previous N iterations includes:

[0022] If the label of the category in the sample image is the missing label and the second feature clustering diameter is less than or equal to the first feature clustering diameter, keep the weight parameter for the missing label unchanged;

[0023] If the label of the category in the sample image is the missing label and the second feature clustering diameter is greater than the first feature clustering diameter, then adjust the weight parameter for the missing label to another weight parameter different from the previous weight parameter.

[0024] Optionally, generating the feature vectors with positive labels and missing labels for the categories in the sample image according to the sample feature mapping graph includes:

[0025] Determine the feature regions with positive labels and missing labels for the categories in the sample image according to the class activation visualization method and the threshold segmentation method;

[0026] Generate the feature vectors with positive labels and missing labels for the categories in the sample image according to the feature regions with positive labels and missing labels for the categories in the sample image.

[0027] Optionally, determining the classification loss value of the sample image according to the weight parameter and the loss function to adjust the label of the sample image includes:

[0028] If the weight parameter is 1, select the first loss function formula to calculate and obtain the first classification loss value;

[0029] According to the first classification loss value, adjust the label of the type in the sample image from the missing label to the negative label;

[0030] If the weight parameter is 0, select the second loss function formula to calculate and obtain the second classification loss value;

[0031] According to the second classification loss value, adjust the label of the type in the sample image from the missing label to the positive label;

[0032] Among them, the first loss function formula is:

[0033]

[0034] The second loss function formula is:

[0035]

[0036] Among them, i represents the i-th category in the sample image, c represents the number of categories in the sample image; Y is the label value; Y i = 1 is the positive label; Y i = 0 is the negative label or the missing label; y i represents the output probability value of the i-th category in the sample image; L 1 represents the first classification loss value; L 2 represents the second classification loss value.

[0037] In a second aspect, an embodiment of the present invention also discloses a device for label classification, the device comprising:

[0038] A dataset acquisition module, configured to acquire a sample image dataset and a label dataset; the sample image dataset includes a plurality of sample images;

[0039] A vector and feature map generation module, configured to iteratively perform the following steps: input the label dataset into a multi-label classification model, and sequentially input a plurality of sample images in the sample image dataset into the multi-label classification model, so as to obtain a sample prediction probability vector and a sample feature map corresponding to each of the plurality of sample images one by one;

[0040] A label determination module, configured to determine labels of each category in the sample image according to the sample prediction probability vector; the labels include positive labels, negative labels, and missing labels;

[0041] A clustering diameter calculation module, configured to calculate a feature clustering diameter of each category according to the sample feature map after each iteration;

[0042] A clustering diameter acquisition module, configured to acquire the feature clustering diameters obtained in the previous N iterations when the number of iterations meets a preset number;

[0043] A weight parameter adjustment module, configured to adjust a weight parameter for a missing label according to the feature clustering diameters obtained in the previous N iterations, where the weight parameter is used to measure the credibility of the missing label;

[0044] A label adjustment module, configured to determine a classification loss value of a category in the sample image according to the weight parameter and a loss function, so as to adjust the label of the category in the sample image.

[0045] Optionally, the clustering diameter calculation module includes:

[0046] A feature vector determination sub-module, configured to determine feature vectors of positive labels and missing labels of the labels of the categories in the sample image according to the sample feature map;

[0047] A clustering diameter calculation sub-module, configured to calculate a feature clustering diameter of each category by using the K-means clustering algorithm for the feature vectors.

[0048] Optionally, the multi-label classification model includes a convolutional hidden layer and a fully connected hidden layer, and the vector and feature map generation module includes:

[0049] A vector and feature map generation sub-module, configured to input the label data set into the convolutional hidden layer of the multi-label classification model, and sequentially input multiple sample images in the sample image data set into the convolutional hidden layer of the multi-label classification model, generate the sample feature maps corresponding to the sample images one by one through the convolutional hidden layer of the multi-label classification model, and input the sample feature maps into the fully connected hidden layer of the multi-label classification model to output the sample prediction probability vectors corresponding to the sample images one by one, where the sample prediction probability vectors include the classification prediction probability values of each category.

[0050] Optionally, the label determination module includes:

[0051] A label determination sub-module, configured to determine that the label of the category in the sample image is a missing label if the classification prediction probability value is greater than a preset probability threshold and the label value of the category is 0.

[0052] Optionally, when N is 2, the feature clustering diameters obtained in the first N iterations include a first feature clustering diameter and a second feature clustering diameter; the value of the weight parameter is 0 or 1, and the weight parameter adjustment module includes:

[0053] A first weight parameter adjustment sub-module, configured to keep the weight parameter for the missing label unchanged if the label of the category in the sample image is the missing label and the second feature clustering diameter is less than or equal to the first feature clustering diameter;

[0054] A second weight parameter adjustment sub-module, configured to adjust the weight parameter for the missing label to another weight parameter different from the previous weight parameter if the label of the category in the sample image is the missing label and the second feature clustering diameter is greater than the first feature clustering diameter.

[0055] Optionally, the feature vector determination sub-module includes:

[0056] A region determination unit, configured to determine the feature regions of the positive label and the missing label of the category in the sample image according to the class activation visualization method and the threshold segmentation method;

[0057] A feature vector generation unit, configured to generate the feature vectors of the positive label and the missing label of the category in the sample image according to the feature regions of the positive label and the missing label of the category in the sample image.

[0058] Optionally, the label adjustment module includes:

[0059] A first loss value calculation sub-module, configured to select the first loss function formula for calculation to obtain the first classification loss value if the weight parameter is 1;

[0060] The first label adjustment sub-module is configured to adjust the label of the type in the sample image from a missing label to a negative label according to the first classification loss value;

[0061] The second loss value calculation sub-module is configured to, if the weight parameter is 0, select a second loss function formula for calculation to obtain a second classification loss value;

[0062] The second label adjustment sub-module is configured to adjust the label of the type in the sample image from a missing label to a positive label according to the second classification loss value;

[0063] Among them, the first loss function formula is:

[0064]

[0065] The second loss function formula is:

[0066]

[0067] Among them, i represents the i-th category in the sample image, c represents the number of categories in the sample image; Y is the label value; Y i = 1 is a positive label; Y i = 0 is a negative label or a missing label; y i represents the output probability value of the i-th category in the sample image; L 12 represents the first classification loss value; L 2 represents the second classification loss value.

[0068] In a third aspect, an embodiment of the present invention further discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0069] The memory is used to store a computer program;

[0070] The processor is configured to, when executing the program stored on the memory, implement the method for label classification as described in the embodiment of the present invention.

[0071] In a fourth aspect, an embodiment of the present invention further discloses one or more computer-readable media, on which instructions are stored. When executed by one or more processors, the instructions cause the processors to execute the method for label classification as described in the embodiment of the present invention.

[0072] The embodiments of the present invention include the following advantages:

[0073] In an embodiment of the present invention, a sample image dataset and a label dataset are obtained, and the following steps are iteratively executed: inputting the label dataset into a multi-label classification model, and sequentially inputting multiple sample images in the sample image dataset into the multi-label classification model to obtain sample prediction probability vectors and sample feature maps corresponding to the multiple sample images one by one; determining labels of each category in the sample images according to the sample prediction probability vectors; after each iteration is completed, calculating the feature clustering diameter of each category according to the sample feature maps; when the number of iterations meets a preset number, obtaining the feature clustering diameters obtained in the previous N iterations; adjusting the weight parameter for missing labels according to the feature clustering diameters obtained in the previous N iterations, and determining the classification loss value of the categories in the sample images according to the weight parameter and the loss function to adjust the labels of the categories in the sample images. The present invention adjusts the weight parameter for missing labels by setting a weight parameter in the loss function and obtaining the feature clustering diameters obtained in the previous N iterations, and adjusts the labels of the categories in the sample images according to the adjusted weight parameter and the loss function, so as to realize mining potential label information as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flowchart of the steps of a method for label classification provided by an embodiment of the present invention;

[0075] Figure 2 is a structural diagram of a multi-label classification model provided by an embodiment of the present invention;

[0076] Figure 3 is a flowchart of model training provided by an embodiment of the present invention;

[0077] Figure 4 is a structural block diagram of a label classification device provided by an embodiment of the present invention;

[0078] Figure 5 is a structural block diagram of an electronic device provided by an embodiment of the present invention;

[0079] Figure 6 is a structural block diagram of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0080] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0081] In the related art, there is a problem that different instance images of the same category in the feature correlation between images also contain other categories, resulting in the inability to accurately measure the feature correlation and label correlation between images.

[0082] One of the core concepts of the embodiments of the present invention is to set a weight parameter in the loss function, and adjust the weight parameter for the missing labels by obtaining the feature clustering diameters obtained in the previous N iterations. Through the adjusted weight parameter and the loss function, the labels of the categories in the sample images are adjusted according to the training effect, so as to realize mining potential label information as much as possible.

[0083] Referring to Figure 1 , a flowchart of steps of a method for label classification provided by an embodiment of the present invention is shown. The method may specifically include the following steps:

[0084] Step 101, obtain a sample image dataset and a label dataset; the sample image dataset includes multiple sample images;

[0085] Exemplarily, a sample image dataset that has been preprocessed can be obtained. Among them, the preprocessing process may include methods such as brightness normalization, noise filtering, scale normalization, cropping, and flipping. Specifically, the preprocessing method can be selected according to the actual situation, and those skilled in the art do not make limitations here.

[0086] In one example, the OpenCV image processing library can be used to preprocess the samples, and the preprocessing method is as follows:

[0087] 1) Cropping: Increasing the randomness of the image data can achieve data augmentation;

[0088] 2) Flipping / rotation: Flipping left and right, up and down to increase randomness;

[0089] 3) Interpolation: Adjust the image size to the model input image size;

[0090] 4) Brightness normalization: Normalize the image based on the mean and variance of the image brightness, which can make the neural network model converge more easily;

[0091] In the embodiments of the present invention, a sample image dataset and a label dataset can be obtained. Among them, the sample image dataset can include multiple sample images, and the label dataset can include multiple list labels.

[0092] Step 102, iteratively execute the following steps:

[0093] Input the label dataset into a multi-label classification model, and sequentially input multiple sample images in the sample image dataset into the multi-label classification model to obtain a sample prediction probability vector and a sample feature map corresponding to each of the multiple sample images;

[0094] In an embodiment of the present invention, the label dataset can be input into the multi-label classification model, and multiple sample images in the sample image dataset can be sequentially input into the multi-label classification model to obtain sample prediction probability vectors and sample feature maps corresponding one-to-one to the multiple sample images.

[0095] In an embodiment of the present invention, the multi-label classification model includes a convolutional hidden layer and a fully-connected hidden layer, and step 102 may include sub-step S11:

[0096] Sub-step S11: Input the label dataset into the convolutional hidden layer of the multi-label classification model, and sequentially input multiple sample images in the sample image dataset into the convolutional hidden layer of the multi-label classification model. Generate the sample feature maps corresponding one-to-one to the sample images through the convolutional hidden layer of the multi-label classification model, and input the sample feature maps into the fully-connected hidden layer of the multi-label classification model to output the sample prediction probability vectors corresponding one-to-one to the sample images. The sample prediction probability vectors include classification prediction probability values for each category.

[0097] Exemplarily, the multi-label classification model may include a convolutional hidden layer and a fully-connected hidden layer. Among them, the convolutional hidden layer can be used for feature extraction to obtain feature map images of several channels; the fully-connected hidden layer can output classification prediction probability values for the classifier model.

[0098] Exemplarily, the multi-label classification model can be constructed by itself. Specifically, the number of neural network layers, the number and dimension of neurons in each layer, the type of activation function, etc. can be set; or a pre-trained model architecture can be directly used. Among them, common neural networks include Residual Neural Network (ResNet), GoogleNet, VGG, InceptionNet, etc.

[0099] In an embodiment of the present invention, the multi-label classification model may include a convolutional hidden layer and a fully-connected hidden layer. The label dataset can be input into the convolutional hidden layer of the multi-label classification model, and multiple sample images in the sample image dataset can be sequentially input into the convolutional hidden layer of the multi-label classification model. Generate the sample feature maps corresponding one-to-one to the sample images through the convolutional hidden layer, and input the sample feature maps into the fully-connected hidden layer of the multi-label classification model to output the sample probability prediction vectors corresponding one-to-one to the sample images. Among them, the sample prediction probability vectors may include classification prediction probability values for each category.

[0100] Among them, the sample feature map can be denoted as [M, N, C], where C is the number of channels of the feature map, and M and N are the image sizes.

[0101] To better understand the process of using a multi-label classification model to output a sample feature map and a sample probability prediction vector corresponding to a sample image one by one, the following uses an example for illustration. Refer to Figure 2 , which shows the structural diagram of a multi-label classification model provided by an embodiment of the present invention.

[0102] Input the first sample image into the convolutional hidden layer of the multi-label classification model to obtain a first sample feature map corresponding to the first sample image one by one; input the first sample feature map into the fully connected hidden layer of the multi-label model to obtain a first sample probability prediction vector corresponding to the first sample image one by one.

[0103] Step 103, determine the labels of each category in the sample image according to the sample prediction probability vector; the labels include positive labels, negative labels, and missing labels;

[0104] In an embodiment of the present invention, the labels of each category in the sample image can be determined according to the sample prediction probability vector, where the labels can include positive labels, negative labels, and missing labels.

[0105] Exemplarily, if the label of a category is a positive label, it can be represented by the label value Y i = 1, and the label value Y i = 0 can be represented as a negative label or a missing label. Specifically, if a certain category is manually marked in the image, it is represented by Y i = 1, which is called a positive label; if there is no manual marking, the label Y i = 0; in fact, there may be two cases for unmarked labels. One is that the category exists in the image but is not marked, which is a missing label; if the category does not exist in the image, the label is a negative label.

[0106] In an embodiment of the present invention, step 103 may include sub-step S21:

[0107] Sub-step S21, if the classification prediction probability value is greater than a preset probability threshold and the label value of the category is 0, determine that the label of the category in the sample image is a missing label.

[0108] In an embodiment of the present invention, the sample prediction probability vector may include the classification prediction probability values of each category. If the classification preset probability value of a category is greater than the preset probability threshold and the label value of the category is 0, it can be determined that the label of the category in the sample image is a missing label.

[0109] Exemplarily, the determination of the missing label can be obtained through forward inference, where the determination condition can be that the classification prediction probability value is greater than the preset probability threshold and the label value Y i= 0, not manually labeled. Specifically, the preset probability threshold is set to 0.5, and the formula for determining whether the label of the category is a missing label is as follows:

[0110]

[0111] where represents the missing label of the j-th sample in the i-th category, represents the classification prediction probability value of the j-th sample in the i-th category; represents the label value of the j-th sample in the i-th category.

[0112] When it means that the predicted label of the j-th sample in the i-th category is a missing label;

[0113] When it means that the predicted label of the j-th sample in the i-th category is not a missing label.

[0114] Exemplarily, the preset probability threshold can be set according to empirical values, and the present invention does not limit this here.

[0115] Step 104, after each iteration is completed, calculate the feature clustering diameter of each category according to the sample feature mapping diagram;

[0116] In the embodiment of the present invention, after each iteration is completed, calculate the feature clustering diameter of each category according to the corresponding sample feature mapping diagram in the sample image dataset.

[0117] Among them, the smaller the feature clustering diameter, the higher the feature similarity in the category, and the better the effect of model training.

[0118] In the embodiment of the present invention, step 104 may include sub-steps S31 - S32:

[0119] Sub-step S31, determine the feature vectors of the positive labels and missing labels of the categories in the sample image according to the sample feature mapping diagram;

[0120] In the embodiment of the present invention, the feature vectors of the positive labels and missing labels of the categories in the sample image can be determined according to the sample feature mapping diagram.

[0121] In the embodiment of the present invention, sub-step S31 may include sub-steps S311 - S312:

[0122] Sub-step S311, determine the feature regions of the positive labels and missing labels of the categories in the sample image according to the class activation visualization method and the threshold segmentation method;

[0123] In the embodiments of the present invention, according to the class activation visualization method and the threshold segmentation method, the feature regions with positive labels and missing labels of the categories in the sample image can be determined.

[0124] Exemplarily, the feature map can be mapped to the original sample image to obtain the importance of each category in different regions. Specifically, gradCAM can be used for calculation, and the calculation formula of the class activation image is:

[0125]

[0126]

[0127] where fM k represents the feature map of k channels, represents the weight of the feature map image of the k-th channel for category i, represents the gradient value of the output probability value of category i with respect to the pixel point on the feature map of channel k, and M and N represent the sizes of the feature map.

[0128] Using the threshold segmentation method, the class activation image of category i is binarized to separate the high-weight region and the low-weight region, where the high-weight region is the feature region of category i.

[0129] Specifically, the high-weight region may include the feature regions with positive labels and missing labels of the categories in the sample image.

[0130] Sub-step S312, generate the feature vectors with positive labels and missing labels of the categories in the sample image according to the feature regions with positive labels and missing labels of the categories in the sample image.

[0131] In the embodiments of the present invention, the feature vectors with positive labels and missing labels of the categories in the sample image can be generated according to the feature regions with positive labels and missing labels of the categories in the sample image; specifically, by calculating the image mean within the feature region on the sample feature map, the feature vectors with positive labels and missing labels of the corresponding categories can be obtained.

[0132] In the embodiments of the present invention, by separately extracting the feature regions and feature vectors of each category in the image, the interference of other categories is reduced.

[0133] Sub-step S32, use the K-means clustering algorithm for the feature vectors to calculate the feature clustering diameter of each category.

[0134] In the embodiment of the present invention, the feature vector may be passed through a K-means clustering algorithm to calculate the feature cluster diameter of each category.

[0135] For example, after completing a round of iterative training for multiple sample images in the sample image data set, the feature vectors of the positive label and the missing label of each sample are obtained, and the K-means clustering algorithm is used to divide the sample set into k categories so that the distance between each sample and the center of the class to which it belongs is minimized. The number of clusters is set to k = 1, and a feature cluster diameter is calculated for each category. The feature cluster diameter is the maximum distance between any two feature vectors in each category. The smaller the cluster diameter, the higher the feature similarity within the category, and the better the model training effect.

[0136] Step 105, when the number of iterations meets the preset number, obtaining the feature cluster diameter obtained in the previous N iterations;

[0137] In the embodiment of the present invention, when the number of iterations meets the preset number, the feature cluster diameters obtained in the previous N iterations can be obtained. Specifically, when the number of iterations is greater than the preset number, the feature cluster diameters obtained in the previous N iterations can be obtained.

[0138] Specifically, the preset number of times can be set by those skilled in the art according to actual conditions, and the present invention does not limit this herein.

[0139] Step 106, adjusting the weight parameter for the missing label according to the feature cluster diameter obtained in the previous N iterations, wherein the weight parameter is used to measure the credibility of the missing label;

[0140] In the embodiment of the present invention, the weight parameter for the missing label can be adjusted according to the feature cluster diameter obtained in the previous N iterations, wherein the weight parameter can be used to measure the credibility of the missing label.

[0141] In the embodiment of the present invention, when N is 2, the feature cluster diameters obtained in the first N iterations include the first feature cluster diameter and the second feature cluster diameter; the value of the weight parameter is 0 or 1, and step 106 may include sub-steps S41-S42:

[0142] Sub-step S41, if the label of the category in the sample image is the missing label, and the second feature cluster diameter is less than or equal to the first feature cluster diameter, then the weight parameter for the missing label is kept unchanged;

[0143] In an embodiment of the present invention, when N is 2, the feature cluster diameters obtained in the first two iterations are obtained, wherein the feature cluster diameters obtained in the first two iterations may include a first feature cluster diameter and a second feature cluster diameter; the value of the weight parameter may be 0 or 1.

[0144] If the class label in the sample image is a missing label and the second feature clustering diameter is less than or equal to the first feature clustering diameter, then the weight parameter for the missing label remains unchanged.

[0145] Sub-step S42: If the label of the class in the sample image is the missing label and the second feature clustering diameter is greater than the first feature clustering diameter, then adjust the weight parameter for the missing label to another weight parameter different from the previous weight parameter.

[0146] In the embodiment of the present invention, if the label of the class in the sample image is a missing label and the second feature clustering diameter is greater than the first feature clustering diameter, then adjust the weight parameter for the missing label to another weight parameter different from the previous weight parameter.

[0147] Specifically, if the label of the class in the sample image is a missing label, the second feature clustering diameter is greater than the first feature clustering diameter, and the previous weight parameter is 0, then adjust the weight parameter for the missing label to 1;

[0148] If the previous weight parameter is 1, then adjust the weight parameter for the missing label to 0.

[0149] Specifically, assume that the weight parameters of the class label of a sample in the previous two iterative trainings are α i and α' i , the first feature clustering diameter is D, the second feature clustering diameter is D', and the update rule of the weight parameter α'' i is as follows:

[0150]

[0151] Among them, if the label of the class in the sample image is not a missing label, then the weight parameter α'' i = 1; specifically, the weight parameter can also measure the credibility of the positive label and the negative label.

[0152] If the label of the class in the sample image is a missing label and the second feature clustering diameter is less than or equal to the first feature clustering diameter, it indicates that the clustering diameter after the previous iterative training has decreased and the clustering performance has improved. Therefore, the weight parameter remains unchanged;

[0153] If the label of the class in the sample image is a missing label and the second feature clustering diameter is greater than the first feature clustering diameter, it indicates that the clustering diameter after the previous iterative training has increased and the clustering performance has decreased. Therefore, adjust the weight parameter for the missing label to another weight parameter different from the previous weight parameter.

[0154] In the embodiment of the present invention, it is necessary to update the weight parameters according to the feature clustering diameter after the first two iterative trainings. Therefore, the weight parameters for missing labels can be updated starting from the 3rd iteration.

[0155] Specifically, in the first iteration of the dataset, the weight parameter for missing labels is set to 1, and in the second iteration of the dataset, the weight parameter for missing labels is set to 0.

[0156] Step 107: Determine the classification loss value of the category in the sample image according to the weight parameter and the loss function, so as to adjust the label of the category in the sample image.

[0157] In the embodiment of the present invention, the classification loss value of the category in the sample image can be determined according to the weight parameter and the loss function, and the label of the category in the sample image can be adjusted.

[0158] In the embodiment of the present invention, step 107 may include sub-steps S51 - S54:

[0159] Sub-step S51: If the weight parameter is 1, select the first loss function formula for calculation to obtain the first classification loss value;

[0160] Sub-step S52: According to the first classification loss value, adjust the label of the type in the sample image from a missing label to a negative label;

[0161] Sub-step S53: If the weight parameter is 0, select the second loss function formula for calculation to obtain the second classification loss value;

[0162] Sub-step S54: According to the second classification loss value, adjust the label of the type in the sample image from a missing label to a positive label;

[0163] Among them, the first loss function formula is:

[0164]

[0165] The second loss function formula is:

[0166]

[0167] Among them, i represents the i-th category in the sample image, c represents the number of categories in the sample image; Y is the label value; Y i = 1 is a positive label; Y i = 0 is a negative label or a missing label; y i represents the output probability value of the i-th category in the sample image; L 2 represents the first classification loss value; L 2 represents the second classification loss value.

[0168] In an embodiment of the present invention, if the weight parameter is 1, the first loss function formula is selected for calculation to obtain the first classification loss value; according to the first classification loss value, the label of the type in the sample image is adjusted from a missing label to a negative label; if the weight parameter is 0, the second loss function formula is selected for calculation to obtain the second classification loss value; according to the second classification loss value, the label of the type in the sample image is adjusted from a missing label to a positive label.

[0169] Specifically, the formula of the loss function is:

[0170] L = L 1 + L 2

[0171] According to the magnitude of the weight parameter, the classification loss value is calculated to determine whether the missing label exists;

[0172] When the weight parameter is 1, it is determined that the missing label does not exist, and the label of the type in the sample image is adjusted from a missing label to a negative label;

[0173] When the weight parameter is 0, it is determined that the missing label exists, and the label of the type in the sample image is adjusted from a missing label to a positive label.

[0174] In an embodiment of the present invention, the model parameters can also be updated through the loss function and the weight parameter, where the model parameters can include model weights and bias parameters; specifically, the gradient of the model weights and bias parameters can be calculated by using the model optimization method of stochastic gradient descent to update the model weights and bias parameters.

[0175] In an embodiment of the present invention, after each iteration is completed, the loss function, model weights, and bias parameters are updated.

[0176] The above steps are the training process of the multi-label classification model, and the standard for the completion of model training can be that the number of iterations of all images in the image dataset is greater than a preset iteration threshold.

[0177] Exemplarily, if the preset iteration threshold is too small, the training is fast but the model classification performance is poor; if the preset iteration threshold is too large, the classification performance is good but the training is slow and the generalization ability is poor. Generally, for a large model, it is 10 - 20 times. Specifically, the magnitude of the preset iteration threshold can be selected by those skilled in the art according to the actual situation, and the present invention does not make a limitation here.

[0178] To enable those skilled in the art to better understand the usage process of the model in the embodiment of the present invention, the following uses a complete example to illustrate. Specifically, it can include:

[0179] Step 1: Obtain an image from the image dataset, preprocess the image to generate a first image; the preprocessing includes methods such as brightness normalization, noise filtering, scale normalization, cropping, and flipping.

[0180] Step 2: Input the first image into a pre-trained multi-label classification model, generate a feature map through the convolutional hidden layer of the multi-label classification model, and input the feature map into the fully connected hidden layer of the multi-label classification model to output a prediction probability vector, where the prediction probability vector includes the classification prediction probability values for each category;

[0181] Among them, the multi-label classification model includes a convolutional hidden layer and a fully connected hidden layer.

[0182] Step 3: If the classification prediction probability value is greater than the preset probability threshold, it is determined that this category exists (positive label) in the first image

[0183] To enable those skilled in the art to better understand the process of label classification in the embodiments of the present invention, the following uses a complete example for illustration. Refer to Figure 3 , which is a flowchart of a model training provided by an embodiment of the present invention, and specifically may include the following steps:

[0184] Step 301: Perform forward calculation on multiple images in the sample image dataset to obtain corresponding sample prediction probability vectors and sample feature maps, and screen out missing labels; specifically, input the label dataset into the multi-label classification model, and sequentially input multiple sample images in the sample image dataset into the multi-label classification model to obtain sample prediction probability vectors and sample feature maps corresponding to the multiple sample images one by one; according to the sample prediction probability vectors, determine the labels for each category in the sample images; the labels include positive labels, negative labels, and missing labels.

[0185] Step 302: Adopt a class activation visualization method and a threshold segmentation method to extract the feature regions and feature vectors of positive labels and missing labels; specifically, according to the sample feature maps, determine the feature vectors of the labels for the categories in the sample images that are positive labels and missing labels.

[0186] Step 303: Update the weight parameters when the number of iterations meets the preset number of times; specifically, update the weight parameters by obtaining the feature clustering diameters obtained in the previous N iterations.

[0187] Step 304: Update the weights and bias parameters of the model according to the updated weight parameters;

[0188] Step 305: After completing this iteration, calculate the feature clustering diameter for each category according to the feature vectors;

[0189] Step 306, until the number of iterations is greater than a preset iteration threshold, and the training is completed.

[0190] In an embodiment of the present invention, a sample image data set and a label data set are obtained, and the following steps are iteratively executed: input the label data set into a multi-label classification model, and sequentially input multiple sample images in the sample image data set into the multi-label classification model to obtain sample prediction probability vectors and sample feature maps corresponding to the multiple sample images one by one; determine the labels of each category in the sample image according to the sample prediction probability vectors; after each iteration is completed, calculate the feature clustering diameter of each category according to the sample feature map; when the number of iterations meets a preset number, obtain the feature clustering diameters obtained in the previous N iterations; adjust the weight parameters for missing labels according to the feature clustering diameters obtained in the previous N iterations, and determine the classification loss values of the categories in the sample image according to the weight parameters and the loss function, so as to adjust the labels of the categories in the sample image. The present invention sets a weight parameter in the loss function, and adjusts the weight parameters for missing labels by obtaining the feature clustering diameters obtained in the previous N iterations. Through the adjusted weight parameters and the loss function, the labels of the categories in the sample image are adjusted according to the training effect, so as to realize the excavation of potential label information as much as possible.

[0191] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequences, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0192] Refer to Figure 4 , which shows a structural block diagram of a device for label classification provided by an embodiment of the present invention, and specifically may include the following modules:

[0193] A data set acquisition module 401, configured to acquire a sample image data set and a label data set; the sample image data set includes multiple sample images;

[0194] A vector and feature map generation module 402, configured to iteratively execute the following steps: input the label data set into a multi-label classification model, and sequentially input multiple sample images in the sample image data set into the multi-label classification model to obtain sample prediction probability vectors and sample feature maps corresponding to the multiple sample images one by one;

[0195] A label determination module 403, configured to determine the labels of each category in the sample image according to the sample prediction probability vectors; the labels include positive labels, negative labels, and missing labels;

[0196] The clustering diameter calculation module 404 is configured to calculate the feature clustering diameter of each category according to the sample feature mapping graph after each iteration is completed.

[0197] The clustering diameter obtaining module 405 is configured to obtain the feature clustering diameters obtained in the previous N iterations when the number of iterations meets a preset number of times.

[0198] The weight parameter adjustment module 406 is configured to adjust the weight parameter for the missing label according to the feature clustering diameters obtained in the previous N iterations, and the weight parameter is used to measure the credibility of the missing label.

[0199] The label adjustment module 407 is configured to determine the classification loss value of the category in the sample image according to the weight parameter and the loss function, so as to adjust the label of the category in the sample image.

[0200] In an embodiment of the present invention, the clustering diameter calculation module includes:

[0201] The feature vector determination sub-module is configured to determine the feature vectors of the positive label and the missing label of the category in the sample image according to the sample feature mapping graph.

[0202] The clustering diameter calculation sub-module is configured to calculate the feature clustering diameter of each category by using the K-means clustering algorithm for the feature vectors.

[0203] In an embodiment of the present invention, the multi-label classification model includes a convolutional hidden layer and a fully connected hidden layer, and the vector and feature map generation module includes:

[0204] The vector and feature map generation sub-module is configured to input the label data set into the convolutional hidden layer of the multi-label classification model, and sequentially input multiple sample images in the sample image data set into the convolutional hidden layer of the multi-label classification model, generate the sample feature mapping graph corresponding to each sample image through the convolutional hidden layer of the multi-label classification model, and input the sample feature mapping graph into the fully connected hidden layer of the multi-label classification model to output the sample prediction probability vector corresponding to each sample image, and the sample prediction probability vector includes the classification prediction probability values of each category.

[0205] In an embodiment of the present invention, the label determination module includes:

[0206] The label determination sub-module is configured to determine that the label of the category in the sample image is a missing label if the classification prediction probability value is greater than a preset probability threshold and the label value of the category is 0.

[0207] In an embodiment of the present invention, when N is 2, the characteristic clustering diameters obtained in the first N iterations include a first characteristic clustering diameter and a second characteristic clustering diameter; the value of the weight parameter is 0 or 1, and the weight parameter adjustment module includes:

[0208] A first weight parameter adjustment sub-module, configured to keep the weight parameter for the missing label unchanged if the label of the category in the sample image is the missing label and the second characteristic clustering diameter is less than or equal to the first characteristic clustering diameter;

[0209] A second weight parameter adjustment sub-module, configured to adjust the weight parameter for the missing label to another weight parameter different from the previous weight parameter if the label of the category in the sample image is the missing label and the second characteristic clustering diameter is greater than the first characteristic clustering diameter.

[0210] In an embodiment of the present invention, the feature vector determination sub-module includes:

[0211] A region determination unit, configured to determine the feature regions of the positive label and the missing label of the category in the sample image according to the class activation visualization method and the threshold segmentation method;

[0212] A feature vector generation unit, configured to generate the feature vectors of the positive label and the missing label of the category in the sample image according to the feature regions of the positive label and the missing label of the category in the sample image.

[0213] In an embodiment of the present invention, the label adjustment module includes:

[0214] A first loss value calculation sub-module, configured to select a first loss function formula for calculation to obtain a first classification loss value if the weight parameter is 1;

[0215] A first label adjustment sub-module, configured to adjust the label of the type in the sample image from the missing label to the negative label according to the first classification loss value;

[0216] A second loss value calculation sub-module, configured to select a second loss function formula for calculation to obtain a second classification loss value if the weight parameter is 0;

[0217] A second label adjustment sub-module, configured to adjust the label of the type in the sample image from the missing label to the positive label according to the second classification loss value;

[0218] Among them, the first loss function formula is:

[0219]

[0220] The second loss function formula is:

[0221]

[0222] Among them, i represents the i-th category in the sample image, c represents the number of categories of the sample image; Y is the label value; Y i = 1 is a positive label; Y i = 0 is a negative label or a missing label; y i represents the output probability value of the i-th category in the sample image; L 1 represents the first classification loss value; L 2 represents the second classification loss value.

[0223] In an embodiment of the present invention, a sample image dataset and a label dataset are obtained, and the following steps are iteratively executed: inputting the label dataset into a multi-label classification model, and sequentially inputting multiple sample images in the sample image dataset into the multi-label classification model to obtain a sample prediction probability vector and a sample feature map corresponding to each of the multiple sample images; determining the labels of each category in the sample image according to the sample prediction probability vector; after each iteration is completed, calculating the feature clustering diameter of each category according to the sample feature map; when the number of iterations meets a preset number, obtaining the feature clustering diameters obtained in the previous N iterations; adjusting the weight parameter for the missing label according to the feature clustering diameters obtained in the previous N iterations, and determining the classification loss value of the category in the sample image according to the weight parameter and the loss function, so as to adjust the label of the category in the sample image. The present invention adjusts the weight parameter for the missing label by setting a weight parameter in the loss function and obtaining the feature clustering diameters obtained in the previous N iterations, and realizes adjusting the label of the category in the sample image according to the training effect through the adjusted weight parameter and the loss function, so as to realize mining potential label information as much as possible.

[0224] For the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the related parts, refer to the partial description of the method embodiment.

[0225] Referring to Figure 5 , there is shown an electronic device 50 provided by an embodiment of the present invention, including:

[0226] including a processor 501, a memory 502, and a computer program 5021 stored on the memory and capable of running on the processor. When the computer program 5021 is executed by the processor, it implements each process of the method embodiment of the above label classification and can achieve the same technical effect. To avoid repetition, it will not be described here again.

[0227] Referring to Figure 6, which shows a computer-readable storage medium 60 provided by an embodiment of the present invention. A computer program 601 is stored on the computer-readable storage medium. When the computer program 601 is executed by a processor, it implements each process of the method embodiment for label classification described above and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0228] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0229] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0230] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0231] These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing terminal devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0232] These computer program instructions can also be loaded onto the computer or other programmable data processing terminal devices, so that a series of operation steps are executed on the computer or other programmable terminal devices to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable terminal devices provide for implementing the functions in the processFigure 1 one process or multiple processes and / or blocks Figure 1 steps of the functions specified in one block or multiple blocks.

[0233] Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0234] Finally, it should also be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0235] The above has introduced in detail a method for label classification, a device for label classification, an electronic device and a computer-readable storage medium provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for label classification, characterized in that, the method includes: Obtaining a sample image data set and a label data set; the sample image data set includes multiple sample images; Iteratively execute the following steps: Input the label data set into a multi-label classification model, and sequentially input multiple sample images in the sample image data set into the multi-label classification model to obtain a sample prediction probability vector and a sample feature map corresponding to each of the multiple sample images; Determine the labels of each category in the sample image according to the sample prediction probability vector; the labels include positive labels, negative labels, and missing labels; After each iteration is completed, calculate the feature clustering diameter of each category according to the sample feature map; When the number of iterations meets a preset number, obtain the feature clustering diameters obtained in the previous N iterations; Adjust the weight parameter for the missing label according to the feature clustering diameters obtained in the previous N iterations, and the weight parameter is used to measure the credibility of the missing label; Determine the classification loss value of the category in the sample image according to the weight parameter and the loss function to adjust the label of the category in the sample image; The calculating the feature clustering diameter of each category according to the sample feature map includes: Determine the feature vectors of the labels of the categories in the sample image as positive labels and missing labels according to the sample feature map; Pass the feature vectors through the K-means clustering algorithm to calculate the feature clustering diameter of each category.

2. The method according to claim 1, characterized in that, the multi-label classification model includes a convolutional hidden layer and a fully connected hidden layer. The inputting the label data set into the multi-label classification model and sequentially inputting multiple sample images in the sample image data set into the multi-label classification model to obtain a sample prediction probability vector and a sample feature map corresponding to each of the multiple sample images includes: Input the label data set into the convolutional hidden layer of the multi-label classification model, and sequentially input multiple sample images in the sample image data set into the convolutional hidden layer of the multi-label classification model. Generate the sample feature map corresponding to the sample image through the convolutional hidden layer of the multi-label classification model, and input the sample feature map into the fully connected hidden layer of the multi-label classification model to output the sample prediction probability vector corresponding to the sample image, and the sample prediction probability vector includes the classification prediction probability values of each category.

3. The method according to claim 2, characterized in that, the determining the labels of each category in the sample image according to the sample prediction probability vector includes: If the classification prediction probability value is greater than a preset probability threshold and the label value of the category is 0, then determine the label of the category in the sample image as a missing label.

4. The method according to claim 1, characterized in that, When N is 2, the feature clustering diameters obtained in the previous N iterations include a first feature clustering diameter and a second feature clustering diameter; The value of the weight parameter is 0 or 1. Adjusting the weight parameter for the missing label according to the feature clustering diameter obtained from the previous N iterations includes: If the label of the category in the sample image is the missing label and the second feature clustering diameter is less than or equal to the first feature clustering diameter, the weight parameter for the missing label remains unchanged; If the label of the category in the sample image is the missing label and the second feature clustering diameter is greater than the first feature clustering diameter, the weight parameter for the missing label is adjusted to another weight parameter different from the previous weight parameter.

5. The method according to claim 1, wherein, Generating the feature vectors of the positive label and the missing label for the category in the sample image according to the sample feature map includes: Determining the feature regions of the positive label and the missing label for the category in the sample image according to the class activation visualization method and the threshold segmentation method; Generating the feature vectors of the positive label and the missing label for the category in the sample image according to the feature regions of the positive label and the missing label for the category in the sample image.

6. The method according to claim 4, wherein, Determining the classification loss value of the sample image according to the weight parameter and the loss function to adjust the label of the sample image includes: If the weight parameter is 1, select the first loss function formula for calculation to obtain the first classification loss value; According to the first classification loss value, adjust the label of the category in the sample image from the missing label to the negative label; If the weight parameter is 0, select the second loss function formula for calculation to obtain the second classification loss value; According to the second classification loss value, adjust the label of the category in the sample image from the missing label to the positive label; wherein, the first loss function formula is: The second loss function formula is: Among them, i represents the i-th category in the sample image, c represents the number of categories of the sample image; Y is the label value; 1 is the positive label; 0 is the negative label or missing label; represents the output probability value of the i-th category in the sample image; represents the first classification loss value; represents the second classification loss value.

7. A device for label classification, wherein, The device includes: A dataset acquisition module for acquiring a sample image dataset and a label dataset; the sample image dataset includes multiple sample images; A vector and feature map generation module for iteratively performing the following steps: inputting the label dataset into a multi-label classification model, and sequentially inputting multiple sample images in the sample image dataset into the multi-label classification model to obtain sample prediction probability vectors and sample feature maps corresponding to the multiple sample images one by one; A label determination module for determining the labels of each category in the sample image according to the sample prediction probability vector; the labels include positive labels, negative labels and missing labels; A clustering diameter calculation module for calculating the feature clustering diameter of each category according to the sample feature map after each iteration is completed; A clustering diameter acquisition module for acquiring the feature clustering diameters obtained from the previous N iterations when the number of iterations meets a preset number; A weight parameter adjustment module for adjusting the weight parameter for the missing label according to the feature clustering diameters obtained from the previous N iterations, and the weight parameter is used to measure the credibility of the missing label; A label adjustment module, configured to determine a classification loss value of a category in the sample image according to the weight parameter and the loss function, so as to adjust the label of the category in the sample image; The clustering diameter calculation module includes: A feature vector determination sub-module, configured to determine feature vectors of positive labels and missing labels of the category in the sample image according to the sample feature mapping graph; A clustering diameter calculation sub-module, configured to calculate a feature clustering diameter of each category by using the K-means clustering algorithm for the feature vectors.

8. An electronic device, characterized in that it includes: A processor, a memory, and a computer program stored on the memory and capable of running on the processor, where when the computer program is executed by the processor, the steps of the label classification method according to any one of claims 1-6 are implemented.

9. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the label classification method according to any one of claims 1-6 are implemented.

Images