Image classification model training method, image classification method, electronic device and storage medium

Abstract

Embodiments of the present invention provide an image classification model training method, an image classification method, an electronic device, and a storage medium, and relate to the field of computer vision. Obtain multiple image samples that have been annotated with type labels. For each image sample, input the image sample into an image classification model, and in any convolutional layer in the image classification model, extract multiple feature images with different channels from the image sample. Among them, different channels represent different image features of the image sample. Using the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample, calculate the loss value of the image classification model, and adjust the parameters of the image classification model based on the loss value. Repeat the above steps so that the expected training goal can be achieved. In this way, since the number of image samples input to the image classification model is not increased, each model training takes less time.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image classification model training method, an image classification method, an electronic device and a storage medium. Background Art

[0002] As an important learning method, self-supervised learning has been widely used in neural networks. Based on a large amount of unlabeled data, through a semi-automatic process, self-supervised learning generates pseudo labels to train the network, making it possible for the network to extract features without manual labeling. Due to the simplicity and power of the self-supervised learning method, researchers have applied it to other fields of deep learning.

[0003] However, when training image classification models based on self-supervised learning methods, in order to ensure the training effect of the final network model, a large number of data sets are generally used, which makes it time-consuming, especially when the network of the image classification model is complex. Summary of the invention

[0004] The objectives of the present invention include, for example, providing an image classification model training method, an image classification method, an electronic device and a storage medium, which can reduce the time consumption during model training.

[0005] The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides an image classification model training method, which is applied to an electronic device, and the method comprises:

[0007] Get multiple image samples with labeled type labels;

[0008] For each of the image samples, the image sample is input into an image classification model, and in any convolutional layer in the image classification model, multiple feature images are extracted from the image sample, and the type of the image sample and the type labels of the multiple feature images are predicted; wherein different feature images represent image samples lacking different channels, and different channels represent different image features of image samples;

[0009] The loss value of the image classification model is calculated using the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample;

[0010] Determine whether the loss value reaches the expected training target;

[0011] If not, adjusting the parameters of the image classification model based on the loss value, and returning to the step of inputting the image sample into the image classification model for each image sample until the latest loss value reaches the expected training target;

[0012] If yes, the training is terminated and the trained image classification model is obtained.

[0013] In an optional embodiment, the step of extracting a plurality of feature images from the image sample in any convolutional layer in the image classification model includes:

[0014] Based on the multiple channels included in the image sample, multiple channel groups are obtained; wherein the channels included in each channel group are different, and the channels missing from each channel group constitute the multiple channels included in the image sample;

[0015] One channel group among the multiple channel groups included in the image sample is discarded respectively to obtain multiple feature images including different channel groups.

[0016] In an optional embodiment, the step of respectively discarding one channel group from the multiple channel groups included in the image sample to obtain a plurality of feature images including different channel groups includes:

[0017] Obtain channels included in a target channel group from multiple channel groups and determine them as target channels;

[0018] Generate a binary mask; wherein the binary mask contains a matrix corresponding to each of the channels, and the value of each element in the matrix is ​​to be determined;

[0019] In the binary mask, the matrix corresponding to the target channel is set to a zero matrix, and the value of each element in the matrix that does not correspond to the target channel is set to 1 to obtain the target binary mask;

[0020] multiplying the target binary mask by the image sample to obtain a feature image;

[0021] Repeat the above steps until all feature images of the image sample are obtained.

[0022] In an optional implementation, after the step of extracting a plurality of feature images from the image sample, the method further includes:

[0023] Based on the type labels that have been labeled for the image samples, labeling the type labels of each feature image extracted from the image samples;

[0024] According to the channels missing from the feature images, label the channel labels of the image samples and each of the feature images and the channel labels;

[0025] The step of predicting the type of the image sample and the type labels of the multiple feature images includes:

[0026] Based on the joint classifier, predict the type label and channel label of the image sample and the type label and channel label of the feature image;

[0027] Based on the joint classifier, predict the type label of the image sample and the type label of the feature image;

[0028] Based on a single classifier, predict the type label of the image sample;

[0029] The step of calculating the loss value of the image classification model by using the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample, comprises:

[0030] Calculate a joint loss value of the image classification model using the predicted type label of the image sample, the channel label of the image sample, the type label of the feature image, the channel label of the feature image, the annotated type label of the image sample, the annotated channel label of the image sample, the annotated type label of the feature image, and the annotated channel label of the feature image;

[0031] The comprehensive loss value of the image classification model is calculated using the type labels of the image samples predicted by the joint classifier and the type labels of the feature images, the type labels of the image samples predicted by the single classifier, the type labels of the image samples that have been annotated, and the type labels of the feature images that have been annotated;

[0032] Using the predicted type label of the image sample and the labeled type label of the image sample, a single loss value of the image classification model is calculated;

[0033] The loss value is obtained according to the combined loss value, the single loss value and the comprehensive loss value.

[0034] In an optional embodiment, the step of predicting the type label and channel label of the image sample and the type label and channel label of the feature image based on the joint classifier includes:

[0035] Based on the joint classifier, determine the joint probabilities of the image samples being various joint labels, and the joint probabilities of each feature image being various joint labels; wherein the joint label is composed of a type label and a channel label, and different types of the channel labels are different joint labels;

[0036] The step of calculating the joint loss value of the image classification model by using the predicted type label of the image sample, the channel label of the image sample, the type label of the feature image, the channel label of the feature image, the type label of the image sample, the channel label of the image sample, the type label of the feature image, and the channel label of the feature image includes:

[0037] Based on multiple joint probabilities of image samples, type labels and channel labels annotated by the image samples, multiple unions of feature images, and type labels and channel labels annotated by the feature images, a joint loss value of the image classification model is calculated.

[0038] In an optional implementation, the step of predicting the type label of the image sample based on a single classifier includes:

[0039] Based on the single classifier, determining the self-distillation probabilities of the image samples being respectively various types of labels;

[0040] The step of predicting the type label of the image sample and the type label of the feature image based on the joint classifier includes:

[0041] Based on the joint classifier, determining the single probabilities that the image samples are respectively various types of labels, and the single probabilities that the feature images are respectively various types of labels; based on multiple single probabilities of the image samples, and multiple single probabilities of the feature images, determining the aggregated probabilities that the image samples are respectively various types of labels;

[0042] The step of calculating the comprehensive loss value of the image classification model by using the type labels of the image samples predicted by the joint classifier and the type labels of the feature images, the type labels of the image samples predicted by the single classifier, the type labels of the image samples that have been annotated, and the type labels of the feature images that have been annotated includes:

[0043] The comprehensive loss value is calculated based on multiple aggregation probabilities of each of the image samples, multiple self-distillation probabilities of each of the image samples, and the type labels that have been annotated with the image samples.

[0044] In an optional implementation, the step of predicting the type label of the image sample based on a single classifier includes:

[0045] Based on a single classifier, determining the self-distillation probabilities of the image samples being respectively various types of labels;

[0046] The step of calculating a single loss value of the image classification model using the predicted type label of the image sample and the labeled type label of the image sample comprises:

[0047] The single loss value is calculated based on multiple self-distillation probabilities of each of the image samples and the type labels that have been annotated with the image samples.

[0048] In a second aspect, the present invention provides an image classification method, applied to an electronic device, the method comprising:

[0049] Acquire the image to be detected;

[0050] The image to be detected is input into an image classification model obtained by using the image classification model training method as described in any one of the aforementioned implementation modes -7 to obtain a predicted type of the image to be detected.

[0051] In a third aspect, the present invention provides an electronic device comprising: a memory, a processor, and a program stored on the memory and executable on the processor, wherein when the program is executed by the processor, the program implements the image classification model training method as described in any one of the aforementioned embodiments or the various steps of the image classification method as described in the aforementioned embodiments.

[0052] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, executes the image classification model training method of any one of the aforementioned embodiments or the various steps of the image classification method of the aforementioned embodiments.

[0053] The beneficial effects of the embodiments of the present invention include, for example: after obtaining an image sample with an annotated type label, and inputting the image sample into an image classification model. In any convolutional layer of the image classification model, multiple feature images with different channels are extracted from the image sample. So that the loss value of the image classification model can be determined based on the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample. Subsequently, the network parameters of the image classification model can be repeatedly adjusted based on the loss value to obtain a successfully trained image classification model. Among them, different channels represent different image features.

[0054] Since the feature image is only extracted in the convolutional layer, the data set of the input image classification model is not expanded, that is, the number of images of the input image classification model is not increased, so that the time for each model training and learning can be reduced, that is, less time can be wasted. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A schematic structural diagram of an electronic device is provided according to an embodiment of the present invention.

[0057] Figure 2 One of the flow charts of a method for training a graph classification model is provided for an embodiment of the present invention.

[0058] Figure 3 A second flowchart of a method for training a graph classification model is provided for an embodiment of the present invention.

[0059] Figure 4 A schematic diagram of discarding feature images of different channels is provided for an embodiment of the present invention.

[0060] Figure 5 A schematic diagram of feature transformation provided by an embodiment of the present invention.

[0061] Figure 6 The third flowchart of a method for training a graphics classification model provided by an embodiment of the present invention.

[0062] Figure 7 A schematic diagram of a network structure for model training provided in an embodiment of the present invention.

[0063] Figure 8 A fourth flowchart of a method for training a graphics classification model provided in an embodiment of the present invention.

[0064] Fig. 9 A fifth flow chart of a method for training a graphics classification model provided in an embodiment of the present invention.

[0065] Fig.10 FIG6 is a flowchart of a method for training a graphics classification model provided in an embodiment of the present invention.

[0066] Fig.11 A schematic diagram of functional modules of an image classification model training device is provided for an embodiment of the present invention.

[0067] Icon: 100 - electronic device; 110 - communication unit; 120 - memory; 130 - processor;

[0068] 200 - image classification model training device; 210 - training sample acquisition module; 220 - image type prediction module; 230 - loss value calculation module; 240 - training progress judgment module. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0070] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0072] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0073] Among existing self-supervised learning technologies, for example, SLA is a technology that applies self-supervised learning methods to supervised learning, which enhances supervised learning by identifying proxy tasks for image transformations (such as image rotation). However, the SLA algorithm requires generating multiple transformed images for each image, which is equivalent to expanding the dataset several times. This image-level proxy task will inevitably incur a lot of time consumption and memory overhead.

[0074] Based on the problems existing in existing self-supervised learning technology, please refer to Figure 1 , is a schematic diagram of a structure of an electronic device 100 according to an embodiment of the present invention. The electronic device 100 includes a memory 120 , a processor 130 and a communication unit 110 .

[0075] The memory 120, the processor 130 and the communication unit 110 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The image classification model training method can be stored in the memory 120 in the form of software or firmware or a software function module solidified in the operating system (OS) of the electronic device 100. The processor 130 is used to execute the executable module stored in the memory 120.

[0076] The memory 120 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 120 is used to store a program, and the processor 130 executes the program after receiving an execution instruction. The communication unit 110 is used to communicate with an external system.

[0077] The processor 130 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0078] Based on the problem of long model training time caused by the large expansion of data sets in the prior art, an embodiment of the present invention provides a flow chart of a method for training a graph classification model, such as Figure 2 As shown. It can reduce the time consumption of image classification model training while ensuring the classification accuracy of the image classification model. Its specific implementation method is as follows:

[0079] Step S101: Obtain multiple image samples that have been annotated with type labels.

[0080] Step S102: for each image sample, input the image sample into the image classification model, extract multiple feature images from the image sample in any convolution layer in the image classification model, and predict the type of the image sample and the type labels of the multiple feature images.

[0081] Among them, different feature images represent image samples lacking different channels, and different channels represent different image features of image samples;

[0082] Step S103: Calculate the loss value of the image classification model using the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample.

[0083] Step S104: Determine whether the loss value reaches the expected training target. If not, execute step S105. If yes, execute step S106.

[0084] Step S105: Adjust the parameters of the image classification model based on the loss value, and return to execute steps S102 to S104;

[0085] Step S106, end the training and obtain the trained image classification model.

[0086] After obtaining multiple image samples with annotated type labels, the multiple image samples are input into the image classification model. In the convolution layer of the image classification model, multiple feature images can be extracted from one image sample. Among them, different feature images lack different channels, and each channel can represent different information of the image sample. Using the predicted type labels of the image samples and the type labels of the feature images, as well as the annotated type labels of the image samples, the loss value of the image classification model is determined, and the parameters of the image classification model are adjusted according to the loss value. And return to the execution step Return to execute steps S102 to S104 until the latest loss value reaches the expected training target.

[0087] Since the feature image is only extracted in the convolution layer, the data set of the input image classification model is not expanded, that is, the number of pictures of the input image classification model is increased, so that the time for each model training and learning can be reduced, that is, less time can be wasted.

[0088] In addition, since there is no expansion of the input dataset, less memory resources are consumed.

[0089] Exemplarily, the training method provided in the present method embodiment can be trained on the basis of any network model to obtain an image classification model. For example, the network model can be ResNet, Wide ResNet1, DenseNet, PyramidNet, etc.

[0090] Exemplarily, any convolution layer may be used, or multiple convolution layers may be selected to extract feature images according to actual application requirements.

[0091] For example, there is a training image set of M image samples Among them, x i Represents image samples, and M represents the number of image samples. Then the expected training target formula can be expressed by the following formula: Among them, θ represents the parameters of the network model, and the loss function loss(·) represents the distance between the network output and the actual label. Generally, the cross entropy loss L is used. CE As the loss function to train the network. The loss function loss(·) can be expressed as follows: The value of K represents the number of feature images plus the number of image samples. Indicates the difference between the predicted type label of an image and the labeled type label.

[0092] Exemplarily, after step S102, the type label of each feature image extracted from the image sample can be labeled based on the type label that has been labeled with the image sample. According to the channels that the feature image lacks, the channel label of the image sample and each feature image and the channel label are labeled. This allows the loss value to be determined later based on the type label and the channel label.

[0093] How feature images are extracted from image samples can be found in Figure 3 , which is a second flow chart of a method for training a graphic classification model provided by an embodiment of the present invention. In step S02, in any convolutional layer in the image classification model, the step of extracting multiple feature images from the image sample can be implemented as follows:

[0094] Step S102-1: obtain multiple channel groups based on multiple channels included in the image sample.

[0095] Each channel group contains different channels.

[0096] Exemplarily, one channel group among the multiple channel groups included in the image sample can be discarded respectively to obtain multiple feature images including different channel groups. The specific implementation method can be achieved through the following steps:

[0097] Step S102 - 2 : Acquire channels included in a target channel group from multiple channel groups, and determine them as target channels.

[0098] Step S102 - 3 : In the binary mask, the matrix corresponding to the target channel is set to a zero matrix, and the value of each element in the matrix not corresponding to the target channel is set to 1, so as to obtain a target binary mask.

[0099] The binary mask contains a matrix corresponding to each channel, and the value of each element in the matrix is ​​to be determined.

[0100] Step S102-4: multiply the target binary mask and the image sample to obtain a feature image.

[0101] Repeat the above steps S102 - 2 to S102 - 4 until all feature images of the image sample are obtained.

[0102] When images are passed into the network model for training, they need to go through multiple convolutions. The convolution layer can interact between channels and then generate new channels in the next layer. Therefore, by considering discarding different channels for feature transformation, the accuracy of the image classification model after training can be ensured without increasing the data set, that is, without increasing the training time. And by retaining most of the channels for each feature image, the integrity of most semantic feature information is guaranteed, so that the original classification task can be carried out smoothly.

[0103] It should be noted that an image has multiple channels, and different channels correspond to different aspects of semantic feature information. For example, an RGB color image contains red, green, and blue channels, while a grayscale image with a single-color channel (such as the red channel) can be regarded as a feature image of a channel, which can indicate certain semantic features. By discarding different channels to perform feature transformation, the model can learn more subtle semantic features by distinguishing which channels are missing during repeated training, that is, it has a deeper understanding of the entire image.

[0104] It should be explained that each image can be considered as consisting of three dimensions, the first dimension is height, the second dimension is width, and the third dimension is channel. Each channel is composed of a two-dimensional matrix, the length and width of which are the sizes of the first dimension and the second dimension respectively, and the value of each pixel corresponding to the element in the two-dimensional matrix is ​​the pixel value corresponding to each pixel in the channel.

[0105] To facilitate understanding, channels can identify different semantic feature information, which can be combined Figure 4For reference, a schematic diagram of a feature image of discarding different channels provided by an embodiment of the present invention is provided. From left to right, the images in each column are respectively the original image, the CAMs of the baseline model, the CAMs of model one, the CAMs of model two, the CAMs of model three, and the CAMs of model four. In each row, the ResNet-18 model trained on the Stanford Dogs dataset is used. Based on the original image of the row, the feature images formed by 4 groups of different channels are discarded on the last layer of the convolution layer of the model (the images corresponding to the CAMs of model one, model two, model three, and model four, respectively). The image samples corresponding to the annotations of the baseline model CAMs in the figure are image samples without discarding any channels. Since different channels are discarded, each feature image focuses on different semantic information, that is, the feature image is different from the part highlighted or ignored by the CAMs of the baseline model. Therefore, the feature transformation, that is, discarding different channels of the image sample, can be used as a supervisory signal to construct a proxy task in self-supervision to explore deeper visual representations and improve classification performance.

[0106] For example, if the size of an image sample is height×width×n channels , randomly divide its channels into k groups. Each channel group contains n channels / k channels. Each feature image can be transformed by mathematical operations, that is, multiple feature images are extracted from the image sample. Among them, the feature image can be obtained by the following formula:

[0107] Among them, t j Characterization feature transformation, M j It represents a binary mask for mathematical operation, so that multiple feature images can be obtained from the image sample based on the binary mask.

[0108] To facilitate understanding of the above formula, the following is a brief description of how to generate a feature image. First, determine the channels that need to be discarded to generate the feature image, and then generate height×width×n channels The binary mask M j Determine the target channel included in the channel group that needs to be discarded to generate the feature image, set the matrix corresponding to the target channel to a zero matrix, and set the elements in the matrices that are not corresponding to the target channel to 1. j Multiply to get the feature image In this way, other feature images can be obtained through the above method.

[0109] To understand the process of discarding channels, Figure 5A schematic diagram of feature transformation provided by an embodiment of the present invention. Figure 5 As shown, f can be regarded as an image sample, They are the feature images after discarding different channels. Each square in the figure represents a channel group. The square in the dotted box represents the discarded channel group. Figure 5 The four feature transformations in , just discard all channels in the image sample once. It should be noted that The discarded channels are not adjacent, i.e. The discarded channels can be Figure 5 middle The channels discarded in , correspondingly, The discarded channels can be Figure 5 middle The channel that is discarded.

[0110] Exemplarily, the determination of the channel group can be set according to the actual application situation to discard those channels when determining the feature image. For example, multiple channel groups can be determined by randomly dividing all channels, and the channels contained in all channel groups are different. Channels can also be divided artificially to determine multiple channel groups. It can also be preset to discard only some channels, and multiple channel groups are determined based on the discarded channels.

[0111] For example, by randomly dividing all channels and determining multiple channel groups in a manner where all channel groups contain different channels, it can be ensured that all channels are only discarded once, so that the performance is better in practical applications. When the program is allowed to discard channels, the following formula can be used: Let the program know that all channels will only be discarded once. Where f is the image sample, is the feature image extracted from the image sample, j is the channel group number, and k is the total number of channel groups.

[0112] In order to restore the basic network structure during the testing phase, the embodiment of the present invention introduces the idea of ​​self-distillation to transfer the knowledge of the joint classifier σ(·;μ) to another single classifier σ(·;v). Figure 6 A third flow chart of a method for training a graph classification model provided by an embodiment of the present invention is as follows: Figure 6 The step of predicting the type of the image sample and the type labels of the multiple feature images in step S102 may be specifically implemented as follows:

[0113] Step S102-5: Based on the joint classifier, predict the type label and channel label of the image sample, and the type label and channel label of the feature image.

[0114] Step S102-6: Based on the joint classifier, predict the type label of the image sample and the type label of the feature image.

[0115] Step S102-7: Based on a single classifier, predict the type label of the image sample.

[0116] The specific implementation of step S103 may be as follows:

[0117] Step S103-1, using the predicted type labels of image samples, channel labels of image samples, type labels of feature images, channel labels of feature images, annotated type labels of image samples, annotated channel labels of image samples, annotated type labels of feature images, and annotated channel labels of feature images, calculate the joint loss value of the image classification model.

[0118] Step S103-2, using the type labels of the image samples predicted by the joint classifier and the type labels of the feature images, the type labels of the image samples predicted by the single classifier, the type labels of the image samples that have been labeled, and the type labels of the feature images that have been labeled, calculate the comprehensive loss value of the image classification model.

[0119] Step S103-3: Calculate a single loss value of the image classification model using the predicted type label of the image sample and the labeled type label of the image sample.

[0120] Step S103-4: Obtain the loss value according to the combined loss value, the single loss value and the comprehensive loss value.

[0121] For example, a single classifier can be added after the last layer in the network. Since no convolutional layers need to be added, there will be no increase in computational overhead.

[0122] For example, the KL divergence can be used as a criterion to distill the knowledge learned by the joint classifier σ(·; μ) into a single classifier σ(·; v). After the training is completed, the network can remove the joint classifier σ(·; μ) and no longer need to perform feature transformation on the feature map. Only a single classifier needs to be used to solve the original classification task without aggregation after the joint classification output, that is, only a network model that is exactly the same as the baseline model can be used to classify the image.

[0123] For example, it can be combined with Figure 7 For reference, a schematic diagram of a network structure for model training provided by an embodiment of the present invention is shown. As shown in the figure, an image sample with a type label of dog is input into the image classification model to be trained, and feature conversion is performed in the last convolutional layer of the image classification model to generate multiple feature images. Then the feature image and image samples are transferred to other parts of the model and finally to the joint classifier. The joint label consists of type labels and channel labels, with a total of n*k. n represents the number of type labels, and k represents the number of channel labels. For example, a joint label is dog and the second set of channels is discarded.

[0124] When predicting the type labels and channel labels of feature images and image samples, we are actually predicting the probabilities of various labels. Figure 8 The fourth flowchart of a method for training a graph classification model provided by an embodiment of the present invention. Figure 8 As shown, the specific implementation of step S102-5 can be as follows:

[0125] Step S201: Based on the joint classifier, determine the joint probability that the image samples are various joint labels, and the joint probability that each feature image is various joint labels.

[0126] The joint label is composed of a type label and a channel label. Different types of labels and channel labels are different joint labels.

[0127] The specific implementation of step S103-1 can be as follows:

[0128] Step S202: Based on multiple joint probabilities of image samples, type labels and channel labels annotated with the image samples, multiple combinations of feature images, and type labels and channel labels annotated with the feature images, a joint loss value of the image classification model is calculated.

[0129] Exemplarily, the joint classifier will obtain a feature image and an image sample, and the joint classifier predicts the probability that the feature image is a joint label and the probability that the image sample is a joint label. For example, the probability that one of the feature images is a dog and the first channel group is discarded is 80%, the probability that it is a dog and the second channel group is discarded is 10%, and so on.

[0130] Among them, the joint probability can be expressed by the following formula: This formula represents the probability that an image is of a certain type label (i) and channel label (j).

[0131] Exemplarily, after obtaining the joint probability, the joint loss value of the image classification model is calculated based on the above loss function.

[0132] When predicting the type labels of feature images and image samples, we are actually predicting the probabilities of various type labels. Fig. 9 FIG5 is a flowchart of a method for training a graph classification model provided by an embodiment of the present invention. Fig. 9 As shown, step S103-7 can be implemented in the following manner:

[0133] Step S301: Based on a single classifier, determine the self-distillation probability of image samples being various types of labels.

[0134] The specific implementation of step S103-3 can be as follows:

[0135] Step S302: Calculate a single loss value based on multiple self-distillation probabilities of each image sample and the type label of the image sample.

[0136] For example, for a single classifier σ(·; v), based on self-distillation inference, the calculation of self-distillation probability can be simply expressed as follows: Among them, the self-distillation probability represents the probability of each type of label. For example, the probability of predicting that the image sample is a dog is 80%, and the probability of being a pig is 10%.

[0137] When predicting the type labels of feature images and image samples, we are actually predicting the probabilities of feature images and image samples being various type labels. Fig.10 , is a sixth flow chart of a method for training a graph classification model provided by an embodiment of the present invention. Fig.10 As shown, step S102-6 can be implemented in the following manner:

[0138] Step S401: Based on the joint classifier, determine the single probability that the image samples are respectively various types of labels, and the single probability that the feature images are respectively various types of labels.

[0139] Step S402: Based on multiple single probabilities of the image samples and multiple single probabilities of the feature images, determine the aggregated probabilities that the image samples are respectively various types of labels.

[0140] Step S102-7 can be implemented as follows:

[0141] Step S403: Based on a single classifier, determine the self-distillation probabilities of the image samples being various types of labels.

[0142] Step S103-2 can be implemented in the following manner:

[0143] Step S404: Calculate a comprehensive loss value based on multiple aggregation probabilities of each image sample, multiple self-distillation probabilities of each image sample, and the type labels annotated with the image samples.

[0144] For example, the joint classifier can calculate the single probability of the image sample by a single inference method. The formula for calculating the single probability is shown on the right: The formula represents the probability that an image sample is a certain type of label. For example, the probability of being a dog is 80%, the probability of being a pig is 10%, and so on.

[0145] The single probability of the feature image can be obtained as follows: Similarly, this formula represents the probability that the feature image is a certain type of label.

[0146] After obtaining the single probability of the feature image and image sample, we can introduce aggregate reasoning to obtain the aggregate probability of the image sample. The formula for calculating the aggregate probability is as follows: Among them, P agg (i|x) means the aggregate probability that the original label of input x is i. i It can be further expressed as:

[0147] For example, during the training of the network, an additional single classifier σ(·; v) is introduced, so an additional cross entropy loss function L needs to be calculated CE (σ(f;v),y), and a KL-loss D is also required KL (P agg (·|x)||σ(f;v)) enables it to learn the knowledge of the joint classifier. agg (·|x) represents the probability of aggregate inference, and σ(f;v) represents the self-distillation probability based on a single classifier.

[0148] For example, the aggregated probability calculated by the joint classifier and the self-distillation probability obtained based on the single classifier can be used to calculate a comprehensive loss value. The comprehensive loss value is added to the total loss value so that when the image classification model is adjusted based on the loss value, the single classifier can learn the knowledge of the joint classifier.

[0149] For example, when image recognition is performed based on a trained image classification model, the specific implementation method is as follows:

[0150] Get the image to be detected.

[0151] The image to be detected is input into the trained image classification model to obtain the predicted type of the image to be detected.

[0152] The image classification model is obtained according to the image classification model training method in the aforementioned embodiment, and the prediction accuracy of the model can be improved to a certain extent by the method provided by the embodiment of the present invention. That is, compared with other network models, when the same stage of repeated training is performed, that is, the parameters of the network model are repeatedly adjusted the same number of times, the classification accuracy is higher.

[0153] For example, ResNet-110 can be selected as the backbone model, and a learning rate of 0.1, a momentum of 0.9, and a weight decay of stochastic gradient descent (SGD) can be used. In the case of the same 300 trainings, the training batch size is 128, the test batch size is 100, and the learning rate is reduced by 0.1 times at the 150th and 225th stages, which reduces the learning difficulty. The batch size refers to the number of images input each time.

[0154] It should be noted that in this field, single reasoning, aggregate reasoning and self-distillation reasoning can be used to judge the classification accuracy (accuracy) of the model after training to evaluate the model classification results. Among them, the higher the classification accuracy, the better the model training. By using the CIFAR-100 validation set, the classification accuracy comparison between the image classification model provided by the embodiment of the present invention and other models under the same number of training times (for example, 300 times) can be referred to the following table:

[0155]

[0156] Wherein, baseline represents the classification accuracy of the network model for the validation set, and Ours-SI, Ours-AG and Ours-SD all represent the classification accuracy of the image classification model trained by the training method provided by the embodiment of the present invention. Wherein, Ours-SI represents the classification accuracy obtained by single inference based on the validation set, Ours-AG represents the classification accuracy obtained by aggregate inference based on the validation set. Ours-SD represents the classification accuracy obtained by aggregate inference based on the validation set.

[0157] It can be seen from the above table that no matter which model the training method provided by the embodiment of the present invention is applied to, its classification accuracy is improved under various evaluation criteria. Therefore, the training method provided by the embodiment of the present invention is applicable to various network models and has corresponding improvement effects.

[0158] For example, the training method provided by the embodiment of the present invention can also achieve good results on other image datasets such as CIFAR-10 and tiny ImageNet. Based on the ResNet-110 model, the classification accuracy comparison effect of using the training method provided by the embodiment of the present invention and not using the training method provided by the embodiment of the present invention can be shown in the following table:

[0159]

[0160] It can be seen from the above table that no matter what evaluation criteria are used for evaluation, the classification accuracy is significantly improved. In particular, the accuracy of training Tiny-ImageNet on ResNet-110 reaches 63.54%. As a result, the image classification model trained by the training method provided by the embodiment of the present invention has wide applicability, versatility and data set compatibility for various data sets.

[0161] In addition, since the embodiment of the present invention trains and learns the image classification model by discarding different channels, the effect of fine-grained recognition is also better.

[0162] It should be explained that the purpose of fine-grained image classification is to identify similar subcategories of objects under the same basic category. The difference between fine-grained image recognition and general image recognition is that subcategories of fine-grained images usually share the same parts, so they can only be distinguished by subtle differences in the texture and color features of these parts. For example, in dog pictures, they are specifically identified as huskies or Alaskans, and in bird pictures, they are specifically identified as magpies or swallows. These subcategory pictures are difficult to distinguish with the naked eye alone, and the same subcategory of different ages and the same subcategory taken from different angles are also different. These are factors that interfere with the classification results. In order to solve the fine-grained classification task, the network must dig deeper and more detailed features to find common features of the same category or find the differences between different subcategories. In theory, fine-grained image classification is a more difficult task than cross-species image classification, so the image feature gap across species is more obvious, while for species of different subcategories, the feature gap is more subtle, which requires the network to have stronger feature extraction capabilities.

[0163] For example, the ResNet-18 model is used as the baseline network, and three commonly used fine-grained image datasets, CUB-200, Stanford Dogs and Stanford Cars, are used for training. Based on the ResNet-110 model, the classification accuracy comparison effect of using the training method provided by the embodiment of the present invention and not using the training method provided by the embodiment of the present invention can be shown in the following table:

[0164]

[0165] It can be seen from the above table that no matter what evaluation criteria are used for evaluation, the classification accuracy is significantly improved. In particular, when training on the Stanford Dogs dataset, the classification accuracy based on aggregate reasoning is 6.62 percentage points higher than that of the baseline network. As a result, the image classification model obtained by the training method provided in the embodiment of the present invention also has a significant effect on fine-grained recognition.

[0166] Fig.11 The present invention provides a functional module diagram of an image classification model training device 200. The basic principle and technical effects of the device are the same as those of the corresponding method embodiment described above. For the sake of brief description, the parts not mentioned in this embodiment can be referred to the corresponding contents in the method embodiment. Fig.11 As shown, the device includes a training sample module 210, an image type prediction module 220, a loss value calculation module 230 and a training progress judgment module 240.

[0167] The training sample module 210 is used to obtain a plurality of image samples that have been annotated with type labels.

[0168] The image type prediction module 220 is used to input the image sample into the image classification model for each image sample, extract multiple feature images from the image sample in any convolution layer in the image classification model, and predict the type of the image sample and the type labels of the multiple feature images.

[0169] The loss value calculation module 230 is used to calculate the loss value of the image classification model using the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample.

[0170] The training progress judgment module 240 is used to judge whether the loss value reaches the expected training target; if not, the parameters of the image classification model are adjusted based on the loss value, and the step of inputting the image sample into the image classification model for each image sample is returned to be executed until the latest loss value reaches the expected training target. If yes, the training is terminated to obtain the trained image classification model.

[0171] The loss value calculation module 230 is also used to annotate the type label of each feature image extracted from the image sample based on the type label annotated by the image sample, and annotate the channel label of the image sample and each feature image and channel label according to the channels missing from the feature image.

[0172] In some embodiments of the present application, an image classification device is provided. The image classification device may include:

[0173] The image acquisition module is used to acquire the image to be detected.

[0174] The image classification module is used to input the image to be detected into the image classification model obtained by the above-mentioned image classification model training method to obtain the predicted type of the image to be detected.

[0175] An embodiment of the present invention further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method embodiment are executed.

[0176] In summary, the embodiments of the present invention provide an image classification model training method, an image classification method, an apparatus, an electronic device, and a storage medium. After obtaining an image sample with an annotated type label, and inputting the image sample into the image classification model. In any convolutional layer of the image classification model, multiple feature images with different channels are extracted from the image sample. So that the loss value of the image classification model can be determined based on the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample. Subsequently, the network parameters of the image classification model can be repeatedly adjusted based on the loss value to obtain a successfully trained image classification model. Because different channels represent different image features.

[0177] Since the feature image is only extracted in the convolution layer, the data set of the input image classification model is not expanded, that is, the number of images of the input image classification model is increased, so that the time for each model training and learning can be reduced, that is, less time can be wasted. And because multiple feature images are obtained by discarding different channels in the convolution layer, the accuracy of the image classification model after training can also be ensured.

[0178] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0179] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0180] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0181] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for training an image classification model, characterized in that: Applied to electronic equipment, the method comprises: Get multiple image samples with labeled type labels; For each of the image samples, the image sample is input into an image classification model, and in any convolutional layer in the image classification model, multiple feature images are extracted from the image sample, and the type of the image sample and the type labels of the multiple feature images are predicted; wherein different feature images represent image samples lacking different channels, and different channels represent different image features of image samples; The loss value of the image classification model is calculated using the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample; Determine whether the loss value reaches the expected training target; If not, adjusting the parameters of the image classification model based on the loss value, and returning to the step of inputting the image sample into the image classification model for each image sample until the latest loss value reaches the expected training target; If yes, the training is terminated and the trained image classification model is obtained.

2. The method according to claim 1, characterized in that The step of extracting a plurality of feature images from the image samples in any convolutional layer in the image classification model comprises: Based on the multiple channels included in the image sample, multiple channel groups are obtained; wherein the channels included in each channel group are different, and the channels missing from each channel group constitute the multiple channels included in the image sample; One channel group among the multiple channel groups included in the image sample is discarded respectively to obtain multiple feature images including different channel groups.

3. The method according to claim 2, characterized in that The step of discarding one channel group from the multiple channel groups included in the image sample to obtain multiple feature images including different channel groups includes: Obtain channels included in a target channel group from multiple channel groups and determine them as target channels; Generate a binary mask; wherein the binary mask contains a matrix corresponding to each of the channels, and the value of each element in the matrix is ​​to be determined; In the binary mask, the matrix corresponding to the target channel is set to a zero matrix, and the value of each element in the matrix that does not correspond to the target channel is set to 1 to obtain the target binary mask; multiplying the target binary mask by the image sample to obtain a feature image; Repeat the above steps until all feature images of the image sample are obtained.

4. The method according to claim 1, characterized in that: After the step of extracting a plurality of feature images from the image sample, the method further comprises: Based on the type labels that have been labeled for the image samples, labeling the type labels of each feature image extracted from the image samples; According to the channels missing from the feature images, label the channel labels of the image samples and each of the feature images and the channel labels; The step of predicting the type of the image sample and the type labels of the multiple feature images includes: Based on the joint classifier, predict the type label and channel label of the image sample and the type label and channel label of the feature image; Based on the joint classifier, predict the type label of the image sample and the type label of the feature image; Based on a single classifier, predict the type label of the image sample; The step of calculating the loss value of the image classification model by using the predicted type label of the image sample and the type label of the feature image, as well as the annotated type label of the image sample, comprises: Calculate a joint loss value of the image classification model using the predicted type label of the image sample, the channel label of the image sample, the type label of the feature image, the channel label of the feature image, the annotated type label of the image sample, the annotated channel label of the image sample, the annotated type label of the feature image, and the annotated channel label of the feature image; The comprehensive loss value of the image classification model is calculated using the type labels of the image samples predicted by the joint classifier and the type labels of the feature images, the type labels of the image samples predicted by the single classifier, the type labels of the image samples that have been annotated, and the type labels of the feature images that have been annotated; Using the predicted type label of the image sample and the labeled type label of the image sample, a single loss value of the image classification model is calculated; The loss value is obtained according to the combined loss value, the single loss value and the comprehensive loss value.

5. The method according to claim 4, characterized in that The step of predicting the type label and channel label of the image sample and the type label and channel label of the feature image based on the joint classifier includes: Based on the joint classifier, determine the joint probabilities of the image samples being various joint labels, and the joint probabilities of each feature image being various joint labels; wherein the joint label is composed of a type label and a channel label, and different types of the channel labels are different joint labels; The step of calculating the joint loss value of the image classification model by using the predicted type label of the image sample, the channel label of the image sample, the type label of the feature image, the channel label of the feature image, the type label of the image sample, the channel label of the image sample, the type label of the feature image, and the channel label of the feature image includes: Based on multiple joint probabilities of image samples, type labels and channel labels annotated by the image samples, multiple unions of feature images, and type labels and channel labels annotated by the feature images, a joint loss value of the image classification model is calculated.

6. The method according to claim 4, characterized in that The step of predicting the type label of the image sample based on a single classifier includes: Based on the single classifier, determining the self-distillation probabilities of the image samples being respectively various types of labels; The step of predicting the type label of the image sample and the type label of the feature image based on the joint classifier includes: Based on the joint classifier, determining the single probabilities that the image samples are respectively various types of labels, and the single probabilities that the feature images are respectively various types of labels; based on multiple single probabilities of the image samples, and multiple single probabilities of the feature images, determining the aggregated probabilities that the image samples are respectively various types of labels; The step of calculating the comprehensive loss value of the image classification model by using the type labels of the image samples predicted by the joint classifier and the type labels of the feature images, the type labels of the image samples predicted by the single classifier, the type labels of the image samples that have been annotated, and the type labels of the feature images that have been annotated includes: The comprehensive loss value is calculated based on multiple aggregation probabilities of each of the image samples, multiple self-distillation probabilities of each of the image samples, and the type labels that have been annotated with the image samples.

7. The method according to claim 4, characterized in that The step of predicting the type label of the image sample based on a single classifier includes: Based on a single classifier, determining the self-distillation probabilities of the image samples being respectively various types of labels; The step of calculating a single loss value of the image classification model using the predicted type label of the image sample and the labeled type label of the image sample comprises: The single loss value is calculated based on multiple self-distillation probabilities of each of the image samples and the type labels that have been annotated with the image samples.

8. An image classification method, characterized in that: Applied to electronic equipment, the method comprises: Acquire the image to be detected; The image to be detected is input into an image classification model obtained by using the image classification model training method as described in any one of claims 1 to 7 to obtain a predicted type of the image to be detected.

9. An electronic device, characterized in that: include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, each step of the image classification model training method according to any one of claims 1 to 7 or the image classification method according to claim 8 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes each step of the image classification model training method according to any one of claims 1 to 7 or the image classification method according to claim 8.

Images