Image classification model training method, image classification method and related device

Abstract

The embodiments of the present invention provide an image classification model training method, an image classification method and related devices, and relate to the field of image processing. First, an original image and a type label of the original image are obtained; the original image is input into a pre-constructed image classification model, the image classification model includes a feature extraction network and a type prediction network, and the feature extraction network includes N convolutional layers connected in series in sequence; the feature extraction network is then used to extract features from the original image to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer; based on the first feature map, the type label and a plurality of preset binary masks, a first training set is generated, and based on the second feature map, the type label and a plurality of binary masks, a second training set is generated; finally, the type prediction network is trained using the first training set and the second training set to obtain a trained image classification model, thereby reducing the additional overhead generated during the model training process.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image classification model training method, an image classification method and related devices. Background Art

[0002] With the development of artificial intelligence technology, there are more and more application scenarios for calling image classification models to predict the type of images. Before calling the image classification model to predict the type of image, the image classification model needs to be trained.

[0003] In order to improve the accuracy of the model, the existing technology usually expands the original image data several times and then inputs it into the convolutional neural network for training. However, since the convolution operation in the convolutional neural network is time-consuming, the model training process will generate additional overhead that is several times the original training overhead. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, an embodiment of the present invention provides an image classification model training method, an image classification method and related devices, which can reduce the additional overhead generated in the model training process.

[0005] The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides an image classification model training method, the method comprising:

[0007] Obtaining an original image and a type label of the original image;

[0008] Inputting the original image into a pre-built image classification model, wherein the image classification model includes a feature extraction network and a type prediction network, wherein the feature extraction network includes N convolutional layers connected in series;

[0009] Performing feature extraction on the original image using the feature extraction network to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer;

[0010] Generate a first training set based on the first feature map, the type label and a plurality of preset binary masks;

[0011] Generate a second training set based on the second feature map, the type label and the multiple binary masks;

[0012] The type prediction network is trained using the first training set and the second training set to obtain a trained image classification model.

[0013] In an optional implementation, the step of generating a first training set based on the first feature map, the type label and a plurality of preset binary masks includes:

[0014] Processing the first feature map using the multiple binary masks to obtain multiple first training samples, where the multiple first training samples correspond one-to-one to the multiple binary masks;

[0015] A label of each of the first training samples is generated according to the multiple binary masks and the type labels to obtain a first training set, wherein the first training set includes multiple first training samples and a label of each of the first training samples.

[0016] In an optional implementation, the side length of each of the binary masks is the same as the side length of the first feature map, and each of the binary masks includes an all-1 region and / or an all-0 region;

[0017] The step of using the multiple binary masks to process the first feature map to obtain multiple first training samples includes:

[0018] For any one target binary mask among the multiple binary masks, retain an area in the first feature map that overlaps with an all-1 area of ​​the target binary mask, and erase an area in the first feature map that overlaps with an all-0 area of ​​the target binary mask, to obtain a first training sample corresponding to the target binary mask;

[0019] Each of the binary masks is traversed to obtain a first training sample corresponding to each of the binary masks.

[0020] In an optional implementation, the step of generating a label for each first training sample according to the multiple binary masks and the type label includes:

[0021] The identifier of the binary mask corresponding to each of the first training samples is associated with the type label to obtain a label of each of the first training samples.

[0022] In an optional implementation, the step of generating a second training set based on the second feature map, the type label and the multiple binary masks includes:

[0023] Processing the second feature map using the multiple binary masks to obtain multiple second training samples, where the multiple second training samples correspond one-to-one to the multiple binary masks;

[0024] A label of each second training sample is generated according to the multiple binary masks and the type label to obtain a second training set, wherein the second training set includes multiple second training samples and a label of each second training sample.

[0025] In an optional implementation, the step of training the type prediction network using the first training set and the second training set to obtain a trained image classification model includes:

[0026] Using the first training set to train the type prediction network to obtain a first type prediction network;

[0027] Using the second training set to train the type prediction network to obtain a second type prediction network;

[0028] The parameters of the first type prediction network and the parameters of the second type prediction network are distilled to obtain the parameters of the type prediction network to complete the training of the image classification model.

[0029] In a second aspect, the present invention provides an image classification method, the method comprising:

[0030] Get the image to be classified;

[0031] The image to be classified is input into an image classification model obtained by the image classification model training method described in any one of the aforementioned implementation modes to predict the type of the image to be classified.

[0032] In a third aspect, the present invention provides an image classification model training device, the device comprising:

[0033] A first acquisition unit, used to acquire an original image and a type label of the original image;

[0034] A processing unit for:

[0035] Inputting the original image into a pre-built image classification model, wherein the image classification model includes a feature extraction network and a type prediction network, wherein the feature extraction network includes N convolutional layers connected in series;

[0036] Performing feature extraction on the original image using the feature extraction network to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer;

[0037] Generate a first training set based on the first feature map, the type label and a plurality of preset binary masks;

[0038] Generate a second training set based on the second feature map, the type label and the multiple binary masks;

[0039] A training unit is used to train the type prediction network using the first training set and the second training set to obtain a trained image classification model.

[0040] In a fourth aspect, the present invention provides an image classification device, the device comprising:

[0041] A second acquisition unit, used for acquiring an image to be classified;

[0042] A prediction unit is used to input the image to be classified into an image classification model obtained by the image classification model training method described in any of the aforementioned implementation modes, and predict the type of the image to be classified.

[0043] In a fifth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the image classification model training method as described in any one of the aforementioned embodiments, and / or the image classification method as described in the aforementioned embodiments.

[0044] Compared with the prior art, the image classification model training method, image classification method and related devices provided by the embodiments of the present invention first obtain the original image and the type label of the original image; then, the original image is input into a pre-constructed image classification model, the image classification model includes a feature extraction network and a type prediction network, and the feature extraction network includes N convolutional layers connected in series in sequence; then, the feature extraction network is used to extract features of the original image to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer; then, based on the first feature map, the type label and a plurality of preset binary masks, a first training set is generated, and based on the second feature map, the type label and a plurality of binary masks, a second training set is generated; finally, the type prediction network is trained using the first training set and the second training set to obtain a trained image classification model. Since the embodiment of the present invention generates a first training set based on the first feature map, type label and preset multiple binary masks output by the Nth convolutional layer, generates a second training set based on the second feature map, type label and multiple binary masks output by the N-1th convolutional layer, and then uses the first training set and the second training set to train the type prediction network to obtain a trained image classification model, it avoids convolution operations on the original image data expanded several times and reduces the additional overhead generated by the model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0046] Figure 1A schematic diagram of a flow chart of an image classification model training method provided by an embodiment of the present invention;

[0047] Figure 2 A flowchart of an implementation method of step S104 provided in an embodiment of the present invention;

[0048] Figure 3 An example diagram of a binary mask provided by an embodiment of the present invention;

[0049] Figure 4 An exemplary diagram of a first training sample acquisition process provided by an embodiment of the present invention;

[0050] Figure 5 An exemplary diagram of a process of generating a label of a first training sample provided by an embodiment of the present invention;

[0051] Figure 6 A flowchart of an implementation method of step S105 provided in an embodiment of the present invention;

[0052] Figure 7 Another schematic diagram of a flow chart of an image classification model training method provided by an embodiment of the present invention;

[0053] Figure 8 An example diagram of the image classification model training process provided by an embodiment of the present invention;

[0054] Fig. 9 A comparison chart of network test results provided by an embodiment of the present invention;

[0055] Fig.10 A graph showing the ablation learning experimental results of the hyperparameter β provided in an embodiment of the present invention;

[0056] Fig.11 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention;

[0057] Fig.12 A functional unit block diagram of an image classification model training device provided by an embodiment of the present invention;

[0058] Fig.13 A block diagram of a functional unit of an image classification device provided in an embodiment of the present invention.

[0059] Icon: 100 - computer device; 110 - memory; 120 - processor; 200 - image classification model training device; 201 - first acquisition unit; 202 - processing unit; 203 - training unit; 300 - image classification device; 301 - second acquisition unit; 302 - prediction unit. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0063] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0064] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0065] In order to avoid performing convolution operations on the original image data that has been expanded several times and reduce the additional overhead generated in the model training process, an embodiment of the present invention provides an image classification model training method, which will be described in detail below.

[0066] Please refer to Figure 1 , Figure 1 A process of an image classification model training method provided by an embodiment of the present invention is shown. The image classification model training method includes steps S101 to S106.

[0067] S101, obtaining an original image and a type label of the original image.

[0068] The original image refers to the image required for training the image classification model. Usually, a corresponding type label can be assigned to the original image according to its type. The type label can be a digital label, a text label, a symbol label, or other labels defined by the user according to actual needs.

[0069] S102, inputting the original image into a pre-built image classification model.

[0070] The image classification model includes a feature extraction network and a type prediction network. The feature extraction network includes N convolutional layers connected in series. The type prediction network includes a fully connected layer and a softmax classifier.

[0071] S103, using a feature extraction network to extract features from the original image, to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer.

[0072] Among them, N convolutional layers in series in the feature extraction network are used to perform convolution operations on the original image to extract features. Since the features extracted by the shallow convolutional layers are often rough and not suitable for training type prediction networks, the embodiment of the present invention trains the type prediction network based on the feature maps output by the last two convolutional layers in the feature extraction network, i.e., the first feature map output by the Nth convolutional layer and the second feature map output by the N-1th convolutional layer.

[0073] S104, generating a first training set based on the first feature map, the type label and a plurality of preset binary masks.

[0074] Among them, for the first feature map from the Nth convolutional layer, use multiple different binary masks to erase different areas on the first feature map to obtain multiple new first feature maps. Use multiple binary masks to expand the type labels of the original image to obtain multiple new labels. Match the new first feature maps corresponding to the same binary mask with the new labels to obtain the first training set.

[0075] S105, generating a second training set based on the second feature map, the type label and the multiple binary masks.

[0076] Among them, for the second feature map from the N-1th convolutional layer, use multiple different binary masks to erase different areas on the second feature map to obtain multiple new second feature maps. Use multiple binary masks to expand the type labels of the original image to obtain multiple new labels. Match the new second feature maps corresponding to the same binary mask with the new labels to obtain a second training set.

[0077] S106, training the type prediction network using the first training set and the second training set to obtain a trained image classification model.

[0078] The first training set is input into the type prediction network for training to obtain the first softmax classifier. The second training set is input into the type prediction network for training to obtain the second softmax classifier. The information contained in the joint reasoning results output by the first softmax classifier and the second softmax classifier is distilled into a new softmax classifier, and the type prediction network in the trained image classification model is obtained by the fully connected layer and the new softmax classifier.

[0079] The above method provided by the embodiment of the present invention has the beneficial effect of generating a first training set based on the first feature map, type label and preset multiple binary masks output by the Nth convolutional layer, generating a second training set based on the second feature map, type label and multiple binary masks output by the N-1th convolutional layer, and then using the first training set and the second training set to train the type prediction network to obtain a trained image classification model, thereby avoiding convolution operations on the original image data expanded several times and reducing the additional overhead generated by the model training process.

[0080] The following is a detailed description of step S104.

[0081] Please refer to Figure 2 , Figure 2 A process of implementing step S104 provided in an embodiment of the present invention is shown. Step S104 includes sub-steps S104 - 1 to S104 - 2 .

[0082] S104-1, using multiple binary masks to process the first feature map to obtain multiple first training samples.

[0083] The plurality of first training samples and the plurality of binary masks correspond one to one. In the embodiment of the present invention, the binary masks include M0, M1, M2, M3 and M4 (see Figure 3 ), each binary mask includes all-1 regions (i.e., white regions) and / or all-0 regions (i.e., black regions), and the boundary coordinates of the all-0 regions can be expressed as

[0084] like Figure 3 As shown, the boundary coordinates of the all-zero region in M0 are B0 = (0, 0, 0, 0), and the boundary coordinates of the all-zero region in M1 are Boundary coordinates of the all-zero region in M2 Boundary coordinates of the all-zero region in M3 Boundary coordinates of the all-zero region in M4 In the above expression, c is the side length of the first feature graph f. as well as They are the coordinates of the four vertices on the boundary of the all-0 region.

[0085] Use binary masks M0, M1, M2, M3 and M4 to process the first feature map f and obtain the first training sample and The processing process can be represented by ⊙, that is,

[0086] Specifically, the implementation process of sub-step S104-1 may be as follows:

[0087] First, for any target binary mask among multiple binary masks, the area in the first feature map that overlaps with the all-1 area of ​​the target binary mask is retained, and the area in the first feature map that overlaps with the all-0 area of ​​the target binary mask is erased to obtain a first training sample corresponding to the target binary mask;

[0088] Then, each binary mask is traversed to obtain the first training sample corresponding to each binary mask.

[0089] The first feature map f has the same side length as the binary masks M0, M1, M2, M3 and M4. Figure 4 As shown, M0 does not contain all zero regions, and the first training sample corresponding to M0 All regions in the first feature map f are retained in the first training sample. This is the first feature map f. The all-0 regions of M1, M2, M3, and M4 are in the upper left, upper right, lower left, and lower right, respectively. It can be understood that the upper left, upper right, lower left, and lower right regions in the first feature map f are erased respectively to obtain the first training sample. and

[0090] In the above method embodiment, since the features of some areas in the first feature map f are erased, the subsequent type prediction network can deepen the learning of local features in the process of learning the semantic information of different areas of the feature map, and have a clearer understanding of the target area. At the same time, the training difficulty of the type prediction network is also improved to a certain extent, which can effectively curb the overfitting phenomenon. In addition, the target area features and the background area features are separated to a certain extent, which reduces the possibility of the network forming "shortcut learning" and effectively alleviates the coupling effect of the target area and the background area brought about by "shortcut learning".

[0091] S104-2, generating a label for each first training sample according to the multiple binary masks and type labels, to obtain a first training set.

[0092] The first training set includes multiple first training samples and labels for each first training sample. In order to reduce the cost of manual annotation and enable the obtained first training samples to be directly applied to network training in supervised learning scenarios, the embodiment of the present invention expands the type label y of the original image according to the binary masks M0, M1, M2, M3 and M4, and obtains the first training samples and Tags and

[0093] Specifically, the implementation process of sub-step S104-2 is as follows:

[0094] The identifier of the binary mask corresponding to each first training sample is associated with the type label to obtain the label of each first training sample.

[0095] Among them, the first training sample Tags The expression can be like Figure 5 As shown in the figure, the type labels of the original image are "dog", "cat", "fish" and "bird". For any type label, such as "dog", it is expanded using binary masks M0, M1, M2, M3 and M4 to obtain the first training sample and The labels are (dog, M0), (dog, M1), (dog, M2), (dog, M3) and (dog, M4).

[0096] The following is a detailed description of step S105.

[0097] Please refer to Figure 6 , Figure 6 A process of implementing step S105 provided in an embodiment of the present invention is shown. Step S105 includes sub-steps S105 - 1 to S105 - 2 .

[0098] S105-1, use multiple binary masks to process the second feature map to obtain multiple second training samples.

[0099] Among them, the multiple second training samples and the multiple binary masks correspond one to one. It can be understood that since the implementation process of sub-step S105-1 is the same as the above sub-step S104-1, it will not be described in detail;

[0100] S105 - 2 , generating a label for each second training sample according to the multiple binary masks and the type labels, to obtain a second training set.

[0101] The second training set includes a plurality of second training samples and a label of each second training sample. Since the implementation process of sub-step S105-2 is the same as that of the above sub-step S104-2, it will not be described in detail.

[0102] The following is a detailed description of step S106.

[0103] Please refer to Figure 7 , Figure 7 Another process of the image classification model training method provided by an embodiment of the present invention is shown, where step S106 includes sub-steps S106 - 1 to S106 - 3 .

[0104] S106-1, using the first training set to train a type prediction network to obtain a first type prediction network.

[0105] The first type prediction network refers to a type prediction network obtained by inputting the first training set data into the type prediction network for training, and the first type prediction network includes a first softmax classifier.

[0106] S106-2, using the second training set to train a type prediction network to obtain a second type prediction network.

[0107] The second type prediction network refers to a type prediction network obtained by inputting the second training set data into the type prediction network for training, and the second type prediction network includes a second softmax classifier.

[0108] S106-3, distilling the parameters of the first type prediction network and the parameters of the second type prediction network to obtain the parameters of the type prediction network to complete the training of the image classification model.

[0109] Among them, the parameters of the first type prediction network refer to the information contained in the joint reasoning result output by the first softmax classifier, and the parameters of the second type prediction network refer to the information contained in the joint reasoning result output by the second softmax classifier. The self-distillation learning method (Self-Distillation) is used to distill the information contained in the joint reasoning result output by the first softmax classifier and the second softmax classifier to a new softmax classifier through knowledge migration, and then the type prediction network in the trained image classification model is obtained by the fully connected layer and the softmax classifier.

[0110] For a more vivid explanation of step S106, please refer to Figure 8 , Figure 8An example diagram of the image classification model training process provided in an embodiment of the present invention, in which a feature map processing module, a fully connected layer and a softmax classifier are provided after the two last convolutional layers of the convolutional neural network.

[0111] make is the softmax classifier attached to the i-th convolutional layer, where ω is the weight parameter in the classifier and f i It is the feature map output by the i-th convolutional layer.

[0112] The raw probability output by the existing softmax classifier Fc(f;ω) for N-class classification tasks Among them, y represents the original category of the input image x.

[0113] In the embodiment of the present invention, a joint softmax classifier is used to replace the original softmax classifier to process the feature map after binary mask processing. Make a prediction. At this point, the joint softmax classifier outputs the joint probability Among them, j represents the label generated based on the binary mask and the type label of the input image x, which is called “pseudo label” here.

[0114] The training loss function obtained by connecting the joint softmax classifier after the last convolutional layer The cross entropy loss term It can be expressed as:

[0115] At this time, for the output of the joint softmax classifier connected after the last convolutional layer, the single inference result will be used to represent the network's ability to classify the original dataset.

[0116] In the embodiment of the present invention, two joint softmax classifiers Fc are introduced. m (·;θ) and Fc m-1 (·; θ). The total loss function of network training at this time The loss functions from the two joint classifiers are combined. and It can be expressed as Where β is a network training hyperparameter that needs to be set in advance to limit the joint classifier Fc m-1 (·; θ) affects the network prediction results, so β ​​will be taken in [0,1]. According to the results of ablation learning, the value of β is usually 1.

[0117] For the mini-batch training strategy, during each batch of network training, the network optimization goal is to make The average value of is the smallest.

[0118] The optimization process can be expressed as Here, M is the preset mini-batch size, that is, the number of images input into the network each time.

[0119] In an embodiment of the present invention, each feature map after binary mask processing is assigned a label composed of the same original label and an independent self-supervised "pseudo-label", which provides a possibility for the network to integrate all feature maps to predict the original label during the testing phase.

[0120] Therefore, in the embodiment of the present invention, the “single inference” result P SI Based on (y|x), a more comprehensive "aggregate reasoning" result is proposed as the final prediction result of the network. The "aggregate reasoning" result integrates the prediction results of each feature map output by all joint classifiers to predict the original category y of the input image x. This can effectively improve the robustness and generalization of the network, and also improve the fault tolerance of network prediction to a certain extent.

[0121] Here is the source process of the "aggregate reasoning" result:

[0122] The “aggregate inference” result under a single joint softmax classifier can be expressed as:

[0123]

[0124] Where m indicates that the joint softmax classifier is connected after the last convolutional layer group.

[0125] Since the embodiment of the present invention uses two joint softmax classifiers to optimize the joint loss, the network can integrate the "aggregate reasoning" results of the two classifiers to obtain a more comprehensive "aggregate reasoning" result. AG The calculation method of (y|x) can be expressed as:

[0126]

[0127] Among them, P m,AG (y|x) and P m-1,AG (y|x) represents the independent “aggregate inference” results given by the joint softmax classifiers connected to the last and the second to last convolutional layers, respectively.

[0128] In order to restore the original structure of the network, the embodiment of the present invention utilizes the existing self-distillation learning method (Self-Distillation) to restore the feature map processing module and the softmax classifier added to the network.

[0129] The network achieves a classification improvement effect similar to that of ensemble learning by distilling the knowledge learned during its own training, without the need to introduce an additional teacher network during the entire training process. In addition, in the method of the embodiment of the present invention, the introduction of the self-distillation learning method can also restore the original structure of the network and compress the number of network parameters in the test phase. At the same time, the convolutional layer in the original deep convolutional neural network learns effective supervised learning representation content.

[0130] After the self-distillation learning method is introduced into the method of the embodiment of the present invention, the network will transfer knowledge so that the “aggregate reasoning” result P output by the network is AG The information contained in (·|x) is distilled into a new softmax classifier Fc(·;μ). At this point, it should be noted that the weight parameter μ in this new classifier Fc(·;μ) should have the same scale as the weight parameter ω in the classifier Fc(·;ω) originally used for the N classification task, thereby achieving the purpose of restoring the network classifier.

[0131] After introducing self-distillation learning to complete network training, this new classifier Fc(·; μ) should provide better test classification performance than the baseline network when the network no longer uses feature maps and labels obtained based on binary masks.

[0132] At this point, the optimization goal of the network during training is Replace with It can be expressed as:

[0133]

[0134] in, Represents the Kullback-Leibler divergence, which can make the classifier Fc(f;μ) approach the "aggregate reasoning" result P output by the network. AG (·|x), thus achieving the purpose of self-distillation learning.

[0135] Kullback-Leibler divergence It can be expressed as:

[0136]

[0137] The probability result output by the additional classifier introduced by self-distillation learning is referred to as the “self-distillation” result P in this paper. SD(y|x), which can be expressed as:

[0138]

[0139] Figure 8 The implementation process of the method of the embodiment of the present invention and three different output results (“single reasoning” result P SI (y|x), “aggregate reasoning” result P AG (y|x) and the “self-distillation” result P SD The process of obtaining (y|x)).

[0140] The embodiment of the present invention also sets up a series of experiments to further demonstrate the effectiveness of the aforementioned image classification model training method.

[0141] First, the software and hardware configuration of the experiment and the training strategy of the network are described. In the embodiment of the present invention, in order to ensure the objectivity and effectiveness of the experiment, a unified software and hardware configuration is used to complete the relevant experiments. The specific software and hardware configuration information is shown in Table 1.

[0142] Table 1

[0143]

[0144] The experiment uses deep convolutional neural networks to complete image classification tasks. The network structures involved include ResNet, Wide ResNet, and PyramidNet. The image datasets involved include CIFAR-10 / 100 datasets and Tiny-ImageNet datasets. Fine-grained image datasets include Stanford Dog / Car datasets and Caltech-UCSD Birds-200-2011 (CUB200-2011) datasets.

[0145] All experiments were repeated four times, and the average and variance of each indicator are given. The highest classification accuracy is indicated in bold. The initial learning rate of the network is 0.1, the initial momentum is 0.9, and the initial weight decay is 10-4. For general image classification tasks, the network will be trained for 300 cycles, the batch size of the CIFAR-10 / 100 dataset is set to 128, and the batch size of the Tiny-ImageNet dataset is set to 256. For fine-grained image classification tasks, the network will also be trained for 300 cycles, and the batch size of the Stanford Dog / Car dataset and the CUB200-2011 dataset is set to 16. In addition, the stochastic gradient descent strategy will be uniformly used to optimize the network weights during network training. The learning rate of the network will be multiplied by a decay factor of 0.1 when the network is trained to the 150th and 225th cycles.

[0146] Tables 2 and 3 respectively show the classification performance of the method provided by the embodiment of the present invention on different data sets and different network structures, which include the classification results of the benchmark network and the prediction results of the network output after training by the method of the embodiment of the present invention. The highest classification accuracy is represented by bold words. It should be noted that the data set generalization experiment uniformly uses the ResNet-110 network; the network structure generalization experiment uniformly uses the CIFAR-100 data set. In addition, due to the small amount of test set data in Tiny-ImageNet, the experiment incorporates the validation set data in the data set into the test set.

[0147] Table 2

[0148]

[0149] Table 3

[0150]

[0151] It can be seen from the experimental results that the method of the embodiment of the present invention can significantly improve the supervised learning representation capabilities of different networks on different data sets. At the same time, thanks to the gain effect of self-distillation learning on the network, the "self-distillation" result can achieve a higher classification accuracy than the "aggregate reasoning" result.

[0152] Table 4 shows the results of the superposition experiment, including the classification accuracy of the baseline network and the classification accuracy of the network output after being trained by the method of the embodiment of the present invention (Ours in the table), the existing regularization method, and the existing self-supervised learning method. The experiment selected the "aggregate reasoning" result of the network output for display.

[0153] Table 4

[0154]

[0155] It can be seen from the experimental results that when the method of the embodiment of the present invention and the existing method are superimposed, the benchmark network achieves a higher classification accuracy than the method of the embodiment of the present invention or the existing method alone. In particular, when the self-supervised learning method SLA and the method of the embodiment of the present invention are superimposed on ResNet-110, the "aggregate reasoning" result output by the network can achieve the highest classification accuracy of 80.61% in the superposition experiment.

[0156] The experiment applies the method of the embodiment of the present invention and the existing self-supervised learning method SLA to the ResNet-110 network, and performs image classification tasks under the CIFAR-100 dataset to compare the classification performance and computational overhead. Table 5 shows the comparison results of the GPU memory consumption of the image processor, the time consumed per batch of network training, and the classification accuracy. It should be noted that the classification accuracy of the SLA method and the method of the embodiment of the present invention in the table are given by the "single inference" results.

[0157] Table 5

[0158]

[0159] It can be seen from Table 5 that compared with the 252.1% additional GPU memory consumption and 258.1% additional training time overhead brought by the SLA method, the method of the embodiment of the present invention only brings 10.1% additional GPU memory consumption and 11.7% additional training time overhead, and the accuracy gain brought to the network by the two is almost the same. It can be seen that the method of the embodiment of the present invention effectively improves the classification performance of the network without bringing a large amount of additional training overhead to the network.

[0160] Fig. 9 The focus areas of target objects on different test samples from the baseline network and the network trained by the method of the embodiment of the present invention are shown. It should be noted that the test samples are randomly selected from the Stanford Dog dataset, which can objectively demonstrate the effect of the method of the embodiment of the present invention.

[0161] Compared with the reference network, the network trained by the method of the embodiment of the present invention has a more accurate judgment on the target object area, and the classification performance of the network has been improved. In addition, it can be seen from the visualization results that the method of the embodiment of the present invention effectively alleviates the coupling effect between the target area and the background area brought about by "shortcut learning", and has a clearer understanding of the characteristics of the target object during the network training process. The network will no longer continue to pay attention to the background object area that appears in large numbers in the data set. Therefore, the network will reduce its dependence on "shortcut learning" and achieve decoupling of the target object area and the background redundant area.

[0162] In the embodiment of the present invention, ablation learning is mainly used to illustrate the effectiveness of the hyperparameter β, and the optimal value range of the hyperparameter β is given through ablation experiments. The hyperparameter introduced in the method of the embodiment of the present invention is to limit the output result of the joint classifier connected to the penultimate convolutional layer, and the hyperparameter β should be taken in [0,1]. Therefore, the method of the embodiment of the present invention is used to train the ResNet-110 network to classify the CIFAR-100 dataset, and the hyperparameter β is taken in [0,1] in turn, and the influence of the hyperparameter β is observed by the changing trend of the network classification performance under different values.

[0163] Fig.10 The experimental results of ablation learning on the hyperparameter β are shown. It can be seen that, except for the case of β=0.1, the method of the embodiment of the present invention shows a very high tolerance for the hyperparameter β. When β=0.1, it is difficult to train the joint classifier connected to the penultimate convolutional layer. As the hyperparameter β increases, the classification performance of the network can be effectively improved. Therefore, in order to ensure the versatility of the method of the embodiment of the present invention, the value of the hyperparameter β defaults to 1.

[0164] Through the above experimental results, it can be clearly seen that the image classification model training method provided by the embodiment of the present invention has a beneficial effect compared with the prior art.

[0165] The embodiment of the present invention further provides an image classification method, which includes steps S201 to S202.

[0166] S201, obtaining an image to be classified.

[0167] S202, inputting the image to be classified into an image classification model to predict the type of the image to be classified.

[0168] Among them, the image classification model is obtained according to the image classification model training method in the aforementioned method embodiment.

[0169] Furthermore, the embodiment of the present invention also provides a schematic block diagram of the structure of the computer device 100, please refer to Fig.11 , the computer device 100 may include a memory 110 and a processor 120 .

[0170] Among them, the processor 120 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the image classification model training method provided by the above-mentioned method embodiment, and / or the program execution of the image classification method.

[0171] The memory 110 may be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 110 may exist independently and be connected to the processor 120 via a communication bus. The memory 110 may also be integrated with the processor 120. Among them, the memory 110 is used to store machine executable instructions for executing the scheme of the present application. The processor 120 is used to execute the machine executable instructions stored in the memory 110 to implement the above-mentioned method embodiment.

[0172] An embodiment of the present invention also provides a computer-readable storage medium containing a computer program. When the computer program is executed, it can be used to perform the image classification model training method provided by the above-mentioned method embodiment, and / or related operations in the image classification method.

[0173] Please refer to Fig.12 , Fig.12 A functional unit block diagram of an image classification model training device 200 provided in an embodiment of the present invention. The image classification model training device 200 is applied to a computer device 100, and may include a first acquisition unit 201, a processing unit 202, and a training unit 203, wherein the first acquisition unit 201, the processing unit 202, and the training unit 203 can all be stored in a memory or a computer-readable storage medium in the form of software. It should be noted that the basic principle and technical effect of the image classification model training device 200 provided in an embodiment of the present invention are the same as those in the above-mentioned embodiment, and for the sake of brief description, they are not mentioned in the embodiment of the present invention.

[0174] The first acquisition unit 201 is used to acquire an original image and a type label of the original image.

[0175] The processing unit 202 is used to input the original image into a pre-built image classification model, where the image classification model includes a feature extraction network and a type prediction network, where the feature extraction network includes N convolutional layers connected in series; extract features from the original image using the feature extraction network to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer; generate a first training set based on the first feature map, the type label and a plurality of preset binary masks; and generate a second training set based on the second feature map, the type label and a plurality of binary masks;

[0176] The training unit 203 is used to train the type prediction network using the first training set and the second training set to obtain a trained image classification model.

[0177] In one implementation, the processing unit 202 is further specifically used to process the first feature map using multiple binary masks to obtain multiple first training samples, and the multiple first training samples and the multiple binary masks correspond one to one; based on the multiple binary masks and type labels, a label for each first training sample is generated to obtain a first training set, wherein the first training set includes multiple first training samples and a label for each first training sample.

[0178] In one implementation, the side length of each binary mask is the same as the side length of the first feature map, and each binary mask includes an all-1 area and / or an all-0 area. When the processing unit 202 is used to process the first feature map using multiple binary masks to obtain multiple first training samples, it is also specifically used to retain the area in the first feature map that overlaps with the all-1 area of ​​the target binary mask, and erase the area in the first feature map that overlaps with the all-0 area of ​​the target binary mask, for any target binary mask among the multiple binary masks, to obtain the first training sample corresponding to the target binary mask; traverse each binary mask to obtain the first training sample corresponding to each binary mask.

[0179] In one implementation, when the processing unit 202 is used to generate a label for each first training sample based on multiple binary masks and type labels, it is also specifically used to associate the identifier of the binary mask corresponding to each first training sample with the type label to obtain the label of each first training sample.

[0180] In one implementation, the processing unit 202 is further specifically used to process the second feature map using multiple binary masks to obtain multiple second training samples, and the multiple second training samples correspond one-to-one to the multiple binary masks; based on the multiple binary masks and type labels, a label for each second training sample is generated to obtain a second training set, wherein the second training set includes multiple second training samples and a label for each second training sample.

[0181] In one implementation, the training unit 203 is further specifically used to train the type prediction network using the first training set to obtain the first type prediction network; train the type prediction network using the second training set to obtain the second type prediction network; and distill the parameters of the first type prediction network and the parameters of the second type prediction network to obtain the parameters of the type prediction network to complete the training of the image classification model.

[0182] Please refer to Fig.13 , Fig.13 A functional unit block diagram of an image classification device 300 provided in an embodiment of the present invention. The image classification device 300 is applied to a computer device 100 and may include a second acquisition unit 301 and a prediction unit 302, wherein the second acquisition unit 301 and the prediction unit 302 can be stored in a memory or a computer-readable storage medium in the form of software. It should be noted that the basic principle and technical effect of the image classification device 300 provided in an embodiment of the present invention are the same as those in the above-mentioned embodiment, and for the sake of brief description, they are not mentioned in the embodiment of the present invention.

[0183] The second acquisition unit 301 is used to acquire the image to be classified;

[0184] The prediction unit 302 is used to input the image to be classified into the image classification model obtained by the above-mentioned image classification model training method, and predict the type of the image to be classified.

[0185] In summary, the image classification model training method, image classification method and related devices provided by the embodiments of the present invention first obtain the original image and the type label of the original image; then, the original image is input into a pre-constructed image classification model, the image classification model includes a feature extraction network and a type prediction network, and the feature extraction network includes N convolutional layers connected in series in sequence; then, the feature extraction network is used to extract features of the original image to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer; then, based on the first feature map, the type label and a plurality of preset binary masks, a first training set is generated, and based on the second feature map, the type label and a plurality of binary masks, a second training set is generated; finally, the type prediction network is trained using the first training set and the second training set to obtain a trained image classification model. Since the embodiment of the present invention generates a first training set based on the first feature map, type label and preset multiple binary masks output by the Nth convolutional layer, generates a second training set based on the second feature map, type label and multiple binary masks output by the N-1th convolutional layer, and then uses the first training set and the second training set to train the type prediction network to obtain a trained image classification model, it avoids convolution operations on the original image data expanded several times and reduces the additional overhead generated by the model training process.

[0186] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for training an image classification model, characterized in that: The method comprises: Obtaining an original image and a type label of the original image; Inputting the original image into a pre-built image classification model, wherein the image classification model includes a feature extraction network and a type prediction network, wherein the feature extraction network includes N convolutional layers connected in series; Performing feature extraction on the original image using the feature extraction network to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer; Generate a first training set based on the first feature map, the type label and a plurality of preset binary masks; Generate a second training set based on the second feature map, the type label and the multiple binary masks; The type prediction network is trained using the first training set and the second training set to obtain a trained image classification model.

2. The method according to claim 1, characterized in that The step of generating a first training set based on the first feature map, the type label and a plurality of preset binary masks comprises: Processing the first feature map using the multiple binary masks to obtain multiple first training samples, where the multiple first training samples correspond one-to-one to the multiple binary masks; A label of each of the first training samples is generated according to the multiple binary masks and the type labels to obtain a first training set, wherein the first training set includes multiple first training samples and a label of each of the first training samples.

3. The method according to claim 2, characterized in that The side length of each of the binary masks is the same as the side length of the first feature map, and each of the binary masks includes an all-1 region and / or an all-0 region; The step of using the multiple binary masks to process the first feature map to obtain multiple first training samples includes: For any one target binary mask among the multiple binary masks, retain an area in the first feature map that overlaps with an all-1 area of ​​the target binary mask, and erase an area in the first feature map that overlaps with an all-0 area of ​​the target binary mask, to obtain a first training sample corresponding to the target binary mask; Each of the binary masks is traversed to obtain a first training sample corresponding to each of the binary masks.

4. The method according to claim 2, characterized in that The step of generating a label for each of the first training samples according to the multiple binary masks and the type labels comprises: The identifier of the binary mask corresponding to each of the first training samples is associated with the type label to obtain a label of each of the first training samples.

5. The method according to claim 1, characterized in that The step of generating a second training set based on the second feature map, the type label and the multiple binary masks comprises: Processing the second feature map using the multiple binary masks to obtain multiple second training samples, where the multiple second training samples correspond one-to-one to the multiple binary masks; A label of each second training sample is generated according to the multiple binary masks and the type label to obtain a second training set, wherein the second training set includes multiple second training samples and a label of each second training sample.

6. The method according to claim 1, characterized in that The step of training the type prediction network using the first training set and the second training set to obtain a trained image classification model comprises: Using the first training set to train the type prediction network to obtain a first type prediction network; Using the second training set to train the type prediction network to obtain a second type prediction network; The parameters of the first type prediction network and the parameters of the second type prediction network are distilled to obtain the parameters of the type prediction network to complete the training of the image classification model.

7. An image classification method, characterized in that: The method comprises: Get the image to be classified; The image to be classified is input into an image classification model obtained by the image classification model training method according to any one of claims 1 to 6 to predict the type of the image to be classified.

8. An image classification model training device, characterized in that: The device comprises: A first acquisition unit, used to acquire an original image and a type label of the original image; A processing unit for: Inputting the original image into a pre-built image classification model, wherein the image classification model includes a feature extraction network and a type prediction network, wherein the feature extraction network includes N convolutional layers connected in series; Performing feature extraction on the original image using the feature extraction network to obtain a first feature map output by the Nth convolutional layer and a second feature map output by the N-1th convolutional layer; Generate a first training set based on the first feature map, the type label and a plurality of preset binary masks; Generate a second training set based on the second feature map, the type label and the multiple binary masks; A training unit is used to train the type prediction network using the first training set and the second training set to obtain a trained image classification model.

9. An image classification device, characterized in that: The device comprises: A second acquisition unit, used for acquiring an image to be classified; A prediction unit is used to input the image to be classified into an image classification model obtained by the image classification model training method according to any one of claims 1 to 6, and predict the type of the image to be classified.

10. A computer device, characterized in that: It comprises a memory and a processor, the memory stores a computer program, and the processor implements the image classification model training method according to any one of claims 1 to 6 and / or the image classification method according to claim 7 when executing the computer program.

Images