Neural network training and image classification method, device, storage medium, and equipment

Abstract

The present disclosure provides a neural network training and image classification method, apparatus, storage medium, and device, wherein the method comprises: inputting a sample image into a teacher network and a student network respectively, obtaining a first feature map output by the teacher network, and a second feature map output by the student network; determining first associated feature information based on the feature information included in the first feature map, and determining second associated feature information based on the feature information included in the second feature map; and performing supervised training on the student network based on the difference between the second associated feature information and the first associated feature information.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning, and in particular to a neural network training and image classification method, device, storage medium, and equipment. Background Art

[0002] At present, knowledge distillation has become a critical technical issue in the field of deep learning. It is usually used for model compression, aiming to transfer the sample relationship information learned by a network model with more parameters and better performance (teacher model) to a lightweight network model with fewer parameters and faster speed (student model), thereby improving the accuracy of the student model. Summary of the invention

[0003] The present invention provides a neural network training and image classification method, device, storage medium, and equipment.

[0004] According to a first aspect of an embodiment of the present disclosure, a neural network training method is provided, comprising: inputting a sample image into a teacher network and a student network respectively, obtaining a first feature map output by the teacher network, and a second feature map output by the student network; determining first associated feature information based on feature information included in the first feature map, and determining second associated feature information based on feature information included in the second feature map; and performing supervised training on the student network based on a difference between the second associated feature information and the first associated feature information.

[0005] In some optional embodiments, the method also includes: determining a target area based on the first feature map; determining a first target feature map based on the target area and the first feature map, and determining a second target feature map based on the target area and the second feature map; determining first associated feature information based on the feature information included in the first feature map, and determining second associated feature information based on the feature information included in the second feature map, includes: determining the first associated feature information based on the feature information included in the first target feature map, and determining the second associated feature information based on the feature information included in the second target feature map.

[0006] In some optional embodiments, the target area of ​​the sample image includes a target object, and / or the area of ​​the sample image other than the target area includes at least one of the following: a background portion, an occluded portion of the target object.

[0007] In some optional embodiments, determining the target area based on the first feature map includes: inputting the first feature map into a pre-trained first neural network to obtain a pixel normalization value corresponding to each of a plurality of areas included in the sample image output by the first neural network; and taking an area whose pixel normalization value is greater than or equal to a preset value as the target area.

[0008] In some optional embodiments, determining a first target feature map based on the target area and the first feature map, and determining a second target feature map based on the target area and the second feature map, includes: performing a dot multiplication of a normalized pixel value corresponding to each area of ​​the multiple areas with the feature information included in the first feature map to obtain a first attention feature map, and performing a dot multiplication of a normalized pixel value corresponding to each area of ​​the multiple areas with the feature information included in the second feature map to obtain a second attention feature map; obtaining the first target feature map based on the target area on the first attention feature map, and obtaining the second target feature map based on the target area on the second attention feature map.

[0009] In some optional embodiments, the first neural network includes multiple first network layers and a first output layer, and the second neural network includes multiple second network layers and a second output layer, wherein the network structure of the multiple second network layers is the same as the network structure of the multiple first network layers; the method also includes: using the first feature map as the input of the second neural network to obtain the classification result output by the second neural network; based on the difference between the classification result output by the second neural network and the classification result marked in the sample image, performing supervised training on the second neural network; using the network parameters of the multiple second network layers included in the trained second neural network as the network parameters of the multiple first network layers included in the first neural network.

[0010] In some optional embodiments, determining the first associated feature information based on the feature information included in the first feature map includes: determining the first associated feature information based on the similarity between the feature information of different regions on the first feature map; determining the second associated feature information based on the feature information included in the second feature map includes: determining the second associated feature information based on the similarity between the feature information of different regions on the second feature map.

[0011] In some optional embodiments, the number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple; determining the first associated feature information based on the feature information included in the first feature map includes: determining the first associated feature information based on the similarity between the feature information of the same area on the first feature map corresponding to different sample images respectively; determining the second associated feature information based on the feature information included in the second feature map includes: determining the second associated feature information based on the similarity between the feature information of the same area on the second feature map corresponding to different sample images respectively.

[0012] In some optional embodiments, the number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple; determining the first associated feature information based on the feature information included in the first feature map includes: determining the first associated feature information based on the similarity between feature information of different regions on the first feature map corresponding to different sample images respectively; determining the second associated feature information based on the feature information included in the second feature map includes: determining the second associated feature information based on the similarity between feature information of different regions on the second feature map corresponding to different sample images respectively.

[0013] In some optional embodiments, there are multiple first feature maps and multiple second feature maps, and the multiple first feature maps correspond to different resolutions, and the multiple second feature maps correspond to different resolutions; determining the first associated feature information based on the feature information included in the first feature map, and determining the second associated feature information based on the feature information included in the second feature map, includes: determining the first associated feature information based on the feature information included in the first feature maps with different resolutions, and determining the second associated feature information based on the feature information included in the second feature maps with different resolutions, in order of resolution from large to small.

[0014] In some optional embodiments, the supervised training of the student network includes: using knowledge distillation to supervise the student network.

[0015] In some optional embodiments, the sample image includes a physical image with a labeled classification result.

[0016] According to a second aspect of an embodiment of the present disclosure, there is provided an image classification method, the method comprising:

[0017] The image to be detected is input into an image classification network to obtain a classification result to which the target object included in the image to be detected belongs, which is output by the image classification network; the image classification network is a student network trained by the method described in any one of the first aspects.

[0018] According to a third aspect of an embodiment of the present disclosure, there is provided a neural network training device, comprising: an acquisition module, used to input a sample image into a teacher network and a student network respectively, to obtain a first feature map output by the teacher network, and a second feature map output by the student network; a first determination module, used to determine first associated feature information based on feature information included in the first feature map, and to determine second associated feature information based on feature information included in the second feature map; and a first training module, used to perform supervised training on the student network based on the difference between the second associated feature information and the first associated feature information.

[0019] In some optional embodiments, the device also includes: a second determination module, used to determine the target area based on the first feature map; a third determination module, used to determine the first target feature map based on the target area and the first feature map, and to determine the second target feature map based on the target area and the second feature map; the first determination module includes: a first determination submodule, used to determine the first associated feature information based on the feature information included in the first target feature map, and to determine the second associated feature information based on the feature information included in the second target feature map.

[0020] In some optional embodiments, the target area of ​​the sample image includes a target object, and / or the area of ​​the sample image other than the target area includes at least one of the following: a background portion, an occluded portion of the target object.

[0021] In some optional embodiments, the second determination module includes: a second determination submodule, used to input the first feature map into a pre-trained first neural network to obtain a pixel normalization value corresponding to each of the multiple regions included in the sample image output by the first neural network; and a third determination submodule, used to take the region whose pixel normalization value is greater than or equal to a preset value as the target region.

[0022] In some optional embodiments, the third determination module includes: a fourth determination submodule, used to perform a dot product of the normalized pixel value corresponding to each of the multiple regions with the feature information included in the first feature map to obtain a first attention feature map, and perform a dot product of the normalized pixel value corresponding to each of the multiple regions with the feature information included in the second feature map to obtain a second attention feature map; a fifth determination submodule, used to obtain the first target feature map based on the target area on the first attention feature map, and to obtain the second target feature map based on the target area on the second attention feature map.

[0023] In some optional embodiments, the first neural network includes multiple first network layers and a first output layer, and the second neural network includes multiple second network layers and a second output layer, wherein the network structure of the multiple second network layers is the same as the network structure of the multiple first network layers; the device also includes: a classification result acquisition module, used to use the first feature map as the input of the second neural network to obtain the classification result output by the second neural network; a second training module, used to supervise the second neural network based on the difference between the classification result output by the second neural network and the classification result marked in the sample image; a fourth determination module, used to use the network parameters of the multiple second network layers included in the trained second neural network as the network parameters of the multiple first network layers included in the first neural network.

[0024] In some optional embodiments, the first determination module includes: a sixth determination submodule, used to determine the first associated feature information based on the similarity between feature information of different regions on the first feature map; and a seventh determination submodule, used to determine the second associated feature information based on the similarity between feature information of different regions on the second feature map.

[0025] In some optional embodiments, the number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple; the first determination module includes: an eighth determination submodule, used to determine the first associated feature information based on the similarity between the feature information of the same area on the first feature map corresponding to different sample images; a ninth determination submodule, used to determine the second associated feature information based on the similarity between the feature information of the same area on the second feature map corresponding to different sample images.

[0026] In some optional embodiments, the number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple; the first determination module includes: a tenth determination submodule, used to determine the first associated feature information based on the similarity between the feature information of different regions on the first feature map corresponding to different sample images; an eleventh determination submodule, used to determine the second associated feature information based on the similarity between the feature information of different regions on the second feature map corresponding to different sample images.

[0027] In some optional embodiments, there are multiple first feature maps and multiple second feature maps, and the multiple first feature maps correspond to different resolutions, and the multiple second feature maps correspond to different resolutions; the first determination module includes: a twelfth determination submodule, which is used to determine the first associated feature information based on the feature information included in the first feature maps of different resolutions in order of resolution from large to small, and to determine the second associated feature information based on the feature information included in the second feature maps of different resolutions.

[0028] In some optional embodiments, the first training module includes: a training submodule, which is used to perform supervised training on the student network by using knowledge distillation.

[0029] In some optional embodiments, the sample image includes a physical image with a labeled classification result.

[0030] According to a fourth aspect of an embodiment of the present disclosure, an image classification device is provided, comprising: an image classification module, configured to input an image to be detected into an image classification network, and obtain a classification result to which a target object included in the image to be detected output by the image classification network belongs; the image classification network is a student network trained by the method described in any one of the first aspects.

[0031] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the neural network training method described in any one of the first aspect or the image classification method described in the second aspect.

[0032] According to a sixth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory to implement any one of the neural network training methods described in the first aspect or the image classification method described in the second aspect.

[0033] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0034] In the embodiment of the present disclosure, the sample image is input into the teacher network and the student network respectively, and a first feature map and a second feature map can be obtained. Based on the feature information included in the first feature map, the first associated feature information can be determined, and based on the feature information included in the second feature map, the second associated feature information can be determined. Further, based on the difference between the second associated feature information and the first associated feature information, the student network is supervised and trained. The present disclosure performs supervised training on the student network based on the feature information included in the first feature map output by the teacher network, so that the teacher network and the student network can be consistent in local associated features, and the performance of the student network finally trained is better. In addition, since the student network is usually smaller in scale and has more efficient processing performance than the teacher network, the student network trained using the technical solution provided by the present disclosure can be deployed in mobile devices such as mobile phones and tablets to achieve functions such as image classification.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0037] Figure 1 is a flow chart of a neural network training method according to an exemplary embodiment of the present disclosure;

[0038] Figure 2 is a flow chart of another neural network training method according to an exemplary embodiment of the present disclosure;

[0039] Figure 3 is a flow chart of another neural network training method according to an exemplary embodiment of the present disclosure;

[0040] Figure 4 is a flow chart of another neural network training method according to an exemplary embodiment of the present disclosure;

[0041] Figure 5 is a flow chart of another neural network training method according to an exemplary embodiment of the present disclosure;

[0042] Fig. 6A is a schematic diagram of a neural network training architecture according to an exemplary embodiment of the present disclosure;

[0043] Figure 6B is a schematic diagram showing a method of determining associated feature information according to an exemplary embodiment of the present disclosure;

[0044] Figure 7 is a block diagram of a neural network training device according to an exemplary embodiment of the present disclosure;

[0045] Figure 8 is a block diagram of an image classification device according to an exemplary embodiment of the present disclosure;

[0046] Fig. 9 It is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0048] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0049] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0050] Before introducing the neural network training scheme based on knowledge distillation provided by the present disclosure, the relevant content of knowledge distillation (KD) is first introduced. Knowledge distillation is essentially closer to transfer learning, and a student model that is more suitable for reasoning is obtained based on the trained teacher model. Among them, the teacher model has more parameters and better performance, while the student model has fewer parameters, but the reasoning speed is faster.

[0051] The loss function corresponding to the more commonly used KD method can be expressed by the following formula 1:

[0052]

[0053] Among them, τ is a hyperparameter called temperature, the teacher model is T, the student model is S, and x i For the sample set The i-th sample image in f T (x i ) and f S (x i ) respectively correspond to the teacher model and the student model for x i Output.

[0054] The student network is trained by minimizing the relative entropy (Kullback-Leibler, KL) divergence between the student model and the teacher model.

[0055] However, existing knowledge distillation methods focus on maintaining instance-level feature consistency or instance-level relationship consistency, but local features usually contain more detailed information and are important information for distinguishing different targets. For example, an image of an ocelot and an image of a leopard, because the two animals look very similar overall, if the overall instance-level feature consistency is compared, it will be misjudged that the two belong to the same category. However, if local features, such as the features of the head or feet, are used, the two can be accurately distinguished.

[0056] In addition, the classification sample images input to the network usually contain a lot of interference information that is irrelevant to the category, such as the background of the image or objects that block the main body of the image. In the process of knowledge distillation, existing methods will indiscriminately pass all the information contained in the image to the student model, which will cause greater redundancy or noise.

[0057] In order to solve the above problems, the embodiments of the present disclosure provide a neural network training scheme based on knowledge distillation, which can enable the student model to learn local features and improve the performance of the student network.

[0058] For example Figure 1 As shown, Figure 1 A neural network training method based on knowledge distillation is shown according to an exemplary embodiment, comprising the following steps:

[0059] In step 101, a sample image is input into a teacher network and a student network respectively to obtain a first feature map output by the teacher network and a second feature map output by the student network.

[0060] In the embodiments of the present disclosure, the sample images may include physical images with labeled classification results. The objects included in the physical images usually have an impact on the image classification results. The objects refer to content that is different from the image background, such as animals, plants, and vehicles. During the training process, the sample images may include but are not limited to images such as animal images, plant images, and vehicle images.

[0061] In the embodiment of the present disclosure, the resolution of the first feature map is higher than that of the second feature map, and / or the feature information included in the first feature map is richer than that in the second feature map.

[0062] In step 102, first associated feature information is determined based on feature information included in the first feature map, and second associated feature information is determined based on feature information included in the second feature map.

[0063] In the disclosed embodiment, the correlation features between different regions of the first feature map can be determined. For example, taking a sample image as an example, a first feature map corresponding to the sample image includes multiple regions, and the correlation features between each two regions can be used as the correlation features between different regions of the first feature map, so that the correlation relationship between different autumn rains in the same sample image can be reflected; in the case where the number of sample images is multiple, the correlation features between the same region and / or different regions of the first feature map corresponding to different sample images can also be determined. Taking two sample images as an example, in the case where each sample image corresponds to a first feature map, the correlation features between the same region in the first feature map corresponding to the two sample images, and / or the correlation features between different regions in the first feature map corresponding to the two sample images, can be used as the above-mentioned correlation features, so that the correlation relationship between the same region and / or different regions of different sample images can be reflected. Based on at least one of the above items, the first correlation feature information is obtained. Similarly, the second correlation feature information can be determined in the same or similar manner, which will not be repeated here.

[0064] In step 103, supervised training is performed on the student network based on the difference between the second associated feature information and the first associated feature information.

[0065] In the disclosed embodiment, the loss function may be determined based on the difference between the second associated feature information and the first associated feature information. Furthermore, the student network may be supervised and trained by using knowledge distillation.

[0066] In the above embodiment, based on the feature information included in the first feature map output by the teacher network, the student network is supervised and trained so that the teacher network and the student network can be consistent in local correlation features, and the performance of the student network finally trained is better.

[0067] In some optional embodiments, for example Figure 2 As shown, the above method may also include:

[0068] In step 104, a target area is determined based on the first feature map.

[0069] In the disclosed embodiment, the target area of ​​the sample image includes the target object (i.e., the content that plays a leading role in the image classification process, such as animal category, plant variety, vehicle style, etc.). In order to make the knowledge distillation process focus more on the target area on the image, the area irrelevant to the category features can be removed, that is, the area other than the target area in the sample image can be removed. These areas include but are not limited to the background part of the sample image and / or the part where the target object is blocked.

[0070] In step 105, a first target feature map is determined based on the target area and the first feature map, and a second target feature map is determined based on the target area and the second feature map.

[0071] In the embodiment of the present disclosure, after determining the target area of ​​the sample image, a first target feature map may be determined based on the target area and the first feature map, where the first target feature map is a feature map including feature information of the target area on the first feature map. Similarly, a second target feature map may be determined based on the target area and the second feature map. Accordingly, step 102 may include:

[0072] The associated feature information is determined based on the feature information included in the first target feature map, and the second associated feature information is determined based on the feature information included in the second target feature map.

[0073] In the above embodiment, the target area where the target object is located can be determined based on the first feature map, and then the first target feature map and the second target feature map can be determined respectively. The first associated feature information is determined based on the feature information included in the first target feature map, and the second associated feature information is determined based on the feature information included in the second target feature map, thereby reducing the redundancy or noise caused by information irrelevant to the category features in the knowledge distillation, improving the performance of the student network after knowledge distillation, and increasing the recognition accuracy of the student network.

[0074] In some optional embodiments, for example Figure 3 As shown, step 104 may include:

[0075] In step 104-1, the first feature map is input into a pre-trained first neural network to obtain a pixel normalization value corresponding to each of a plurality of regions included in the sample image output by the first neural network.

[0076] In an embodiment of the present disclosure, the first neural network includes a plurality of first network layers and a first output layer connected sequentially, wherein the plurality of first network layers include but are not limited to a convolutional layer (conv), a batch normalization layer (bn), and an activation layer (relu), and a threshold (sigmoid) layer can be sequentially connected after the plurality of first network layers as the first output layer.

[0077] Considering that the first feature map learned in the teacher model has more obvious feature information for category-related areas and less obvious feature information for category-irrelevant areas, it can better distinguish the target area from other areas. Therefore, the first feature map can be used as the input of the first neural network, and the first neural network outputs the pixel normalization value corresponding to each of the multiple areas included in the sample image. Among them, the multiple areas included in the sample image refer to part or all of the pixel areas of the entire sample image, and the pixel normalization value can be a continuous value in the range of [0, 1].

[0078] In one example, the first neural network may output an image mask whose size is consistent with the sample image size, and different regions identify the normalized pixel values ​​corresponding to the region.

[0079] In step 104 - 2 , the region where the pixel normalization value is greater than or equal to the preset value is taken as the target region.

[0080] In one example, the pixel normalization value corresponding to each of the multiple regions included in the sample image may be thresholded, for example, the pixel normalization value corresponding to the region whose pixel normalization value is greater than the preset value is set to threshold 1, and the pixel normalization values ​​corresponding to other regions are set to threshold 0. The boundary range of the target region can be determined based on the region whose pixel normalization value is 1 after thresholding.

[0081] In the above embodiment, the pixel normalization value corresponding to each of the multiple regions included in the sample image can be obtained through the pre-trained first neural network, and then the target region of the sample image can be determined, which is simple to implement and has high usability.

[0082] In some optional embodiments, for example Figure 4 As shown, step 105 may include;

[0083] In step 105-1, the normalized pixel value corresponding to each of the multiple regions is dot-multiplied with the feature information included in the first feature map to obtain a first attention feature map, and the normalized pixel value corresponding to each of the multiple regions is dot-multiplied with the feature information included in the second feature map to obtain a second attention feature map.

[0084] In an embodiment of the present disclosure, the normalized pixel value corresponding to each of the multiple areas included in the sample image can be dot-multiplied with the feature information included in the first feature map to obtain a first attention feature map that can highlight the foreground information related to the category and weaken the information unrelated to the category.

[0085] In the same way, the normalized pixel value corresponding to each area of ​​the multiple areas included in the sample image is multiplied by the feature information included in the second feature map to obtain the second attention feature map.

[0086] In step 105-2, the first target feature map is obtained based on the target area on the first attention feature map, and the second target feature map is obtained based on the target area on the second attention feature map.

[0087] In the disclosed embodiment, the boundary range of the target area is determined according to the thresholded mask, and the first candidate feature map can be cropped out from the first attention feature map based on the boundary range of the target area.

[0088] If the size of the first candidate feature map meets the preset size, such as k×k, the first candidate feature can be directly used as the first target feature map. If the size of the first candidate feature map does not meet the preset size, the first target feature map meeting the preset size can be obtained by reshaping the first candidate feature.

[0089] In the same way, multiple second target feature maps can be obtained.

[0090] In the above embodiment, an attention mechanism can be used to focus on the target area where the target object is located on the feature map, so that knowledge distillation of the student network can be performed based on the associated features of the target area to improve the performance of the student network.

[0091] In some optional embodiments, the first neural network includes multiple first network layers and a first output layer. In order to obtain a trained first neural network, a second neural network may be provided, the second neural network having multiple second network layers and a second output layer, wherein the network structure of the multiple second network layers is the same as the network structure of the multiple first network layers. For example, the multiple first network layers include M1 conv layers, M2 bn layers, and M3 relu layers, which are connected in sequence, then the multiple second network layers also include M1 conv layers, M2 bn layers, and M3 relu layers connected in sequence. Among them, M1, M2, and M3 can be integers greater than or equal to zero. If the value is zero, it means that the network layer does not exist. If the value is an integer greater than or equal to 2, it means that there are multiple identical network layers connected in sequence. For example, M1 is 2, and the multiple first network layers include 2 conv layers connected in sequence.

[0092] For example Figure 5 As shown, the above method may also include:

[0093] In step 100-1, the first feature map is used as an input of a second neural network to obtain a classification result output by a classification layer of the second neural network.

[0094] In the embodiment of the present disclosure, the second output layer included in the second neural network can adopt a classification layer to output a classification result corresponding to the first feature map.

[0095] In step 100-2, supervised training is performed on the second neural network based on the difference between the classification result output by the second neural network and the classification result annotated in the sample image.

[0096] In the disclosed embodiment, the classification results of the target objects have been pre-annotated in the sample image. The classification results annotated in the sample image are used as the true value of the classification results. According to the difference between the classification results output by the second neural network and the true value of the classification results, the corresponding loss function is determined, thereby performing supervised training on the second neural network.

[0097] In step 100-3, the network parameters of the multiple second network layers included in the trained second neural network are used as the network parameters of the multiple first network layers included in the first neural network.

[0098] In the embodiment of the present disclosure, since the structures of the first neural network and the second neural network are the same, the network parameters of the multiple second network layers included in the trained second neural network can be directly used as the network parameters of the multiple first network layers included in the first neural network to obtain the trained first neural network.

[0099] In the above embodiment, the classification labels in the sample images can be used as supervision, and a trained second neural network can be obtained by minimizing the difference between the classification results output by the second neural network and the true value of the classification results. Therefore, the network parameters of multiple first network layers of the first neural network can be determined based on the network parameters of the second neural network, which is simple to implement and has high usability.

[0100] In some optional embodiments, the process of determining the first associated feature information based on the feature information included in the first feature map in step 102 may include:

[0101] The first associated feature information is determined based on the similarity between feature information of different regions on the first feature map.

[0102] Accordingly, in step 102, the process of determining the second associated feature information based on the feature information included in the second feature graph may include:

[0103] The second associated feature information is determined based on the similarity between feature information of different regions on the second feature map.

[0104] Furthermore, step 103 can determine the corresponding loss function L according to the difference between the second associated feature information and the first associated feature information. intra , using knowledge distillation to supervise the training of the student network. The loss function L intra It can be expressed by the following formula 2:

[0105]

[0106] Where L is the total number of stages of knowledge distillation, which can be a positive integer. G(.) is a function for constructing the association matrix. is the L2 loss function. They represent the xth i The similarity between the feature information of different regions on the first feature map corresponding to the sample images, represents the xth i The similarity between the feature information of different regions on the second feature map corresponding to the sample images.

[0107] In some optional embodiments, the number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple. Generally, one sample image may correspond to one or more first feature maps, and one or more second feature maps.

[0108] In step 102, the process of determining the first associated feature information based on the feature information included in the first feature graph may include:

[0109] Determining the first associated feature information based on similarities between feature information of the same area on the first feature map corresponding to different sample images;

[0110] Accordingly, in step 102, the process of determining the second associated feature information based on the feature information included in the second feature graph may include:

[0111] The second associated feature information is determined based on the similarity between the feature information of the same area on the second feature map corresponding to different sample images.

[0112] Accordingly, the loss function L determined in step 103 is inter -s can be expressed by the following formula 3:

[0113]

[0114] in, represents the similarity between the feature information of the same region i on the first feature map corresponding to different sample images in the teacher network, Represents the similarity between the feature information of the same region i on the second feature map corresponding to different sample images in the student network.

[0115] Based on the loss function L inter -s, the student network can be supervised and trained by using knowledge distillation.

[0116] In some optional embodiments, there are multiple sample images, and there are multiple first feature maps and multiple second feature maps.

[0117] In step 102, the process of determining the first associated feature information based on the feature information included in the first feature graph may include:

[0118] Determining the first associated feature information based on similarities between feature information of different regions on the first feature map corresponding to different sample images;

[0119] Accordingly, in step 102, the process of determining the second associated feature information based on the feature information included in the second feature graph may include:

[0120] The second associated feature information is determined based on the similarity between feature information of different regions on the second feature map corresponding to different sample images.

[0121] Accordingly, the loss function L determined in step 103 is inter-d It can be expressed by the following formula 4:

[0122]

[0123] in, Represents different sample images x of the teacher network i and x j The similarity between the feature information of different regions p and q on the corresponding first feature map, Represents different sample images x of the student network i and x j The similarity between the feature information of different regions p and q on the corresponding second feature map.

[0124] Based on the loss function L inter-d , the knowledge distillation method can be used to supervise the training of the student network.

[0125] In the above embodiment, at least one method can be used to determine the first associated feature information and the second associated feature information, thereby obtaining a corresponding loss function for supervised training of the student network, so that the trained student network pays more attention to the local associated features of the image and has higher performance.

[0126] In some optional embodiments, a loss function used in knowledge distillation can be obtained based on the above loss function, as shown in Formula 5:

[0127] L LKD =L intra +L inter-s +L inter-d Formula 5

[0128] In the disclosed embodiment, since it is hoped that the student network will pay more attention to the feature information of the target area, Formula 5 can be transformed into Formula 6:

[0129]

[0130] in, and The loss function obtained by corresponding to the difference between the first associated feature information determined based on the first target feature map and the second associated feature information determined based on the second target feature map.

[0131] In some optional embodiments, in order to further improve the performance of the student network, Formula 7 may be used as the loss function during knowledge distillation:

[0132]

[0133] Among them, α and β are hyperparameters, L KD is the loss function determined by the commonly used knowledge distillation method, which can be obtained by formula 1, L CE Corresponding to the loss function determined by the cross entropy method.

[0134] In some optional embodiments, there are multiple first feature maps and multiple second feature maps, and the multiple first feature maps correspond to different resolutions, and the multiple second feature maps correspond to different resolutions.

[0135] Accordingly, step 102 may include:

[0136] In order from large to small resolutions, the first associated feature information is determined based on the feature information included in the first feature maps of different resolutions, and the second associated feature information is determined based on the feature information included in the second feature maps of different resolutions.

[0137] In the embodiments of the present disclosure, knowledge distillation may be performed in stages in the order of decreasing resolution, and the method of knowledge distillation in each stage may adopt the method provided in the above embodiments.

[0138] In the above embodiment, knowledge distillation can be performed in stages to improve the performance of the student network.

[0139] In some optional embodiments, for example Fig. 6A As shown, multiple sample images with classification labels are respectively input into the teacher network and the student network, and multiple first feature maps and multiple second feature maps can be obtained respectively. The process of knowledge distillation is divided into multiple stages, and the first associated feature information is determined based on the feature information included in the first feature maps of different resolutions, and the second associated feature information is determined based on the feature information included in the second feature maps of different resolutions in order from large to small resolutions.

[0140] For example, a sample image corresponds to multiple first feature maps with different resolutions, and the sample image corresponds to multiple second feature maps with different resolutions. In the first stage of knowledge distillation, the first feature map with the highest resolution corresponding to the sample image and the second feature map with the highest classification rate can be used. In the subsequent stages of knowledge distillation, the resolutions of the first feature map and the second feature map are gradually reduced. In the final stage of knowledge distillation, the first feature map with the lowest resolution corresponding to the sample image and the second feature map with the lowest resolution are used. Fig. 6A In the method, for multiple first feature maps of any stage, the target areas of different sample images are determined. The corresponding first attention feature map can be obtained by performing a dot multiplication operation on the target areas of different sample images and the feature information included in the corresponding first feature map. Similarly, the corresponding second attention feature map can be obtained by performing a dot multiplication operation on the target areas of different sample images and the feature information included in the corresponding first feature map. The target area is intercepted from the first attention feature map to obtain the first target feature map, and the target area is intercepted from the second attention feature map in the same way to obtain the second target feature map. According to the feature information included in the first target feature map, the first associated feature information can be determined. In the same way, based on the feature information included in the second target feature map, the second associated feature information can be determined. Then, based on the difference between the second associated feature information and the first associated feature information, the corresponding loss function is determined, and the student network is trained by using knowledge distillation.

[0141] The student network obtained in this way pays more attention to the feature information of the target area, reduces the redundancy and noise brought by the knowledge distillation process, and uses local correlation features to establish a loss function, so that the performance of the trained student network is better and the results are more accurate when performing image classification.

[0142] exist Fig. 6A In the method, each sample image is input into the teacher network and the student network respectively to obtain the first feature map output by the teacher network and the second feature map group output by the student network.

[0143] At any stage of knowledge distillation, the first feature map can be input into the pre-trained first neural network to obtain the normalized pixel value corresponding to each of the multiple regions included in the sample image, that is, to obtain the image mask corresponding to the sample image, and to perform dot multiplication of the image mask and the feature information included in the first feature map to obtain the first attention feature map. The second attention feature map can be obtained in the same way.

[0144] Furthermore, the image mask is thresholded to determine the boundary range of the target area, and the target area is cut out from the first attention feature map to obtain the first target feature map. In the same way, the second target feature map can be obtained.

[0145] In the embodiment of the present disclosure, the number of sample images is multiple, the number of first target feature maps and second feature maps is also multiple, and further, it can be adopted Figure 6B In the manner shown, the first associated feature information and the second associated feature information are determined. Figure 6B That is, for the feature information included in the multiple first target feature maps obtained in a certain knowledge distillation stage, the first associated feature information is determined, and for the feature information included in the multiple second target feature maps obtained in this stage, the second associated feature information is determined.

[0146] The first associated feature information can be obtained by determining at least one of the similarity between feature information of different regions on the same first feature map, the similarity between feature information of the same region on first feature maps corresponding to different sample images, and the similarity between feature information of different regions on first feature maps corresponding to different sample images. The second associated feature information can be obtained in the same manner.

[0147] After determining the corresponding loss function according to the above formula 7, the student network is supervised and trained at different stages using knowledge distillation.

[0148] In the above embodiment, more attention is paid to the target area related to the category on the image, and knowledge distillation is performed based on local relationship features, which not only reduces the redundancy and noise in the knowledge distillation process, but also improves the performance of the student network.

[0149] In some optional embodiments, the present disclosure further provides an image classification method, comprising the following steps:

[0150] In step 201, the image to be detected is input into an image classification network, and a classification result to which the target object included in the image to be detected belongs is obtained which is output by the image classification network.

[0151] In the disclosed embodiment, the image classification network is a student network trained by any of the methods described above. After the image to be detected is input into the trained student network, the student network outputs the classification result to which the target object included in the image to be detected belongs, and the classification result includes but is not limited to the category to which the target object belongs, or the categories to which the target object may belong, and the corresponding probabilities and / or scores, etc. For example, the probability that the target object belongs to the first category is 60%, the probability that it belongs to the second category is 15%, the probability that it belongs to the third category is 15%, and the probability that it belongs to the fourth category is 10%, wherein the first to fourth categories may be dissimilar or partially similar categories.

[0152] In the above embodiment, the image classification network adopts the student network obtained by the above training. Since during the training process, the student network pays more attention to the target area where the target object is located and the associated features of the target area, the trained student network can highlight the response value of the target area in the image to be detected during the inference process of image classification, so that the output classification result is more accurate and reliable.

[0153] Corresponding to the aforementioned method embodiments, the present disclosure also provides device embodiments.

[0154] like Figure 7 As shown, Figure 7 It is a block diagram of a neural network training device shown in accordance with an exemplary embodiment of the present disclosure, the device comprising: an acquisition module 310, used to input sample images into a teacher network and a student network respectively, to obtain a first feature map output by the teacher network, and a second feature map output by the student network; a first determination module 320, used to determine first associated feature information based on feature information included in the first feature map, and to determine second associated feature information based on feature information included in the second feature map; a first training module 330, used to perform supervised training on the student network based on the difference between the second associated feature information and the first associated feature information.

[0155] In some optional embodiments, the device also includes: a second determination module, used to determine the target area based on the first feature map; a third determination module, used to determine the first target feature map based on the target area and the first feature map, and to determine the second target feature map based on the target area and the second feature map; the first determination module includes: a first determination submodule, used to determine the first associated feature information based on the feature information included in the first target feature map, and to determine the second associated feature information based on the feature information included in the second target feature map.

[0156] In some optional embodiments, the target area of ​​the sample image includes a target object, and / or the area of ​​the sample image other than the target area includes at least one of the following: a background portion, an occluded portion of the target object.

[0157] In some optional embodiments, the second determination module includes: a second determination submodule, used to input the first feature map into a pre-trained first neural network to obtain a pixel normalization value corresponding to each of the multiple regions included in the sample image output by the first neural network; and a third determination submodule, used to take the region whose pixel normalization value is greater than or equal to a preset value as the target region.

[0158] In some optional embodiments, the third determination module includes: a fourth determination submodule, used to perform a dot product of the normalized pixel value corresponding to each of the multiple regions with the feature information included in the first feature map to obtain a first attention feature map, and perform a dot product of the normalized pixel value corresponding to each of the multiple regions with the feature information included in the second feature map to obtain a second attention feature map; a fifth determination submodule, used to obtain the first target feature map based on the target area on the first attention feature map, and to obtain the second target feature map based on the target area on the second attention feature map.

[0159] In some optional embodiments, the first neural network includes multiple first network layers and a first output layer, and the second neural network includes multiple second network layers and a second output layer, wherein the network structure of the multiple second network layers is the same as the network structure of the multiple first network layers; the device also includes: a classification result acquisition module, used to use the first feature map as the input of the second neural network to obtain the classification result output by the second neural network; a second training module, used to supervise the second neural network based on the difference between the classification result output by the second neural network and the classification result marked in the sample image; a fourth determination module, used to use the network parameters of the multiple second network layers included in the trained second neural network as the network parameters of the multiple first network layers included in the first neural network.

[0160] In some optional embodiments, the first determination module includes: a sixth determination submodule, used to determine the first associated feature information based on the similarity between feature information of different regions on the first feature map; and a seventh determination submodule, used to determine the second associated feature information based on the similarity between feature information of different regions on the second feature map.

[0161] In some optional embodiments, the number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple; the first determination module includes: an eighth determination submodule, used to determine the first associated feature information based on the similarity between the feature information of the same area on the first feature map corresponding to different sample images; a ninth determination submodule, used to determine the second associated feature information based on the similarity between the feature information of the same area on the second feature map corresponding to different sample images.

[0162] In some optional embodiments, the number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple; the first determination module includes: a tenth determination submodule, used to determine the first associated feature information based on the similarity between the feature information of different regions on the first feature map corresponding to different sample images; an eleventh determination submodule, used to determine the second associated feature information based on the similarity between the feature information of different regions on the second feature map corresponding to different sample images.

[0163] In some optional embodiments, there are multiple first feature maps and multiple second feature maps, and the multiple first feature maps correspond to different resolutions, and the multiple second feature maps correspond to different resolutions; the first determination module includes: a twelfth determination submodule, which is used to determine the first associated feature information based on the feature information included in the first feature maps of different resolutions in order of resolution from large to small, and to determine the second associated feature information based on the feature information included in the second feature maps of different resolutions.

[0164] In some optional embodiments, the first training module includes: a training submodule, which is used to perform supervised training on the student network by using knowledge distillation.

[0165] In some optional embodiments, the sample image includes a physical image with a labeled classification result.

[0166] like Figure 8 As shown, Figure 8 It is a block diagram of an image classification device shown in the present disclosure according to an exemplary embodiment, and the device includes: an image classification module 410, which is used to input the image to be detected into an image classification network to obtain a classification result to which the target object included in the image to be detected output by the image classification network belongs; the image classification network is a student network trained by any of the methods described above.

[0167] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art may understand and implement it without creative work.

[0168] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and the computer program is used to execute any of the above-mentioned neural network training methods or image classification methods.

[0169] In some optional embodiments, the embodiments of the present disclosure provide a computer program product, including computer-readable code. When the computer-readable code is run on a device, a processor in the device executes instructions for implementing the neural network training method or the image classification method provided in any of the above embodiments.

[0170] In some optional embodiments, the embodiments of the present disclosure further provide another computer program product for storing computer-readable instructions, which, when executed, enable the computer to execute the neural network training method or the image classification method provided in any of the above embodiments.

[0171] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0172] An embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to call the executable instructions stored in the memory to implement any of the above-mentioned neural network training methods or image classification methods.

[0173] Fig. 9 The hardware structure diagram of an electronic device provided in an embodiment of the present disclosure is shown in FIG. The electronic device 510 includes a processor 511, and may also include an input device 512, an output device 513, and a memory 514. The input device 512, the output device 513, the memory 514, and the processor 511 are interconnected via a bus.

[0174] The memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable read-only memory (CD-ROM), which is used for related instructions and data.

[0175] The input device is used to input data and / or signals, and the output device is used to output data and / or signals. The output device and the input device can be independent devices or an integrated device.

[0176] The processor may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0177] The memory is used to store program codes and data of the network device.

[0178] The processor is used to call the program code and data in the memory to execute the steps in the above method embodiment. For details, please refer to the description in the method embodiment, which will not be repeated here.

[0179] Understandably, Fig. 9 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, controllers, memories, etc., and all devices that can implement the neural network training method or the image classification method of the embodiment of the present disclosure are within the protection scope of the present disclosure.

[0180] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0181] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A neural network training method, characterized in that: include: Inputting the sample image into the teacher network and the student network respectively, obtaining a first feature map output by the teacher network and a second feature map output by the student network, wherein the number of the first feature map and the number of the second feature map are multiple, and the multiple first feature maps correspond to different resolutions, and the multiple second feature maps correspond to different resolutions; Determining first associated feature information based on feature information included in the first feature map, and determining second associated feature information based on feature information included in the second feature map, includes: determining the first associated feature information based on feature information included in the first feature maps with different resolutions, and determining the second associated feature information based on feature information included in the second feature maps with different resolutions, in descending order of resolution; The associated feature information includes first associated feature information and second associated feature information; in the case of one sample image, the associated feature information refers to the associated features between different regions in the feature map of the sample image; in the case of multiple sample images, the associated feature information refers to the associated features between the same region and / or different regions in the feature maps corresponding to different sample images; Based on the difference between the second associated feature information and the first associated feature information, supervised training is performed on the student network.

2. The method according to claim 1, characterized in that: The method further comprises: Based on the first feature map, determining a target area; Determine a first target feature map based on the target area and the first feature map, and determine a second target feature map based on the target area and the second feature map; The determining of first associated feature information based on feature information included in the first feature graph, and determining of second associated feature information based on feature information included in the second feature graph, includes: The first associated feature information is determined based on the feature information included in the first target feature map, and the second associated feature information is determined based on the feature information included in the second target feature map.

3. The method according to claim 2, characterized in that The target area of ​​the sample image includes a target object, and / or the area other than the target area in the sample image includes at least one of the following: a background part, and a part of the target object that is blocked.

4. The method according to claim 2, characterized in that: The determining the target area based on the first feature map includes: Inputting the first feature map into a pre-trained first neural network to obtain a pixel normalization value corresponding to each of a plurality of regions included in the sample image output by the first neural network; The area where the pixel normalization value is greater than or equal to the preset value is taken as the target area.

5. The method according to claim 4, characterized in that The determining a first target feature map based on the target area and the first feature map, and determining a second target feature map based on the target area and the second feature map, comprises: Performing a dot multiplication of the normalized pixel value corresponding to each of the multiple regions with the feature information included in the first feature map to obtain a first attention feature map, and performing a dot multiplication of the normalized pixel value corresponding to each of the multiple regions with the feature information included in the second feature map to obtain a second attention feature map; Based on the target area on the first attention feature map, the first target feature map is obtained, and based on the target area on the second attention feature map, the second target feature map is obtained.

6. The method according to claim 4, characterized in that The first neural network includes a plurality of first network layers and a first output layer, and the second neural network includes a plurality of second network layers and a second output layer, wherein the network structure of the plurality of second network layers is the same as the network structure of the plurality of first network layers; The method further comprises: Using the first feature map as an input of the second neural network to obtain a classification result output by the second neural network; Based on the difference between the classification result output by the second neural network and the classification result marked in the sample image, performing supervised training on the second neural network; The network parameters of the multiple second network layers included in the trained second neural network are used as the network parameters of the multiple first network layers included in the first neural network.

7. The method according to claim 1, characterized in that The determining, based on the feature information included in the first feature graph, first associated feature information includes: Determining the first associated feature information based on the similarity between feature information of different regions on the first feature map; The determining the second link feature information based on the feature information included in the second feature graph includes: The second associated feature information is determined based on the similarity between feature information of different regions on the second feature map.

8. The method according to claim 1, characterized in that The number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple; The determining, based on the feature information included in the first feature graph, first associated feature information includes: Determining the first associated feature information based on similarities between feature information of the same area on the first feature map corresponding to different sample images; The determining the second associated feature information based on the feature information included in the second feature graph includes: The second associated feature information is determined based on the similarity between the feature information of the same area on the second feature map corresponding to different sample images.

9. The method according to claim 1, characterized in that: The number of the sample images is multiple, and the number of the first feature maps and the second feature maps is multiple; The determining, based on the feature information included in the first feature graph, first associated feature information includes: Determining the first associated feature information based on similarities between feature information of different regions on the first feature map corresponding to different sample images; The determining the second associated feature information based on the feature information included in the second feature graph includes: The second associated feature information is determined based on the similarity between feature information of different regions on the second feature map corresponding to different sample images.

10. The method according to claim 1, characterized in that The supervised training of the student network comprises: The student network is supervised and trained by using knowledge distillation.

11. The method according to any one of claims 1 to 10, characterized in that: The sample images include physical images with labeled classification results.

12. An image classification method, characterized in that: The method comprises: Inputting the image to be detected into an image classification network, and obtaining a classification result to which the target object included in the image to be detected belongs, output by the image classification network; The image classification network is a student network trained by the method described in any one of claims 1 to 10.

13. A neural network training device, characterized in that: include: An acquisition module is used to input the sample image into the teacher network and the student network respectively, to obtain a first feature map output by the teacher network and a second feature map output by the student network, wherein the number of the first feature map and the second feature map is multiple, and the multiple first feature maps correspond to different resolutions, and the multiple second feature maps correspond to different resolutions; A first determination module, used to determine first associated feature information based on feature information included in the first feature map, and to determine second associated feature information based on feature information included in the second feature map, including: determining the first associated feature information based on feature information included in the first feature map with different resolutions, and determining the second associated feature information based on feature information included in the second feature map with different resolutions in order of resolution from large to small; the associated feature information includes first associated feature information and second associated feature information; in the case of one sample image, the associated feature information refers to associated features between different regions in the feature map of the sample image; in the case of multiple sample images, the associated feature information refers to associated features between the same region and / or different regions in feature maps corresponding to different sample images; The first training module is used to perform supervised training on the student network based on the difference between the second associated feature information and the first associated feature information.

14. An image classification device, characterized in that: The device comprises: An image classification module is used to input the image to be detected into an image classification network, and obtain a classification result to which the target object included in the image to be detected belongs, which is output by the image classification network; The image classification network is a student network trained by the method described in any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the neural network training method described in any one of claims 1 to 11 or the image classification method described in claim 12.

16. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to call the executable instructions stored in the memory to implement the neural network training method described in any one of claims 1 to 11 or the image classification method described in claim 12.

Images