Student network processing method and apparatus, and electronic device

Abstract

The present application provides a kind of student network processing method, device and electronic equipment.Therein, the method comprises: obtaining the image sample of student network;Image sample is input into the teacher network of training completion, and the first output image and at least one first intermediate feature map of image sample are obtained;First intermediate feature map and first output image are based on the first saliency feature of image sample extraction, and first intermediate feature map is based on the first position feature of image sample extraction;First saliency feature and first position feature are based on, and image sample trains student network.The way, student network can learn the relative as position information between object and object according to first position feature, to improve the performance of model;Student network is based on first saliency feature training, not only learn the saliency feature in image from spatial angle, also extract the saliency of image from channel feature, to strengthen the learning ability of model to saliency feature.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and in particular to a student network processing method and device and electronic equipment. BACKGROUND

[0002] At present, most of the knowledge distillation schemes are designed for image classification networks, and only a few knowledge distillation schemes for semantic segmentation are based on the structural feature knowledge of the teacher network output space to distill the student network. This method inherited from the traditional image recognition distillation scheme is not efficient enough, and also ignores the characteristics of the semantic segmentation task itself.

[0003] The semantic segmentation task usually requires the model to learn the saliency information and pixel position information of the image. The existing technical solutions do not design a knowledge distillation scheme for the characteristics of the semantic segmentation task. Secondly, the existing technical solutions usually design a complex and inefficient adversarial learning network to obtain the structural feature knowledge of the teacher output space, making the training process particularly complex. And this structural feature knowledge is only for the teacher output space, and the label containing rich semantic structural information is missed, and the performance is poor. SUMMARY

[0004] Therefore, the present application aims to provide a student network processing method and device and electronic equipment to improve the performance of neural networks and enhance the model's learning ability of saliency features.

[0005] In a first aspect, the present application provides a student network processing method, which comprises: obtaining an image sample of a student network; for each image sample, performing the following operations: inputting the image sample into a trained teacher network to obtain a first output image of the image sample and at least one first intermediate feature map; wherein the first intermediate feature map is output by at least one intermediate feature layer of the teacher network, and the first output image is output by an output layer of the teacher network; extracting a first saliency feature of the image sample based on the first intermediate feature map and the first output image, and extracting a first position feature of the image sample based on the first intermediate feature map; wherein the first saliency feature represents the saliency information of the first intermediate feature map and the first output image, and the first position feature represents the position information of each pixel of the first intermediate feature map; and training the student network based on the first saliency feature and the first position feature, and the image sample.

[0006] In the preferable embodiment of the present application, the step of extracting the first saliency feature of the image sample based on the first intermediate feature map and the first output image comprises: performing a preset first feature probability calculation on the first intermediate feature map to obtain a first channel feature probability; performing the first feature probability calculation on the first output image to obtain a second channel feature probability; and taking the first channel feature probability and the second channel feature probability as the first saliency feature.

[0007] In the preferable embodiment of the present application, the step of extracting the first saliency feature of the image sample based on the first intermediate feature map and the first output image comprises: performing a preset first feature probability calculation on the first intermediate feature map to obtain a first channel feature probability; performing the first feature probability calculation on the first output image to obtain a second channel feature probability; and taking the first channel feature probability and the second channel feature probability as the first saliency feature. The step of extracting the second saliency feature of the image sample based on the second intermediate feature map and the second output image comprises: performing the first feature probability calculation on the second intermediate feature map to obtain a third channel feature probability; performing the first feature probability calculation on the second output image to obtain a fourth channel feature probability; and taking the third channel feature probability and the fourth channel feature probability as the second saliency feature. The step of training the student network based on the first saliency feature and the second saliency feature comprises: calculating a first KL divergence of the first channel feature probability and the third channel feature probability, and adjusting the intermediate feature layer parameters of the student network based on the first KL divergence; calculating a second KL divergence of the second channel feature probability and the fourth channel feature probability; adjusting the output layer parameters of the student network based on the second KL divergence; and continuing to train the student network until the training is completed.

[0008] In the preferable embodiment of the present application, the feature probability calculation comprises: converting the 3-dimensional tensor feature of the input image into a 1-dimensional tensor feature; and calculating the 1-dimensional tensor feature based on a preset first Softmax function to obtain the channel feature probability corresponding to the input image.

[0009] In the preferred embodiment of the present application, the step of extracting the first position feature of the image sample based on the first intermediate feature map comprises: performing a preset second feature probability calculation on the first intermediate feature map to obtain a first position feature probability, and taking the first position feature probability as the first position feature; the step of extracting the second position feature of the image sample based on the second intermediate feature map comprises: performing a second feature probability calculation on the second intermediate feature map to obtain a second position feature probability, and taking the second position feature probability as the second position feature; and the step of training the student network based on the first position feature and the second position feature comprises: calculating a first L2 loss value of the first position feature probability and the second position feature probability; adjusting the parameters of the intermediate feature layer of the student network based on the first L2 loss value, and continuing to train the student network until the training is completed.

[0010] In the preferred embodiment of the present application, the position feature probability calculation comprises: for each target pixel of the input image, calculating a nonlinear feature of the target pixel; based on the nonlinear feature and a preset second Softmax function, calculating a response value of each other pixel of the input image to the target pixel, and taking each response value as a position feature probability of the target pixel; wherein the other pixel is a pixel of the input image other than the target pixel, and the response value of the other pixel to the target pixel represents the position information of the target pixel relative to the other pixel.

[0011] In the preferred embodiment of the present application, after the step of inputting the image sample into the trained teacher network, the method further comprises: obtaining a first label of the image sample; and training the student network based on the first output image, the first label and the image sample.

[0012] In the preferred embodiment of the present application, the step of training the student network based on the first output image and the first label comprises: training an autoencoder based on the first output image and the first label; wherein the autoencoder comprises an encoder and a decoder; adjusting the parameters of the output layer of the student network based on the trained autoencoder and the image sample, and continuing to train the student network until the training is completed.

[0013] In the preferred embodiment of the present application, the step of training the autoencoder based on the first output image and the first label comprises: inputting the first output image and the first label into the encoder to perform dimension reduction and obtain a reduced dimension image; performing dimension increase on the reduced dimension image through the decoder to obtain an increased dimension image; calculating a second L2 loss value based on the first output image, the first label and the increased dimension image; adjusting the parameters of the autoencoder based on the second L2 loss value, and continuing to train the autoencoder until the training is completed.

[0014] In the preferred embodiment of the present application, the step of adjusting the parameters of the output layer of the student network based on the trained autoencoder and the image sample comprises: inputting the image sample into the student network to obtain a third output image of the image sample and a second label of the image sample; inputting the third output image and the second label into the autoencoder to obtain an upsampled image of the image sample in the student network; calculating a third L2 loss value based on the third output image, the second label and the upsampled image in the student network; and adjusting the parameters of the output space of the student network based on the third L2 loss value.

[0015] In the preferred embodiment of the present application, the teacher network and the student network are both semantic segmentation neural networks.

[0016] In the preferred embodiment of the present application, after the step of training the student network based on the first saliency feature and the first position feature and the image sample, the method further comprises: obtaining a target image; and inputting the target image into the trained student network to obtain an optimized image.

[0017] In a second aspect, the embodiments of the present application further provide a student network processing apparatus, which comprises: an image sample obtaining module configured to obtain image samples of a student network; and for each image sample, the following operations are performed: a teacher network output module configured to input the image sample into a trained teacher network to obtain a first output image of the image sample and at least one first intermediate feature map; wherein the first intermediate feature map is output by at least one intermediate feature layer of the teacher network, and the first output image is output by an output layer of the teacher network; a saliency feature and position feature extraction module configured to extract a first saliency feature of the image sample based on the first intermediate feature map and the first output image, and extract a first position feature of the image sample based on the first intermediate feature map; wherein the first saliency feature represents saliency information of the first intermediate feature map and the first output image, and the first position feature represents position information of each pixel of the first intermediate feature map; and a student network training module configured to train the student network based on the first saliency feature and the first position feature and the image sample.

[0018] In a third aspect, the embodiments of the present application further provide an electronic device, which comprises a processor and a memory, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the steps of the student network processing method described above.

[0019] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the steps of the student network processing method described above.

[0020] The embodiments of the present application bring the following beneficial effects:

[0021] The student network processing method, device and electronic equipment provided by the embodiments of the present application output a first output image and at least one first intermediate feature map of an image sample from a trained teacher network, extract a first saliency feature of the image sample from the first output image and the first intermediate feature map, extract a first position feature of the image sample from the first intermediate feature map, and train a student network according to the first saliency feature, the first position feature and the image sample. In this way, the student network can learn the relative position information between objects and objects according to the first position feature, thereby improving the performance of the model; the student network is trained based on the first saliency feature, not only learning the saliency feature in the image from the spatial angle, but also extracting the saliency of the image from the channel feature, thereby strengthening the learning ability of the model to the saliency feature.

[0022] Other features and advantages of the present disclosure will be described in the following description, or can be inferred from the description or determined without doubt, or can be known by implementing the above-mentioned technologies of the present disclosure.

[0023] In order to make the above-mentioned purposes, features and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0025] Figure 1 The flowchart of the student network processing method provided by the embodiments of the present application is shown in the figure;

[0026] Figure 2 The flowchart of another student network processing method provided by the embodiments of the present application is shown in the figure;

[0027] Figure 3 The schematic diagram of the student network processing method provided by the embodiments of the present application is shown in the figure;

[0028] Figure 4 The structural schematic diagram of the student network processing device provided by the embodiments of the present application is shown in the figure;

[0029] Figure 5 The structural schematic diagram of another student network processing device provided by the embodiments of the present application is shown in the figure;

[0030] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0032] At present, the method inherited from the traditional image recognition distillation scheme is not efficient enough, and the characteristics of the semantic segmentation task itself are also ignored. The existing technical solutions usually design complex and inefficient adversarial learning networks to obtain the structural feature knowledge of the teacher output space, which makes the training process particularly complex, only for the teacher output space, and misses the labels containing rich semantic structural information, and the performance is poor. Based on this, the student network processing method, device and electronic equipment provided by the embodiments of the present application can be applied to servers, mobile phones, tablet computers, computers and other devices. The technology can belong to the field of computers and deep learning, and specifically relates to a visual-based deep neural network model compression method. The technology can be implemented by using corresponding software and hardware, and the embodiments of the present application will be described in detail below.

[0033] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0034] Embodiment one:

[0035] The embodiment provides a student network processing method, and a flowchart of a student network processing method is shown in the figure. Figure 1 The student network processing method comprises the following steps:

[0036] Step S102, acquiring an image sample of a student network.

[0037] The method provided in the embodiment can be used for knowledge distillation, wherein the knowledge distillation can refer to inducing the training of a student network (student network: simplified, low complexity) by introducing a soft target (soft-target) related to a teacher network (teacher network: complex, but superior in reasoning performance) as part of a loss function, to realize knowledge transfer (knowledge transfer). The software target in the embodiment can refer to the saliency feature and the position feature below.

[0038] In simple terms, knowledge distillation can be understood as using a complex teacher network to help train a simplified student network. Therefore, the teacher network can be a large neural network, and the student network can be a small neural network. In this embodiment, the image sample of the student network can be obtained first. The image sample can be an image, a video frame in a video, etc. The image sample can be obtained from a training set, which can be: ImageNet dataset, PASCAL VOC dataset, Labelme dataset, COCO dataset, etc., or a user's own image collection.

[0039] For each image sample, the following operations are performed:

[0040] Step S104, input the image sample into the trained teacher network to obtain a first output image of the image sample and at least one first intermediate feature map; wherein the first intermediate feature map is output by at least one intermediate feature layer of the teacher network, and the first output image is output by the output layer of the teacher network.

[0041] The teacher network in this embodiment is pre-trained, so the parameters in the teacher network can be fixed and no longer adjusted. The teacher network can include at least one intermediate feature layer and an output layer. After inputting the image sample into the trained teacher network, the intermediate feature layer of the teacher network can output the first intermediate feature map, and the output layer of the teacher network can output the first output image.

[0042] It should be noted here that generally, each intermediate feature layer of the teacher network can output an intermediate feature map, however, in this embodiment, a certain number of intermediate feature maps can be selected as the first intermediate feature map. Because if the number of first intermediate feature maps is too large, it may affect the speed of training the student network. For example: the teacher network includes 10 intermediate feature layers, which can output 10 intermediate feature maps respectively, and 5 of them can be selected as the first intermediate feature map.

[0043] Step S106, extracting a first saliency feature of the image sample based on the first intermediate feature map and the first output image, and extracting a first position feature of the image sample based on the first intermediate feature map; wherein the first saliency feature represents the saliency information of the first intermediate feature map and the first output image, and the first position feature represents the position information of each pixel of the first intermediate feature map.

[0044] In this embodiment, the first saliency feature of the image sample can be extracted from the first intermediate feature map and the first output image. By squeezing the first intermediate feature map and the first output image, the saliency information of the entire feature map is contained in each channel dimension of the first intermediate feature map and the first output image.

[0045] The first position feature of the image sample can be extracted from the first intermediate feature map in the embodiment, and the first position feature can be relative position information of each pixel of the first intermediate feature map to other pixels of the first intermediate feature map.

[0046] In summary, the first saliency feature of the image sample can be extracted based on the first intermediate feature map and the first output image, and the first position feature of the image sample can be extracted based on the first intermediate feature map, so that the first saliency feature and the first position feature are used to assist in training the student network subsequently.

[0047] In step S108, the student network is trained based on the first saliency feature and the first position feature and the image sample.

[0048] The student network is trained based on the first saliency feature, so that the trained student network can imitate the channel saliency information learned by the teacher network; the student network is trained based on the first position feature, so that the trained student network can imitate the position information learned by the teacher network, and the learning mode of the student network combines the characteristics of the semantic segmentation task itself, is efficient, and has good performance.

[0049] The student network processing method provided in the embodiment comprises the following steps: a first output image and at least one first intermediate feature map of an image sample are output by a trained teacher network; a first saliency feature of the image sample is extracted from the first output image and the first intermediate feature map; a first position feature of the image sample is extracted from the first intermediate feature map; and a student network is trained based on the first saliency feature, the first position feature and the image sample. In this way, the student network can learn the relative position information between objects based on the first position feature, so as to improve the performance of the model; the student network is trained based on the first saliency feature, so that the saliency of the image is extracted not only from the spatial angle but also from the channel feature, thereby enhancing the learning ability of the model to the saliency feature.

[0050] Embodiment two

[0051] The embodiment provides another student network processing method, which is implemented on the basis of the above-mentioned embodiment; and the embodiment mainly describes a specific implementation manner of training the student network based on the first saliency feature and the first position feature and the image sample. Referring to the flowchart of another student network processing method shown in Figure 2 The student network processing method in the embodiment comprises the following steps:

[0052] In step S202, an image sample of a student network is acquired.

[0053] Currently, the knowledge distillation scheme for semantic segmentation ignores the characteristics of the semantic segmentation task itself. The semantic segmentation task usually requires the model to learn saliency features and pixel position features. The method provided in this embodiment learns saliency features and pixel position features, so the method provided in this embodiment can be applied in a semantic segmentation neural network, that is, the teacher network and the student network are both semantic segmentation neural networks. The teacher network can be a large semantic segmentation neural network, and the student network can be a small semantic segmentation neural network.

[0054] For each image sample, the following operations are performed:

[0055] Step S204, inputting the image sample into the trained teacher network to obtain a first output image of the image sample and at least one first intermediate feature map; wherein the first intermediate feature map is output by at least one intermediate feature layer of the teacher network, and the first output image is output by an output layer of the teacher network.

[0056] Referring to Figure 3 , a schematic diagram of a student network processing method is shown as Figure 3 , a schematic diagram of a teacher network is shown above Figure 3 , it can be seen that Figure 3 The teacher network in the teacher network includes two intermediate feature layers and an output layer, which respectively output two first intermediate feature maps and a first output image.

[0057] The neural network is divided into input, intermediate layer and output layer, and the overall part of the intermediate network (including convolutional layer, batch-normalization layer and nonlinear layer, etc.) can be called intermediate layer. The output layer can refer to the probability distribution of an output composed of all outputs, which can be simply understood as a very large number of output items.

[0058] Step S206, extracting a first saliency feature of the image sample based on the first intermediate feature map and the first output image, and extracting a first position feature of the image sample based on the first intermediate feature map; wherein the first saliency feature represents the saliency information of the first intermediate feature map and the first output image, and the first position feature represents the position information of each pixel of the first intermediate feature map.

[0059] In this embodiment, the channel feature probability can be used as the saliency feature, so the first saliency feature can be calculated by the following method: performing a preset first feature probability calculation on the first intermediate feature map to obtain a first channel feature probability; performing the first feature probability calculation on the first output image to obtain a second channel feature probability; and taking the first channel feature probability and the second channel feature probability as the first saliency feature.

[0060] Let Figure 3As shown, in the embodiment, the channel saliency module can be configured to calculate saliency features of the teacher network and the student network, and train the student network according to the calculated saliency features. When calculating the saliency features, the tensor features of the input image can be squeezed, for example, the 3D tensor features of the input image are converted into 1D tensor features; the 1D tensor features are calculated based on a preset first Softmax function to obtain channel feature probabilities corresponding to the input image.

[0061] The input image for feature probability calculation in the embodiment can be the first intermediate feature map and the first output image of the teacher network, and the second intermediate feature map and the second output image of the student network. Taking the teacher network as an example, the 3D tensor of the first intermediate feature map and the first output image of the teacher network can be squeezed, and the feature space of the original CxHxW (where C represents the channel, H represents the height, and W represents the length) is changed into 1D tensor features, that is, CxHxW→C. Therefore, the saliency information of the entire feature map is contained in each channel dimension of the squeezed 1D tensor features.

[0062] The feature space of the 3D tensor features is converted into the feature space of the 1D tensor features, which reduces the dimension, but the saliency information of the entire feature map is contained in each channel dimension of the 1D tensor features. The saliency information (or saliency map) can include important foreground objects in the feature image in computer vision. Each dimension C actually includes the result of HxW addition, so the current 1D tensor features actually include spatial information.

[0063] After obtaining the 1D tensor features, the channel feature probabilities corresponding to the input image can be obtained using a preset first Softmax function. The Softmax function, also known as the normalized exponential function, is a generalization of the binary classification function Sigmoid in multi-classification, and the purpose is to present the results of multi-classification in the form of probability. For example, the probability that the current vector x belongs to the jth classification is:

[0064] In the embodiment, the feature probability can be used as the position feature, so the first position feature can be calculated by the following method: performing a preset second feature probability calculation on the first intermediate feature map to obtain a first position feature probability, and taking the first position feature probability as the first position feature.

[0065] Let Figure 3 As shown, in the embodiment, the position feature module can be configured to calculate the position features of the teacher network and the student network, and train the student network according to the calculated position features. For example, the position feature module can calculate the position feature probability by the following method:

[0066] For each target pixel of the input image, a nonlinear feature of the target pixel is calculated; based on the nonlinear feature and a preset second Softmax function, a response value of each other pixel of the input image to the target pixel is calculated, and each response value is taken as a position feature probability of the target pixel; wherein the other pixel is a pixel of the input image other than the target pixel, and the response value of the other pixel to the target pixel represents position information of the target pixel relative to the other pixel.

[0067] For each target pixel of the input image, the Softmax probability (i.e. the response value) of the target pixel and other pixels of the input image can be calculated, so that the relative position information of each pixel can be obtained.

[0068] For example, for each target pixel, three learnable nonlinear features can be obtained first, and then the second Softmax function is calculated after merging the three nonlinear features, so as to determine the position of the target pixel and which other pixel in the image can obtain the strongest response.

[0069] The second Softmax function can obtain the probability distribution of the current specific target pixel to all the remaining pixels, and a larger value represents a higher response, which also indicates that the pixel at this position has a higher correlation with the target pixel, that is, the relative position is obtained.

[0070] For example, the position feature module can be used to determine the relative position information of the object in the input image, such as the positions of the facial features in a face, which is the information that the position feature module wants to obtain. Specifically, for each target pixel i in the image, there are (w x h-1) pixel points, where w and h represent the length and width of the input image, and i represents the coordinate position of the target pixel. For each i, the response of any other pixel point to j can be obtained, and the response can be obtained by calculating the second Softmax function.

[0071] Specifically, for a position feature module F, we can write:

[0072] For the pixel position i in the image, we want to obtain the response of all j positions. f u is a unary equation that does not require the information of position i, but only the response of the current position j. Usually, we can use f u (x j )=W m x j / ∑ j W m x j , where Wm is a set of weights, which we obtain through a linear transformation. p is a dual equation to compute the relative relationship of i and j, and we usually compute f p using the following formula:

[0073] Here W q is also a set of weights obtained through a linear transformation. k and the above W m are all weights obtained through a linear transformation. Here is the total number of samples. Among them, the above W m , W q and W m are nonlinear features.

[0074] Step S208, inputting the image sample into the student network to obtain a second output image of the image sample and at least one second intermediate feature map; wherein the second intermediate feature map is output by at least one intermediate feature layer of the student network, and the second output image is output by an output layer of the student network.

[0075] In order to train the student network, it is necessary to input the image sample into the student network to obtain the second intermediate feature map output by the at least one intermediate feature layer of the student network and the second output image output by the output layer of the student network.

[0076] Step S210, extracting a second saliency feature of the image sample based on the second intermediate feature map and the second output image, and training the student network based on the first saliency feature and the second saliency feature.

[0077] It should be noted here that the way of calculating the second saliency feature in step S210 can be the same as the way of calculating the first saliency feature in step S206.

[0078] For example, the second saliency feature can be calculated by the following way: performing first feature probability calculation on the second intermediate feature map to obtain third channel feature probability; performing first feature probability calculation on the second output image to obtain fourth channel feature probability; and taking the third channel feature probability and the fourth channel feature probability as the second saliency feature.

[0079] The calculation method of the second saliency feature can be calculated by the channel saliency module, and the specific calculation method can be referred to step S206, which will not be repeated here.

[0080] The first channel feature probability and the third channel feature probability are calculated according to the first intermediate feature map and the second intermediate feature map respectively, and the second channel feature probability and the fourth channel feature probability are calculated according to the first output image and the second output image respectively.

[0081] Therefore, the intermediate feature layer parameters of the student network can be adjusted according to the first channel feature probability and the third channel feature probability, and the output layer parameters of the student network can be adjusted according to the second channel feature probability and the fourth channel feature probability.

[0082] Specifically, the channel feature probability in the teacher network and the channel feature probability in the student network can be optimized using KL divergence, so that the student network can imitate the channel saliency information learned by the teacher network.

[0083] For example, the student network can be trained by calculating the first KL divergence of the first channel feature probability and the third channel feature probability, adjusting the intermediate feature layer parameters of the student network based on the first KL divergence, calculating the second KL divergence of the second channel feature probability and the fourth channel feature probability, adjusting the output layer parameters of the student network based on the second KL divergence, and continuing to train the student network until the training is completed.

[0084] The KL divergence is usually used to measure the difference between two probability distributions, and can be used as an optimization task target to make the distribution of the network approach another target distribution by optimizing the KL divergence, as follows: Loss = KLD(Student(x), Teacher(x)).

[0085] The above formula can make the output distribution of the student network approach the output distribution of the teacher network after optimization by the above Loss, and the input of the formula can be replaced by various probability distributions, which are not limited here.

[0086] In step S212, the second position feature of the image sample is extracted based on the second intermediate feature map, and the student network is trained based on the first position feature and the second position feature.

[0087] The way of calculating the second position feature in step S210 can be the same as the way of calculating the first position feature in step S206. For example, the second position feature can be calculated by performing second feature probability calculation on the second intermediate feature map to obtain the second position feature probability, and taking the second position feature probability as the second position feature. The calculation of the second position feature can be calculated by a position feature module, and the specific calculation method can be referred to step S206, which will not be repeated here.

[0088] Therefore, the intermediate feature layer parameters of the student network can be adjusted according to the first position feature probability and the second position feature probability. Specifically, the first position feature probability in the teacher network and the second position feature probability in the student network can be used to calculate the loss function (i.e., the first L2 loss value) of the student network using the L2 norm, so that the student network can imitate the position information learned by the teacher network.

[0089] For example, the student network can be trained by calculating the first L2 loss value of the first position feature probability and the second position feature probability; adjusting the parameters of the intermediate feature layer of the student network based on the first L2 loss value, and continuing to train the student network until the training is completed.

[0090] The first L2 loss value can be calculated by the following formula: represents the sum of the squares of the elements of the vector and then taking the square root, which can measure the difference between the first position feature probability and the second position feature probability.

[0091] As shown in Figure 3 In addition to the channel saliency module and the position feature module, the output feature module can also be included in the embodiment. Among them, the saliency module (1) acts on the intermediate layer and the output layer of the teacher network, respectively, and is used before the decoder of the backbone network in the semantic segmentation network. The position feature module (2) only acts on the intermediate layer of the teacher network, and is used in the intermediate layer of the backbone network. The output feature module (3) acts on the teacher output layer and the label at the same time, respectively extracts the features of the output of the teacher network and the label, and then uses this information to supervise the student network.

[0092] The output feature module can greatly simplify the design of the complex output space module in the past. In addition to the teacher output space, the features of the label are also extracted in the same way. The specific implementation way is to use a self-encoder to learn the structural features of the output space through the optimization method of the reconstruction function. After obtaining such features, the L2 loss function can be used to constrain the parameters of the student network, thereby improving the performance of the student network. For example: obtaining a first label of an image sample; training the student network based on the first output image, the first label, and the image sample.

[0093] The label (Ground Truth) can also be called annotation. In supervised learning, the data is annotated in the form of (x, t), where x is the input data and t is the annotation. The correct t annotation is the ground truth, and the incorrect annotation is not.

[0094] The optimization manner of the reconstruction function can use an L2 norm optimization manner, and a specific implementation manner is L2(x, y), that is, comparing the output distribution distance between the output x and the input y. Specifically, the autoencoder can be trained through the following steps: training the autoencoder based on the first output image and the first label; wherein the autoencoder comprises an encoder and a decoder; adjusting the parameters of the output layer of the student network based on the trained autoencoder and the image sample, and continuing to train the student network until the training is completed.

[0095] Assuming that the input is x and the output is y, an autoencoder can be constructed. The autoencoder is composed of an encoder and a decoder. The encoder preliminarily reduces the dimension of the input image (i.e., the first output image and the first label), and the decoder preliminarily increases the dimension of the image after being reduced in dimension, so as to achieve the effect of reconstructing the image. Assuming that the autoencoder here is F, the reconstruction loss function Loss can be obtained: Loss = L2(x, F(x)); wherein x is the input image, y is F(x), which refers to the image after being reduced in dimension and then increased in dimension.

[0096] After the autoencoder is trained, the output layer of the student network can be trained using the autoencoder, for example: inputting the image sample into the student network to obtain the third output image of the image sample and the second label of the image sample; inputting the third output image and the second label into the autoencoder to obtain the image in the student network; calculating the third L2 loss value based on the third output image, the second label and the image in the student network; and adjusting the parameters of the output space of the student network based on the third L2 loss value.

[0097] The third output image and the second label of the student network are reduced in dimension and then increased in dimension through the trained autoencoder, the third L2 loss value of the image after being increased in dimension and the input image is calculated, and the parameters of the output space of the student network are adjusted based on the third L2 loss value.

[0098] After the student network is trained, the student network can be used to optimize the target image, for example: obtaining the target image; inputting the target image into the trained student network to obtain the optimized image. The optimized image obtained by using the student network has high performance and efficiency.

[0099] In summary, the embodiments of the present application provide more of the above methods, which use the intermediate feature layer of the teacher network and the channel saliency feature of the output space to distill the student network, use the position feature of the intermediate feature layer of the teacher network to distill the student network, and use the output space and the label of the teacher network to distill the student network. These modules are designed around important feature information in the semantic segmentation task.

[0100] The embodiment of the present application designs a high-performance knowledge distillation algorithm for a semantic segmentation task. The method provided by the embodiment of the present application solves the problem of lacking learning of position information and saliency information in the semantic segmentation knowledge distillation algorithm, improves the output feature module for obtaining a complex output space feature in the past, and combines label information, so that a small semantic segmentation network (that is, a student network) can obtain a performance close to that of a large semantic segmentation network (that is, a teacher network) while being more efficient. This method is suitable for edge computing and can be used in a computing resource limited scene such as an embedded chip, a CPU (Central Processing Unit), an NPU (Neural Network Processing Unit) and the like.

[0101] The above method provided by the embodiment of the present application has the following effects:

[0102] Firstly, the position feature module in the technical solution can enable the model to learn the relative position information between objects and objects, so that some objects that appear at the same time (such as a car and a lane line) can be better learned, thereby improving the performance of the model.

[0103] Secondly, the model improves the learning scheme of the saliency map, not only learns the saliency features in the image from the spatial perspective, but also extracts the saliency of the image from the channel features, thereby strengthening the learning ability of the model for saliency features. This effect is achieved thanks to the channel saliency feature module, which calculates the saliency on the channel by simultaneously calculating the saliency of the intermediate feature map and the output map of the teacher model, and distills this knowledge to the student network.

[0104] Finally, the output feature module designed by us learns the structured features in the data label and the teacher network output at the same time to achieve the effect of optimizing the output space of the student network, so that the student network can have a clear optimization direction.

[0105] The above model provided by the embodiment greatly improves the prediction effect of the student network by distilling important features for the task from the teacher network or the label to the student network from different feature angles of the semantic segmentation task.

[0106] In addition, it should be noted that the algorithm provided in the embodiment can only be used when training the semantic segmentation network. The operation process is to first train a teacher network, then freeze all parameters of the teacher network, then train the related saliency module, position feature module and output feature module, and then use the designed distillation algorithm to train a student network. The algorithm framework of the embodiment can be adapted to various teacher networks and student networks, including but not limited to using backbone networks such as ResNet, MobileNet, EfficientNet and ESPNet, or using semantic algorithm segmentation structures such as PSPNet and DeepLabV3. In addition, the position feature module proposed in the embodiment belongs to a general module, and various self-attention algorithms or non-local algorithms can be used to extract the position information of the model.

[0107] Embodiment three:

[0108] Corresponding to the method embodiment, the embodiment of the application provides a student network processing device, which refers to a structure schematic diagram of a student network processing device as shown in Figure 4 The student network processing device comprises:

[0109] An image sample acquisition module 41 is configured to acquire an image sample of a student network.

[0110] For each image sample, the following operations are performed:

[0111] A teacher network output module 42 is configured to input the image sample into a trained teacher network to obtain a first output image of the image sample and at least one first intermediate feature map. The first intermediate feature map is output by at least one intermediate feature layer of the teacher network, and the first output image is output by an output layer of the teacher network.

[0112] A saliency feature and position feature extraction module 43 is configured to extract a first saliency feature of the image sample based on the first intermediate feature map and the first output image, and extract a first position feature of the image sample based on the first intermediate feature map. The first saliency feature represents the saliency information of the first intermediate feature map and the first output image, and the first position feature represents the position information of each pixel of the first intermediate feature map.

[0113] A student network training module 44 is configured to train a student network based on the first saliency feature and the first position feature, and the image sample.

[0114] The student network processing device provided by the embodiment of the present application extracts the first saliency feature of the image sample from the first output image and the first intermediate feature map of the image sample, extracts the first position feature of the image sample from the first intermediate feature map, and trains the student network according to the first saliency feature, the first position feature and the image sample. In this way, the student network can learn the relative position information between objects and objects according to the first position feature, thereby improving the performance of the model; the student network is trained based on the first saliency feature, not only learns the saliency feature in the image from the spatial angle, but also extracts the saliency of the image from the channel feature, thereby strengthening the learning ability of the model to the saliency feature.

[0115] The student network training module is configured to input the image sample into the student network to obtain a second output image and at least one second intermediate feature map of the image sample, wherein the second intermediate feature map is output by at least one intermediate feature layer of the student network, and the second output image is output by an output layer of the student network; extract a second saliency feature of the image sample based on the second intermediate feature map and the second output image, and train the student network based on the first saliency feature and the second saliency feature; extract a second position feature of the image sample based on the second intermediate feature map, and train the student network based on the first position feature and the second position feature.

[0116] The saliency feature and position feature extraction module is configured to perform a preset first feature probability calculation on the first intermediate feature map to obtain a first channel feature probability, perform the first feature probability calculation on the first output image to obtain a second channel feature probability, and take the first channel feature probability and the second channel feature probability as the first saliency feature; the student network training module is configured to perform the first feature probability calculation on the second intermediate feature map to obtain a third channel feature probability, perform the first feature probability calculation on the second output image to obtain a fourth channel feature probability, and take the third channel feature probability and the fourth channel feature probability as the second saliency feature; the student network training module is configured to calculate a first KL divergence of the first channel feature probability and the third channel feature probability, adjust the intermediate feature layer parameters of the student network based on the first KL divergence, calculate a second KL divergence of the second channel feature probability and the fourth channel feature probability, adjust the output layer parameters of the student network based on the second KL divergence, and continue to train the student network until the training is completed.

[0117] The feature probability calculation includes: converting a 3-dimensional tensor feature of an input image into a 1-dimensional tensor feature; calculating the 1-dimensional tensor feature based on a preset first Softmax function to obtain a channel feature probability corresponding to the input image.

[0118] The significant feature and position feature extraction module is configured to perform preset second feature probability calculation on the first intermediate feature map to obtain a first position feature probability, and take the first position feature probability as the first position feature.

[0119] The position feature probability calculation includes: for each target pixel of the input image, calculating a nonlinear feature of the target pixel; based on the nonlinear feature and a preset second Softmax function, calculating a response value of each other pixel of the input image to the target pixel, and taking each response value as a position feature probability of the target pixel; wherein the other pixel is a pixel of the input image other than the target pixel, and the response value of the other pixel to the target pixel represents position information of the target pixel relative to the other pixel.

[0120] The student network training module is further configured to obtain a first label of the image sample; and train the student network based on the first output image, the first label and the image sample.

[0121] The student network training module is further configured to train an autoencoder based on the first output image and the first label; wherein the autoencoder includes an encoder and a decoder; adjust parameters of an output layer of the student network based on the trained autoencoder and the image sample, and continue to train the student network until the training is completed.

[0122] The student network training module is further configured to input the first output image and the first label into the encoder to obtain a reduced dimension image; input the reduced dimension image into the decoder to obtain an up-sampled image; calculate a second L2 loss value based on the first output image, the first label and the up-sampled image; adjust parameters of the autoencoder based on the second L2 loss value, and continue to train the autoencoder until the training is completed.

[0123] The student network training module is further configured to input the image sample into the student network to obtain a third output image of the image sample and a second label of the image sample; input the third output image and the second label into the autoencoder to obtain an up-sampled image of the image sample in the student network; calculate a third L2 loss value based on the third output image, the second label and the up-sampled image in the student network; and adjust parameters of an output space of the student network based on the third L2 loss value.

[0124] The teacher network and the student network are both semantic segmentation neural networks.

[0125] Referring to Figure 5 Another structural schematic diagram of a student network processing apparatus is shown, which comprises a student network use module 45 connected with a student network training module 44, the student network training module is used for obtaining a target image; and the target image is input into the trained student network to obtain an optimized image.

[0126] The student network processing apparatus provided in the embodiments of the present application has the same implementation principle and technical effects as the foregoing student network processing method embodiments, and for brief description, the parts not mentioned in the student network processing apparatus embodiments can refer to the corresponding contents in the foregoing student network processing method embodiments.

[0127] The embodiments of the present application further provide an electronic device for running the foregoing student network processing method; referring to Figure 6 A structural schematic diagram of an electronic device is shown, which comprises a memory 100 and a processor 101, wherein the memory 100 is used for storing one or more computer instructions, and the one or more computer instructions are executed by the processor 101 to realize the foregoing student network processing method.

[0128] Further, Figure 6 The electronic device further comprises a bus 102 and a communication interface 103, and the processor 101, the communication interface 103 and the memory 100 are connected through the bus 102.

[0129] The memory 100 can contain a high-speed random access memory (RAM) and can also include a non-volatile memory, for example, at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 102 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0130] The processor 101 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 101 or the instruction in the form of software. The processor 101 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 100, and the processor 101 reads the information in the storage 100, and combines the hardware to complete the steps of the method of the above embodiment.

[0131] The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores computer executable instructions, when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to implement the above student network processing method, and the specific implementation can be referred to the method embodiment, and will not be repeated here.

[0132] The computer program product of the student network processing method, device and electronic equipment provided by the embodiment of the present application includes a computer readable storage medium storing program codes, the instructions included in the program codes can be used to execute the method in the foregoing method embodiment, and the specific implementation can be referred to the method embodiment, and will not be repeated here.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and / or device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0134] In addition, in the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the term "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through an intermediate medium; it can be internal connection of two elements. For those skilled in the art, the specific meaning of the above-mentioned term in the present application can be understood according to the specific circumstances.

[0135] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0136] In the description of the present application, it should be noted that the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0137] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, and are not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of student network processing, the method comprising: The method comprises: obtaining an image sample of a student network; for each of the image samples, the following operations are performed: inputting the image sample into a trained teacher network to obtain a first output image and at least one first intermediate feature map of the image sample; wherein the first intermediate feature map is output by at least one intermediate feature layer of the teacher network, and the first output image is output by an output layer of the teacher network; extracting first saliency features of the image sample based on the first intermediate feature map and the first output image, and extracting first position features of the image sample based on the first intermediate feature map; wherein the first saliency features represent saliency information of the first intermediate feature map and the first output image, and the first position features represent position information of each pixel of the first intermediate feature map; training the student network based on the first saliency features and the first position features, and the image sample; after the step of inputting the image sample into the trained teacher network, the method further comprises: obtaining a first label of the image sample; and training the student network based on the first output image, the first label, and the image sample; the step of training the student network based on the first output image, the first label, and the image sample comprises: training an autoencoder based on the first output image and the first label; wherein the autoencoder comprises an encoder and a decoder; adjusting parameters of an output layer of the student network based on the trained autoencoder and the image sample, and continuing to train the student network until the training is completed.

2. The method of claim 1, wherein, the step of training the student network based on the first saliency features and the first position features, and the image sample comprises: inputting the image sample into the student network to obtain a second output image and at least one second intermediate feature map of the image sample; wherein the second intermediate feature map is output by at least one intermediate feature layer of the student network, and the second output image is output by an output layer of the student network; extracting second saliency features of the image sample based on the second intermediate feature map and the second output image, and training the student network based on the first saliency features and the second saliency features; extracting second position features of the image sample based on the second intermediate feature map, and training the student network based on the first position features and the second position features.

3. The method of claim 2, wherein, the step of extracting first saliency features of the image sample based on the first intermediate feature map and the first output image comprises: performing a first feature probability calculation on the first intermediate feature map to obtain a first channel feature probability, and performing the first feature probability calculation on the first output image to obtain a second channel feature probability; and taking the first channel feature probability and the second channel feature probability as the first saliency features; the step of extracting second saliency features of the image sample based on the second intermediate feature map and the second output image comprises: performing the first feature probability calculation on the second intermediate feature map to obtain third channel feature probability; performing the first feature probability calculation on the second output image to obtain fourth channel feature probability; taking the third channel feature probability and the fourth channel feature probability as second saliency features; The step of training the student network based on the first saliency features and the second saliency features comprises: calculating a first KL divergence of the first channel feature probability and the third channel feature probability, and adjusting the intermediate feature layer parameters of the student network based on the first KL divergence; calculating a second KL divergence of the second channel feature probability and the fourth channel feature probability, and adjusting the output layer parameters of the student network based on the second KL divergence; continuing to train the student network until the training is completed.

4. The method of claim 3, wherein, The feature probability calculation comprises: converting the 3-dimensional tensor feature of the input image into a 1-dimensional tensor feature; calculating the 1-dimensional tensor feature based on a preset first Softmax function to obtain the channel feature probability corresponding to the input image.

5. The method of claim 2, wherein, The step of extracting the first position feature of the image sample based on the first intermediate feature map comprises: performing a preset second feature probability calculation on the first intermediate feature map to obtain a first position feature probability, and taking the first position feature probability as the first position feature. The step of extracting the second position feature of the image sample based on the second intermediate feature map comprises: performing the second feature probability calculation on the second intermediate feature map to obtain a second position feature probability, and taking the second position feature probability as the second position feature. The step of training the student network based on the first position feature and the second position feature comprises: calculating a first L2 loss value of the first position feature probability and the second position feature probability, adjusting the parameters of the intermediate feature layer of the student network based on the first L2 loss value, and continuing to train the student network until the training is completed.

6. The method of claim 5, wherein, The position feature probability calculation comprises: for each target pixel of an input image, calculating a nonlinear feature of the target pixel; based on the nonlinear feature and a preset second Softmax function, calculating a response value of each other pixel of the input image to the target pixel, and taking each response value as a position feature probability of the target pixel; wherein the other pixels are pixels of the input image other than the target pixel, and the response value of the other pixels to the target pixel represents the position information of the target pixel relative to the other pixels.

7. The method of claim 1, wherein, The step of training the autoencoder based on the first output image and the first label comprises: inputting the first output image and the first label into the encoder to perform dimension reduction to obtain a reduced dimension image; performing dimension increase on the reduced dimension image through the decoder to obtain an increased dimension image; calculating a second L2 loss value based on the first output image, the first label and the increased dimension image; adjusting the parameters of the autoencoder based on the second L2 loss value, continuing to train the autoencoder until the training is completed.

8. The method of claim 1, wherein, The step of adjusting parameters of an output layer of the student network based on the trained autoencoder and the image sample comprises: inputting the image sample into the student network to obtain a third output image of the image sample and a second label of the image sample; inputting the third output image and the second label into the autoencoder to obtain an upsampled image of the image sample in the student network; calculating a third L2 loss value based on the third output image, the second label and the upsampled image in the student network; adjusting parameters of an output space of the student network based on the third L2 loss value.

9. The method according to any one of claims 1 to 8, characterized in that, The teacher network and the student network are both semantic segmentation neural networks.

10. The method according to any one of claims 1 to 8, characterized in that, After the step of training the student network based on the first saliency feature and the first position feature and the image sample, the method further comprises: obtaining a target image; inputting the target image into the trained student network to obtain an optimized image.

11. A student network processing device, characterized by The apparatus comprises: an image sample obtaining module configured to obtain image samples of a student network; for each of the image samples, the following operations are performed: a teacher network outputting module configured to input the image sample into a trained teacher network to obtain a first output image of the image sample and at least one first intermediate feature map; wherein the first intermediate feature map is output by at least one intermediate feature layer of the teacher network, and the first output image is output by an output layer of the teacher network; a saliency feature and position feature extracting module configured to extract a first saliency feature of the image sample based on the first intermediate feature map and the first output image, and extract a first position feature of the image sample based on the first intermediate feature map; wherein the first saliency feature represents saliency information of the first intermediate feature map and the first output image, and the first position feature represents position information of each pixel of the first intermediate feature map; a student network training module configured to train the student network based on the first saliency feature and the first position feature and the image sample; the student network training module is further configured to obtain a first label of the image sample, and train the student network based on the first output image, the first label and the image sample; the student network training module is configured to train an autoencoder based on the first output image and the first label; wherein the autoencoder comprises an encoder and a decoder; and adjust parameters of an output layer of the student network based on the trained autoencoder and the image sample, and continue to train the student network until the training is completed.

12. An electronic device, comprising: The apparatus comprises a processor and a memory, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the steps of the student network processing method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, when the computer executable instructions are invoked and executed by the processor, the computer executable instructions cause the processor to implement the steps of the student network processing method in any one of claims 1-10.

Images