A deep learning knowledge distillation method and system based on frequency domain supervision

Abstract

The present invention discloses a deep learning knowledge distillation method and system based on frequency domain supervision. The method and system train a teacher network, and then design a student network based on the trained teacher network. Then, using the same sample data set to be trained, the teacher network supervises the student network in the frequency domain, thereby ensuring that the student network can finely imitate the edge features of the teacher network, thereby making up for the defect that the student network does not learn the areas where the edge changes of the teacher network features are more obvious, thereby achieving the purpose of improving the performance of the student network. At the same time, the technical solution provided by the embodiment of the present invention can be applied to the knowledge distillation of various networks, can realize the compression of complex networks, and is conducive to realizing the needs of deploying lightweight networks to mobile terminals.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and computer vision technology, and in particular to a deep learning knowledge distillation method and system based on frequency domain supervision. Background Art

[0002] With the development of civil aviation, projects such as computer vision-based target detection and pedestrian attribute classification have received more and more attention in the implementation of video surveillance scenarios such as civil aviation and security. Most of the existing recognition of flight ground support status is based on image-based target detection methods. However, it is difficult to identify the ground support status based on a single image, and it is impossible to accurately determine the current status of the target. Although the video data collected by the flight monitoring system can help improve the recognition effect of the flight ground support status, it requires processing a large amount of video data. Therefore, it is necessary to improve the inference speed of the model.

[0003] It can be seen that in order to achieve real-time intelligent management, provide security for airports and improve operational efficiency, it is necessary to take into account various requirements such as accurate identification, fast operation speed, and easy deployment on mobile terminals. Since knowledge distillation is an effective solution for model compression, it can improve model compression and model accuracy. Therefore, knowledge distillation can be applied to the identification of flight ground support status.

[0004] At present, the existing knowledge distillation scheme only uses the feature map space domain to supervise the student network. However, the feature map space domain supervision usually uses the mean square error loss, which can only ensure that the feature map of the student network is as close as possible to the feature map of the teacher network, but cannot ensure that the student network imitates the edge features of the teacher network in detail. That is, in some areas where the feature map of the student network mutates, it cannot be completely consistent with the teacher network, and thus cannot effectively improve the performance of the student network. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide a deep learning knowledge distillation method and system based on frequency domain supervision, which is used to ensure that the student network can finely imitate the edge features of the teacher network to improve the performance of the student network.

[0006] In a first aspect, an embodiment of the present invention provides a deep learning knowledge distillation method based on frequency domain supervision, comprising:

[0007] Obtaining a first sample data set to be trained and a teacher network to be trained, and inputting the first sample data set to be trained into the teacher network to be trained until convergence, to obtain a trained first teacher network;

[0008] Based on the first teacher network, design a student network;

[0009] Obtain a second sample data set to be trained, and input the second sample data set to the first teacher network and the student network simultaneously for Fourier transform and Gaussian weight distribution processing, thereby obtaining a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network;

[0010] Calculating a frequency domain supervision loss based on the first frequency domain feature map and the second frequency domain feature map;

[0011] Determine whether the frequency domain supervision loss converges; if the frequency domain supervision loss does not converge, obtain a new sample data set to be trained, replace the new sample data set to be trained with the second sample data set to be trained, return to execute the second sample data set to be trained simultaneously input into the first teacher network and the student network for Fourier transform and Gaussian weight allocation processing, and obtain a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network; or, if the frequency domain supervision loss has converged, determine that the knowledge distillation is completed.

[0012] In a possible design, based on the first teacher network, a student network is designed, including:

[0013] The structural framework of the student network is designed to be the same as the structural framework of the first teacher network, and the number of feature map channels or layers of the student network is designed to be less than the number of feature map channels or layers of the first teacher network.

[0014] In a possible design, after simultaneously inputting the second sample data set to be trained into the first teacher network and the student network for Fourier transform and Gaussian weight distribution processing, a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network are obtained, including:

[0015] Inputting the second sample data set to be trained into the first teacher network and the student network at the same time to obtain a first feature map corresponding to the first teacher network and a second feature map corresponding to the student network;

[0016] Performing Fourier transform on the first feature map and the second feature map to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map;

[0017] The high-frequency signals and low-frequency signals in the third feature map and the fourth feature map are sorted and weighted according to a preset arrangement to obtain the first frequency domain feature map and the second frequency domain feature map.

[0018] In a possible design, performing Fourier transform on the first feature map and the second feature map to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map includes:

[0019] Performing Fourier transform on the first feature map and the second feature map by using a two-dimensional discrete Fourier transform to obtain the third feature map and the fourth feature map.

[0020] In one possible design, the two-dimensional discrete Fourier transform follows the following formula:

[0021]

[0022] Among them, F(u,v) represents the frequency domain signal of the third feature map or the fourth feature map, f(x,y) represents the value at the coordinate (x,y) on the first feature map or the second feature map, M and N represent the number of rows and columns of the first feature map or the second feature map, respectively, and j and π are constants.

[0023] In a possible design, the high-frequency signals and the low-frequency signals in the third feature map and the fourth feature map are sorted and weighted according to a preset arrangement to obtain the first frequency domain feature map and the second frequency domain feature map, including:

[0024] Generate a Gaussian weighted initial image, wherein the Gaussian weighted initial image is an image with a white block in the middle and black blocks around it;

[0025] Obtaining a Gaussian filter weight template, and using the Gaussian filter weight template to perform convolution on the Gaussian weight initial image to obtain a Gaussian weight mask;

[0026] Based on the Gaussian weight mask, the high-frequency signals and the low-frequency signals in the third feature map and the fourth feature map are sorted and weights are set to obtain the first frequency domain feature map and the second frequency domain feature map.

[0027] In one possible design, the Gaussian filter weight template follows the following formula:

[0028]

[0029] Among them, the center of the Gaussian filter weight template is the origin, G(x1,y1) represents the weight of each point in the Gaussian filter weight template, (x1,y1) represents the coordinates of each point in the Gaussian filter weight template, and σ is a scalar. The larger its value, the larger the range covered by the Gaussian filter weight template.

[0030] In one possible design, calculating the frequency domain supervision loss based on the first frequency domain feature map and the second frequency domain feature map includes:

[0031] Based on the first frequency domain feature map and the second frequency domain feature map, using a loss function to calculate the frequency domain supervision loss;

[0032] The loss function is expressed as:

[0033]

[0034] in, Represented as the value of the i-th point in the first frequency domain feature map, y i It is represented by the value of the i-th point in the second frequency domain feature map, n is represented by the number of all coordinate points in the first frequency domain feature map, n=W×H×C, W, H, and C respectively represent the width, height, and number of feature map channels of the first frequency domain feature map, and the number of all coordinate points in the second frequency domain feature map is the same as the number of all coordinate points in the first frequency domain feature map.

[0035] In a second aspect, an embodiment of the present invention further provides a deep learning knowledge distillation system based on frequency domain supervision, comprising:

[0036] A teacher network training unit is used to obtain a first sample data set to be trained and a teacher network to be trained, and input the first sample data set to be trained into the teacher network to be trained until convergence, so as to obtain a trained first teacher network;

[0037] A student network design unit, used for designing a student network based on the first teacher network;

[0038] Student online training modules for:

[0039] Obtain a second sample data set to be trained, and input the second sample data set to the first teacher network and the student network simultaneously for Fourier transform and Gaussian weight distribution processing, thereby obtaining a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network;

[0040] Calculating a frequency domain supervision loss based on the first frequency domain feature map and the second frequency domain feature map;

[0041] Determine whether the frequency domain supervision loss converges; if the frequency domain supervision loss does not converge, obtain a new sample data set to be trained, replace the new sample data set to be trained with the second sample data set to be trained, return to execute the second sample data set to be trained simultaneously input into the first teacher network and the student network for Fourier transform and Gaussian weight allocation processing, and obtain a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network; or, if the frequency domain supervision loss has converged, determine that the knowledge distillation is completed.

[0042] In a possible design, the student network design unit is specifically used for:

[0043] The structural framework of the student network is designed to be the same as the structural framework of the first teacher network, and the number of feature map channels or layers of the student network is designed to be less than the number of feature map channels or layers of the first teacher network.

[0044] In a possible design, the student network training unit is specifically used for:

[0045] Inputting the second sample data set to be trained into the first teacher network and the student network at the same time to obtain a first feature map corresponding to the first teacher network and a second feature map corresponding to the student network;

[0046] Performing Fourier transform on the first feature map and the second feature map to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map;

[0047] The high-frequency signals and low-frequency signals in the third feature map and the fourth feature map are sorted and weighted according to a preset arrangement to obtain the first frequency domain feature map and the second frequency domain feature map.

[0048] In a possible design, the student network training unit is specifically used for:

[0049] Performing Fourier transform on the first feature map and the second feature map by using a two-dimensional discrete Fourier transform to obtain the third feature map and the fourth feature map.

[0050] In one possible design, the two-dimensional discrete Fourier transform follows the following formula:

[0051]

[0052] Among them, F(u,v) represents the frequency domain signal of the third feature map or the fourth feature map, f(x,y) represents the value at the coordinate (x,y) on the first feature map or the second feature map, M and N represent the number of rows and columns of the first feature map or the second feature map, respectively, and j and π are constants.

[0053] In a possible design, the student network training unit is specifically used for:

[0054] Generate a Gaussian weighted initial image, wherein the Gaussian weighted initial image is an image with a white block in the middle and black blocks around it;

[0055] Obtaining a Gaussian filter weight template, and using the Gaussian filter weight template to perform convolution on the Gaussian weight initial image to obtain a Gaussian weight mask;

[0056] Based on the Gaussian weight mask, the high-frequency signals and the low-frequency signals in the third feature map and the fourth feature map are sorted and weights are set to obtain the first frequency domain feature map and the second frequency domain feature map.

[0057] In one possible design, the Gaussian filter weight template follows the following formula:

[0058]

[0059] Among them, the center of the Gaussian filter weight template is the origin, G(x1,y1) represents the weight of each point in the Gaussian filter weight template, (x1,y1) represents the coordinates of each point in the Gaussian filter weight template, and σ is a scalar. The larger its value, the larger the range covered by the Gaussian filter weight template.

[0060] In a possible design, the student network training unit is specifically used for:

[0061] Based on the first frequency domain feature map and the second frequency domain feature map, using a loss function to calculate the frequency domain supervision loss;

[0062] The loss function is expressed as:

[0063]

[0064] in, Represented as the value of the i-th point in the first frequency domain feature map, y i It is represented by the value of the i-th point in the second frequency domain feature map, n is represented by the number of all coordinate points in the first frequency domain feature map, n=W×H×C, W, H, and C respectively represent the width, height, and number of feature map channels of the first frequency domain feature map, and the number of all coordinate points in the second frequency domain feature map is the same as the number of all coordinate points in the first frequency domain feature map.

[0065] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one program; when the at least one program is executed by a processor, the method involved in any possible design of the first aspect described above is implemented.

[0066] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings.

[0067] The beneficial technical effects of the present invention are as follows:

[0068] In the technical solution provided by the embodiment of the present invention, by allowing the teacher network to supervise the student network in the frequency domain, it can be ensured that the student network can finely imitate the edge features of the teacher network, thereby making up for the defect that the student network does not learn well enough in areas where the edge changes of the teacher network features are more obvious, thereby achieving the purpose of improving the performance of the student network. At the same time, the technical solution provided by the embodiment of the present invention can be applied to the knowledge distillation of various networks, can realize the compression of complex networks, and is conducive to realizing the needs of deploying lightweight networks to mobile terminals, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of a flow chart of a deep learning knowledge distillation method based on frequency domain supervision provided by an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of a process for executing step S104 provided in an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of a method for performing knowledge distillation based on frequency domain supervision using a ResNnet network provided in an embodiment of the present invention;

[0072] Figure 4 A schematic diagram of using a VGG network to perform knowledge distillation based on frequency domain supervision provided by an embodiment of the present invention;

[0073] Figure 5 A schematic diagram of a passenger elevator and pedestrian daytime image and its corresponding first / second frequency domain feature graph provided by an embodiment of the present invention;

[0074] Figure 6 A schematic diagram of a night image of a food truck and its corresponding first / second frequency domain feature graphs provided by an embodiment of the present invention;

[0075] Figure 7 A schematic diagram of a daytime image of an aircraft and its corresponding first / second frequency domain feature graphs provided by an embodiment of the present invention;

[0076] Figure 8A schematic diagram of a daytime image of a small cargo truck and its corresponding first / second frequency domain feature graphs provided by an embodiment of the present invention;

[0077] FIG9( a) is a schematic diagram of a Gaussian weighted initial image provided by an embodiment of the present invention;

[0078] FIG9( b) is a schematic diagram of a Gaussian weight mask provided in an embodiment of the present invention;

[0079] Fig.10 A schematic diagram of the framework of a deep learning knowledge distillation system based on frequency domain supervision provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The directional terms such as up, down, left, right, front, back, front, back, top, bottom, etc. mentioned or may be mentioned in this specification are defined relative to their structure and are relative concepts. Therefore, they may change accordingly according to their different locations and different usage conditions. Therefore, these or other directional terms should not be interpreted as restrictive terms.

[0081] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of embodiments consistent with some aspects of the present disclosure.

[0082] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "the" and "the" used in this disclosure are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0083] Unless otherwise specified, ordinal numbers such as "first" and "second" in the embodiments of the present invention are used to distinguish multiple objects and are not used to limit the order, timing, priority or importance of the multiple objects.

[0084] The shapes and sizes of various components in the drawings do not reflect the true scale, and are intended only to illustrate the implementations described in the following exemplary embodiments.

[0085] The deep learning knowledge distillation method based on frequency domain supervision involved in the embodiment of the present invention can be applied to target classification, target detection, image semantic segmentation, human posture estimation and other aspects, which are not limited by the embodiment of the present invention.

[0086] Please refer to Figure 1 As shown, it is a flow chart of a deep learning knowledge distillation method based on frequency domain supervision provided by an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0087] S101, obtaining a first sample data set to be trained and a teacher network to be trained, and inputting the first sample data set to be trained into the teacher network to be trained until convergence, to obtain a trained first teacher network.

[0088] In the specific implementation process, the first sample data set to be trained can be used to train the teacher network to be trained until the teacher network to be trained converges, that is, the relevant parameters of the teacher network no longer change or change very little. At this time, the trained first teacher network can be obtained.

[0089] As an example, taking ResNet network knowledge distillation and taking ResNet50 as the teacher network to be trained, the teacher network to be trained may have 49 convolutional layers and 1 fully connected layer, so that a first teacher network with higher accuracy can be trained.

[0090] As another example, taking VGG network knowledge distillation and using VGG16(D) network as the teacher network to be trained, the teacher network to be trained can have 5 convolution groups, each convolution group has 2 or 3 convolution layers, and 5 convolution groups have a total of 16 convolution layers, so that a first teacher network with higher accuracy can be trained.

[0091] S102. Design a student network based on the first teacher network.

[0092] In some embodiments, the structural framework of the student network can be designed to be the same as the structural framework of the first teacher network, and the feature map channels or layers of the student network can be designed to be less than the feature map channels or layers of the first teacher network, so that the student network can reduce the amount of calculation compared to the first teacher network and achieve lightweight.

[0093] As an example, taking the ResNet network knowledge distillation as an example, combined with Table 1 and Figure 3 As shown in the figure, if the ResNet50 network is used as the first teacher network, the ResNet18 network can be used as the student network, and the number of feature layers output by the corresponding part of the student network is less than the number of feature layers output by the corresponding part of the teacher network.

[0094] Table 1 Structural comparison between ResNet18 network and ResNet50 network

[0095]

[0096] For example, stride 2 means a stride of 2, FC-1000 means a 1000-dimensional fully connected layer, maxpool means a maximum pooling layer, average pool means an average pooling layer, and soft-max is used to calculate the probability of classification.

[0097] As another example, taking VGG network knowledge distillation as an example, combining Table 2 and Figure 4 As shown in the figure, if the VGG16 (D) network is used as the first teacher network, the VGG11 (A) network can be used as the student network. The number of layers corresponding to the student network (including 11 layers of convolutional layers and fully connected layers) is less than the number of layers corresponding to the teacher network (including 16 layers of convolutional layers and fully connected layers).

[0098] Table 2 Structural comparison of VGG11 (A) network and VGG16 (D) network

[0099]

[0100] Exemplarily, FC-4096 represents a 4096-dimensional fully connected layer.

[0101] S103: Obtain a second sample data set to be trained.

[0102] In some embodiments, the second sample data set to be trained and the first sample data set to be trained may be the same or different, which is not limited in the embodiment of the present invention.

[0103] S104, inputting the second sample data set to be trained into the first teacher network and the student network simultaneously for Fourier transform and Gaussian weight distribution processing, thereby obtaining a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network.

[0104] In some embodiments, in combination Figure 1-2 As shown, when executing step S104, the following steps may be specifically performed:

[0105] S201. Input a second sample data set to be trained into the first teacher network and the student network at the same time, so as to obtain a first feature map corresponding to the first teacher network and a second feature map corresponding to the student network.

[0106] As an example, combining the above Table 1 and Figure 3As shown, taking the ResNet network knowledge distillation as an example, the main structure of the first teacher network and the student network may include 4 residual structures. After the second sample data set to be trained is input into the first teacher network and the student network, the first feature map obtained may include 4 feature maps, namely, feature map 11a, feature map 11b, feature map 11c and feature map 11d, and the second feature map obtained may include 4 feature maps, namely, feature map 21a, feature map 21b, feature map 21c and feature map 21d. Since the number of feature layers output by the corresponding part of the student network is less than the number of feature layers output by the first teacher network, the feature maps 21a, feature map 21b, feature map 21c and feature map 21d output by the corresponding part of the student network may be increased in dimension by 1×1 convolution, and feature maps 22a, feature map 22b, feature map 22c and feature map 22d may be obtained accordingly. Among them, the number of layers of feature map 22a, feature map 22b, feature map 22c and feature map 22d corresponds to the same number of layers of feature map 11a, feature map 11b, feature map 11c and feature map 11d.

[0107] As another example, combining the above Table 2 and Figure 4 As shown, taking the VGG network knowledge distillation as an example, the main structure of the first teacher network and the student network may include 5 convolutional layers. After the second sample data set to be trained is input into the first teacher network and the student network, the first feature map obtained may include 5 feature maps, namely, feature map 31a, feature map 31b, feature map 31c, feature map 31d and feature map 31e, and the second feature map obtained may include 5 feature maps, namely, feature map 41a, feature map 41b, feature map 41c, feature map 41d and feature map 41e. Among them, the number of layers of feature map 31a, feature map 31b, feature map 31c, feature map 31d and feature map 31e corresponds to the same number of layers of feature map 41a, feature map 41b, feature map 41c, feature map 41d and feature map 41e.

[0108] S202, performing Fourier transform on the first feature map and the second feature map to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map.

[0109] In some embodiments, a two-dimensional discrete Fourier transform may be used to perform a Fourier transform on the first feature map and the second feature map, and the first feature map and the second feature map may be converted into a frequency domain to obtain a third feature map and a fourth feature map, that is, the third feature map and the fourth feature map are frequency domain feature maps. It can be understood that, through Fourier transform, the two-dimensional image of the xy plane can be converted into a frequency domain signal of the uv plane.

[0110] As an example, combining the above Table 1 and Figure 3As shown, taking ResNet network knowledge distillation as an example, the third feature map may include four feature maps, namely feature map 12a, feature map 12b, feature map 12c, and feature map 12d, and the fourth feature map may include four feature maps, namely feature map 23a, feature map 23b, feature map 23c, and feature map 23d.

[0111] As another example, combining the above Table 2 and Figure 4 As shown, taking VGG network knowledge distillation as an example, the third feature map may include five feature maps, namely feature map 32a, feature map 32b, feature map 32c, feature map 32d and feature map 32e, and the fourth feature map may include five feature maps, namely feature map 42a, feature map 42b, feature map 42c, feature map 42d and feature map 42e.

[0112] In a specific implementation process, the above two-dimensional discrete Fourier transform can follow the following formula (1):

[0113]

[0114] Among them, F(u,v) represents the frequency domain signal of the third feature map or the fourth feature map, f(x,y) represents the value at the coordinate (x,y) on the first feature map or the second feature map, M and N represent the number of rows and columns of the first feature map or the second feature map, respectively, and j and π are constants.

[0115] S203 , sort the high-frequency signals and low-frequency signals in the third feature map and the fourth feature map according to a preset arrangement method and set weights to obtain a first frequency domain feature map and a second frequency domain feature map.

[0116] In some embodiments, the high-frequency signals and low-frequency signals in the third feature map and the fourth feature map can be sorted and weighted according to Gaussian distribution to obtain the first frequency domain feature map and the second frequency domain feature map. Figure 5-8 As shown, the first frequency domain feature map and the second frequency domain feature map are frequency domain feature maps with higher weights for high-frequency signals and lower weights for low-frequency signals.

[0117] As an example, combining the above Table 1 and Figure 3 As shown, taking ResNet network knowledge distillation as an example, the first frequency domain feature map may include the first frequency domain feature Figure 1 , the first frequency domain feature Figure 2 , the first frequency domain feature Figure 3 and the first frequency domain feature Figure 4 The four frequency domain feature maps and the second frequency domain feature map may include the second frequency domain feature Figure 1 , the second frequency domain feature Figure 2 , the second frequency domain feature Figure 3 and the second frequency domain feature Figure 4 These 4 frequency domain feature maps.

[0118] As another example, combining the above Table 2 and Figure 4 As shown, taking VGG network knowledge distillation as an example, the first frequency domain feature map may include the first frequency domain feature Figure 1 , the first frequency domain feature Figure 2 , the first frequency domain feature Figure 3 , the first frequency domain feature Figure 4 and the first frequency domain feature Figure 5 The five frequency domain feature graphs and the second frequency domain feature graph may include the second frequency domain feature Figure 1 , the second frequency domain feature Figure 2 , the second frequency domain feature Figure 3 , the second frequency domain feature Figure 4 and the second frequency domain feature Figure 5 These 5 frequency domain feature maps.

[0119] In a specific implementation process, a Gaussian weighted initial image can be generated, wherein the Gaussian weighted initial image can be an image with a white block in the middle and black blocks around. For example, the Gaussian weighted initial image can be shown in FIG9(a). Then, a Gaussian filter weight template can be obtained, and the Gaussian filter weight template can be used to convolve the Gaussian weighted initial image to obtain a Gaussian weighted mask. For example, the Gaussian weighted mask can be shown in FIG9(b). Afterwards, based on the Gaussian weighted mask, the high-frequency signals and low-frequency signals in the third feature map and the fourth feature map can be sorted and weights can be set, that is, the first frequency domain feature map and the second frequency domain feature map.

[0120] In the specific implementation process, the Gaussian filter weight template can follow the following formula (2):

[0121]

[0122] Among them, the center of the Gaussian filter weight template is the origin, G(x1,y1) represents the weight of each point in the Gaussian filter weight template, (x1,y1) represents the coordinates of each point in the Gaussian filter weight template, and σ is a scalar. The larger its value, the larger the range covered by the Gaussian filter weight template.

[0123] In an embodiment of the present invention, by performing Fourier transform and Gaussian weight assignment on the feature maps output by the corresponding parts of the first teacher network and the second teacher network, the student network can pay more attention to the high-frequency signals in the frequency domain feature map. For example, since the high-frequency signals mainly reflect the contour part where the features of the main target and the background meet in the spatial domain feature map, by giving the high-frequency signals a larger weight during supervision, the student network can more accurately imitate the high-frequency signals of the feature map of the first teacher network, which can make up for the shortcomings of spatial domain supervision, and is conducive to learning the details of the feature map output by the corresponding part of the first teacher network. Further, it helps to improve the performance of the student network.

[0124] S104: Calculate the frequency domain supervision loss based on the first frequency domain feature map and the second frequency domain feature map.

[0125] In some embodiments, the frequency domain supervision loss may be calculated using a loss function based on the first frequency domain feature map and the second frequency domain feature map.

[0126] In the specific implementation process, the loss function can be expressed as the following formula (3):

[0127]

[0128] in, Represented as the value of the i-th point in the first frequency domain feature map, y i It is represented by the value of the i-th point in the second frequency domain feature map, n is represented by the number of all coordinate points in the first frequency domain feature map, n = W × H × C, W, H, C represent the width, height, and number of feature map channels of the first frequency domain feature map, respectively, and the number of all coordinate points in the second frequency domain feature map is the same as the number of all coordinate points in the first frequency domain feature map.

[0129] S105: Determine whether the frequency domain supervision loss has converged. If not, execute step S106; otherwise, execute step S107.

[0130] In the specific implementation process, by judging whether the frequency domain supervision loss converges, we can further complete the knowledge distillation, that is, complete the training of the student network.

[0131] Exemplarily, the convergence of the frequency domain supervision loss can be expressed as: the performance of the student network is close to or better than the performance of the teacher network and no longer improves, that is, the frequency domain feature map loss no longer decreases significantly.

[0132] S106: Obtain a new sample data set to be trained, replace the new sample data set to be trained with the second sample data set to be trained, and return to execute step S104.

[0133] In the specific implementation process, if the frequency domain supervision loss has not converged, at this time, a new sample data set to be trained can be obtained, and the new sample data set to be trained can be replaced with the second sample data set to be trained, and the process returns to step S104 to continue supervising the student network until the frequency domain supervision loss converges, thereby effectively improving the performance of the student network.

[0134] S107. Complete knowledge distillation.

[0135] From the above description, it can be seen that in the technical solution provided by the embodiment of the present invention, by allowing the teacher network to supervise the student network in the frequency domain, it can be ensured that the student network can finely imitate the edge features of the teacher network, thereby making up for the defect that the student network is not ideal in learning the areas where the edge changes of the teacher network features are more obvious, and then the purpose of improving the performance of the student network can be achieved. At the same time, the technical solution provided by the embodiment of the present invention can be applied to the knowledge distillation of various networks, can realize the compression of complex networks, and is conducive to realizing the needs of deploying lightweight networks to mobile terminals.

[0136] Based on the same inventive concept, an embodiment of the present invention also provides a deep learning knowledge distillation system based on frequency domain supervision, such as Fig.10 As shown, the deep learning knowledge distillation system 300 based on frequency domain supervision may include:

[0137] The teacher network training unit 301 is used to obtain a first sample data set to be trained and a teacher network to be trained, and input the first sample data set to be trained into the teacher network to be trained until convergence, so as to obtain a trained first teacher network;

[0138] A student network design unit 302, configured to design a student network based on the first teacher network;

[0139] The student network training unit 303 is used to:

[0140] Obtain a second sample data set to be trained, and input the second sample data set to the first teacher network and the student network simultaneously for Fourier transform and Gaussian weight distribution processing, thereby obtaining a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network;

[0141] Calculating a frequency domain supervision loss based on the first frequency domain feature map and the second frequency domain feature map;

[0142] Determine whether the frequency domain supervision loss converges; if the frequency domain supervision loss does not converge, obtain a new sample data set to be trained, replace the new sample data set to be trained with the second sample data set to be trained, return to execute the second sample data set to be trained simultaneously input into the first teacher network and the student network for Fourier transform and Gaussian weight allocation processing, and obtain a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network; or, if the frequency domain supervision loss has converged, determine that the knowledge distillation is completed.

[0143] In a possible design, the student network design unit 302 is specifically used for:

[0144] The structural framework of the student network is designed to be the same as the structural framework of the first teacher network, and the number of feature map channels or layers of the student network is designed to be less than the number of feature map channels or layers of the first teacher network.

[0145] In a possible design, the student network training unit 303 is specifically used for:

[0146] Inputting the second sample data set to be trained into the first teacher network and the student network at the same time to obtain a first feature map corresponding to the first teacher network and a second feature map corresponding to the student network;

[0147] Performing Fourier transform on the first feature map and the second feature map to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map;

[0148] The high-frequency signals and low-frequency signals in the third feature map and the fourth feature map are sorted and weighted according to a preset arrangement to obtain the first frequency domain feature map and the second frequency domain feature map.

[0149] In a possible design, the student network training unit 303 is specifically used for:

[0150] The first feature map and the second feature map are Fourier transformed by using a two-dimensional discrete Fourier transform to obtain the third feature map and the fourth feature map.

[0151] In one possible design, the two-dimensional discrete Fourier transform follows the following formula:

[0152]

[0153] Among them, F(u,v) represents the frequency domain signal of the third feature map or the fourth feature map, f(x,y) represents the value at the coordinate (x,y) on the first feature map or the second feature map, M and N represent the number of rows and columns of the first feature map or the second feature map, respectively, and j and π are constants.

[0154] In a possible design, the student network training unit 303 is specifically used for:

[0155] Generate a Gaussian weighted initial image, wherein the Gaussian weighted initial image is an image with a white block in the middle and black blocks around it;

[0156] Obtaining a Gaussian filter weight template, and using the Gaussian filter weight template to perform convolution on the Gaussian weight initial image to obtain a Gaussian weight mask;

[0157] Based on the Gaussian weight mask, the high-frequency signals and the low-frequency signals in the third feature map and the fourth feature map are sorted and weights are set to obtain the first frequency domain feature map and the second frequency domain feature map.

[0158] In one possible design, the Gaussian filter weight template follows the following formula:

[0159]

[0160] Among them, the center of the Gaussian filter weight template is the origin, G(x1,y1) represents the weight of each point in the Gaussian filter weight template, (x1,y1) represents the coordinates of each point in the Gaussian filter weight template, and σ is a scalar. The larger its value, the larger the range covered by the Gaussian filter weight template.

[0161] In a possible design, the student network training unit 303 is specifically used for:

[0162] Based on the first frequency domain feature map and the second frequency domain feature map, using a loss function to calculate the frequency domain supervision loss;

[0163] The loss function is expressed as:

[0164]

[0165] in, Represented as the value of the i-th point in the first frequency domain feature map, y iIt is represented by the value of the i-th point in the second frequency domain feature map, n is represented by the number of all coordinate points in the first frequency domain feature map, n=W×H×C, W, H, and C respectively represent the width, height, and number of feature map channels of the first frequency domain feature map, and the number of all coordinate points in the second frequency domain feature map is the same as the number of all coordinate points in the first frequency domain feature map.

[0166] The deep learning knowledge distillation system 300 based on frequency domain supervision in the embodiment of the present invention is similar to the above Figure 1 The deep learning knowledge distillation method based on frequency domain supervision shown is an invention based on the same concept. Through the above detailed description of the deep learning knowledge distillation method based on frequency domain supervision, those skilled in the art can clearly understand the implementation process of the deep learning knowledge distillation system 300 based on frequency domain supervision in this embodiment, so for the sake of brevity of the specification, it will not be repeated here.

[0167] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, which can store at least one program. When the at least one program is executed by a processor, the above Figure 1 The deep learning knowledge distillation method based on frequency domain supervision is shown.

[0168] It should be understood that a computer-readable storage medium is any data storage device that can store data or programs, which can then be read by a computer system. Examples of computer-readable storage media include read-only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, and optical data storage devices, etc.

[0169] The computer readable storage medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

[0170] The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the above.

[0171] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A deep learning knowledge distillation method based on frequency domain supervision, characterized in that: include: Obtaining a first sample data set to be trained and a teacher network to be trained, and inputting the first sample data set to be trained into the teacher network to be trained until convergence, to obtain a trained first teacher network; Based on the first teacher network, design a student network; Obtain a second sample data set to be trained, and input the second sample data set to the first teacher network and the student network simultaneously for Fourier transform and Gaussian weight distribution processing, thereby obtaining a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network; Calculating a frequency domain supervision loss based on the first frequency domain feature map and the second frequency domain feature map; Determine whether the frequency domain supervision loss converges; if the frequency domain supervision loss does not converge, obtain a new sample data set to be trained, replace the new sample data set to be trained with the second sample data set to be trained, return to execute the second sample data set to be trained simultaneously input into the first teacher network and the student network for Fourier transform and Gaussian weight distribution processing, and obtain a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network; or, if the frequency domain supervision loss has converged, determine that knowledge distillation is completed; Wherein, after simultaneously inputting the second sample data set to be trained into the first teacher network and the student network for Fourier transform and Gaussian weight distribution processing, obtaining a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network, including: Inputting the second sample data set to be trained into the first teacher network and the student network at the same time to obtain a first feature map corresponding to the first teacher network and a second feature map corresponding to the student network; Performing Fourier transform on the first feature map and the second feature map to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map; Generate a Gaussian weighted initial image, wherein the Gaussian weighted initial image is an image with a white block in the middle and black blocks around it; Obtaining a Gaussian filter weight template, and using the Gaussian filter weight template to perform convolution on the Gaussian weight initial image to obtain a Gaussian weight mask; Based on the Gaussian weight mask, the high-frequency signals and the low-frequency signals in the third feature map and the fourth feature map are sorted and weights are set to obtain the first frequency domain feature map and the second frequency domain feature map.

2. The method according to claim 1, characterized in that Based on the first teacher network, a student network is designed, including: The structural framework of the student network is designed to be the same as the structural framework of the first teacher network, and the number of feature map channels or layers of the student network is designed to be less than the number of feature map channels or layers of the first teacher network.

3. The method according to claim 1, characterized in that Performing Fourier transform on the first feature map and the second feature map to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map, including: The first feature map and the second feature map are Fourier transformed by using a two-dimensional discrete Fourier transform to obtain the third feature map and the fourth feature map.

4. The method according to claim 3, characterized in that The two-dimensional discrete Fourier transform follows the following formula: in, a frequency domain signal representing the third feature map or the fourth feature map, represents the coordinates on the first feature map or the second feature map , M and N represent the number of rows and columns of the first feature map or the second feature map, respectively. is a constant.

5. The method according to claim 1, characterized in that The Gaussian filter weight template follows the following formula: Wherein, the center of the Gaussian filter weight template is the origin, represents the weight of each point in the Gaussian filter weight template, represents the coordinates of each point in the Gaussian filter weight template, It is a scalar. The larger its value is, the larger the range covered by the Gaussian filter weight template is.

6. The method according to any one of claims 1 to 5, characterized in that: Calculating the frequency domain supervision loss based on the first frequency domain feature map and the second frequency domain feature map includes: Based on the first frequency domain feature map and the second frequency domain feature map, using a loss function to calculate the frequency domain supervision loss; The loss function is expressed as: in, It is represented as i The value of the point, It is represented as i The value of the point, It is represented by the number of all coordinate points in the first frequency domain feature map, , , , They are respectively represented by the width, height, and number of feature map channels of the first frequency domain feature map, and the number of all coordinate points in the second frequency domain feature map is the same as the number of all coordinate points in the first frequency domain feature map.

7. A deep learning knowledge distillation system based on frequency domain supervision, characterized in that: include: A teacher network training unit is used to obtain a first sample data set to be trained and a teacher network to be trained, and input the first sample data set to be trained into the teacher network to be trained until convergence, so as to obtain a trained first teacher network; A student network design unit, used for designing a student network based on the first teacher network; Student online training modules for: Obtain a second sample data set to be trained, and input the second sample data set to the first teacher network and the student network simultaneously for Fourier transform and Gaussian weight distribution processing, thereby obtaining a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network; Calculating a frequency domain supervision loss based on the first frequency domain feature map and the second frequency domain feature map; Determine whether the frequency domain supervision loss converges; if the frequency domain supervision loss does not converge, obtain a new sample data set to be trained, replace the new sample data set to be trained with the second sample data set to be trained, return to execute the second sample data set to be trained simultaneously input into the first teacher network and the student network for Fourier transform and Gaussian weight distribution processing, and obtain a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network; or, if the frequency domain supervision loss has converged, determine that knowledge distillation is completed; Wherein, when the student network training unit is used to simultaneously input the second sample data set to be trained into the first teacher network and the student network for Fourier transform and Gaussian weight distribution processing, and obtains a first frequency domain feature map corresponding to the first teacher network and a second frequency domain feature map corresponding to the student network, it is specifically used to: Inputting the second sample data set to be trained into the first teacher network and the student network at the same time to obtain a first feature map corresponding to the first teacher network and a second feature map corresponding to the student network; Performing Fourier transform on the first feature map and the second feature map to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map; Generate a Gaussian weighted initial image, wherein the Gaussian weighted initial image is an image with a white block in the middle and black blocks around it; Obtaining a Gaussian filter weight template, and using the Gaussian filter weight template to perform convolution on the Gaussian weight initial image to obtain a Gaussian weight mask; Based on the Gaussian weight mask, the high-frequency signals and the low-frequency signals in the third feature map and the fourth feature map are sorted and weights are set to obtain the first frequency domain feature map and the second frequency domain feature map.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program; when the at least one program is executed by a processor, the method according to any one of claims 1 to 6 is executed.

Images