Knowledge distillation method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application disclose a knowledge distillation method and device, electronic equipment and a storage medium. A feature extraction layer of a to-be-classified network extracts a to-be-processed feature of a to-be-processed picture, and a classification layer of the to-be-classified network classifies the to-be-processed feature to obtain a classification result; wherein the to-be-classified network comprises a teacher network and a student network; the feature weight of the to-be-classified network is determined according to the to-be-processed feature of the to-be-classified network; the target loss function when the teacher network and the student network perform knowledge distillation is determined according to the to-be-processed feature, the feature weight and the classification result of different to-be-classified networks; and the network parameter of the student network is adjusted according to the target loss function. The embodiments of the present application improve the classification accuracy and speed, and improve the applicability of the classification network.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to a knowledge distillation method, apparatus, electronic device, and storage medium. Background Technology

[0002] During the production and construction of various power grid projects, case files are stored in the archive management system as scanned copies. When archivists verify various approval procedures, a crucial task is to confirm whether signatures, dates, and approval comments on relevant documents are handwritten or typed. Manual verification is not only inefficient but also prone to errors due to visual fatigue.

[0003] In existing technologies, deep learning models can be used to determine whether the font in a specified area of ​​a power grid project file is handwritten or machine-printed.

[0004] However, shallow network layers in deep learning models cannot meet accuracy requirements, while increasing the number of network layers will affect the inference speed of the model, consume more resources, and have poor applicability. Summary of the Invention

[0005] This application provides a knowledge distillation method, apparatus, electronic device, and storage medium to improve classification accuracy and speed, and enhance the applicability of classification networks.

[0006] In a first aspect, embodiments of this application provide a knowledge distillation method, which includes:

[0007] The process involves obtaining the unprocessed features of the image extracted by the feature extraction layer of the network to be classified, and the classification results obtained by the classification layer of the network to be classified in classifying the unprocessed features; wherein, the network to be classified includes the teacher network and the student network.

[0008] Based on the features to be processed in the network to be classified, determine the feature weights of the network to be classified.

[0009] Based on the features to be processed, feature weights, and classification results of different networks to be classified, the target loss function for knowledge distillation of teacher and student networks is determined.

[0010] Adjust the network parameters of the student network based on the target loss function.

[0011] Secondly, embodiments of this application also provide a knowledge distillation apparatus, which includes:

[0012] The parameter acquisition module is used to acquire the features to be processed of the image extracted by the feature extraction layer of the network to be classified, and the classification results obtained by the classification layer of the network to be classified to classify the features to be processed; wherein, the network to be classified includes the teacher network and the student network.

[0013] The feature weight determination module is used to determine the feature weights of the network to be classified based on the features to be processed in the network to be classified.

[0014] The objective loss function determination module is used to determine the objective loss function for knowledge distillation of teacher and student networks based on the features to be processed, feature weights and classification results of different networks to be classified.

[0015] The network parameter adjustment module is used to adjust the network parameters of the student network according to the target loss function.

[0016] Thirdly, embodiments of this application also provide an electronic device, which includes:

[0017] One or more processors;

[0018] Storage device for storing one or more programs;

[0019] When one or more programs are executed by one or more processors, the one or more processors implement any of the knowledge distillation methods provided in the embodiments of this application.

[0020] Fourthly, embodiments of this application also provide a storage medium including computer-executable instructions, which, when executed by a computer processor, are used to perform any of the knowledge distillation methods provided in embodiments of this application.

[0021] This application obtains the unprocessed features of the image extracted by the feature extraction layer of the network to be classified, and the classification results obtained by the classification layer of the network to be classified to classify the unprocessed features. The network to be classified includes a teacher network and a student network, facilitating subsequent supervised learning of the corresponding layers of the student network based on the feature extraction and classification layers of the teacher network. Based on the unprocessed features of the network to be classified, feature weights are determined, which strengthen the learning of important features and improve the classification accuracy of the student network. Based on the unprocessed features, feature weights, and classification results of different networks to be classified, a target loss function is determined for knowledge distillation between the teacher and student networks. The network parameters of the student network are adjusted according to the target loss function. The target loss function includes the loss functions of the feature extraction and classification layers. During knowledge distillation, the teacher and student networks can simultaneously learn the parameters of the teacher network's feature extraction and classification layers, improving the consistency between the student and teacher networks, increasing the classification accuracy and speed of the student network, and improving applicability due to the shallower number of layers in the student network and lower resource consumption. Therefore, the technical solution of this application solves the problems that shallow network layers cannot meet the accuracy requirements, and that increasing the number of network layers will affect the inference speed of the model, consume more resources, and have poor applicability. It achieves the effect of improving classification accuracy and speed, and improving the applicability of classification networks. Attached Figure Description

[0022] Figure 1 This is a flowchart of a knowledge distillation method according to Embodiment 1 of this application;

[0023] Figure 2a This is a flowchart of a knowledge distillation method according to Embodiment 2 of this application;

[0024] Figure 2b This is a schematic diagram of a feature weight determination process in Embodiment 2 of this application;

[0025] Figure 3a This is a flowchart of a knowledge distillation method according to Embodiment 3 of this application;

[0026] Figure 3b This is a schematic diagram of a target loss function determination process in Embodiment 3 of this application;

[0027] Figure 4 This is a schematic diagram of the structure of a knowledge distillation apparatus according to Embodiment 4 of this application;

[0028] Figure 5 This is a schematic diagram of the structure of an electronic device according to Embodiment 5 of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Example 1

[0032] Figure 1 This is a flowchart of a knowledge distillation method provided in Embodiment 1 of this application. This embodiment can be applied to the situation where knowledge distillation is performed on the classification network when classifying handwritten fonts in the archives of power grid construction. The method can be executed by a knowledge distillation device, which can be implemented in software and / or hardware and specifically configured in an electronic device, such as a server.

[0033] See Figure 1 The knowledge distillation method shown includes the following steps:

[0034] S110. Obtain the features to be processed of the image extracted by the feature extraction layer of the network to be classified, and the classification result obtained by the classification layer of the network to be classified to classify the features to be processed; wherein, the network to be classified includes the teacher network and the student network.

[0035] The network to be classified can be a network capable of classifying both handwritten and typed text, used for classifying handwritten text in documents related to power grid construction projects. For example, the network to be classified can be a convolutional neural network, a residual network, or a deep belief network. Specifically, the network to be classified includes a teacher network and a student network. The teacher network can be a pre-trained classification network, i.e., a network with fixed parameters, used to train the student network; the student network is the network that needs to be trained, i.e., the network parameters need to be adjusted, and its parameters are relatively shallow.

[0036] Knowledge distillation is a method that uses transferred knowledge to generate a more suitable student network for reasoning from a trained teacher network. The teacher network can be a deep network, while the student network can be a shallow network. For example, the teacher network could be ResNet34 (Residual Network 34, a 34-layer residual network), and the student network could be ResNet18 (Residual Network 18, an 18-layer residual network). Specifically, after constructing the teacher network, it is trained using a sample training set to obtain a well-trained teacher network, which is then fixed. The trained teacher network is then used to supervise the training of the student network, allowing the student network to learn the superior performance of the teacher network while reducing the number of parameters, thereby improving the reasoning accuracy of the student network.

[0037] The feature extraction layer can be a convolutional layer in the network to be classified, used to extract features from the input image. For example, the feature extraction layer can be the last layer in a multi-layer convolutional network, meaning there are no other convolutional layers between the feature extraction layer and the output layer. The image to be processed can be sample data, such as a signature image in a power grid construction project document. The features to be processed in the image can be the output image after feature extraction by the feature extraction layer, representing the features extracted by the feature extraction layer. The classification layer can be the output layer of the network to be classified, used to determine the classification result based on the features obtained from the feature extraction layer. For example, the classification layer can be a fully connected layer. The classification result can be the result of classifying the features to be processed. For example, when the network to be classified is used to classify whether a signature image is handwritten, the classification result can include handwritten and typed characters.

[0038] S120. Determine the feature weights of the network to be classified based on the features to be processed in the network to be classified.

[0039] Feature weights, determined based on the features to be processed, are used to supervise the training of the student network, enabling it to learn richer discriminative features from the teacher network. For example, an attention mechanism can be used to determine the feature weights of the network to be classified, allowing the student network to focus more on specific information in the teacher network, such as edge features. Specifically, based on the attention mechanism, the feature weights of the teacher network are obtained from its feature extraction layer; similarly, the feature weights of the student network are obtained from its feature extraction layer.

[0040] S130. Based on the features to be processed, feature weights, and classification results of different networks to be classified, determine the target loss function for knowledge distillation of the teacher network and the student network.

[0041] The target loss function can be used for student networks to perform distillation learning on teacher networks, supervising the learning of the student networks through the loss function. For example, the loss function can be at least one of KL divergence (KL divergence), mean squared error (MSE), L1 loss (mean absolute error), and deep mutual learning loss. The target loss function can be the total loss function of the distilled network composed of the networks to be classified.

[0042] Specifically, the loss function for the student network to learn the feature extraction layer of the teacher network can be determined based on the unprocessed features and feature weights of different networks to be classified; the loss function for the student network to learn the classification layer of the teacher network can be determined based on the classification results of different networks to be classified; and the target loss function can be obtained by weighted sum of the two loss functions.

[0043] S140. Adjust the network parameters of the student network according to the objective loss function.

[0044] The network parameters can be the parameters learned during the training of the student network. For example, the network parameters can be convolutional kernels. After obtaining the target loss function, the gradient descent algorithm is used to train the entire distillation network based on the target loss function, adjusting the network parameters of the student network.

[0045] The archives of power grid construction projects include a large number of project files. Many documents in these files require handwritten signatures and approvals from the responsible personnel to ensure their validity. These files are stored as scanned copies in the archives department, which then verifies various approval procedures. A crucial part of this verification is confirming whether signatures, dates, and approval comments on relevant documents are handwritten or typed, and issuing alerts for any non-compliance documents containing typed text. A single project awaiting review typically generates thousands of documents, each containing numerous items for review. Currently, these are all manually checked, which is not only inefficient but also prone to errors due to visual fatigue. Utilizing computer technology to locate items for review and automatically classify whether they are handwritten would significantly reduce the workload of staff. Currently, there is no automatic handwriting classification model in the review of power grid construction projects.

[0046] Distinguishing between machine-typed and handwritten text can be abstracted as an image binary classification task. Existing technologies have made significant strides in image classification, with deep learning-based network models becoming the mainstream. Classic networks include LeNet-5 (a 5-layer convolutional neural network), AlexNet (an 8-layer network), VGG-16 (a 16-layer network), GoogLeNet (a 22-layer network), and ResNet-101 (a 101-layer network). As the number of network layers increases, the classification accuracy of the model improves, and the algorithm becomes more robust. However, the number of parameters also increases, affecting the computational speed and consuming more computing resources, which is detrimental to the practical application of the technology. Conversely, shallow network models suffer from poor classification accuracy and also fail to meet the requirements for practical applications.

[0047] The technical solution of this embodiment obtains the unprocessed features of the image extracted by the feature extraction layer of the network to be classified, and the classification result obtained by the classification layer of the network to be classified to classify the unprocessed features. The network to be classified includes a teacher network and a student network, facilitating subsequent supervised learning of the corresponding layers of the student network based on the feature extraction and classification layers of the teacher network. Based on the unprocessed features of the network to be classified, the feature weights of the network to be classified are determined. These feature weights enhance the learning of important features, improving the classification accuracy of the subsequent student network. Based on the unprocessed features, feature weights, and classification results of different networks to be classified, the target loss function for knowledge distillation of the teacher and student networks is determined. The network parameters of the student network are adjusted according to the target loss function. The target loss function includes the loss functions of the feature extraction and classification layers. During knowledge distillation, the teacher and student networks can simultaneously learn the parameters of the teacher network's feature extraction and classification layers, improving the consistency between the student and teacher networks, increasing the classification accuracy and speed of the student network. Furthermore, since the student network itself has fewer layers, it consumes fewer resources, thus improving its applicability. Therefore, the technical solution of this application solves the problems that shallow network layers cannot meet the accuracy requirements, and that increasing the number of network layers will affect the inference speed of the model, consume more resources, and have poor applicability. It achieves the effect of improving classification accuracy and speed, and improving the applicability of classification networks.

[0048] Example 2

[0049] Figure 2a This is a flowchart of a knowledge distillation method provided in Embodiment 2 of this application. The technical solution of this embodiment is further refined based on the above technical solution.

[0050] Furthermore, the phrase "determine the feature weights of the network to be classified based on the features to be processed of the network to be classified" is further refined into: "extract the edge features of the features to be processed of the network to be classified; determine the feature weights of the network to be classified based on the edge features and through a weight mapping function" to determine the feature weights of the network to be classified.

[0051] See Figure 2a A knowledge distillation method is shown, comprising:

[0052] S210. Obtain the features to be processed of the image extracted by the feature extraction layer of the network to be classified, and the classification result obtained by the classification layer of the network to be classified to classify the features to be processed; wherein, the network to be classified includes the teacher network and the student network.

[0053] S220. Extract the edge features of the network to be classified.

[0054] Edge features can be discontinuous features among the features to be processed, used to determine weighted features. For example, edge features can be extracted using one of the following operators: Roberts operator, Sobel operator, and Laplacian edge detection operator. Edge features are important features in images, and learning from them can improve the accuracy of image classification. For handwritten and typed text, edge features are crucial for classification; extracting edge features improves classification accuracy.

[0055] In an optional embodiment, the edge features of the features to be processed in the network to be classified are extracted, including: determining the frequency domain features of the features to be processed by Fourier transform; determining the high-frequency features of the frequency domain features by inverse Fourier transform; and determining the edge features of the high-frequency features by the Sobel operator.

[0056] Frequency domain features can be the corresponding features of the feature to be processed in the frequency domain, that is, the feature data corresponding to the feature to be processed after transforming it from the time domain to the frequency domain. Specifically, the frequency domain features of the feature to be processed can be obtained by performing a Fourier transform on the feature to be processed. High-frequency features can be features that change drastically in the image and are used to extract edge features. Specifically, the high-frequency features of the frequency domain features can be obtained by performing an inverse Fourier transform on the frequency domain features. The Sobel operator is easy to implement spatially, has good edge detection performance, and is less affected by noise. Therefore, the Sobel operator is used to extract the edge features of high-frequency features.

[0057] The frequency domain features of the features to be processed are determined by Fourier transform; the high-frequency features of the frequency domain features are determined by inverse Fourier transform. Obtaining the high-frequency features of the features to be processed by Fourier transform and inverse Fourier transform of the image is beneficial for the subsequent extraction of edge features and improves the extraction efficiency of subsequent edge features. The edge features of the high-frequency features are determined by the Sobel operator. Based on the easy implementation, low noise and high accuracy of the Sobel operator, the extraction efficiency and accuracy of edge features are improved.

[0058] S230. Based on edge features, determine the feature weights of the network to be classified using a weight mapping function.

[0059] A weight mapping function can be a function that maps edge features to a weight matrix, used to determine feature weights based on edge features. For example, the weight mapping function can be a softmax function, which transforms edge features into a weight matrix between 0 and 1, making the network to be classified pay more attention to edge features in the image. During subsequent distillation learning, the feature weights can be used to guide the student network to focus on learning the teacher network's edge feature extraction method, enabling the student network to learn more robust edge features.

[0060] Figure 2b This is a schematic diagram illustrating the process of determining feature weights. For example... Figure 2b The diagram shows how to obtain frequency domain features by performing a Fourier transform on the features to be processed; how to obtain high-frequency features by performing an inverse Fourier transform on the frequency domain features; how to obtain edge features by processing the high-frequency features using the Sobel operator; and how to obtain feature weights by mapping the edge features using a weight mapping function.

[0061] S240. Based on the features to be processed, feature weights, and classification results of different networks to be classified, determine the target loss function for knowledge distillation of the teacher network and the student network.

[0062] S250. Adjust the network parameters of the student network according to the objective loss function.

[0063] The technical solution of this embodiment extracts the edge features of the network to be classified. Based on the edge features, the feature weights of the network to be classified are determined through a weight mapping function. Edge features are an important feature basis for image classification. By extracting edge features and mapping them to feature weights, the feature weights of the teacher network can supervise the feature weights of the student network when knowledge distillation is performed on the student network by the teacher network. This allows the student network to learn more refined edge features from the teacher network, improving the accuracy of feature extraction in the student network and providing a more accurate classification basis for subsequent classification, thus improving the accuracy of classification.

[0064] Example 3

[0065] Figure 3a This is a flowchart of a knowledge distillation method provided in Embodiment 3 of this application. The technical solution of this embodiment is further refined based on the above technical solution.

[0066] Furthermore, the statement "determine the target loss function for knowledge distillation of teacher and student networks based on the unprocessed features, feature weights, and classification results of different networks to be classified" is further refined into: "determine the feature loss function based on the unprocessed features and feature weights of different networks to be classified; determine the classification loss function based on the classification results of different networks to be classified; and determine the target loss function for knowledge distillation of teacher and student networks based on the feature loss function and the classification loss function," thus refining the method for determining the target loss function.

[0067] See Figure 3a A knowledge distillation method is shown, comprising:

[0068] S310. Obtain the features to be processed of the image extracted by the feature extraction layer of the network to be classified, and the classification result obtained by the classification layer of the network to be classified to classify the features to be processed; wherein, the network to be classified includes the teacher network and the student network.

[0069] S320. Determine the feature weights of the network to be classified based on the features to be processed in the network to be classified.

[0070] S330. Determine the feature loss function based on the features to be processed and the feature weights of different networks to be classified.

[0071] The feature loss function can be a loss function determined based on the features to be processed and the feature weights of the teacher network and the student network. It is used to supervise the learning of the feature extraction layer parameters of the teacher network by the student network, thereby improving the feature extraction capability of the feature extraction layer of the student network.

[0072] The feature loss function can be at least one type. For example, there can be two types: a loss function that includes feature weights and a loss function that does not include feature weights. Specifically, the feature loss function can be the mean squared error function.

[0073] In one optional embodiment, determining the feature loss function based on the features to be processed and the feature weights of different networks to be classified includes: determining a first loss function based on the features to be processed and the corresponding feature weights of different networks to be classified; determining a second loss function based on the features to be processed of different networks to be classified; and determining the feature loss function based on the first loss function and the second loss function.

[0074] The first loss function can be a loss function that includes feature weights, used to supervise the student network in learning the edge features of the feature to be processed. For example, the first loss function can be a mean squared error function. In this case, the product of the feature to be processed in the teacher network and its corresponding feature weight, and the product of the feature to be processed in the student network and its corresponding feature weight, can be used as two parameters in the mean squared error function to determine the first loss function. By using the first loss function, the student network can be supervised to learn richer edge features, with the aim of enabling the student network to learn the features of the teacher network, and even surpassing the teacher's performance.

[0075] The second loss function can be a loss function that does not include feature weights, used to supervise the student network's learning of the features to be processed. For example, the first loss function can be a mean squared error function, in which case the features to be processed in the teacher network and the features to be processed in the student network can be used as two parameters in the mean squared error function to determine the second loss function. In feature learning, it is desirable for the student network and the teacher network's last feature extraction layer to be as consistent as possible. Therefore, the second loss function prevents the student network from deliberately learning detailed edge features and deviating from the parameter distribution characteristics of the teacher model.

[0076] The weighted sum of the first loss function and the second loss function is used as the feature loss function. The weighting coefficients for the first and second loss functions can be adjusted based on experience and experimentation, and this application does not impose specific limitations. Optionally, the sum of the first and second loss functions can be used as the feature loss function, that is, the weights of the first and second loss functions are both 1.

[0077] A first loss function is determined based on the features to be processed and the corresponding feature weights of different networks to be classified. This allows the student network to learn rich edge features. A second loss function is determined based on the features to be processed of different networks to be classified. This allows the student network to maintain maximum consistency with the feature map extraction layer of the teacher network. Based on the first and second loss functions, a feature loss function is determined, enabling the student network to learn more edge features while also learning the feature extraction performance of the teacher network.

[0078] In an optional embodiment, determining a first loss function based on the features to be processed and the corresponding feature weights of different networks to be classified includes: obtaining teacher loss parameters based on the features to be processed and the corresponding feature weights of the teacher network; obtaining student loss parameters based on the features to be processed and the corresponding feature weights of the student network; and determining the first loss function using a mean squared loss function based on the teacher loss parameters and the student loss parameters.

[0079] The teacher loss parameter can be a calculated parameter in the first loss function determined based on the teacher network. Specifically, the teacher loss parameter can be the product of the feature to be processed in the teacher network and its corresponding feature weight. The student loss parameter can be a calculated parameter in the first loss function determined based on the student network. Specifically, the student loss parameter can be the product of the feature to be processed in the teacher network and its corresponding feature weight. The teacher loss parameter and the student loss parameter are used as two parameters of the mean squared loss function to determine the first loss function.

[0080] The teacher loss parameter is obtained based on the features to be processed and the corresponding feature weights of the teacher network; the student loss parameter is obtained based on the features to be processed and the corresponding feature weights of the student network; based on the teacher loss parameter and the student loss parameter, the first loss function is determined by the mean square loss function, and the feature weights are used as weighting coefficients for the features to be processed. This allows for a focus on learning edge features, improving the ability to extract edge features, and increasing the accuracy of subsequent classification.

[0081] In an optional embodiment, the student loss parameters are obtained based on the features to be processed and the corresponding feature weights of the student network, including: using the difference between the identity matrix and the corresponding feature weights of the student network as the weighted weights of the student network; and determining the student loss parameters based on the features to be processed and the weighted weights of the student network.

[0082] Using the difference between the identity matrix and the corresponding feature weights of the student network as the weights of the student network is equivalent to reducing the weight of edge detail texture regions of the features to be processed in the student network and increasing the weight of background regions. For example, when the student network is used to classify handwritten and typed characters, using the difference between the identity matrix and the corresponding feature weights of the student network as the weights ensures that smaller weights are assigned to font edges and larger weights to background regions. During supervised training, to reduce the first loss function, the student network is encouraged to focus more on the edge detail texture information of the features to be processed, learning richer features at the edges and weakening the learning of background regions, thus optimizing the student network. The product of the features to be processed in the student network and the weights is determined as the student loss parameter.

[0083] By using the difference between the identity matrix and the corresponding feature weights of the student network as the weights of the student network, and determining the student loss parameters based on the features to be processed and the weights of the student network, the student network is encouraged to focus more on the edge detail texture information of the features to be processed, thereby optimizing the student network.

[0084] S340. Determine the classification loss function based on the classification results of different networks to be classified.

[0085] The classification loss function can be a loss function determined based on the classification results of the teacher network and the student network. It is used to supervise the student network's learning of the classification layer parameters of the teacher network, thereby improving the classification accuracy of the student network's classification layer. For example, the classification loss function can be the KL divergence loss function, where the classification results of the teacher network and the student network are used as two parameters in the KL divergence loss function to determine the classification loss function.

[0086] S350. Based on the feature loss function and the classification loss function, determine the target loss function for knowledge distillation of the teacher network and the student network.

[0087] The target loss function for knowledge distillation of the teacher and student networks is determined based on the weighted sum of the feature loss function and the classification loss function. The weighting coefficients for the feature loss function and the classification loss function can be adjusted based on experience or experimentation, and this application does not impose specific limitations.

[0088] Figure 3b A schematic diagram illustrating the process of determining a target loss function. For example... Figure 3b As shown, the image to be processed is simultaneously input into both the teacher network and the student network. The features to be processed obtained from the last convolutional layer of each network are then acquired, such as... Figure 3b As shown in the dashed box on the left, the second loss function is determined based on the feature to be processed; the corresponding feature to be processed is weighted according to the feature weights of the teacher network and the weighted weights of the student network, as follows: Figure 3b As shown in the multiplier, Figure 3b The weighting coefficients used by the multipliers in the upper middle section are the feature weights corresponding to the teacher network. Figure 3b The multipliers in the lower middle section use weighting coefficients that are the difference between the identity matrix and the corresponding feature weights of the student network, i.e., the weighting weights. After the weighting operation, the weighted features to be processed are obtained, such as... Figure 3b As shown in the dashed box on the right, based on the feature to be processed, the first loss function is determined; the classification results obtained from the classification layers of the student network and the teacher network are obtained respectively, and the classification loss function is determined; the target loss function is obtained based on the first loss function, the second loss function, and the classification loss function, that is... Figure 3b The target loss function is shown in the figure.

[0089] S360. Adjust the network parameters of the student network according to the objective loss function.

[0090] The technical solution of this embodiment, by determining the feature loss function based on the unprocessed features and feature weights of different networks to be classified, enables the student network to learn the feature extraction ability of the teacher network; by determining the classification loss function based on the classification results of different networks to be classified, the student network can learn the ability of the teacher network to classify based on the extracted unprocessed features; by determining the target loss function when the teacher network and the student network perform knowledge distillation based on the feature loss function and the classification loss function, the student network can learn more robust unprocessed features that are beneficial for image classification, thereby improving the learning ability and classification accuracy of the student network, and achieving excellent classification performance even with a smaller student network.

[0091] Example 4

[0092] Figure 4 The diagram shown is a schematic representation of a knowledge distillation device according to Embodiment 4 of this application. This embodiment is applicable to the situation where knowledge distillation is performed on the classification network when classifying handwritten fonts in the archives of power grid construction. The specific structure of the knowledge distillation device is as follows:

[0093] The parameter acquisition module 410 is used to acquire the features to be processed of the image extracted by the feature extraction layer of the network to be classified, and the classification result obtained by the classification layer of the network to be classified to classify the features to be processed; wherein, the network to be classified includes the teacher network and the student network.

[0094] The feature weight determination module 420 is used to determine the feature weights of the network to be classified based on the features to be processed in the network to be classified.

[0095] The target loss function determination module 430 is used to determine the target loss function for knowledge distillation of teacher network and student network based on the features to be processed, feature weights and classification results of different networks to be classified.

[0096] The network parameter adjustment module 440 is used to adjust the network parameters of the student network according to the target loss function.

[0097] The technical solution of this embodiment obtains the unprocessed features of the image extracted by the feature extraction layer of the network to be classified, and the classification results obtained by the classification layer of the network to be classified, through the parameter acquisition module. The network to be classified includes a teacher network and a student network, facilitating supervised learning of the corresponding layers of the student network based on the feature extraction and classification layers of the teacher network. The feature weight determination module determines the feature weights of the network to be classified based on the unprocessed features, thereby strengthening the learning of important features and improving the classification accuracy of the student network. The target loss function determination module determines the target loss function for knowledge distillation between the teacher and student networks based on the unprocessed features, feature weights, and classification results of different networks to be classified. The network parameter adjustment module adjusts the network parameters of the student network according to the target loss function. The target loss function includes the loss functions of the feature extraction and classification layers. During knowledge distillation, the teacher and student networks can simultaneously learn the parameters of the feature extraction and classification layers of the teacher network, improving the consistency between the student and teacher networks, increasing the classification accuracy and speed of the student network, and improving applicability due to the shallower layers of the student network and lower resource consumption. Therefore, the technical solution of this application solves the problems that shallow network layers cannot meet the accuracy requirements, and that increasing the number of network layers will affect the inference speed of the model, consume more resources, and have poor applicability. It achieves the effect of improving classification accuracy and speed, and improving the applicability of classification networks.

[0098] Optionally, the feature weight determination module 420 includes:

[0099] The edge feature extraction unit is used to extract the edge features of the features to be processed by the network to be classified.

[0100] The edge feature mapping unit is used to determine the feature weights of the network to be classified based on edge features through a weight mapping function.

[0101] Optional, edge feature extraction unit, including:

[0102] The frequency domain feature determination subunit is used to determine the frequency domain features of the feature to be processed through Fourier transform.

[0103] The high-frequency feature determination subunit is used to determine the high-frequency features of the frequency domain features through inverse Fourier transform.

[0104] The edge feature determination subunit is used to determine the edge features of high-frequency features using the Sobel operator.

[0105] Optionally, the target loss function determination module 430 includes:

[0106] The feature loss function determination unit is used to determine the feature loss function based on the features to be processed and the feature weights of different networks to be classified.

[0107] The classification loss function determination unit is used to determine the classification loss function based on the classification results of different networks to be classified.

[0108] The target loss function combination unit is used to determine the target loss function for knowledge distillation of the teacher network and the student network based on the feature loss function and the classification loss function.

[0109] Optionally, the feature loss function determination unit includes:

[0110] The first loss function determination sub-unit is used to determine the first loss function based on the features to be processed and the corresponding feature weights of different networks to be classified.

[0111] The second loss function determination subunit is used to determine the second loss function based on the features to be processed in different networks to be classified.

[0112] The feature loss function combination subunit is used to determine the feature loss function based on the first loss function and the second loss function.

[0113] Optionally, the first loss function determines the sub-unit, specifically for: obtaining teacher loss parameters based on the features to be processed and the corresponding feature weights of the teacher network; obtaining student loss parameters based on the features to be processed and the corresponding feature weights of the student network; and determining the first loss function through the mean square loss function based on the teacher loss parameters and the student loss parameters.

[0114] Optionally, the first loss function determines the sub-unit and is also specifically used to: use the difference between the identity matrix and the corresponding feature weights of the student network as the weighting weights of the student network; and determine the student loss parameters based on the features to be processed and the weighting weights of the student network.

[0115] The knowledge distillation apparatus provided in this application can execute the knowledge distillation method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the knowledge distillation method.

[0116] Example 5

[0117] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of this application, as shown below. Figure 5 As shown, the electronic device includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of processors 510 in the electronic device can be one or more. Figure 5Taking a processor 510 as an example; the processor 510, memory 520, input device 530, and output device 540 in the electronic device can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0118] The memory 520, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the knowledge distillation method in this embodiment (e.g., parameter acquisition module 410, feature weight determination module 420, target loss function determination module 430, and network parameter adjustment module 440). The processor 510 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 520, thereby implementing the knowledge distillation method described above.

[0119] The memory 520 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 520 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 520 may further include memory remotely located relative to the processor 510, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] Input device 530 can be used to receive input character information and generate key signal inputs related to user settings and function control of the electronic device. Output device 540 may include display devices such as a display screen.

[0121] Example 6

[0122] Embodiment Six of this application also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to execute a knowledge distillation method. The method includes: acquiring the unprocessed features of the image to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by the classification layer of the network to be classified classifying the unprocessed features; wherein the network to be classified includes a teacher network and a student network; determining the feature weights of the network to be classified based on the unprocessed features of the network to be classified; determining the target loss function for knowledge distillation of the teacher network and the student network based on the unprocessed features, feature weights, and classification results of different networks to be classified; and adjusting the network parameters of the student network according to the target loss function.

[0123] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the method operations described above, but can also perform related operations in the knowledge distillation method provided in any embodiment of this application.

[0124] Based on the above description of the implementation methods, those skilled in the art can clearly understand that this application can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0125] It is worth noting that in the embodiments of the search device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0126] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A method of knowledge distillation, the method comprising: The method comprises the following steps: obtaining a to-be-processed feature of a to-be-processed picture extracted by a feature extraction layer of a to-be-classified network and a classification result obtained by classifying the to-be-processed feature by a classification layer of the to-be-classified network; wherein the to-be-classified network comprises a teacher network and a student network; determining a feature weight of the to-be-classified network according to the to-be-processed feature of the to-be-classified network; determining a target loss function when the teacher network and the student network perform knowledge distillation according to the to-be-processed feature, the feature weight and the classification result of different to-be-classified networks; adjusting network parameters of the student network according to the target loss function; the target loss function when the teacher network and the student network perform knowledge distillation is determined according to the to-be-processed feature, the feature weight and the classification result of different to-be-classified networks, comprising: determining a feature loss function according to the to-be-processed feature and the feature weight of different to-be-classified networks; determining a classification loss function according to the classification result of different to-be-classified networks; and determining the target loss function when the teacher network and the student network perform knowledge distillation according to the feature loss function and the classification loss function; the feature loss function is determined according to the to-be-processed feature and the corresponding feature weight of different to-be-classified networks, comprising: determining a first loss function according to the to-be-processed feature and the corresponding feature weight of the teacher network; determining a second loss function according to the to-be-processed feature of the to-be-classified network; determining the feature loss function according to the first loss function and the second loss function; the first loss function is determined according to the to-be-processed feature and the corresponding feature weight of the teacher network, comprising: obtaining a teacher loss parameter according to the to-be-processed feature and the corresponding feature weight of the teacher network; obtaining a student loss parameter according to the to-be-processed feature and the corresponding feature weight of the student network; determining the first loss function by a mean square loss function according to the teacher loss parameter and the student loss parameter.

2. The method of claim 1, wherein, the feature weight of the to-be-classified network is determined according to the to-be-processed feature of the to-be-classified network, comprising: extracting an edge feature of the to-be-processed feature of the to-be-classified network; determining the feature weight of the to-be-classified network by a weight mapping function according to the edge feature.

3. The method of claim 2, wherein, the edge feature of the to-be-processed feature of the to-be-classified network is extracted, comprising: determining a frequency domain feature of the to-be-processed feature by Fourier transform; determining a high frequency feature of the frequency domain feature by inverse Fourier transform; determining the edge feature of the high frequency feature by a Sobel operator.

4. The method of claim 1, wherein, the student loss parameter is obtained according to the to-be-processed feature and the corresponding feature weight of the student network, comprising: taking a difference value between a unit matrix and the corresponding feature weight of the student network as a weighting weight of the student network; determining the student loss parameter according to the to-be-processed feature and the weighting weight of the student network.

5. A knowledge distillation apparatus, characterized by, The method comprises the following steps: The parameter acquisition module is configured to acquire a to-be-processed feature of a to-be-processed picture extracted by a feature extraction layer of a to-be-classified network and a classification result obtained by classifying the to-be-processed feature by a classification layer of the to-be-classified network, wherein the to-be-classified network comprises a teacher network and a student network; The feature weight determination module is configured to determine a feature weight of the to-be-classified network according to the to-be-processed feature of the to-be-classified network; The target loss function determination module is configured to determine a target loss function for knowledge distillation of the teacher network and the student network according to the to-be-processed feature, the feature weight and the classification result of different to-be-classified networks; The network parameter adjustment module is configured to adjust a network parameter of the student network according to the target loss function; The target loss function determination module is specifically configured to: determine a feature loss function according to the to-be-processed feature and the feature weight of different to-be-classified networks, determine a classification loss function according to the classification result of different to-be-classified networks, and determine the target loss function for knowledge distillation of the teacher network and the student network according to the feature loss function and the classification loss function; The determination of the feature loss function according to the to-be-processed feature and the feature weight of different to-be-classified networks comprises: determining a first loss function according to the to-be-processed feature and the corresponding feature weight of different to-be-classified networks; determining a second loss function according to the to-be-processed feature of different to-be-classified networks; determining the feature loss function according to the first loss function and the second loss function; The determination of the first loss function according to the to-be-processed feature and the corresponding feature weight of different to-be-classified networks comprises: obtaining a teacher loss parameter according to the to-be-processed feature and the corresponding feature weight of the teacher network; obtaining a student loss parameter according to the to-be-processed feature and the corresponding feature weight of the student network; determining the first loss function by a mean square loss function according to the teacher loss parameter and the student loss parameter.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the knowledge distillation method in any one of claims 1-4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the knowledge distillation method in any one of claims 1-4.

Images