Knowledge distillation method, device, equipment, storage medium and program product

Abstract

The invention relates to a knowledge distillation method, device and equipment, a storage medium and a program product, and relates to the technical field of machine learning. The method comprises the following steps: determining a loss function, output distribution and feature information of a teacher model; determining a loss function of the initial student model based on the loss function of the teacher model and the output distribution and feature information; the initial student model has partial functions of the teacher model; performing knowledge distillation on the initial student model based on the feature information, the output distribution and a loss function of the initial student model to obtain a target student model; the similarity between the output distribution of the target student model and the output distribution of the teacher model is greater than a similarity threshold. Therefore, the calculation complexity and the resource consumption of the student model can be effectively reduced, and the performance of the student model can be guaranteed.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, in particular to the field of model compression technology, and specifically to a knowledge distillation method, device, equipment, storage medium and program product. Background Art

[0002] The widespread application of artificial intelligence technology has led to the increasing complexity of deep learning models, and the demand for computing resources and storage space has continued to grow. However, mobile devices such as vehicles are limited by hardware conditions and it is difficult to deploy and maintain large models. In addition, autonomous driving and intelligent monitoring require high real-time performance. Therefore, model compression technology has emerged. This technology can significantly reduce the demand for computing resources and storage space while maintaining a certain performance by reducing the number of model parameters and the amount of calculation.

[0003] Related technology CN116740019A proposes one-to-one learning of intermediate layer feature information at different downsampling stages, but there are problems of information loss and overfitting, resulting in limited improvement in algorithm accuracy.

[0004] Related technology CN114565045A proposes to use foreground attention mask and background attention mask to perform knowledge transfer respectively, but does not distinguish the ratio of foreground and background, resulting in limited improvement in algorithm accuracy.

[0005] Therefore, it is necessary to explore effective ways to reduce computational complexity and resource consumption while ensuring the performance of the student model. Summary of the invention

[0006] The present application provides a knowledge distillation method, device, equipment, storage medium and program product to at least solve the technical problem that important features are lost in the related art, resulting in low algorithm accuracy. The technical solution of the present application is as follows:

[0007] According to the first aspect provided by the present application, a knowledge distillation method is provided, including: determining the loss function, output distribution and feature information of a teacher model; determining the loss function of an initial student model based on the loss function, output distribution and feature information of the teacher model; the initial student model has some functions of the teacher model; performing knowledge distillation on the initial student model based on the feature information, output distribution and the loss function of the initial student model to obtain a target student model; the similarity between the output distribution of the target student model and the output distribution of the teacher model is greater than a similarity threshold.

[0008] According to the above-mentioned technical means, the present application can effectively transfer the knowledge contained in a large and high-precision model (i.e., the teacher model) to a small and low-complexity model (i.e., the student model) by using knowledge distillation technology to adapt to application environments with relatively limited resources, and divide the feature information into different dimensions to ensure that the student model does not miss any key feature information during the learning process, thereby avoiding the problem of losing important features in related technologies. Therefore, the present application can effectively reduce the computational complexity and resource consumption of the student model while ensuring the performance of the student model.

[0009] In one possible manner, the feature information includes spatial feature information and channel feature information; based on the teacher model output, the feature information, and the loss function of the teacher model, the loss function of the initial student model is determined, including: determining a first loss function corresponding to the spatial feature information, a second loss function corresponding to the channel feature information, and a third loss function corresponding to the output distribution; weighting the loss function of the teacher model, the first loss function corresponding to the spatial feature information, the second loss function corresponding to the first channel feature information, and the third loss function corresponding to the output distribution to obtain the loss function of the initial student model.

[0010] According to the above-mentioned technical means, this application designs the first loss function and the second loss function for the spatial feature information and the channel feature information respectively, so that the student model can more comprehensively capture the key features in the data, and by weighting the loss function of the teacher model, the first loss function of the spatial feature information, the second loss function of the channel feature information and the third loss function of the output distribution, the difference between the student model and the teacher model can be more accurately measured, thereby guiding the student model to perform more effective optimization.

[0011] In one possible manner, determining the first loss function corresponding to the spatial feature information includes: determining the adaptive weight of the spatial feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model; and determining the first loss function corresponding to the spatial feature information based on the adaptive weight of the spatial feature information.

[0012] According to the above technical means, the present application can reflect the importance of spatial feature information in the model learning process through adaptive weights, ensure that the student model can focus on these key features during the training process, and by determining the first loss function, enable the student model to more accurately extract and retain spatial feature information similar to the teacher model, thereby improving the performance of the student model.

[0013] In one possible approach, based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model, the adaptive weight of the spatial feature information is determined, including: when the channel dimensions of the intermediate layer features of the teacher model and the intermediate layer features of the initial student model are aligned, average pooling is performed on the intermediate layer features of the teacher model to obtain the first channel feature information of the teacher model, and average pooling is performed on the intermediate layer features of the initial student model to obtain the second channel feature information of the initial student model; dot product calculation is performed on the first channel feature information and the second channel feature information to obtain the adaptive weight of the spatial feature information.

[0014] According to the above technical means, the present application can determine the adaptive weights of spatial feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model, which can significantly improve the learning efficiency, generalization ability and effectiveness of knowledge distillation of the model.

[0015] In one possible manner, based on the adaptive weight of the spatial feature information, a first loss function corresponding to the spatial feature information is determined, including: when the spatial dimensions between the teacher model and the initial student model are aligned, determining the first spatial feature information of the teacher model in the first dimension and the second spatial feature information of the initial student model in the first dimension; the first dimension is the dimension after the spatial dimensions between the teacher model and the initial student model are aligned; based on the adaptive weight of the spatial feature information, weighting the first spatial feature information and the second spatial feature information to obtain the first loss function corresponding to the spatial feature information.

[0016] According to the above-mentioned technical means, the present application can determine the first loss function corresponding to the spatial feature information based on the adaptive weight of the spatial feature information, and operate under the condition of spatial dimension alignment, which can significantly improve the model's targetedness, spatial feature learning ability, generalization performance, efficiency of knowledge distillation and stability of the training process.

[0017] In one possible manner, determining the second loss function corresponding to the channel feature information includes: determining the adaptive weight of the channel feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model; and determining the second loss function corresponding to the channel feature information based on the adaptive weight of the channel feature information.

[0018] According to the above technical means, the present application can reflect the importance of channel feature information in the model learning process through adaptive weights, ensure that the student model can focus on these key features during the training process, and by determining the second loss function, enable the student model to more accurately extract and retain channel feature information similar to the teacher model, thereby improving the performance of the student model.

[0019] In one possible method, based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model, the adaptive weights of the channel features are determined, including: performing dimension swapping on the intermediate layer features of the teacher model to obtain the first intermediate layer features, and performing dimension swapping on the intermediate layer features of the initial student model to obtain the second intermediate layer features; determining the first height dimension information and the first width dimension information based on the first intermediate feature value, and determining the second height dimension information and the second width dimension information based on the second intermediate layer feature; performing weighted processing on the first dot product and the second dot product to obtain the adaptive weights of the channel features; the first dot product is the dot product between the first height dimension information and the second height dimension information; the second dot product is the dot product between the first width dimension information and the second width dimension information.

[0020] According to the above technical means, the present application can flexibly adjust the dimensional order of the intermediate layer features through dimensional exchange, making it easier to process and analyze, and by extracting the height dimension information and the width dimension information respectively and calculating the dot product between them, it can more carefully reflect the similarities and differences in the channel characteristics between the teacher model and the initial student model.

[0021] In one possible manner, based on the adaptive weight of the channel feature, a second loss function corresponding to the channel feature information is determined, including: based on the adaptive weight of the channel feature information, weighted processing is performed on the first channel feature information and the second channel feature information to obtain the second loss function corresponding to the channel feature information.

[0022] According to the above-mentioned technical means, the present application can reflect the importance of different channel feature information in the model learning process through adaptive weights, and by weighted processing of the first channel feature information and the second channel feature information, the second loss function can pay more attention to those channel features that have a greater impact on the model performance, thereby improving the targetedness of the loss function.

[0023] In one possible manner, determining a third loss function corresponding to the output distribution includes: taking the difference between the output distribution of the teacher model and the output distribution of the initial student model to determine the third loss function corresponding to the output distribution.

[0024] According to the above technical means, the present application can directly reflect the gap in task performance between the teacher model and the initial student model by calculating the difference between the outputs, thereby determining the third loss function corresponding to the output distribution through the difference, which can significantly promote the robustness and generalization ability of knowledge distillation.

[0025] According to the second aspect provided by the present application, a knowledge distillation device is provided, including: a determination unit and a training unit; the determination unit is used to determine the loss function, output distribution and feature information of a teacher model; the determination unit is also used to determine the loss function of an initial student model based on the loss function, output distribution and feature information of the teacher model; the initial student model has some functions of the teacher model; the training unit is used to perform knowledge distillation on the initial student model based on the feature information, output distribution and loss function of the initial student model to obtain a target student model; the similarity between the output distribution of the target student model and the output distribution of the teacher model is greater than a similarity threshold.

[0026] In one possible manner, the determination unit is specifically used to: determine a first loss function corresponding to the spatial feature information, a second loss function corresponding to the channel feature information, and a third loss function corresponding to the output distribution; weightedly process the loss function of the teacher model, the first loss function corresponding to the spatial feature information, the second loss function corresponding to the first channel feature information, and the third loss function corresponding to the output distribution to obtain the loss function of the initial student model.

[0027] In one possible manner, the determination unit is specifically used to: determine the adaptive weight of the spatial feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model; and determine the first loss function corresponding to the spatial feature information based on the adaptive weight of the spatial feature information.

[0028] In one possible manner, the determination unit is specifically used to: when the channel dimensions of the intermediate layer features of the teacher model are aligned with those of the intermediate layer features of the initial student model, perform average pooling on the intermediate layer features of the teacher model to obtain the first channel feature information of the teacher model, and perform average pooling on the intermediate layer features of the initial student model to obtain the second channel feature information of the initial student model; perform dot product calculation on the first channel feature information and the second channel feature information to obtain the adaptive weight of the spatial feature information.

[0029] In one possible manner, the determination unit is specifically used to: determine the first spatial feature information of the teacher model in the first dimension and the second spatial feature information of the initial student model in the first dimension when the spatial dimensions between the teacher model and the initial student model are aligned; the first dimension is the dimension after the spatial dimensions between the teacher model and the initial student model are aligned; based on the adaptive weight of the spatial feature information, the first spatial feature information and the second spatial feature information are weighted to obtain a first loss function corresponding to the spatial feature information.

[0030] In one possible manner, the determination unit is specifically used to: determine the adaptive weight of the channel feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model; and determine the second loss function corresponding to the channel feature information based on the adaptive weight of the channel feature information.

[0031] In one possible manner, the determination unit is specifically used to: perform dimension swapping on the intermediate layer features of the teacher model to obtain the first intermediate layer features, and perform dimension swapping on the intermediate layer features of the initial student model to obtain the second intermediate layer features; determine the first height dimension information and the first width dimension information based on the first intermediate feature value, and determine the second height dimension information and the second width dimension information based on the second intermediate layer feature; perform weighted processing on the first dot product and the second dot product to obtain the adaptive weight of the channel feature; the first dot product is the dot product between the first height dimension information and the second height dimension information; the second dot product is the dot product between the first width dimension information and the second width dimension information.

[0032] In a possible manner, the determination unit is specifically used to: perform weighted processing on the first channel feature information and the second channel feature information based on the adaptive weight of the channel feature information to obtain a second loss function corresponding to the channel feature information.

[0033] In one possible manner, the determination unit is specifically used to: determine a third loss function corresponding to the output distribution by taking the difference between the output distribution of the teacher model and the output distribution of the initial student model.

[0034] According to the third aspect provided by the present application, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement the method of the above-mentioned first aspect and any possible implementation manner thereof.

[0035] According to the fourth aspect provided by the present application, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method in the above-mentioned first aspect and any possible implementation method thereof.

[0036] According to the fifth aspect provided by the present application, a computer program product is provided, the computer program product comprising computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method of the above-mentioned first aspect and any possible implementation manner thereof.

[0037] Therefore, the above technical features of the present application have the following beneficial effects:

[0038] (1) By using knowledge distillation technology, the knowledge contained in a large and high-precision model (i.e., the teacher model) can be effectively transferred to a small and low-complexity model (i.e., the student model) to adapt to application environments with relatively limited resources, and the feature information can be divided into different dimensions to ensure that the student model does not miss any key feature information during the learning process, thereby avoiding the problem of losing important features in related technologies. Therefore, the present application can effectively reduce the computational complexity and resource consumption of the student model while ensuring the performance of the student model.

[0039] (2) By designing the first loss function and the second loss function for spatial feature information and channel feature information respectively, the student model can capture the key features of the data more comprehensively. Moreover, by weighting the loss function of the teacher model, the first loss function of spatial feature information, the second loss function of channel feature information and the third loss function of output distribution, the difference between the student model and the teacher model can be measured more accurately, thereby guiding the student model to perform more effective optimization.

[0040] (3) Adaptive weights can be used to reflect the importance of spatial feature information in the model learning process, ensuring that the student model can focus on these key features during training. By determining the first loss function, the student model can more accurately extract and retain spatial feature information similar to that of the teacher model, thereby improving the performance of the student model.

[0041] (4) The adaptive weights of spatial feature information can be determined based on the intermediate-layer features of the teacher model and the intermediate-layer features of the initial student model, which can significantly improve the learning efficiency, generalization ability, and effectiveness of knowledge distillation of the model.

[0042] (5) The first loss function corresponding to the spatial feature information can be determined based on the adaptive weight of the spatial feature information, and the operation can be performed under the condition of spatial dimension alignment, which can significantly improve the model's targeting, spatial feature learning ability, generalization performance, efficiency of knowledge distillation and stability of the training process.

[0043] (6) Adaptive weights can be used to reflect the importance of channel feature information in the model learning process, ensuring that the student model can focus on these key features during training. By determining the second loss function, the student model can more accurately extract and retain channel feature information similar to that of the teacher model, thereby improving the performance of the student model.

[0044] (7) The dimensional order of the intermediate layer features can be flexibly adjusted through dimension exchange, making them easier to process and analyze. In addition, by extracting the height dimension information and the width dimension information respectively and calculating the dot product between them, the similarities and differences in the channel characteristics between the teacher model and the initial student model can be more carefully reflected.

[0045] (8) The importance of different channel feature information in the model learning process can be reflected through adaptive weights, and by weighted processing of the first channel feature information and the second channel feature information, the second loss function can pay more attention to those channel features that have a greater impact on the model performance, thereby improving the pertinence of the loss function.

[0046] (9) By calculating the difference between the output of the teacher model and the initial student model, the gap in task performance between the two can be directly reflected. The third loss function corresponding to the output distribution can be determined by the difference, which can significantly promote the robustness and generalization ability of knowledge distillation.

[0047] It should be noted that the technical effects brought about by any implementation method in the second to fifth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here.

[0048] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0050] Figure 1 is a flow chart of a knowledge distillation method according to an exemplary embodiment;

[0051] Figure 2 is a schematic diagram of a loss function determination process corresponding to a provided spatial feature information according to an exemplary embodiment;

[0052] Figure 3 is a schematic diagram of a loss function determination process corresponding to a provided channel feature information according to an exemplary embodiment;

[0053] Figure 4 is a schematic diagram showing a model compression process according to an exemplary embodiment;

[0054] Figure 5 is a schematic diagram of a knowledge distillation process according to an exemplary embodiment;

[0055] Figure 6 is a block diagram of a knowledge distillation device according to an exemplary embodiment;

[0056] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0058] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0059] In the embodiments of the present application, words such as "exemplary", "for example", or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary", "for example", or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example", or "for example" is intended to present related concepts in a specific way.

[0060] First, the relevant technologies involved in this application are explained to facilitate understanding by those skilled in the art.

[0061] With the vigorous development of artificial intelligence technology, deep learning models are increasingly used in various fields. However, these models are often very complex and require huge computing resources and storage space. Especially on mobile devices such as vehicles, the deployment and maintenance of large models become particularly difficult due to hardware limitations. At the same time, in scenarios with extremely high real-time requirements such as autonomous driving and intelligent monitoring, the efficient operation of the model is even more crucial.

[0062] In the face of these challenges, model compression technology has emerged. The main goal of this technology is to reduce the number of model parameters and the amount of calculation, thereby achieving lightweight models. Through lightweight processing, the model can significantly reduce the demand for computing resources and storage space while maintaining a certain performance, making it more suitable for the use environment of mobile devices.

[0063] Among the various means of model compression, knowledge distillation is a particularly eye-catching method. The basic idea of ​​knowledge distillation is to transfer the knowledge in the large, high-precision "teacher" model to the small, lightweight "student" model. In this process, the "teacher" model will provide rich supervision information to help the "student" model learn more effective feature representation and decision logic. In this way, the "student" model can significantly reduce its size and increase its running speed without sacrificing too much performance.

[0064] The advantage of knowledge distillation is that it can achieve lightweight models while maintaining model performance. This is especially important for mobile devices, because lightweight models can significantly reduce power consumption and improve operating efficiency, thereby extending the battery life of the device and improving user experience. In addition, knowledge distillation can also promote the migration and deployment of models between different devices, providing strong support for the widespread application of artificial intelligence technology.

[0065] The application of knowledge distillation is even more significant in scenarios with high real-time requirements such as autonomous driving and intelligent monitoring. Lightweight models can achieve rapid response and accurate judgment of complex scenarios, thereby improving the safety and reliability of the system. At the same time, lightweight models can also reduce the demand for hardware resources, allowing more devices to support advanced artificial intelligence functions.

[0066] Based on this, knowledge distillation, as an important means of model compression, provides the possibility for the application of artificial intelligence technology on mobile devices. By reducing model parameters and computational complexity to achieve model lightweighting, knowledge distillation can meet the urgent needs of mobile devices for low power consumption and high efficiency while maintaining performance. With the continuous development of technology, knowledge distillation will play an important role in more fields and promote the widespread application and in-depth development of artificial intelligence technology.

[0067] However, during the feature distillation process, information is often lost due to the significant differences in feature levels and expressiveness between the teacher network and the student network. To address this challenge, an effective multi-stage knowledge distillation strategy needs to be designed and implemented. The strategy should ensure that the student model can effectively extract knowledge from the teacher model at different feature levels, thereby minimizing information loss. This requires us to have a deep understanding of the differences in feature representation between the teacher network and the student network, and to develop a suitable knowledge distillation scheme accordingly.

[0068] Moreover, in the object recognition task, feature selection and importance determination are crucial. Not all features are equally important to the final recognition result. Therefore, we need to filter out the most critical features for the recognition task from the teacher model and accurately determine the importance of these features at different stages. This requires us to use advanced feature selection techniques and importance assessment methods to ensure that the selected features can maximize the recognition performance.

[0069] In addition, in image recognition tasks, the ratio of foreground to background varies significantly, and the impact on recognition results varies. Therefore, in the distillation process, it is necessary to effectively adjust the ratio of foreground and background features to maximize recognition accuracy. Therefore, it is necessary to deeply understand the role of foreground and background features in recognition tasks and formulate appropriate adjustment strategies accordingly.

[0070] Related technology 1 proposes one-to-one learning of the intermediate layer feature information at different downsampling stages, but there are problems of information loss and overfitting, resulting in limited improvement in algorithm accuracy.

[0071] Related technology 2 proposes to use foreground attention mask and background attention mask to transfer knowledge separately, but does not distinguish the ratio of foreground and background, resulting in limited improvement in algorithm accuracy.

[0072] Therefore, it is necessary to explore effective ways to reduce computational complexity and resource consumption while ensuring the performance of the student model.

[0073] As mentioned in the background technology, in order to solve the problem of low accuracy of the student model after knowledge distillation in the related technology, the present application provides a knowledge distillation method, which can determine the loss function, output distribution and feature information of the teacher model, and determine the loss function of the initial student model based on the loss function, output distribution and feature information of the teacher model; the initial student model has some functions of the teacher model, so as to further perform knowledge distillation on the initial student model based on the feature information, output distribution and the loss function of the initial student model to obtain a target student model, wherein the similarity between the output distribution of the target student model and the output distribution of the teacher model is greater than the similarity threshold.

[0074] Based on this, the present application can effectively transfer the knowledge contained in a large and high-precision model (i.e., the teacher model) to a small and low-complexity model (i.e., the student model) by using knowledge distillation technology to adapt to application environments with relatively limited resources, and divide the feature information into different dimensions to ensure that the student model does not miss any key feature information during the learning process, avoiding the problem of losing important features in related technologies. Therefore, the present application can effectively reduce the computational complexity and resource consumption of the student model while ensuring the performance of the student model.

[0075] For ease of understanding, the knowledge distillation method provided in this application is specifically introduced below with reference to the accompanying drawings.

[0076] Figure 1 is a flow chart of a knowledge distillation method according to an exemplary embodiment. Figure 1 As shown, the knowledge distillation method includes the following steps: S101-S103.

[0077] S101. Determine the loss function, output distribution and feature information of the teacher model.

[0078] In one possible implementation, the knowledge distillation device may obtain a teacher model. The teacher model is a pre-trained model. The teacher model may be a model that is pre-trained and stored in the knowledge distillation device, or a model that is obtained by the knowledge distillation device after receiving the knowledge distillation task and training the initial teacher model through the data set.

[0079] In one embodiment, the knowledge distillation device can determine the loss function used when training the teacher model, as well as the distribution of the teacher's output on the test set after training is completed.

[0080] The output distribution of the teacher model is usually a probability distribution, which is used to represent the model's predicted probability that the input data belongs to each category. In classification problems, this output distribution is usually a discrete probability distribution, where the probability value of each category is between 0 and 1, and the sum of the probability values ​​of all categories is 1.

[0081] In addition, the loss function of the teacher model can be set according to actual needs. For example, the loss function of the teacher model can be mean square error or cross entropy loss. This application does not make specific restrictions on this.

[0082] In another embodiment, the knowledge distillation device can also determine the feature information of the teacher model. The feature information of the teacher model generally refers to the feature representation of the input data learned by the model during the training process. These feature representations are the basis for the model to make predictions and classifications. The feature information may include intermediate layer features, spatial feature information, channel feature information, etc.

[0083] In deep learning, the teacher model usually contains multiple layers of neurons and connections, each of which can extract different features of the input data. For example, in Convolutional Neural Networks (CNNs), neurons in lower layers may be responsible for extracting local features of the input image (such as edges, textures, etc.), while neurons in higher layers may be responsible for extracting more abstract and global features (such as the shape and contour of objects).

[0084] In one possible implementation, before determining the loss function, output distribution, and feature information of the teacher model, the knowledge distillation device can freeze the parameters of the pre-trained teacher model and load it into the algorithm framework to avoid updating the parameters of the teacher model during the distillation process, which may introduce uncertainty and lead to poor distillation results. In addition, freezing the parameters of the teacher model can ensure that during the distillation process, the student model can learn feature representations consistent with the teacher model.

[0085] S102. Determine the loss function of the initial student model based on the loss function, output distribution and feature information of the teacher model.

[0086] Among them, the initial student model has some functions of the teacher model.

[0087] It should be noted that the initial student model is a smaller, simpler model that is designed to learn and imitate the behavior or prediction results of a more complex and better performing teacher model. Before training begins, the initial student model can have some of the functions or knowledge of the teacher model, so that it can converge to the optimal solution faster during training, reduce training time, and improve training efficiency.

[0088] Exemplarily, in the case where the teacher model is a classification model, the initial student model may include basic classification capabilities.

[0089] In one possible implementation, the knowledge distillation device can obtain a student model. The student model can be a pre-trained model or an untrained student model, and this application does not impose any restrictions on this.

[0090] Specifically, when the initial student model is a pre-trained model, the initial student model may be a model that is pre-trained and stored in the knowledge distillation device, or a model that is obtained by the knowledge distillation device training the initial teacher model through a data set after receiving the knowledge distillation task. When the initial student model is a model that has not been pre-trained, the weight coefficients of the parameters included in the initial student model may be determined by random initialization or may be preset.

[0091] In one embodiment, the knowledge distillation device can determine a first loss function corresponding to the spatial feature information, a second loss function corresponding to the channel feature information, and a third loss function corresponding to the output distribution.

[0092] Among them, the knowledge distillation device can determine the adaptive weight of the spatial feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model. The knowledge distillation device can determine the second loss function based on the adaptive weight of the spatial feature information. The specific implementation method of the knowledge distillation device determining the first loss function corresponding to the spatial feature information can refer to the following S201-S202. It will not be repeated here.

[0093] The knowledge distillation device can determine the adaptive weight of the channel feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model. The knowledge distillation device can determine the second loss function corresponding to the channel feature information based on the adaptive weight of the channel feature information. The specific implementation method of the knowledge distillation device determining the first loss function corresponding to the channel feature information can refer to the following S301-S302. It will not be repeated here.

[0094] In addition, the knowledge distillation device can determine the third loss function corresponding to the output distribution based on the difference between the output distribution of the teacher model and the output distribution of the initial student model. The specific implementation of the knowledge distillation device determining the third loss function corresponding to the output distribution can refer to the following S401. No further details will be given here.

[0095] In another embodiment, the knowledge distillation device can determine the loss function of the teacher model, the first loss function corresponding to the spatial feature information, the second loss function corresponding to the first channel feature information, and the third loss function corresponding to the output distribution to obtain the loss function of the initial student model.

[0096] Exemplarily, the loss function of the initial student model satisfies the following first formula:

[0097] L s =λ1L SAT +λ2L CAT +λ3L OT +λ4L GT First formula

[0098] Among them, L s It can be used to characterize the loss function of the initial student. SAT It can be used to characterize the first loss function corresponding to the spatial feature information. CAT It can be used to characterize the second loss function corresponding to the channel feature information. OT It can be used to characterize the third loss function corresponding to the output distribution. GTIt can be used to characterize the loss function of the teacher model, that is, the loss function between the teacher model and the true label. λ1 can be used to characterize the weight coefficient of the first loss function corresponding to the spatial feature information. λ2 can be used to characterize the weight coefficient of the second loss function corresponding to the channel feature information. λ3 can be used to characterize the weight coefficient of the third loss function corresponding to the output distribution. λ4 can be used to characterize the weight coefficient of the loss function of the teacher model.

[0099] Optionally, λ1, λ2, λ3 and λ4 can be set according to actual needs, and the sum of λ1, λ2, λ3 and λ4 is 1.

[0100] S103, performing knowledge distillation on the initial student model based on feature information, output distribution and the loss function of the initial student model to obtain a target student model.

[0101] Among them, the similarity between the output distribution of the target student model and the output distribution of the teacher model is greater than the similarity threshold.

[0102] Optionally, the similarity threshold can be set according to actual needs, for example, the similarity threshold can be 99% or 99.5%. This application does not impose any specific restrictions on this.

[0103] It can be understood that when the similarity between the output distribution of the target student model and the output distribution of the teacher model is greater than the similarity threshold, it can be used to characterize that the target student model has successfully learned the key information or knowledge in the teacher model.

[0104] In one possible implementation, the knowledge distillation device can train the student model through the feature information and output distribution of the dataset and the teacher model. In each iteration, the knowledge distillation device can calculate the loss value based on the loss function of the initial student model. The knowledge distillation device can evaluate the performance of the student model through the loss value, and the training process ends when the performance reaches the expected level. Otherwise, the knowledge distillation device can adjust the structure of the student model, the weight of the distillation loss function, or other hyperparameters, and retrain.

[0105] In another possible implementation, the present application can deploy the target student model on the vehicle side. Due to the limited computing resources on the vehicle side, before deploying the target student model on the vehicle side, the knowledge distillation device can quantize the target student model, and reduce the weight accuracy of the model to further improve the real-time performance of the model and reduce the consumption of on-board computing resources.

[0106] Exemplarily, the knowledge distillation device can quantize the target student model by using the int8 quantization method. The int8 quantization method satisfies the following second formula:

[0107]

[0108] Among them, W new Can be used to characterize the quantized weight parameters. pre It can be used to characterize the weight parameter before quantization. α can be used to characterize the maximum real number in the current input data distribution. round() can be used to characterize the rounding function. clip() can be used to characterize the truncation function.

[0109] based on Figure 1 The technical solution in the present application, the knowledge distillation method provided by the present application can effectively transfer the knowledge contained in a large and high-precision model (i.e., the teacher model) to a small and low-complexity model (i.e., the student model) by using knowledge distillation technology to adapt to application environments with relatively limited resources, and divide the feature information into different dimensions to ensure that the student model does not miss any key feature information in the learning process, avoiding the problem of losing important features in related technologies. Therefore, the present application can effectively reduce the computational complexity and resource consumption of the student model while ensuring the performance of the student model.

[0110] In some embodiments, in order to determine the first loss function corresponding to the spatial feature information, the knowledge distillation method provided in the embodiment of the present application also includes the following steps: S201-S202.

[0111] S201. Determine the adaptive weight of the spatial feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model.

[0112] In one possible implementation, the knowledge distillation device may perform average pooling on the intermediate layer features of the teacher model when the channel dimensions of the intermediate layer features of the teacher model are aligned with those of the initial student model to obtain the first channel feature information of the teacher model, and perform average pooling on the intermediate layer features of the initial student model to obtain the second channel feature information of the initial student model. The knowledge distillation device may perform dot product calculation on the first channel feature information and the second channel feature information to obtain the adaptive weight of the spatial feature information.

[0113] For example, Figure 2 As shown, it is a schematic diagram of a loss function determination process corresponding to a spatial feature information provided in this application. In the adaptive weight generation part, the knowledge distillation device can use 1×1 convolution to align the intermediate layer feature X of the teacher model. t ∈R H×W×C and the intermediate layer features X of the initial student model s ∈R H×W×C The channel dimension of the teacher model is averaged and the intermediate layer features of the teacher model are averaged and pooled to obtain the first channel feature information Q of the teacher model.t ∈R 1×1×C , and average pooling is performed on the intermediate layer features of the initial student model to obtain the second channel feature information Q of the initial student model s ∈R 1×1×C The knowledge distillation device can perform a dot product calculation on the first channel feature information and the second channel feature information to obtain the adaptive weight of the spatial feature information. i,j , which is the channel dimension weight.

[0114] Among them, the first channel feature information and the second channel information satisfy the following third formula, the adaptive weight w of the spatial feature information i,j The following fourth formula and third formula are satisfied:

[0115] Q=F2σ(BN(F1(X))) The third formula

[0116] The fourth formula:

[0117]

[0118] Among them, F1() can be used to characterize 1×1 convolution. BN() can be used to characterize batch normalization. σ() can be used to characterize nonlinear activation function. F2() can be used to characterize global average pooling. Q s，i It can be used to characterize the channel feature information of the i-th intermediate layer features of the student model. t，j It can be used to represent the channel feature information of the jth intermediate layer feature of the teacher model. d can be used to represent the number of channels of the teacher model and the student model after 1×1 convolution transformation. softmax() can be used to represent the normalized exponential function. w i,j Adaptive weights that can be used to characterize spatial feature information.

[0119] S202: Determine a first loss function corresponding to the spatial feature information based on the adaptive weight of the spatial feature information.

[0120] In one possible implementation, the knowledge distillation device can determine the first spatial feature information of the teacher model in the first dimension and the second spatial feature information of the initial student model in the first dimension when the spatial dimensions between the teacher model and the initial student model are aligned. The knowledge distillation device can perform weighted processing on the first spatial feature information and the second spatial feature information based on the adaptive weight of the spatial feature information to obtain a first loss function corresponding to the spatial feature information. The first dimension is the dimension after the spatial dimensions between the teacher model and the initial student model are aligned.

[0121] Exemplary, combined Figure 2 , the knowledge distillation device can upsample or downsample the intermediate layer features of the teacher model and the intermediate layer features of the initial student model to align the spatial dimensions between the teacher model and the initial student model. The knowledge distillation device can perform channel-by-channel average pooling on the teacher model and the initial student model to obtain the first spatial feature information of the teacher model in the first dimension and the second spatial feature information of the initial student model in the first dimension, that is, determine the spatial dimension similarity between the intermediate layer features of the teacher model and the intermediate layer features of the initial student model. When the teacher model is an image processing model, the knowledge distillation device can also generate a foreground mask M∈R according to the true label. h×w , background mask, and generate corresponding weights according to the target proportion γ. The knowledge distillation device can perform weighted processing on the first spatial feature information and the second spatial feature information based on the adaptive weight of the spatial feature information to obtain a first loss function corresponding to the spatial feature information.

[0122] Alternatively, the knowledge distillation device can perform weighted processing on the first spatial feature information, the second spatial feature information, the foreground mask, the background mask and the weight based on the adaptive weight of the spatial feature information to obtain a first loss function corresponding to the spatial feature information.

[0123] The first spatial feature information and the second spatial dimension information satisfy the fifth formula, and the first loss function satisfies the sixth formula. The fifth formula:

[0124] S=F4σ(BN(F3(X))) Fifth formula

[0125] The sixth formula:

[0126]

[0127] Among them, F3() can be used to represent upsampling or downsampling. F3() can be used to represent channel dimension average pooling. s,i It can be used to characterize the spatial information of the i-th intermediate layer features of the student model. t,i It can be used to represent the spatial information of the jth intermediate layer features of the teacher model. I can be used to represent the unit matrix with the same dimension as M. BN() can be used to represent batch normalization. σ() can be used to represent the nonlinear activation function. L SAT It can be used to characterize the first loss function corresponding to the spatial feature information. i,j Adaptive weights that can be used to characterize spatial feature information.

[0128] Based on the above technical solution, the knowledge distillation method provided in this application can reflect the importance of spatial feature information in the model learning process through adaptive weights, ensure that the student model can focus on these key features during the training process, and by determining the first loss function, enable the student model to more accurately extract and retain spatial feature information similar to the teacher model, thereby improving the performance of the student model.

[0129] In some embodiments, in order to determine the second loss function corresponding to the channel feature information, the knowledge distillation method provided in the embodiment of the present application also includes the following steps: S301-S302.

[0130] S301. Determine adaptive weights of channel feature information based on intermediate layer features of the teacher model and intermediate layer features of the initial student model.

[0131] In one possible implementation, the knowledge distillation device may perform dimension swapping on the intermediate layer features of the teacher model to obtain the first intermediate layer features, and perform dimension swapping on the intermediate layer features of the initial student model to obtain the second intermediate layer features. The knowledge distillation device may determine the first height dimension information and the first width dimension information based on the first intermediate feature value, and determine the second height dimension information and the second width dimension information based on the second intermediate layer features. The knowledge distillation device may perform weighted processing on the first dot product and the second dot product to obtain an adaptive weight of the channel feature. The first dot product may be the dot product between the first height dimension information and the second height dimension information, and the second dot product may be the dot product between the first width dimension information and the second width dimension information.

[0132] For example, Figure 3 As shown, it is a schematic diagram of a loss function determination process corresponding to a channel feature information provided in this application. In the adaptive weight generation part, the knowledge distillation device can use 1×1 convolution to align the intermediate layer feature X of the teacher model. t ∈R H×W×C and the intermediate layer features X of the initial student model s ∈R H×W×C The channel dimension of the teacher model is averaged and pooled, and the intermediate layer features of the teacher model are dimensionally swapped to obtain the first intermediate layer features, and the intermediate layer features of the initial student model are dimensionally swapped to obtain the second intermediate layer features. The first intermediate layer features may include X th ∈R C×W×H and X tw ∈R H×C×W The first intermediate layer features may include X sh ∈R C ×W×H and X sw ∈RH×C×W .

[0133] In addition, the knowledge distillation device can determine the height dimension information and the width dimension information, that is, the knowledge distillation device can determine the first height dimension information Y based on the first intermediate eigenvalue t ∈R 1×1×d and the first width dimension information Z t ∈R 1×1×d The knowledge distillation device can determine the second high-dimensional information Y based on the second intermediate eigenvalue t ∈R 1×1×d and the second width dimension information Z t ∈R 1×1×d .

[0134] Furthermore, the knowledge distillation device can perform weighted processing on the first dot product and the second dot product to obtain the adaptive weight of the channel feature. The adaptive weight of the channel feature satisfies the following seventh formula:

[0135]

[0136] Among them, w' i,j Can be used to characterize the adaptive weight of the channel characteristics. s,i It can be used to represent the second height dimension information. Can be used to represent the transpose of the first height dimension information. s,i It can be used to represent the second width dimension information. Can be used to represent the transpose of the first height dimension information.

[0137] S302: Determine a second loss function corresponding to the channel feature information based on the adaptive weight of the channel feature information.

[0138] In one possible implementation, the knowledge distillation device may perform weighted processing on the first channel feature information and the second channel feature information based on the adaptive weight of the channel feature information to obtain a second loss function corresponding to the channel feature information.

[0139] For example, the knowledge distillation device can use 1×1 convolution to align the intermediate layer features X of the teacher model t ∈R H ×W×C and the intermediate layer features X of the initial student model s ∈R H×W×C The channel dimension is calculated, and a global average pooling process is performed to obtain the similarity of the channel dimension, that is, the similarity between the first channel feature information and the second channel feature information, and the first channel feature information and the second channel feature information are weighted to obtain a second loss function corresponding to the channel feature information.

[0140] Among them, the second loss function corresponding to the channel feature information satisfies the following eighth formula:

[0141]

[0142] Among them, L CAT It can be used to characterize the second loss function corresponding to the channel feature information. i,j It can be used to characterize the adaptive weight of the channel characteristics. s,i It can be used to characterize the channel feature information of the i-th intermediate layer features of the student model. t,j It can be used to characterize the channel feature information of the jth intermediate layer features of the teacher model.

[0143] Based on the above technical solution, the knowledge distillation method provided in this application can reflect the importance of channel feature information in the model learning process through adaptive weights, ensure that the student model can focus on these key features during the training process, and by determining the second loss function, enable the student model to more accurately extract and retain channel feature information similar to the teacher model, thereby improving the performance of the student model.

[0144] In some embodiments, in order to determine the third loss function corresponding to the output distribution, the knowledge distillation method provided in the embodiment of the present application also includes the following steps: S401.

[0145] S401. Determine a third loss function corresponding to the output distribution by taking the difference between the output distribution of the teacher model and the output distribution of the initial student model.

[0146] In one possible implementation, the loss function corresponding to the output distribution is determined to enable the student model to model the decision-making process of the teacher model. The knowledge distillation device can soften the softmax output to increase the amount of information output by the teacher model, and then use the information (Kullback-Leibler, KL) divergence to supervise the student model to learn the output response knowledge of the teacher model, thereby determining the third loss function corresponding to the output distribution. The third loss function corresponding to the output distribution satisfies the following ninth formula, the ninth formula:

[0147] L OT =D KL (S t (T)||S s (T)) Ninth Formula

[0148] Among them, L OT It can be used to characterize the third loss function corresponding to the output distribution. KL Can be used to characterize KL divergence. t (T) can be used to characterize the softened teacher model output. Ss (T) can be used to characterize the student model output after softening. T can be used to characterize the temperature parameter.

[0149] Based on the above technical solution, the knowledge distillation method provided in this application can directly reflect the gap in task performance between the teacher model and the initial student model by calculating the difference between the outputs, thereby determining the third loss function corresponding to the output distribution through the difference, which can significantly promote the robustness and generalization ability of knowledge distillation.

[0150] In some embodiments, Figure 4 As shown, it is a schematic diagram of a model compression process provided by this application.

[0151] In one possible implementation, the knowledge distillation device can train the teacher model according to the data set. The knowledge distillation device can continue to train the teacher model according to the data set when the training is not completed. The knowledge distillation device can freeze the parameters of the teacher model and load it into the knowledge distillation process when the training is completed. The knowledge distillation device can jointly train the initial student model according to the data set and the trained teacher model. The knowledge distillation device can continue to jointly train the initial student model according to the data set and the trained teacher model when the training is not completed. The knowledge distillation device can quantize the target student model and deploy it to the vehicle-side device when the training is completed.

[0152] In some embodiments, in combination Figure 4 ,like Figure 5 Shown is a schematic diagram of a knowledge distillation process provided in this application.

[0153] In one possible implementation, the knowledge distillation device can input data to determine the teacher model and the initial student model. The knowledge distillation device can adaptively transfer the multi-stage features of the teacher model to the stage features of the initial student model, that is, the knowledge distillation device can adaptively transfer the spatial feature information to the initial student model. The knowledge distillation device can adaptively transfer the channel feature information of the teacher model to the initial student model. The knowledge distillation device can transfer the output response of the teacher model to the initial student model. The knowledge distillation device can determine the true loss, that is, the loss function of the initial student model, based on the migrated initial student model and the true label.

[0154] Based on this, in the feature distillation process, the present application can effectively perform multi-stage knowledge distillation, so that the student model can effectively learn from the teacher model at different feature levels, thereby avoiding the problem of information loss caused by differences in feature levels and expression capabilities between the teacher network and the student network.

[0155] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, the knowledge distillation device or electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0156] The embodiments of the present application can exemplarily divide the functional modules of the knowledge distillation device or electronic device according to the above method. For example, the knowledge distillation device or electronic device may include various functional modules corresponding to the various functional divisions, or two or more functions may be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0157] Figure 6 is a block diagram of a knowledge distillation device according to an exemplary embodiment. Figure 6 , the knowledge distillation device includes: a determination unit 501 and a training unit 502.

[0158] In one possible manner, the determination unit 501 is used to determine the loss function, output distribution and feature information of the teacher model. The determination unit is also used to determine the loss function of the initial student model based on the loss function, output distribution and feature information of the teacher model.

[0159] In one possible approach, the training unit 502 is used to perform knowledge distillation on the initial student model based on feature information, output distribution, and the loss function of the initial student model to obtain a target student model.

[0160] In one possible manner, the determination unit 501 is specifically used to: determine a first loss function corresponding to the spatial feature information, a second loss function corresponding to the channel feature information, and a third loss function corresponding to the output distribution. The loss function of the teacher model, the first loss function corresponding to the spatial feature information, the second loss function corresponding to the first channel feature information, and the third loss function corresponding to the output distribution are weighted to obtain the loss function of the initial student model.

[0161] In one possible manner, the determination unit 501 is specifically configured to: determine the adaptive weight of the spatial feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model, and determine the first loss function corresponding to the spatial feature information based on the adaptive weight of the spatial feature information.

[0162] In one possible manner, the determination unit 501 is specifically used to: when the channel dimensions of the intermediate layer features of the teacher model and the intermediate layer features of the initial student model are aligned, average pooling is performed on the intermediate layer features of the teacher model to obtain the first channel feature information of the teacher model, and average pooling is performed on the intermediate layer features of the initial student model to obtain the second channel feature information of the initial student model. Dot product calculation is performed on the first channel feature information and the second channel feature information to obtain the adaptive weight of the spatial feature information.

[0163] In one possible manner, the determination unit 501 is specifically used to: determine the first spatial feature information of the teacher model in the first dimension and the second spatial feature information of the initial student model in the first dimension when the spatial dimensions between the teacher model and the initial student model are aligned. The first dimension is the dimension after the spatial dimensions between the teacher model and the initial student model are aligned. Based on the adaptive weight of the spatial feature information, the first spatial feature information and the second spatial feature information are weighted to obtain a first loss function corresponding to the spatial feature information.

[0164] In one possible manner, the determination unit 501 is specifically configured to: determine the adaptive weight of the channel feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model, and determine the second loss function corresponding to the channel feature information based on the adaptive weight of the channel feature information.

[0165] In one possible manner, the determination unit 501 is specifically used to: perform dimension swapping on the intermediate layer features of the teacher model to obtain the first intermediate layer features, and perform dimension swapping on the intermediate layer features of the initial student model to obtain the second intermediate layer features. Based on the first intermediate feature value, the first height dimension information and the first width dimension information are determined, and based on the second intermediate layer features, the second height dimension information and the second width dimension information are determined. The first dot product and the second dot product are weighted to obtain the adaptive weight of the channel feature. The first dot product is the dot product between the first height dimension information and the second height dimension information. The second dot product is the dot product between the first width dimension information and the second width dimension information.

[0166] In a possible manner, the determination unit 501 is specifically configured to: perform weighted processing on the first channel feature information and the second channel feature information based on the adaptive weight of the channel feature information to obtain a second loss function corresponding to the channel feature information.

[0167] In one possible manner, the determination unit 501 is specifically used to: determine a third loss function corresponding to the output distribution based on the difference between the output distribution of the teacher model and the output distribution of the initial student model.

[0168] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0169] Figure 7 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 7 As shown, the electronic device includes but is not limited to: a processor 601 and a memory 602 .

[0170] The memory 602 is used to store executable instructions of the processor 601. It can be understood that the processor 601 is configured to execute instructions to implement the knowledge distillation method in the above embodiment.

[0171] It should be noted that those skilled in the art can understand that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include Figure 7 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0172] The processor 601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 601 may include one or more processing units. Optionally, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0173] The memory 602 may be used to store software programs and various data. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one functional module (such as a determination unit, a processing unit, etc.), etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0174] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, and the above instructions can be executed by a processor 601 of an electronic device to implement the method in the above embodiment.

[0175] In actual implementation, Figure 6 The functions of the determination unit 501 and the training unit 502 in Figure 7 The processor 601 in the embodiment calls the computer program stored in the memory 602. The specific execution process can refer to the description of the method part in the above embodiment, which will not be repeated here.

[0176] Optionally, the computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0177] In an exemplary embodiment, the present application also provides a computer program product including one or more instructions, and the one or more instructions can be executed by the processor 601 of the electronic device to complete the method in the above embodiment.

[0178] It should be noted that when the instructions in the above-mentioned computer-readable storage medium or one or more instructions in the computer program product are executed by the processor of the electronic device, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect as the above-mentioned method can be achieved. To avoid repetition, they will not be repeated here.

[0179] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the full classification part or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute the full classification part or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or CD and other media that can store program code.

[0184] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A knowledge distillation method, characterized in that: include: Determine the loss function, output distribution, and feature information of the teacher model; Based on the loss function of the teacher model, the output distribution and the feature information, determining the loss function of an initial student model; the initial student model has some functions of the teacher model; Based on the feature information, the output distribution and the loss function of the initial student model, knowledge distillation is performed on the initial student model to obtain a target student model; the similarity between the output distribution of the target student model and the output distribution of the teacher model is greater than a similarity threshold.

2. The method according to claim 1, characterized in that The feature information includes spatial feature information and channel feature information; the loss function of the initial student model is determined based on the teacher model output, the feature information, and the loss function of the teacher model, including: Determine a first loss function corresponding to the spatial feature information, a second loss function corresponding to the channel feature information, and a third loss function corresponding to the output distribution; The loss function of the teacher model, the first loss function corresponding to the spatial feature information, the second loss function corresponding to the first channel feature information, and the third loss function corresponding to the output distribution are weighted to obtain the loss function of the initial student model.

3. The method according to claim 2, characterized in that Determining a first loss function corresponding to the spatial feature information includes: Determining an adaptive weight of the spatial feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model; Based on the adaptive weight of the spatial feature information, a first loss function corresponding to the spatial feature information is determined.

4. The method according to claim 3, characterized in that The step of determining the adaptive weight of the spatial feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model comprises: In the case where the channel dimensions of the intermediate layer features of the teacher model are aligned with the intermediate layer features of the initial student model, average pooling is performed on the intermediate layer features of the teacher model to obtain first channel feature information of the teacher model, and average pooling is performed on the intermediate layer features of the initial student model to obtain second channel feature information of the initial student model; A dot product calculation is performed on the first channel feature information and the second channel feature information to obtain an adaptive weight of the spatial feature information.

5. The method according to claim 3, characterized in that: The determining, based on the adaptive weight of the spatial feature information, a first loss function corresponding to the spatial feature information comprises: In the case where the spatial dimensions between the teacher model and the initial student model are aligned, determining first spatial feature information of the teacher model in a first dimension and second spatial feature information of the initial student model in the first dimension; the first dimension is a dimension after the spatial dimensions between the teacher model and the initial student model are aligned; Based on the adaptive weight of the spatial feature information, the first spatial feature information and the second spatial feature information are weighted to obtain a first loss function corresponding to the spatial feature information.

6. The method according to any one of claims 2 to 5, characterized in that: Determining a second loss function corresponding to the channel feature information includes: Determining an adaptive weight of the channel feature information based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model; Based on the adaptive weight of the channel feature information, a second loss function corresponding to the channel feature information is determined.

7. The method according to claim 6, characterized in that The step of determining the adaptive weight of the channel feature based on the intermediate layer features of the teacher model and the intermediate layer features of the initial student model comprises: Performing dimension swapping on the intermediate layer features of the teacher model to obtain first intermediate layer features, and performing dimension swapping on the intermediate layer features of the initial student model to obtain second intermediate layer features; Determine first height dimension information and first width dimension information based on the first intermediate feature value, and determine second height dimension information and second width dimension information based on the second intermediate layer feature; The first dot product and the second dot product are weighted to obtain an adaptive weight of the channel feature; the first dot product is the dot product between the first height dimension information and the second height dimension information; the second dot product is the dot product between the first width dimension information and the second width dimension information.

8. The method according to claim 6, characterized in that The determining, based on the adaptive weight of the channel feature, a second loss function corresponding to the channel feature information comprises: Based on the adaptive weight of the channel feature information, weighted processing is performed on the first channel feature information and the second channel feature information to obtain a second loss function corresponding to the channel feature information.

9. The method according to any one of claims 2 to 5, characterized in that: Determining a third loss function corresponding to the output distribution includes: The difference between the output distribution of the teacher model and the output distribution of the initial student model is used to determine a third loss function corresponding to the output distribution.

10. A knowledge distillation device, characterized in that: The device comprises: a determination unit, a training unit; The determination unit is used to determine the loss function, output distribution and feature information of the teacher model; The determining unit is further used to determine the loss function of the initial student model based on the loss function of the teacher model, the output distribution and the feature information; the initial student model has some functions of the teacher model; The training unit is used to perform knowledge distillation on the initial student model based on the feature information, the output distribution and the loss function of the initial student model to obtain a target student model; the similarity between the output distribution of the target student model and the output distribution of the teacher model is greater than a similarity threshold.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: When the computer-executable instructions stored in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is capable of performing the method as claimed in any one of claims 1 to 9.

13. A computer program product comprising instructions, characterized in that When the instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 9.

Images