Model lightweight application deployment method based on knowledge distillation and related device
By improving the knowledge distillation algorithm, using adversarial game and multi-angle rotation image self-supervision technology to optimize the characteristics of the student model, the problem of insufficient knowledge transfer between teachers and students is solved, and efficient deep learning model deployment on edge devices is achieved.
Patent Information
- Application Number
- CN202411202638.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
AI Technical Summary
The existing knowledge distillation algorithm ignores the intermediate layer features when passing on teacher model knowledge, resulting in poor performance of student models and it is difficult to effectively deploy deep learning models on resource-constrained edge devices.
Using an improved knowledge distillation algorithm, the teacher network and student network interchange roles through adversarial game, generate an adversarial teacher model, and use it to guide students' network training, combining multi-angle rotation image self-supervision and cross-entropy loss function to optimize the multi-pupilization of student network Scaling features improve the recognition performance of student models.
It effectively improves the recognition performance of student models and generates a lightweight model with fast training speed, fewer parameters and no loss of accuracy, which is suitable for edge device deployment.
Smart Images

Figure CN119067161A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of model deployment, relates to the field of power systems, and particularly to a model lightweight application deployment method based on knowledge distillation and related devices. Background Art
[0002] In the current power system, the application of artificial intelligence has been widely penetrated into the cloud, covering multiple core areas such as dispatching, operation and inspection, and marketing, involving advanced technologies such as intelligent voice processing, image recognition, video analysis, and knowledge graph construction, which has greatly improved the intelligence level of the power system; however, as the power system enters the digital era of the Internet of Everything, the number of various devices and software has shown an unprecedented rapid growth trend. This trend, while promoting the digital transformation of the power industry, has also put forward higher requirements for the edge deployment of power models. Explanatory, compared with the cloud, edge devices have higher requirements for the adaptability and efficiency of model operation due to their large number, wide distribution, relatively weak computing power, and low energy consumption.
[0003] In the application of power systems, deep learning models have shown many advantages such as high accuracy, strong adaptability, and strong portability. However, due to the huge amount of computation and complex network structure of such models, it is difficult to run directly on resource-constrained and low-performance edge devices. Example explanatory, in the important field of power transmission and transformation inspection, the application of deep learning models is particularly prominent; in order to adapt to the actual situation of a wide variety of inspection targets and complex inspection scenarios, deep learning inspection models usually need to be trained with deep networks and large parameters, which also leads to large model size, high hardware resource usage, slow inference speed, high power consumption, over-reliance on backend servers, and insufficient response time, which makes it difficult to meet the needs of mobile devices and embedded devices.
[0004] In order to ensure the application and deployment of deep learning models on power edge devices, compressing network models has become an important technical means. Explanatory, the ultimate goal of compressing network models is to produce small-scale, high-precision and accurate models; model compression specifically refers to the method of streamlining the model while ensuring the accuracy of the model. After model compression, the number of network parameters will be smaller, and the amount of calculation will usually be reduced, which greatly reduces the computing and storage overheads, making it possible to deploy models under limited hardware conditions. The existing model lightweight methods are roughly the following: network pruning, parameter sharing, quantization, knowledge distillation, etc.
[0005] The knowledge distillation method can transfer the knowledge information of the excellent teacher model to the student model. After learning the knowledge of the teacher model, the performance of the student model can reach the level of the teacher model; therefore, knowledge distillation has been widely used in model lightweighting. At present, in the process of knowledge distillation, the parameter scale between the teacher model and the student model is quite different, which leads to the inability to effectively transfer the knowledge information between the two, and the final student model has poor performance. Explanatory examples show that the existing knowledge distillation algorithm is usually a knowledge distillation method based on logits. The core of this method is to achieve effective knowledge transfer by minimizing the KL divergence (Kullback-LeiblerDivergence) between the predicted logits of the teacher model and the student model. This method has shown remarkable results in promoting knowledge transfer, but it does not fully tap the full potential of the teacher model, especially ignoring the rich teaching information contained in the intermediate layer features. Therefore, the student model cannot obtain more knowledge information of the teacher model, which leads to poor performance of the student model. Summary of the invention
[0006] The purpose of the present invention is to provide a method and related device for lightweight model application deployment based on knowledge distillation to solve one or more of the above-mentioned technical problems. The technical solution provided by the present invention can realize lightweight model application deployment and effectively improve the recognition accuracy of lightweight models, and has good application prospects and high application value.
[0007] In order to achieve the above object, the present invention adopts the following technical solutions:
[0008] In a first aspect, the present invention provides a method for deploying a model lightweight application based on knowledge distillation, comprising the following steps:
[0009] Get the student network and the trained teacher network;
[0010] Based on the trained teacher network, the improved knowledge distillation algorithm is used to train and update the student network and reach the preset convergence conditions to obtain the trained target student network;
[0011] The step of improving the knowledge distillation algorithm includes:
[0012] Based on the adversarial game, the teacher network and the student network exchange roles; the student network is used as the teacher model, and the teacher network is used as the student model;
[0013] An adversarial teacher model is generated through the KL divergence between the teacher model and the student model, and the adversarial teacher model is used as the guiding teacher model of the student network to train the student network.
[0014] A further improvement of the present invention is that in the step of generating an adversarial teacher model by using the KL divergence between the teacher model and the student model,
[0015] The training process of the adversarial teacher model is expressed as:
[0016] F * =θ(F s core );
[0017]
[0018] In the formula, F * represents the probability distribution of the initial non-rotated image; F s core represents the probability distribution of logits output by the student backbone network; θ represents a quarter-slice operation; N represents a data sample set; i represents the i-th image sample; Represents the original label corresponding to the slice of the i-th image sample; represents the probability distribution of the i-th image sample initially without rotation; β CE represents the cross entropy loss; β KL represents KL divergence; Represent the softened outputs of the backbone networks of the student model and the teacher model respectively.
[0019] A further improvement of the present invention is that, in the data sample set, each original sample image is rotated around the center point by 0 degrees, 90 degrees, 180 degrees and 270 degrees, so that each original sample image is expanded into four versions with different rotation angles for subsequent training.
[0020] A further improvement of the present invention is that the step of training the student network using the adversarial teacher model as the guiding teacher model of the student network comprises:
[0021] Freeze the parameters of the adversarial teacher model and use KL divergence calculation and The improved similarity between MKL ; In the process of optimizing the student network, the branch auxiliary network in the student network deepens the multi-scale features output by the backbone network and generates the probability distribution of image logits under multi-angle supervision
[0022] The true labels c of images supervised by multiple angles are used i , using cross loss To assist the optimization direction of the network;
[0023] By calculating the probability distribution and the original label The cross entropy loss To train student networks;
[0024] in,
[0025]
[0026] A further improvement of the present invention is that the total loss function β of the improved knowledge distillation algorithm is total It is expressed as:
[0027]
[0028] In the formula, The loss function for the adversarial teacher model stage is generated by the KL divergence between the teacher model and the student model; The loss function for the training phase of the student network using the adversarial teacher model as its guiding teacher model;
[0029]
[0030] In the formula, α and ω are weight coefficients.
[0031] A further improvement of the present invention is that
[0032] The teacher network is a deep convolutional neural network for detecting one or more of four power scenarios: silicone discoloration, insulator damage, abnormal door closure, and climbing over fences;
[0033] The student network is a lightweight convolutional neural network.
[0034] A further improvement of the present invention is that after obtaining the trained target student network, the method further comprises:
[0035] Combining the horizontal and vertical characteristics of Docker containers in the form of hybrid elasticity, the adaptive elastic encapsulation of the target student network is achieved by using dynamic scaling decisions of containers.
[0036] A second aspect of the present invention provides a model lightweight application deployment system based on knowledge distillation, comprising:
[0037] Model acquisition module, used to obtain the student network and the trained teacher network;
[0038] The training update module is used to train and update the student network based on the trained teacher network using an improved knowledge distillation algorithm and achieve the preset convergence conditions to obtain a trained target student network;
[0039] The step of improving the knowledge distillation algorithm includes:
[0040] Based on the adversarial game, the teacher network and the student network exchange roles; the student network is used as the teacher model, and the teacher network is used as the student model;
[0041] An adversarial teacher model is generated through the KL divergence between the teacher model and the student model, and the adversarial teacher model is used as the guiding teacher model of the student network to train the student network.
[0042] A further improvement of the present invention is that in the step of generating an adversarial teacher model by using the KL divergence between the teacher model and the student model,
[0043] The training process of the adversarial teacher model is expressed as:
[0044] F * =υ(F s core );
[0045]
[0046] In the formula, F * represents the probability distribution of the initial non-rotated image; F s core It represents the probability distribution of logits output by the student backbone network; υ represents the quarter-slice operation; N represents the data sample set; i represents the i-th image sample; Represents the original label corresponding to the slice of the i-th image sample; represents the probability distribution of the i-th image sample initially without rotation; β CE represents the cross entropy loss; β KL represents KL divergence; Represent the softened outputs of the backbone networks of the student model and the teacher model respectively.
[0047] A further improvement of the present invention is that, in the data sample set, each original sample image is rotated around the center point by 0 degrees, 90 degrees, 180 degrees and 270 degrees, so that each original sample image is expanded into four versions with different rotation angles for subsequent training.
[0048] A further improvement of the present invention is that the step of training the student network using the adversarial teacher model as the guiding teacher model of the student network comprises:
[0049] Freeze the parameters of the adversarial teacher model and use KL divergence calculation and The improved similarity between MKL ; In the process of optimizing the student network, the branch auxiliary network in the student network deepens the multi-scale features output by the backbone network and generates the probability distribution of image logits under multi-angle supervision
[0050] The true labels c of images supervised by multiple angles are used i , using cross loss To assist the optimization direction of the network;
[0051] By calculating the probability distribution and the original label The cross entropy loss To train student networks;
[0052] in,
[0053]
[0054] A further improvement of the present invention is that the total loss function β of the improved knowledge distillation algorithm is total It is expressed as:
[0055]
[0056] In the formula, The loss function for the adversarial teacher model stage is generated by the KL divergence between the teacher model and the student model; The loss function for the training phase of the student network using the adversarial teacher model as its guiding teacher model;
[0057]
[0058] In the formula, α and ω are weight coefficients.
[0059] A further improvement of the present invention is that
[0060] The teacher network is a deep convolutional neural network for detecting one or more of four power scenarios: silicone discoloration, insulator damage, abnormal door closure, and climbing over fences;
[0061] The student network is a lightweight convolutional neural network.
[0062] A further improvement of the present invention is that it also includes:
[0063] The encapsulation module is used to combine the horizontal and vertical characteristics of the Docker container in a hybrid elastic form after obtaining the trained target student network, and adopt the dynamic expansion and contraction decision of the container to achieve adaptive elastic encapsulation of the target student network.
[0064] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model lightweight application deployment method as described in any one of the first aspects of the present invention is implemented.
[0065] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model lightweight application deployment method as described in any one of the first aspects of the present invention is implemented.
[0066] Compared with the prior art, the present invention has the following beneficial effects:
[0067] In the model lightweight application deployment method based on knowledge distillation provided by the present invention, model lightweighting is performed based on an improved knowledge distillation algorithm, which enables the student model to absorb more knowledge information of the teacher model, and ultimately improves the recognition performance of the target student model, and has good application prospects and high application value. Further explanatory, the improved knowledge distillation algorithm of the present invention adopts a binary game idea based on adversarial learning to perform two-stage adversarial training. The first stage of adversarial training is to convert the roles of the teacher network and the student network so that the generated adversarial teacher model obtains more knowledge information of the student network; the second stage is to use the generated adversarial teacher model to guide the initial student network, and finally construct a target student network with fast training speed, fewer parameters and no loss of training accuracy. The improved knowledge distillation algorithm of the present invention generates an adversarial teacher model through two-stage adversarial training, which solves the problem caused by the large difference in model scale between the teacher and the student, is conducive to the teacher network to transfer knowledge information to the student network, can improve the training accuracy of the student network, and can effectively improve the recognition accuracy of the lightweight model. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0069] Figure 1 It is a flow chart of a method for deploying a model lightweight application based on knowledge distillation in an embodiment of the present invention.
[0070] Figure 2 It is a schematic diagram of the principle of improving the knowledge distillation algorithm in an embodiment of the present invention.
[0071] Figure 3 It is a schematic diagram of dynamic scaling decision based on containers in an embodiment of the present invention.
[0072] Figure 4 It is a schematic diagram of a model lightweight application deployment system based on knowledge distillation in an embodiment of the present invention. DETAILED DESCRIPTION
[0073] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.
[0074] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0075] See also Figure 1 In an embodiment of the present invention, a method for deploying a model lightweight application based on knowledge distillation is provided, comprising the following steps:
[0076] Step 1, obtain the student network and the trained teacher network;
[0077] Step 2: Based on the trained teacher network, the improved knowledge distillation algorithm is used to train and update the student network and reach the preset convergence condition to obtain the trained target student network;
[0078] The steps of improving the knowledge distillation algorithm specifically include:
[0079] Based on the adversarial game, the teacher network and the student network exchange roles; the student network is used as the teacher model, and the teacher network is used as the student model;
[0080] An adversarial teacher model is generated through the KL divergence between the teacher model and the student model, and the adversarial teacher model is used as the guiding teacher model of the student network to train the student network.
[0081] In the improved knowledge distillation algorithm provided by the embodiment of the present invention, the teacher network and the student network are first exchanged based on the adversarial game and an adversarial teacher model is generated, and finally the adversarial teacher model is used as the guidance teacher model of the student network; wherein, the adversarial teacher model is generated through the first stage adversarial training, and the adversarial teacher model contains more knowledge information of the initial teacher and student models; the second stage adversarial training uses the adversarial teacher model to guide the training of the student network. The use of the adversarial teacher model shortens the relationship between the student network and the teacher network, which is more conducive to the teacher network to transfer knowledge information to the student network, and can effectively improve the detection performance of the student network.
[0082] See also Figure 2 In one embodiment of the present invention, the step of exchanging the roles of the teacher network and the student network based on the adversarial game can be considered as the first stage of adversarial training. The process of exchanging the roles of the teacher network and the student network in this stage can be specifically as follows: the student network with fewer parameters and simpler structure is used as the teacher model F t (·), the teacher network with many parameters and complex structure is used as the student model F s (·), taking the pre-acquired sample data set as input, the teacher model and the student model are used to extract features respectively, and the obtained probability distribution is as follows:
[0083]
[0084] In the formula, F t core and F t aux They represent the probability distribution of the teacher backbone and auxiliary network output logits respectively; F s core and F s aux They represent the probability distribution of logits output by the student backbone and auxiliary networks respectively;
[0085] In the technical solution of the embodiment of the present invention, an adversarial teacher model is generated by the KL divergence between the teacher model and the student model, so that the adversarial teacher model can learn more knowledge of the student model;
[0086] Among them, F s core Take a quarter slice to get the probability distribution F of the initial non-rotated image * , the training process of the adversarial teacher model can be expressed as:
[0087] F * =θ(F s core );
[0088]
[0089] Where N represents the data sample set; θ represents the quarter-slice operation; represents the true label corresponding to the slice, i represents the i-th image sample; F * and Minimize the cross entropy loss between to ensure the accuracy of the student model; and They represent the softened outputs of the backbone networks of the student and teacher models respectively, and minimize the KL divergence between the two to ensure the knowledge transfer between the student and teacher models.
[0090] In the technical solution of the embodiment of the present invention, after the first stage of adversarial training, the adversarial teacher model contains a large amount of knowledge of the target student network; therefore, the adversarial teacher model is used in the second stage to train the student network to obtain the final target student network; the training process of the adversarial generative network in the second stage can be expressed as:
[0091]
[0092] The specific explanation is: First, freeze the parameters of the adversarial teacher model and use KL divergence to calculate and The improved similarity between MKL At the same time, in the process of optimizing the target student network architecture, the branch auxiliary network in the target student network deepens the multi-scale features output by the backbone network, further extracts details, and generates the probability distribution of image logits under multi-angle supervision. The true labels c of the images supervised by multiple angles i , and adopt cross loss To determine the optimization direction of the auxiliary network; finally, by calculating the probability distribution F * and the original label y * The cross entropy loss To further train the student network.
[0093] In one embodiment of the present invention, in the first stage of adversarial training, the core goal is to obtain an adversarial teacher model tailored for teaching the student network, which has rich and convenient knowledge for the target student network to learn. Therefore, the loss function of the first stage is for:
[0094]
[0095] The purpose of the second stage of training is to guide the target student network to learn by adversarial teacher model, and finally build a lightweight compressed student model with fast computation speed and no loss of accuracy; therefore, the loss function of the second stage is for:
[0096]
[0097] In the formula, α and ω represent weight coefficients, which are used to weigh the impact of each department in the model.
[0098] Combining the two-stage losses, the loss function of the improved knowledge organization algorithm is optimized, and the total loss function β total for:
[0099]
[0100] In a preferred embodiment of the present invention, the classical self-supervised learning strategy often focuses on designing auxiliary tasks, aiming to explore and utilize the inherent representation characteristics of the data itself as an implicit supervisory force. Such strategies include, for example, self-supervised learning methods. Specifically, the image is rotated around its center point accurately to an integer multiple of 90 degrees, including four basic angles of 0, 90, 180 and 270 degrees. After the data set has undergone standard data preprocessing and enhancement procedures, this rotation transformation technique is used to expand each original image into four versions with different rotation angles as training inputs. This greatly enriches the data diversity and enables the neural network to capture richer and more effective feature information.
[0101] As an example, suppose a training model is F(·), and the image obtained by rotating the initial data clockwise by 90 degrees is The probability distribution obtained after feature extraction by model F(·) is:
[0102]
[0103] Where: F core represents the probability distribution of the output logits of the backbone network; F aru Represents the probability distribution of the auxiliary network output logits; It represents the operation of applying the geometric transformation containing label y to the data x to obtain the data after geometric transformation; K represents the number of discrete geometric variables.
[0104] In a specific embodiment of the present invention, for four types of power scenarios, namely, silicone discoloration, insulator damage, abnormal door closure, and fence crossing detection, the corresponding models of each scenario are used as the teacher network (deep convolutional neural network, such as ResNet, VGG, DenseNet, etc.) in this knowledge distillation technology system, and the small model previously obtained through neural architecture search is used as the student network (lightweight convolutional neural network, such as MobileNet, ShuffleNet, SqueezeNet, etc.) in the technology system. Through knowledge distillation, the knowledge of the large model in each scenario is transferred to the new small model to achieve the purpose of lightweight model, successfully build a lightweight model that meets the hardware resource limitations of edge devices, and while compressing the model, minimize the accuracy loss after model training and reasoning, and ensure the smooth completion of various power inspection tasks in actual scenarios. The main idea of knowledge distillation is to use a pre-trained large network or integrated network to guide the training of a small network model. This training mode is also called teacher-student relationship, where the large network is the teacher and the small network is the student. In knowledge distillation, the teacher model minimizes the loss function of the prediction result in the process of training the student model, thereby transferring knowledge to the student model. In the classification problem, the last layer is the output of the Softmax function, which represents the predicted probability distribution of the teacher model. Since the parameter scales between the teacher and student models of the traditional knowledge distillation algorithm are very different, it is difficult to transfer knowledge between the models, resulting in poor recognition performance of the student model. Therefore, three strategies, including multi-angle rotation image self-supervision, multi-stage adversarial training and loss function optimization, are used to improve the traditional knowledge distillation algorithm.
[0105] To be more specific and exemplary, according to the flowchart of the algorithm, taking the silicone discoloration scene as an example, first, the training data set of the corresponding scene must be prepared, and then the images must be preprocessed so that the format size of the input images is 224×224. The input images are then rotated at multiple angles, and the model is self-supervised by the obtained multi-angle rotated images. Secondly, based on the idea of adversarial game, the teacher network and the student network are swapped, and through multi-stage adversarial training, the student network is enabled to better learn and absorb the knowledge information in the teacher network; finally, the losses of the two stages are superimposed, and after multiple iterative training, the training is terminated when the adversarial training model converges, and a trained student network is obtained.
[0106] In the technical solution of the embodiment of the present invention, the Docker containerization technology is used to realize the elastic allocation of edge device resources. The horizontal and vertical elasticity characteristics of the Docker container are combined. According to the heterogeneous resource configuration of the edge device and the workload requirements of the power inspection task, the number of copies of the lightweight model container is horizontally expanded and shrunk, and the computing resources of the container packaging unit are vertically expanded and shrunk to meet the requirements of the adaptive model packaging deployment. According to the model packaging requirements of power inspection tasks such as silicone discoloration and insulator damage, a copy of the lightweight model container is added or deleted in the horizontal elasticity. In the vertical elasticity, the computing resources (such as CPU, memory, etc.) in the container can be increased or decreased.
[0107] Specifically, a feasible implementation plan is to combine the horizontal and vertical characteristics of Docker containers in the form of hybrid elasticity, directly design dynamic container scaling decisions to achieve adaptive elastic packaging of the model, and thereby achieve the purpose of automatically evolving the lightweight model container scaling solution deployed on the device with the device resources and inspection task workload requirements.
[0108] like Figure 3 As shown in the figure, for the detection tasks of silicone discoloration, insulator damage, abnormal door closure and crossing fence in actual power scenarios, according to the model reasoning task latency and accuracy requirements, as well as the memory and computing power resources of edge power equipment such as control balls and drones, the dynamic expansion and contraction decision of containers is designed based on Docker container technology. The number of container copies and the allocation of resources in the container on the power edge devices are dynamically decided from the horizontal and vertical perspectives, so as to complete the packaging and deployment of inspection model containers for power scenarios.
[0109] In the present invention, for the models of silicone discoloration, insulator damage, abnormal door closure, and fence crossing detection tasks in power scenarios, the power inspection model application and its operating environment are packaged into a standard packaging unit based on Docker's containerized packaging technology, providing a unified functional function interface to development users, thereby realizing standardized packaging of power models.
[0110] In summary, the embodiment of the present invention proposes a model lightweight application and deployment method based on knowledge distillation; wherein, due to the characteristics of large number, wide distribution, weak computing power, and low energy consumption of edge devices, the deep learning network cannot be directly run in resource-constrained and weak performance edge devices; the technical solution of the embodiment of the present invention uses the knowledge distillation technology to guide and train a more streamlined and efficient small network model by using the powerful capabilities of pre-trained large networks or integrated networks, so as to achieve the purpose of model lightweight, so that it can be run on resource-constrained and weak performance edge devices, and finally uses Docker's container packaging technology to design API packaging rules and processes for silicone discoloration, insulator damage, abnormal door closure, and climbing over fences to detect power scenarios, and package the power inspection model application and its operating environment into a standard packaging unit, and provide a unified function function interface, so that developers can easily integrate it into the existing power inspection system. Through standardized packaging, the present invention brings higher efficiency and maintainability to the field of power inspection, while also reducing development costs and technical barriers.
[0111] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.
[0112] See also Figure 4 In an embodiment of the present invention, a model lightweight application deployment system based on knowledge distillation is provided, including:
[0113] Model acquisition module, used to obtain the student network and the trained teacher network;
[0114] The training update module is used to train and update the student network based on the trained teacher network using an improved knowledge distillation algorithm and achieve the preset convergence conditions to obtain a trained target student network;
[0115] The step of improving the knowledge distillation algorithm includes:
[0116] Based on the adversarial game, the teacher network and the student network exchange roles; the student network is used as the teacher model, and the teacher network is used as the student model;
[0117] An adversarial teacher model is generated through the KL divergence between the teacher model and the student model, and the adversarial teacher model is used as the guiding teacher model of the student network to train the student network.
[0118] In one embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the model lightweight application deployment method.
[0119] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM (Random Access Memory) memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the model lightweight application deployment method in the above embodiment.
[0120] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) that contain computer-usable program code.
[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A model lightweight application deployment method based on knowledge distillation, It is characterized in that The following steps are involved: Get the student network and the trained teacher network; Based on the trained teacher network, the improved knowledge distillation algorithm is used to train and update the student network and reach the preset convergence conditions to obtain the trained target student network; The step of improving the knowledge distillation algorithm includes: Based on the adversarial game, the teacher network and the student network exchange roles; the student network is used as the teacher model, and the teacher network is used as the student model; An adversarial teacher model is generated through the KL divergence between the teacher model and the student model, and the adversarial teacher model is used as the guiding teacher model of the student network to train the student network.
2. According to the method for deploying lightweight model applications based on knowledge distillation according to claim 1, It is characterized in that In the step of generating an adversarial teacher model by the KL divergence between the teacher model and the student model, The training process of the adversarial teacher model is expressed as: F * =υ(F s core ); In the formula, F * represents the probability distribution of the initial non-rotated image; F s core It represents the probability distribution of logits output by the student backbone network; υ represents the quarter-slice operation; N represents the data sample set; i represents the i-th image sample; Represents the original label corresponding to the slice of the i-th image sample; represents the probability distribution of the i-th image sample initially without rotation; β CE represents the cross entropy loss; β KL represents KL divergence; Represent the softened outputs of the backbone networks of the student model and the teacher model respectively.
3. According to the method for deploying lightweight model applications based on knowledge distillation according to claim 2, It is characterized in that In the data sample set, each original sample image is rotated around a center point by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, so that each original sample image is expanded into four versions with different rotation angles for subsequent training.
4. According to the method for deploying lightweight model applications based on knowledge distillation according to claim 2, It is characterized in that The step of training the student network using the adversarial teacher model as the guiding teacher model of the student network comprises: Freeze the parameters of the adversarial teacher model and use KL divergence calculation and The improved similarity between MLL ; In the process of optimizing the student network, the branch auxiliary network in the student network deepens the multi-scale features output by the backbone network and generates the probability distribution of image logits under multi-angle supervision The true labels c of images supervised by multiple angles are used i , using cross loss To assist the optimization direction of the network; By calculating the probability distribution and the original label The cross entropy loss To train student networks; in, 5. According to the method for deploying lightweight model applications based on knowledge distillation according to claim 4, It is characterized in that The total loss function β of the improved knowledge distillation algorithm total It is expressed as: In the formula, The loss function for the adversarial teacher model stage is generated by the KL divergence between the teacher model and the student model; The loss function for the training phase of the student network using the adversarial teacher model as its guiding teacher model; In the formula, α and ω are weight coefficients.
6. According to the method for deploying lightweight model applications based on knowledge distillation in claim 1, It is characterized in that The teacher network is a deep convolutional neural network for detecting one or more of four power scenarios: silicone discoloration, insulator damage, abnormal door closure, and climbing over fences; The student network is a lightweight convolutional neural network.
7. According to claim 6, a method for deploying lightweight applications based on knowledge distillation, It is characterized in that After obtaining the trained target student network, it also includes: Combining the horizontal and vertical characteristics of Docker containers in the form of hybrid elasticity, the adaptive elastic encapsulation of the target student network is achieved by using dynamic scaling decisions of containers.
8. A model lightweight application deployment system based on knowledge distillation, It is characterized in that include: Model acquisition module, used to obtain the student network and the trained teacher network; The training update module is used to train and update the student network based on the trained teacher network using an improved knowledge distillation algorithm and achieve the preset convergence conditions to obtain a trained target student network; The step of improving the knowledge distillation algorithm includes: Based on the adversarial game, the teacher network and the student network exchange roles; the student network is used as the teacher model, and the teacher network is used as the student model; An adversarial teacher model is generated through the KL divergence between the teacher model and the student model, and the adversarial teacher model is used as the guiding teacher model of the student network to train the student network.
9. A model lightweight application deployment system based on knowledge distillation according to claim 8, It is characterized in that In the step of generating an adversarial teacher model by the KL divergence between the teacher model and the student model, The training process of the adversarial teacher model is expressed as: F * =θ(F s core ); In the formula, F * represents the probability distribution of the initial non-rotated image; F s core represents the probability distribution of logits output by the student backbone network; θ represents a quarter-slice operation; N represents a data sample set; i represents the i-th image sample; Represents the original label corresponding to the slice of the i-th image sample; represents the probability distribution of the i-th image sample initially without rotation; β CE represents the cross entropy loss; β KL represents KL divergence; Represent the softened outputs of the backbone networks of the student model and the teacher model respectively.
10. A model lightweight application deployment system based on knowledge distillation according to claim 9, It is characterized in that In the data sample set, each original sample image is rotated around a center point by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, so that each original sample image is expanded into four versions with different rotation angles for subsequent training.
11. A model lightweight application deployment system based on knowledge distillation according to claim 9, It is characterized in that The step of training the student network using the adversarial teacher model as the guiding teacher model of the student network comprises: Freeze the parameters of the adversarial teacher model and use KL divergence calculation and The improved similarity between NKL ; In the process of optimizing the student network, the branch auxiliary network in the student network deepens the multi-scale features output by the backbone network and generates the probability distribution of image logits under multi-angle supervision The true labels c of images supervised by multiple angles are used i , using cross loss To assist the optimization direction of the network; By calculating the probability distribution and the original label The cross entropy loss To train student networks; in, 12. A model lightweight application deployment system based on knowledge distillation according to claim 11, It is characterized in that The total loss function β of the improved knowledge distillation algorithm total It is expressed as: In the formula, The loss function for the adversarial teacher model stage is generated by the KL divergence between the teacher model and the student model; The loss function for the training phase of the student network using the adversarial teacher model as its guiding teacher model; In the formula, α and ω are weight coefficients.
13. A model lightweight application deployment system based on knowledge distillation according to claim 8, It is characterized in that The teacher network is a deep convolutional neural network for detecting one or more of four power scenarios: silicone discoloration, insulator damage, abnormal door closure, and climbing over fences; The student network is a lightweight convolutional neural network.
14. A model lightweight application deployment system based on knowledge distillation according to claim 13, It is characterized in that Also includes: The encapsulation module is used to combine the horizontal and vertical characteristics of the Docker container in a hybrid elastic form after obtaining the trained target student network, and adopt the dynamic expansion and contraction decision of the container to achieve adaptive elastic encapsulation of the target student network.
15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, it implements the model lightweight application deployment method as described in any one of claims 1 to 7.
16. A non-transitory computer readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the model lightweight application deployment method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method for simultaneously carrying out vertical and horizontal scheduling on container application program by machine learning
CN116700885A
Cited By
Large-model end-to-end distillation deployment method, device and equipment for low-computing-power equipment and medium
CN120066803A
Knowledge distillation-based high-speed flow interpretable artificial intelligence control method
CN120370720A
Wireless gesture recognition interaction system for deep learning optimization
CN120928950A
Cross-network collaborative alarm noise reduction method based on knowledge distillation large model
CN121603348A
