A model compression-based power image recognition model lightweight application method
By optimizing the power image recognition model through knowledge distillation and neural network architecture search, the problems of model complexity and resource constraints on edge terminal devices are solved, realizing lightweight and efficient power image recognition applications.
Patent Information
- Application Number
- CN202310308366.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-28
- Publication Date
- 2025-12-05
- Estimated Expiration
- 2043-03-28
AI Technical Summary
Existing power image recognition models suffer from complex network structures, large number of parameters, and high computational and storage resource requirements when deployed on edge terminal devices. Furthermore, existing model compression technologies struggle to balance compression ratio and performance, resulting in poor application performance.
The power image recognition model is compressed using knowledge distillation and neural network architecture search methods. The model structure is optimized to adapt to embedded edge devices by calibrating the model prediction process. By combining knowledge distillation and neural network architecture search, the model can be made lightweight and deployed efficiently.
The power image recognition model has been made lightweight on edge devices, reducing computational and storage overhead while improving the model's performance and adapting it to diverse edge terminal devices.
Smart Images
Figure CN116362325B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to a model compression-based power image recognition model lightweight application method, and belongs to the technical field of image recognition model optimization for edge devices. BACKGROUND
[0002] At present, artificial intelligence technology has been widely applied in the power field, such as power transmission channel visualization, unmanned aerial vehicle inspection, and on-site operation violation monitoring. The power transmission channel visualization prevents external damage by monitoring cameras to take pictures, and uses an artificial intelligence model to analyze possible external damage behaviors and issue warnings.
[0003] However, there are still many problems that have been restricting the further improvement of the professional intelligent application level. The traditional deep learning models such as image recognition mainly used in the process of power artificial intelligence analysis have complex network structures and large parameter quantities, and the model training and deployment running require high basic software and hardware environment and resource overhead. The power edge terminal device is limited by the software and hardware resource configuration, and the normal deployment and running of the model is under great pressure. At the same time, the computing devices / terminals in the power application scene are diverse, and the software and hardware architectures are different, so it is difficult to adapt the intelligent model. Therefore, the existing application mode mainly collects data through edge devices and then transmits them back to the backend / cloud for centralized analysis based on intelligent models.
[0004] Such architecture mode seriously depends on centralized cloud service resources (artificial intelligence platform and video platform), which is easy to cause performance bottleneck. At the same time, the fluctuation of edge application task requests causes the change of load in the time domain, which is easy to cause load superposition to cause response violation in some periods. The running of multiple types of heterogeneous services in the edge also causes uneven distribution of system load in space, resulting in low application performance. In terms of data management, due to the limitation of the current application mode and platform architecture, the processing and mining of massive data collected by edge devices are not enough, and the utilization level is not high, so the data value and data assistance role have not been fully played.
[0005] In order to explore the computing resources and capabilities of edge devices, reduce the dependence on cloud resources and edge-cloud communication channels, and improve the stability and efficiency of intelligent analysis applications, there are still some deficiencies. In the field of image recognition, the performance gap between embedded lightweight network structure and deep learning-based neural network model is large, and the practical effect is poor, which cannot meet the business requirements. The existing model compression technology cannot balance the compression ratio and performance when solving the power image recognition model compression problem, and cannot meet the business application requirements. SUMMARY
[0006] The application aims at the problems and deficiencies in the prior art, and provides a power image recognition model lightweight application method based on model compression, which uses knowledge distillation to compress the power image recognition model, and cooperates with neural network architecture search and model prediction process calibration, so that the DNN model for image recognition can be applied to embedded edge devices in the power image recognition application scene, the storage cost and calculation cost of the power image recognition model are reduced, and the performance effect of the compressed power image recognition model is improved.
[0007] The technical scheme is a power image recognition model lightweight application method based on model compression, which supports embedded edge devices to realize power image recognition, and takes into account the model compression ratio and model performance; comprising the following steps: 1) deploying a camera and a DNN model in an embedded edge device, for collecting power images and performing power image recognition; 2) using a knowledge distillation-based model compression method to compress the DNN model deployed on the embedded edge device; 3) using a neural network architecture search method to search for and prune the optimal structure of the student model; 4) using the output information of the intermediate layer of the model to optimize the knowledge distillation-based model compression method, and realizing model prediction process calibration; 5) deploying the lightweight power image recognition model obtained after compression on a power grid embedded edge device, and completing power image data collection and analysis on the power grid edge device.
[0008] The DNN model comprises a neural network structure of an input layer, an output layer and a plurality of hidden layers, and usually comprises a convolution layer, a pooling layer, a batch normalization layer, a full connection layer and an activation layer. The commonly used DNN models in the image classification field include ResNet_v1, ResNet_v2, WideResNet (WRN), ResNeXt, VGG and the like, and the commonly used models in the target detection field include Faster R-CNN, SSD, YOLO and the like.
[0009] In the step 2), the model compression will distinguish between the training model and the deployed model. Because there is a certain inconsistency between the training environment and the deployment environment: in the training stage, a complex model and a large amount of calculation resources are needed to extract information from a very large and highly redundant data set; and in the deployment stage, the calculation cost and storage cost of the model are usually strictly limited. Therefore, the model used in the training stage needs to be converted into a model effective in the deployment stage through model compression.
[0010] The model compression method based on knowledge distillation uses a teacher-student model, in which the teacher model is the "knowledge" outputter and the student model is the "knowledge" accepter. Knowledge distillation generally includes two stages of teacher model training and student model training. The teacher model training stage is the process of training the teacher model, which is characterized by a relatively complex model, with model accuracy as the primary goal, without considering the limitations of model architecture, parameter quantity, and computational overhead; the student model training stage is the process of training the student model, which is characterized by a relatively simple model structure and fewer parameters, and the training goal is to obtain the best performance model under the limitations of model architecture, parameter quantity, and computational overhead.
[0011] The teacher model training stage is a process of maximizing the likelihood of the true label (Ground Truth) of the input power image data, that is, the training goal is to make the prediction output of the model as close as possible to the true label of the power image. Taking a classification task as an example, in the teacher model training stage, the cross-entropy (Cross-Entropy) of the prediction output of the model and the true label is calculated as the loss function (Loss = CrossEntropy), and the error is back propagated to update the parameters of the entire DNN model to obtain a model with the smallest loss, which is the teacher model.
[0012] In the student model training stage, in addition to the training set and test set, the trained model obtained in the teacher model training stage is also needed. The training goal of this stage includes two parts: the first part is still to make the prediction output of the model as close as possible to the true label of the power image data, and this part uses the true label value information of the power image data; the second part is to make the performance of the student model consistent with that of the teacher model as much as possible, that is, to produce as similar as possible final representations (Logits) of the model for the same input of power image data, and this part uses the information of the teacher model. According to the two-part training goal of this stage, the loss function of this part of training can be divided into two parts, which are: the cross-entropy (Cross-Entropy) of the prediction output of the model and the true label, and the similarity between the student model and the teacher model in terms of data final representation, measured by KL divergence. At this time, the form of the loss function is: Loss = a·CrossEntropy + b·KL Logits The hyperparameters a and b are used to adjust the weights of the two parts of the loss function, and then the error back propagation method is used to update the parameters of the student model to obtain a student model with the smallest loss.
[0013] In the 3), the neural network architecture search is a technology for automatically designing a neural network, which can automatically design a high-performance network structure according to a sample set through an algorithm. The student model indicated in the 2) is a neural network model with an uncertain architecture, and the expectation is to obtain a model with the best performance under the condition of satisfying the compression rate. In order to obtain the best student model architecture, a coarse-grained search is performed using the neural network architecture search.
[0014] In order to reduce the search time of the neural network architecture search, the search space is compressed, and a coarse-grained neural network architecture search method is used, that is, a block combination method. The neural network layers of each layer of the DNN model are divided according to their input and output specifications, convolution kernel size, channel number and other characteristics. After division, the DNN model can be regarded as being composed of a plurality of blocks connected in series. The block combination is a unit organization form of the DNN model, which can be regarded as a basic unit of the DNN model. Each block is composed of a plurality of neuron layers, including convolution layers, fully connected layers, activation layers and batch normalization layers. Considering that the size of the neuron layer is closely related to the depth of the block in which the neuron layer is located in the DNN model, this rule is that the deeper the block in the DNN, the more neurons in the neuron layer in the block, that is, the more parameters and the greater the calculation overhead. Under the premise of limiting the total parameter amount of the DNN model, the number of neuron layers in the blocks of different depths is combined to build a plurality of candidate student models with different structures. After pruning these student models, training and evaluation are performed to select the best student model, and the neural network architecture search of the student model is completed.
[0015] The number of neural network layers in each block is independent of each other and has different parameter amounts, which can be adjusted as needed. Through a breadth-first search method, the combination method of the number of blocks in each block in the candidate model structure is searched.
[0016] Because the number of candidate models generated by the search is large, the search space is large, and the subsequent training overhead is also large, the search process is pruned to reduce the search results to reduce the overhead. The pruning rule is that in the same block, the number of network layers used by the student model should not exceed the number of network layers used by the corresponding block of the teacher model. That is, the upper limit of the number of network layers of the student model in each block is the number of network layers of the teacher model in the corresponding block, and the lower limit is 1. This rule can be understood as follows: in order to learn the knowledge of a certain block of the teacher model as much as possible, only the number of network layers in the corresponding block of the teacher model is used, and the number of network layers that exceeds the number of network layers in the corresponding block of the teacher model is meaningless. In practice, the actual results of the model also verify this idea. Therefore, this pruning idea is adopted in this process, which greatly improves the system performance.
[0017] Therefore, the block-combined neural network architecture search is to search the combination of the number of neural network layers in each block of the DNN model and prune, thereby obtaining a plurality of candidate student models with different structures. The student model training phase in the knowledge distillation indicated in 2) is sequentially performed on these student models, and the training results are evaluated to obtain a student model with the best performance effect, thereby completing the neural network architecture search process.
[0018] In 4), the model prediction process calibration is a supplement to the student model training process in the knowledge distillation in 2) according to the neural network architecture search strategy in 3). According to 3), the DNN model can be regarded as being formed by a plurality of blocks connected in series, and the blocks are one-to-one corresponding for the teacher model and the student model. This correspondence is not only structural correspondence, but also functional correspondence, that is, the corresponding blocks in the teacher model and the student model perform similar feature mapping, and the output results of each block can be regarded as an intermediate representation obtained by the model for the input of the block. As the depth of the blocks in the model increases, the intermediate representation of the model becomes more complex. The process of the model producing a complex intermediate representation is also the process of the model predicting the input power image data. In this process, the additional calibration is to calibrate the intermediate representation produced by the model. Similar to knowledge distillation, the student model needs to learn the final representation (Logits) of the teacher model; and in the intermediate calibration process, the student model also needs to learn the intermediate representation of the teacher model. Through the calibration in the model prediction process, the prediction of the student model is closer to the teacher model, thereby improving the performance of the student model.
[0019] In the intermediate calibration process, the similarity of the intermediate representations of the blocks of the student model and the teacher model is additionally introduced, measured by KL divergence, and added to the final loss function to obtain a loss function containing three terms, and the proportions of the three terms, classification loss (CrossEntropy), Logits similarity loss (KL Logits ) and intermediate representation similarity loss (∑KL Block ) are adjusted by hyperparameters a, b and g, in the form of: Loss = a·CrossEntropy + b·KL Logits + g·∑KL Block .
[0020] In 5), the compressed model deployment, the lightweight model obtained by compressing the original complex teacher model using the above model compression strategy, can meet the computing resources of the power grid edge embedded device, and realize data collection and analysis on the edge device. The input is the real-time power image collected by the camera of the edge device, the model performs prediction analysis on the power image, and the image that may contain alarm information is sent to the backend / cloud for confirmation.
[0021] 2) Model compression method based on knowledge distillation compresses the DNN model deployed on the embedded edge device, and the implementation process is as follows:
[0022] 201) Teacher model training stage:
[0023] i) Construct the teacher model architecture and randomly initialize the model parameters;
[0024] ii) Read the power image data in batches and input it to the teacher model;
[0025] iii) Calculate the cross-entropy (Cross-Entropy) of the model prediction value and the true label value to get the loss Loss of the model;
[0026] iv) Update the parameters of each neural network layer of the model according to the loss Loss using the error back propagation algorithm;
[0027] v) Train the model until the model converges to get the trained teacher model.
[0028] 202) Student model training stage:
[0029] i) Construct the student model architecture and randomly initialize the model parameters;
[0030] ii) Read the power image data in batches and input it to the teacher model and the student model;
[0031] iii) Calculate the cross-entropy (Cross-Entropy) of the prediction value of the student model and the true label value, and calculate the KL divergence between the final representation Logits of the student model and the final representation Logits of the teacher model, and combine the two to get the loss Loss of the model;
[0032] iv) Update the parameters of each neural network layer of the student model according to the loss Loss using the error back propagation algorithm, but the parameters of each layer of the teacher model need to remain unchanged and not be updated;
[0033] v) Train the student model until the model converges to get the trained student model.
[0034] 3) Based on the method of neural network architecture search, the optimal structure of the student model is searched, and the implementation process is as follows:
[0035] 301) According to the compression rate requirement, select a candidate student model structure. For example, to compress ResNet-110, the compression rate requirement is about 20%, and the candidate student model structure is ResNet-20.
[0036] 302) The network structure of the candidate student model is divided into blocks. For example, both ResNet-110 and ResNet-20 networks can be divided into a network main part composed of 3 block groups, but there are 18 layers of neural networks in each block in ResNet-110, while there are only 3 layers of neural networks in each block in ResNet-20.
[0037] 303) Calculate the parameter proportion of the neural network layers in each block. For example, in the ResNet network architecture, the parameter quantity of a neural network layer in a deeper layer is 4 times that of the previous layer.
[0038] 304) According to the above parameter proportion and the structure of the candidate student model, a breadth-first method is used to search for other candidate student model structures. For example, the structure of ResNet-20 can be represented as {3, 3, 3}, and on this basis, other candidate student model structures such as {3, 7, 2}, {7, 10, 1}, {19, 3, 2}, {23, 6, 1} and the like can be derived.
[0039] 305) According to the number of network layers in each block in the teacher model structure, the above search process is pruned to avoid unnecessary search. For example, the structure of ResNet-110 as the teacher model is {18, 18, 18}, so some candidate structures in the search results of ResNet-20 (such as {19, 3, 2}, {23, 6, 1}, etc.) will be pruned and deleted.
[0040] 306) The pruned candidate student model structures are sequentially subjected to knowledge distillation model compression training, and the best performing model is selected as the structure of the student model. For example, according to experimental results, the student model structure of the ResNet-20 system is usually {7, 10, 1}.
[0041] 4) Based on the intermediate layer output information in the model, the knowledge distillation model compression method is optimized to realize model prediction process calibration, and the process is as follows:
[0042] 401) Use the optimal student model structure obtained in process 3) and perform random parameter initialization.
[0043] 402) During the student model training stage in process 202), a third loss function ∑KL Block is introduced to calculate the KL divergence of the intermediate output structure of each block of the student model and the teacher model, which is used to calibrate the model prediction process. The remaining processes remain unchanged, and a trained compressed lightweight model, also known as a lightweight power image recognition model, is obtained.
[0044] 5) Deploy the compressed lightweight model to the power grid embedded edge device, and complete the data acquisition and analysis on the power grid edge device. The process is as follows:
[0045] 501) Deploy the compressed, lightweight model to embedded edge devices in the power grid;
[0046] 502) Real-time power images collected by edge devices are input into the model;
[0047] The 503 model performs predictive analysis on the input power image data and sends power images that may contain alarm information to the backend / cloud for confirmation.
[0048] On the backend / cloud, manual methods are used to handle anomalies and conduct spot checks on the predicted information. Due to the environmental diversity and uncertainty in power identification application scenarios, there is a problem of excessive false alarm rate in the model. Using manual calibration to provide feedback and adjust the model predictions helps to improve the model's performance.
[0049] Compared with existing technical solutions, the present invention has the following characteristics:
[0050] 1) Use knowledge distillation to compress the model, thereby improving the performance of DNN models running on resource-constrained devices such as embedded edge devices in the power grid;
[0051] 2) The neural network architecture search method optimizes the compressed model structure and can be applied to various existing model compression methods to improve their compression effect;
[0052] 3) Based on the output information of the intermediate layer of the model, the model prediction process is calibrated. This, combined with knowledge distillation and neural network architecture search, makes the compressed student model closer to the teacher model.
[0053] 4) This method is universal, decoupled from the system implementation, and can be adapted to deep learning tasks on devices with limited resources. Attached Figure Description
[0054] Figure 1 This is a flowchart illustrating the overall execution process of the present invention.
[0055] Figure 2 A flowchart illustrating the calibration process for the model prediction process provided by this invention;
[0056] Figure 3 This is a flowchart illustrating the process of searching for the optimal structure of a model in an embodiment of the present invention. Detailed Implementation
[0057] The application will be further illustrated by the following specific examples, which are only used to illustrate the application and not used to limit the scope of the application. After reading the application, those skilled in the art can make various modifications to the application, and the modifications fall within the scope defined by the appended claims.
[0058] A model compression-based power image recognition model lightweight application method, characterized in that it comprises: 1) deploying a camera and a DNN model in an embedded edge device to collect power images and perform power image recognition; 2) using a knowledge distillation-based model compression method to compress the DNN model deployed on the embedded edge device; 3) using a neural network architecture search method to search for and prune the optimal structure of the student model; 4) using the output information of the intermediate layer of the model to optimize the knowledge distillation-based model compression method and realize model prediction process calibration; and 5) deploying the lightweight power image recognition model obtained after compression to the power grid embedded edge device to complete the collection and analysis of power image data on the power grid edge device.
[0059] On an embedded edge device with limited computing resources, a lightweight DNN model is deployed to perform prediction analysis on images collected by a camera on the edge device.
[0060] In the application scenario of power image recognition, there are many movable or immovable terminals, such as smart phones, unmanned vehicles, wearable devices, etc. These devices have many limitations in computing power, storage space, and power endurance, but at the same time, they may need to perform complex computing processes such as image detection, face recognition, and augmented reality. A feasible method is to deploy a lightweight model after model compression on these edge devices.
[0061] The lightweight DNN model deployed on the embedded edge device includes common ResNet_v1, ResNet_v2, WideResNet(WRN), ResNeXt, VGG, etc. The neuron layers contained include convolutional layers, pooling layers, batch normalization layers, fully connected layers, and activation layers, etc. By changing the number of layers, the size of each layer, and the combination of neural networks, the parameter quantity and the calculation quantity of the neural network can be changed, thereby changing the architecture of the model. The method of changing the architecture of the model based on this is called neural network architecture search.
[0062] For the model compression method using knowledge distillation, the difference between the output (i.e. Logits) of the teacher model and the student model is minimized, and the loss function Kullback-Leibler (KL) divergence is used to measure the difference value.
[0063] Given the same power image as input of the teacher model and the student model, the final representation (Logits, i.e. "knowledge") of the input power image of the two models is obtained through the calculation of the model, and the process of setting the loss function to make the student model as close as possible to the teacher model is the knowledge distillation. Through knowledge distillation, the more lightweight student model learns the representation of the input data of the teacher model, thereby realizing the maintenance of the model accuracy while reducing the calculation and storage overhead, and completing the model compression.
[0064] Figure 1 The overall execution flowchart of the application is described, including model compression, model deployment, and intelligent platform analysis. Figure 2 The overall flowchart of model compression in the application is specifically described, including knowledge distillation, student model architecture search, and model prediction process calibration.
[0065] The specific implementation is described below. First, the model compression part is described:
[0066] Step one: first, the teacher model is trained. The teacher model architecture is constructed, and the model parameters are randomly initialized; the power image data is read in batches and input to the teacher model; the cross-entropy of the model prediction value and the true label value is calculated, and the loss function is calculated; the error back propagation algorithm is used to update the parameters of each neural network layer of the model according to the loss; the model is trained until convergence, and the trained model is obtained as the teacher model in the subsequent steps.
[0067] Step two: second, the search space of the student model structure is constructed. According to the size of the trained teacher model and the compression rate requirement, the basic candidate structure of the student model is determined. Taking the trained teacher model as ResNet-110 and the compression rate requirement of about 20% as an example, the basic candidate structure of the student model can be determined as ResNet-20.
[0068] The basic candidate structure of the student model is divided into blocks. The main part of the ResNet series network structure can be divided into 3 blocks.
[0069] The number of neural network layers in each block of the basic candidate structure of the student model is determined, and the number relationship of the parameter amount of the neural network layers between the blocks is determined. In the ResNet-20 structure, each block contains 3 layers of neural network; the parameter amount of the neural network layer in the deeper layer is 4 times that of the upper layer.
[0070] Other candidate structures are derived from the basic candidate structure of the student model. According to the number of neural network layers in each block and the number relationship of the neural network layer parameters between the blocks, different neural network structures with the same parameter amount can be obtained and expanded for search. The search process adopts a breadth-first method, first decreases the number of network layers in the deepest block, and for each decrease of one layer, the number of network layers in the previous block is increased by a corresponding number of network layers, and so on for the number of network layers in each block in all blocks as the upper limit of the number of network layers in the corresponding block in the candidate model structure, pruning the search process.
[0071] For example, the basic structure of ResNet-20{3,3,3}, that is, containing three blocks, each block has three layers of neural network, according to the ResNet network architecture, the deeper layer of neural network is 4 times the parameter amount of the previous neural network, therefore the derived structures {3,7,2}, {7,10,1}, {19,3,2}, {23,6,1} have the same parameter amount as the structure {3,3,3}, but the latter two do not meet the pruning rule and need to be deleted, the former two are the retained candidate student model structures, which constitute part of the search space of the student model structure.
[0072] Step three: Then the candidate student model structure is trained. For each candidate student model structure in the student model structure search space, construct and randomly initialize the model parameters; read in the power image data in batches, and input them into the teacher model and the candidate student model; calculate the cross-entropy between the predicted value of the student model and the true label value, and calculate the KL divergence between the intermediate output result and the final output result Logits of each block of the student model and the intermediate output result and Logits of the teacher model, and combine to obtain the loss function; use the error back propagation algorithm to update the parameters of each neural network layer of the student model according to the loss, but the parameters of each layer of the teacher model need to remain unchanged and not be updated; finally, train the student model until it converges, and take the best one among all candidate student models as the model for subsequent deployment.
[0073] Next is the execution process part of the model deployment:
[0074] Step four: The model trained in step three is a lightweight model for power image recognition based on the knowledge distillation model compression technology considering storage volume, decision accuracy, inference speed and other indicators, considering edge devices, which can be directly deployed on the embedded edge devices of the power grid, and according to the power image information collected by the external input devices such as cameras of the devices, real-time power image data collection, data screening and prediction analysis are performed.
[0075] Next is the execution process part of the intelligent platform analysis:
[0076] Step five: the predicted result in step four will be returned to the intelligent platform in the back end / cloud, and in the cloud, the cloud server will use the resource advantage to verify and confirm the power image data returned from the power grid edge device using a model with higher accuracy and better performance (such as the teacher model trained in step one), and introduce artificial inspection to further improve the model alarm effectiveness.
[0077] Through the above method, the application realizes a power image model lightweight application technology based on model compression. The method adopts a model compression method based on knowledge distillation, and adopts a construction method based on neural network architecture search for the compressed model architecture, and adopts a prediction process calibration method to optimize the performance of the compressed model.
[0078] Obviously, those skilled in the art should understand that the steps of the power image model lightweight application technology based on model compression of the above-mentioned embodiments of the application can be realized by general computing devices, which can be realized on embedded edge devices with limited computing resources, or on cloud servers with unlimited resources. Alternatively, they can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be made into individual integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module. Thus, the embodiments of the application are not limited to any specific combination of hardware and software.
Claims
1. A model compression-based power image recognition model lightweight application method, characterized in that, Comprise, 1) deploy camera and DNN model in embedded edge device, for collecting power image and power image recognition; 2) the model compression method based on knowledge distillation is compressed to the DNN model deployed on the embedded edge device; 3) the optimal structure of student model is searched and pruned based on the method of neural network architecture search; 4) based on the output information of the intermediate layer of the model, the model compression method based on knowledge distillation is optimized, and the model prediction process is calibrated; 5) the lightweight power image recognition model obtained after compression is deployed on the power grid embedded edge device, and the power image data collection and analysis are completed on the power grid edge device; In the 2), the student model training stage, the training target contains two parts: the first part is still to make the prediction output of the model as close as possible to the true label of the power image data, and the true label value information of the power image data is used; The second part is to make the student model and the teacher model as consistent as possible, that is, to generate as similar as possible final representations of the model for the same power image data input, and the information of the teacher model is used in this part; according to the two-part training target of this stage, the loss function of this part is divided into two parts, which are respectively: the cross entropy of the calculation model's prediction output and the real label and the similarity of the student model and the teacher model about the final representation of the data, which is measured by the KL divergence; at this time, the form of the loss function is: Loss = α·CrossEntropy + β·KL Logits Additional hyperparameters are used to adjust the weights of the two parts of the loss function, and then the error back propagation method is used to update the parameters of the student model to obtain the student model with the minimum loss. The 3) based on the method of neural network architecture search, the optimal structure of student model is searched, and the process is as follows: 301) according to the compression rate requirement, a candidate student model structure is selected; 302) the network structure of the candidate student model is divided into blocks; 303) the parameter quantity proportion of each block neural network layer is calculated; 304) according to the above parameter quantity proportion and the candidate student model structure, the breadth first method is used to search other candidate student model structures; 305) according to the number of network layers in each block in the teacher model structure, the above search process is pruned; 306) the knowledge distillation model compression training is carried out on each candidate student model structure after pruning in turn, and the best model is selected as the structure of the student model.
2. The model compression-based power image recognition model lightweight application method according to claim 1, characterized in that, In the 2), the model compression method based on knowledge distillation usually contains two stages of teacher model training and student model training; The teacher model training stage is the process of maximum likelihood of the true label of the input power image data. For classification task, in the teacher model training stage, the cross entropy of the prediction output and the true label of the model is calculated as the loss function, the error is propagated back to update the parameters of the whole DNN model by minimizing the loss function, and the model with the minimum loss is obtained, that is, the teacher model. 3.The method of claim 1, wherein, In the 3), a coarse-grained search is carried out by using neural network architecture search. In order to reduce the search time of neural network architecture search, the search space is compressed, and a coarse-grained neural network architecture search method is used, that is, the block combination method; the layers of neural network of DNN model are divided according to its input and output specifications, convolution kernel size and channel number characteristics. After division, the DNN model is regarded as being composed of a plurality of blocks connected in series, and the number of neural network layers in each block is independent and different in parameter quantity, which is adjusted according to the need, and the block number combination method in the candidate model structure is searched by breadth first search. 4.The method of claim 1, wherein, Pruning is performed on the search process, and the pruning rule is: in the same block, the number of network layers used by the student model should not exceed the number of network layers used by the corresponding block of the teacher model; that is, the upper limit of the number of network layers of the student model in each block is the number of network layers of the teacher model in the corresponding block, and the lower limit is 1; The block-combined neural network architecture search is to search and prune the combination of the number of neural network layers in each block of the DNN model, thereby obtaining a plurality of candidate student models with different structures, sequentially performing the student model training stage in the knowledge distillation indicated in 2), and evaluating the training results to obtain a student model with the best performance effect, and completing the neural network architecture search process. 5.The method of claim 1, wherein, In 4), the model prediction process calibration is to calibrate the intermediate representation generated by the model, and the intermediate calibration process, the student model also needs to learn the intermediate representation of the teacher model; In the intermediate calibration process, the similarity of the intermediate representations of the student model and the teacher model is introduced, measured by KL divergence, and added to the final loss function to obtain a loss function containing three terms, and the proportion of each term is determined by the hyperparameter, in the form of: Loss = a CrossEntropy + b KL Logits + g SumKL Block . 6.The method of claim 1, wherein, The model compression method based on knowledge distillation in 2) compresses the DNN model deployed on the embedded edge device, and the implementation process is as follows: 201) Teacher model training stage: i) Construct the teacher model architecture and randomly initialize the model parameters; ii) Read the power image data in batches and input it to the teacher model; iii) Calculate the cross-entropy (Cross-Entropy) of the model prediction value and the true label value to obtain the loss Loss of the model; iv) Update the parameters of each neural network layer of the model according to the loss Loss using the error back propagation algorithm; v) Train the model until the model converges to obtain the trained teacher model; 202) Student model training stage: i) Construct the student model architecture and randomly initialize the model parameters; ii) Read the power image data in batches and input it to the teacher model and the student model; iii) Calculate the cross-entropy (Cross-Entropy) of the prediction value of the student model and the true label value, and calculate the KL divergence between the final representation Logits of the student model and the final representation Logits of the teacher model, and combine the two to obtain the loss Loss of the model; iv) Update the parameters of each neural network layer of the student model according to the loss Loss using the error back propagation algorithm, but the parameters of each layer of the teacher model need to remain unchanged and not be updated; v) Train the student model until the model converges to obtain the trained student model. 7.The method of claim 1, wherein, 4) Based on the intermediate layer output information of the model, the model compression method of knowledge distillation is optimized to realize the model prediction process calibration, and the implementation process is as follows: 401) Use the optimal student model structure obtained in 3) and perform parameter random initialization; 402) In the student model training stage in the process of 202), a new loss function is introduced to calculate the KL divergence of the intermediate output structure of each block of the student model and the teacher model, which is used to calibrate the model prediction process; the remaining process remains unchanged to obtain the trained compressed lightweight model, also known as the lightweight power image recognition model. 8.The method of claim 1, wherein, The 5) deploy the lightweight model obtained after compression to the power grid embedded edge device, complete data collection and analysis on the power grid edge device, and realize the process as follows: 501) deploy the compressed lightweight model to the power grid embedded edge device; 502) input the real-time power image collected by the edge device to the model; 503) the model performs prediction analysis on the input power image data, and sends the power image that may contain alarm information to the backend / cloud for confirmation.