Improved knowledge distillation method for edge intelligent ecosystem
The method uses a real teacher model and Gaussian noise-based virtual teachers to streamline knowledge distillation, addressing computational overhead and convergence speed issues in edge devices, ensuring high accuracy and efficient performance on large-scale datasets.
Patent Information
- Application Number
- CN202411733353.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2024-12-31
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing multi-teacher distillation methods for edge intelligent ecosystems require training multiple pre-trained teacher models, leading to significant computational overhead and making them unsuitable for large-scale datasets, while current single-teacher methods struggle with performance and convergence speed on edge devices with limited computing power.
A knowledge distillation method using a real teacher model and simulated virtual teacher models generated by Gaussian noise to guide the student model, reducing computational overhead and improving convergence speed without additional environmental deployment costs.
The method significantly reduces computational overhead and inference costs, enhances convergence speed, and maintains performance comparable to multi-teacher methods, achieving high accuracy in large-scale datasets without altering the edge device environment.
Smart Images
Figure CN119227735A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of knowledge distillation technology, and in particular to an improved knowledge distillation method for edge intelligent ecosystem. Background Art
[0002] Deep learning has made significant progress in recent years, and in the field of smart ecosystems, deep learning models are being integrated into production and daily life devices. However, edge devices with limited computing power are usually unable to smoothly apply deep learning algorithms. Therefore, improving the performance of smaller models without changing the original architecture of deep learning algorithms is a major challenge in the field of smart communication systems.
[0003] Knowledge distillation refers to a deep learning algorithm that uses a teacher model to guide the training of a student model, and allows the student model to learn the knowledge of the teacher model through distillation. In the compression of the model, the teacher model is a complex model that has been trained in advance, while the student model is a smaller model. Knowledge distillation does not change the original structure of the model, it just replaces the original weight file to improve the performance of the algorithm. The knowledge distillation method effectively solves the problem of computing performance limits of edge intelligent devices in the field of intelligent communication systems.
[0004] In order to improve the process of transferring knowledge in more complex environments, many different knowledge distillation algorithms have emerged, including adversarial distillation methods, multi-teacher distillation methods, cross-mode distillation methods, data-free distillation methods, and quantized distillation methods. At present, the prior art generally applies multi-teacher distillation methods to edge intelligence ecosystems. However, the current multi-teacher distillation method requires training multiple pre-trained teacher models in the same dataset, which makes testing the multi-teacher distillation method in large-scale datasets not only generate a lot of additional computational overhead, so the multi-teacher distillation method in the prior art cannot be applied to large-scale datasets. Summary of the invention
[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an improved knowledge distillation method for edge intelligent ecosystem, which is used to improve the convergence speed of the knowledge distillation method and reduce the computational overhead on large-scale data sets.
[0006] To achieve the above object, the present invention provides the following technical solution: an improved knowledge distillation method for edge intelligent ecosystem, comprising:
[0007] Step S1, introducing a real teacher model, inputting input data into the real teacher model to obtain a real feature map;
[0008] Step S2, configuring at least two Gaussian noises using a preset noise generation formula, using the Gaussian noises as a virtual teacher model, and introducing the at least two Gaussian noises into the real feature map to form at least two virtual feature maps;
[0009] Step S3: using the plurality of virtual feature maps to guide the preset student model to learn.
[0010] Furthermore, in step S2, the noise generation formula is configured as:
[0011] ,
[0012] in, For the Gaussian noise, The expression used to represent the Gaussian noise is, Used to represent input data, is used to represent the average value of the Gaussian noise, It is used to represent the variance of the Gaussian noise.
[0013] Furthermore, in step S2, each of the Gaussian noises and the real feature map is input into a preset virtual feature calculation formula to calculate the corresponding virtual feature map, and the virtual feature calculation formula is configured as:
[0014] ,
[0015] in, For representing the virtual feature map, For representing the real feature map, is used to represent the mean of the true feature map, is used to represent the variance of the true feature map, Used to indicate the first hyperparameter set in advance.
[0016] Furthermore, the step S2 includes:
[0017] Step S21, configure a virtual teacher mask matrix, and the virtual teacher mask matrix is used to verify the prediction accuracy of the virtual feature map using the classifier of the real teacher model. During the verification process of the prediction accuracy, the samples of the virtual feature map that are correctly predicted are set to 1, and the samples of the virtual feature map that are incorrectly predicted are set to 0.
[0018] Furthermore, the step S2 further includes:
[0019] Step S22, inputting the input data into the student model to obtain a student feature map, and inputting the virtual teacher mask matrix, the virtual feature map and the student feature map into a preset loss calculation formula to calculate the feature distillation loss.
[0020] Furthermore, the loss calculation formula is configured as:
[0021] ,
[0022] in, For virtual mask features, Used to represent student mask features, For representing the student characteristics graph, is used to represent the characteristic distillation loss, A mask matrix used to represent the virtual teacher.
[0023] Furthermore, the step S2 further includes:
[0024] Step S23, input the real feature map and the student feature map into a preset directional norm comprehensive calculation equation to calculate the directional norm loss, and then input the directional norm loss into a preset overall loss function equation to calculate the overall loss.
[0025] Furthermore, the directional norm comprehensive calculation equation is configured as:
[0026] ,
[0027] in, Used to represent norm loss, Used to indicate directional loss, Used to indicate the size of small batches of data, Used to represent the Euclidean norm, Used to represent a preconfigured second hyperparameter.
[0028] Furthermore, the overall loss function equation is configured as:
[0029] ,
[0030] in, is used to represent the overall loss, Used to represent the cross entropy loss, Used to represent KL divergence, , , They are respectively used to represent the pre-configured first loss coefficient, second loss coefficient and third loss coefficient.
[0031] Furthermore, the edge intelligent ecosystem is configured with a number of intelligent ecological detection devices, each of which is used to detect intelligent ecological environment parameters, and the intelligent ecological environment parameters include PM2.5 index, sulfur dioxide concentration, ecological temperature, ecological humidity, and oxygen concentration;
[0032] In step S2, each of the smart ecological environment parameters is input into a preset hyperparameter calculation formula to calculate the first hyperparameter;
[0033] The hyperparameter calculation formula is configured as:
[0034] ,
[0035] in, It is used to represent the preset first initial hyperparameters. , Used to represent the PM2.5 index, Used to indicate the preset standard PM2.5 index. Used to express the sulfur dioxide concentration, Used to indicate the preset standard sulfur dioxide concentration. Used to indicate ecological temperature, Used to indicate the preset standard ecological temperature. Used to indicate the ecological humidity, Used to indicate the preset standard ecological humidity. Used to express the oxygen concentration, Used to indicate the preset standard oxygen concentration. , , , , They are respectively used to represent the pre-configured first environment adjustment coefficient, second environment adjustment coefficient, third environment adjustment coefficient, fourth environment adjustment coefficient and fifth environment adjustment coefficient.
[0036] Beneficial effects of the present invention:
[0037] The present invention uses only one real teacher model, uses the real teacher model to obtain a real feature map, and uses at least two pre-configured Gaussian noises as virtual teacher models, and introduces at least two Gaussian noises into the real feature map to form at least two virtual feature maps, and then uses at least two virtual feature maps to train the student model to complete the learning process of knowledge distillation;
[0038] The present invention uses a virtual teacher model based on Gaussian noise to replace the additional real teacher model that needs to be pre-trained in the multi-teacher distillation method, which significantly reduces the additional computing overhead and inference cost of knowledge distillation in large-scale data sets. At the same time, due to the simplification of the model pre-training process, the convergence speed is improved. In addition, the present invention does not need to change the deployment environment of the existing edge intelligent ecosystem, and will not increase additional environmental deployment costs while improving the knowledge distillation performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a flowchart of the steps of the improved knowledge distillation method for edge intelligent ecosystem in the present invention;
[0040] Figure 2 is a sub-flow chart of the steps in the present invention;
[0041] Figure 3 It is a plot of the prediction accuracy of the student model versus the number of true teacher models. DETAILED DESCRIPTION
[0042] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The same parts are represented by the same reference numerals. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "bottom surface" and "top surface", "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.
[0043] Example 1, reference Figure 1 FIG. 1 is a first embodiment of the present invention, which provides an improved knowledge distillation method for edge intelligent ecosystem, which can improve the convergence speed of the knowledge distillation method and reduce the computational overhead on large-scale data sets, including:
[0044] Step S1, introducing a real teacher model, inputting input data into the real teacher model to obtain a real feature map;
[0045] Step S2, configuring at least two Gaussian noises using a preset noise generation formula, using the Gaussian noises as a virtual teacher model, introducing the at least two Gaussian noises into the real feature map, and forming at least two virtual feature maps;
[0046] Step S3, using multiple virtual feature maps to guide the preset student model to learn.
[0047] Working principle of embodiment 1:
[0048] In this embodiment, only one real teacher model is used, and a real feature map is obtained by using the real teacher model. At least two pre-configured Gaussian noises are used as virtual teacher models, and at least two Gaussian noises are introduced into the real feature map to form at least two virtual feature maps. Then, the student model is trained by using the at least two virtual feature maps to complete the learning process of knowledge distillation.
[0049] This embodiment uses a virtual teacher model based on Gaussian noise to replace the additional real teacher model that needs to be pre-trained in the multi-teacher distillation method, which significantly reduces the additional computing overhead and inference cost of knowledge distillation in large-scale data sets. At the same time, due to the streamlining of the model pre-training process, the convergence speed is improved. In addition, this embodiment does not need to change the deployment environment of the existing edge intelligent ecosystem, and will not increase additional environmental deployment costs while improving the knowledge distillation performance.
[0050] This embodiment uses the CIFAR-100 dataset and the IMAGENET-1K dataset as large-scale datasets to experimentally test the distillation effect of the improved knowledge distillation method in this embodiment.
[0051] Table 1 is a comparison table of the TOP-1 test accuracy (%) of the improved knowledge distillation method in this embodiment and other single-teacher distillation methods based on the CIFAR-100 dataset. The comparison of the TOP-1 test accuracy (%) of the improved knowledge distillation method based on the CIFAR-100 dataset and other single-teacher distillation methods is shown in Table 1:
[0052] Table 1
[0053]
[0054] In Tables 1 and 2, Ours represents the test accuracy of the improved knowledge distillation method, Student represents the student model, and Teacher represents the teacher model. According to Table 1, in the CIFAR-100 dataset, the improved knowledge distillation method outperforms other knowledge distillation methods in the accuracy pairs of most teacher models and student models. Although it performs slightly worse in the distillation experiments of VGG-13, VGG-8, ResNet-56, and ResNet20, the gap with the best result does not exceed 0.13%. Therefore, the improved knowledge distillation method performs well in the test experiments of the CIFAR-100 dataset.
[0055] Table 2 is a comparison table of the TOP-1 and TOP-5 test accuracy (%) of the improved knowledge distillation method based on the IMAGENET-1K dataset and other single-teacher distillation methods. The comparison of the TOP-1 and TOP-5 test accuracy (%) of the improved knowledge distillation method based on the IMAGENET-1K dataset and other single-teacher distillation methods is shown in Table 2:
[0056] Table 2
[0057]
[0058] In the ImageNet-lk experiment, the improved knowledge distillation method not only achieved higher final accuracy, but also showed the fastest convergence speed in the early stage. Therefore, experiments on the CIFAR-100 dataset and ImageNet-1k dataset proved the robustness and generalization ability of the improved knowledge distillation method.
[0059] Table 3 is a comparison table of the TOP-1 test accuracy (%) of the improved knowledge distillation method based on the CIFAR-100 dataset and the multi-teacher distillation method. The comparison of the TOP-1 test accuracy (%) of the improved knowledge distillation method based on the CIFAR-100 dataset and the multi-teacher distillation method is shown in Table 3:
[0060] Table 3
[0061]
[0062] In Table 3, the improved knowledge distillation method is compared with the current multi-teacher distillation method. In the experiments of these multi-teacher models, the test results of the multi-teacher distillation method are obtained by three teacher models jointly guiding the student model. The comparison results in the above table show that despite the additional guidance of two other teacher models in the existing multi-teacher knowledge distillation method, the improved knowledge distillation method achieves comparable or even higher accuracy in most cases.
[0063] Based on Table 1, Table 2, and Table 3, this embodiment selects one real teacher model and two virtual teacher models to make a fair comparison with the existing multi-teacher distillation method. However, the introduction of the virtual teacher model allows the number of real teacher models that do not require training to be increased. This advantage enables further exploration of how the performance of the student model changes with the increase in the number of real teacher models.
[0064] refer to Figure 3 ,exist Figure 3 In the above figure, the horizontal axis Teacher num is used to represent the teacher number, and the vertical axis ACC represents the prediction accuracy of the student model. The teacher number refers to the total number of real teacher models and virtual teacher models. For example, a teacher number of 1 means there is only one real teacher model, while 3 means there are two virtual teacher models in addition to one real teacher model. Figure 3 It can be observed that when the teacher number is 5, the student model has the highest accuracy of 77.14%. The real teacher model and student model used in this process are ResNet32x4 model and ResNet8x4 model respectively.
[0065] Table 4 is a comparison table of the TOP-1 test accuracy (%) of the real teacher model and the virtual teacher model based on the CIFAR-100 dataset. The comparison of the TOP-1 test accuracy (%) of the real teacher model and the virtual teacher model based on the CIFAR-100 dataset is shown in Table 4:
[0066] Table 4
[0067]
[0068] In Table 4, Virtual+ represents the virtual teacher model. According to the above table, although the virtual teacher model is used instead of the real teacher model in the improved knowledge distillation method, the test accuracy of the improved knowledge distillation method still reaches nearly 99% of the performance of the original method on average. It is worth noting that the improved knowledge distillation method only uses one real teacher model, which results in the computational cost of teacher model inference being only 1 / 3 of that of the original method. At the same time, in some cases, such as CA-MKD and AVER, the test accuracy of the improved knowledge distillation method is slightly higher than that of the original method when the virtual teacher model is used. This phenomenon shows that the existing multi-teacher knowledge distillation method has significant defects.
[0069] Preferably, in step S2, the noise generation formula is configured as:
[0070] ,
[0071] in, Used to represent Gaussian noise, The expression used to represent Gaussian noise is, Used to represent input data, Used to represent the average value of Gaussian noise, Used to represent the variance of Gaussian noise.
[0072] Specifically, in this embodiment, in order to reduce the additional computational overhead, the goal of this embodiment is to use only one teacher model, and the rest of the teacher models are simulated with Gaussian noise to form a virtual teacher model. In order to simulate the feature graphs of the additional teacher models, we introduce Gaussian noise into the feature graphs of a single teacher model to form a virtual feature graph. This process does not require additional reasoning, so additional reasoning overhead can be avoided.
[0073] Preferably, in step S2, each Gaussian noise and the real feature map are input into a preset virtual feature calculation formula to calculate the corresponding virtual feature map, and the virtual feature calculation formula is configured as:
[0074] ,
[0075] in, Used to represent virtual feature maps, Used to represent the real feature map, Used to represent the mean of the true feature map, Used to represent the variance of the true feature map, Used to indicate the first hyperparameter set in advance.
[0076] Specifically, in this embodiment, the first hyperparameter is a perturbation parameter pre-configured for different teacher models. Since the characteristics of the original real teacher model are disturbed during the use of the virtual teacher model, the size of the disturbance is reduced as much as possible by setting the first hyperparameter.
[0077] Table 5 is a comparison table of the first hyperparameter and the performance of the virtual teacher model. The comparison of the first hyperparameter and the performance of the virtual teacher model is shown in Table 5:
[0078] Table 5
[0079]
[0080] In Table 5, the first column represents the first hyperparameter, and the second column shows the angle difference between the virtual feature map and the real feature map, which is calculated using the cosine similarity formula. The third column shows the accuracy of the virtual teacher model obtained by comparing the features of the classifier passed to the real teacher model with the label. According to Table 5, as the first hyperparameter increases, the angle difference between the virtual teacher model and the real teacher model increases, resulting in a decrease in the performance of the virtual teacher. In this embodiment, the first hyperparameter is set to 0.5 to ensure that the performance of the virtual teacher model does not exceed 1% of the performance of the real teacher model.
[0081] Example 2, reference Figure 2 , which is the second embodiment of the present invention. Different from the previous embodiment, this embodiment provides a virtual teacher mask matrix, which can refine the characteristics of the virtual teacher model and further reduce the disturbance. In step S2, it includes:
[0082] Step S21, configure the virtual teacher mask matrix, the virtual teacher mask matrix is used to verify the prediction accuracy of the virtual feature map using the classifier of the real teacher model. During the verification of the prediction accuracy, the samples of the correctly predicted virtual feature map are set to 1, and the samples of the incorrectly predicted virtual feature map are set to 0.
[0083] Preferably, step S2 also includes:
[0084] Step S22, input the input data into the student model to obtain the student feature map, and input the virtual teacher mask matrix, the virtual feature map and the student feature map into the preset loss calculation formula to calculate the feature distillation loss.
[0085] Preferably, the loss calculation formula is configured as:
[0086] ,
[0087] in, For virtual mask features, Used to represent student mask features, Used to represent student characteristics graph, Used to represent feature distillation loss, Used to represent the virtual teacher mask matrix.
[0088] Working principle of embodiment 2:
[0089] Specifically, in this embodiment, the virtual teacher mask matrix is named " ”, The main purpose of is to use the classifier of the real teacher model to verify whether the virtual feature map is correctly predicted. The samples of the correctly predicted virtual feature map are set to 1, and the samples of the incorrectly predicted virtual feature map are set to 0. In this way, before calculating the loss, the virtual teacher mask matrix is multiplied by the student feature map and the virtual feature map respectively to ensure that the wrong features will not appear. By setting the virtual teacher mask matrix, it is possible to avoid affecting the subsequent loss calculation, refine the features of the virtual teacher model, and further reduce the disturbance.
[0090] Example 3, reference Figure 2 , which is the third embodiment of the present invention. Different from the previous embodiment, this embodiment provides a directional norm loss, which can standardize the difference between the real feature map and the student feature map and improve the performance of the student model in the knowledge distillation process. Step S2 also includes:
[0091] Step S23, input the real feature map and the student feature map into the preset directional norm comprehensive calculation equation to calculate the directional norm loss, and then input the directional norm loss into the preset overall loss function equation to calculate the overall loss.
[0092] Preferably, the directional norm comprehensive calculation equation is configured as:
[0093] ,
[0094] in, Used to represent norm loss, Used to indicate directional loss, Used to indicate the size of small batches of data, Used to represent the Euclidean norm, Used to represent a preconfigured second hyperparameter.
[0095] Preferably, the overall loss function equation is configured as:
[0096] ,
[0097] in, Used to represent the overall loss, Used to represent the cross entropy loss, Used to represent KL divergence, , , They are respectively used to represent the pre-configured first loss coefficient, second loss coefficient and third loss coefficient.
[0098] Working principle of embodiment 3:
[0099] In this embodiment, the loss between the student feature map and the real feature map is calculated by two dimensions, norm and direction. Since the norm of the real feature map is usually larger than the norm of the smaller model feature map. Even if the classification layer structure is the same, this difference in feature specification will prevent the smaller model from achieving performance comparable to the real teacher model. Therefore, this embodiment regulates the norm in the loss design by incorporating the difference in the norm between the student feature map and the real feature map. In this embodiment, the Euclidean norm is used to measure the size of the feature vector between the student feature map and the real feature map. In this embodiment, additional regularization is also required for the feature vector direction. Cosine similarity is a common measure for evaluating the directional difference between vectors. Therefore, in this embodiment, by calculating the cosine similarity between the student feature map and the real feature map, the directional dimension is processed to obtain the directional loss. Then the directional loss and the norm loss are combined, and a second hyperparameter is introduced to adjust the ratio between the directional loss and the norm loss to obtain the final distillation loss: directional norm loss. The smaller the directional norm loss, the smaller the difference between the real feature map and the student feature map. Therefore, the directional norm loss can standardize the difference between the real feature map and the student feature map, thereby improving the performance of the student model in the knowledge distillation process. The final overall loss can be combined with the directional norm loss, cross entropy loss, and KL divergence to achieve a comprehensive calculation of the overall loss in the knowledge distillation process. Among them, when training the student model on the CIFAR-100 dataset and the ImageNet-1K dataset, the student model follows the preset experimental training configuration to maintain the loss balance between the different components of the distillation process. The second hyperparameter, the first loss coefficient, and the second loss coefficient are set to 1, and the third loss coefficient is set to 3.
[0100] Embodiment 4 is the third embodiment of the present invention. Different from the previous embodiment, this embodiment provides a hyperparameter calculation formula, which can dynamically adjust the size of the first hyperparameter according to the edge intelligent ecosystem, further reduce the disturbance of the virtual teacher model, and improve the performance of the improved knowledge distillation method. In which, the edge intelligent ecosystem is equipped with a number of intelligent ecological detection devices, each of which is used to detect intelligent ecological environment parameters, and the intelligent ecological environment parameters include PM2.5 index, sulfur dioxide concentration, ecological temperature, ecological humidity, and oxygen concentration;
[0101] In step S2, each intelligent ecological environment parameter is input into a preset hyperparameter calculation formula to calculate a first hyperparameter;
[0102] The hyperparameter calculation formula is configured as:
[0103] ,
[0104] in, It is used to represent the preset first initial hyperparameters. , Used to indicate PM2.5 index, Used to indicate the preset standard PM2.5 index. Used to indicate sulfur dioxide concentration. Used to indicate the preset standard sulfur dioxide concentration. Used to indicate ecological temperature, Used to indicate the preset standard ecological temperature. Used to indicate ecological humidity, Used to indicate the preset standard ecological humidity. Used to indicate oxygen concentration. Used to indicate the preset standard oxygen concentration. , , , , They are respectively used to represent the pre-configured first environment adjustment coefficient, second environment adjustment coefficient, third environment adjustment coefficient, fourth environment adjustment coefficient and fifth environment adjustment coefficient.
[0105] Specifically, in this embodiment, the intelligent ecological detection equipment includes a sensing detection equipment for intelligent ecological environment detection, including a temperature sensor, a humidity sensor, a PM2.5 detector, a sulfur dioxide concentration detector, and an oxygen concentration detector, which detect ecological temperature, ecological humidity, PM2.5 index, sulfur dioxide concentration, and oxygen concentration respectively. The standard sulfur dioxide concentration, standard ecological temperature, standard ecological humidity, standard oxygen concentration, and standard PM2.5 index are parameters measured under ideal environmental conditions, which have a certain gap with the actually measured ecological temperature, ecological humidity, PM2.5 index, sulfur dioxide concentration, and oxygen concentration.
[0106] The first initial hyperparameter can be the first hyperparameter when the virtual teacher performs best under ideal environmental conditions, and its value is 0.5. During the use of the virtual teacher model, since the virtual teacher model disturbs the real features in the original real teacher model, the first hyperparameter is used to reduce the disturbance. The intelligent ecological environment parameters in the ecological environment are dynamically changing, and the dynamically changing intelligent ecological environment parameters affect the performance and life of the edge detection equipment in the edge intelligent ecosystem, causing the detection data of the edge detection equipment to deviate, thereby interfering with the data entered into the large-scale data set, affecting the performance of the virtual teacher model. By introducing dynamically changing intelligent ecological environment parameters to adjust the size of the first hyperparameter in the knowledge distillation process, the impact of the actual ecological environment on the construction and testing of the virtual teacher model is reduced, thereby further reducing the disturbance of the virtual teacher model and improving the performance of the improved knowledge distillation method.
[0107] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. An improved knowledge distillation method for edge intelligence ecosystem, characterized in that: include: Step S1, introducing a real teacher model, inputting input data into the real teacher model to obtain a real feature map; Step S2, configuring at least two Gaussian noises using a preset noise generation formula, using the Gaussian noises as a virtual teacher model, and introducing at least two Gaussian noises into the real feature map to form at least two virtual feature maps; Step S3, using the plurality of virtual feature maps to guide the preset student model to learn.
2. The improved knowledge distillation method for edge intelligent ecosystem according to claim 1, characterized in that: In step S2, the noise generation formula is configured as: ， in, For the Gaussian noise, The expression used to represent the Gaussian noise is, Used to represent input data, is used to represent the average value of the Gaussian noise, It is used to represent the variance of the Gaussian noise.
3. The improved knowledge distillation method for edge intelligent ecosystem according to claim 2, characterized in that: In step S2, each of the Gaussian noises and the real feature map is input into a preset virtual feature calculation formula to calculate the corresponding virtual feature map, and the virtual feature calculation formula is configured as: ， in, For representing the virtual feature map, For representing the real feature map, is used to represent the mean of the real feature map, is used to represent the variance of the true feature map, Used to indicate the first hyperparameter set in advance.
4. The improved knowledge distillation method for edge intelligent ecosystem according to claim 3, characterized in that: The step S2 includes: Step S21, configure a virtual teacher mask matrix, and the virtual teacher mask matrix is used to verify the prediction accuracy of the virtual feature map using the classifier of the real teacher model. During the verification process of the prediction accuracy, the samples of the virtual feature map that are correctly predicted are set to 1, and the samples of the virtual feature map that are incorrectly predicted are set to 0.
5. The improved knowledge distillation method for edge intelligent ecosystem according to claim 4, characterized in that: The step S2 also includes: Step S22, inputting the input data into the student model to obtain a student feature map, and inputting the virtual teacher mask matrix, the virtual feature map and the student feature map into a preset loss calculation formula to calculate the feature distillation loss.
6. The improved knowledge distillation method for edge intelligent ecosystem according to claim 5, characterized in that: The loss calculation formula is configured as: ， in, For virtual mask features, Used to represent student mask features, For representing the student characteristics graph, is used to represent the characteristic distillation loss, A mask matrix used to represent the virtual teacher.
7. The improved knowledge distillation method for edge intelligent ecosystem according to claim 5, characterized in that: The step S2 further comprises: Step S23, input the real feature map and the student feature map into a preset directional norm comprehensive calculation equation to calculate the directional norm loss, and then input the directional norm loss into a preset overall loss function equation to calculate the overall loss.
8. The improved knowledge distillation method for edge intelligent ecosystem according to claim 7, characterized in that: The directional norm comprehensive calculation equation is configured as: ， in, Used to represent norm loss, Used to indicate directional loss, Used to indicate the size of small batches of data, Used to represent the Euclidean norm, Used to represent a preconfigured second hyperparameter.
9. The improved knowledge distillation method for edge intelligent ecosystem according to claim 8, characterized in that: The overall loss function equation is configured as: ， in, is used to represent the overall loss, Used to represent the cross entropy loss, Used to represent KL divergence, , , They are respectively used to represent the pre-configured first loss coefficient, second loss coefficient and third loss coefficient.
10. The improved knowledge distillation method for edge intelligent ecosystem according to claim 3, characterized in that: The edge smart ecosystem is equipped with a number of smart ecological detection devices, each of which is used to detect smart ecological environment parameters, including PM2.5 index, sulfur dioxide concentration, ecological temperature, ecological humidity, and oxygen concentration; In step S2, each of the smart ecological environment parameters is input into a preset hyperparameter calculation formula to calculate the first hyperparameter; The hyperparameter calculation formula is configured as: ， in, It is used to represent the preset first initial hyperparameters. , Used to represent the PM2.5 index, Used to indicate the preset standard PM2.5 index. Used to express the sulfur dioxide concentration, Used to indicate the preset standard sulfur dioxide concentration. Used to indicate ecological temperature, Used to indicate the preset standard ecological temperature. Used to indicate the ecological humidity, Used to indicate the preset standard ecological humidity. Used to express the oxygen concentration, Used to indicate the preset standard oxygen concentration. , , , , They are respectively used to represent the pre-configured first environment adjustment coefficient, second environment adjustment coefficient, third environment adjustment coefficient, fourth environment adjustment coefficient and fifth environment adjustment coefficient.