Multi-student elimination type knowledge distillation method for image classification

Abstract

The invention discloses a multi-student elimination type knowledge distillation method for image classification, and the method comprises the steps: firstly carrying out the analysis and research of the related technology and current situation of wheat scab, carrying out the expansion of the number of data set samples through a data enhancement measure, and selecting a classic classification model; the method comprises the following steps of: researching, analyzing and experimenting the classification capability of the data on a wheat scab data set, and finally, on the basis of an original distillation framework, providing a new distillation mode, adding a plurality of student models and an elimination mechanism, expanding the knowledge acquisition channel of a single model, and enabling the model to learn and fit the respective characteristics of various different data. The model has stronger generalization ability and classification ability; finally, an ablation experiment is carried out, the superiority of an elimination mechanism adopted in the scheme is proved, the model fitting ability is also improved by using the method provided by the invention aiming at the classification of the wheat scab, the highest classification accuracy rate is finally achieved by 93.4%, and the method has a remarkable effect on the field of crop disease detection.

Description

Technical Field

[0001] The invention relates to the technical field of image classification methods, and in particular to the technical field of wheat scab image classification methods. Background Art

[0002] The wide distribution of wheat makes it a major food crop in the world, carrying the main dietary needs of mankind. Its importance to human survival and development is self-evident. Given the key position of wheat, its food security has been studied in depth. However, Fusarium head blight, as a common fungal disease, often occurs during the growth period of wheat, which can significantly affect the yield and quality of wheat. If not discovered and intervened in time, Fusarium head blight may cause a large-scale reduction in wheat yield or even a total crop failure. Infected wheat releases a variety of toxins, such as deoxynivalenol (DON), which can cause health problems such as acute poisoning and immunosuppression in humans and animals. Therefore, it is crucial to effectively identify wheat Fusarium head blight and provide relevant information for its prevention and control.

[0003] In recent years, with the improvement of computer hardware performance, artificial intelligence and image recognition technology have developed rapidly, providing new methods for crop disease identification. These efficient image recognition technologies can not only significantly improve the accuracy of identification, but also reduce labor costs. Deep learning algorithms, especially convolutional neural networks (CNNs), have demonstrated powerful capabilities in the field of image recognition.

[0004] Convolutional neural network is a deep learning architecture widely used in image processing tasks. The model uses key components such as convolutional layers, pooling layers, and fully connected layers to efficiently identify and classify complex data such as images. The core advantages of CNN are its parameter sharing mechanism and local perception ability, which make CNN perform well in image recognition tasks. CNN consists of several key components: convolutional layers are used to capture local features of images; pooling layers reduce the dimension of these features to improve the generalization of the model; non-linear activation layers enhance the expressiveness of the model by introducing non-linearity; and finally, fully connected layers map features to the output layer to complete classification or regression tasks. Through this hierarchical design, CNN can automatically learn the hierarchical feature representation of data without manually designing feature extractors, which significantly improves the efficiency and accuracy of image processing tasks. By automatically learning the representation of data, CNN eliminates the step of manual feature extraction and simplifies the preprocessing process. In the field of computer vision, CNN has achieved remarkable results in competitions such as ImageNet, and its application has been extended to speech recognition, pedestrian detection, and agricultural fields such as identification of crop diseases and pests.

[0005] In the diagnosis of wheat fusarium, the traditional method that relies on manual experience has certain limitations. This method is easily affected by personal subjective judgment and is limited by the individual's knowledge of wheat fusarium. Long-term reliance on this empirical judgment standard may lead to misjudgment due to lack of experience or bias, thereby delaying the opportunity for farmers to take necessary remedial measures for the disease. Therefore, relying solely on traditional manual judgment methods is not enough in large-scale, cross-regional wheat disease diagnosis. In order to improve the accuracy of diagnosis, some researchers have constructed a prediction model using statistical data. Some researchers have established a prediction model by analyzing the occurrence of wheat fusarium in previous years. The model can make predictions based on time, climate and environmental factors, and issue an alarm when the disease may occur. The accuracy of the prediction depends on the analytical ability of the researchers, and the results will be presented to farmers in the form of graphics and text. In addition, a variety of chemical analysis techniques are used in the laboratory, including high-performance liquid chromatography (HPLC), infrared spectroscopy, gas chromatography (GC) and enzyme-linked immunosorbent assay (ELISA), which are also used for the diagnosis of wheat fusarium. However, these methods also have certain limitations, such as strict requirements on experimental time and huge consumption of manpower and material resources during the experiment, which limits their effectiveness in practical applications. Therefore, finding more efficient and accurate diagnostic methods is crucial for the prevention and control of wheat fusarium head blight.

[0006] In view of the above, the present invention provides a multi-student elimination knowledge distillation method for image classification to solve the above problems. Summary of the invention

[0007] The purpose of this invention is to provide a multi-student elimination knowledge distillation method for image classification, and to conduct relevant research on the classification problem of wheat scab, so as to assist farmers in screening wheat scab. In view of the relevant problems in the wheat scab image classification process, this method adopts a new solution to improve the classification accuracy of wheat scab while reducing the model volume, and conducts the following research:

[0008] A multi-student elimination knowledge distillation method for image classification is characterized by first collecting, constructing and processing a data set, then training an original classification network, then selecting a teacher network and a student network from the trained network, and then using the original knowledge distillation method to perform distillation learning on the teacher network and the student network, and finally improving the distillation mode of the original knowledge distillation method.

[0009] The beneficial effects of the technology of the present invention are:

[0010] Based on the original distillation framework, this paper proposes a new distillation mode, adds multiple student models and elimination mechanisms, expands the channels for a single model to acquire knowledge, and allows the model to learn to fit the characteristics of a variety of different data, so that the model has stronger generalization and classification capabilities, increases the robustness of image recognition and classification, and improves recognition efficiency.

[0011] Furthermore, the elimination mechanism ensures that poorly performing models do not negatively impact the training of other student networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a schematic diagram of a wheat fusarium disease sample;

[0013] Figure 2 It is a schematic diagram of the overall technical route of a specific implementation method of the present invention;

[0014] Figure 3 Schematic diagram of the training model for distillation learning of existing technical knowledge;

[0015] Figure 4 Schematic diagram of the original basic architecture for the distillation of prior art knowledge;

[0016] Figure 5 Schematic diagram of two different label probability distributions for distillation of prior art knowledge;

[0017] Figure 6 Schematic diagram of Softmax output at different temperatures for the distillation of prior art knowledge;

[0018] Figure 7 A schematic diagram of part of a data set of wheat fusarium head blight disease grades according to a specific embodiment of the present invention;

[0019] Figure 8 A diagram showing the distribution details of a data set according to a specific embodiment of the present invention;

[0020] Fig. 9 The figure is a graph showing the loss and accuracy changes of several classification models on a data set according to a specific embodiment of the present invention;

[0021] Fig.10 This is a strategy framework diagram of the knowledge distillation model improved by the present invention;

[0022] Fig.11 A schematic diagram of confusion matrices of original and improved knowledge distillation on a test set according to a specific embodiment of the present invention;

[0023] Fig.12 Schematic diagram of the worst student record when training two models according to a specific implementation mode of the present invention. DETAILED DESCRIPTION

[0024] The purpose of the present invention is to provide a multi-student elimination knowledge distillation method for image classification, which is used to screen the image appearance features of classified diseases to determine the disease type and propose corresponding prevention and control measures.

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] 1. Establishment of technical route:

[0027] The technical route of this method is as follows Figure 2 As shown in the figure, firstly, the dataset is collected, constructed and processed, then the original classification network is trained, and then the teacher network and student network are selected from the trained network. Then, the distillation experiment is performed using the original method and the method proposed in this paper respectively. By comparing the model effects, the advantages of the method proposed in this paper are analyzed. Finally, an ablation experiment is performed to evaluate the model trained by the method proposed in this paper.

[0028] 2. Introduction of knowledge distillation:

[0029] The rapid development of convolutional neural networks has led to a significant increase in network depth, which has significantly improved their ability to fit complex data patterns. However, this has been accompanied by a huge increase in the number of model parameters, which has made it difficult to deploy some hardware devices with limited computing power. In some scenarios that require high real-time performance of the model, such as real-time image processing and speech recognition, large deep learning models usually require a long inference time. Therefore, scholars have begun to explore ways to compress the model.

[0030] Currently, the main model compression technologies include quantization, pruning, and knowledge distillation. These methods help speed up the model's reasoning and meet the real-time requirements by reducing the number of model parameters or reducing its computational complexity.

[0031] Knowledge distillation is a model compression technology that has attracted widespread attention since it was proposed by Hinton in 2014. It aims to transfer the knowledge of complex large models to simplified small models, thereby achieving the purpose of model compression and reasoning acceleration. In the process of knowledge distillation, two main models are involved: a highly complex and high-performance teacher model, usually a large deep neural network; and a student model with a simpler structure and smaller size, usually a lightweight deep neural network. This model is like Figure 3 As shown in the figure, the teacher transfers the "knowledge" to the students through "distillation", and the students simulate and learn the teacher's fitting ability through "learning knowledge" to complete the training. Figure 4The simple knowledge distillation framework is presented. The process of knowledge distillation can be simply described as:

[0032] (1) Use the teacher model to predict the training data and obtain the soft label (Soft target), that is, the probability distribution of the output; use the soft label as an additional supervision signal to assist the student model in training;

[0033] (2) The student model learns the knowledge of the teacher model by minimizing the cross entropy loss function of the soft labels (usually combined with the student model’s own loss function);

[0034] (3) During the training process, in addition to soft labels, hard labels (hard target), i.e., true labels, are usually used for supervision to ensure that the student model can adapt to real data.

[0035] In knowledge distillation, hard labels and soft labels are two different probability distributions used to guide the training of the model. Figure 5 shows the difference between the two labels.

[0036] Hard target refers to a specific category label annotated by humans, usually a vector in the form of one-hot encoding, where one element is 1 and the rest are 0. In traditional supervised learning, the training goal of the model is to minimize the distance between the prediction and the true hard label, that is, the calculation of loss functions such as cross-entropy. Hard targets are more direct, and the training goal of the model is to predict results that are completely consistent with the true label.

[0037] A soft target is a probability distribution generated by a large, already trained model (usually a teacher model). This probability distribution is usually smoother and contains more information, not only about the category, but also about the relationship between different categories. Compared with hard labels, soft labels have higher information entropy. In knowledge distillation techniques, soft targets are a method used to guide the training of small models. The goal of the student model is to mimic the probability distribution output by the teacher model, rather than simply copying the deterministic labels of the teacher model. By minimizing the difference between the probability distribution predicted by the student model and the output of the teacher model, the student model is able to learn the deep feature representation of the teacher model, inherit the knowledge of the teacher model, and thus improve its performance.

[0038] In order to further emphasize the information in the soft target, a temperature parameter T is usually introduced into the softmax function. A higher temperature will make the probability distribution smoother and emphasize the subtle differences between categories. The formula of the softmax function with temperature is as follows:

[0039]

[0040] Under hard targets, error propagation is only for a single correct class, while under soft targets, error propagation covers all classes. This means that even on incorrect classes, the student model can learn useful information from the output of the teacher model, thereby improving the performance of the model more comprehensively. Figure 6 The results of softening the array [0.8, 0.06, 0.1, 0.02, 0.04] at temperatures T = 1, 2, 3, 4, and 5 are shown.

[0041] By using soft targets and appropriate distillation temperature, the knowledge distillation process can effectively transfer rich information from the teacher model to the student model, improving the generalization ability and overall performance of the student model. These technologies work together to enable the student model to not only learn the correct classification, but also to master more comprehensive and detailed category distinction information, thereby performing better in practical applications.

[0042] 3. Implementation of technical route:

[0043] 3.1. Dataset collection and processing:

[0044] The dataset studied in this invention is a disease grade image set of wheat fusarium wilt. Data collection is divided into two phases. The first phase was conducted at the experimental farm of Henan Agricultural University in Xuchang City on May 8, 2022, and the second phase was completed on May 12, 2023 at the experimental base of Henan Agricultural University in Yuanyang County, Xinxiang City. When collecting images, black cardboard was used as the background, and the side ears of wheat were photographed at an angle close to 90°. The shooting device used was an iPhone13, equipped with a 12-megapixel rear dual camera. The shooting distance was maintained between 20 and 30 centimeters. The shooting time was scheduled between 9 am and 6 pm, and the original resolution of the image was 3024×3024 pixels.

[0045] The collected images were classified according to the 1S01 standard of the Technical Specifications for Wheat Fusarium Scab Detection and Forecasting (GBT15796-2011) and under the guidance of the wheat fusarium scab resistance breeding experts of the Agricultural College. The disease level is divided into five levels: Level 0 represents completely healthy wheat ears, while Levels 1 to 4 are divided according to the proportion of the diseased area, which are 0-25%, 25%-50%, 50%-75% and 75%-100% respectively. Figure 7 Five categories of disease grade images in the dataset are shown.

[0046] A total of 1,089 wheat ear images with wheat scab were collected and classified according to the degree of disease, as follows: 124 healthy wheat ear images, 283 first-level disease images, 272 second-level disease images, 206 third-level disease images, and 204 fourth-level disease images. Based on these images, a wheat scab dataset was constructed. Figure 8 The specific distribution of the data set is shown in detail.

[0047] In order to enhance the diversity of the dataset and improve the generalization ability of the model, this paper uses a variety of data enhancement techniques, including Gaussian blur, contrast adjustment, brightness adjustment, sharpness adjustment, adding Gaussian noise, geometric transformation, and salt and pepper noise, and finally obtains 8297 enhanced images. After completing the data enhancement, the dataset is divided into training set, validation set, and test set in a ratio of 7:2:1 to facilitate model training, tuning, and performance evaluation.

[0048] 3.2. Introduction, comparison and analysis of classic classification models:

[0049] This paper focuses on several well-known convolutional neural network classification models, including ResNet18, MobileNetV2, DenseNet121 and ShuffleNetV2. The core of the study is to evaluate the performance and classification ability of these models on the wheat fusarium image dataset. Brief information is shown in Table 1.

[0050] Table 1 Basic information of classic classification models

[0051]

[0052] Since ResNet18, MobileNetV2, DenseNet121 and ShuffleNetV2 are all existing technologies, those skilled in the art are already familiar with and use them, and no further details are given.

[0053] 3.3. Experimental results and analysis of classic classification model:

[0054] 3.3.1. Experimental setup and parameter settings:

[0055] The experiment was completed under the Ubuntu 20.04 64-bit operating system, using the Pytorch 1.11.0 open source deep learning framework that supports GPU acceleration, and the CUDA version is 11.3. The programming language used is Python 3.8.10. The hardware used is an AMD EPYC 9654 processor with 96 cores, 60GB of memory, and a graphics card of RTX 4090 with 24GB of video memory.

[0056] The experiment selected ResNet18, MobileNetV2, DenseNet121 and ShuffleNetV2 for research. The training scheme is as follows: the initial learning rate of the model is 0.001, and it decays to 0.1 times of the original every 30 rounds. The batch size is 16, and the total iteration is 100 rounds. The optimizer selects the stochastic gradient descent method.

[0057] 3.3.2. Recognition results of different models:

[0058] In this experiment, the training set and validation set data were used to train the model. The training set was used to optimize and adjust the model parameters, while the validation set was used to evaluate the model's fitting ability. After the model training was completed, in order to finally evaluate the performance of the model on unseen data, the experiment used an independent test set for testing. Fig. 9 The changes in loss value and accuracy of each model on the validation set during training are shown. Table 2 shows the performance of the model on the test set.

[0059] Table 2 Performance of each classification model on the test set

[0060]

[0061]

[0062] By observing the above accuracy and loss change curves, we can know that the model adopted has fully converged within 100 rounds and the accuracy has stabilized. It can be clearly seen from the chart that DenseNet121 shows the highest accuracy and the lowest loss value, thanks to its dense connection structure and feature reuse mechanism.

[0063] Overall, all classification models have low recall rates for categories 3 and 4. This is because the leaf infections in these two categories account for 25-50% and 50-75% respectively, and many samples have similar features, making it difficult to distinguish them effectively. It is worth noting that ShuffleNetV2 has a very low recall rate for category 4, which indicates that the network has not learned the features of category 4 very well. According to the distribution of the highest recall values ​​for each category, the models have different recognition capabilities for different categories. This shows that the features learned by the model differ between categories, so not all categories can be optimized to the same degree. This phenomenon also shows that even large-scale models may not perform well in the recognition of all categories.

[0064] 4. Identification of wheat scab based on original knowledge distillation:

[0065] 4.1. Selection of teacher and student models:

[0066] Generally speaking, the teacher model is larger and more complex than the student model, and performs well on a given task. Among the models explored in the previous article, ResNet18 (R18) has the largest model size, the most parameters, and multiplication and addition operations, and is not suitable for deployment on hardware devices with relatively scarce computing resources. However, it performs well in the wheat fusarium classification task, so it is suitable as a teacher model. Although DenseNet121 (D121) has a smaller model size and parameter count than ResNet18, its recognition accuracy is higher than ResNet18. In addition, due to its more multiplication and addition operations, the complexity and computational complexity of the model have increased, so DenseNet121 is considered as a teacher model and a student model respectively. MobileNetV2 (MV2) and ShuffleNetV2 (SV2) have smaller model sizes, fewer parameters, and lower recognition accuracy, and are suitable as student models.

[0067] 4.2. Original knowledge distillation experimental results and analysis:

[0068] 4.2.1. Experimental setup and parameter settings:

[0069] The experimental environment setting is the same as that in Section 3.3. During the distillation process, there is only one teacher model and one student model at a time, the distillation temperature T is 7, and the distillation loss, mutual learning loss, and true label loss each account for 1 / 3.

[0070] 4.2.2. Analysis of the effect of the original distillation scheme:

[0071] The baseline distillation structure obtained by distilling the student model from the selected teacher model is shown in Table 3.

[0072] Table 3 Benchmark knowledge distillation results

[0073]

[0074] The results show that knowledge distillation can not only allow the student model to learn the knowledge of the teacher model, thereby achieving the task of model compression, but also improve its own performance to a certain extent and have a better generalization effect. It is worth noting that the student model MobileNetV2 has achieved an improvement of 1.3 and 2.01 percentage points after distillation to ResNet18 and DenseNet121, respectively, which fully demonstrates that the model after knowledge distillation is more robust in image classification tasks. DenseNet121 has better classification performance than ResNet18. In the results of the distillation tasks ResNet18-ShuffleNetV2 and DenseNet-ShuffleNet, it can be seen that the former has a greater improvement in the accuracy of the student model. In the results of ResNet18-MobilenetV2 and DenseNet121-MobileNetV2, it can be observed that the latter has a greater improvement in the accuracy of the student model. This shows that a better teacher model is not necessarily better than a student model distilled from a poor teacher model.

[0075] In DenseNet121-ShuffleNetV2, it can be observed that the overall accuracy of the student model after distillation has not improved. However, it should be noted that the network's classification ability for specific categories has been greatly improved. Specifically, the study found that for category 4, the classification accuracy of the student model after distillation increased by 5.48 percentage points, which is a significant improvement. Table 4 shows the classification accuracy of each category and the improvement relative to the undistilled model in detail.

[0076] Table 4 ShuffleNetV2 classification performance changes

[0077]

[0078] The original DenseNet121 model performs poorly in the recognition of category 4, with a recall rate much lower than that of the other categories. The improved model improves the recall rate of category 4 while slightly losing the accuracy of other categories. However, despite the slight performance degradation on other categories, the overall accuracy remains unchanged. This shows that knowledge distillation effectively improves the robustness and stability of the overall performance while improving the recognition ability of specific categories. This also further proves the role of soft label loss in improving the generalization performance of the model.

[0079] 5. Wheat scab image recognition based on elimination-based multi-student knowledge distillation:

[0080] 5.1. Improving the original knowledge distillation model strategy:

[0081] In traditional knowledge distillation, there is only one student model and one teacher model, and the size of the student model is usually smaller than that of the teacher model. The source of knowledge acquired by the student model is the real label and the softened teacher output. According to the conclusion of the previous article, different models can capture different image features. Based on this, this paper proposes a distillation method based on the traditional knowledge distillation framework. The structure of the distillation is as follows: Fig.10 As shown:

[0082] In this structure, multiple identical or different student models are added to participate in the training at the same time, so that they can learn from each other and acquire knowledge from the teacher model and the true label. At the same time, the proposed method eliminates some of them in the prescribed training stage and leaves the best performing model.

[0083] The loss function consists of three parts: label loss, distillation loss, and mutual learning loss. First, the teacher model outputs a smoother soft label through the Softmax function adjusted by the temperature parameter T. Then, the student model 1 uses the same temperature parameter T to perform the Softmax operation to obtain the soft label of its own output. Then the distillation loss is obtained by calculating the KL divergence of the soft labels of the teacher model and the student model 1. Then model 1 performs the Softmax operation with the same temperature as other models and calculates the KL divergence. After obtaining the average, the mutual learning loss is obtained. Finally, the true label and the softened student output are calculated to obtain the label loss.

[0084] As the training progresses, each model can not only learn the knowledge of the teacher model and simulate the distribution of the teacher model, but also learn the distribution of other student models to improve its own generalization ability. The model will also adjust the learned distribution in time according to the real label. When the specified number of rounds is reached, the framework eliminates one or more student models based on the performance of these student models. The metric for elimination is the weighted sum of their accuracy and loss values ​​calculated on the validation set. The model with the smallest weighted sum has poor overall performance and will affect the learning of other networks. Therefore, it will be eliminated and no longer participate in training, leaving the remaining models to continue learning from each other. In the end, the model outputs one or a small number of student models as the result of training.

[0085] 5.2. Experimental results and analysis of improved model strategy:

[0086] 5.2.1. Experimental setup and parameter settings:

[0087] In the improved knowledge distillation scheme, one trained teacher model and three identical or different student models are selected for distillation. For example, one trained DenseNet121 is selected to distill three ShuffleNetV2s and finally retained by selection (D121-SV2*3-1, three SV2 models, and one is retained in the end). The total loss consists of distillation loss, label loss, and mutual learning loss, and each part of the loss accounts for 1 / 3 of the total loss. Models with poor performance will be selectively eliminated at 30 and 60 rounds, and finally one student model will be left to complete the last 40 rounds of training. The accuracy and loss value of the model are defined as half of the weighted difference as the score. The model with the lowest score is regarded as the worst model and is eliminated, and then no longer participates in training.

[0088] 5.2.2. Analysis of the effect of improved distillation scheme:

[0089] The training results of the elimination-type multi-student knowledge distillation are evaluated on the test set, as shown in Table 5. As can be seen from the table, the recognition accuracy has been improved by using the proposed method to perform elimination-type distillation on multiple models. Among them, the original classification accuracy of SV2 is low, and the maximum recognition accuracy can be improved by 1.41 percentage points after elimination-type distillation using D121. For D121 with a high original classification accuracy, elimination-type distillation using R18 can also improve it by 0.36 percentage points. It is worth noting that when D121, SV2, and MV2 are put together for distillation, the model left in the end is D121. This phenomenon shows that models with strong classification capabilities are not easily eliminated.

[0090] Table 5. Training results of the elimination multi-student model

[0091]

[0092] In order to further compare the superiority of the proposed distillation method over the original distillation method, this paper selects the results of the proposed method D121-SV2*3-1 and the original distillation method D121-SV2 to display the confusion matrix on the test set. The confusion matrix is ​​an important evaluation indicator in the classification model. The results of the model prediction are displayed in the matrix to compare the difference between the true value and the predicted value of the model. Fig.11 .

[0093] exist Fig.11 In the figure, the horizontal axis is the predicted disease category, the vertical axis is the true category of wheat scab, the numbers on the diagonal represent the prediction accuracy of the corresponding true category, and the numbers in the remaining positions represent the proportion of the corresponding category prediction errors. Fig.11(a) It can be seen that the D121 model after using the original distillation method is easy to misclassify between category 3 and category 4. The main reason is that the infection degree of the two categories of diseases is somewhat similar, which causes interference to the model. Fig.11 (b) It can be seen that the improved distillation model has improved the accuracy of identifying the two types of diseases to a certain extent, alleviating the situation where the diseases are misclassified. This once again proves that the elimination distillation method has excellent performance and can increase the probability of correct identification of similar disease features.

[0094] 5.3 Ablation Experiment

[0095] In order to verify the impact of the proposed method on the model, an ablation experiment is conducted on the elimination-type multi-student knowledge distillation, and the experimental environment is the same as before. In the experiment, the same number of students as the proposed method, that is, 3 student models, are added each time, but no elimination is performed and they are retained until the end of training. The control experiment performs elimination-type knowledge distillation on the same 3 student models under the same teacher model. The final retained model is compared with the proposed distillation method, and the results are shown in Table 6.

[0096] Table 6 Ablation experiment results

[0097]

[0098] From Table 6, we can see that eliminating poorly performing student models can effectively improve the quality of the remaining student models. To further illustrate the role of the elimination mechanism, we take the three SV2 models distilled from R18 in the table as an example to show the records and elimination process of the worst students in the two schemes during training. Fig.12 shown.

[0099] In this experiment, the definition of the worst is the same as before, that is, the accuracy and loss value of the model each account for half of the weights and the difference is used as the score. The model with the lowest score is considered the worst model. Fig.12 (a) The worst performing model in each round is recorded without eliminating any model. Fig.12(b) records the elimination of the worst model in the specified number of rounds. From the ablation experiment results, it can be seen that as the training progresses, the same student model shows different classification capabilities. If the worst student model is retained, although it can learn the knowledge of the good student model, due to its small number and low weight, the remaining models will learn more knowledge from the poorer models and converge to the worst model, thereby lowering the fitting ability of the entire student group and reducing the overall performance of the student model cluster. In the elimination distillation, the worst student model is eliminated once in 30 rounds, leaving two student models with stronger fitting ability to learn from each other and promote each other's convergence. At 60 rounds, the models have fully learned each other's knowledge, and the poorer model of the two is eliminated again, leaving only one model to learn the teacher and the true label knowledge to complete the final convergence. This shows that eliminating the worst model promotes the mutual learning of models, and the method proposed in this paper can achieve better results than the original distillation.

[0100] In summary, with the continuous advancement of deep learning technology, its application in the agricultural field has received increasing attention, especially in the identification of crop diseases. Compared with traditional manual feature extraction methods, convolutional neural networks have significant advantages in automatically extracting image features, which enables it to provide faster and more accurate recognition results in crop disease image recognition tasks. Using CNN, farmers can quickly identify diseases and take preventive measures in time, thereby effectively improving the yield and quality of crops. In addition, other applications of deep learning technology in agriculture, such as intelligent monitoring and management of crop growth environments, also show their value. Real-time monitoring of crop growth conditions through intelligent systems can optimize resource allocation, reduce unnecessary waste, and improve the efficiency of agricultural production.

[0101] Although the present invention has been illustrated and described with respect to the preferred embodiments, it will be appreciated by those skilled in the art that various changes and modifications may be made to the present invention without departing from the scope of the present invention as defined by the claims.

Claims

1. A multi-student elimination knowledge distillation method for image classification, characterized in that: First, the data set is collected, constructed and processed, and then the original classification network is trained. Next, a teacher network and a student network are selected from the trained network. Then, the teacher network and the student network are distilled and learned using the original knowledge distillation method. Finally, the distillation mode of the original knowledge distillation method is improved.

2. The multi-student elimination knowledge distillation method for image classification according to claim 1, characterized in that: The collection, construction and processing of the data set specifically include: When collecting images, use black cardboard as the background, shoot the side ears of wheat at an angle close to 90°, and keep the shooting distance between 20 and 30 cm; The collected images were classified according to the technical specifications and standards for wheat fusarium rust detection and forecasting, and a wheat fusarium rust image dataset was constructed; To enhance the diversity of the data set and improve the generalization ability of the model, data enhancement techniques are used, including Gaussian blur, contrast adjustment, brightness adjustment, sharpness adjustment, adding Gaussian noise, geometric transformation, and salt and pepper noise, to enhance the image.

3. The multi-student elimination knowledge distillation method for image classification according to claim 1, characterized in that: The original classification networks include ResNet18, MobileNetV2, DenseNet121 and ShuffleNetV2.

4. The multi-student elimination knowledge distillation method for image classification according to claim 3, characterized in that: The selection of the teacher network and the student network specifically includes: ResNet18 has the largest model size, the most parameters, and the most multiplication and addition operations, and is not suitable for deployment on hardware devices with relatively scarce computing resources. However, it performs well in the wheat fusarium classification task, and is therefore suitable as a teacher model. Although the model size and parameter amount of DenseNet121 are smaller than those of ResNet18, its recognition accuracy is higher than that of ResNet18. In addition, due to its more multiplication and addition operations, the complexity and calculation amount of the model are increased. Therefore, DenseNet121 is considered as the teacher model and student model respectively; MobileNetV2 and ShuffleNetV2 have smaller model sizes, fewer parameters, and lower recognition accuracy, making them suitable as student models.

5. The multi-student elimination knowledge distillation method for image classification according to claim 4, characterized in that: The improvement of the distillation mode of the original knowledge distillation method specifically includes: Join multiple identical or different student models to participate in training at the same time, so that they can learn from each other and acquire knowledge from the teacher model and the true label; At the same time, the proposed method eliminates some of them in the specified training stage and keeps the best performing model.

6. The multi-student elimination knowledge distillation method for image classification according to claim 5, characterized in that: The elimination method specifically includes: first calculating the weighted sum of the accuracy and loss value of the student models participating in the training on the validation set. The student model with the smallest weighted sum has poor overall performance, which will affect the learning of other networks. As the training proceeds, each student model can not only learn the knowledge of the teacher model and simulate the distribution of the teacher model, but also learn the distribution of other student models to improve its own generalization ability. The student model will also adjust the learned distribution in time according to the real label. When the specified number of rounds is reached, one or more student models are eliminated according to the performance of these student models, leaving the remaining student models to continue to learn from each other. Finally, the model outputs one or a small number of student models as the result of training.

7. The multi-student elimination knowledge distillation method for image classification according to claim 6, characterized in that: The loss value, i.e., the loss function, specifically includes label loss, distillation loss, and mutual learning loss. The calculation process of the loss function specifically includes: First, the teacher model outputs smoother soft labels through the Softmax function adjusted by the temperature parameter T; Next, a student model 1 in the student model performs a Softmax operation using the same temperature parameter T to obtain a soft label of its own output, and then calculates the KL divergence of the soft labels of the teacher model and the student model 1 to obtain the distillation loss; Then, the student model 1 performs the Softmax operation of the same temperature with the other student models and calculates the KL divergence, and obtains the mutual learning loss after obtaining the mean; Finally, the true label and the softened student model output are calculated to obtain the label loss.

Images