A lightweight image classification neural network architecture system based on knowledge distillation

Abstract

This invention belongs to the field of image classification neural network technology and discloses a lightweight image classification neural network architecture system based on knowledge distillation. The system includes: a dataset preprocessing module, a teacher model preprocessing module, a subnet model construction module, a personalized training module, a module for calculating the true loss, a module for calculating the distillation loss, a module for calculating the total loss, and a module for constructing the student model. This invention provides a lightweight neural network architecture system based on knowledge distillation. Leveraging the advantages of knowledge distillation in extracting weights from the teacher model, a novel training mode is designed: this invention combines knowledge distillation of the teacher and student models with deep compression methods. The proposed lightweight network system can be used to train various complex image classification neural networks to improve network performance.

Description

Technical Field

[0001] This invention belongs to the field of image classification neural network technology, and particularly relates to a lightweight image classification neural network architecture system based on knowledge distillation. Background Technology

[0002] With the development of deep learning technology, neural networks have been widely used in computer vision, especially in image classification tasks. Image classification is a core problem in computer vision, aiming to categorize input images into known classes. Compared to traditional manual feature extraction, neural networks can automatically learn information such as edges, textures, shapes, and semantics in images through hierarchical structures. They can also capture complex patterns and regularities in image data, acquiring robust features through end-to-end learning, giving the model excellent generalization capabilities. This technology has wide-ranging applications in many fields, including identifying crop pests and diseases, monitoring livestock growth status, recognizing faces and other targets in public safety, airport, and bank security systems, and playing a crucial role in various scenarios such as terrain surveying and water condition assessment using drones.

[0003] Currently, many methods exist for image classification using neural networks. However, in practical deployments to end devices, insufficient available computational resources and limited training data make it difficult to ensure robust operation of the developed image classification models on resource-constrained devices (such as mobile devices and embedded systems). Knowledge distillation (KD) is a promising method for building lightweight networks. It transfers knowledge from a large, complex teacher model to a smaller student model, allowing the student model to maintain high performance while significantly reducing model parameters and computational cost. This process not only effectively improves the generalization ability of the student model but also alleviates the problem of insufficient training data to some extent.

[0004] In "An Image Classification Method Based on Ensemble Knowledge Distillation (Invention Patent No. CN112199535B)," this invention proposes a teacher model that jointly guides a student model through three stages to further improve the performance of the student model, ensuring image classification accuracy while improving model response speed. "An Image Classification Model Training Method Based on Ensemble and Knowledge Distillation (Invention Patent No. CN113822373B)" proposes a method for training image classification models based on ensemble and knowledge distillation for imbalanced data. This method first trains multiple models of different depths based on the long-tail effect of the dataset, then uses these models to extract features from the training data and integrates these deep models in the feature space. Finally, knowledge distillation is used to fuse the ensemble models into a lower-complexity inference model, reducing inference complexity. The invention, "A Compression Method and Platform for Pre-trained Language Models Based on Knowledge Distillation (Invention Authorization No.: CN111767711B)," proposes to first design a universal knowledge distillation strategy for feature transfer. In the process of knowledge transfer in the classroom model, it focuses on the feature representation ability of small samples in the intermediate layer of the teacher model and uses these features to guide the feature mapping of the student model to approximate the teacher model. Finally, it constructs a self-attention cross-knowledge distillation method based on the self-attention distribution detection function of the teacher model, realizing the automatic compression of pre-trained language models for multi-task applications.

[0005] The existing knowledge distillation technologies, represented by the three patents mentioned above, all distill the knowledge from the integrated model into a single model. However, the distilled model is still quite large and does not support large-scale deployment on mobile devices. Among them, CN113822373B performs feature extraction during the training of the image classification model, which solves the problem of performance degradation of the student model. However, the calculation process is complex and the model response time is long, resulting in poor performance of the integrated model.

[0006] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0007] (1) Traditional knowledge distillation techniques distill the knowledge of the integrated model into a single model. However, they often use the method of averaging the teacher model parameters to the student end, and the integrated model often only undergoes one round of pruning, ignoring the differences in the prediction results of different teacher models on the same data. The parameters learned by the student model are flattened, resulting in the model still having a large number of parameters after distillation, which makes the model perform poorly in mobile deployment tasks.

[0008] (2) Existing techniques for enhancing the diversity of student models mainly involve extracting data sample features in advance or improving the accuracy of student models by enhancing the expression of small samples. However, these techniques generally suffer from drawbacks such as large data feature dimensions that the model needs to learn during the distillation process, high computational complexity, and low model training efficiency. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention provides a lightweight image classification neural network architecture system based on knowledge distillation.

[0010] This invention is implemented as follows: a lightweight image classification neural network architecture system based on knowledge distillation includes:

[0011] The module includes: dataset preprocessing module, teacher model preprocessing module, subnet model construction module, personalized training module, real loss calculation module, distillation loss calculation module, total loss calculation module, and student model construction module.

[0012] The dataset preprocessing module, connected to the teacher model preprocessing module, is used to perform image transformations on the data in the dataset, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding, to generate an image training set.

[0013] The teacher model preprocessing module, connected to the dataset preprocessing module and the subnet model construction module, is used to train the teacher model with a large amount of data, and the teacher model ultimately generates soft labels.

[0014] The subnet model construction module is connected to the teacher model preprocessing module, personalized training module, real loss calculation module, distillation loss calculation module, total loss calculation module, and student model construction module. It is used to input the image training set and the soft labels obtained by the teacher pre-training into the student subnet model, output the image classification results of the student subnet, and save the weights of the student subnet.

[0015] The personalized training module, connected to the subnet model building module, is used to introduce a rank factor to transfer teacher knowledge to the subnet model. During training, adversarial perturbations are added to the input image, the loss value of the student subnet is calculated, and the subnet shared parameters are updated based on the loss value to improve the robustness of the model.

[0016] The module for calculating the true loss is connected to the module for building the subnet model. It is used to compare the images reclassified by the subnet model with the true image labels and calculate the cross-entropy loss between the true labels and the output of the subnet model.

[0017] The module for calculating distillation loss is connected to the module for building subnet models. It is used to compare the category probability distribution obtained by the subnet model reclassification with the soft labels of the teacher model and calculate the knowledge distillation loss between the soft labels and the predictions of the subnet model.

[0018] The total loss calculation module, connected to the subnet model construction module, is used to calculate the total loss by weighting the two losses. The total loss is used as the network training error, and the final subnet model is obtained after iterative training.

[0019] The student model building module is connected to the subnet model building module. It is used to compress the knowledge of the subnet model into a single model. The model is then retrained using the above image training set. Simultaneously, pruning is performed and the pruned weights are quantized. The quantized weights are further compressed using Huffman coding to generate the student model.

[0020] Furthermore, the teacher model includes a pre-trained ResNet-50 network.

[0021] Furthermore, the subnet model adopts the MobileNetV2 architecture; the forward propagation process of the subnet model includes the following steps: the input image is subjected to feature extraction through several convolutional layers; a low-dimensional feature map is generated after several convolution and pooling operations; the feature map is subjected to a global average pooling layer to obtain global features; the global features are used to generate output through a fully connected layer.

[0022] Furthermore, the knowledge distillation process includes the following steps: Training phase: The teacher model and the subnet model receive the input image respectively and generate the teacher output and the subnet output; the teacher output is used as a soft label, and the error between the subnet output and the teacher output is calculated by KL divergence; the training objective of the subnet model is to minimize the weighted sum of the soft label error and the true label error; Inference phase: Only the trained student model is used for the image classification task.

[0023] Furthermore, the soft label of the teacher model is a label probability value calculated by a temperature-sensitive softmax function, where the temperature parameter T is used to smooth the probability distribution; the specific representation of the probability value is as follows:

[0024]

[0025] Among them, z i These are the logits output by the teacher model, where T is the distillation temperature;

[0026] As a preferred technical solution, the total loss is calculated by proportionally dividing the cross-entropy loss and distillation loss, as follows:

[0027] L=α·T 2 ·H(q T ,p T) +(1-α)·H(y,p)

[0028] Where H represents the cross-entropy loss, q T and p T These are the output probability distributions of the teacher model and the subnet model at temperature T, respectively. y is the hard label, and α is the weight used to balance the losses of the soft label and the hard label.

[0029] Furthermore, the student model is obtained by knowledge compression from the subnet model, including three important steps: weight pruning, weight quantization, and weight compression.

[0030] The weight pruning part requires transferring the knowledge of the subnet model to the student model, inputting the training dataset into the student model, and dynamically removing connections with smaller weights during the training process.

[0031] The weighted pruning is implemented through dynamic pruning techniques, including dynamic evaluation of parameter importance, selection of pruning strategies, setting of pruning thresholds, and periodic pruning.

[0032] Dynamic evaluation of parameter importance refers to calculating the parameter importance score based on the absolute value of the model parameters or the absolute value of the product of the parameter values ​​and the gradient in the initial stage.

[0033] The selection of pruning strategies involves determining which parameters or layers need to be pruned, and dynamically updating and adjusting the importance score and uncertainty value based on the global step count;

[0034] The pruning threshold setting is based on the needs of different stages in the training process. The pruning intensity is dynamically adjusted by setting different thresholds for the warm-up step, the middle stage of training, and the end stage.

[0035] Periodic pruning refers to performing pruning operations periodically at different stages of training, with the decision on whether to prune based on a set number of steps.

[0036] The weight quantization part includes two parts: weight quantization and shared weight.

[0037] Another objective of this invention is to provide a lightweight image classification neural network architecture method based on knowledge distillation, comprising:

[0038] Step 1: The dataset preprocessing module performs image transformations on the data in the dataset, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding, to generate an image training set; the teacher model preprocessing module trains the teacher model with a large amount of data, and the teacher model finally generates soft labels.

[0039] Step 2: By constructing the subnet model module, the image training set and the soft labels obtained by the teacher pre-training are respectively input into the student subnet model, the image classification results of the student subnet are output and the weights of the student subnet are saved;

[0040] Step 3: The teacher's knowledge is transferred to the subnet model by introducing a rank factor through the personalized training module. During the training process, adversarial perturbations are added to the input image, the loss value of the student subnet is calculated, and the subnet shared parameters are updated according to the loss value to improve the robustness of the model.

[0041] Step 4: By using the real loss calculation module, the images reclassified by the subnet model are compared with the real image labels, and the cross-entropy loss between the real labels and the subnet model output is calculated.

[0042] Step 5: By using the distillation loss calculation module, the category probability distribution obtained by the subnet model reclassification is compared with the soft labels of the teacher model, and the knowledge distillation loss between the soft labels and the subnet model prediction is calculated.

[0043] Step 6: The total loss is calculated by weighting the two losses using the total loss calculation module. The total loss is used as the network training error. After updating and iterating the training, the final subnet model is obtained.

[0044] Step 7: By constructing a student model module, the knowledge of the subnet model is compressed into a single model. The image training set mentioned above is input and iteratively trained again. At the same time, pruning is performed and the pruned weights are quantized. The quantized weights are further compressed using Huffman coding to generate the student model.

[0045] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the lightweight image classification neural network architecture method based on knowledge distillation.

[0046] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the lightweight image classification neural network architecture method based on knowledge distillation.

[0047] Another objective of this invention is to provide an information data processing terminal for implementing the lightweight image classification neural network architecture system based on knowledge distillation.

[0048] Based on the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solution to be protected by this invention from the following aspects:

[0049] First, addressing the technical problems existing in the prior art and the difficulty in solving them, this paper closely analyzes, in conjunction with the technical solution to be protected by this invention and the results and data obtained during the research and development process, how the technical solution of this invention solves the technical problems, and the inventive technical effects brought about by solving these problems. The specific description is as follows:

[0050] This invention provides a lightweight neural network architecture system based on knowledge distillation. Leveraging the superiority of knowledge distillation in extracting weights for teacher models, a novel training mode is designed: multiple teacher models are pre-trained on the same dataset; distillation introduces a rank factor to shrink the search space and adds perturbations, improving training efficiency while maintaining model performance, resulting in a subnet model with reduced parameter size. The subnet model is then compressed to obtain the student model, and dynamic pruning is performed during the iterative training of the student model. Through the combined effect of these two methods, model training efficiency is effectively improved and parameter size is reduced, enhancing the quality and efficiency of image classification. This invention combines knowledge distillation of teacher and student models with deep compression methods, and the proposed system for constructing lightweight networks can be used to train various complex image classification neural networks to improve network performance.

[0051] (1) The present invention introduces a rank factor combined with a residual network for knowledge distillation to address the problem of large search space and low training efficiency when deep neural networks extract target features. In practical applications, image training data often contains high-dimensional features, which often leads to slow training of the teacher model. Therefore, the introduction of a rank factor to fully extract image features and residual convolution aims to help improve the training efficiency of the model.

[0052] (2) The present invention adds perturbation during training to solve the problem of performance degradation due to diversity loss when the subnet learns from the teacher model (for example, the teacher model has three soft labels, but the subnet model only learns one soft label that appears more often and ignores the other results), so that the subnet model can fully learn the differences between the knowledge of different teacher networks, reduce the probability of the subnet making positive and negative sample classification errors when learning image features, and improve the accuracy and robustness of the student model.

[0053] (3) This invention adds deep compression to the traditional distillation process, further compressing the parameter scale to generate a personalized, lightweight student model that is easy to deploy on terminals. Compared with traditional weight processing methods, this invention does not directly observe the weight values ​​but measures the importance of the weights. By setting a mask, it continuously attempts to subtract network connections with less influence during training, thereby reducing the number of network connections, refining the subnet model, and effectively saving computing resources.

[0054] The positive effects of this invention are as follows: it preserves important information from the teacher model through knowledge distillation, improving the accuracy and generalization ability of the student model; it also enhances model accuracy and robustness under complex classification problems and noise interference; and it significantly reduces the number of model parameters through pruning techniques, thus reducing computational resource consumption and inference time. This makes the generated student model both lightweight and efficient, suitable for resource-constrained environments such as mobile devices and embedded systems. Furthermore, this method can achieve rapid model training and deployment while ensuring model performance, and has broad application prospects.

[0055] Secondly, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0056] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0057] In the technical solution of this invention, by deploying the image recognition model to mobile devices, such as smartphones, farmers can monitor the growth status of crops in greenhouses in real time and accurately grasp key operation time nodes (such as pest control, watering, etc.), thereby increasing crop yield.

[0058] Taking the monitoring and water spraying facilities inside the greenhouse as an example, assuming each farmer owns 10 mu of land, the investment required to deploy this technical solution is 50,000 yuan, which specifically includes the following:

[0059] 1. Equipment cost:

[0060] (1) Smartphones and related applications: approximately RMB 10,000.

[0061] (2) Monitoring equipment and sensors: approximately RMB 20,000.

[0062] (3) Sprinkler system and automatic control system: approximately 20,000 yuan.

[0063] 2. Operating costs:

[0064] (1) Data traffic and daily maintenance costs: approximately RMB 10,000 per year.

[0065] By utilizing data observed through a platform equipped with a personalized lightweight image classification neural network and employing precise crop management techniques, crop yields are projected to increase by 30%. Based on a traditional planting model yield of 15,000 yuan per mu per year, this could increase to 19,500 yuan per mu after the yield increase. The total increased yield for 10 mu of land would be 45,000 yuan.

[0066] Comprehensive benefit analysis:

[0067] With the aforementioned investment, the increased production revenue within one year can reach 45,000 yuan. After deducting operating costs of 10,000 yuan, the net increased production revenue is 35,000 yuan. Calculated over a two-year period, the total expected revenue can reach 90,000 yuan. After deducting the initial investment of 50,000 yuan, the net profit is 40,000 yuan.

[0068] By promoting this technology on a large scale, domestic farmers will be able to achieve higher yields on limited land resources, drive the modernization of agriculture, and promote the sustainable development of the agricultural economy.

[0069] (2) The technical solution of this invention solves a long-standing technical problem that people have long desired to solve but have never been able to: namely, reducing the parameter size of the model while ensuring the performance of the student model, and constructing a personalized lightweight neural network system. This reduces the model parameter size while ensuring the performance of the student model, enabling the deployment of large models on mobile devices and promoting the large-scale application of research results in neural networks.

[0070] Compared to traditional multi-teacher student models, this invention allows the subnet to fully learn features extracted by different teachers, effectively compensating for the diversity loss that occurs during the learning process of multi-teacher models. Furthermore, this process achieves a smaller computational scale by randomly adding perturbations. It also incorporates deep compression techniques on top of traditional knowledge distillation, further compressing model parameters to generate a personalized, lightweight student model that is easy to deploy on terminals. This effectively solves the problem of balancing the size and performance of deep neural network models.

[0071] Third, this invention addresses the problems of low efficiency and insufficient accuracy in the training and inference processes of existing lightweight image classification neural network models. Traditional methods typically rely on large deep neural network models, which, while performing well in terms of accuracy, are extremely demanding in terms of computational resources and storage space, making them unsuitable for deployment on resource-constrained devices. To address this, this invention introduces knowledge distillation technology to effectively transfer the rich knowledge of the teacher model to the student model, thereby achieving efficient training of the lightweight model and significantly reducing computational costs during inference.

[0072] Furthermore, this invention proposes a modular system architecture that systematically solves the challenges of loss calculation and model compression in the knowledge transfer process through the collaborative work of multiple modules, including data preprocessing, teacher model training, student subnet model construction and training, loss calculation, and student model compression. This modular design not only improves the system's flexibility but also enhances the overall model performance through optimized coordination among modules. In particular, the robustness of the student model is significantly improved after introducing adversarial perturbations and a rank factor, further enhancing the model's stability and generalization ability in complex environments.

[0073] Through the implementation of this invention, the training efficiency of lightweight image classification neural networks has been significantly improved, while maintaining model accuracy and greatly reducing the computational resource requirements and storage space usage. This advancement enables the more widespread deployment of efficient image classification models in practical applications, especially in resource-constrained scenarios such as mobile devices and embedded systems, showing great application potential. Simultaneously, the model quantization and compression techniques significantly reduce latency during the inference phase, enabling applications with high real-time requirements to also be met.

[0074] Overall, this invention achieves significant technological advancements by addressing the limitations of existing technologies. Through innovative knowledge distillation strategies and modular design, this invention not only improves the training and inference efficiency of lightweight neural network models but also achieves remarkable technological breakthroughs in multiple aspects such as model compression, robustness enhancement, and computational resource optimization, providing a more efficient and practical solution for the field of image classification. Attached Figure Description

[0075] Figure 1 This is a block diagram of a lightweight image classification neural network architecture system based on knowledge distillation provided in an embodiment of the present invention.

[0076] Figure 2 This is a flowchart of a lightweight image classification neural network architecture system based on knowledge distillation provided in an embodiment of the present invention.

[0077] Figure 3 This is a detailed flowchart of the lightweight image classification neural network architecture based on knowledge distillation provided in this embodiment of the invention.

[0078] Figure 4 This is a diagram illustrating the lightweight image classification neural network construction technology based on knowledge distillation provided in this embodiment of the invention.

[0079] Figure 5 This is a diagram of the multi-teacher model distillation algorithm for generating highly robust and highly generalizable subnet models provided in this embodiment of the invention.

[0080] Figure 6 This is a hierarchical diagram of a lightweight image classification neural network structure based on knowledge distillation provided in an embodiment of the present invention.

[0081] Figure 7 This is a diagram illustrating the algorithm for forming a student model through deep compression of a subnet model, as provided in an embodiment of the present invention.

[0082] Figure 8 This is a schematic diagram of the pruned network structure provided in an embodiment of the present invention.

[0083] Figure 9This is a schematic diagram of the system interface and a predicted effect derivation diagram provided in the embodiments of the present invention.

[0084] Figure 1 The modules are as follows: 1. Dataset preprocessing module; 2. Teacher model preprocessing module; 3. Subnet model construction module; 4. Personalized training module; 5. Calculation of true loss module; 6. Calculation of distillation loss module; 7. Calculation of total loss module; 8. Student model construction module. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0086] like Figure 1 As shown, the lightweight image classification neural network architecture system based on knowledge distillation provided in this embodiment of the invention includes:

[0087] Dataset preprocessing module 1, teacher model preprocessing module 2, subnet model construction module 3, personalized training module 4, real loss calculation module 5, distillation loss calculation module 6, total loss calculation module 7, student model construction module 8;

[0088] Dataset preprocessing module 1, connected to teacher model preprocessing module 2, is used to perform image transformations on the data in the dataset, including transformation chain implementation, image scaling, horizontal flipping, random cropping, padding, etc., to generate an image training set.

[0089] Teacher model preprocessing module 2, connected to dataset preprocessing module 1 and subnet model construction module 3, is used to train the teacher model with a large amount of data, and finally the teacher model generates soft labels;

[0090] The subnet model construction module 3 is connected to the teacher model preprocessing module 2, the personalized training module 4, the real loss calculation module 5, the distillation loss calculation module 6, the total loss calculation module 7, and the student model construction module 8. It is used to input the image training set and the soft labels obtained by the teacher pre-training into the student subnet model, output the image classification results of the student subnet, and save the weights of the student subnet.

[0091] Personalized training module 4, connected to subnet model construction module 3, is used to introduce a rank factor to transfer teacher knowledge to the subnet model. During training, adversarial perturbations are added to the input image, the loss value of the student subnet is calculated, and the subnet shared parameters are updated based on the loss value to improve the robustness of the model.

[0092] The module 5 for calculating the true loss is connected to the module 3 for building the subnet model. It is used to compare the images reclassified by the subnet model with the true image labels and calculate the cross-entropy loss between the true labels and the output of the subnet model.

[0093] The distillation loss calculation module 6 is connected to the subnet model construction module 3. It is used to compare the category probability distribution obtained by the subnet model reclassification with the soft labels of the teacher model and calculate the knowledge distillation loss between the soft labels and the subnet model prediction.

[0094] The total loss calculation module 7, connected to the subnet model construction module 3, is used to calculate the total loss by weighting the two losses. The total loss is used as the network training error, and the final subnet model is obtained after iterative training.

[0095] The student model construction module 8 is connected to the subnet model construction module 3. It is used to compress the knowledge of the subnet model into a single model, input the above image training set for iterative training, and simultaneously perform pruning and quantize the pruned weights. The quantized weights are further compressed through Huffman coding to generate the student model.

[0096] As a preferred technical solution, the teacher model includes a pre-trained ResNet-50 network.

[0097] This invention provides a lightweight image classification neural network architecture system based on knowledge distillation. Through modular design, it achieves efficient image classification model training and optimization. First, a dataset preprocessing module performs a series of image transformation operations on the original dataset, including image scaling, horizontal flipping, random cropping, and padding, to generate an image set suitable for training. Subsequently, a teacher model preprocessing module trains the teacher model using a large amount of data and generates soft labels as an important reference for subsequent student model training.

[0098] Next, the subnet model construction module inputs the preprocessed image training set and the soft labels generated by the teacher model into the student subnet model and outputs the image classification results. Simultaneously, this module preserves the weights of the student subnet model to ensure effective utilization of existing knowledge during model training. The personalized training module introduces a rank factor during training to transfer knowledge from the teacher model to the student subnet model. Furthermore, by adding adversarial perturbations to the input images, this module improves the robustness of the student subnet model and dynamically adjusts the shared parameters of the model based on the calculated loss value.

[0099] In terms of loss calculation, the module for calculating the true loss compares the classification results of the student subnet model with the true labels to calculate the cross-entropy loss, thereby measuring how well the model fits the real data. Simultaneously, the module for calculating the distillation loss compares the prediction results of the student subnet model with the soft labels of the teacher model to calculate the knowledge distillation loss, ensuring that the student model effectively inherits the knowledge from the teacher model. Finally, the module for calculating the total loss weighted averages the true loss and distillation loss to obtain the total loss value, which is used as the network training error to iteratively optimize the student subnet model.

[0100] Finally, a student model module compresses the knowledge from the subnet models into a single model. This module retrains the model iteratively after inputting the image training set, performing model pruning and quantization during the process to reduce model complexity and computational overhead. The pruned weights are further compressed using Huffman coding, ultimately generating an efficient and lightweight student model that meets the performance and resource consumption requirements of practical applications.

[0101] As a preferred technical solution, the subnet model adopts the MobileNetV2 architecture. The forward propagation process of the subnet model includes the following steps: the input image is subjected to feature extraction through several convolutional layers; a low-dimensional feature map is generated after several convolution and pooling operations; the feature map is passed through a global average pooling layer to obtain global features; and the global features are passed through a fully connected layer to generate the output.

[0102] As a preferred technical solution, the knowledge distillation process includes the following steps: Training phase: The teacher model and the subnet model respectively receive the input image and generate teacher output and subnet output; the teacher output serves as a soft label, and the error between the subnet output and the teacher output is calculated using KL divergence; the training objective of the subnet model is to minimize the weighted sum of the soft label error and the true label error. Inference phase: Only the trained student model is used for the image classification task.

[0103] As a preferred technical solution, the soft labels of the teacher model are calculated using a temperature-sensitive softmax function, where the temperature parameter T is used to smooth the probability distribution. The specific representation of the probability value is as follows:

[0104]

[0105] Where, q i z is the probability value of image i belonging to a certain category. i This is the logits output by the teacher model, where T is the distillation temperature.

[0106] As a preferred technical solution, the total loss is calculated by proportionally dividing the cross-entropy loss and distillation loss, as follows:

[0107] L=α·T 2 ·H(q T ,p T) +(1-α)·H(y,p)

[0108] Where L represents the total loss of the model, H represents the cross-entropy loss, and q T and p T These are the output probability distributions of the teacher model and the subnet model at temperature T (usually set to 1), respectively. y is the hard label, and α is the weight used to balance the soft label and hard label losses.

[0109] As a preferred technical solution, the student model is obtained by knowledge compression of the subnet model, including three important steps: weight pruning, weight quantization, and weight compression.

[0110] The weight pruning part requires transferring the knowledge of the subnet model to the student model, inputting the training dataset into the student model, dynamically removing connections with smaller weights during training to reduce the weight scale and obtain a sparse network model, and retraining (fine-tuning training) the sparse model to restore its accuracy.

[0111] As a preferred technical solution, the weighted pruning is implemented through dynamic pruning technology, including dynamic evaluation of parameter importance, selection of pruning strategy, setting of pruning threshold, and periodic pruning.

[0112] Dynamic evaluation of parameter importance refers to calculating parameter importance scores based on the absolute values ​​of model parameters or the absolute values ​​of the products of parameter values ​​and gradients in the initial stage. As training progresses, an exponential moving average method is used to evaluate parameter importance and uncertainty in conjunction with changes in parameter values ​​and gradients.

[0113] Pruning strategy selection involves determining which parameters or layers need to be pruned, and dynamically updating and adjusting the importance score and uncertainty value based on the global step count.

[0114] The pruning threshold setting is based on the needs of different stages in the training process. The pruning intensity is dynamically adjusted by setting different thresholds for the warm-up step, the middle stage of training, and the end stage.

[0115] Periodic pruning refers to performing pruning operations periodically at different stages of training. The decision to prune is based on a set number of steps, thereby effectively controlling model complexity and maintaining model performance.

[0116] The weight quantization part includes two components: weight quantization and shared weights. Weight quantization refers to quantizing the model's weight values ​​into a finite number of discrete values, thereby reducing the precision of the weights and further lowering computational complexity and storage requirements. Shared weights involve storing the quantized weight values ​​in a shared weight table, with all weights in the model referencing values ​​from the shared weight table, further reducing storage requirements.

[0117] As a preferred technical solution, weight quantization is achieved by performing k-means clustering on the model weight values ​​and quantizing the weight values ​​to the nearest cluster center.

[0118] The weight compression part uses Huffman coding to compress the quantized weights.

[0119] like Figure 2 As shown, the lightweight image classification neural network architecture method based on knowledge distillation provided in this embodiment of the invention includes:

[0120] S101 uses the dataset preprocessing module to perform image transformations on the data in the dataset, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding, to generate an image training set; the teacher model preprocessing module trains the teacher model with a large amount of data, and the teacher model finally generates soft labels.

[0121] S102, by constructing a subnet model module, the image training set and the soft labels obtained by the teacher pre-training are respectively input into the student subnet model, the image classification results of the student subnet are output and the weights of the student subnet are saved;

[0122] S103 introduces a rank factor through a personalized training module to transfer teacher knowledge to the subnet model. During training, adversarial perturbations are added to the input image, the loss value of the student subnet is calculated, and the subnet shared parameters are updated based on the loss value to improve the robustness of the model.

[0123] S104, by comparing the images reclassified by the subnet model with the real image labels through the real loss calculation module, the cross-entropy loss between the real labels and the subnet model output is calculated;

[0124] S105, by comparing the category probability distribution obtained by the subnet model reclassification with the soft label of the teacher model through the distillation loss calculation module, the knowledge distillation loss between the soft label and the subnet model prediction is calculated.

[0125] S106, the total loss is calculated by weighting the two losses in the total loss calculation module. The total loss is used as the network training error. After updating and iterating the training, the final subnet model is obtained.

[0126] S107: By constructing a student model module, the knowledge of the subnet model is compressed into a single model. The above image training set is input for iterative training. At the same time, pruning is performed and the pruned weights are quantized. The quantized weights are further compressed through Huffman coding to generate the student model.

[0127] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the lightweight image classification neural network architecture method based on knowledge distillation.

[0128] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the lightweight image classification neural network architecture method based on knowledge distillation.

[0129] Another objective of this invention is to provide an information data processing terminal for implementing the lightweight image classification neural network architecture system based on knowledge distillation.

[0130] Example 1

[0131] As shown in the figure, this embodiment provides a lightweight image classification neural network architecture system based on knowledge distillation, including the following steps:

[0132] Dataset preprocessing: Taking a publicly available dataset as an example, we first address issues such as missing values ​​in the dataset. Then, we perform image transformation operations on the processed dataset, including discrete mapping, horizontal flipping, random cropping, edge padding, and chained applications. The processed dataset is then used to supervise network learning.

[0133] In this embodiment, the publicly available dataset CIFAR-100 is used, which comprises 60,000 32*32 pixel color images. Each image belongs to one of 100 different categories, covering a wide range of everyday objects and animals, including cars, birds, cats, dogs, boats, airplanes, etc. Each category has 600 images, with 500 used for training and 100 for testing. Therefore, the dataset is divided into 45,000 training images, 5,000 validation images, and 10,000 test images for supervised learning, validation, and testing.

[0134] Teacher network pre-training: For existing networks, if a pre-trained network model is provided, it is loaded directly; for unpublished network models, the training dataset is input first for pre-training to obtain the trained network model and the corresponding dataset soft labels, which are then saved before loading.

[0135] In this embodiment, the teacher network used is a pre-trained ResNet-50 network, and a total of 4 teacher models are trained depending on the number of seeds set during training.

[0136] Constructing a subnet model: After adding perturbations to the training dataset and soft labels, input them into the subnet model, and output the subnet model weights and reclassification results.

[0137] The subnet reclassification results are compared with the soft labels to obtain the distillation loss, and compared with the hard labels to obtain the true loss. The two losses are weighted and summed and then smoothed with temperature to obtain the final loss. The subnet parameters are updated through backpropagation, and the final subnet model is trained.

[0138] The sub-network model is a custom convolutional Gateconv2d network structure. It uses convolution to control the input and output channel weights of image features, performing weight selection to achieve distillation. This convolutional layer includes two convolutional operations (conv1 and conv2) and introduces a learnable parameter `gate` to control the weight selection for each channel. After the input data passes through the two convolutional layers, the output is a weighted sum of these two layers. Several knowledge distillations are performed, and the loss is continuously updated via backpropagation, achieving finer-grained feature extraction and fusion.

[0139] Compressing the student model: knowledge from the subnet model is transferred to the student model, the student model is trained using the input training dataset, flexible pruning is performed during training to remove network connections with smaller weights to reduce the weight scale, and then the sparse model is fine-tuned to restore accuracy.

[0140] Further compress the parameter size: The pruned subnet model is weighted by k-means clustering to map the parameter values ​​to discrete values; then, the parameter size is further compressed by Huffman coding to obtain a sparse and high-performance image classification neural network.

[0141] In this embodiment, although the subnet model obtained by distilling the teacher network can reduce the parameter size and improve the robustness and training efficiency of the model to some extent, it still cannot be called a lightweight model due to the large parameter size and deployment difficulties. Therefore, the subnet model is deeply compressed to further reduce the parameter size while ensuring the performance of the model as much as possible.

[0142] As shown in the figure, this embodiment first trains a high-precision teacher network to generate soft labels for the training dataset, which are then used for model distillation and knowledge transfer. Next, a sub-network model is constructed, and teacher knowledge transfer is performed using a rank-one factor. The sub-network model is trained with perturbed data as input. The total loss is calculated based on the distillation loss and the true loss, and the model parameters are updated using the total loss. The generated sub-network model has a certain level of accuracy and is more efficient than teacher training. Knowledge transfer is performed based on the sub-network model, and the training data is re-inputted for retraining. The importance and uncertainty of the parameters are dynamically evaluated, and periodic pruning is performed to reduce the weight size. The pruned model parameters are quantized and weights are compressed using Huffman coding, resulting in a final high-precision, high-efficiency, lightweight image classification neural network model.

[0143] In this embodiment, both the teacher and student models contain two convolutional blocks (ConvBlocks), each containing two convolutional layers. Max pooling (max_pool) is performed after each convolutional block, with a pooling window size of 2 and a stride of 2. After image input, local features are extracted at each convolutional layer, followed by batch normalization to stabilize and accelerate training. The ReLU activation function is used to introduce non-linearity. After training each convolutional block, downsampling is performed at the max-pooling layer to reduce the feature map size while retaining the most important features.

[0144] As shown in the figure, the knowledge distillation process is as follows: First, a complex teacher model, typically a deep neural network, is trained with high accuracy. The teacher model is used to predict training data and generate soft labels. These soft labels are obtained using a softmax function with a temperature parameter. Next, a smaller student model is built, which learns the knowledge from the teacher model using a rank-one factor. Adversarial perturbations are added to the training dataset and fed into the student model for training, personalizing the model through these perturbations. A total loss function is defined, which includes the difference between the student model's predictions and the predictions of the teacher model's soft labels. The cross-entropy loss function is used to calculate the difference between the student model's output and the two types of labels. The total loss function L is optimized using backpropagation to train the student model, adjusting its parameters. During training, the same temperature parameter T (used for generating soft labels as the teacher model) is used to calculate the softmax. After multiple iterations of training, the student model gradually learns the knowledge from the teacher model. The final student model should significantly reduce model complexity and the number of parameters while maintaining high performance.

[0145] In this embodiment, the teacher model uses `optax.softmax_cross_entropy_with_integer_labels` to calculate the cross-entropy loss; the student model calculates a weighted sum of the true loss and the distillation loss. The true loss is obtained by calculating the cross-entropy loss using the predicted labels and the true labels; the distillation loss is obtained by calculating the cross-entropy loss using the predicted labels and the soft labels after temperature scaling.

[0146] For the actual loss, the update and iteration formula is:

[0147] L real =H(y,p)

[0148] The update formula for distillation loss is:

[0149] L kd =T 2 ·H(q T ,p T )

[0150] In this embodiment, the overall loss function formula is:

[0151] L=α·L kd +(1-α)·L real

[0152] Where L represents the total loss, L real L represents the true loss (the loss between the predicted result and the true label of the image). kd H represents the distillation loss (the loss between the predicted result and the soft label), H represents the cross-entropy loss, and q represents the distillation loss. T and p T These are the output probability distributions (soft labels) of the teacher model and the student model at temperature T (usually set to 1), respectively. y is the hard label, and α is the weight used to balance the soft label and hard label losses.

[0153] The formula for calculating the soft label is:

[0154]

[0155] Where q i z represents the probability value that an image belongs to category i. i This is the logits output by the teacher model, where T is the temperature coefficient.

[0156] This embodiment is trained based on a deep learning open-source framework. In terms of training parameter configuration, the SGD optimizer is used, the learning rate is set to 0.001, the total number of training rounds is 10000, the learning rate decays to 0.9 times the current rate every 200 training rounds, the number of training images (batch_size) in each round is set to 128, α is set to 1.0, and the distillation temperature is set to 4.0.

[0157] The soft targets are the probability distribution output by the teacher model, containing rich information about inter-class similarity, rather than just a hard classification. This probability distribution can be adjusted using the temperature parameter T to make the output probability distribution smoother, thus better conveying knowledge.

[0158] The temperature parameter T is used to smooth the softmax output. During training, the teacher model's output is processed by the softmax function, and T is introduced. When T > 1, the softmax output becomes smoother and better reflects the similarity between different categories. The student model also uses the same temperature parameter during training to learn the teacher model's soft labels.

[0159] As shown in the figure, in this embodiment, the image and category are first input into convolutional layers. Each convolutional layer extracts local features of the image through a 3x3 kernel. The convolutional layers use a certain number of filters to perform feature extraction. After each convolutional layer, the features are normalized using BatchNorm. Then, the ReLU activation function is used to introduce non-linearity, setting all negative values ​​to zero while retaining positive values, allowing the model to learn more complex features. At the end of each ConvBlock training, the feature map is downsampled using a 2x2 window with a stride of 2, reducing the size of the feature map while retaining the most important features. Through the combined action of convolutional and pooling layers, multi-level and multi-scale features are extracted from the input, providing high-quality feature representations for subsequent fully connected (dense) layers and classification tasks. Finally, the category of the output image is calculated in the fully connected layer based on the extracted feature representations.

[0160] The filters, conv1 and conv2, are convolved with different numbers of filters (feature maps), the number of which is determined by the parameter `filters`. In this embodiment, the teacher model's ConvBlock uses 32 and 64 filters respectively; the student model's ConvBlock uses 16 and 32 filters respectively. Both the teacher and student models use 3*3 convolutional kernels, performing max pooling after two convolutional layers (one convolutional block).

[0161] As shown in the figure, the subnet model depth compression process is as follows: In the initial stage, the parameter importance score is calculated based on the absolute value of the product of the model weight (parameter) value and the gradient. As the training process progresses, exponential moving average is used to dynamically evaluate the parameter importance and uncertainty. During training, the pruning threshold is determined by sorting the importance and uncertainty scores of all parameters and retaining the weight values ​​corresponding to the score sorting vector proportionally. The theoretical pruning threshold Ts changes with the time step S during training and is dynamically adjusted based on the initial threshold Ts_ini and the final threshold Ts_fin. Each layer dynamically adjusts the pruning intensity based on the threshold during the initial warm-up, intermediate, and final stages. Simultaneously, at different stages of training, the value of the periodic pruning strategy (mask_ind) determines whether to prune at the current stage, effectively controlling model complexity and maintaining model performance. The network structure before and after pruning is shown in the figure. Next, k-means clustering is performed on the model parameter values ​​to quantize the parameter values ​​into the nearest cluster centers and store them in a shared weight table. Huffman coding is used to compress the quantized parameters, further reducing the parameter size.

[0162] The dynamic scoring formula for parameter importance is as follows:

[0163]

[0164] The importance of the smoothing parameter was scored using an exponential moving average. The formula for calculating the exponential moving average of the parameter is:

[0165] S exp [ω]=β1·S exp [ω]+(1-β1)·S ipt [ω]

[0166] For uncertainty, an uncertainty exponential moving average was used, and its calculation formula is as follows:

[0167]

[0168] Where ω represents the current weight, L represents the loss value of the current training step, and S ipt [ω] represents the importance score of parameter ω, S exp [ω] represents the exponential moving average of ω, S unc [ω] represents the exponentially moving average of the uncertainty of ω, U s This represents the updated training steps (current training steps). β1 and β2 are two probability values ​​no greater than 1, which determine the pruning ratio and are also specified parameters.

[0169] The proportion P of retaining weight based on score ranking is...

[0170] P = 1 - T s

[0171] Where P is the ratio of reserved network connections, and T S The theoretical pruning threshold (a floating-point number less than 1).

[0172] The pruning threshold in the intermediate training stage is set by smoothing the threshold from the initial to the final value using a cubic coefficient based on the ratio of the current step to the intermediate step, thus achieving dynamic adjustment of the pruning threshold. The formula for calculating the pruning threshold is:

[0173] In this embodiment, the pruning threshold during the initial preheating stage is:

[0174] T s =T s_ini

[0175] The pruning threshold for ending the preheating phase is:

[0176] T s =T s_fin

[0177] In this embodiment, when entering the intermediate preheating stage, the formula for calculating the number of preheating steps in the current stage is:

[0178] S sw =(W ini +W fin )×S w

[0179] The formula for calculating the cubic coefficient used for the smoothing threshold is:

[0180]

[0181] The threshold calculation formula for the intermediate stage is:

[0182] T S =T s_fin +(T s_ini -T s_fin )×(C mul ) 3

[0183] Where T s_ini and T s_fin S represents the initial threshold and the final threshold, respectively. sw W represents the number of preheating steps in a stage. ini and W fin S represents the initial preheating value and the final preheating value, respectively. w C represents the initial preheating step number. mul S represents the cubic coefficient, S0, S1 sw S w S tThese represent the current step count, the number of steps in the pre-warm-up phase, the initial pre-warm-up steps, and the total steps, respectively.

[0184] The theoretical pruning threshold for the entire pruning strategy is calculated as follows:

[0185]

[0186] The periodic pruning strategy mask_ind is set to a boolean value, based on the current training step G. S With the number of training steps in a cycle δ T The ratio is calculated and used to control whether pruning is performed in the current step. The calculation method for mask_ind is as follows:

[0187]

[0188] Regarding training parameter configuration, the initial warm-up steps S W Set to 5, β1 and β2 are set to 0.85 and 0.95 respectively, δ T Set to 10, T s_ini T s_fin W ini and W fin Set them to 1, 0.1, 1 and 5 respectively.

[0189] Example 2

[0190] This embodiment provides a storage medium, which may be a ROM, RAM, disk, optical disk, or other storage medium. The storage medium stores one or more programs. When the programs are executed by a processor, they implement the lightweight image classification neural network architecture system based on knowledge distillation of Embodiment 1.

[0191] Example 3

[0192] This embodiment provides a computing device, which may be a desktop computer, laptop computer, smartphone, tablet computer or other terminal device with display function. The computing device includes a processor and a memory. The memory stores one or more programs. When the processor executes the program stored in the memory, it implements the lightweight image classification neural network architecture system based on knowledge distillation of Embodiment 1.

[0193] The technical solution of this invention can be widely applied to agricultural equipment, especially agricultural machinery equipped with intelligent classification programs, such as drones and automated agricultural machinery systems. The following are specific application scenarios:

[0194] 1. Drone monitoring:

[0195] Equipped with an intelligent image classification system, the drone can conduct large-scale real-time monitoring of fruit trees and quickly identify the presence of pests and diseases. Through aerial photography and image processing, the drone can automatically detect the health status of fruit trees, generate detailed pest and disease distribution maps, and transmit the data to farmers' mobile phones in real time.

[0196] 2. Automated agricultural machinery and equipment:

[0197] Modern agricultural machinery, such as automated tillers and sprayers, can be equipped with this technology to achieve precision planting and pesticide application. For example, sprayers can automatically adjust the amount of pesticide applied based on image classification results, ensuring that pest-infested areas are properly treated and avoiding pesticide waste.

[0198] 3. Intelligent greenhouse management:

[0199] In greenhouse environments, monitoring equipment equipped with intelligent classification programs can analyze plant growth status in real time, identify pests and diseases or abnormal growth, and automatically trigger irrigation or ventilation systems to optimize the crop growth environment.

[0200] Practical application results:

[0201] These intelligent agricultural machines enable farmers to manage crops more efficiently, reduce labor costs, improve production efficiency, and minimize yield losses due to pests and diseases. Such intelligent agricultural machinery will greatly promote the development of intelligent agriculture, increase agricultural output, and enhance the sustainability of the agricultural economy.

[0202] The effectiveness of the invention model is analyzed in depth from a theoretical perspective.

[0203] 1. Advantages of Model Structure: The invented model adopts a more streamlined and efficient network architecture in its structural design. For example, by combining feature fusion and multi-scale feature extraction, the model can reduce the number of parameters while maintaining high classification accuracy, thereby improving the model's inference efficiency.

[0204] 2. Optimization of distillation technology: Compared with traditional knowledge distillation methods, the invention model has conducted a deeper level of mining and refinement of the knowledge of the teacher model. For example, by using weighted average or other methods, it more reasonably integrates the features of multiple teacher models, enabling the student model to learn more effectively.

[0205] 3. Model Adaptability: The invented model was specifically optimized for resource-constrained devices (such as mobile devices). This optimization is reflected not only in model size and inference time, but also in reducing reliance on high-precision data, enabling it to maintain good classification performance on low-quality images (such as low-resolution or noisy images).

[0206] 4. Software Simulation: Simulation can predict the performance of the invention model in real-world scenarios. For example, in a simulation environment, the inference time, accuracy variations, and robustness under various image interferences (such as changes in lighting, blurring, and occlusion) can be tested under different computing resource conditions.

[0207] Expected results from the model:

[0208] High accuracy: Due to the full utilization of the teacher's model knowledge, the invented model should maintain a high classification accuracy on the test dataset.

[0209] Smaller model size: Due to the effective compression and refinement of features, the model size should be significantly smaller than the original teacher model.

[0210] Fast inference: Through optimization, inference time should be significantly reduced, especially on mobile devices.

[0211] Adaptable to diverse scenarios: The model should maintain high robustness under different application scenarios (such as low light, occlusion, and noise) to prove its effectiveness in actual deployment.

[0212] Through these theoretical analyses and simulations, we can predict that the invention model will exhibit superior performance in practical applications, especially in resource-constrained environments.

[0213] like Figure 9 This diagram illustrates the system interface and predicts the expected performance of the invention. The graph shows the normalized line graphs of the theoretical performance of each model across different metrics. The invention's model is expected to perform well across all metrics, particularly in terms of inference time and model size, outperforming the teacher model. Accuracy and ROC metrics should also be superior to the KD model. Overall, the invention's model is expected to achieve a good balance between performance and efficiency, making it suitable for deployment needs in practical applications.

[0214] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0215] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A lightweight image classification neural network architecture system based on knowledge distillation, characterized in that, include: The module includes: dataset preprocessing module, teacher model preprocessing module, subnet model construction module, personalized training module, real loss calculation module, distillation loss calculation module, total loss calculation module, and student model construction module. The dataset preprocessing module, connected to the teacher model preprocessing module, is used to perform image transformations on the data in the dataset, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding, to generate an image training set. The teacher model preprocessing module, connected to the dataset preprocessing module and the subnet model construction module, is used to train the teacher model with a large amount of data, and the teacher model ultimately generates soft labels. The subnet model construction module is connected to the teacher model preprocessing module, personalized training module, real loss calculation module, distillation loss calculation module, total loss calculation module, and student model construction module. It is used to input the image training set and the soft labels obtained by the teacher pre-training into the student model, output the image classification results of the student model, and save the weights of the student model. The personalized training module, connected to the subnet model building module, is used to introduce a rank factor to transfer teacher knowledge to the subnet model. During training, adversarial perturbations are added to the input image, the loss value of the student model is calculated, and the subnet shared parameters are updated based on the loss value. The module for calculating the true loss is connected to the module for building the subnet model. It is used to compare the images reclassified by the subnet model with the true image labels and calculate the cross-entropy loss between the true labels and the output of the subnet model. The module for calculating distillation loss is connected to the module for building subnet models. It is used to compare the category probability distribution obtained by the subnet model reclassification with the soft labels of the teacher model and calculate the knowledge distillation loss between the soft labels and the predictions of the subnet model. The total loss calculation module, connected to the subnet model construction module, is used to calculate the total loss by weighting the two losses. The total loss is used as the network training error, and the final subnet model is obtained after iterative training. The student model building module is connected to the subnet model building module. It is used to compress the knowledge of the subnet model into a single model. The model is then retrained using the above image training set. Simultaneously, pruning is performed and the pruned weights are quantized. The quantized weights are further compressed using Huffman coding to generate the student model.

2. The lightweight image classification neural network architecture system based on knowledge distillation as described in claim 1, characterized in that, The teacher model includes a pre-trained ResNet-50 network.

3. The lightweight image classification neural network architecture system based on knowledge distillation as described in claim 1, characterized in that, The sub-network model is a custom convolutional Gateconv2d network structure. It uses convolution to control the input and output channel weights of image features to perform distillation. The convolutional layer includes two convolutional operations, conv1 and conv2, and introduces a learnable parameter gate to control the selection weight of each channel. After the input data passes through the two convolutional layers, the weighted sum of the two convolutional layers is output. Several knowledge distillations are performed and the loss is continuously updated by backpropagation to achieve finer-grained feature extraction and fusion.

4. The lightweight image classification neural network architecture system based on knowledge distillation as described in claim 1, characterized in that, The knowledge distillation process includes the following steps: Training phase: The teacher model and the subnet model receive the input image respectively and generate the teacher output and the subnet output; the teacher output is used as a soft label, and the error between the subnet output and the teacher output is calculated by KL divergence; the training objective of the subnet model is to minimize the weighted sum of the soft label error and the true label error; Inference phase: Only the trained student model is used for the image classification task.

5. The lightweight image classification neural network architecture system based on knowledge distillation as described in claim 1, characterized in that, The soft labels of the teacher model are calculated using a temperature-sensitive softmax function, where the temperature parameter T is used to smooth the probability distribution. The specific representation of the probability values ​​is as follows: ， Where zi is the logits output by the teacher model, and T is the distillation temperature; The total loss is calculated by proportionally dividing the cross-entropy loss and distillation loss, and is specifically expressed as follows: ， Where H represents the cross-entropy loss, qT and pT are the output probability distributions of the teacher model and the subnet model at temperature T, respectively, y is the hard label, and α is the weight used to balance the soft label and hard label losses.

6. The lightweight image classification neural network architecture system based on knowledge distillation as described in claim 1, characterized in that, The student model is obtained by knowledge compression from the subnet model, which includes three important steps: weight pruning, weight quantization, and weight compression. The weight pruning part requires transferring the knowledge of the subnet model to the student model, inputting the training dataset into the student model, and dynamically removing connections with smaller weights during the training process. The weighted pruning is implemented through dynamic pruning techniques, including dynamic evaluation of parameter importance, selection of pruning strategies, setting of pruning thresholds, and periodic pruning. Dynamic evaluation of parameter importance refers to calculating the parameter importance score based on the absolute value of the model parameters or the absolute value of the product of the parameter values ​​and the gradient in the initial stage. The selection of pruning strategies involves determining which parameters or layers need to be pruned, and dynamically updating and adjusting the importance score and uncertainty value based on the global step count; The pruning threshold setting is based on the needs of different stages in the training process. The pruning intensity is dynamically adjusted by setting different thresholds for the warm-up step, the middle stage of training, and the end stage. Periodic pruning refers to performing pruning operations periodically at different stages of training, with the decision on whether to prune based on a set number of steps. The weight quantization part includes two parts: weight quantization and shared weight.

7. A method for implementing a lightweight image classification neural network architecture based on knowledge distillation as described in any one of claims 1-6, characterized in that, The lightweight image classification neural network architecture method based on knowledge distillation includes: Step 1: The dataset preprocessing module performs image transformations on the data in the dataset, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding, to generate an image training set; The teacher model is trained with a large amount of data through the teacher model preprocessing module, and finally the teacher model generates soft labels. Step 2: By constructing a subnet model module, the image training set and the soft labels obtained from teacher pre-training are respectively input into the student model, the image classification results of the student model are output, and the weights of the student model are saved; Step 3: Introduce the rank factor through the personalized training module to transfer teacher knowledge to the subnet model. During the training process, add adversarial perturbation to the input image, calculate the loss value of the student model, and update the subnet shared parameters based on the loss value. Step 4: By using the real loss calculation module, the images reclassified by the subnet model are compared with the real image labels, and the cross-entropy loss between the real labels and the subnet model output is calculated. Step 5: By using the distillation loss calculation module, the category probability distribution obtained by the subnet model reclassification is compared with the soft labels of the teacher model, and the knowledge distillation loss between the soft labels and the subnet model prediction is calculated. Step 6: The total loss is calculated by weighting the two losses using the total loss calculation module. The total loss is used as the network training error. After updating and iterating the training, the final subnet model is obtained. Step 7: By constructing a student model module, the knowledge of the subnet model is compressed into a single model. The image training set mentioned above is input and iteratively trained again. At the same time, pruning is performed and the pruned weights are quantized. The quantized weights are further compressed using Huffman coding to generate the student model.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the lightweight image classification neural network architecture method based on knowledge distillation as described in claim 7.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the lightweight image classification neural network architecture method based on knowledge distillation as described in claim 7.

10. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the lightweight image classification neural network architecture system based on knowledge distillation as described in any one of claims 1-6.

Images