Lightweight image classification neural network architecture system based on knowledge distillation

Abstract

The invention belongs to the technical field of image classification neural networks, and discloses a lightweight image classification neural network architecture system based on knowledge distillation. The invention provides a lightweight image classification neural network architecture system based on knowledge distillation. The lightweight image classification neural network architecture system comprises a data set preprocessing module, a teacher model preprocessing module, a subnet model construction module, a personalized training module, a real loss calculation module, a distillation loss calculation module, a total loss calculation module and a student model construction module. The invention provides a lightweight neural network architecture system based on knowledge distillation, and designs a brand new training mode by using the superiority of knowledge distillation for teacher model weight extraction: the knowledge distillation of teacher and student models is combined with a deep compression method; the system for constructing the lightweight network can be used for training various complex image classification neural networks so as to improve the network performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification neural network, and in particular relates to a lightweight image classification neural network architecture system based on knowledge distillation. Background Art

[0002] With the development of deep learning technology, neural networks have been widely used in the field of computer vision, especially in image classification tasks. Image classification is a core problem in the field of computer vision, and its goal is to classify input images into known categories. Compared with traditional manual feature extraction, neural networks can automatically learn information such as edges, textures, shapes, and semantics in images through a hierarchical structure; at the same time, they can also capture complex patterns and regularities in image data, obtain sample robust features through end-to-end learning, and make the model have good generalization ability. This technology has a wide range of application needs in many fields, including tasks such as identifying crop diseases and pests, detecting the growth status of animal husbandry, identifying targets such as faces in security systems such as public security, airports, and banks, and using drones for terrain surveys, water conditions inspections, and other scenarios. It has played an important role.

[0003] There are many methods for image classification through neural networks. However, in the actual process of deploying models to terminals, the available computing cost is insufficient and the training data is limited, making it difficult to support the robust operation of the developed image classification models on resource-constrained devices (such as mobile devices, embedded systems, etc.). Knowledge distillation (KD) is a highly regarded method for building lightweight networks. It transfers the knowledge of a large and complex teacher model to a smaller student model, so that the student model can significantly reduce model parameters and computation while maintaining high performance. This process can not only effectively improve the generalization ability of the student model, but also alleviate the problem of insufficient training data to a certain extent.

[0004] In "An Image Classification Method Based on Integrated Knowledge Distillation (Invention Authorization Number: CN112199535B)", the invention proposes a teacher model that jointly guides the student model through three stages to further improve the performance of the student model, while improving the model response speed and ensuring the accuracy of image classification. "An Image Classification Model Training Method Based on Integration and Knowledge Distillation (Invention Authorization Number: CN113822373B)" proposes a method of first training multiple models of different depths based on the long-tail effect of the data set, then using these models to extract features from the training data and integrating these deep models in the feature space, and finally using knowledge distillation to fuse the integrated model into a less complex reasoning model, thereby reducing the reasoning complexity of the image classification model training method based on integration and knowledge distillation for imbalanced data. "Compression method and platform based on knowledge distillation and training language model (invention authorization number: CN111767711B)" proposes to first design a universal knowledge distillation strategy for feature transfer. In the process of classroom model knowledge transfer, it focuses on the feature expression ability of small samples in the middle layer of the teacher model and uses these features to guide the feature mapping of the student model to approach the teacher model. Finally, it uses the self-attention distribution detection function of the teacher model to construct a self-attention based cross-knowledge distillation method, which realizes the automatic compression of pre-trained language models for multi-tasks.

[0005] The existing knowledge distillation technologies represented by the above three patents all distill the knowledge of the integrated model into a single model, but the distilled model is still large in scale and does not support large-scale deployment on mobile terminals. Among them, CN113822373B performs feature extraction during the image classification model training process, which solves the problem of student model performance degradation, but the calculation process is complicated and the model response time is long, and the performance of the integrated fusion model is poor.

[0006] Through the above analysis, the problems and defects of the prior art are as follows:

[0007] (1) Traditional knowledge distillation technology distills the knowledge of the integrated model into a single model. However, it often adopts the method of averaging the parameters of the teacher model to the student side and often only undergoes one round of pruning after integration, ignoring the differences in the prediction results of different teacher models on the same data. The parameters learned by the student model are flattened, resulting in the model after distillation still having large-scale parameters, which makes the model perform poorly in the task of deploying on the mobile terminal.

[0008] (2) Existing technologies for enhancing the diversity of student models mainly extract data sample features in advance or correct the features learned by the student model by enhancing the expression of small samples to improve the accuracy of the student model. There are common defects such as large dimensions of data features that the model needs to learn during the distillation process, high computational complexity, and low model training efficiency. Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a lightweight image classification neural network architecture system based on knowledge distillation.

[0010] The present invention is implemented as follows: a lightweight image classification neural network architecture system based on knowledge distillation includes:

[0011] Dataset preprocessing module, teacher model preprocessing module, subnet model building module, personalized training module, true loss calculation module, distillation loss calculation module, total loss calculation module, student model building module;

[0012] The dataset preprocessing module is connected to the teacher model preprocessing module and is used to perform image transformation on the data in the dataset, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding, to generate an image training set;

[0013] The teacher model preprocessing module is connected with the data set preprocessing module and the subnet model building module to train the teacher model with a large amount of data, and finally the teacher model generates soft labels;

[0014] A subnet model module is constructed, connected with the teacher model preprocessing module, the personalized training module, the true loss calculation module, the distillation loss calculation module, the total loss calculation module, and the student model construction module, and is used to input the image training set and the soft labels obtained by the teacher pre-training into the student subnet model respectively, output the image classification result of the student subnet and save the weight of the student subnet;

[0015] The personalized training module is connected to the subnetwork model building module and is used to introduce a rank-one factor to transfer the teacher knowledge to the subnetwork model. During the training process, adversarial perturbations are added to the input image, the loss value of the student subnetwork is calculated, and the subnetwork shared parameters are updated according to the loss value to improve the robustness of the model.

[0016] The module for calculating the true loss is connected to the module for building the subnetwork model. It is used to compare the images reclassified by the subnetwork model with the true image labels and calculate the cross entropy loss between the true labels and the output of the subnetwork model.

[0017] The module for calculating the distillation loss is connected to the module for building the subnetwork model. It is used to compare the category probability distribution obtained by reclassifying the subnetwork model with the soft label of the teacher model, and calculate the knowledge distillation loss between the soft label and the prediction of the subnetwork model.

[0018] The total loss calculation module is connected to the subnet model construction module and is used to calculate the total loss by weighted average of the two losses. The total loss is used as the network training error. After updating the iterative training, the final subnet model is obtained.

[0019] The student model module is constructed and connected to the subnet model construction module to compress the knowledge of the subnet model into a single model, input the above image training set for re-iterative training, and prune and quantize the pruned weights at the same time. The quantized weights are further compressed through Huffman coding to generate a student model.

[0020] Furthermore, the teacher model includes a pre-trained ResNet-50 network.

[0021] Furthermore, the subnet model adopts the MobileNetV2 architecture; the forward propagation process of the subnet model includes the following steps: the input image is subjected to feature extraction through several convolutional layers; a low-dimensional feature map is generated after several convolution and pooling operations; the feature map is subjected to a global average pooling layer to obtain a global feature; and the global feature is outputted through a fully connected layer.

[0022] Furthermore, the knowledge distillation process includes the following steps: training phase: the teacher model and the subnetwork model receive input images respectively to generate teacher output and subnetwork output; the teacher output is used as a soft label, and the error between the subnetwork output and the teacher output is calculated by KL divergence; the training goal of the subnetwork model is to minimize the weighted sum of the soft label error and the true label error; inference phase: only the trained student model is used for image classification tasks.

[0023] Furthermore, the soft label of the teacher model is a label probability value calculated by a softmax function with temperature, where the temperature parameter T is used to smooth the probability distribution; the specific expression of the probability value is:

[0024]

[0025] Among them, z i are the logits output by the teacher model, T is the distillation temperature;

[0026] As a preferred technical solution, the total loss is calculated by proportionally combining the cross entropy loss and the distillation loss, which is specifically expressed as:

[0027] L=α·T 2 ·H(q T ,p T) +(1-α)·H(y,p)

[0028] Among them, H represents the cross entropy loss, q T and p T are the output probability distributions of the teacher model and the subnetwork model at temperature T, respectively, y is the hard label, and α is the weight used to balance the soft label and hard label losses.

[0029] Furthermore, the student model is obtained by knowledge compression of the subnet model, including three important steps: weight pruning, weight quantization, and weight compression;

[0030] The weight pruning part needs to transfer the subnet model knowledge to the student model, input the training data set to the student model, and dynamically remove the connections with smaller weights during the training process;

[0031] The weight pruning is achieved through a dynamic pruning technology, including dynamic evaluation of parameter importance, pruning strategy selection, pruning threshold setting, and periodic pruning;

[0032] Dynamic evaluation of parameter importance means that in the initial stage, the parameter importance score is calculated based on the absolute value of the model parameter or the absolute value of the product of the parameter value and the gradient;

[0033] Pruning strategy selection involves determining which parameters or layers need to be pruned and dynamically updating and adjusting the importance scores and uncertainty values ​​based on the global step number;

[0034] The pruning threshold setting is based on the needs of different stages of the training process. The pruning intensity is dynamically adjusted by setting different thresholds in the warm-up step, the middle stage of training, and the end stage.

[0035] Periodic pruning refers to periodic pruning at different stages of training, and whether to prune is determined based on the set step interval;

[0036] The weight quantization part includes two parts: weight quantization and shared weight.

[0037] Another object of the present invention is to provide a lightweight image classification neural network architecture method based on knowledge distillation, including:

[0038] Step 1: Perform image transformation on the data in the data set through the data set preprocessing module, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding to generate an image training set; train the teacher model with a large amount of data through the teacher model preprocessing module, and finally the teacher model generates soft labels;

[0039] Step 2, inputting the image training set and the soft labels obtained by teacher pre-training into the student subnet model respectively by constructing a subnet model module, outputting the image classification result of the student subnet and saving the weight of the student subnet;

[0040] Step 3: Introduce the rank-one factor through the personalized training module to transfer the teacher knowledge to the subnetwork model. During the training process, add adversarial perturbations to the input image, calculate the loss value of the student subnetwork, and update the subnetwork shared parameters according to the loss value to improve the robustness of the model.

[0041] Step 4: Compare the image reclassified by the subnetwork model with the real image label by calculating the real loss module, and calculate the cross entropy loss between the real label and the subnetwork model output;

[0042] Step 5: Compare the category probability distribution obtained by reclassifying the subnetwork model with the soft label of the teacher model through the calculation distillation loss module, and calculate the knowledge distillation loss between the soft label and the prediction of the subnetwork model;

[0043] Step 6: The total loss is calculated by adding the weighted average of the two losses through the total loss calculation module. The total loss is used as the network training error. After updating the iterative training, the final subnetwork model is obtained.

[0044] Step 7: Compress the knowledge of the subnetwork model into a single model by constructing a student model module, input the above image training set for re-iterative training, prune and quantize the pruned weights at the same time, and further compress the quantized weights through Huffman coding to generate a student model.

[0045] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the lightweight image classification neural network architecture method based on knowledge distillation.

[0046] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the lightweight image classification neural network architecture method based on knowledge distillation.

[0047] Another object of the present invention is to provide an information data processing terminal, which is used to implement the lightweight image classification neural network architecture system based on knowledge distillation.

[0048] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0049] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:

[0050] The present invention provides a lightweight neural network architecture system based on knowledge distillation. By utilizing the superiority of knowledge distillation for extracting teacher model weights, a new training mode is designed: multiple teacher models are pre-trained on the same data set, and a rank-one factor is introduced by distillation to reduce the search space while adding disturbances, thereby improving the training efficiency while ensuring the model performance, and obtaining a subnet model with a reduced parameter scale. The subnet model is compressed to obtain a student model, and dynamic pruning is performed during the iterative training process of the student model. Under the joint action of the two, the model training efficiency is effectively improved and the parameter scale is reduced, thereby improving the quality and efficiency of image classification. The present invention combines the knowledge distillation of the teacher-student model with the deep compression method, and the proposed system for constructing a lightweight network can be used to train various complex image classification neural networks to improve network performance.

[0051] (1) The purpose of introducing the rank-one factor and combining the residual network for knowledge distillation in this invention is to solve the problem of large search space and low training efficiency when deep neural networks extract target features. In practical applications, image training data often contains high-dimensional features, which often leads to slow training of the teacher model. Therefore, the rank-one factor is introduced to fully extract image features and residual convolution to help improve the efficiency of model training.

[0052] (2) The present invention adds disturbances during training to solve the problem of decreased diversity loss performance when the subnet learns from the teacher model (for example, the teacher model has three soft labels, but the subnet model only learns one of the soft labels that appears more frequently and ignores the other results). This allows the subnet model to fully learn the differences between the knowledge of different teacher networks, reduces the probability of positive and negative sample classification errors when the subnet learns image features, and improves the accuracy and robustness of the student model.

[0053] (3) The present invention adds deep compression to the traditional distillation, further compresses the parameter scale, and generates a personalized lightweight student model that is easy to deploy on terminals. Compared with the traditional weight processing method, the present invention does not directly observe the weight value but measures the weight importance. By setting a mask, it continuously tries to subtract network connections with less influence during the training process, thereby reducing the number of network connections and refining the subnetwork model to effectively save computing resources.

[0054] The positive effects of the present invention are: it not only retains important information in the teacher model through knowledge distillation, improves the accuracy and generalization ability of the student model, but also has the ability to improve the accuracy and robustness of the model under complex classification problems and noise interference, and at the same time greatly reduces the number of model parameters through pruning technology, reduces the consumption of computing resources and reasoning time. This makes the generated student model both lightweight and efficient, and is suitable for resource-constrained environments such as mobile devices and embedded systems. At the same time, this method can also achieve rapid training and deployment of the model while ensuring model performance, and has broad application prospects.

[0055] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0056] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0057] In the technical solution of the present invention, by deploying the image recognition model to mobile devices, such as smart phones, farmers can monitor the growth status of crops in the greenhouse in real time and accurately grasp the key operation time nodes (such as pest control, watering, etc.), thereby increasing the yield of crops.

[0058] Taking the monitoring and watering facilities in the greenhouse as an example, assuming that each farmer owns 10 mu of land, the investment required to deploy this technical solution is 50,000 yuan, which specifically includes the following parts:

[0059] 1. Equipment cost:

[0060] (1) Smartphone and supporting applications: approximately RMB 10,000.

[0061] (2) Monitoring equipment and sensors: approximately RMB 20,000.

[0062] (3) Water spraying facilities and automatic control system: approximately RMB 20,000.

[0063] 2. Operating costs:

[0064] (1) Data traffic and daily maintenance costs: approximately RMB 10,000 per year.

[0065] Using data observed by the platform equipped with a personalized lightweight image classification neural network, through precise crop management technology, the yield of crops is expected to increase by 30%. Based on the annual yield of 15,000 yuan per mu under the traditional planting model, the yield per mu can be increased to 19,500 yuan after the increase. The total yield increase of 10 mu of land is 45,000 yuan.

[0066] Comprehensive benefit analysis:

[0067] With the above investment, the profit from increased production can reach 45,000 yuan within one year, and after deducting the operating cost of 10,000 yuan, the net profit from increased production is 35,000 yuan. Calculated on a two-year cycle, the total profit is expected to reach 90,000 yuan, and after deducting the initial investment of 50,000 yuan, a net profit of 40,000 yuan will be achieved.

[0068] By promoting this technology on a large scale, domestic farmers will be able to achieve higher yields on limited land resources, advance the process of agricultural modernization, and promote the sustainable development of the agricultural economy.

[0069] (2) The technical solution of the present invention solves a technical problem that people have been eager to solve but have never succeeded in solving: that is, while ensuring the performance of the student model, the parameter scale of the model is reduced, and a personalized lightweight neural network system is constructed. While ensuring the performance of the student model, the parameter scale of the model is reduced, and the mobile terminal deployment of large models is realized, and the scientific research results in neural networks are promoted and implemented on a large scale.

[0070] Compared with the traditional multi-teacher student model, the subnet of the present invention can fully learn the features extracted by different teachers, fully make up for the defect of diversity loss in the process of subnet learning multi-teacher model, and the process achieves a small calculation scale by randomly adding disturbances. At the same time, it also integrates the idea of ​​deep compression on the basis of traditional knowledge distillation, further compresses the model parameters, and generates a personalized lightweight student model, which is easy to deploy on the terminal. It effectively solves the problem that the size and performance of the deep neural network model cannot be taken into account at the same time.

[0071] Third, the present invention solves the problems of low efficiency and insufficient model accuracy in the training and reasoning process of lightweight image classification neural network models in the prior art. Traditional methods usually rely on large deep neural network models. Although these models perform well in terms of accuracy, they are very demanding in terms of computing resources and storage space, and are not suitable for deployment on resource-constrained devices. To this end, the present invention effectively transfers the rich knowledge of the teacher model to the student model by introducing knowledge distillation technology, thereby achieving efficient training of lightweight models and significantly reducing computing costs during the reasoning process.

[0072] In addition, the present invention proposes a modular system architecture, which systematically solves the loss calculation and model compression problems in the knowledge transfer process through the collaborative work of multiple modules such as data preprocessing, teacher model training, student subnet model construction and training, loss calculation and student model compression. This modular design not only improves the flexibility of the system, but also improves the overall performance of the model through optimization and coordination between modules. In particular, after the introduction of adversarial perturbations and rank-one factors, the robustness of the student model has been significantly improved, further enhancing the stability and generalization ability of the model in complex environments.

[0073] Through the implementation of the present invention, the training efficiency of the lightweight image classification neural network has been significantly improved, and the computing resource requirements and storage space occupied by the model have been greatly reduced while maintaining the model accuracy. This progress has enabled efficient image classification models to be more widely deployed in practical applications, especially in resource-constrained scenarios such as mobile devices and embedded systems, and has great application prospects. At the same time, the quantization and compression technology of the model also significantly reduces the delay of the model in the inference stage, so that application scenarios with high real-time requirements can also be met.

[0074] In general, the present invention has achieved significant technical progress on the basis of solving the limitations of the existing technology. Through innovative knowledge distillation strategies and modular design, the present invention not only improves the training and reasoning efficiency of lightweight neural network models, but also achieves significant technical breakthroughs in model compression, robustness enhancement, and computing resource optimization, providing a more efficient and practical solution for the field of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a structural block diagram of a lightweight image classification neural network architecture system based on knowledge distillation provided in an embodiment of the present invention.

[0076] Figure 2 It is a flowchart of a lightweight image classification neural network architecture system based on knowledge distillation provided by an embodiment of the present invention.

[0077] Figure 3 It is a detailed method flow chart of a lightweight image classification neural network architecture based on knowledge distillation provided in an embodiment of the present invention.

[0078] Figure 4 This is a technical diagram of building a lightweight image classification neural network based on knowledge distillation provided by an embodiment of the present invention.

[0079] Figure 5 This is a technical diagram of an algorithm for generating a highly robust and highly generalizable subnetwork model by distilling multiple teacher models provided in an embodiment of the present invention.

[0080] Figure 6 It is a structural hierarchical diagram of a lightweight image classification neural network based on knowledge distillation provided by an embodiment of the present invention.

[0081] Figure 7 It is an algorithm technology diagram for deeply compressing a subnet model to form a student model provided by an embodiment of the present invention.

[0082] Figure 8 It is a schematic diagram of the network structure of pruning provided by an embodiment of the present invention.

[0083] Fig. 9It is a schematic diagram of the system interface and an effect derivation and estimation diagram provided by an embodiment of the present invention.

[0084] Figure 1 In: 1. Dataset preprocessing module; 2. Teacher model preprocessing module; 3. Subnet model building module; 4. Personalized training module; 5. True loss calculation module; 6. Distillation loss calculation module; 7. Total loss calculation module; 8. Student model building module; DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0086] like Figure 1 As shown, a lightweight image classification neural network architecture system based on knowledge distillation provided by an embodiment of the present invention includes:

[0087] Dataset preprocessing module 1, teacher model preprocessing module 2, subnet model construction module 3, personalized training module 4, true loss calculation module 5, distillation loss calculation module 6, total loss calculation module 7, student model construction module 8;

[0088] The data set preprocessing module 1 is connected to the teacher model preprocessing module 2 and is used to perform image transformation on the data in the data set, including transformation chain implementation, image scaling, horizontal flipping, random cropping, padding, etc., to generate an image training set.

[0089] The teacher model preprocessing module 2 is connected with the data set preprocessing module 1 and the subnet model building module 3, and is used to train the teacher model with a large amount of data, and finally the teacher model generates soft labels;

[0090] A subnet model building module 3 is connected to the teacher model preprocessing module 2, the personalized training module 4, the true loss calculation module 5, the distillation loss calculation module 6, the total loss calculation module 7, and the student model building module 8, and is used to input the image training set and the soft labels obtained by the teacher pre-training into the student subnet model respectively, output the image classification results of the student subnet and save the weight of the student subnet.

[0091] The personalized training module 4 is connected to the subnetwork model building module 3, and is used to introduce a rank-one factor to transfer the teacher knowledge to the subnetwork model. During the training process, adversarial perturbations are added to the input image, the loss value of the student subnetwork is calculated, and the subnetwork shared parameters are updated according to the loss value to improve the robustness of the model.

[0092] The real loss calculation module 5 is connected to the subnetwork model construction module 3, and is used to compare the image reclassified by the subnetwork model with the real image label, and calculate the cross entropy loss between the real label and the subnetwork model output;

[0093] The distillation loss calculation module 6 is connected to the subnet model construction module 3, and is used to compare the category probability distribution obtained by reclassifying the subnet model with the soft label of the teacher model, and calculate the knowledge distillation loss between the soft label and the subnet model prediction;

[0094] The total loss calculation module 7 is connected to the subnetwork model construction module 3 and is used to calculate the total loss by weighted average of the two losses. The total loss is used as the network training error. After updating the iterative training, the final subnetwork model is obtained.

[0095] The student model building module 8 is connected to the subnet model building module 3, and is used to compress the knowledge of the subnet model into a single model, input the above-mentioned image training set for re-iterative training, and prune and quantize the pruned weights at the same time, and further compress the quantized weights through Huffman coding to generate a student model.

[0096] As a preferred technical solution, the teacher model includes a pre-trained ResNet-50 network.

[0097] The embodiment of the present invention provides a lightweight image classification neural network architecture system based on knowledge distillation, which realizes efficient image classification model training and optimization through modular design. First, the data set preprocessing module performs a series of image transformation operations on the original data set, including image scaling, horizontal flipping, random cropping and padding, etc., to generate an image set suitable for training. Subsequently, the teacher model preprocessing module uses a large amount of data to train the teacher model and generates soft labels as an important reference for subsequent student model training.

[0098] Next, the subnetwork model building module inputs the preprocessed image training set and the soft labels generated by the teacher model into the student subnetwork model and outputs the image classification results. At the same time, this module saves the weights of the student subnetwork model to ensure that the existing knowledge can be effectively used during the model training process. The personalized training module introduces a rank-one factor during the training process to transfer the knowledge of the teacher model to the student subnetwork model. In addition, by adding adversarial perturbations to the input image, this module can improve the robustness of the student subnetwork model and dynamically adjust the shared parameters of the model according to the calculated loss value.

[0099] In terms of loss calculation, the module for calculating the true loss is responsible for comparing the classification results of the student subnetwork model with the true labels and calculating the cross entropy loss, thereby measuring the degree of fit of the model to the real data. At the same time, the module for calculating the distillation loss compares the prediction results of the student subnetwork model with the soft labels of the teacher model and calculates the knowledge distillation loss, thereby ensuring that the student model can effectively inherit the knowledge of the teacher model. Finally, the module for calculating the total loss takes the weighted average of the true loss and the distillation loss to obtain the total loss value, and uses this as the network training error to continuously iterate and optimize the student subnetwork model.

[0100] Finally, the student model module is constructed to compress the knowledge of the subnetwork model into a single model. After inputting the image training set, the module is re-trained through iteration, and the model is pruned and quantized in the process to reduce the model complexity and computational overhead. The pruned weights are further compressed through Huffman coding, and finally an efficient and lightweight student model is generated to meet the requirements of model performance and resource usage in practical applications.

[0101] As a preferred technical solution, the subnet model adopts the MobileNetV2 architecture. The forward propagation process of the subnet model includes the following steps: the input image is subjected to feature extraction through several convolutional layers; a low-dimensional feature map is generated after several convolution and pooling operations; the feature map is subjected to a global average pooling layer to obtain a global feature; the global feature is outputted through a fully connected layer.

[0102] As a preferred technical solution, the knowledge distillation process includes the following steps: training phase: the teacher model and the sub-network model receive input images respectively, generate teacher output and sub-network output; the teacher output is used as a soft label, and the error between the sub-network output and the teacher output is calculated by KL divergence; the training goal of the sub-network model is to minimize the weighted sum of the soft label error and the true label error. Inference phase: only the trained student model is used for image classification tasks.

[0103] As a preferred technical solution, the soft label of the teacher model is a label probability value calculated by a softmax function with temperature, where the temperature parameter T is used to smooth the probability distribution. The specific expression of the probability value is:

[0104]

[0105] Among them, q i is the probability value that image i belongs to a certain category, z i are the logits output by the teacher model and T is the distillation temperature.

[0106] As a preferred technical solution, the total loss is calculated by proportionally combining the cross entropy loss and the distillation loss, which is specifically expressed as:

[0107] L=α·T 2 ·H(q T ,p T) +(1-α)·H(y,p)

[0108] Among them, L represents the total loss of the model, H represents the cross entropy loss, and q T and p T are the output probability distributions of the teacher model and the subnetwork model at temperature T (usually set to 1), y is the hard label, and α is the weight used to balance the soft label and hard label losses.

[0109] As a preferred technical solution, the student model is obtained by knowledge compression of the subnet model, including three important steps: weight pruning, weight quantization, and weight compression;

[0110] The weight pruning part requires migrating the subnet model knowledge to the student model, inputting the training data set into the student model, dynamically removing connections with smaller weights during the training process to reduce the weight scale to obtain a sparse network model, and retraining (fine-tuning training) the sparse model to restore its accuracy.

[0111] As a preferred technical solution, the weight pruning is achieved through a dynamic pruning technology, including dynamic evaluation of parameter importance, pruning strategy selection, pruning threshold setting, and periodic pruning;

[0112] Dynamic evaluation of parameter importance means that in the initial stage, the parameter importance score is calculated based on the absolute value of the model parameter or the absolute value of the product of the parameter value and the gradient. As the training steps progress, the exponential moving average calculation method is used to evaluate the parameter importance and uncertainty by combining the parameter value and gradient changes.

[0113] Pruning strategy selection involves determining which parameters or layers need to be pruned and dynamically updating the importance scores and uncertainty values ​​based on the global step number.

[0114] The pruning threshold setting is based on the needs of different stages of the training process. The pruning intensity is dynamically adjusted by setting different thresholds in the warm-up step, the middle stage of training, and the end stage.

[0115] Periodic pruning refers to periodic pruning operations at different stages of training, deciding whether to prune based on the set step interval, thereby effectively controlling model complexity and maintaining model performance.

[0116] The weight quantization part includes two parts: weight quantization and shared weight. The weight quantization refers to quantizing the weight value of the model into a limited number of discrete values, which further reduces the computational complexity and storage requirements by reducing the precision of the weight; the shared weight refers to storing the quantized weight value in a shared weight table, and the weights in the model all refer to the values ​​in the shared weight table, further reducing the storage requirements.

[0117] As a preferred technical solution, weight quantization is achieved by performing k-means clustering on the model weight values ​​and quantizing the weight values ​​to the nearest cluster center.

[0118] The weight compression part uses Huffman coding to compress the quantized weights.

[0119] like Figure 2 As shown, a lightweight image classification neural network architecture method based on knowledge distillation provided by an embodiment of the present invention includes:

[0120] S101, using the data set preprocessing module to perform image transformation on the data in the data set, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding, to generate an image training set; using the teacher model preprocessing module to train the teacher model with a large amount of data, the teacher model finally generates soft labels;

[0121] S102, inputting the image training set and the soft labels obtained by teacher pre-training into the student subnet model respectively by constructing a subnet model module, outputting the image classification result of the student subnet and saving the weight of the student subnet;

[0122] S103, introducing a rank-one factor through a personalized training module to transfer the teacher knowledge to the subnetwork model, adding adversarial perturbations to the input image during training, calculating the loss value of the student subnetwork, and updating the subnetwork shared parameters according to the loss value to improve the robustness of the model;

[0123] S104, comparing the image reclassified by the subnetwork model with the real image label by calculating the real loss module, and calculating the cross entropy loss between the real label and the subnetwork model output;

[0124] S105, comparing the category probability distribution obtained by reclassifying the sub-network model with the soft label of the teacher model through the calculation distillation loss module, and calculating the knowledge distillation loss between the soft label and the prediction of the sub-network model;

[0125] S106, a total loss is calculated by weighting the two losses together through a total loss calculation module, and the total loss is used as a network training error. The final subnetwork model is obtained after updating and iterative training.

[0126] S107, compressing the knowledge of the sub-network model into a single model by constructing a student model module, inputting the above-mentioned image training set for re-iterative training, and pruning and quantizing the pruned weights at the same time, and further compressing the quantized weights through Huffman coding to generate a student model.

[0127] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the lightweight image classification neural network architecture method based on knowledge distillation.

[0128] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the lightweight image classification neural network architecture method based on knowledge distillation.

[0129] Another object of the present invention is to provide an information data processing terminal, which is used to implement the lightweight image classification neural network architecture system based on knowledge distillation.

[0130] Example 1

[0131] As shown in the figure, this embodiment provides a lightweight image classification neural network architecture system based on knowledge distillation, including the following steps:

[0132] Dataset preprocessing: Taking the public dataset as an example, we first deal with problems such as missing values ​​in the dataset, and perform image conversion operations on the processed dataset, including discrete mapping, horizontal flipping, random cropping, edge filling, chain application, etc. The processed dataset is used for supervised network learning.

[0133] In this embodiment, the public dataset CIFAR-100 is used, which includes 60,000 32*32 pixel color pictures. Each picture belongs to one of 100 different categories, which cover a wide range of everyday objects and animals, including cars, birds, cats, dogs, ships, airplanes, etc. There are 600 images in each category, 500 of which are used for training and 100 for testing. Therefore, the dataset is divided into 45,000 training images, 5,000 verification images, and 10,000 test images for supervised learning, verification, and testing.

[0134] Teacher network pre-training: For existing networks, if there is a provided trained network model, it can be loaded directly; for unpublished network models, the training data set is first input for pre-training, and the trained network model and the soft labels of the corresponding data set are obtained and saved before loading.

[0135] In this embodiment, the teacher network used is a pre-trained ResNet-50 network. Depending on the number of seeds set during training, a total of 4 teacher models are trained.

[0136] Build a subnetwork model: add perturbations to the training dataset and soft labels and input them into the subnetwork model, and output the subnetwork model weights and reclassification results.

[0137] The subnet reclassification result is compared with the soft label to obtain the distillation loss, and compared with the hard label to obtain the real loss. The two losses are weighted summed and smoothed with temperature to obtain the final loss. The subnet parameters are updated through back propagation, and the final subnet model is trained.

[0138] The subnet model is a custom convolutional Gateconv2d network structure. Through convolution, the input and output channel weights of the image features are controlled to select the weights to achieve distillation. The convolution layer includes two convolution operations (conv1 and conv2), and introduces a learnable parameter gate to control the selection weight of each channel. After the input data passes through two convolutional layers, the weighted sum of the two convolutional layers is output. Several knowledge distillations are performed and the loss is continuously updated in reverse to achieve feature extraction and fusion at a finer level.

[0139] Compress the student model: Transfer the subnetwork model knowledge to the student model, input the training data set to train the student model, perform flexible pruning during the training process, remove network connections with small weights to reduce the weight scale, and then fine-tune the sparse model to restore accuracy.

[0140] Further compress the parameter scale: The pruned subnetwork model is weighted quantized through k-means clustering, and the parameter values ​​are mapped to discrete values. Then, the parameter scale is further compressed through Huffman coding to obtain a sparse and high-performance image classification neural network.

[0141] In this embodiment, the subnet model obtained by distilling the teacher network can reduce the parameter scale to a certain extent and improve the robustness and training efficiency of the model. However, due to the large parameter scale and difficulty in deployment, the subnet model cannot be called a lightweight model. Therefore, the subnet model is deeply compressed to further reduce the parameter scale while ensuring the performance of the model as much as possible.

[0142] As shown in the figure, this embodiment first trains a high-precision teacher network to generate soft labels for training data sets for model distillation and knowledge transfer. Then, a subnet model is constructed, and the teacher knowledge is transferred through the rank-one factor. The perturbed data is input to train the subnet model, and the total loss value is calculated based on the distillation loss and the true loss. The model parameters are updated by the total loss. The generated subnet model has a certain accuracy and is more efficient than teacher training. Knowledge transfer is performed according to the subnet model and the training data is re-entered for retraining. The importance and uncertainty of the parameters are dynamically evaluated, and the weight scale is reduced periodically and flexibly. The pruned model parameters are quantized and weight compression is performed using Huffman coding to obtain the final high-precision, high-efficiency, lightweight image classification neural network model.

[0143] The teacher model and the student model of this embodiment both contain two convolution blocks (ConvBlock), each of which contains two convolution layers. After each convolution block, the maximum average pooling (max_pool) is used for pooling operation, the pooling window size is 2, and the step size is 2. After the image is input, local features are extracted in each convolution layer, followed by batch normalization to stabilize and accelerate training, and nonlinearity is introduced using the ReLU activation function. After each convolution block is trained, downsampling is performed in the maximum pooling layer to reduce the size of the feature map and retain the most important features.

[0144] As shown in the figure, the process of knowledge distillation is as follows: First, train a complex teacher model, usually a deep neural network with high accuracy. Use the teacher model to predict the training data and generate soft labels. These soft labels are obtained through the softmax function with a temperature parameter. Then build a smaller student model, which will learn the knowledge of the teacher model using a rank-one factor. Add adversarial perturbations to the training data set and input it into the student model for training. The addition of perturbations makes the student model personalized. Define a total loss function, which includes the difference between the student model's prediction of the true label and the difference between the teacher model's soft label. The cross entropy loss function is used to calculate the difference between the student model output and the two labels. The total loss function L is optimized by the back-propagation algorithm, and the student model is trained and the parameters of the student model are adjusted. During the training process, the softmax is calculated using the same temperature parameter T (which is used to generate soft labels for the teacher). After multiple iterations of training, the student model will gradually learn the knowledge of the teacher model. The final student model should significantly reduce the model complexity and the number of parameters while maintaining high performance.

[0145] In this embodiment, the teacher model uses optax.softmax_cross_entropy_with_integer_labels to calculate the cross entropy loss; the student model calculates the weighted sum of the true loss and the distillation loss. The true loss is obtained by calculating the cross entropy loss using the predicted label and the true label; the distillation loss is obtained by calculating the cross entropy loss using the predicted label and the soft label after temperature scaling.

[0146] For the real loss, the update iteration formula is:

[0147] L real =H(y,p)

[0148] For the distillation loss, the update formula is:

[0149] L kd =T 2 ·H(q T ,p T )

[0150] In this embodiment, the total loss function formula is:

[0151] L=α·L kd +(1-α)·L real

[0152] Among them, L represents the total loss, L real represents the true loss (the loss between the predicted result and the true label of the image), L kd represents the distillation loss (the loss between the prediction result and the soft label), H represents the cross entropy loss, and q T and p T are the output probability distributions (soft labels) of the teacher model and the student model at temperature T (usually set to 1), y is the hard label, and α is the weight used to balance the soft label and hard label losses.

[0153] The calculation formula of the soft tag is:

[0154]

[0155] where q i Indicates the probability value that the image belongs to category i, z i are the logits output by the teacher model and T is the temperature coefficient.

[0156] This embodiment is trained based on the deep learning open source framework. In terms of training parameter configuration, the SGD optimizer is used, the learning rate is set to 0.001, the total training rounds are 10000, and the rate is decayed to 0.9 times the current rate every 200 training rounds. The number of images trained in each round (batch_size) is set to 128, α is set to 1.0, and the distillation temperature is set to 4.0.

[0157] The soft labels are the probability distributions output by the teacher model, which contain rich information about the similarities between classes, rather than just a hard classification. This probability distribution can be adjusted by the temperature parameter T to make the output probability distribution smoother, thereby better transferring knowledge.

[0158] The temperature parameter T is used to smooth the output of softmax. During the training process, the output of the teacher model will be processed by the softmax function and T will be introduced. When T>1, the output of softmax will become smoother and can reflect the similarity between different categories. The student model will also use the same temperature parameter to learn the soft labels of the teacher model during training.

[0159] As shown in the figure, the image and category of the present embodiment are first input into the convolution layer, and each convolution layer performs a convolution operation through a 3*3 convolution kernel to extract the local features of the image. The convolution layer implements the feature extraction operation through a certain number of filters. After each convolution layer, the features are normalized by batch normalization (BatchNorm). Then the ReLU activation function is used to introduce nonlinearity, set all negative values ​​to zero, keep the positive values ​​unchanged, and let the model learn more complex features. At the end of each ConvBlock training, the feature map is downsampled by the maximum pooling operation with a window size of 2*2 and a step length of 2, reducing the size of the feature map while retaining the most important features. Multi-level and multi-scale features are extracted from the input through the joint action of the convolution layer and the pooling layer, providing high-quality feature representation for the subsequent fully connected (Dense) layer and classification tasks. Then, the category to which the output image belongs is calculated in the fully connected layer according to the extracted feature representation.

[0160] The filters, conv1 and conv2, are convolved with different numbers of filters (feature maps), respectively, and the number of filters is determined by the parameter filters. In this embodiment, the ConvBlock of the teacher model uses 32 and 64 filters respectively; the ConvBlock of the student model uses 16 and 32 filters respectively. Both the teacher and the student use a 3*3 convolution kernel, and perform maximum pooling after completing two layers of convolution (one convolution block).

[0161] As shown in the figure, the deep compression process of the subnet model is as follows: in the initial stage, the parameter importance score is calculated according to the absolute value of the product of the model weight (parameter) value and the gradient. As the training process progresses, the exponential moving average is used to dynamically evaluate the parameter importance and uncertainty. During the training process, the pruning threshold is sorted by the importance and uncertainty scores of all parameters, and the weight value corresponding to the score sorting vector is retained in proportion. During training, the theoretical pruning threshold Ts changes with the time step S and is dynamically adjusted according to the initial threshold Ts_ini and the final threshold Ts_fin. Each layer dynamically adjusts the pruning intensity according to the threshold in the initial warm-up, intermediate stage and final stage. At the same time, at different stages of training, it is determined whether to prune in the current stage according to the value of the periodic pruning strategy (mask_ind), effectively controlling the model complexity and maintaining the model performance. The network structure before and after pruning is shown in the figure. Then, the model parameter values ​​are clustered by k-means to quantize the parameter values ​​into the nearest cluster center and store them in the shared weight table. Huffman coding is used to achieve quantized parameter compression, further reducing the parameter scale.

[0162] The parameter importance dynamic scoring formula used is:

[0163]

[0164] To score the importance of the smoothing parameters, an exponential moving average is used. The formula for calculating the exponential moving average of the parameters is:

[0165] S exp [ω]=β1·S exp [ω]+(1-β1)·S ipt [ω]

[0166] For uncertainty, the uncertainty exponential moving average is used, and its calculation formula is:

[0167]

[0168] Where ω represents the current weight, L represents the loss value of the current training step, and S ipt [ω] represents the importance score of parameter ω, S exp [ω] represents the exponential moving average of ω, S unc [ω] represents the exponential moving average of the uncertainty of ω, U s It represents the updated number of training steps (current number of training steps), β1 and β2 are two probability values ​​not greater than 1, which determine the proportion of pruning and are also specified parameters.

[0169] The proportion P of the weights retained according to the score ranking is:

[0170] P=1-T s

[0171] Among them, P is the ratio of the number of reserved network connections, T S is the theoretical pruning threshold (a floating point number less than 1).

[0172] The pruning threshold in the intermediate stage of training is set to use a cubic coefficient to smooth the threshold from the initial to the final value based on the ratio of the current step to the intermediate step, so as to achieve dynamic adjustment of the pruning threshold. The calculation formula of the pruning threshold is:

[0173] In this embodiment, the pruning threshold in the initial warm-up phase is:

[0174] T s =T s_ini

[0175] The pruning threshold for ending the warm-up phase is:

[0176] T s =T s_fin

[0177] In this embodiment, when entering the preheating intermediate stage, the calculation formula for the number of preheating steps in the current stage is:

[0178] S sw =(W ini +W fin )×S w

[0179] The cubic coefficient used for the smoothing threshold is calculated as:

[0180]

[0181] The threshold calculation formula for the intermediate stage is:

[0182] T S =T s_fin +(T s_ini -T s_fin )×(C mul ) 3

[0183] Where T s_ini and T s_fin Represent the initial threshold and the end threshold respectively, S sw Indicates the number of preheating steps, W ini and W fin Respectively represent the initial preheating value and the final preheating value, S w Indicates the number of initial warm-up steps, C mul represents the cubic coefficient, S, S sw , S w , S tThey represent the current number of steps, stage preheating number of steps, initial preheating number of steps, and total number of steps respectively.

[0184] The theoretical pruning threshold calculation method of the entire pruning strategy is as follows:

[0185]

[0186] The periodic pruning strategy mask_ind is set to a bool value, according to the current training step number G S and the number of training steps δ T The ratio of is used to control whether to perform pruning in the current step. The calculation method of mask_ind is:

[0187]

[0188] In the training parameter configuration, the initial warm-up step number S W is set to 5, β1 and β2 are set to 0.85 and 0.95 respectively, and δ T Set to 10, T s_ini , T s_fin , W ini and W fin Set them to 1, 0.1, 1 and 5 respectively.

[0189] Example 2

[0190] This embodiment provides a storage medium, which may be a ROM, RAM, disk, CD or other storage medium, and the storage medium stores one or more programs. When the program is executed by the processor, the lightweight image classification neural network architecture system based on knowledge distillation of Embodiment 1 is implemented.

[0191] Example 3

[0192] This embodiment provides a computing device, which can be a desktop computer, a laptop computer, a smart phone, a tablet computer or other terminal device with a display function. The computing device includes a processor and a memory, and the memory stores one or more programs. When the processor executes the program stored in the memory, the lightweight image classification neural network architecture system based on knowledge distillation of Example 1 is implemented.

[0193] The technical solution of the present invention can be widely used in agricultural equipment, especially agricultural equipment equipped with intelligent classification programs, such as drones and automated agricultural machinery systems. The following are specific application scenarios:

[0194] 1. Drone monitoring:

[0195] Drones are equipped with intelligent image classification systems that can monitor fruit trees in real time over a large area and quickly identify whether there are pests and diseases. Through high-altitude photography and image processing, drones can automatically detect the health of fruit trees, generate detailed pest and disease distribution maps, and transmit data to farmers' mobile phones in real time.

[0196] 2. Automated agricultural machinery and equipment:

[0197] Modern agricultural machinery and equipment, such as automated tillers and sprayers, can be equipped with this technology to achieve precise crop planting and pesticide application. For example, a sprayer can automatically adjust the spraying amount based on the image classification results to ensure that the pest-infested area is properly treated and avoid wasting pesticides.

[0198] 3. Intelligent greenhouse management:

[0199] In a greenhouse environment, monitoring equipment equipped with intelligent classification programs can analyze plant growth status in real time, identify pests and diseases or growth abnormalities, and automatically trigger irrigation or ventilation systems to optimize the crop growth environment.

[0200] Actual application effect:

[0201] Through these intelligent agricultural machinery and equipment, farmers can manage crops more efficiently, reduce labor costs, improve production efficiency, and also reduce yield losses caused by pests and diseases. Such intelligent agricultural machinery and equipment will greatly promote the development of intelligent agriculture, increase agricultural output, and enhance the sustainability of the agricultural economy.

[0202] Conduct an in-depth theoretical analysis of the effects of the invention model.

[0203] 1. Model structure advantages: The inventive model adopts a more streamlined and efficient network architecture in its structural design. For example, by combining feature fusion, multi-scale feature extraction and other methods, the model reduces the number of parameters while maintaining a high classification accuracy, thereby improving the model's reasoning efficiency.

[0204] 2. Optimization of distillation technology: Compared with traditional knowledge distillation methods, the invented model conducts deeper mining and refinement of the teacher model’s knowledge. For example, it adopts weighted average or other methods to more reasonably integrate the features of multiple teacher models, so that the student model can learn more effectively.

[0205] 3. Model adaptability: The inventive model is designed to be optimized for resource-constrained devices (such as mobile devices). This optimization is not only reflected in the model size and inference time, but also reduces the reliance on high-precision data, so that it can still maintain good classification performance on low-quality images (such as low-resolution or noisy images).

[0206] 4. Software simulation: Through simulation, the performance of the invented model in actual scenarios can be estimated. For example, in a simulation environment, the inference time, accuracy change, and robustness of the model under various image interferences (such as lighting changes, blur, and occlusion) under different computing resource conditions can be tested.

[0207] The expected results of the model are:

[0208] High accuracy: Due to the full utilization of the teacher model’s knowledge, the invented model should maintain high classification accuracy on the test dataset.

[0209] Smaller model size: Due to the effective compression and refinement of features, the model size should be significantly smaller than the original teacher model.

[0210] Fast inference: Through optimizations, inference time should be significantly reduced, especially on mobile devices.

[0211] Adapt to diverse scenarios: The model should maintain high robustness in different application scenarios (such as low light, occlusion, and noise) to prove its effectiveness in actual deployment.

[0212] Through these theoretical analyses and simulations, we can infer that the invented model will show superior performance in practical applications, especially in resource-constrained environments.

[0213] like Fig. 9 This is a schematic diagram of the system interface of the present invention and a graph of the effect derivation and estimation. This graph is a normalized line graph of the theoretical performance of each model on different indicators. The model of the present invention is expected to have a good comprehensive performance in various indicators, especially in terms of reasoning time and model size, which should be better than the teacher model; the accuracy and ROC indicators should be better than the KD model. In general, the model of the invention is expected to achieve a good balance between performance and efficiency, and is suitable for deployment requirements in practical applications.

[0214] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0215] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A lightweight image classification neural network architecture system based on knowledge distillation, characterized in that: include: Dataset preprocessing module, teacher model preprocessing module, subnet model building module, personalized training module, true loss calculation module, distillation loss calculation module, total loss calculation module, student model building module; The dataset preprocessing module is connected to the teacher model preprocessing module and is used to perform image transformation on the data in the dataset, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding, to generate an image training set; The teacher model preprocessing module is connected with the data set preprocessing module and the subnet model building module to train the teacher model with a large amount of data, and finally the teacher model generates soft labels; A subnet model module is constructed, connected with the teacher model preprocessing module, the personalized training module, the true loss calculation module, the distillation loss calculation module, the total loss calculation module, and the student model construction module, and is used to input the image training set and the soft labels obtained by the teacher pre-training into the student subnet model respectively, output the image classification result of the student subnet and save the weight of the student subnet; The personalized training module is connected to the subnetwork model building module and is used to introduce a rank-one factor to transfer the teacher knowledge to the subnetwork model. During the training process, adversarial perturbations are added to the input image, the loss value of the student subnetwork is calculated, and the subnetwork shared parameters are updated according to the loss value. The module for calculating the true loss is connected to the module for building the subnetwork model. It is used to compare the images reclassified by the subnetwork model with the true image labels and calculate the cross entropy loss between the true labels and the output of the subnetwork model. The module for calculating the distillation loss is connected to the module for building the subnetwork model. It is used to compare the category probability distribution obtained by reclassifying the subnetwork model with the soft label of the teacher model, and calculate the knowledge distillation loss between the soft label and the prediction of the subnetwork model. The total loss calculation module is connected to the subnet model construction module and is used to calculate the total loss by weighted average of the two losses. The total loss is used as the network training error. After updating the iterative training, the final subnet model is obtained. The student model module is constructed and connected to the subnet model construction module to compress the knowledge of the subnet model into a single model, input the above image training set for re-iterative training, and prune and quantize the pruned weights at the same time. The quantized weights are further compressed through Huffman coding to generate a student model.

2. The lightweight image classification neural network architecture system based on knowledge distillation as claimed in claim 1, characterized in that: The teacher model includes a pre-trained ResNet-50 network.

3. The lightweight image classification neural network architecture system based on knowledge distillation as claimed in claim 1, characterized in that: The subnet model adopts the MobileNetV2 architecture; the forward propagation process of the subnet model includes the following steps: the input image is subjected to feature extraction through several convolutional layers; a low-dimensional feature map is generated after several convolution and pooling operations; the feature map is subjected to a global average pooling layer to obtain a global feature; and the global feature is outputted through a fully connected layer.

4. The lightweight image classification neural network architecture system based on knowledge distillation as claimed in claim 1, characterized in that: The knowledge distillation process includes the following steps: training phase: the teacher model and the subnetwork model receive input images respectively to generate teacher output and subnetwork output; the teacher output is used as a soft label, and the error between the subnetwork output and the teacher output is calculated by KL divergence; the training goal of the subnetwork model is to minimize the weighted sum of the soft label error and the true label error; inference phase: only the trained student model is used for image classification tasks.

5. The lightweight image classification neural network architecture system based on knowledge distillation as claimed in claim 1, characterized in that: The soft label of the teacher model is a label probability value calculated by a softmax function with temperature, where the temperature parameter T is used to smooth the probability distribution; the specific expression of the probability value is: Where zi is the logits output by the teacher model, and T is the distillation temperature; As a preferred technical solution, the total loss is calculated by proportionally combining the cross entropy loss and the distillation loss, which is specifically expressed as: L=α·T 2 ·H(q T ,p T) +(1-α)·H(y,p) Where H represents the cross entropy loss, qT and pT are the output probability distributions of the teacher model and the subnetwork model at temperature T, respectively, y is the hard label, and α is the weight used to balance the soft label and hard label losses.

6. The lightweight image classification neural network architecture system based on knowledge distillation as claimed in claim 1, characterized in that: The student model is obtained by knowledge compression of the subnet model, including three important steps: weight pruning, weight quantization, and weight compression; The weight pruning part needs to transfer the subnet model knowledge to the student model, input the training data set to the student model, and dynamically remove the connections with smaller weights during the training process; The weight pruning is achieved through a dynamic pruning technology, including dynamic evaluation of parameter importance, pruning strategy selection, pruning threshold setting, and periodic pruning; Dynamic evaluation of parameter importance means that in the initial stage, the parameter importance score is calculated based on the absolute value of the model parameter or the absolute value of the product of the parameter value and the gradient; Pruning strategy selection involves determining which parameters or layers need to be pruned and dynamically updating and adjusting the importance scores and uncertainty values ​​based on the global step number; The pruning threshold setting is based on the needs of different stages of the training process. The pruning intensity is dynamically adjusted by setting different thresholds in the warm-up step, the middle stage of training, and the end stage. Periodic pruning refers to periodic pruning at different stages of training, and whether to prune is determined based on the set step interval; The weight quantization part includes two parts: weight quantization and shared weight.

7. A lightweight image classification neural network architecture method based on knowledge distillation for implementing the lightweight image classification neural network architecture system based on knowledge distillation as claimed in any one of claims 1 to 6, characterized in that: The lightweight image classification neural network architecture method based on knowledge distillation includes: Step 1: Perform image transformation on the data in the data set through the data set preprocessing module, including transformation chain implementation, image scaling, horizontal flipping, random cropping, and padding to generate an image training set; train the teacher model with a large amount of data through the teacher model preprocessing module, and finally the teacher model generates soft labels; Step 2, inputting the image training set and the soft labels obtained by teacher pre-training into the student subnet model respectively by constructing a subnet model module, outputting the image classification result of the student subnet and saving the weight of the student subnet; Step 3: The teacher knowledge is transferred to the subnetwork model by introducing the rank-one factor through the personalized training module. During the training process, adversarial perturbations are added to the input image, the loss value of the student subnetwork is calculated, and the subnetwork shared parameters are updated according to the loss value. Step 4: Compare the image reclassified by the subnetwork model with the real image label by calculating the real loss module, and calculate the cross entropy loss between the real label and the subnetwork model output; Step 5: Compare the category probability distribution obtained by reclassifying the subnetwork model with the soft label of the teacher model through the calculation distillation loss module, and calculate the knowledge distillation loss between the soft label and the prediction of the subnetwork model; Step 6: The total loss is calculated by adding the weighted sum of the two losses through the total loss calculation module. The total loss is used as the network training error. After updating the iterative training, the final subnetwork model is obtained. Step 7: Compress the knowledge of the subnetwork model into a single model by constructing a student model module, input the above image training set for re-iterative training, prune and quantize the pruned weights at the same time, and further compress the quantized weights through Huffman coding to generate a student model.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the lightweight image classification neural network architecture method based on knowledge distillation as claimed in claim 7.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the lightweight image classification neural network architecture method based on knowledge distillation as described in claim 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the lightweight image classification neural network architecture system based on knowledge distillation as described in any one of claims 1-6.

Images