A neural network classification model knowledge distillation method of passive domain data

Abstract

The application relates to a passive domain data neural network classification model knowledge distillation method, data is crawled, all samples of a candidate data set are pretreated as candidate samples, a category similarity matrix is constructed based on the classification layer weight of a teacher model, logits values of all candidate samples are calculated based on the teacher model and are subjected to SoftMaxt operation, and the difference degrees of all candidate samples and the teacher model field distribution are calculated; all samples are sorted from small to large according to the difference degrees, the first M samples are selected as a knowledge distillation data set, and a student model is obtained by distilling the knowledge of the teacher model. The application avoids that an unreasonable high-confidence out-of-domain sample is wrongly selected, does not completely need source domain data, has higher classification accuracy and wider model applicability, has faster selection speed, can perform knowledge distillation on a pre-training model under the condition that resources are less, and can solve various downstream tasks based on knowledge distillation.

Description

Technical Field

[0001] This invention relates to the technical field of computation, calculation, or counting, and particularly to a knowledge distillation method for neural network classification models of passive domain data in the fields of machine learning and data mining. Background Technology

[0002] In recent years, deep neural networks have achieved remarkable results in visual classification and have been widely used across various industries. A prerequisite for the superior performance of neural networks is that the test and training data follow an independent and identically distributed (i.i.d.) distribution. However, in the real world, when deploying trained models on various platforms, as business grows, we often need to upgrade and maintain these models. For example, we might need to compress the model for deployment on edge devices or increase its classification capabilities (incremental learning). High-performance networks are often accompanied by massive network architectures, meaning that achieving high performance requires significant computational and memory consumption. This makes it impossible to deploy most large, high-performance networks on terminal devices with limited memory resources or requiring fast response times, such as traffic cameras and autonomous driving systems. Furthermore, upgrading and maintaining these models also presents considerable challenges.

[0003] Significant progress has been made in this technical challenge in recent years, particularly in model compression based on knowledge distillation. Many existing knowledge distillation methods are highly effective for training compact deep models when we have direct access to the source domain training dataset. However, the effectiveness of knowledge distillation relies on the availability of source domain data. Current emphasis on data privacy and security dictates that the training set should be kept confidential after model training, hindering many downstream tasks based on large models. Knowledge distillation-based model compression also faces this challenge. Furthermore, the large size of the source domain dataset makes transportation and storage inconvenient, preventing access to the source domain data. In reality, we often only obtain a pre-trained deep neural network model.

[0004] In existing technologies, methods based on maximum confidence or minimum entropy are used to measure whether a sample belongs to the learning domain distribution of a given model. When a neural network model is actually running, it is very likely to fail when the input data is inconsistent with the distribution of the training dataset, because it may give a high confidence prediction for samples outside the domain. This will result in a large number of samples outside the domain in the filtered pseudo-domain samples.

[0005] Given a pre-trained large neural network model, if the model's source domain data can be accessed, it will be easy to achieve the purpose of knowledge transfer by performing knowledge distillation on the model. However, in reality, there is often only one pre-trained model, and it is not possible to effectively access the model's source domain data. Summary of the Invention

[0006] This invention addresses the problems existing in the prior art and provides a knowledge distillation method for neural network classification models with passive domain data. This method involves selecting image data that conforms to the distribution of the model's training set from pre-trained models for neural network knowledge distillation. It considers the internal category information of the predicted sample values ​​to avoid the incorrect selection of unreasonably high-confidence out-of-domain samples. The method uses the filtered data, which approximates the distribution within the model's domain, to perform knowledge distillation on the pre-trained model, thereby achieving the purpose of transferring knowledge from the pre-trained model. This improves the transferability of the pre-trained model to downstream tasks and the scalability of its subsequent maintenance. It can be applied to many deep learning fields such as model compression, incremental learning, and transfer learning based on knowledge distillation.

[0007] The technical solution adopted in this invention is a knowledge distillation method for a neural network classification model of passive domain data, the method comprising the following steps:

[0008] Step 1: Crawl data to obtain candidate datasets;

[0009] Step 2: Preprocess all samples in the candidate dataset to obtain candidate samples;

[0010] Step 3: Construct a category similarity matrix C based on the classification layer weights of the teacher model;

[0011] Step 4: Calculate the logits values ​​of all candidate samples based on the teacher model and perform the SoftMaxt operation. Finally, calculate the difference between all candidate samples and the domain distribution of the teacher model.

[0012] Step 5: Sort all samples from smallest to largest according to their dissimilarity;

[0013] Step 6: Select the first M samples as the knowledge distillation dataset, M≥100;

[0014] Step 7: Perform knowledge distillation on the teacher model based on the knowledge distillation dataset to obtain the student model.

[0015] Preferably, in step 1, if the categories of the teacher model are known, the data corresponding to the category keywords are crawled directly; otherwise, it is determined whether the known teacher model is classifying a certain superclass. If so, the superclass name is used as a keyword to crawl relevant data; otherwise, relevant type data of any category is crawled or a publicly available dataset is used.

[0016] Preferably, the preprocessing includes data cleaning and adjusting the input candidate samples to a uniform format.

[0017] Preferably, step 3 includes the following steps:

[0018] Step 3.1: Calculate the correlation of each category using the classification layer weights of the teacher model to obtain equation (1).

[0019]

[0020] Where C(i,j) represents the similarity between category i and category j, w i and w j Here, i and j are the weights for category i and category j, respectively, where i ≠ j;

[0021] Step 3.2: Repeat step 3.1 until the similarity between all categories is calculated, and matrix C′ is obtained;

[0022] Step 3.3: Normalize matrix C′ so that the sum of each row of the matrix is ​​1, and obtain the final category similarity matrix C.

[0023] Preferably, step 4 includes the following steps:

[0024] Step 4.1: For any input candidate sample x, predict it using the teacher model to obtain the output logits value T(x,θ). T ), where T represents the teacher model, θ T The parameters representing the teacher model;

[0025] Step 4.2: Calculate the SoftMax output of the logits value for each sample to obtain the predicted label. Temp is a temperature constant, S i (x; Temp) represents the SoftMax operation, as shown in equation (2).

[0026]

[0027] Among them, T i and T j Let represent the outputs of the i-th and j-th nodes of the teacher model, and N represent the number of categories; for a given sample x, ... Let mc be the maximum confidence level.

[0028] Preferably, in step 4.1, for any candidate sample x, the logits value predicted as class k is given by equation (3).

[0029] T i (x,θ T )=[g (x) ,1] T w k (3)

[0030] Among them, g (x) This represents the feature vector of the penultimate layer activation of the teacher model, which is concatenated with "1" to explain bias, wk These are the weights and biases of the k-th class in the teacher model.

[0031] Preferably, in step 4, the degree of difference is given by equation (4).

[0032]

[0033] Among them, X W Given a candidate sample dataset, mc(.) represents the maximum confidence score of the predicted sample, X sub For the selected knowledge distillation dataset, C k For the k-th row of the category similarity matrix, D KL Let S be the KL divergence and S represent the SoftMax operation.

[0034] Preferably, the selected domain dataset is input into the teacher model and the student model, and the distillation loss L is calculated. kD Fixed teacher model parameters, using L kD Loss update student model, distillation loss L kD Satisfying equation (5),

[0035] L ＞D =αL soft +βL hard (5)

[0036] Among them, α, β≥0 and α+β=1, L soft The loss between the teacher model and the student model is calculated using soft labels, L hard It is the cross-entropy loss of the student model under the real labels.

[0037] Preferably,

[0038] in,

[0039] This represents the output of the teacher model predicting a given sample as class i. This represents the output of the student model predicting class i for a given sample, where Temp is the temperature parameter, N represents the number of classes, and k corresponds to each class.

[0040] Preferably,

[0041] in,

[0042] With c i For the true label of the sample, This represents the output of the student model predicting class i for a given sample, where Temp is the temperature parameter, N represents the number of classes, and k corresponds to each class.

[0043] This invention relates to a knowledge distillation method for a neural network classification model of passive domain data. The method involves crawling data to obtain a candidate dataset, preprocessing all samples in the candidate dataset as candidate samples, constructing a class similarity matrix C based on the classification layer weights of the teacher model, calculating the logits values ​​of all candidate samples based on the teacher model and performing a SoftMaxt operation, and calculating the difference between the domain distribution of all candidate samples and the teacher model. Based on the difference, all samples are sorted from smallest to largest, and the top M samples are selected as the knowledge distillation dataset. Knowledge distillation is then performed on the teacher model based on the knowledge distillation dataset to obtain the student model.

[0044] The beneficial effects of this invention are as follows:

[0045] (1) Make full use of the information between categories provided by the teacher classifier model to avoid unreasonable high-confidence out-of-domain samples being wrongly selected to satisfy the domain distribution of the pre-trained model;

[0046] (2) Even if some neural network models can only provide the weights of the classifier layer of the teacher model, knowledge distillation can be performed by filtering candidate data using this method, without requiring source domain data.

[0047] (3) Compared with the traditional method of using maximum confidence or minimum entropy to select candidate datasets, the present invention has higher classification accuracy and wider model applicability;

[0048] (4) Compared with other passive domain data knowledge distillation techniques, the present invention has a faster filtering speed and requires less computing resources, enabling knowledge distillation of pre-trained models with limited resources.

[0049] (5) The effectiveness was verified on the CIFAR-10 and CIFAR-100 datasets. The accuracy of the knowledge distillation student model reached 94.68% on the CIFAR-10 validation set and 72.33% on the CIFAR-100 validation set. It can solve various downstream tasks based on knowledge distillation, including incremental learning and transfer learning based on knowledge distillation. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of passive domain data knowledge distillation according to the present invention;

[0051] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0053] This invention relates to a knowledge distillation method for a neural network classification model of passive domain data. The method involves acquiring a dataset from the Internet that has a domain distribution difference of less than a threshold from the training set of the pre-trained model (teacher model) using a pre-trained network model, and then transferring the knowledge of the pre-trained model through knowledge distillation.

[0054] The method includes the following steps:

[0055] Step 1: Crawl data to obtain candidate datasets;

[0056] In step 1, if the categories of the teacher model are known, the data corresponding to the category keywords are crawled directly. Otherwise, it is determined whether the known teacher model is classifying a certain superclass. If so, the superclass name is used as a keyword to crawl relevant data. Otherwise, relevant type data of any category is crawled or a publicly available dataset is used.

[0057] In this invention, to obtain a dataset close to the source domain of the teacher model, a large amount of data needs to be collected first. Taking image data as an example, generally, data is crawled from the internet or public large-scale datasets are used as candidate datasets. For example, when crawling data from Google, if the specific category of the teacher model classifier is known, images with category keywords are directly crawled from Google, such as cars, cats, and dogs in the CIFAR-10 dataset. If the specific category is unknown but it is known that the teacher model classifies a certain superclass, the superclass name is used as a keyword to crawl images with relevant keywords, such as animals, numbers, buildings, and flowers. When only the teacher model is available, and there is no other classifier category information, images of any category are directly crawled from Google or the ImageNet dataset is directly used as a candidate dataset.

[0058] In this invention, the size of the candidate dataset is determined by factors such as the size of the teacher model and the number of categories. Generally, a larger candidate dataset yields better results, but also requires more computational resources. Therefore, the size of the candidate dataset can be determined based on computational resources and the desired effect. Typically, the size of the candidate dataset should be no less than 20 times the size of the teacher model's training set. If the size of the teacher model's training set is unknown, the candidate dataset should be set to at least 1 million data points.

[0059] Step 2: Preprocess all samples in the candidate dataset to obtain candidate samples;

[0060] The preprocessing includes data cleaning and adjusting the input candidate samples to a uniform format.

[0061] In this invention, data cleaning refers to cleaning out data with obvious format errors or missing data. Taking images as an example, adjusting the input candidate samples to a uniform format means adjusting the images to the same size so that the length and width of each image are equal, and uniformly converting the image format to png or jpeg, etc.

[0062] Step 3: Construct a category similarity matrix C based on the classification layer weights of the teacher model;

[0063] In the pre-trained model, the last layer is a fully connected layer with softmax non-linearity. The low-dimensional features obtained by the teacher encoder will be predicted as each class by the weights of the classifier. The weight of a class in the classifier is actually the modeling of the low-dimensional features of that class of samples by the pre-trained model classifier.

[0064] Step 3 includes the following steps:

[0065] Step 3.1: Calculate the correlation of each category using the classification layer weights of the teacher model to obtain equation (1).

[0066]

[0067] Where C(i,j) represents the similarity between category i and category j, w i and w j Here, i and j are the weights for category i and category j, respectively, where i ≠ j;

[0068] Step 3.2: Repeat step 3.1 until the similarity between all categories is calculated, and matrix C′ is obtained;

[0069] Step 3.3: Normalize matrix C′ so that the sum of each row of the matrix is ​​1, and obtain the final category similarity matrix C.

[0070] In this invention, the key to determining whether any sample x is predicted as class k is the combination of the features before the classifier and the template w of class k. k The alignment of the weights and biases of the classifier for the kth class; when the encoded features are consistent with the weights, the softmax value predicted for the kth class is the largest, and conversely, when the encoded features are opposite to the weights, the softmax value predicted for the kth class is the smallest; based on this, it is easy to calculate the correlation of each class through the classifier weights.

[0071] In this invention, a category similarity matrix is ​​obtained by calculating the similarity between each category. In addition to the highest similarity between the categories themselves, for example, the similarity between trucks and cars is obviously higher than that between cars and birds or deer.

[0072] In this invention, in order to reflect the similarity between the categories of the predicted vectors, the similarity matrix needs to be Softmax normalized, and a temperature is given to make the normalized probability distribution smoother. The category similarity matrix is ​​normalized row by row so that the sum of the probabilities of each row is 1.

[0073] Step 4: Calculate the logits values ​​of all candidate samples based on the teacher model and perform the SoftMaxt operation. Finally, calculate the difference between all candidate samples and the domain distribution of the teacher model.

[0074] Step 4 includes the following steps:

[0075] Step 4.1: For any input candidate sample x, predict it using the teacher model to obtain the output logits value T(x,θ). T ), where T represents the teacher model, θ T The parameters representing the teacher model;

[0076] In step 4.1, for any candidate sample x, the predicted logits value of class k is given by equation (3).

[0077] T i (x,θ T )=[g (x) ,1] T w > (3)

[0078] Among them, g (x) This represents the feature vector of the penultimate layer activation of the teacher model, which is concatenated with "1" to explain bias, w k These are the weights and biases of the k-th class in the teacher model.

[0079] Step 4.2: Calculate the SoftMax output of the logits value for each sample to obtain the predicted label. Temp is a temperature constant, S i (x; Temp) represents the SoftMax operation, as shown in equation (2).

[0080]

[0081] Among them, T i and T j Let represent the outputs of the i-th and j-th nodes of the teacher model, and N represent the number of categories; for a given sample x, ... Let mc be the maximum confidence level.

[0082] In step 4, the degree of difference is given by equation (4).

[0083]

[0084] Among them, X W Given a candidate sample dataset, mc(.) represents the maximum confidence score of the predicted sample, X sub For the selected knowledge distillation dataset, C k For the k-th row of the category similarity matrix, D KL Let S be the KL divergence and S represent the SoftMax operation.

[0085] Step 5: Sort all samples from smallest to largest according to the difference; the smaller the difference, the more the sample conforms to the distribution of the pre-trained model's source domain, and the larger the difference, the more the sample deviates from the distribution of the pre-trained model's domain.

[0086] Step 6: Select the first M samples as the knowledge distillation dataset, M≥100; Select the first M samples from the sorted samples as the selected dataset that is close to the domain distribution of the pre-trained model. The smaller M is, the closer the data is to the domain distribution of the pre-trained model. The larger M is, the greater the deviation between the data distribution and the domain distribution of the pre-trained model. M is a hyperparameter and will be determined depending on the downstream task.

[0087] Step 7: Perform knowledge distillation on the teacher model based on the knowledge distillation dataset to obtain the student model.

[0088] Input the selected domain dataset into the teacher model and student model, and calculate the distillation loss L. kD Fixed teacher model parameters, using L kD Loss update student model, distillation loss L kD Satisfying equation (5),

[0089] L kD =αL sofZ +βL hard (5)

[0090] Among them, α, β≥0 and α+β=1, L sofZ The loss between the teacher model and the student model is calculated using soft labels, L hard It is the cross-entropy loss of the student model under the real labels.

[0091]

[0092] in,

[0093] This represents the output of the teacher model predicting a given sample as class i. This represents the output of the student model predicting class i for a given sample, where Temp is the temperature parameter, N represents the number of classes, and k corresponds to each class.

[0094]

[0095] in,

[0096] With c i For the true label of the sample, This represents the output of the student model predicting class i for a given sample, where Temp is the temperature parameter, N represents the number of classes, and k corresponds to each class.

[0097] In this invention, α is generally set to 0.7, β is generally set to 0.3, and T is usually set to 3.

[0098] In this invention, a specific embodiment is given:

[0099] Step 1: Given a pre-trained model on CIFAR-10 with no other classifier category information, the ImageNet2012 dataset is used directly as the candidate dataset; the candidate dataset contains 1,281,167 RGB images in JPEG format, with 1,000 categories.

[0100] Step 2: Delete obvious erroneous data obtained due to garbled characters during the collection process, delete data with missing format errors, and uniformly change the non-JPEG format images to JPEG format; after resizing the images, uniformly adjust the images in the CIFAR-10 dataset to 32×32 pixel color images, or crop all images to the same size as needed for prediction by the pre-trained model.

[0101] The above operations can be performed on the dataset using deep learning frameworks such as PyTorch and TensorFlow. The torchvision package in PyTorch already provides operations such as image cropping and certain data augmentation, i.e., image normalization. After completion, the image input can be loaded into DataLoader through the provided framework interface for subsequent training.

[0102] Step 3: If a teacher model is already provided, it can be used directly, or a pre-trained teacher model can be downloaded from the Internet; select the ResNet34 network model as the teacher model, and train the teacher model using the stochastic gradient descent optimizer provided by PyTorch according to the cross-entropy loss function. After 200 iterations, the model of the last iteration is used as the teacher model.

[0103] For the teacher model, the weight matrix W of its classification layer is retained, with dimensions of 512×10, where 512 represents the feature length and 10 represents the number of categories in the teacher model classifier; the similarity between category i and category j is calculated, w kThe weights and biases of the k-th class in the classifier are used to obtain a class similarity matrix with a dimension of 10×10. The weight correlation matrix is ​​mapped to the probability space of (0,1) to obtain a probability class correlation matrix C with a dimension of 10×10, where each row sums to 1.

[0104] Step 4: Input the candidate data from ImageNet2012 into the teacher model for prediction, and obtain the logits output with a dimension of 128×10, where 128 is the batch size and 10 represents the number of classes. Perform the SoftMax operation to obtain the probability prediction value with a dimension of 128×10.

[0105] Calculate the difference between each sample and the domain distribution of the pre-trained model;

[0106] Step 5: Sort all samples in ascending order of dissimilarity using Python's built-in sort() method and save them in a txt text file. The data structure in the txt text file is a dictionary type, that is, each row represents a sample, the first column indicates the index position of the image, and the second column indicates its calculated dissimilarity.

[0107] Step 6: Select the top M samples from the sorted samples as the selected dataset that closely approximates the domain distribution of the pre-trained model; in this embodiment, the training set of the CIFAR-10 pre-trained model consists of 50,000 images, and M is set to 100,000; the selected images are read from the indexes of the first 100,000 images saved in Step 6 using Python code as the knowledge distillation dataset.

[0108] Step 7: Train a ResNet34 neural network as the teacher model using the source domain dataset, and a ResNet18 neural network with randomized parameters as the student network. Both the teacher and student networks are optimized using Nesterov Accelerated Gradient (NAG), with weight decay and momentum set to 0.0005 and 0.9, respectively, for a total of 200 epochs. The hyperparameter temperature T is set to 3, the initial learning rate is 0.1, and it decays to 0 with cosine decay over epochs. α is set to 0.7, and β is set to 0.3.

[0109] For CIFAR-10, 100,000 images were selected from the ImageNet2012 dataset as distillation datasets. Before knowledge distillation, the selected data were resized to 32*32 images, the image edges were filled with 4 pixels and cropped to 32*32 images, randomly flipped horizontally, and standardized, among other processes. Finally, after training convergence, the trained student model was saved.

[0110] To achieve the above, after obtaining the student model, the student model is deployed to an embedded device to implement the application.

[0111] For example, the Raspberry Pi is a microcomputer running Raspbian OS (the official operating system), characterized by its ease of secondary development and high integration. Therefore, it was used as the hardware platform for deploying neural network models to demonstrate practical image classification tasks. In this implementation, a Raspberry Pi 4 Model B was used, equipped with a Broadcom BCM2711 (quad-core Cortex-A72) processor with a 1.5GHz clock speed, 2GB of RAM, and an external 32GB SanDisk MicroSD card. The deep learning framework on the Raspberry Pi was PyTorch version 1.6 for the arm71 instruction set. This microcomputer implemented the aforementioned knowledge distillation method for neural network classification models with passive domain data, thus solving the problem in existing technologies where the inability to effectively access source domain data prevented knowledge distillation.

[0112] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0117] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A knowledge distillation method for a neural network classification model of passive domain data, characterized in that: The method includes the following steps: Step 1: Crawl image data to obtain candidate datasets; if the categories classified by the teacher model are known, directly crawl the data corresponding to the category keywords; otherwise, determine whether the known teacher model is classifying a certain superclass. If so, crawl the relevant data using the superclass name as the keyword; otherwise, crawl the relevant type data of any category or use a publicly available dataset. Step 2: Preprocess all samples in the candidate dataset to obtain candidate samples; Step 3: Construct the category similarity matrix C based on the classification layer weights of the teacher model, including the following steps: Step 3.1: Calculate the correlation of each category using the classification layer weights of the teacher model to obtain equation (1). (1) in, Indicate category With category Similarity between them and Categories With category The weight, ; Step 3.2: Repeat step 3.1 until the similarity between all categories is calculated, resulting in a matrix. ; Step 3.3: Convert the matrix Normalization is performed so that the sum of each row of the matrix is ​​1, resulting in the final category similarity matrix C. Step 4: Calculate the logits values ​​of all candidate samples based on the teacher model and perform the SoftMaxt operation. Finally, calculate the difference between all candidate samples and the domain distribution of the teacher model; the difference is given by equation (4). (4) in, For the candidate sample dataset, This represents the maximum confidence level of the predicted sample. For the selected knowledge distillation dataset, The first class similarity matrix OK, Let KL divergence be the KL divergence. Indicates a SoftMax operation; Step 5: Sort all samples from smallest to largest according to their dissimilarity; Step 6: Select the first M samples as the knowledge distillation dataset, M≥100; Step 7: Perform knowledge distillation on the teacher model based on the knowledge distillation dataset to obtain the student model. Deploy the obtained student model to a resource-constrained embedded device to implement the image classification task application.

2. The knowledge distillation method for a neural network classification model of passive domain data according to claim 1, characterized in that: The preprocessing includes data cleaning and adjusting the input candidate samples to a uniform format.

3. The knowledge distillation method for a neural network classification model of passive domain data according to claim 1, characterized in that: Step 4 includes the following steps: Step 4.1: For any input candidate sample The teacher model is used to predict the output logits value. ,in Representing the teacher model, The parameters representing the teacher model; Step 4.2: Calculate the SoftMax output of the logits value for each sample to obtain the predicted label. , For temperature constant, The SoftMax operation is represented by equation (2). (2) in, and The teacher model is represented by the first... and the The output of each node, where N represents the number of categories; for a given sample ,by For maximum confidence .

4. The knowledge distillation method for a neural network classification model of passive domain data according to claim 3, characterized in that: In step 4.1, for any candidate sample Predicted as category The value of logits is given by equation (3). (3) in, This represents the feature vector of the penultimate layer activation in the teacher model, which is concatenated with "1" to account for bias. It is the teacher model number Weights and biases of each class.

5. The knowledge distillation method for a neural network classification model of passive domain data according to claim 1, characterized in that: Input the selected domain dataset into the teacher and student models, and calculate the distillation loss. Fixed teacher model parameters, using Loss update student model, distillation loss Satisfying equation (5), (5) in, , The loss between the teacher model and the student model is calculated using soft labels. It is the cross-entropy loss of the student model under the real labels.

6. The knowledge distillation method for a neural network classification model of passive domain data according to claim 5, characterized in that: ， in, , ; This represents the output of the teacher model predicting a given sample as class i. This represents the output of the student model predicting a class i for a given sample. Here, N represents the number of categories, and the parameter is temperature. Each category corresponds to this.

7. The knowledge distillation method for a neural network classification model of passive domain data according to claim 5, characterized in that: ， in, ; by For the true label of the sample, This represents the output of the student model predicting a class i for a given sample. Here, N represents the number of categories, and the parameter is temperature. Each category corresponds to this.

Images