Image classification method and device based on student model, equipment and medium

Abstract

The invention provides an image classification method and device based on a student model, equipment and a medium, and relates to the technical field of knowledge distillation, and the method comprises the steps: obtaining a to-be-classified image; inputting the to-be-classified image into a target student model to obtain a category corresponding to the to-be-classified image; inputting the initial image set into a trained teacher model and a preset initial student model, and obtaining a teacher model imbalance category probability and a student model imbalance category probability; inputting the mixed image set into a trained teacher model and a preset initial student model, and obtaining a teacher model imbalance category probability and a student model imbalance category probability; determining a corresponding loss; according to the loss, performing back propagation to update student model parameters so as to obtain a target student model; when the image is classified, the resource computing power required by computer hardware can be reduced, the response speed of a computer is improved, and the phenomenon of computer jamming is avoided.

Description

Technical Field

[0001] The present invention relates to the field of knowledge distillation technology, and in particular to an image classification method, device, equipment and medium based on a student model. Background Art

[0002] Deep Neural Networks (DNNs) is a machine learning model that is mainly used to process complex data. DNNs learn the characteristics of data through the connection of multiple layers of neurons and are widely used in tasks such as image and speech recognition. Deep learning models can be established based on DNNs. In the process of deep learning model training, using a large amount of training data can make the ultra-large-scale model achieve good performance, but in actual use, the ultra-large-scale model needs to be compressed to obtain a lightweight model and deploy the lightweight model to the platform. However, in most actual scenarios where lightweight models are obtained, data annotation information is difficult to obtain, the real available data is limited, and there may be an imbalance in data distribution, resulting in the performance of the obtained lightweight model being weak. When using lightweight models to classify images, since the operation of lightweight models depends on the support of computer hardware computing power, the efficiency of lightweight models is low, which can easily lead to insufficient computer computing power, resulting in technical problems such as poor response speed and jamming of lightweight models during image classification. Summary of the invention

[0003] The present invention provides an image classification method, device, equipment and medium based on a student model to solve the technical problems in the prior art that in actual scenarios where most lightweight models are obtained, data annotation information is difficult to obtain, the real available data is limited, and there may be an imbalance in data distribution, resulting in the obtained lightweight model having weak performance.

[0004] According to a first aspect of the present application, a method for image classification based on a student model is provided, comprising:

[0005] Get the image to be classified;

[0006] The image to be classified is input into the target student model to obtain the category corresponding to the image to be classified; the target student model is obtained by the following steps:

[0007] Get an initial set of images;

[0008] Randomly mix two initial images of different categories in the initial image set to obtain a mixed image set;

[0009] Input the initial image set into the trained teacher model and the preset initial student model, and obtain the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image based on knowledge distillation;

[0010] The mixed image set is input into the trained teacher model and the preset initial student model, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image are obtained based on knowledge distillation;

[0011] Determine the corresponding loss according to the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image;

[0012] According to the loss, backpropagation updates the student model parameters to obtain the target student model.

[0013] Furthermore, the randomly mixing two initial images of different categories and corresponding labels in the initial image set to obtain a mixed image set includes:

[0014] Randomly select a majority class initial image and a minority class initial image from the initial image set;

[0015] Obtaining a first mixing coefficient corresponding to the majority class initial image and the minority class initial image and a second mixing coefficient corresponding to the label;

[0016] Using a first mixing coefficient, the majority class initial image and the minority class initial image are mixed to obtain a mixed image;

[0017] The labels of the majority class initial image and the minority class initial image are mixed using the second mixing coefficient to obtain a mixed label corresponding to the mixed image.

[0018] Furthermore, the initial image set is input into the trained teacher model and the preset initial student model, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image are obtained based on knowledge distillation, including:

[0019] For any initial image, obtain a first logarithmic probability corresponding to each category of images obtained by inputting the initial image into a trained teacher model, and obtain a second logarithmic probability corresponding to each category of images obtained by inputting the initial image into a student model;

[0020] Get the number of images in each category;

[0021] According to the first logarithmic probability of each category of images and the number of images of each category, each first logarithmic probability is scaled to obtain the teacher model unbalanced category probability corresponding to the initial image;

[0022] According to the second logarithmic probability of each category of images and the number of images of each category, each second logarithmic probability is scaled to obtain the unbalanced category probability of the student model corresponding to the initial image.

[0023] Furthermore, the mixed image set is input into the trained teacher model and the preset initial student model, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image are obtained based on knowledge distillation, including:

[0024] For any mixed image, obtain three logarithmic probabilities corresponding to each category of images obtained by inputting the mixed image into a trained teacher model, and obtain a fourth logarithmic probability corresponding to each category of images obtained by inputting the mixed image into a student model;

[0025] Get the number of images in each category;

[0026] According to the third logarithmic probability of each category of images and the number of images of each category, each third logarithmic probability is scaled to obtain the unbalanced category probability of the teacher model corresponding to the mixed image;

[0027] According to the fourth logarithmic probability of each category of images and the number of images of each category, each fourth logarithmic probability is scaled to obtain the unbalanced category probability of the student model corresponding to the mixed image.

[0028] Furthermore, the mixed image is equal to the majority class initial image times the first mixing coefficient plus the minority class initial image minus the minority class initial image times the first mixing coefficient; the label corresponding to the mixed image is equal to the label corresponding to the majority class initial image times the second mixing coefficient plus the label corresponding to the minority class initial image minus the label corresponding to the minority class initial image times the second mixing coefficient.

[0029] Furthermore, the number of initial images included in the initial image set is 64.

[0030] Furthermore, the first mixing coefficient is obtained through a β distribution probability density function.

[0031] According to another aspect of the present application, there is also provided an image classification device based on a student model, the device comprising:

[0032] An image acquisition module to be classified is used to acquire images to be classified;

[0033] A category determination module, used for inputting the image to be classified into a target student model to obtain a category corresponding to the image to be classified;

[0034] The target student model acquisition module includes:

[0035] An initial image set acquisition module, used to acquire an initial image set;

[0036] An image mixing module, used for randomly mixing two initial images of different categories in the initial image set to obtain a mixed image set;

[0037] A first imbalanced class probability acquisition module is used to input the initial image set into the trained teacher model and the preset initial student model, and obtain the teacher model imbalanced class probability and the student model imbalanced class probability corresponding to each initial image based on knowledge distillation;

[0038] A second imbalanced category probability acquisition module is used to input the mixed image set into the trained teacher model and the preset initial student model, and obtain the teacher model imbalanced category probability and the student model imbalanced category probability corresponding to each mixed image based on knowledge distillation;

[0039] A loss determination module, used to determine the corresponding loss according to the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image;

[0040] The model parameter updating module is used to update the student model parameters according to the loss and back propagation to obtain the target student model.

[0041] According to another aspect of the present application, a non-transitory computer-readable storage medium is also provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned student model-based image classification method.

[0042] According to another aspect of the present application, an electronic device is provided, including a processor and the above-mentioned non-transitory computer-readable storage medium.

[0043] The present invention has at least the following beneficial effects:

[0044] The image classification method based on the student model of the present invention obtains an image to be classified; inputs the image to be classified into a target student model to obtain a category corresponding to the image to be classified; the target student model is obtained by the following steps: inputs an initial image set into a trained teacher model and a preset initial student model, and obtains the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image based on knowledge distillation; inputs a mixed image set into a trained teacher model and a preset initial student model, and obtains the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image based on knowledge distillation; according to the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image, and the teacher model unbalanced category probability corresponding to each mixed image The probability and the probability of imbalanced categories in the student model are used to determine the corresponding loss; according to the loss, the student model parameters are updated by back propagation to obtain the target student model; the present invention constructs sufficient virtual samples at the sample level, further expands the input space, and can more fully retain the teacher's knowledge; the image mixing method of the present invention starts from the imbalanced perspective, and designs a Mixup label selection method between the majority class and the minority class, thereby improving the classification performance of the model in an unbalanced scenario; through the method of the present invention, a large model can be effectively compressed into a small model without significantly losing its performance, thereby achieving efficient model deployment and reasoning, and the obtained lightweight model has stronger performance; when classifying images, it can reduce the resource computing power required by computer hardware, improve the computer response speed, and avoid the occurrence of computer freezes. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A schematic diagram of a knowledge distillation model in the prior art provided for an embodiment of the present invention;

[0047] Figure 2 A flow chart of an image classification method based on a student model provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a general knowledge distillation model framework based on Mixup data enhancement provided in an embodiment of the present invention;

[0049] Figure 4 An application scenario diagram of the image classification method based on the student model provided by an embodiment of the present invention;

[0050] Figure 5 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] It should be noted that, based on the present disclosure, those skilled in the art should understand that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement a device and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0053] Related knowledge in this application:

[0054] (1) Deep Neural Networks (DNNs)

[0055] Deep neural network is a machine learning model that is mainly used to process complex data. DNNs learn the characteristics of data through the connection of multiple layers of neurons and are widely used in tasks such as image and speech recognition.

[0056] (2) Knowledge Distillation

[0057] Knowledge distillation is a model compression technique that uses a complex model (teacher model) to train a smaller model (student model). The student model inherits the knowledge and generalization ability of the teacher model by learning its output.

[0058] (3) Image Mixup

[0059] Mixup is a data augmentation technique that generates new training samples by linearly interpolating two images and their labels. This can increase the diversity of data and improve the generalization ability of the model.

[0060] (4) Teacher Model

[0061] The teacher model is a large model in the knowledge distillation process, which is used to guide the training of the student model. The teacher model is usually very complex and has strong performance.

[0062] (5) Student Model

[0063] The student model is a small model in the knowledge distillation process, which improves its own performance by learning the output of the teacher model. The student model is relatively simple and suitable for resource-constrained devices.

[0064] (6) Cross Entropy

[0065] Cross entropy is a loss function used to measure the difference between the predicted value and the true value, and is often used in classification tasks.

[0066] (7)KL Divergence (Kullback-Leibler Divergence)

[0067] KL divergence is an indicator that measures the difference between two probability distributions and is often used to calculate distillation loss in knowledge distillation.

[0068] (8) Imbalanced Classification

[0069] Imbalanced classification refers to the situation where the number of samples of different categories in the training data set varies greatly, which will affect the training effect of the model.

[0070] (9) Logits

[0071] Logits are the probability values ​​output by the neural network without being processed by the normalized exponential function (softmax), and are often used to calculate loss functions.

[0072] (10) Normalized exponential function (Softmax)

[0073] Softmax is a function that converts log probability into probability distribution and is often used in multi-classification tasks.

[0074] The present application relates to the field of knowledge distillation in image classification, and aims to study the model compression problem in different data scale scenarios. In the process of deep learning model training, good performance is obtained using data and ultra-large-scale models, but in actual use, model compression is required, and a lightweight model is deployed on the platform. In most practical scenarios, on the one hand, data annotation information is difficult to obtain, the real available data is limited, and there may be situations such as unbalanced data distribution. In order to be applicable to a wider range of scenarios, the present invention designs a general knowledge distillation method based on Mixup data enhancement for scenarios with sufficient data, sparse data, and unbalanced data: on the basis of classical knowledge distillation, the unbalanced classification learning strategy and the Mixup data enhancement method are integrated to achieve model compression in conventional data scenarios, few sample data scenarios, and unbalanced data scenarios.

[0075] Introduction to knowledge distillation: The success of deep neural networks (DNNs) usually depends on the careful design of the DNNs architecture. In large-scale machine learning, especially in tasks such as image and speech recognition, most DNNs-based models are over-parameterized, extracting significant features to ensure the generalization ability of the model. Such over-parameterized models are usually very deep and wide, with too large model parameters, requiring a lot of computation for training, and difficult to reason in real time. Therefore, in order to achieve faster speeds, researchers have tried to use trained large models to obtain lightweight models that can be deployed in edge devices. In other words, after training, complex models can be used to "guide" a small model that can be deployed in real-time applications. For details, see Figure 1 shown.

[0076] Knowledge distillation uses a teacher-student model, where the teacher model is the output of "knowledge" and the student model is the receiver of "knowledge". The process of knowledge distillation is divided into two stages:

[0077] (1) Training the teacher model: The teacher model is relatively complex and can be integrated from multiple separately trained models. Generally, there are no restrictions on the model architecture, number of parameters, or whether it is integrated. The only requirement is that for the input image X, the teacher model can output the corresponding output logit, denoted as Y. Y is converted into a probability value through the Softmax function, corresponding to the probability value of the corresponding category.

[0078] (2) Training the student model: The student model has a small number of parameters and a relatively simple model structure. Similarly, for input X, it can output Y, and Y can also output the probability value corresponding to the corresponding category after Softmax mapping. During the student model training process, the prediction results output by the teacher model are used as soft labels, and then the distillation loss is calculated and back-propagated together with the classification loss, so that the student model can learn the generalization ability of the teacher model while classifying.

[0079] Image mixing: Mixup is a common data enhancement method and a simple data enhancement principle that is independent of data. Its core is to construct new training samples and labels in a linear interpolation manner. The final processing of the label is shown in the following formula:

[0080] x ij =λx i +(1-λ)x j

[0081] y ij =λy i +(1-λ)y j

[0082] Among them, (x i ,y i ) and (x j ,y j ) The two data pairs are training sample pairs in the original data set, where x represents the training sample and y represents the corresponding label. Among them, λ obeys the Beta distribution with parameter α, λ~β(α,α).

[0083] The above-mentioned prior art deduced by causal reasoning has the following defects:

[0084] Defect 1: Currently, traditional knowledge distillation brings limited improvement to small models. Traditional knowledge distillation adds distillation loss on top of classification loss. As shown in Table 1, simple knowledge distillation can only improve the original classification by 1-2 percentage points, which leads to a decrease in the utilization rate of large-scale pre-trained models obtained at a high cost. How to make full use of the dark knowledge of pre-trained models is a technical problem that needs to be explored urgently.

[0085] Table 1

[0086]

[0087] Defect 2: In most real-world scenarios, real data is often difficult to obtain. On the one hand, the amount of real data is small, and on the other hand, real data often presents a long-tail distribution or data imbalance. For example, for animal classification, the amount of data for common cats and dogs is extremely large, but the amount of data for rare animals such as pandas and penguins is very small. Existing knowledge distillation methods are not applicable to scenarios with small amounts of data or imbalanced data. In addition, such data will further reduce the gains brought by knowledge distillation.

[0088] The following will refer to Figure 2 The flowchart of the image classification method based on the student model is shown, and an image classification method based on the student model is introduced.

[0089] The method of the present invention mainly includes modules such as logits knowledge distillation, unbalanced general knowledge distillation, and Mixup data enhancement. Conventional knowledge distillation mainly includes classification loss and distillation loss. On this basis, we expand it from two aspects: data and loss function. At the data level, we use the Mixup strategy to expand the data. On the one hand, it alleviates the challenges brought by data scarcity, and on the other hand, it uses the proposed Mixup data mixing strategy to alleviate the constraints brought by data imbalance. In terms of loss function, on the one hand, an unbalanced general classification loss is designed to maintain classification performance, and on the other hand, an unbalanced distillation loss is designed to be suitable for knowledge distillation in different data scenarios. The algorithm model framework of the present invention is as follows: Figure 3 shown.

[0090] The image classification method based on the student model may include the following steps:

[0091] S100, obtaining an image to be classified.

[0092] In this embodiment, any image containing a target object may be acquired, for example, an image to be classified containing an animal or a plant.

[0093] S200, inputting the image to be classified into a target student model to obtain a category corresponding to the image to be classified.

[0094] In this embodiment, before the image to be classified is input into the target student model to obtain the category corresponding to the image to be classified, it also includes: a process of training the initial model to obtain the target student model, which may include the following steps:

[0095] S310, obtaining an initial image set.

[0096] In this embodiment, K classification data sets are preset, and the K classification data sets include K types of image samples, and each type of image includes a number of image samples; for example, the image samples of the cat type include 1,000 cat images; a certain number of images can be randomly obtained from the K classification data sets as initial images, thereby obtaining an initial image set.

[0097] S320, randomly mixing two initial images of different categories in the initial image set to obtain a mixed image set.

[0098] In this embodiment, the data of a batch is obtained as<x,y> , the batch size can be 64, that is, 64 images are input each time, and virtual samples are constructed cyclically through the following steps:

[0099] S321, randomly select an initial image of the majority class and an initial image of the minority class in the initial image set.

[0100] In this embodiment, the initial image set includes majority class images and minority class images, and two samples x of different classes can be randomly selected. i ,x j .

[0101] S322, obtaining first mixing coefficients corresponding to the majority class initial image and the minority class initial image and second mixing coefficients corresponding to the labels.

[0102] In this embodiment, the first mixing coefficient λ can be obtained by sampling from the Beta distribution β(α,α): x ; That is, you only need to set this distribution, and you can randomly sample according to this distribution; specifically, the result of sampling countless times according to this distribution is that the final data distribution is consistent with the given distribution; the second mixing coefficient λ y The definition is as follows:

[0103]

[0104] Among them, n i ,n j Represents sample x i ,x j The number of samples of the corresponding category. k,τ are the hyperparameters of the method:

[0105] When τ = 0, the method degenerates into the traditional Mixup, that is, λ y =λ x Because, when τ=0, due to λ x Belongs to [0,1]. Substituting it into the above formula, we can find that only the third part of the above formula is satisfied.

[0106] When τ=1, if n i / nj ≥k, that is, sample x i The category is the majority class, sample x j The category is the minority class; in this case, the label of the mixed image is set to the minority class label.

[0107] When 0<τ<1, if n i / n j ≥k, and at this time the sample x i If the majority class weight λ is small, it is set as the minority class label.

[0108] In this embodiment, it should be noted that in traditional unbalanced classification, the classification boundary is always close to the minority class, which leads to a decrease in the performance of the minority class. Here, the relaxation factor τ is introduced to control its size, which can ensure that the minority class accounts for a larger proportion in the label corresponding to the Mixup image, thereby expanding the minority class proportion in the label space and alleviating the impact of imbalance.

[0109] S323: Use the first mixing coefficient to mix the majority class initial image and the minority class initial image to obtain a mixed image.

[0110] S324, using a second mixing coefficient to mix the labels of the majority class initial image and the minority class initial image to obtain a mixed label corresponding to the mixed image.

[0111] Furthermore, the mixed image is equal to the majority class initial image times the first mixing coefficient plus the minority class initial image minus the minority class initial image times the first mixing coefficient; the label corresponding to the mixed image is equal to the label corresponding to the majority class initial image times the second mixing coefficient plus the label corresponding to the minority class initial image minus the label corresponding to the minority class initial image times the second mixing coefficient.

[0112] In this embodiment, Mixup image mixing: x ij =λ x x i +(1-λ x )x j ; Calculate sample x i ,x j The number of samples in the category n,n j , then according to the above λ y The formula for calculating λ is y ; Get the mixed label corresponding to the mixed image: y ij =λ y y i +(1-λ y )y j ; The mixed image is <x mixup ,y mixup >.

[0113] S330, inputting the initial image set into the trained teacher model and the preset initial student model, and obtaining the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image based on knowledge distillation.

[0114] Further, step S330 may include the following steps:

[0115] S331, for any initial image, obtain the first logarithmic probability corresponding to each category of image obtained by inputting the initial image into the trained teacher model, and obtain the second logarithmic probability corresponding to each category of image obtained by inputting the initial image into the student model.

[0116] S332, obtaining the number of images of each category.

[0117] S333, scaling each first logarithmic probability according to the first logarithmic probability of each category of images and the number of images of each category to obtain the teacher model unbalanced category probability corresponding to the initial image.

[0118] S334, scaling each second logarithmic probability according to the second logarithmic probability of each category of images and the number of images of each category to obtain the student model unbalanced category probability corresponding to the initial image.

[0119] In this embodiment, logits knowledge distillation, the current mainstream deep neural network used for image classification tasks can be regarded as a combination of a feature encoder with multiple nonlinear layers and a classifier that usually contains a single fully connected layer with a Softmax activation function. Both parts are trained end-to-end through the back-propagation algorithm.

[0120] Given a training sample x and the corresponding one-hot encoded label y, the data comes from a K-classification dataset. Define the student model F s The output feature of the penultimate layer is recorded as where θ s is the parameter of the student model, C s It is the feature dimension output by the student model. Feature f s Input to the student model’s classifier Get logits g from s , i.e. g s =W s f s ∈R K . Logits g s The final probability p of each category is obtained by the Softmax function s =σ(g s / T)∈R K :

[0121]

[0122] The superscript s represents student, and the subscript i represents the i-th category. i s The probability of the i-th category of the student model output result. T represents the temperature, which is used to adjust the distribution of logit. You can set T=2.

[0123] Similarly, all the symbols of the teacher model are similar to those of the student model, except that the superscript is changed to t(p i t ). Traditional logits knowledge distillation consists of two parts of loss, one is the traditional cross entropy loss, and the other is the distillation loss, which is generally the Kullback-Leibler divergence to constrain the prediction p s With soft target p t The alignment loss between:

[0124] L=L ce (y,p s )+T 2 L KL (p t ,p s )

[0125] Compared with the cross entropy loss, the introduced prediction alignment loss provides additional information about the wrong classes to facilitate the training of the student. Since the probabilities assigned to those incorrect classes are often small after the softmax transformation, the temperature T in this term needs to be increased to produce a softer distribution in order to pass more information.

[0126] Due to the data imbalance scenario, unbalanced general knowledge distillation is introduced. The biggest impact of unbalanced data is the imbalance of data brightness in each category, which leads to unbalanced gradients in the final classification head. In order to avoid gradient imbalance, we introduce weight factors for different categories. The number of samples in category i is n i , based on the original logits, we scale the logits of different categories separately:

[0127]

[0128] Finally, the general knowledge distillation loss function for imbalanced scenarios is defined as KL(φ t ||φ s ).

[0129] S340, inputting the mixed image set into the trained teacher model and the preset initial student model, and obtaining the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image based on knowledge distillation.

[0130] Further, step S340 may include the following steps:

[0131] S341, for any mixed image, obtain three logarithmic probabilities corresponding to each category of image obtained by inputting the mixed image into the trained teacher model, and obtain a fourth logarithmic probability corresponding to each category of image obtained by inputting the mixed image into the student model.

[0132] S342, obtaining the number of images of each category.

[0133] S343, scaling each third logarithmic probability according to the third logarithmic probability of each category of images and the number of images of each category to obtain the teacher model unbalanced category probability corresponding to the mixed image.

[0134] S344, scaling each fourth logarithmic probability according to the fourth logarithmic probability of each category of images and the number of images of each category to obtain the student model unbalanced category probability corresponding to the mixed image.

[0135] In this embodiment, the unbalanced category probability of the mixed image can also be obtained using the above-mentioned method for obtaining the unbalanced category probability of the initial image, which will not be described in detail here.

[0136] S350, determining the corresponding loss according to the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image.

[0137] In this embodiment, the initial image<x,y> The category probability is obtained through the teacher network and the student network, and the category probability is adjusted by unbalanced knowledge distillation to obtain φ real t ,φ real s .

[0138] Data and labels corresponding to the mixed images constructed by Mixup <x mixup ,y mixup >, similarly, <x mixup ,y mixup > Also get φ through the teacher network and student network mixup t ,φ mixup s . The final loss function is:

[0139] L=L ce +λL kd

[0140] L ce=L ce (y,φ real s )+βL ce (y mixup ,φ mixup s )

[0141] L kd =KL(φ real t ||φ real s )+γL(φ mixup t ||φ mixup s )

[0142] Among them, λ is used to balance the classification loss and distillation loss, β and γ are used to balance the loss weights of real samples and Mixup samples, and the default setting is β=γ=1.

[0143] S360, based on the loss, back-propagation updates the student model parameters to obtain the target student model.

[0144] In this embodiment, after obtaining the target student model, only a more lightweight student model needs to be used subsequently. Using the method in this application, the performance of the target student model obtained is stronger.

[0145] It should be noted that the above-mentioned training step for the target student model can be performed after step S100 and before step S200, or can be performed before step S100.

[0146] The image classification method based on the student model of the present application has the following beneficial effects:

[0147] (1) Compared with traditional knowledge distillation: The present invention has a wider scope of application and can be applied to scenarios with small data volume and unbalanced data. By introducing Mixup data enhancement, the present invention constructs sufficient virtual samples at the sample level, further expands the input space, and can more fully retain the teacher's knowledge. The present invention can be combined with other knowledge distillation methods as a plug-and-play module to improve the performance of other knowledge distillation methods and has a wider range of applications.

[0148] (2) Compared with traditional Mixup data enhancement: The image mixing method of the present invention designs a Mixup label selection method between the majority class and the minority class from an imbalanced perspective, thereby improving the classification performance of the model in an unbalanced scenario.

[0149] (3) The present invention is particularly suitable for scenarios where lightweight models need to be deployed, such as edge computing devices and mobile devices. In these scenarios, computing resources are limited, but high-performance models are still required to handle complex tasks. Through the method of the present invention, large models can be effectively compressed into small models without significantly losing their performance, thereby achieving efficient model deployment and reasoning.

[0150] The image classification method based on the student model of this embodiment obtains an image to be classified;

[0151] The image to be classified is input into the target student model to obtain the category corresponding to the image to be classified; the target student model is obtained by the following steps: the initial image set is input into the trained teacher model and the preset initial student model, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image are obtained based on knowledge distillation; the mixed image set is input into the trained teacher model and the preset initial student model, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image are obtained based on knowledge distillation; according to the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image rate, determine the corresponding loss; according to the loss, back-propagate to update the student model parameters to obtain the target student model; the present invention constructs sufficient virtual samples at the sample level, further expands the input space, and can more fully retain the teacher's knowledge; the image mixing method of the present invention designs a Mixup label selection method between the majority class and the minority class from an imbalanced perspective, thereby improving the classification performance of the model in an unbalanced scenario; through the method of the present invention, a large model can be effectively compressed into a small model without significantly losing its performance, thereby achieving efficient model deployment and reasoning, and the obtained lightweight model has stronger performance; when classifying images, it can reduce the resource computing power required by computer hardware, improve computer response speed, and avoid computer freezes.

[0152] In an exemplary embodiment, a student model-based image classification device is provided, the device comprising:

[0153] The module for acquiring images to be classified is used to acquire images to be classified.

[0154] The category determination module is used to input the image to be classified into the target student model to obtain the category corresponding to the image to be classified.

[0155] The target student model acquisition module includes:

[0156] The initial image set acquisition module is used to acquire the initial image set.

[0157] The image mixing module is used to randomly mix two initial images of different categories in the initial image set to obtain a mixed image set.

[0158] The first unbalanced category probability acquisition module is used to input the initial image set into the trained teacher model and the preset initial student model, and obtain the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image based on knowledge distillation.

[0159] The second unbalanced category probability acquisition module is used to input the mixed image set into the trained teacher model and the preset initial student model, and obtain the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image based on knowledge distillation.

[0160] The loss determination module is used to determine the corresponding loss according to the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image.

[0161] The model parameter updating module is used to update the student model parameters according to the loss and back propagation to obtain the target student model.

[0162] In an exemplary embodiment, an application scenario based on the method in the above embodiment is provided, which is as follows:

[0163] Through the method in this embodiment, for image classification scenarios, a trained target student model can be obtained, and the target student model can be deployed on an edge device with limited computing resources, for example: the edge device is an Edge Box, or a similar device; after acquiring the image to be classified, the image to be classified can be input into the target student model to output the category corresponding to the image to be classified through the target student model on the edge device.

[0164] In the above scenario, the classification performance of the target student model obtained by the method in this embodiment is strong, and the computing power requirements of the edge device are low. The edge device can easily run the target student model. While ensuring the accuracy of image classification, it can also reduce the resource computing power required by computer hardware, improve computer response speed, and avoid computer freezes.

[0165] Furthermore, the method in this embodiment can also be applied to the following scenarios:

[0166] (1) Natural Language Processing (NLP): In NLP tasks such as text classification, sentiment analysis, and machine translation, data augmentation and knowledge distillation methods can also improve the generalization ability and efficiency of the model. By using methods such as Mixup, more training data can be generated to alleviate data scarcity and imbalance problems.

[0167] (2) Speech recognition: In speech recognition tasks, data imbalance and insufficient data are common problems. The method of the present invention can enhance speech data and improve the performance and stability of the model. At the same time, knowledge distillation can compress complex speech recognition models into small models suitable for mobile devices.

[0168] (3) Medical image analysis: In the medical field, labeled data is often difficult to obtain and unevenly distributed. Through Mixup data enhancement, more diverse training samples can be generated to improve the diagnostic accuracy of the model under different lesion types. At the same time, knowledge distillation technology can compress large medical image analysis models, making them easier to apply on medical equipment with limited resources.

[0169] (4) Financial data analysis: Imbalanced data is a common problem in financial data analysis tasks, such as credit scoring and fraud detection. The method of the present invention can enhance the model's detection ability in minority class events by enhancing training data, and improve the prediction performance of small models through knowledge distillation.

[0170] (5) Recommendation system: The recommendation system needs to process massive amounts of data, and the amount of interaction data between different users and items is uneven. By applying Mixup data enhancement, more user-item interaction data can be generated to improve the effect of the recommendation model. At the same time, knowledge distillation technology can compress large recommendation models and is suitable for real-time recommendation scenarios.

[0171] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0172] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0173] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0174] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0175] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0176] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. Figure 4 As shown, the program code may be executed entirely on the user's computing device, partially on the user's computing device, as a separate software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0177] like Figure 5 As shown, an embodiment of the present invention further provides an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0178] The electronic device is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0179] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one memory mentioned above, and a bus connecting different system components (including the memory and the processor).

[0180] The memory stores program codes, which can be executed by the processor, so that the processor executes the steps in various embodiments described in this specification.

[0181] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0182] The memory may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0183] The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0184] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0185] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0186] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0187] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. An image classification method based on a student model, characterized in that: include: Get the image to be classified; Inputting the image to be classified into the target student model to obtain the category corresponding to the image to be classified; The target student model is obtained by the following steps: Get an initial set of images; Randomly mix two initial images of different categories in the initial image set to obtain a mixed image set; Input the initial image set into the trained teacher model and the preset initial student model, and obtain the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image based on knowledge distillation; The mixed image set is input into the trained teacher model and the preset initial student model, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image are obtained based on knowledge distillation; Determine the corresponding loss according to the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image; According to the loss, backpropagation updates the student model parameters to obtain the target student model.

2. The image classification method based on the student model according to claim 1, characterized in that: The randomly mixing two initial images of different categories and corresponding labels in the initial image set to obtain a mixed image set includes: Randomly select an initial image of the majority class and an initial image of the minority class from the initial image set; Obtaining a first mixing coefficient corresponding to the majority class initial image and the minority class initial image and a second mixing coefficient corresponding to the label; Using a first mixing coefficient, the majority class initial image and the minority class initial image are mixed to obtain a mixed image; The labels of the majority class initial image and the minority class initial image are mixed using the second mixing coefficient to obtain a mixed label corresponding to the mixed image.

3. The image classification method based on the student model according to claim 1, characterized in that: The initial image set is input into the trained teacher model and the preset initial student model, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image are obtained based on knowledge distillation, including: For any initial image, obtain a first logarithmic probability corresponding to each category of images obtained by inputting the initial image into a trained teacher model, and obtain a second logarithmic probability corresponding to each category of images obtained by inputting the initial image into a student model; Get the number of images in each category; According to the first logarithmic probability of each category of images and the number of images of each category, each first logarithmic probability is scaled to obtain the teacher model unbalanced category probability corresponding to the initial image; According to the second logarithmic probability of each category of images and the number of images of each category, each second logarithmic probability is scaled to obtain the unbalanced category probability of the student model corresponding to the initial image.

4. The image classification method based on the student model according to claim 1, characterized in that: The mixed image set is input into the trained teacher model and the preset initial student model, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image are obtained based on knowledge distillation, including: For any mixed image, obtain three logarithmic probabilities corresponding to each category of images obtained by inputting the mixed image into a trained teacher model, and obtain a fourth logarithmic probability corresponding to each category of images obtained by inputting the mixed image into a student model; Get the number of images in each category; According to the third logarithmic probability of each category of images and the number of images of each category, each third logarithmic probability is scaled to obtain the unbalanced category probability of the teacher model corresponding to the mixed image; According to the fourth logarithmic probability of each category of images and the number of images of each category, each fourth logarithmic probability is scaled to obtain the unbalanced category probability of the student model corresponding to the mixed image.

5. The image classification method based on the student model according to claim 2, characterized in that: The mixed image is equal to the majority class initial image times the first mixing coefficient plus the minority class initial image minus the minority class initial image times the first mixing coefficient; the label corresponding to the mixed image is equal to the label corresponding to the majority class initial image times the second mixing coefficient plus the label corresponding to the minority class initial image minus the label corresponding to the minority class initial image times the second mixing coefficient.

6. The image classification method based on the student model according to claim 1, characterized in that: The number of initial images included in the initial image set is 64.

7. The image classification method based on the student model according to claim 2, characterized in that: The first mixing coefficient is obtained through a β distribution probability density function.

8. An image classification device based on a student model, characterized in that: The device comprises: An image acquisition module to be classified is used to acquire images to be classified; A category determination module, used for inputting the image to be classified into a target student model to obtain a category corresponding to the image to be classified; The target student model acquisition module includes: An initial image set acquisition module, used to acquire an initial image set; An image mixing module, used for randomly mixing two initial images of different categories in the initial image set to obtain a mixed image set; A first imbalanced class probability acquisition module is used to input the initial image set into the trained teacher model and the preset initial student model, and obtain the teacher model imbalanced class probability and the student model imbalanced class probability corresponding to each initial image based on knowledge distillation; A second imbalanced category probability acquisition module is used to input the mixed image set into the trained teacher model and the preset initial student model, and obtain the teacher model imbalanced category probability and the student model imbalanced category probability corresponding to each mixed image based on knowledge distillation; A loss determination module, used to determine the corresponding loss according to the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each initial image, and the teacher model unbalanced category probability and the student model unbalanced category probability corresponding to each mixed image; The model parameter updating module is used to update the student model parameters according to the loss and back propagation to obtain the target student model.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the student model-based image classification method as described in any one of claims 1-7.

10. An electronic device, characterized in that: Includes a processor and the non-transitory computer-readable storage medium of claim 8.

Images