An image feature extraction method and device based on knowledge distillation

Abstract

The application belongs to the technical field of image processing and relates to an image feature extraction method and device based on knowledge distillation. Multiple image samples are input into a teacher model to obtain a first feature vector of each image sample. The multiple image samples are input into a student model to obtain a second feature vector of each image sample. Feature values with the same order in the multiple first feature vectors are combined to obtain m first combined vectors. Feature values with the same order in the multiple second feature vectors are combined to obtain m second combined vectors. A first loss function is obtained based on a target class knowledge distillation loss function of the multiple image samples, a second loss function is obtained based on a Pearson correlation coefficient loss function of the multiple image samples, and a third loss function is obtained based on a Pearson correlation coefficient loss function of the m combined vectors, so that an image feature loss function is obtained to train the student model until the image feature extraction loss function is minimum, and a trained student model is obtained.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image feature extraction method and apparatus based on knowledge distillation. Background Technology

[0002] In recent years, machine learning (ML) and deep learning (DL) have made significant progress in the field of computer vision. Starting with the first deep convolutional neural network, AlexNet, deeper networks with increasingly more parameters and computational costs have been proposed, such as VGGNet, Inception, and ResNet. These models are widely used for image feature extraction tasks. To improve the effectiveness of feature extraction, convolutional neural network models are becoming increasingly complex, with more and more computational costs and parameters, which in turn increases the demand on hardware resources, making it difficult to deploy these models on resource-constrained edge devices.

[0003] Existing technologies use knowledge distillation to compress deep neural network models. This is achieved by transferring knowledge from a large model (teacher model) to a small model (student model), improving the accuracy of the small model without altering its structure. This results in achieving comparable accuracy to the large model at a lower cost, thus lightweighting the model and reducing computational and parameter requirements without compromising image feature extraction accuracy. Current knowledge distillation methods use features from the teacher model's output layer as supervisory information to train the student model. This ensures that the student model's output features are as consistent as possible with those of the teacher model. The trained student model is then used as the feature extraction model, achieving the same feature extraction accuracy as the teacher model with fewer computational costs and parameters, thus achieving model lightweighting without sacrificing image feature extraction precision.

[0004] Currently, knowledge distillation based on output layer features minimizes the differences in output layer features between the student model and the teacher model for the same image sample by constructing intra-class distillation loss. However, when the model architectures of the teacher model and the student model are different, such as the teacher model being a ResNet architecture model and the student model being a MobileNet architecture model, the teacher model and the student model have significantly different understandings of the image. Therefore, the teacher model can provide the student model with more diverse knowledge and feature representations. Training the student model in a heterogeneous teacher-student model solely through intra-class knowledge distillation loss cannot effectively transfer the knowledge of the teacher model to the student model. This results in the student model not being able to fully learn all the knowledge in the teacher model, thus degrading the performance of the student model and affecting the image feature extraction accuracy of the student model.

[0005] In summary, existing image feature extraction methods based on knowledge distillation cannot fully learn the knowledge in the teacher model when training the student model in a heterogeneous teacher-student model, resulting in low image feature extraction accuracy of the student model. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem that when the existing image feature extraction method based on knowledge distillation is applied to heterogeneous teacher-student models, the student model cannot fully learn the knowledge in the teacher model, resulting in low image feature extraction accuracy of the student model.

[0007] To address the aforementioned technical problems, this invention provides an image feature extraction method based on knowledge distillation, comprising:

[0008] Multiple image samples from the training set are input into the teacher model for feature extraction, and a first feature vector corresponding to each image sample is obtained based on the output of the teacher model; the multiple image samples are input into the student model for feature extraction, and a second feature vector corresponding to each image sample is obtained based on the output of the student model.

[0009] The same sorted feature values ​​in the first feature vectors corresponding to multiple image samples are combined to obtain m first combination vectors; the same sorted feature values ​​in the second feature vectors corresponding to multiple image samples are combined to obtain m second combination vectors; where m represents the number of feature values ​​in the first feature vectors and the second feature vectors.

[0010] A target class knowledge distillation loss function is constructed based on the first feature vector and the second feature vector corresponding to each image sample; the first loss function is obtained by summing the target class knowledge distillation loss functions of multiple image samples.

[0011] Based on the Pearson correlation coefficient between the first feature vector and the second feature vector corresponding to each image sample, a Pearson correlation coefficient loss function is constructed for that image sample; the second loss function is obtained by summing the Pearson correlation coefficient loss functions of multiple image samples.

[0012] Based on the Pearson correlation coefficient between the i-th first combined vector and the i-th second combined vector, a Pearson correlation coefficient loss function for the i-th combined vector is constructed; a third loss function is obtained by summing the Pearson correlation coefficient loss functions of the m combined vectors; where i∈[1,m];

[0013] The first loss function, the second loss function, and the third loss function are weighted and summed to obtain the image feature extraction loss function. The student model is then iteratively trained using image samples from the training set until the value of the image feature extraction loss function is minimized, thus obtaining a well-trained student model.

[0014] Preferably, the target class knowledge distillation loss function for image samples is expressed as:

[0015]

[0016] Among them, TCKD j Let KL denote the target class knowledge distillation loss function for the j-th image sample. and Similarity between them Let represent the target category prediction probability obtained after softmax normalization of the first feature vector of the j-th image sample. The target category prediction probability is obtained by normalizing the second feature vector of the j-th image sample using softmax.

[0017] The first loss function is expressed as:

[0018]

[0019] Where TCKD represents the first loss function.

[0020] Preferably, the Pearson correlation coefficient loss function for image samples is expressed as:

[0021]

[0022] Among them, Pearson j Let Pearson correlation coefficient loss function be the value of the j-th image sample. This represents the first feature vector of the j-th image sample. This represents the second feature vector of the j-th image sample;

[0023] The second loss function is expressed as:

[0024]

[0025] Among them, Pearson (P T ||P S ) represents the second loss function.

[0026] Preferably, the Pearson correlation coefficient loss function of the i-th combined vector is expressed as:

[0027]

[0028] Among them, Pearson i Let Pearson correlation coefficient loss function be the value of the i-th combined vector. Represents the i-th first combination vector. Represents the i-th second combination vector;

[0029] The third loss function is expressed as:

[0030]

[0031] Among them, Pearson (P Tz ||P Sz ) represents the third loss function.

[0032] Preferably, the image feature extraction loss function is expressed as:

[0033] KD=α·TCKD+β·(Pearson(P T ||P S )+Pearson(P Tz ||P Sz )),

[0034] Where KD represents the image feature extraction loss function, α represents the weight of the first loss function, TCKD represents the first loss function, β represents the weight of the second and third loss functions, and Pearson(P T ||P S ) represents the second loss function, Pearson(P Tz ||P Sz ) represents the third loss function.

[0035] Preferably, it further includes:

[0036] Normalize the first feature vector corresponding to each image sample and output the class prediction probability value of each image sample; take the average of the class prediction probability values ​​of multiple image samples to obtain the target class prediction probability value.

[0037] The weights of the second and third loss functions are updated based on the predicted probability values ​​of the target category to obtain the target image feature extraction loss function.

[0038] Preferably, the target image feature extraction loss function is expressed as:

[0039]

[0040] Where KD′ represents the target image feature extraction loss function, This represents the predicted probability value of the target category, where k is a preset coefficient.

[0041] Preferably, the method further includes scaling and inverting the predicted probability value of the target category to obtain a temperature factor, thereby adjusting the output probability distribution of the teacher model based on the temperature factor.

[0042] Preferably, the student model includes a stem module and multiple feature extraction modules with different kernel sizes connected in series along the forward propagation direction. Each feature extraction module includes a no-padding downsampling submodule and a feature extraction convolution submodule. Each no-padding downsampling submodule includes a no-padding convolution unit, a no-padding max pooling unit, and a feature fusion unit.

[0043] The present invention also provides an image feature extraction device based on knowledge distillation, comprising:

[0044] The feature extraction module is used to input multiple image samples from the training set into the teacher model for feature extraction, and obtain a first feature vector corresponding to each image sample based on the output of the teacher model; and to input the multiple image samples into the student model for feature extraction, and obtain a second feature vector corresponding to each image sample based on the output of the student model.

[0045] The feature combination module is used to combine the feature values ​​with the same order in the first feature vectors corresponding to multiple image samples to obtain m first combination vectors; and to combine the feature values ​​with the same order in the second feature vectors corresponding to multiple image samples to obtain m second combination vectors; where m represents the number of feature values ​​in the first feature vectors and the second feature vectors.

[0046] The first loss function construction module is used to construct the target class knowledge distillation loss function for each image sample based on the first feature vector and the second feature vector corresponding to that image sample; the first loss function is obtained by summing the target class knowledge distillation loss functions of multiple image samples.

[0047] The second loss function construction module is used to construct the Pearson correlation coefficient loss function of the image sample based on the Pearson correlation coefficient between the first feature vector corresponding to each image sample and the second feature vector corresponding to the image sample; and to obtain the second loss function based on the sum of the Pearson correlation coefficient loss functions of multiple image samples.

[0048] The third loss function construction module is used to construct the Pearson correlation coefficient loss function of the i-th combination vector based on the Pearson correlation coefficient between the i-th first combination vector and the i-th second combination vector; the third loss function is obtained by summing the Pearson correlation coefficient loss functions of the m combination vectors; where i∈[1,m];

[0049] The model training module is used to perform a weighted summation of the first loss function, the second loss function, and the third loss function to obtain an image feature extraction loss function. The student model is then iteratively trained using image samples from the training set until the value of the image feature extraction loss function is minimized, thus obtaining a trained student model.

[0050] The image feature extraction method based on knowledge distillation provided in this application first constructs a target class knowledge distillation loss based on the first and second feature vectors of the same image sample when constructing the loss function. This results in a first loss function based on the target class knowledge distillation loss of all image samples, constraining the student model's prediction of the target class for the image sample to approach that of the teacher model. Simultaneously, a Pearson correlation coefficient loss function is constructed based on the Pearson correlation coefficient between the first and second feature vectors corresponding to the same image sample. This results in a second loss function based on the Pearson correlation coefficient loss function of all image samples, constraining the difference between the feature vectors extracted by the student model and the teacher model for the same sample. Furthermore, it also considers the first feature vectors corresponding to different samples... The Pearson correlation coefficient between the eigenvalues ​​with the same order in the first feature vector and the eigenvalues ​​with the same order in the second feature vector is used to construct a third loss function, enabling the student model to learn the output feature values ​​of the teacher model for the same category in different samples. Finally, based on the first, second, and third loss functions, an image loss function is constructed to iteratively train the student model. This allows the student model to not only learn the teacher model's prediction of the target category of image samples, but also the teacher model's feature representation of the same image sample and the feature representation of the same category in different samples. Even when the model architectures of the student model and the teacher model are significantly different, the student model can still fully learn the knowledge of the teacher model, thereby improving the image feature extraction accuracy of the student model. Attached Figure Description

[0051] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0052] Figure 1 Flowchart of the image feature extraction method based on knowledge distillation provided in this application;

[0053] Figure 2 A schematic diagram of intra-class knowledge distillation and inter-class knowledge distillation provided for this application;

[0054] Figure 3 A schematic diagram illustrating the early hints principle for knowledge learning provided in this application;

[0055] Figure 4 A schematic diagram of the image feature extraction device based on knowledge distillation provided in this application;

[0056] Figure 5 This application provides a schematic diagram of the knowledge distillation principle based on decoupling.

[0057] Figure 6 This application provides a schematic diagram illustrating the degree of discreteness in the feature representation of image feature extraction models obtained using different knowledge distillation methods; wherein, Figure 6 In the diagram, (a) represents the degree of discreteness of the model feature representation obtained based on the traditional knowledge distillation method. Figure 6 In the diagram, (b) represents the degree of discreteness of the model feature representation obtained by the decoupled knowledge distillation method. Figure 6 (c) in the diagram represents the degree of discreteness of the model feature representation obtained by the knowledge distillation method provided in this application;

[0058] Figure 7 A schematic diagram illustrating the correlation between the student model logic output and the teacher model logic output obtained based on different knowledge distillation methods, provided for this application; wherein, Figure 7 (a) in the diagram represents the correlation between the logical output of the student model and the logical output of the teacher model obtained based on the traditional knowledge distillation method. Figure 7 (b) in the diagram represents the correlation between the student model logic output and the teacher model logic output obtained by the decoupled knowledge distillation method. Figure 7 (c) in the diagram represents the correlation between the student model logic output and the teacher model logic output obtained by the knowledge distillation method provided in this application;

[0059] Figure 8 A schematic diagram illustrating the accuracy of the student model obtained based on the knowledge distillation method provided in this application on the validation set. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0061] Please see Figure 1 , Figure 1 The flowchart of the image feature extraction method based on knowledge distillation provided in this application is as follows:

[0062] S10: Input multiple image samples from the training set into the teacher model for feature extraction, and obtain the first feature vector corresponding to each image sample based on the output of the teacher model; input multiple image samples into the student model for feature extraction, and obtain the second feature vector corresponding to each image sample based on the output of the student model;

[0063] S20: Combine the eigenvalues ​​with the same order in the first feature vectors corresponding to multiple image samples to obtain m first combination vectors; combine the eigenvalues ​​with the same order in the second feature vectors corresponding to multiple image samples to obtain m second combination vectors; where m represents the number of eigenvalues ​​in the first and second feature vectors;

[0064] For example, such as Figure 2 As shown in the figure, the data in the two dashed boxes are the output results of the teacher model and the student model after extracting features from three image samples, respectively; each row of data in the dashed box corresponds to the output of one image sample, and each column represents the output feature value of the same category in different samples, that is, the feature values ​​with the same order in the feature vectors corresponding to different samples;

[0065] S30: Construct the target class knowledge distillation loss function for each image sample based on the first feature vector and the second feature vector corresponding to that image sample; obtain the first loss function by summing the target class knowledge distillation loss functions of multiple image samples;

[0066] S40: Based on the Pearson correlation coefficient between the first feature vector and the second feature vector corresponding to each image sample, construct the Pearson correlation coefficient loss function for that image sample; obtain the second loss function by summing the Pearson correlation coefficient loss functions of multiple image samples;

[0067] S50: Based on the Pearson correlation coefficient between the i-th first combined vector and the i-th second combined vector, construct the Pearson correlation coefficient loss function for the i-th combined vector; obtain the third loss function based on the sum of the Pearson correlation coefficient loss functions of the m combined vectors; where i∈[1,m];

[0068] S60: The first loss function, the second loss function, and the third loss function are weighted and summed to obtain the image feature extraction loss function. The student model is iteratively trained using image samples in the training set until the value of the image feature extraction loss function is minimized, thus obtaining the trained student model.

[0069] The image feature extraction method based on knowledge distillation provided in this application differs from existing technologies that only construct intra-class distillation loss. Instead, it constructs a target class knowledge distillation loss based on the first and second feature vectors of the same sample, thus obtaining a first loss function based on the target class knowledge distillation loss of all samples. This constrains the student model's prediction of the target class for the image sample to approach that of the teacher model. Simultaneously, a Pearson correlation coefficient loss function is constructed based on the Pearson correlation coefficient loss function between the first and second feature vectors corresponding to the same sample. This results in a second loss function based on the Pearson correlation coefficient loss function of all samples, constraining the difference between the feature vectors extracted by the student model and the teacher model for the same sample. Furthermore, a third loss function is constructed based on the Pearson correlation coefficient between the same ranked feature values ​​in the first and second feature vectors corresponding to different samples. This enables the student model to learn the output feature values ​​of the teacher model for the same class in different samples. This allows the student model to not only learn the teacher model's prediction of the target class for the image sample but also the teacher model's feature representation for the same image sample and the feature representation for the same class in different samples. This effectively transfers the knowledge of the teacher model to the student model, thereby improving the image feature extraction accuracy of the student model.

[0070] Specifically, the target class knowledge distillation loss function for the image samples in step S30 is expressed as:

[0071]

[0072] Among them, TCKD j Let KL denote the target class knowledge distillation loss function for the j-th image sample. and Similarity between them Let represent the target category prediction probability obtained after softmax normalization of the first feature vector of the j-th image sample. The target category prediction probability is obtained by normalizing the second feature vector of the j-th image sample using softmax.

[0073] For example, the first feature vector, after being normalized by softmax, will yield a class prediction probability vector {p1, p2, ..., p...}. c}, where p c Let p represent the probability that the image sample corresponding to the first feature vector belongs to the c-th class. Assuming the label of the image sample is the c-th class, the predicted probability of the target class obtained after softmax normalization of the first feature vector of the image sample is p. c ;

[0074] Specifically, LK divergence is mainly used in the field of knowledge distillation to measure the difference between two probability distributions. Its inputs are the predictions from two softmax models, as shown in the formula. Where p(x) represents the prediction probability of the teacher model for sample x, and q(x) represents the prediction probability of the student model for sample x;

[0075] The first loss function is expressed as:

[0076]

[0077] Where TCKD represents the first loss function.

[0078] Furthermore, the Pearson correlation coefficient loss function for the image samples in step S40 is expressed as:

[0079]

[0080] Among them, Pearson j Let Pearson correlation coefficient loss function be the value of the j-th image sample. This represents the first feature vector of the j-th image sample. This represents the second feature vector of the j-th image sample;

[0081] Specifically, the Pearson correlation coefficient can be used to represent the strength of the linear relationship between two data sets or vectors. Its value ranges from -1 to 1, where -1 indicates a perfect negative correlation (i.e., an increase in one variable leads to a proportional decrease in the other), 0 indicates no correlation, and 1 indicates a perfect correlation (i.e., an increase in one variable leads to a proportional increase in the other). When used as a loss function in knowledge distillation, it measures the difference between the predicted values ​​of the teacher-student models, aiming for closer similarity between their predictions. The specific calculation formula is as follows:

[0082]

[0083] Where Pearson(x,y) represents the Pearson correlation coefficient between datasets x and y, x l This represents the l-th data point in dataset x. Let L represent the mean of all data in dataset x, and let L represent the total number of data points in datasets x and y. l This represents the l-th data point in dataset y. This represents the mean of all data in dataset y;

[0084] Optionally, in other embodiments of this application, the first feature vector and the second feature vector may be normalized first to obtain two category prediction probability vectors corresponding to the image sample, and then the Pearson correlation coefficient loss function may be constructed based on the Pearson correlation coefficient of the two category prediction probability vectors.

[0085] The second loss function is expressed as:

[0086]

[0087] Among them, Pearson (P T ||P S ) represents the second loss function.

[0088] The Pearson correlation coefficient loss function for the i-th combined vector in step S50 is expressed as:

[0089]

[0090] Among them, Pearson i Let Pearson correlation coefficient loss function be the value of the i-th combined vector. Represents the i-th first combination vector. Represents the i-th second combination vector;

[0091] The third loss function is expressed as:

[0092]

[0093] Among them, Pearson (P Tz ||P Sz ) represents the third loss function.

[0094] Furthermore, based on the first loss function, the second loss function, and the third loss function described above, the image feature extraction loss function in step S60 is expressed as follows:

[0095] KD=α·TCKD+β·(Pearson(P T ||P S )+Pearson(P Tz ||P Sz )),

[0096] Where KD represents the image feature extraction loss function, α represents the weight of the first loss function, TCKD represents the first loss function, β represents the weight of the second and third loss functions, and Pearson(P T ||P S ) represents the second loss function, Pearson(P Tz ||P Sz ) represents the third loss function.

[0097] In the image feature extraction loss function, the first loss function only focuses on the similarity of the student model and the teacher model's predicted values ​​for the target category of the image sample, while the second and third loss functions both focus on the similarity of the student model and the teacher model's predicted values ​​for non-target categories. If the teacher model's predicted value for the target category is higher, the predicted value for the non-target category is lower, containing less information, and in this case, the weight of non-target category distillation should be reduced. Conversely, if the teacher model's predicted value for the target category is lower, the predicted value for the non-target category is higher, containing more information, and in this case, the weight of non-target category distillation should be increased. Therefore, in some embodiments of this application, the weights of the second and third loss functions are adjusted based on the teacher model's predicted probability value for the target category.

[0098] Specifically, the first feature vector corresponding to each image sample is normalized to output the category prediction probability value of each image sample; the average of the category prediction probability values ​​of multiple image samples is taken to obtain the target category prediction probability value.

[0099] The weights of the second and third loss functions are updated based on the predicted probability values ​​of the target category to obtain the target image feature extraction loss function;

[0100] Specifically, the target image feature extraction loss function is expressed as:

[0101]

[0102] Where KD′ represents the target image feature extraction loss function, This represents the predicted probability value of the target category, where k is a preset coefficient;

[0103] For example, in one specific embodiment of this application, the value of k is 1.5, which is to maintain... The value is approximately 1.

[0104] Distillation temperature is a commonly used hyperparameter in knowledge distillation. It can be used to control the strength of knowledge transfer between the teacher model and the student model. Lower distillation temperatures emphasize the detailed information of the teacher model, while higher distillation temperatures make it easier for the student model to learn from the teacher model.

[0105] In existing knowledge distillation methods, the temperature factor is generally set to a fixed value of 4. This application proposes a temperature adaptive algorithm to adjust the temperature factor during the knowledge distillation process, thereby adjusting the probability distribution of the teacher model. The higher the prediction probability of the teacher model for the correct category, the more accurate the teacher's knowledge is, and the temperature factor should be reduced accordingly to retain more knowledge details of the teacher model. Conversely, if the prediction probability of the teacher model for the correct category is low, it indicates that the teacher's knowledge is relatively inaccurate, and the temperature factor should be increased to smooth out the less accurate teacher knowledge, preventing the student model from learning incorrect knowledge and affecting the image feature extraction accuracy of the student model.

[0106] Specifically, in this embodiment of the application, the temperature factor is obtained by scaling and inverting the predicted probability value of the target category, and then the output probability distribution of the teacher model is adjusted based on the temperature factor.

[0107] Table 1 shows the adaptive temperature algorithm provided in the embodiments of this application:

[0108] Table 1

[0109] #correct_labels: Sample labels #logits_teacher: The teacher model's predictions for the sample. #scale: Scaling factor, hyperparameter #r: Responsible for inverting, hyperparameter correct_labels=target.unsqueeze(1) pred_teacher_all = F.softmax(logits_teacher, dim = 1) / / Normalize the predicted values `correct_prediction_probs = torch.gather(pred_teacher_all, 1, correct_labels)` / / Predicted probability values ​​for the correct class. temperature = torch.mean(correct_prediction_probs) / / Use the same temperature factor for a batch of categories temperature[temperature<=0.9]=0.9001 / / Ensure temperature range temperature = (temperature % 1 * 10000).long() / / Get the decimal part of the number. temperature=(temperature / scale) / / scale temperature=r–temperature / / reverse

[0110] Specifically, in Table 1, scale and r are hyperparameters responsible for mapping the predicted probability values ​​to the temperature coefficient. In a specific example of this application, the value of scale is 220, and the value of r varies from 6.5 to 8 depending on the model.

[0111] Regarding the setting of the temperature factor, this application uses the accuracy of the teacher model as the criterion. For teacher models with higher accuracy, the knowledge they provide is more accurate, and reducing the temperature factor can enable student models to learn more accurate teacher knowledge. For teacher models with lower accuracy, the knowledge they provide is limited and the accuracy is not high. A small temperature factor will cause student models to learn contradictory information between the real labels of image samples and teacher knowledge, so the temperature factor should be increased. For example, in the embodiments of this application, a small temperature factor (T≈2.5) is used for the ResNet teacher model, and a larger temperature factor (T≈4) is used for teacher models of other architectures.

[0112] Optionally, in some embodiments of this application, early hints for knowledge learning may also be introduced, such as... Figure 3As shown, not only does the student model learn the knowledge of the teacher model at the output layer, but the intermediate layers of the student model are also connected to the output layer of the teacher model. This allows the output layer of the teacher model to guide not only the output layer of the student model but also the intermediate layers of the student model, enabling the student model to fully learn the task. Theoretically, using more intermediate layers of the student model would yield better results. However, considering factors such as computational cost, this embodiment only uses the output of the penultimate stage of the student model. A mean squared error loss function is constructed based on the output of the intermediate layers of the student model and the output of the output layer of the teacher model. This mean squared error loss function is then weighted and summed with the first, second, and third loss functions in the above embodiment to obtain the image feature extraction loss function for training the student model.

[0113] Based on the image feature extraction method based on knowledge distillation provided in the above embodiments, this application also provides an image feature extraction device based on knowledge distillation, such as... Figure 4 As shown, the device specifically includes:

[0114] The feature extraction module 10 is used to input multiple image samples from the training set into the teacher model for feature extraction, and obtain the first feature vector corresponding to each image sample based on the output of the teacher model; and to input multiple image samples into the student model for feature extraction, and obtain the second feature vector corresponding to each image sample based on the output of the student model.

[0115] The feature combination module 20 is used to combine the feature values ​​with the same order in the first feature vectors corresponding to multiple image samples to obtain m first combination vectors; and to combine the feature values ​​with the same order in the second feature vectors corresponding to multiple image samples to obtain m second combination vectors; where m represents the number of feature values ​​in the first feature vectors and the second feature vectors.

[0116] The first loss function construction module 30 is used to construct the target class knowledge distillation loss function of each image sample based on the first feature vector and the second feature vector of each image sample; and to obtain the first loss function based on the sum of the target class knowledge distillation loss functions of multiple image samples.

[0117] The second loss function construction module 40 is used to construct a Pearson correlation coefficient loss function for an image sample based on the Pearson correlation coefficient between the first feature vector and the second feature vector corresponding to each image sample; and to obtain the second loss function by summing the Pearson correlation coefficient loss functions of multiple image samples.

[0118] The third loss function construction module 50 is used to construct the Pearson correlation coefficient loss function of the i-th combination vector based on the Pearson correlation coefficient between the i-th first combination vector and the i-th second combination vector; and to obtain the third loss function based on the sum of the Pearson correlation coefficient loss functions of the m combination vectors; where i∈[1,m];

[0119] The model training module 60 is used to perform a weighted summation of the first loss function, the second loss function, and the third loss function to obtain the image feature extraction loss function. The student model is then iteratively trained using image samples from the training set until the value of the image feature extraction loss function is minimized, thus obtaining a well-trained student model.

[0120] Example 2

[0121] The image feature extraction method based on knowledge distillation provided in Embodiment 1 above can be applied to both heterogeneous teacher-student models and homogeneous teacher-student models. In other embodiments, the above image feature extraction method can also be combined with existing image feature extraction methods for homogeneous models to obtain an image feature extraction method that is applicable to both homogeneous and heterogeneous teacher-student models.

[0122] Existing image feature extraction methods for isomorphic teacher-student models use a loss function composed of a weighted sum of two parts: target class knowledge distillation and non-target class knowledge distillation. Figure 5 As shown, its specific representation is as follows:

[0123] KD = α·TCKD + β·NCKD,

[0124] Where TCKD represents the target class distillation loss function, α represents the weight of the target class distillation loss function, NCKD represents the non-target class distillation loss function, and β represents the weight of the non-target class distillation loss function. in, This represents the probability that the teacher model predicts for the non-target class of an image sample. This represents the probability of the student model predicting the non-target class of an image sample;

[0125] Because the ability gap between isomorphic teacher-student models is small, the teacher model can only provide limited knowledge. Relaxing the matching precision would further weaken the guidance of the teacher model, making the information provided by the already limited teacher model even weaker. Therefore, precise matching should be strengthened. Based on this, this application embodiment uses the L2 norm to further constrain the existing loss function, making the teacher-student model matching more accurate. Specifically, the image feature extraction loss function of the isomorphic teacher-student model constrained by the L2 norm is expressed as follows:

[0126]

[0127] in, Let be the mean squared error loss function.

[0128] Optionally, early hints for knowledge learning can be introduced in the knowledge distillation of the isomorphic teacher-student model to establish a connection between the intermediate layer of the student model and the output layer of the teacher model. This allows the output layer of the teacher model to guide not only the output layer of the student model but also the intermediate layer of the student model, thereby enabling the student model to fully learn the task.

[0129] Optionally, for the image feature extraction loss function of the isomorphic teacher-student model, the weights of the non-target class distillation loss can also be adjusted based on the target class prediction value of the teacher model. Specifically, the adjusted image feature extraction loss function can be expressed as:

[0130]

[0131] Specifically, based on the image feature extraction loss function for the homogeneous teacher-student model and the image feature extraction loss function for the heterogeneous teacher-student model provided in the above embodiments, the target image feature extraction loss function is obtained as follows:

[0132]

[0133] When training the student model based on the above target image feature extraction loss function, it is necessary to first determine whether the model architecture of the student model and the teacher model are the same, so as to select the appropriate loss function to iteratively train the student model.

[0134] To verify the effectiveness of the above feature extraction method, this application embodiment tested the above method on the general datasets CIFAR-100, ImageNet, Stanford Dogs and the food domain datasets VegFru-292 and Ingredients-51.

[0135] 1. Data Augmentation:

[0136] For the CIFAR-100 dataset, this application embodiment uses the standard data augmentation of the CIFAR dataset, that is, the image edges are zero-padded with 4 pixels and randomly cropped to produce a 32*32 image, while using random horizontal flipping;

[0137] For the ImageNet, Stanford Dogs, VsgFru-292, and Ingredients-51 datasets, this application embodiment uses standard data augmentation to resize the images to 256*256 and randomly crop them to 224*224.

[0138] 2. The hyperparameter settings for the student model on different datasets are shown in Table 3:

[0139] Table 3

[0140] CIFAR-100 64 240 <![CDATA[5×10 -4 ]]> 0.05 / 0.01 150、180、210 ImageNet 512 100 <![CDATA[10 -4 ]]> 0.2 30、60、90 Stanford Dogs 128 100 <![CDATA[10 -4 ]]> 0.05 / 0.02 Cosine annealing decay VegFru-292 128 100 <![CDATA[10 -4 ]]> 0.2 30、60、90 Ingredients-51 128 100 <![CDATA[10 -4 ]]> 0.02 30、60、90

[0141] Specifically, for the CIFAR-100 dataset, SGD was used for 240 training epochs with a batch size of 64. The initial learning rate for ShuffleNet-V2 and MobileNet-V2 was 0.01, while the initial learning rate for other models was 0.05. The learning rate was linearly warmed up in the first 20 epochs and then decayed to 0.1 times its original value at epochs 150, 180, and 210. The weights decayed by 5 × 10⁻⁶. -4 The momentum is 0.9, the weight of the cross-entropy loss is fixed at 1, the weight of the target class distillation is 1, and the weight of the early cue for learning is fixed at 0.5.

[0142] For ImageNet, the SGD optimizer was used to train for 100 epochs with an initial learning rate of 0.2. At epochs 30, 60, and 90, the learning rate was decayed to 0.1 times its original value, and the weight decay was set to 10. -4 Momentum is set to 0.9, and batch size is set to 512;

[0143] For the Stanford Dogs dataset, the teacher model was trained for 200 epochs using the SGD optimizer with an initial learning rate of 0.6 and a cosine annealing training strategy. The student model was trained for 100 epochs, with an initial learning rate of 0.05 for MobileNet-v1 and 0.02 for ResNet18, using a cosine annealing training strategy with weight decay of 10. -4 The momentum is set to 0.9, and the batch size is 128.

[0144] For the VegFru-292 dataset, the teacher model was trained for 200 epochs using the SGD optimizer with an initial learning rate of 0.2, which was then decayed to half its original value at epochs 60, 120, and 180. The student model was trained for 100 epochs with an initial learning rate of 0.2, which was then decayed to half its original value at epochs 30, 60, and 90, with the weight decay set to 10. -4 The momentum was set to 0.9, and the batch size was 128.

[0145] For the Ingredients-51 dataset, the teacher model was trained for 200 epochs using the SGD optimizer and cosine annealing with an initial learning rate of 0.02. The student model was trained for 100 epochs with an initial learning rate of 0.02, and the learning rate was decayed to 0.1 times its original value at epochs 30, 60, and 90, with the weight decay set to 10.-4 The momentum is set to 0.9, and the batch size is 128.

[0146] Table 4 shows the experimental results of the method provided in this application embodiment on the CIFAR-100 dataset when the teacher-student model is a heterogeneous model:

[0147] Table 4

[0148]

[0149] Table 5 shows the experimental results of the method provided in this application embodiment on the CIFAR-100 dataset when the teacher-student model is a homogeneous model:

[0150] Table 5

[0151]

[0152]

[0153] As can be seen from the data in Tables 4 and 5, the method provided in this application embodiment did not achieve the best results on some teacher-student models. This is because the logical information of the output layer of the teacher model is limited. In particular, for isomorphic teacher-student models, the information of their output layer is similar, and students can only obtain very little knowledge from the teacher model.

[0154] Table 6 shows the experimental results of the method provided in the embodiments of this application on the ImageNet dataset:

[0155] Table 6

[0156]

[0157] As can be seen, the method provided in this application surpasses other existing methods in both homogeneous and heterogeneous teacher-student models, achieving better results in Top-1 and Top-5 metrics. In particular, in the ResNet50 and MobileNet-V2 heterogeneous teacher-student models, the method provided in this application improves by nearly 0.8 percentage points compared to the decoupled knowledge distillation method, and also surpasses the feature-based knowledge distillation method.

[0158] Table 7 shows the experimental results of the method provided in the embodiments of this application on the Stanford Dogs fine-grained dataset:

[0159] Table 7

[0160]

[0161]

[0162] The food ingredient recognition dataset is a feature-domain dataset, involving the field of smart health, and has broad application prospects. Furthermore, the similarity between images of different categories of food ingredients is high, involving fine-grained image classification. Tables 8 and 9 show the experimental results of the method provided in this application's embodiments on the VegFru-292 fruit and vegetable dataset and the Ingredients-51 food ingredient dataset, respectively:

[0163] Table 8

[0164]

[0165] Table 9

[0166]

[0167] Furthermore, ablation experiments were conducted on the CIFAR-100 dataset in this embodiment. The homogeneous teacher-student model used ResNet32×4 to ResNet8×4 as the experimental object, and the heterogeneous teacher-student model used ResNet32×4 to Shuvffle-V2 as the experimental object, to verify the effectiveness of the homogeneous teacher-student model distillation strategy, the heterogeneous teacher-student model distillation strategy, the early hints for knowledge learning, the adaptive distillation temperature, and the adaptive non-target class weights provided in the above embodiments.

[0168] Tables 10 and 11 show the results of ablation experiments on the CIFAR-100 dataset, using ResNet32×4—ResNet8×4 and ResNet32×4—Shuvffle-V2 as the experimental subjects, respectively:

[0169] Table 10

[0170] × × × × 75.99％ √ × × × 76.24％ √ √ × × 76.45％ √ √ √ × 76.61％ √ √ √ √ 76.71％

[0171] Table 11

[0172] × × × × 76.93％ √ × × × 77.92％ √ √ × × 77.71％ √ √ √ × 77.97％ √ √ √ √ 78.02％

[0173] The data in the table shows that when only the heterogeneous teacher-student model distillation strategy is used, the accuracy is improved by nearly 1 percentage point. When only the homogeneous teacher-student model distillation strategy is used, the accuracy improvement is limited. It is necessary to combine adaptive distillation temperature, adaptive non-target class weights, and early knowledge hints to achieve better results. This is because the homogeneous teacher-student model has a similar understanding and judgment of images and needs more subtle knowledge hints. On the other hand, the heterogeneous teacher-student model has a different understanding and judgment of images. Therefore, the heterogeneous teacher-student model distillation strategy provided in this application can achieve better results.

[0174] The embodiments of this application also present visualization results from three aspects: the degree of discreteness of feature representation (t-SNE), the logistic relevance of model prediction output, and the validation set accuracy curve. The teacher model is set to ResNet32×4, and the student model is set to ResNet8×4.

[0175] First, t-SNE is used to reduce the dimensionality of the features. Different colors are used to represent different categories, visually displaying the distribution relationship between the features extracted by the model and the categories in three-dimensional space. If the data points of different categories are relatively discretely distributed in space, it means that the model can effectively distinguish different categories when extracting features; conversely, it indicates that the model has difficulty distinguishing certain categories. Figure 6 As shown, where, Figure 6 In the diagram, (a) represents the degree of discreteness of the model feature representation obtained based on the traditional knowledge distillation (KD) method. Figure 6 In the diagram, (b) represents the degree of discreteness of the model feature representation obtained by the decoupled knowledge distillation method (QKD). Figure 6 (c) in the figure represents the degree of dispersion of the model feature representation obtained by the knowledge distillation method provided in this application. As can be seen from the figure, the feature representation output by the student model obtained in this application is more separable. The distribution of data points of different categories is relatively discrete, while the distribution of data points of the same category is relatively concentrated. This indicates that the feature extraction accuracy of the model is high and can better distinguish different categories.

[0176] Furthermore, this application embodiment also uses a difference correlation matrix to visualize the correlation between the logical output of the student model and the logical output of the teacher model, such as... Figure 7 As shown, where, Figure 7 In the diagram, (a) represents the correlation between the logical output of the student model and the logical output of the teacher model obtained based on the traditional knowledge distillation method. Figure 7 In the diagram, (b) represents the correlation between the student model's logical output and the teacher model's logical output obtained using the decoupled knowledge distillation method. Figure 7 In the figure, (c) represents the correlation between the logical output of the student model and the logical output of the teacher model obtained by the knowledge distillation method provided in this application; each cell in the figure represents the degree of difference between the logical outputs of the two models in the corresponding categories;

[0177] To observe the convergence of the student model during training, this application also visualizes the accuracy of the student model on the validation set, such as... Figure 8 As shown in the figure, compared with knowledge distillation based on decoupling, the student model obtained by the knowledge distillation method provided in this application not only has higher feature extraction accuracy, but also converges more easily.

[0178] Optionally, in some embodiments of this application, the student model can be lightweighted by first using a non-filled downsampling module and a hybrid stem layer, and then knowledge distillation can be performed on the modified student model.

[0179] Specifically, the student model includes a stem module and multiple feature extraction modules with different kernel sizes that are sequentially connected in series along the forward propagation direction. Each feature extraction module includes a no-padding downsampling submodule and a feature extraction convolution submodule. Each no-padding downsampling submodule includes a no-padding convolution unit, a no-padding max pooling unit, and a feature fusion unit.

[0180] To verify the feasibility of performing knowledge distillation after lightweighting the student model, this application embodiment conducted experiments on four lightweight models: MobileNet_v3, Parc MobileNet_v2, ShuffleNet_v2, and GhostNet. The hyperparameter settings for the experiments are shown in Table 12, and the experimental results are shown in Table 13.

[0181] Table 12

[0182] CIFAR-100 64 240 <![CDATA[5×10 -4 ]]> 0.05 / 0.01 150、180、210 VegFru-Ingredients-313 128 100 <![CDATA[10 -4 ]]> 0.2 Cosine annealing decay

[0183] Table 13

[0184]

[0185] In addition, experiments were also conducted on the VegFru-Ingredients-313 dataset in this application embodiment, and the experimental results are shown in Table 14.

[0186] Table 14

[0187]

[0188] As can be seen from the data in Tables 13 and 14, the lightweight modification of the model based on the knowledge distillation method provided in this application can not only reduce the computational load of the model and achieve model lightweighting, but also improve the feature extraction accuracy of the model.

[0189] There are many commonly used mobile terminal deployment frameworks, such as TensorFlow Lite, PyTorch_Mobile, and TNN. This application's embodiment selects the PyTorch_Mobile inference framework for model deployment. PyTorch_Mobile is a subset of PyTorch optimized for mobile and edge devices, supporting direct deployment of PyTorch models to Android devices for local inference without server dependency. PyTorch provides a complete end-to-end process for converting PyTorch models to PyTorch_Mobile models. Simply use `torch.jit.trace` to convert the PyTorch-generated model file with the .pth extension to a .pt extension. The algorithm flow is shown in Table 15.

[0190] Table 15

[0191] #float_model: The original model in PyTorch format #light.pt: The converted model file float_model.eval() / / Set the model to inference mode input_tensor = torch.rand(1, 3, 224, 224).to(device) / / Set the input image size traced_model = torch.jit.trace(float_model, input_tensor) / / Model format conversion torch.jit.save(traced_model,"light.pt") / / Save the converted model

[0192] During the model conversion process, PyTorch provides several selectable model optimization APIs that can perform optimization operations such as quantization and pruning on the model.

[0193] By deploying the above model to Android devices, it is also possible to build intelligent food identification software for health and wellness. The software can obtain image information of food by taking pictures with a mobile phone, call a deep learning model to identify the food category, and obtain the weight information of food through an electronic kitchen Bluetooth scale. This can record the user's daily dietary intake, help manage diet, and improve health.

[0194] Specifically, the software has three user types: regular users, nutritionists, and super administrators. Nutritionists can view users' historical food consumption data and provide dietary advice, super administrators are responsible for changing user information, and regular users can take photos for recognition and weigh themselves.

[0195] Users upload photos of ingredients and click the recognition button to use a deep learning model to extract features and classify the ingredients. Once the electronic Bluetooth scale is paired with the Android phone's software, the weighing interface of the software is opened, displaying the currently read weight data, such as process data and locked data, as well as historical weight data such as year, month, day, time, weight type, and weight information. When the ingredients are not placed securely, the weight data will change and the process data will be displayed. After the ingredients are placed securely, the data will be fixed and the locked data will be displayed, and the data will be saved to the historical data record. Nutritionists can query the historical eating history of selected ordinary users and provide corresponding suggestions. The system can automatically record the date and submit the nutritionist's suggestions to the ordinary users.

[0196] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0200] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An image feature extraction method based on knowledge distillation, characterized in that, include: Multiple image samples from the training set are input into the teacher model for feature extraction, and the first feature vector corresponding to each image sample is obtained based on the output of the teacher model. The multiple image samples are input into the student model for feature extraction, and a second feature vector corresponding to each image sample is obtained based on the output of the student model. The student model includes a STEM module and multiple feature extraction modules with different kernel sizes connected in series along the forward propagation direction. Each feature extraction module includes a padding-free downsampling submodule and a feature extraction convolution submodule. Each padding-free downsampling submodule includes a padding-free convolution unit, a padding-free max pooling unit, and a feature fusion unit. The same sorted feature values ​​in the first feature vectors corresponding to multiple image samples are combined to obtain m first combination vectors; the same sorted feature values ​​in the second feature vectors corresponding to multiple image samples are combined to obtain m second combination vectors; where m represents the number of feature values ​​in the first feature vectors and the second feature vectors. A target class knowledge distillation loss function is constructed based on the first feature vector and the second feature vector corresponding to each image sample; the first loss function is obtained by summing the target class knowledge distillation loss functions of multiple image samples. Based on the Pearson correlation coefficient between the first feature vector and the second feature vector corresponding to each image sample, a Pearson correlation coefficient loss function is constructed for that image sample; the second loss function is obtained by summing the Pearson correlation coefficient loss functions of multiple image samples. Based on the Pearson correlation coefficient between the i-th first combined vector and the i-th second combined vector, a Pearson correlation coefficient loss function is constructed for the i-th combined vector; a third loss function is obtained by summing the Pearson correlation coefficient loss functions of the m combined vectors; where... ; The first loss function, the second loss function, and the third loss function are weighted and summed to obtain the image feature extraction loss function. The student model is then iteratively trained using image samples from the training set until the value of the image feature extraction loss function is minimized, thus obtaining a well-trained student model.

2. The image feature extraction method based on knowledge distillation according to claim 1, characterized in that, The target class knowledge distillation loss function for image samples is expressed as: ， in, Indicates the first The target class knowledge distillation loss function for each image sample express and Similarity between them Indicates the first The target category prediction probability is obtained by normalizing the first feature vector of each image sample using softmax. Indicates the first The target category prediction probability is obtained by normalizing the second feature vector of each image sample using softmax. The first loss function is expressed as: ， in, This represents the first loss function.

3. The image feature extraction method based on knowledge distillation according to claim 2, characterized in that, The Pearson correlation coefficient loss function for image samples is expressed as: ， in, Indicates the first The Pearson correlation coefficient loss function for each image sample. Indicates the first The first feature vector of each image sample Indicates the first The second feature vector of each image sample; The second loss function is expressed as: ， in, This represents the second loss function.

4. The image feature extraction method based on knowledge distillation according to claim 3, characterized in that, The Pearson correlation coefficient loss function for the i-th combined vector is expressed as: ， in, Let Pearson correlation coefficient loss function be the value of the i-th combined vector. Represents the i-th first combination vector. Represents the i-th second combination vector; The third loss function is expressed as: ， in, This represents the third loss function.

5. The image feature extraction method based on knowledge distillation according to claim 4, characterized in that, The image feature extraction loss function is expressed as: ， in, This represents the image feature extraction loss function. The weights of the first loss function are represented. Denotes the first loss function. This represents the weights of the second and third loss functions. This represents the second loss function. This represents the third loss function.

6. The image feature extraction method based on knowledge distillation according to claim 5, characterized in that, Also includes: Normalize the first feature vector corresponding to each image sample and output the class prediction probability value of each image sample; The target class prediction probability value is obtained by averaging the class prediction probability values ​​of multiple image samples. The weights of the second and third loss functions are updated based on the predicted probability values ​​of the target category to obtain the target image feature extraction loss function.

7. The image feature extraction method based on knowledge distillation according to claim 6, characterized in that, The loss function for target image feature extraction is expressed as: ， in, This represents the loss function for target image feature extraction. This represents the predicted probability value for the target category. These are preset coefficients.

8. The image feature extraction method based on knowledge distillation according to claim 7, characterized in that, It also includes scaling and inverting the predicted probability value of the target category to obtain a temperature factor, thereby adjusting the output probability distribution of the teacher model based on the temperature factor.

9. An image feature extraction device based on knowledge distillation, characterized in that, The apparatus is used to implement the image feature extraction method based on knowledge distillation as described in any one of claims 1 to 8, comprising: The feature extraction module is used to input multiple image samples from the training set into the teacher model for feature extraction, and obtain a first feature vector corresponding to each image sample based on the output of the teacher model; and to input the multiple image samples into the student model for feature extraction, and obtain a second feature vector corresponding to each image sample based on the output of the student model. The feature combination module is used to combine the feature values ​​with the same order in the first feature vectors corresponding to multiple image samples to obtain m first combination vectors; and to combine the feature values ​​with the same order in the second feature vectors corresponding to multiple image samples to obtain m second combination vectors; where m represents the number of feature values ​​in the first feature vectors and the second feature vectors. The first loss function construction module is used to construct the target class knowledge distillation loss function for each image sample based on the first feature vector and the second feature vector corresponding to that image sample; the first loss function is obtained by summing the target class knowledge distillation loss functions of multiple image samples. The second loss function construction module is used to construct the Pearson correlation coefficient loss function of the image sample based on the Pearson correlation coefficient between the first feature vector corresponding to each image sample and the second feature vector corresponding to the image sample; and to obtain the second loss function based on the sum of the Pearson correlation coefficient loss functions of multiple image samples. The third loss function construction module is used to construct the Pearson correlation coefficient loss function for the i-th combined vector based on the Pearson correlation coefficient between the i-th first combined vector and the i-th second combined vector; the third loss function is obtained by summing the Pearson correlation coefficient loss functions of the m combined vectors; where... ; The model training module is used to perform a weighted summation of the first loss function, the second loss function, and the third loss function to obtain an image feature extraction loss function. The student model is then iteratively trained using image samples from the training set until the value of the image feature extraction loss function is minimized, thus obtaining a trained student model.

Images