Model training method, image recognition method, equipment and storage medium

Abstract

The invention provides a knowledge distillation model-based model training method, an image recognition method, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining similarity data and distance data of a teacher model according to a first feature output vector corresponding to each initial training sample and a second feature output vector corresponding to each enhanced training sample output by an output layer of the teacher model; wherein the enhanced training sample is obtained after the initial training sample is enhanced; determining a similarity weight and a distance weight according to the similarity data and the distance data of the teacher model; based on the similarity weight, the distance weight, and the similarity data and the distance data of the student model, obtaining sample difference data; and training a student model according to the sample difference data, the initial training sample and the enhanced training sample to obtain a trained recognition model. According to the embodiment of the invention, the adaptability of a student model application scene can be improved, and the recognition precision is improved.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of artificial intelligence technology, and in particular to a model training method, image recognition method, device and storage medium based on a knowledge distillation model. Background Art

[0002] Neural network models can be applied to a variety of scenarios, such as image recognition, speech recognition, and intelligent customer dialogue. In related technologies, in order to improve the performance of neural network models (such as recognition accuracy, etc.), neural network models are usually trained based on data enhancement, knowledge distillation, etc., so that the neural network model can reduce complexity and resource dependence during the training process while improving the performance of the trained neural network model. However, in practical applications, the prediction accuracy of the teacher model for the enhanced training samples after data enhancement is usually low. The greater the data enhancement, the lower the teacher model's familiarity with the samples. Therefore, the knowledge distillation is more dependent on the resources of the initial training samples, which leads to the need to rely on more initial training samples for the training of the student model, resulting in low learning efficiency and performance of the student model. The generalization ability of the trained student model is poor and cannot adapt to data analysis in different scenarios. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a model training method, image recognition method, device and storage medium based on a knowledge distillation model, which can solve the technical problem of poor generalization ability of student models in related technologies and improve the adaptability of student models to application scenarios.

[0004] In a first aspect, an embodiment of the present application provides a model training method based on a knowledge distillation model, the method comprising: According to the first feature output vectors corresponding to each initial training sample and the second feature output vectors corresponding to each enhanced training sample outputted by the output layer of the preset teacher model, the similarity data and the distance data of the teacher model are obtained; wherein the enhanced training samples are obtained after the initial training samples are enhanced; Determine a similarity weight and a distance weight according to the similarity data and the distance data of the teacher model; Based on the similarity weight, the distance weight, the similarity data and the distance data of the student model, sample difference data is obtained; wherein the similarity data and the distance data of the student model are obtained according to the third feature output vector corresponding to each of the initial training samples and the fourth feature output vector corresponding to each of the enhanced training samples output by the output layer of the student model; The student model is trained according to the sample difference data, the initial training samples and the enhanced training samples to obtain a trained recognition model.

[0005] In some optional embodiments, obtaining the similarity data and distance data of the teacher model according to the first feature output vector corresponding to the initial training sample and the second feature output vector corresponding to the enhanced training sample output by the output layer of the preset teacher model includes: Input each of the initial training samples and each of the enhanced training samples into the teacher model, and obtain a first feature output vector corresponding to the initial training sample and a second feature output vector corresponding to the enhanced training sample output by the output layer of the teacher model respectively; Normalizing the first feature output vector and the second feature output vector respectively to obtain a first unit vector corresponding to the first feature output vector and a second unit vector corresponding to the second feature output vector; Obtaining a first similarity between each of the initial training samples and the corresponding enhanced training sample according to the transposed vector of the first unit vector and the corresponding second unit vector, and obtaining similarity data of the teacher model according to each of the first similarities; According to the Euclidean distance between the first unit vector and the corresponding second unit vector, the first sample distance between each of the initial training samples and the corresponding enhanced training sample is obtained, and according to each of the first sample distances, the distance data of the teacher model is obtained.

[0006] In some optional embodiments, determining the similarity weight and the distance weight according to the similarity data and the distance data of the teacher model includes: Performing one-dimensional vector conversion processing on the similarity data and the distance data respectively to obtain a similarity vector and a distance vector; Obtaining the similarity weight according to the similarity vector, the sample batch size of the teacher model and a preset temperature distribution parameter; The distance weight is obtained according to the distance vector, the sample batch size of the teacher model and the temperature distribution parameter.

[0007] In some optional embodiments, the acquiring of sample difference data based on the similarity weight, the distance weight, the similarity data of the student model, and the distance data includes: Constructing a similarity square difference matrix according to the similarity data of the teacher model and the student model of the same batch; Constructing a distance square difference matrix according to the distance data of the teacher model and the student model of the same batch; Perform matrix conversion on the similarity weight and the distance weight respectively to obtain a similarity matrix and a distance matrix; The similarity matrix and the distance matrix are respectively subjected to matrix dot multiplication with the similarity square difference matrix and the distance square difference matrix to obtain the sample difference data.

[0008] In some optional embodiments, the student model is trained according to the sample difference data, the initial training sample and the enhanced training sample to obtain a trained recognition model, including: According to the sample difference data, a difference loss value obtained by training the student model based on each of the initial training samples and each of the enhanced training samples of the current batch is obtained; and according to each of the difference loss values, an average difference loss value of the current batch is calculated; According to the average value of the difference loss of the current batch, the model parameters of the student model are adjusted and the step of obtaining the similarity data and distance data of the teacher model is jumped to perform the next batch of model training until the trained student model meets the preset convergence condition; The student model that meets the preset convergence condition is used as the trained recognition model.

[0009] In some optional embodiments, adjusting the model parameters of the student model according to the average difference loss of the current batch includes: Obtaining a classification loss value and a relative entropy of the student model, wherein the relative entropy is used to characterize the degree of difference between the output data of the student model and the teacher model in an output unit; The model parameters of the student model are adjusted according to the classification loss value, the relative entropy and the difference loss average value.

[0010] In some optional embodiments, the similarity data and distance data of the student model are obtained by the following steps: Input each of the initial training samples and each of the enhanced training samples into the student model, and obtain a third feature output vector corresponding to the initial training sample and a fourth feature output vector corresponding to the enhanced training sample output by the output layer of the student model respectively; Normalizing the third characteristic output vector and the fourth characteristic output vector respectively to obtain a third unit vector corresponding to the third characteristic output vector and a fourth unit vector corresponding to the fourth characteristic output vector; Obtaining a second similarity between each of the initial training samples and the corresponding enhanced training sample according to the third unit vector and the corresponding transposed vector of the fourth unit vector, and obtaining similarity data of the student model according to each of the second similarities; According to the Euclidean distance between the third unit vector and the corresponding fourth unit vector, the second sample distance between each of the initial training samples and the corresponding enhanced training sample is obtained, and according to each of the second sample distances, the distance data of the student model is obtained.

[0011] In a second aspect, an embodiment of the present application provides an image recognition method, the image recognition method comprising: Acquire an image recognition model, wherein the image recognition model is trained by any one of the methods described in the first aspect; The recognition image data is input into the image recognition model, and the image recognition data is output.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the model training method based on the knowledge distillation model as described in any one of the first aspects or the steps of the image recognition method as described in any one of the second aspects.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the steps of the model training method based on the knowledge distillation model as described in any one of the first aspects or the steps of the image recognition method as described in any one of the second aspects.

[0014] According to the above-mentioned embodiments of the present application, at least the following beneficial effects are achieved: by respectively determining the subtle differences between the initial training samples and the enhanced training samples based on the similarity data and the distance data of the teacher model, more knowledge can be mined from the limited sample data; at the same time, by introducing similarity weights and distance weights, the student model can learn samples with high differences, effectively reducing the need for a large amount of training data, and the generalization ability is stronger and the recognition accuracy is higher. Therefore, compared with the related art, the embodiments of the present application can improve the generalization ability of the student model and can adapt to data analysis in more scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a model training method based on a knowledge distillation model provided in this application; Figure 2 A schematic diagram of the structure of a teacher model in a knowledge distillation model provided for this application; Figure 3 A schematic diagram of an embodiment of a model training method based on a knowledge distillation model provided in this application; Figure 4A schematic diagram of a module of a device corresponding to a model training method based on a knowledge distillation model provided in this application; Figure 5 A schematic diagram of the hardware structure of a device corresponding to a model training method based on a knowledge distillation model provided in this application. DETAILED DESCRIPTION

[0016] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given: Understandably, referring to Figure 1 As shown, the model training method based on the knowledge distillation model provided in the embodiment of the present application includes: Step S100, obtaining similarity data and distance data of the teacher model according to the first feature output vectors corresponding to each initial training sample and the second feature output vectors corresponding to each enhanced training sample output by the output layer of the preset teacher model; wherein the enhanced training sample is obtained after the initial training sample is enhanced; Step S200, determining a similarity weight and a distance weight according to the similarity data and the distance data of the teacher model; Step S300, based on the similarity weight, the distance weight, the similarity data of the student model and the distance data, obtain the sample difference data; wherein the similarity data and the distance data of the student model are obtained according to the third feature output vector corresponding to each initial training sample and the fourth feature output vector corresponding to each enhanced training sample output by the output layer of the student model; Step S400: Train the student model according to the sample difference data, the initial training samples and the enhanced training samples to obtain a trained recognition model.

[0021] Therefore, by respectively determining the subtle differences between the initial training samples and the enhanced training samples based on the similarity data and the distance data of the teacher model, more knowledge can be mined from limited sample data. At the same time, by introducing similarity weights and distance weights, the student model can learn samples with high differences, effectively reducing the demand for a large amount of training data and having stronger generalization capabilities. Therefore, the embodiments of the present application can improve the generalization capabilities of the student model and can adapt to data analysis in more scenarios.

[0022] Understandably, referring to Figure 2 As shown, the teacher model usually includes an input layer, an intermediate layer and an output layer, wherein the input layer, the intermediate layer and the output layer can determine the relational feature knowledge, the output layer can determine the output feature knowledge, and the intermediate layer can determine the intermediate feature knowledge, wherein the relational feature knowledge refers to the relational knowledge between different layers of the teacher model and different data samples; the output feature knowledge usually refers to the last layer of features of the teacher model, mainly including the knowledge of the output layer and the soft target; and the features are extracted from the network layer in the middle of the teacher to serve as the hint (Hint) of the output of the intermediate layer of the student model; this process is called the knowledge distillation of the intermediate features, which not only requires the use of the output feature knowledge of the teacher model, but also requires the use of the feature map knowledge in the hidden layer of the teacher model. Structural feature knowledge is a complete knowledge system of the teacher model, which includes not only the output feature knowledge, intermediate feature knowledge and relational feature knowledge of the teacher, but also the knowledge of the regional feature distribution of the teacher model. Among them, the intermediate feature knowledge, the relational feature knowledge and the structural feature knowledge are all strongly related to the specific architecture of the teacher model, and in this application, by only using the output feature knowledge output in the output layer of the teacher model, without relying on the specific architecture of the teacher model, the heterogeneity of the teacher and student models can be allowed. At the same time, a large pre-trained teacher model with unknown structure can be used to process unlabeled data, and the prediction of the teacher model on these data can be used to guide the learning of the student model. This not only reduces the dependence on a large amount of labeled data, but also can use the existing large model's understanding of new data (that is, enhanced training samples) to enhance the learning results of the student model.

[0023] It is understandable that the embodiments of the present application do not limit the specific implementation methods of the enhancement processing. Technical personnel in this field can choose different enhancement processing methods according to actual needs, such as flipping, rotating, cropping, deforming, scaling, noise, blurring, color change, erasing, filling, etc.

[0024] It is understandable that the similarity data represents the similarity between each enhanced training sample in the training sample set and each initial training sample, and the distance data represents the Euclidean distance between each enhanced training sample in the training sample set and each initial training sample. An initial training sample can have at least one enhanced training sample. In this regard, the embodiment of the present application does not limit the quantitative relationship between the enhanced samples and the initial samples. For example, assuming that the training sample set includes N data sets, each data set includes an initial training sample and an enhanced training sample, then the similarity data includes the similarity between the initial training sample and each enhanced training sample in the N data sets. For example, for 2x enhancement, each data set can be represented as , where each initial training sample Both with an enhanced training sample pair. represents the true label associated with each initial training sample and each corresponding enhanced training sample, and K is the total number of image classes. For example, in the case of 4-fold enhancement, the dataset is expanded to , including two identical initial training samples and two different enhanced training samples. For higher multiples such as 6 times and 8 times, the data set is further increased by adding additional copies of the initial training samples and enhanced training samples for each initial training sample. Among them, the copies of the initial training samples are increased according to the number of enhanced training samples in the corresponding data set, so that the number of initial training samples and enhanced training samples in each data set is the same, and then each data set can be traversed according to the index value, thereby improving the convenience of data processing. In the following examples, the 2x data enhancement scenario is used as an example. At this time, the similarity data in each batch includes BxB similarities, and the distance data includes BxB Euclidean distances, where B is the number of data sets.

[0025] It can be understood that the similarity weight is the weight of the similarity data, and the distance weight is the weight of the distance data.

[0026] It is understandable that the similarity data and distance data of the student model can be calculated with reference to the similarity data and distance data of the teacher model. Therefore, when the similarity weight and distance weight are determined, the student model can efficiently utilize its capabilities and focus on more challenging differences when training the initial training samples and enhanced training samples based on the similarity weight and distance weight.

[0027] It can be understood that the sample difference data characterizes the difference in knowledge understanding between the student model's differential characteristics of each sample and the teacher model's differential characteristics of each sample.

[0028] It is understandable that step S100, according to the first feature output vector corresponding to the initial training sample and the second feature output vector corresponding to the enhanced training sample output by the output layer of the preset teacher model, obtains the similarity data and distance data of the teacher model, including: Input each initial training sample and each enhanced training sample into the teacher model, and obtain a first feature output vector corresponding to the initial training sample and a second feature output vector corresponding to the enhanced training sample output by the output layer of the teacher model respectively; Normalizing the first characteristic output vector and the second characteristic output vector respectively to obtain a first unit vector corresponding to the first characteristic output vector and a second unit vector corresponding to the second characteristic output vector; Obtaining a first similarity between each initial training sample and the corresponding enhanced training sample according to the transposed vector of the first unit vector and the corresponding second unit vector, and obtaining similarity data of the teacher model according to each first similarity; According to the Euclidean distance between the first unit vector and the corresponding second unit vector, the first sample distance between each initial training sample and the corresponding enhanced training sample is obtained, and the distance data of the teacher model is obtained according to each first sample distance.

[0029] The normalization process is used to normalize the first feature output vector and the second feature output vector to ensure that the size of each vector does not affect the similarity measurement. represents the logical output of the teacher model on the i-th initial training sample. represents the logical output of the teacher model on the i-th augmented training sample. Then a general normalization function g(x) is defined, and the feature vector is normalized by the normalization function g(x). and Normalized to a unit norm, where the normalization function g(x) is as shown in the following formula (1): (1) in Representation vector L 2 norm. At this time, each vector is represented only by its direction, which provides a standardized basis for calculating similarity and distance. Then when the feature vector and is normalized to a unit vector, then the first unit vector ; The second unit vector , at this time, it can be ensured that the comparison between vectors focuses on the angle difference rather than the size, which can further improve the accuracy of evaluating the difference knowledge between the initial training samples and the enhanced training samples.

[0030] It is understandable that, in some embodiments, the first similarity can be determined by multiplying the first unit vector and the transposed vector of the corresponding second unit vector. Exemplarily, the alignment of features between the initial training sample and the enhanced training sample can be determined with reference to formula (2) to obtain similarity data. Formula (2) is as follows: (2) in, is the transposed matrix obtained by transposing the matrix of the enhanced training sample after normalization; i and j are both indexes, i represents the index of the initial training sample in the current batch, and j represents the index of the enhanced training sample in the current batch. When the similarity data is represented by a matrix, is a similarity matrix of size B×B. Each element represents the cosine similarity between the i-th initial training sample and the j-th enhanced training sample. These matrices It can be used as a key indicator to evaluate how the teacher model perceives the difference between the initial training samples and the enhanced training samples. The similarity matrix reflects how each model maintains or changes its feature detection ability in response to data enhancement, as follows: 1) A high value of indicates that the augmented training samples perceived by the teacher model are similar to the initial training samples, indicating that feature extraction remains stable despite the introduced changes; 2) Lower values ​​indicate that the teacher model recognizes and responds to the changes caused by the augmentation, highlighting the dynamic adaptation to the changing data characteristics.

[0031] Similarly, for distance data, referring to formula (3), the alignment of features between the initial training sample and the enhanced training sample can be determined to obtain distance data. Formula (3) is as follows: (3) Where i and j are the indices of the initial training samples and enhanced training samples in the same batch, respectively. When expressed as a matrix, the size of the distance matrix is ​​B×B, where B is the batch size. Each element Quantifies the Euclidean distance between the i-th initial training sample and the j-th enhanced training sample. Values ​​indicate significant differences due to enhancement and vice versa.

[0032] It is understandable that step S200, determining the similarity weight and the distance weight according to the similarity data and the distance data of the teacher model, includes: The similarity data and the distance data are respectively converted into one-dimensional vectors to obtain a similarity vector and a distance vector; The similarity weight is obtained according to the similarity vector, the sample batch size of the teacher model and the preset temperature distribution parameters; The distance weight is obtained according to the distance vector, the sample batch size of the teacher model, and the temperature distribution parameter.

[0033] It is understandable that a sample-level importance weighting mechanism can be introduced to enhance the concentration of the student model in learning from diverse knowledge sources. This mechanism uses the experience of the teacher model to evaluate the relative importance of each differential knowledge in the batch. The one-dimensional vector conversion process can simplify the weight calculation and improve the calculation efficiency. The embodiment of the present application does not limit the specific method of the one-dimensional vector conversion process, and those skilled in the art can refer to the conversion method between a multi-dimensional matrix and a one-dimensional matrix for conversion.

[0034] Taking the similarity data represented in matrix form as an example, the similarity matrix and the distance matrix Convert to a one-dimensional vector and For the similarity matrix ( ), weight The calculation formula is as follows: (4) Wherein, formula (4) means that B The weight matrix is ​​obtained by combining the normalized values. τ is the temperature distribution parameter that controls the weight distribution. τ can be selectively set according to actual needs. Represents a hollow R to indicate the output The dimension size of Represents a one-dimensional vector Normalize each element in so that The value of each element in the calculated one-dimensional vector is between 0 and 1. Represents a one-dimensional vector For formula (4), there is a high similarity between the initial training samples and the enhanced training samples, which means that the enhanced training samples lack diversity and challenge relative to the initial training samples, resulting in lower weights. This ensures that the student model allocates less attention to sample pairs that the teacher model considers to be too similar. . As a result, the student model is able to efficiently use its capabilities and focus on more challenging differences.

[0035] Similarly, for the distance matrix , weight The calculation of is as follows: (5) Wherein, formula (5) means that B The weight matrix is ​​obtained by combining the normalized values. τ is the temperature distribution parameter that controls the weight distribution. τ can be selectively set according to actual needs. Represents a hollow R to indicate the output The dimension size of Represents a one-dimensional vector Normalize each element in so that The value of each element in the calculated one-dimensional vector is between 0 and 1. Represents a one-dimensional vector The sum is calculated for each element in . In the distance matrix, higher values ​​indicate a larger difference between the initial training samples and the enhanced training samples, showing more challenging instances for the student model to learn. Therefore, these higher values ​​are given greater weights. This ensures that the student model pays more attention to sample pairs with significant differences, allowing it to learn more effectively from complex and diverse data.

[0036] It is understandable that step S300, obtaining sample difference data based on the similarity weight, the distance weight, the similarity data of the student model and the distance data, includes: According to the similarity data of the teacher model and the student model in the same batch, a similarity square difference matrix is ​​constructed; According to the distance data of the teacher model and the student model in the same batch, a distance square difference matrix is ​​constructed; The similarity weight and the distance weight are transformed into matrices respectively to obtain the similarity matrix and the distance matrix; The similarity matrix and the distance matrix are respectively multiplied by the similarity square difference matrix and the distance square difference matrix to obtain the sample difference data.

[0037] Matrix conversion is used to convert similarity weights and distance weights into matrices that match the size of the similarity square difference matrix and the distance square difference matrix. The similarity matrix, distance matrix, similarity square difference matrix, and distance square difference matrix can be used to construct a difference loss function. Integrate the difference metric into the knowledge distillation process. The difference loss function aims to align the student model's understanding of sample differences with the insights of the teacher model, using the subtle differences between the identified initial training samples and the enhanced training samples. The difference loss function is as follows: (6) in Represents element-wise multiplication, ensuring Each element of accurately represents the weighted mean square difference between the student model and the teacher model in terms of similarity and distance perception. Each matrix, and and its student model counterpart, play a key role in measuring how the model perceives the impact of data enhancement on sample features. The sample difference data is the data set obtained based on formula (6).

[0038] It is understandable that step S400, training the student model according to the sample difference data, the initial training sample and the enhanced training sample to obtain the trained recognition model, includes: According to the sample difference data, the difference loss value obtained by the student model based on the initial training samples and enhanced training samples of the current batch is obtained; and the average difference loss value of the current batch is calculated according to each difference loss value; According to the average difference loss of the current batch, the model parameters of the student model are adjusted and the step of obtaining the similarity data and distance data of the teacher model is jumped to perform the next batch of model training until the trained student model meets the preset convergence conditions; The student model that meets the preset convergence conditions is used as the trained recognition model.

[0039] Understandably, Each element of corresponds to a difference loss value; the difference loss value represents the weighted mean square difference between the student model and the teacher model's perception of similarity and distance.

[0040] The average difference loss can be obtained by referring to the following formula: (7) Where B is the batch size, i is the index of the initial training sample, and j is the index of the enhanced training sample. The i-th enhanced training sample is obtained after the i-th initial training sample is enhanced.

[0041] It can be understood that according to the average value of the difference loss of the current batch, the model parameters of the student model are adjusted, including: Obtain the classification loss value and relative entropy of the student model, where the relative entropy is used to characterize the degree of difference between the output data of the student model and the teacher model in the output unit; The model parameters of the student model are adjusted according to the classification loss value, relative entropy and the average value of the difference loss.

[0042] In some embodiments, the performance of the student model also involves the integration of class relationship knowledge and correct data labels obtained from the teacher model. The performance of the model can be further improved by classification loss value and relative entropy. The parameters of the student model can be trained with reference to the following loss function: (8) in , , are hyperparameters (i.e., model parameters to be adjusted), is the classification loss function of the student model, based on The classification loss value can be obtained. It is a function corresponding to the logical KL divergence (also known as relative entropy) between the teacher and student models. is the average difference loss.

[0043] It can be understood that the similarity data and distance data of the student model are obtained through the following steps: Input each initial training sample and each enhanced training sample into the student model, and obtain a third feature output vector corresponding to the initial training sample and a fourth feature output vector corresponding to the enhanced training sample output by the output layer of the student model respectively; Normalizing the third characteristic output vector and the fourth characteristic output vector respectively to obtain a third unit vector corresponding to the third characteristic output vector and a fourth unit vector corresponding to the fourth characteristic output vector; Obtaining a second similarity between each initial training sample and the corresponding enhanced training sample according to the third unit vector and the transposed vector of the corresponding fourth unit vector, and obtaining similarity data of the student model according to each second similarity; According to the Euclidean distance between the third unit vector and the corresponding fourth unit vector, the second sample distance between each initial training sample and the corresponding enhanced training sample is obtained, and the distance data of the student model is obtained according to each second sample distance.

[0044] It is understandable that the student model corresponds to the logical vector of the initial training sample and enhanced training samples Also using the same function g(x) for normalization, the third unit vector , the fourth unit vector , then the similarity data of the student model can refer to the following formula (9).

[0045] (9) in, is the transposed matrix (i.e., transposed vector) of the normalized matrix of the augmented training samples in the student model; i and j are both indexes, i represents the index of the initial training sample in the current batch, and j represents the index of the augmented training sample in the current batch. When the similarity data is represented by a matrix, then is a similarity matrix of size B×B. Each element represents the cosine similarity between the i-th initial training sample and the j-th enhanced training sample. These matrices As a key indicator for evaluating how the student model perceives the difference between the initial training samples and the enhanced training samples, the matrix reflects how each model maintains or changes its feature detection ability in response to data enhancement, as follows: 1) A high value of indicates that the student model perceives the augmented training samples as similar to the initial training samples, indicating that feature extraction remains stable despite the introduced changes; 2) Lower values ​​indicate that the student model recognizes and responds to the changes caused by the enhancement, highlighting dynamic adaptation to the changing data characteristics.

[0046] Similarly, for the distance data of the student model, referring to formula (10), the alignment of features between the original sample and the enhanced training sample can be determined to obtain the distance data of the student model. Formula (10) is as follows: ,(10) Where i and j are the indices of the initial training samples and enhanced training samples in the same batch, respectively. When expressed as a matrix, the size of the distance matrix is ​​B×B, where B is the batch size. Each element Quantifies the Euclidean distance between the i-th initial training sample and the j-th enhanced training sample. Values ​​indicate significant differences due to enhancement and vice versa.

[0047] Therefore, the method of the above embodiment of the present application has the following beneficial effects: (1) Efficiently mine rich knowledge from limited samples: By calculating similarity data and distance data, the subtle differences between the initial training samples and the enhanced training samples are accurately captured, effectively mining more knowledge from limited sample data. This enables the student model to improve learning efficiency and better understand and utilize implicit information in the data without adding additional data burden.

[0048] (2) Reduce data acquisition costs: Through the dynamic weighting mechanism, the student model can focus resources on learning samples with high differences, effectively reducing the need for large amounts of training data. This method not only reduces the cost and labor intensity of data collection, but also provides advantages for processing application scenarios, especially those with scarce data. It is particularly suitable for efficient learning of large models in limited data environments.

[0049] (3) Enhance the model’s generalization ability and adaptability to new data: By combining detailed analysis of differential knowledge, the teacher model’s in-depth understanding of the differences between original and enhanced training samples is passed to the student model during the knowledge distillation process. This strategy not only enhances the student model’s generalization ability, but also optimizes its adaptability and responsiveness to unseen samples, especially enhanced training samples.

[0050] In a second aspect, an embodiment of the present application provides an image recognition method, the image recognition method comprising: Obtain an image recognition model, which is trained using the above method; The recognition image data is input into the image recognition model and the image recognition data is output.

[0051] Refer to the following Figure 3 Describe the model training method of the embodiment of the present application, assuming that there are N training sample data sets, each data set , then the size of each training batch is N. Taking the similarity data and distance data as an example, the training steps for each batch are as follows: Step S1: Calculation of similarity and distance matrix: set up represents the logical output of the teacher model on the i-th initial training sample. represents the logical output of the teacher model on the i-th enhanced training sample. Then, through the normalization function g(x) of formula (1), and Normalized to have unit norm, we get the first unit vector and the second unit vector This step ensures that the comparison between vectors focuses on the angle difference rather than the size, which is crucial for accurately evaluating the difference between the initial training sample and the enhanced training sample. Similarly, the logical vector of the student model corresponding to the initial training sample is and enhanced training samples The same function g(x) is also used for normalization to obtain the third unit vector and the fourth unit vector .

[0052] To quantify the alignment of features between the original and augmented training samples, similarity matrices are computed for the teacher and student models. These matrices are obtained by computing the dot product of the normalized logistic matrices of the augmented training samples transposed to properly align the dimensions. Indicates that the column vector is converted into a row vector or vice versa in order to correctly calculate the dot product. Then, referring to formula (2) and formula (9), the similarity matrix of the teacher model can be obtained: ,Similarity matrix of student model ; where i and j index the initial training samples and enhanced training samples in the batch respectively. Similarity matrix and The size of is N×N. Each element and represents the cosine similarity between the i-th initial training sample and the j-th enhanced training sample. These matrices and As a key indicator for evaluating how the teacher model and the student model perceive the difference between the initial training samples and the enhanced training samples. These matrices reflect how each model maintains or changes its feature detection ability in response to data enhancement, as follows: 1) and A high value of indicates that the model perceives the augmented training samples as similar to the initial training samples, indicating that feature extraction remains stable despite the introduced changes; 2) Lower values ​​indicate that the model recognizes and responds to the changes caused by the enhancement, highlighting dynamic adaptation to the changing characteristics of the data.

[0053] The distance matrix can quantify the difference between the model's response to the initial training samples and the enhanced training samples. These matrices are measured by calculating the Euclidean distance between the corresponding feature vectors of the original dataset and the enhanced dataset, thereby evaluating how the enhancement affects the model's feature perception. Referring to formulas (4) and (10), the distance matrix of the teacher model can be obtained: And the distance matrix of the student model .

[0054] Where i and j are the indices of the initial training sample and the enhanced training sample in the same batch, respectively. The size of the distance matrix is ​​B×B, and each dimension corresponds to the batch size B. Each element and Quantifies the Euclidean distance between the i-th initial training sample and the j-th enhanced training sample. and Values ​​indicate significant differences due to enhancement and vice versa.

[0055] Step S2: Sample-level importance weighting mechanism: After calculating the similarity and distance matrices between the initial training samples and the augmented training samples, a sample-level importance weighting mechanism is introduced to enhance the student model's focus on learning from diverse knowledge sources. This mechanism leverages the experience of the teacher model to evaluate the relative importance of each differential knowledge in the batch.

[0056] First, the similarity matrix and the distance matrix Convert to a one-dimensional vector and . Then, according to the reshaped vector and Calculate the weight of each matrix. Then we can get the similarity matrix ( ) .

[0057] where τ is a temperature parameter that controls the weight distribution. A higher similarity indicates that the enhanced training samples lack diversity and challenge relative to the initial training samples, resulting in a lower weight. This approach ensures that the student model allocates less attention to sample pairs that the teacher model considers to be too similar. As a result, the student model is able to efficiently use its capabilities to focus on more challenging differences.

[0058] Similarly, we can get the distance matrix Weight .

[0059] In the distance matrix, higher values ​​indicate larger differences between the initial training samples and the enhanced training samples, showing more challenging instances for the student model to learn. Therefore, these higher values ​​are given greater weights. This ensures that the student model pays more attention to sample pairs with significant differences, allowing it to learn more effectively from complex and diverse data.

[0060] Step S3, calculate the sample difference loss: The weight vector and Transform to match the size of the difference matrix. and Reshape into a matrix and After calculating the weighted similarity and distance matrices, the difference loss function Integrating these differentiability metrics into the knowledge distillation process allows us to align the student model’s understanding of sample differences with the insights of the teacher model, leveraging the subtle differences between the identified initial and augmented training samples.

[0061] in Represents element-wise multiplication, ensuring Each element of accurately represents the weighted mean square difference between the student model and the teacher model in terms of similarity and distance perception. Each matrix, and and its student model counterpart, play a key role in measuring how the model perceives the impact of data augmentation on sample features. The final difference loss for the entire batch is then calculated: .

[0062] Step S4: Adjustment of student model parameters: Through the loss function To train the parameters of the student model; , , is a hyperparameter, is the classification loss function of the student model, is the KL divergence between the teacher and student models in terms of logistic terms.

[0063] Repeat the above steps S1 to S4 until the preset convergence condition is met, thereby obtaining the recognition model, where the convergence condition can be the number of iterations or The value is less than the preset threshold.

[0064] In some embodiments, reference Figure 4 As shown, the present application also provides an image recognition device, the image recognition device comprising: The model training module 100 includes a teacher model training submodule 110 and a student model training submodule 120. The teacher model training submodule 110 is used to obtain the similarity data and distance data of the teacher model according to the first feature output vector corresponding to each initial training sample output by the output layer of the preset teacher model and the second feature output vector corresponding to each enhanced training sample; wherein the enhanced training sample is obtained after the initial training sample is enhanced; according to the similarity data and distance data of the teacher model, the similarity weight and the distance weight are determined; the student model training submodule 120 is used to obtain the sample difference data based on the similarity weight, the distance weight, the similarity data and the distance data of the student model; wherein the similarity data and the distance data of the student model are obtained according to the third feature output vector corresponding to each initial training sample output by the output layer of the student model and the fourth feature output vector corresponding to each enhanced training sample; the student model is trained according to the sample difference data, the initial training sample and the enhanced training sample to obtain a trained recognition model; The recognition module 200 is used to recognize images according to a trained recognition model.

[0065] The above is a schematic scheme of an image recognition device of this embodiment. It should be noted that the technical scheme of the image recognition device and the technical scheme of the model training method and image recognition method based on the knowledge distillation model mentioned above belong to the same concept. For the details not described in detail in the technical scheme of the image recognition device, please refer to the description of the technical scheme of the model training method and image recognition method based on the knowledge distillation model mentioned above.

[0066] like Figure 5 As shown, Figure 5 The structure block diagram of an electronic device 300 provided according to an embodiment of the present application is shown. The components of the electronic device 300 include but are not limited to a memory 310 and a receiver 320. The receiver 320 is connected to the memory 310 via a bus 330.

[0067] The electronic device 300 further includes an access module 340, which enables the electronic device 300 to communicate with one or more terminal devices 400. In one embodiment of the present application, the above components of the electronic device 300 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The structural block diagram of the electronic device shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0068] The terminal device 400 may be any type of stationary or mobile electronic device, including a mobile computer or mobile electronic device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable electronic device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary electronic device such as a desktop computer or a PC. The terminal device 400 may also be a mobile or stationary server.

[0069] Among them, the receiver 320 is used for computer executable instructions of a model training method based on a knowledge distillation model.

[0070] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above object receiving method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above object receiving method.

[0071] An embodiment of the present application also provides a computer-readable storage medium, which is a computer-readable storage medium that stores a computer program. When the computer program is executed by a receiver, it implements the above-mentioned model training method based on the knowledge distillation model.

[0072] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the receiver, and these remote memories may be connected to the receiver via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0073] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a receiver, such as a central receiver, a digital signal receiver, or a microreceiver, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0074] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions under the shared conditions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A model training method based on a knowledge distillation model, characterized in that: The method comprises: According to the first feature output vectors corresponding to each initial training sample and the second feature output vectors corresponding to each enhanced training sample outputted by the output layer of the preset teacher model, the similarity data and the distance data of the teacher model are obtained; wherein the enhanced training samples are obtained after the initial training samples are enhanced; Determine a similarity weight and a distance weight according to the similarity data and the distance data of the teacher model; Based on the similarity weight, the distance weight, the similarity data and the distance data of the student model, sample difference data is obtained; wherein the similarity data and the distance data of the student model are obtained according to the third feature output vector corresponding to each of the initial training samples and the fourth feature output vector corresponding to each of the enhanced training samples output by the output layer of the student model; The student model is trained according to the sample difference data, the initial training samples and the enhanced training samples to obtain a trained recognition model.

2. The model training method based on the knowledge distillation model according to claim 1, characterized in that: The method of obtaining similarity data and distance data of the teacher model according to the first feature output vector corresponding to the initial training sample and the second feature output vector corresponding to the enhanced training sample outputted by the output layer of the preset teacher model comprises: Input each of the initial training samples and each of the enhanced training samples into the teacher model, and obtain a first feature output vector corresponding to the initial training sample and a second feature output vector corresponding to the enhanced training sample output by the output layer of the teacher model respectively; Normalizing the first feature output vector and the second feature output vector respectively to obtain a first unit vector corresponding to the first feature output vector and a second unit vector corresponding to the second feature output vector; Obtaining a first similarity between each of the initial training samples and the corresponding enhanced training sample according to the transposed vector of the first unit vector and the corresponding second unit vector, and obtaining similarity data of the teacher model according to each of the first similarities; According to the Euclidean distance between the first unit vector and the corresponding second unit vector, the first sample distance between each of the initial training samples and the corresponding enhanced training sample is obtained, and according to each of the first sample distances, the distance data of the teacher model is obtained.

3. The model training method based on the knowledge distillation model according to claim 1, characterized in that: Determining the similarity weight and the distance weight according to the similarity data and the distance data of the teacher model includes: Performing one-dimensional vector conversion processing on the similarity data and the distance data respectively to obtain a similarity vector and a distance vector; Obtaining the similarity weight according to the similarity vector, the sample batch size of the teacher model and a preset temperature distribution parameter; The distance weight is obtained according to the distance vector, the sample batch size of the teacher model and the temperature distribution parameter.

4. The model training method based on the knowledge distillation model according to claim 1, characterized in that: The obtaining of sample difference data based on the similarity weight, the distance weight, the similarity data of the student model and the distance data comprises: Constructing a similarity square difference matrix according to the similarity data of the teacher model and the student model of the same batch; Constructing a distance square difference matrix according to the distance data of the teacher model and the student model of the same batch; Perform matrix conversion on the similarity weight and the distance weight respectively to obtain a similarity matrix and a distance matrix; The similarity matrix and the distance matrix are respectively subjected to matrix dot multiplication with the similarity square difference matrix and the distance square difference matrix to obtain the sample difference data.

5. The model training method based on the knowledge distillation model according to claim 4 is characterized in that: The student model is trained according to the sample difference data, the initial training sample and the enhanced training sample to obtain a trained recognition model, including: According to the sample difference data, a difference loss value obtained by training the student model based on each of the initial training samples and each of the enhanced training samples in the current batch is obtained; and an average difference loss value of the current batch is calculated according to each of the difference loss values; According to the average value of the difference loss of the current batch, the model parameters of the student model are adjusted and the step of obtaining the similarity data and distance data of the teacher model is jumped to perform the next batch of model training until the trained student model meets the preset convergence condition; The student model that meets the preset convergence condition is used as the trained recognition model.

6. The model training method based on the knowledge distillation model according to claim 5, characterized in that: The adjusting the model parameters of the student model according to the average difference loss of the current batch includes: Obtaining a classification loss value and a relative entropy of the student model, wherein the relative entropy is used to characterize the degree of difference between the output data of the student model and the teacher model in an output unit; The model parameters of the student model are adjusted according to the classification loss value, the relative entropy and the difference loss average value.

7. The model training method based on the knowledge distillation model according to claim 2, characterized in that: The similarity data and distance data of the student model are obtained by the following steps: Input each of the initial training samples and each of the enhanced training samples into the student model, and obtain a third feature output vector corresponding to the initial training sample and a fourth feature output vector corresponding to the enhanced training sample output by the output layer of the student model respectively; Normalizing the third characteristic output vector and the fourth characteristic output vector respectively to obtain a third unit vector corresponding to the third characteristic output vector and a fourth unit vector corresponding to the fourth characteristic output vector; Obtaining a second similarity between each of the initial training samples and the corresponding enhanced training sample according to the third unit vector and the corresponding transposed vector of the fourth unit vector, and obtaining similarity data of the student model according to each of the second similarities; According to the Euclidean distance between the third unit vector and the corresponding fourth unit vector, the second sample distance between each of the initial training samples and the corresponding enhanced training sample is obtained, and according to each of the second sample distances, the distance data of the student model is obtained.

8. An image recognition method, characterized in that: The image recognition method comprises: Obtaining an image recognition model, wherein the image recognition model is trained by the method according to claim 1; The recognition image data is input into the image recognition model, and the image recognition data is output.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the model training method based on the knowledge distillation model described in any one of claims 1 to 7 or the steps of the image recognition method described in claim 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the steps of the model training method based on the knowledge distillation model as described in any one of claims 1 to 7 or the steps of the image recognition method as described in claim 8.

Images