Model training method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a model training method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining a multimedia sample set; for each multimedia sample, performing prediction processing on the multimedia sample through a teacher model and a student model to obtain prediction information of the multimedia sample, the prediction information of the multimedia sample including teacher prediction information of the teacher model on the multimedia sample and student prediction information of the student model on the multimedia sample; determining prediction difference information of the multimedia sample set according to the difference between the teacher prediction information of any multimedia sample and the student prediction information of any multimedia sample; determining knowledge distillation loss of the student model to the multimedia sample set according to the prediction difference information; and training a student model based on the knowledge distillation loss to obtain a trained student model. According to the invention, the accuracy and prediction capability of the student model obtained through training can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a model training method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Knowledge distillation is a model compression technique that trains a smaller model (student model) to learn the knowledge of a larger model (teacher model). The goal of this training is to maximize the performance of the smaller model while maintaining its computational and storage efficiency. During knowledge distillation, the student model learns the knowledge of the teacher model by mimicking its output probability distribution. This method effectively transfers knowledge from complex models to simpler ones, meeting the needs of deploying models on resource-constrained devices.

[0003] However, existing knowledge distillation processes typically measure the difference between the output probability distributions of the teacher and student models on the same sample to train the student model. This approach is limited to multi-sample distillation and struggles to comprehensively capture the overall differences between the teacher and student models, resulting in poor accuracy and predictive ability of the trained student model. Summary of the Invention

[0004] This application provides a model training method, apparatus, electronic device, storage medium, and program product that can improve the accuracy and predictive ability of the trained student model.

[0005] This application provides a model training method, comprising: acquiring a multimedia sample set, the multimedia sample set including multiple multimedia samples; for each multimedia sample, performing prediction processing on the multimedia sample using a teacher model and a student model respectively, to obtain prediction information for the multimedia sample, the prediction information including teacher prediction information of the multimedia sample by the teacher model and student prediction information of the multimedia sample by the student model; determining prediction difference information of the multimedia sample set based on the difference between the teacher prediction information and the student prediction information of any multimedia sample; determining the knowledge distillation loss of the multimedia sample set by the student model based on the prediction difference information; and training the student model based on the knowledge distillation loss to obtain a trained student model, the trained student model being used for prediction processing of multimedia information.

[0006] This application embodiment also provides a model training apparatus, comprising: an acquisition unit for acquiring a multimedia sample set, the multimedia sample set including multiple multimedia samples; a prediction unit for predicting each multimedia sample using a teacher model and a student model respectively, to obtain prediction information for the multimedia sample, the prediction information including teacher prediction information of the multimedia sample by the teacher model and student prediction information of the multimedia sample by the student model; a difference determination unit for determining prediction difference information of the multimedia sample set based on the difference between the teacher prediction information and the student prediction information of any multimedia sample; a loss determination unit for determining the knowledge distillation loss of the multimedia sample set by the student model based on the prediction difference information; and a training unit for training the student model based on the knowledge distillation loss to obtain a trained student model, the trained student model being used for predicting multimedia information.

[0007] In some implementations, the knowledge distillation loss includes a transfer loss, and the loss determination unit is specifically configured to: calculate a kernel matrix from the prediction difference information using a kernel function, wherein the elements of the kernel matrix represent the similarity between the teacher prediction information and the student prediction information of any multimedia sample; determine a transfer matrix based on the kernel matrix, wherein the elements of the transfer matrix represent the transfer amount between the teacher prediction information and the student prediction information of any multimedia sample; and obtain the transfer loss between the teacher prediction information and the student prediction information based on the transfer matrix and the prediction difference information.

[0008] In some implementations, determining the transition matrix based on the kernel matrix includes: iteratively updating the kernel matrix based on the sample information of the multimedia samples to obtain an updated kernel matrix, wherein the sample information includes teacher sample information and student sample information, the teacher sample information includes at least one of teacher prediction information of the multimedia samples and teacher sample weights of the multimedia samples corresponding to the teacher model, and the student sample information includes at least one of student prediction information of the multimedia samples and student sample weights of the multimedia samples corresponding to the student model; and using the updated kernel matrix as the transition matrix.

[0009] In some implementations, iteratively updating the kernel matrix based on the sample information of the multimedia samples to obtain an updated kernel matrix includes: determining a first scaling weight based on the kernel matrix and the student sample information; obtaining an intermediate kernel matrix from the first scaling weight and the kernel matrix; determining a second scaling weight based on the intermediate kernel matrix and the teacher sample information; obtaining a target kernel matrix from the second scaling weight and the intermediate kernel matrix; and using the target kernel matrix as the kernel matrix to iteratively update the kernel matrix to obtain the updated kernel matrix.

[0010] In some implementations, determining the transition matrix based on the kernel matrix includes: obtaining row scaling vectors corresponding to rows in the kernel matrix and column scaling vectors corresponding to columns in the kernel matrix; diagonally scaling the kernel matrix using the row scaling vectors and column scaling vectors to obtain an initial transition matrix; iteratively updating the row scaling vectors and column scaling vectors to update the initial transition matrix and obtain the transition matrix.

[0011] In some implementations, the knowledge distillation loss further includes at least one of prediction loss and knowledge loss, and the loss determination unit may also be used to: determine the prediction loss based on the difference between the sample labels of the multimedia samples and the student prediction information; and / or, determine the knowledge loss based on the difference between the teacher prediction information and the student prediction information.

[0012] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the model training methods provided in this application.

[0013] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the model training methods provided in this application.

[0014] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in any of the model training methods provided in this application.

[0015] This application embodiment can obtain a multimedia sample set, which includes multiple multimedia samples; for each multimedia sample, a teacher model and a student model are used to perform prediction processing on the multimedia sample to obtain prediction information for the multimedia sample. The prediction information for the multimedia sample includes teacher prediction information of the multimedia sample by the teacher model and student prediction information of the multimedia sample by the student model; based on the difference between the teacher prediction information and the student prediction information of any multimedia sample, prediction difference information of the multimedia sample set is determined; based on the prediction difference information, the knowledge distillation loss of the student model on the multimedia sample set is determined; based on the knowledge distillation loss, the student model is trained to obtain a trained student model, which is used to perform prediction processing on multimedia information.

[0016] In this application, the overall recognition difference across the entire multimedia sample set is determined based on the difference between the teacher's prediction information and the student's prediction information of any multimedia sample within the set. This approach goes beyond focusing solely on the difference between teacher and student prediction information within the same multimedia sample, meaning it is not limited to the difference at the multimedia sample level. It comprehensively measures the overall difference between teacher and student prediction information across the entire multimedia sample set, thus fully capturing the overall difference between the teacher and student models. A knowledge distillation loss is then constructed based on this difference information. This approach goes beyond multimedia sample-level distillation; by comprehensively measuring the overall difference between teacher and student prediction information across the entire multimedia sample set, it fully captures the overall difference between the teacher and student models, thereby improving the accuracy and predictive ability of the trained student model. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1a This is a schematic diagram of a scenario for the model training method provided in an embodiment of this application;

[0019] Figure 1b This is a flowchart illustrating the model training method provided in the embodiments of this application;

[0020] Figure 2a This is a flowchart illustrating a model training method provided in another embodiment of this application;

[0021] Figure 2b This is a flowchart illustrating yet another model training method provided in another embodiment of this application;

[0022] Figure 2c The experimental results of Verification Experiment 1 are shown in Table 1 of the embodiments of this application;

[0023] Figure 2d Table 2 in this application provides the experimental results for verifying Experiment 2;

[0024] Figure 2e Table 3 of this application provides the experimental results for verifying Experiment 3;

[0025] Figure 2f Table 4 in this application provides the experimental results of verification experiment four;

[0026] Figure 2g The experimental results of Experiment 4 are shown in Table 5 of the embodiments of this application.

[0027] Figure 2h The experimental results of verification experiment four are shown in Table 6 of the embodiments of this application;

[0028] Figure 2i The experimental results of Experiment 5 are shown in Table 7 of the embodiments of this application.

[0029] Figure 2j The experimental results of Experiment Six are shown in Table 8 of the embodiments of this application.

[0030] Figure 3 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0031] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The following is a description of the technical terms used in this application:

[0034] Knowledge distillation is a model compression technique that trains a smaller model (student model) to learn the knowledge of a larger model (teacher model). The goal is to maximize performance while maintaining the computational and storage efficiency of the smaller model. During knowledge distillation, the student model learns the teacher model's knowledge by mimicking its output probability distribution. This method effectively transfers knowledge from complex models to simpler ones, meeting the needs of deploying models on resource-constrained devices.

[0035] Computer Vision Model (CVM) is an artificial intelligence model that aims to enable computers or machines to acquire, process, analyze, and understand image or video data from the real world through cameras or other visual sensors. This model can simulate the visual perception capabilities of the human eye, allowing machines to identify objects, features, scenes, and other information in images or videos. It can even perform more complex visual tasks such as face recognition, image segmentation, and object detection.

[0036] Large Language Models (MLMs) are models based on deep learning and Natural Language Processing (NLP) techniques, aiming to understand and generate human language. These models typically consist of billions or even tens of billions of parameters and can learn language patterns, semantic information, and knowledge from massive amounts of text data. MLMs have achieved significant results in various NLP tasks, such as machine translation, text summarization, question answering systems, and sentiment analysis.

[0037] Autoregressive Language Models (ARMs) are probabilistic natural language processing models that aim to predict the next word based on a known sequence of words. In an ALM, the occurrence of each word depends on the preceding word sequence, making it a conditional probability model. The training process typically uses Maximum Likelihood Estimation (MLE) or other optimization methods to learn the model parameters, thereby maximizing the overall probability on the training data. In practical applications, ALMs can be used for tasks such as text generation, machine translation, and speech recognition.

[0038] Masked Language Model (MLM) is a language model trained on a pre-training task (mask modeling), where a portion of the words in the input sequence are replaced with special symbols (such as "[MASK]"). The model's task is to predict these replaced words. This training method is widely used in Transformer models such as BERT, enabling the model to understand the contextual meaning of words in a sentence.

[0039] Discriminative language models (CLMs) are a type of natural language processing method used to distinguish between correct and incorrect word sequences. Unlike generative language models (GLMs), discriminative language models do not focus on the probability of generating correct word sequences; instead, they focus on how to directly predict output labels based on given input features. The goal of discriminative language models is to maximize the log-likelihood function on the training data to better predict new, unseen data. The advantage of discriminative language models is that they can directly learn the relationship between input features and output labels without considering the generation process. This makes them perform better on certain tasks, such as text classification, named entity recognition, and sentiment analysis.

[0040] Generative language models (GLMs) are machine learning algorithms that aim to learn a probability distribution that can generate a series of words or sentences. These models typically use chained rules to generate text, predicting the probability of the next word given preceding words. GLMs can be used in various natural language processing tasks, such as machine translation, speech recognition, and text generation.

[0041] Sinkhorn distance, or Sinkhorn Distance, is a method for measuring the distance between two probability distributions. It is based on optimal transport theory and entropy regularization. Given two probability distributions P and Q, Sinkhorn distance calculates the distance between them by solving a regularized optimal transport problem.

[0042] The optimal transport distance, also known as the Wasserstein distance or Earth travel distance, is a metric defined between probability distributions.

[0043] This application provides a model training method, apparatus, electronic device, storage medium, and program product.

[0044] Specifically, the model training device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0045] In some embodiments, the model training device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the model training method of this application.

[0046] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0047] For example, refer to Figure 1a The model training method is integrated into a server that can acquire a multimedia sample set, which includes multiple multimedia samples. For each multimedia sample, a teacher model and a student model are used to perform prediction processing to obtain prediction information for the multimedia sample. The prediction information includes the teacher model's prediction information for the multimedia sample and the student model's prediction information for the multimedia sample. Based on the difference between the teacher's prediction information and the student's prediction information for any multimedia sample, the prediction difference information of the multimedia sample set is determined. Based on the prediction difference information, the knowledge distillation loss of the student model for the multimedia sample set is determined. Based on the knowledge distillation loss, the student model is trained to obtain the trained student model, which is used to perform prediction processing on multimedia information.

[0048] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0049] The following sections provide detailed descriptions. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments. It is understood that in the specific embodiments of this application, user-related data such as multimedia samples and multimedia information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0050] In this embodiment, a model training method is provided, such as Figure 1b As shown, the specific process of this model training method can be as follows:

[0051] 110. Obtain a multimedia sample set, which includes multiple multimedia samples.

[0052] Multimedia samples are multimedia information used to train student models. Multimedia samples can include one or more combinations of images (i.e., sample images) and text (sample text). For example, sample images can be used for computer vision tasks such as image classification, object detection, or image generation. Or, sample text can be used for natural language processing tasks such as text classification, named entity recognition, sentiment analysis, or question answering.

[0053] For example, a multimedia sample set can be a collection containing multiple different sample images or sample texts. In practical applications, multimedia samples in image or text form can be collected manually, or sample images or sample texts from open datasets can be used as multimedia samples.

[0054] 120. For each multimedia sample, the teacher model and the student model are used to perform prediction processing on the multimedia sample to obtain the prediction information of the multimedia sample. The prediction information of the multimedia sample includes the teacher prediction information of the multimedia sample by the teacher model and the student prediction information of the multimedia sample by the student model.

[0055] In knowledge distillation, the teacher model and the student model are two types of models. The teacher model is typically a larger, more complex model trained on a large amount of data. Its purpose is usually to provide an accurate reference standard or to guide the training of the student model. The student model is typically a smaller, simpler model. It is usually trained through guidance from the teacher model or by knowledge distillation, thereby learning the knowledge and predictive capabilities of the teacher model.

[0056] Predictive processing refers to the process of predicting the outcome of a sample. Predictive processing can be a process of classifying the sample (i.e., classification prediction) or a process of regressing the sample (i.e., regression prediction, etc.). Predictive processing for multimedia samples can be either regression processing or non-regression processing. Predictive information refers to the specific result of the predictive processing task. This predictive information can be the probability distribution of a sample belonging to each category in classification prediction, or the most likely category label, or the continuous numerical output predicted by the model in regression prediction, etc.

[0057] In this application embodiment, the teacher model and student model can be models with various structures, such as a purely discriminative encoder, an encoder-decoder, a purely generative decoder, etc. This application does not impose any limitations and the choice can be made according to actual needs or application scenarios. In practical applications, the teacher model and student model can differ for different multimedia samples. For example, for multimedia sample images, the teacher model and student model can be computer vision models, such as image classification models, image detection models, etc. Specifically, they can be residual network models (ResNet), convolutional neural networks such as the Visual Geometric Group Network (VGG) model, or ShuffleNet, etc. For multimedia sample text, the teacher model and student model can be language models, such as autoregressive language models, masked modeling language models, discriminative language models (such as large language models), etc. Specifically, they can be generative pre-trained transformers (GTP), bidirectional transformer models (BERT), Roberta models (RoBERTa), etc.

[0058] For example, in some implementations, the prediction information includes the probability distribution of multimedia samples belonging to each category, so as to provide richer and more accurate training signals using the predicted probability distribution, thereby improving the performance and generalization ability of the trained model. This allows the model training method of the embodiments of this application to be used not only for training a specific model, but also for training various types of models, such as computer vision models or language models. Specifically, the prediction information includes the predicted probability distribution of multimedia samples. The prediction information is obtained by performing prediction processing on the multimedia samples using both the teacher model and the student model, including:

[0059] Multimedia samples are predicted using both the teacher model and the student model, resulting in prediction vectors for both models.

[0060] The prediction vectors of the teacher model and the student model are activated separately to obtain the prediction probability distributions of the teacher model and the student model for multimedia samples.

[0061] In this context, the prediction vector is the output vector generated by the model after processing the multimedia samples; this vector can be called logits. For example, in classification prediction, the prediction vector can be used to represent the model's predicted probability for each class. The prediction probability distribution refers to the prediction distribution obtained by activating the prediction vector. For example, in classification prediction, the prediction probability distribution represents the probability distribution of the model's prediction result for each possible class. This activation processing can be implemented using activation functions such as softmax. The softmax function can take the model's output vector (usually logits) as input and output the probability of each class, so that the predicted probability of each class is explicitly represented.

[0062] For example, given any multimedia sample x in the multimedia sample set i Its real label is Teacher model f T And student model f S The prediction output logits for this multimedia sample are f. t (x i ) and f S (x i The logits can be activated by the softmax activation function σ. τ The activation process yields the following predicted probability distribution t for the teacher model. i The predicted probability distribution of the student model i :

[0063] t i =σ τ (f T (x i )), s i =σ τ (f S (x i ));

[0064] in, τ is the temperature parameter, and d is the dimension of the output logits. In practical applications, the model's prediction process may include, but is not limited to, extracting features from multimedia samples to obtain higher-level feature representations, performing necessary feature transformations on the extracted features (such as preprocessing, normalization, or converting text data into embedding vectors) to adapt to the model's input requirements, inputting the transformed features into the model for prediction, and producing an output vector.

[0065] In some implementations, when the teacher and student models are regression models (i.e., the prediction process is regression), the model outputs a prediction vector for the multimedia samples not as a probability distribution, but as a prediction vector composed of scalars (such as a continuous numerical value). This scalar can represent a specific attribute or the degree of that attribute of the sample. Therefore, when the teacher and student models are regression models, the prediction information can include prediction labels for the multimedia samples, which can be continuous numerical values ​​(i.e., scalars) predicted by the model. For example, if the teacher and student models output a scalar (d=1), for b multimedia samples in the multimedia sample set, the teacher prediction information and student prediction information output by the teacher and student models can be represented as scalar teacher prediction labels, respectively. and student prediction labels

[0066] 130. Based on the difference between the teacher prediction information and the student prediction information of any multimedia sample, determine the prediction difference information of the multimedia sample set.

[0067] The prediction difference information refers to the difference between the prediction information of the teacher model and the prediction information of the student model for the multimedia sample set. In this embodiment, the prediction difference information may include the difference between the teacher's prediction information and the student's prediction information for any pair of multimedia samples in the multimedia sample set.

[0068] It is understood that in the embodiments of this application, the overall identification difference in the entire multimedia sample set is determined based on the difference between the teacher prediction information and the student prediction information of any multimedia sample in the multimedia sample set (i.e., batch-based prediction difference information). This is not limited to focusing only on the difference between the teacher prediction information and the student prediction information of the same multimedia sample, that is, it is not limited to the prediction information difference information at the multimedia sample level. It can more comprehensively measure the overall difference between the teacher prediction information and the student prediction information of the entire multimedia sample set, so as to fully capture the overall difference between the teacher model and the student model.

[0069] In some implementations, teacher and student prediction information for all multimedia samples in the multimedia sample set can be traversed, and the difference between teacher and student prediction information for all pairwise multimedia samples in the multimedia sample set can be used as the prediction difference information for the multimedia sample set. Alternatively, the difference between teacher and student prediction information for only a subset of multimedia samples in the multimedia sample set can be used as the prediction difference information for the multimedia sample set. It is understood that neither of these methods is limited to prediction difference information at the multimedia sample level; both can comprehensively measure the overall difference between teacher and student prediction information, thereby achieving the effect of comprehensively capturing the overall difference between the teacher and student models.

[0070] For example, in some implementations, the recognition difference degree of any multimedia sample pair in the multimedia sample set can be predicted, and the overall prediction difference information of the multimedia sample set can be determined by combining the recognition difference degrees of all multimedia sample pairs. This allows for a more comprehensive measurement of the overall difference between teacher prediction information and student prediction information across the entire multimedia sample set. Specifically, determining the prediction difference information of the multimedia sample set based on the difference between the teacher prediction information and the student prediction information of any multimedia sample includes:

[0071] The recognition difference of a multimedia sample pair is obtained based on the difference between the teacher's prediction information of one multimedia sample and the student's prediction information of the other multimedia sample in any multimedia sample pair in the multimedia sample set. A multimedia sample pair includes any two multimedia samples in the multimedia sample set.

[0072] Predictive difference information is obtained from the recognition difference of all multimedia sample pairs.

[0073] For example, given the teacher prediction information and student prediction information for all multimedia samples in a multimedia sample set, we can calculate the teacher prediction information (such as the teacher prediction information t for the i-th multimedia sample) for any pair of multimedia samples (i.e., any multimedia sample pair) in the set. i ) and student prediction information (such as the student prediction information s of the j-th multimedia sample) j The degree of difference between the teacher's and student's predictions (i.e., the degree of difference in predicted information) can be calculated, for example, by calculating the difference between the teacher's and student's predictions (t). i -s j ), where i can be equal to j, meaning the two multimedia samples in a multimedia sample pair can be the same. Similarly, the recognition difference between teacher and student prediction information for any pair of multimedia sample pairs in the multimedia sample set can be calculated. By pairing prediction information not limited to the sample level, "batch-based" prediction difference information can be obtained.

[0074] In some implementations, it can be achieved by using l p The norm is used to measure the recognition difference between multimedia sample pairs in a multimedia sample set (i.e., l). p Norm differences). For example, this can be achieved through ||t i -s j || p Calculate the teacher prediction information t for the i-th multimedia sample. i Student prediction information s with the j-th multimedia sample j The pairwise differences, where different p-values ​​correspond to different norms.

[0075] In some implementations, the teacher and student prediction information of multimedia samples in a multimedia sample set can be flattened into a one-dimensional vector, and the difference between the flattened teacher and student prediction information can be used as the recognition difference degree. For example, the prediction probability distribution of multimedia samples in a multimedia sample set is usually a tensor. This tensor can be flattened into a one-dimensional vector to increase the dimension of the prediction probability distribution, consider the distribution of a batch of multimedia samples in the multimedia sample set from a broader perspective, and optimize the model more comprehensively to improve generalization ability. For example, the d-dimensional prediction probability distribution of b samples in a multimedia sample set can be flattened to obtain a bd dimension, which is obviously significantly higher than d. Furthermore, when the multimedia samples are uniformly distributed, the kernel matrix... The teacher sample weights and student sample weights are as follows: The distribution of batch multimedia samples is considered from a broader perspective during the kernel matrix iteration process.

[0076] In some implementations, the predicted difference information can be stored in matrix form (i.e., a difference matrix) to store the recognition difference information of all multimedia sample pairs. For example, this can be achieved by using l p The norm is used to measure the recognition difference of each multimedia sample pair in the multimedia sample set, and the recognition difference of each multimedia sample pair is stored in a matrix to obtain the difference matrix of the prediction difference information of the multimedia sample set. Any element in this difference matrix (i.e., the recognition difference of any multimedia sample pair) can be represented as the following entry D. i,j :D i,j =||t i -s j || p For example, the differences between the flattened teacher prediction information and the flattened student prediction information of each multimedia sample pair can be stored in a matrix to obtain the difference matrix of the prediction difference information of the multimedia sample set. Any element in this difference matrix (i.e., the recognition difference of any multimedia sample pair) can be represented as the following entry D. im,jn =|ti(m) -s j(n) | where bd is the flattened dimension.

[0077] In some implementations, when the teacher and student models are regression models (i.e., the prediction process is a regression process), the difference in labels between multimedia sample pairs in the multimedia sample set can be used as the dissimilarity measure. For example, any element in the dissimilarity matrix (i.e., the recognition dissimilarity measure of any multimedia sample pair) can be represented as the following entry D. i,j =|t i -s j |

[0078] 140. Based on the prediction discrepancy information, determine the knowledge distillation loss of the student model for the multimedia sample set.

[0079] Knowledge distillation loss refers to the loss that measures the difference between the output distribution of the teacher model and the prediction distribution of the student model. The goal of knowledge distillation is to minimize the measured difference in order to transfer knowledge. During knowledge distillation, by minimizing the knowledge distillation loss, the student model can, to some extent, simulate the decision-making process and reasoning ability of the teacher model.

[0080] For example, in the embodiments of this application, knowledge distillation loss can be constructed based on prediction difference information. This is not limited to multimedia sample-level distillation, but rather comprehensively captures the overall difference between teacher prediction information and student prediction information in the entire multimedia sample set by comprehensively measuring the overall difference between teacher prediction information and student prediction information, thereby improving the accuracy and prediction ability of the trained student model.

[0081] In some implementations, kernel functions can be used to effectively capture complex similarity information between multimedia samples, helping the model better understand and generalize the relationships between different samples. The transition matrix determined by the kernel matrix can then effectively transfer the teacher model's predictions to the student model. The transition loss between teacher and student predictions is determined by combining the transition matrix and prediction discrepancy information. This transition loss guides the student model to better simulate the teacher model's behavior, thereby improving the student model's prediction accuracy. Specifically, the knowledge distillation loss includes the transition loss. Based on the prediction discrepancy information, the knowledge distillation loss of the student model for the multimedia sample set is determined, including:

[0082] The kernel matrix is ​​calculated by using the kernel function to obtain the predicted difference information. The elements of the kernel matrix represent the similarity between the teacher's predicted information and the student's predicted information of any multimedia sample.

[0083] Based on the kernel matrix, the transition matrix is ​​determined. The elements of the transition matrix represent the amount of transition between the teacher's predicted information and the student's predicted information for any multimedia sample.

[0084] Based on the transition matrix and prediction difference information, the transition loss between teacher prediction information and student prediction information is obtained.

[0085] A kernel function is a function that maps input data to a higher-dimensional feature space through nonlinear transformation. Kernel functions can include one or more of the following: linear kernel functions, polynomial kernel functions, Gaussian kernel functions, etc. The purpose of a kernel function is to create a kernel matrix (Gram matrix) by calculating the similarity between the teacher's predicted information and the student's predicted information for any multimedia sample. Each element of this kernel matrix represents the similarity between the teacher's and student's predicted information for any pair of multimedia samples, rather than the Euclidean distance in the original feature space.

[0086] The transition matrix, also known as the transfer matrix, is used to characterize the transition pattern between teacher prediction information (i.e., the probability distribution of the teacher model output) and student prediction information (i.e., the probability distribution of the student model output). For example, the transition matrix can be a square matrix whose elements P {ij} ) represents the transition amount (i.e., the transmission amount) from state i to state j, which is state i × P {ij} The system is transitioned to state j, and the transition matrix is ​​used to try to find an optimal way to minimize the overall transition cost or distance. Here, state i can represent the probability distribution of the teacher model's output for any multimedia sample i, and state j can represent the probability distribution of the student model's output for any multimedia sample j.

[0087] For example, a Gaussian kernel function can be applied to the dissimilarity matrix D to calculate the kernel matrix. The process of calculating the kernel matrix using the Gaussian kernel function is as follows:

[0088] In the above formula, λ is the weight for entropy regularization, and λ>0. This formula represents applying a Gaussian kernel function to the dataset (i.e., the difference matrix) D, and combining it with the weight λ for entropy regularization to obtain the kernel matrix. The resulting kernel matrix can then be used to determine the transition matrix. This transition matrix, along with the predicted difference information, is used to determine the transition cost (i.e., transportation cost) between teacher and student predicted information for any pairwise multimedia samples. This yields the transition loss between teacher and student predicted information corresponding to the multimedia sample set, which characterizes the overall transition loss of the multimedia sample set.

[0089] In this application embodiment, the transition matrix can be determined from the kernel matrix in various ways. For example, in some implementations, the kernel matrix can be diagonally scaled using the row scaling vector and column scaling vector. This diagonal scaling can adjust the weights of different rows and columns in the kernel matrix, thereby accurately representing the transition amount between samples, which helps to improve the model's expressive power and learning effect. Simultaneously, by iteratively updating the row scaling vector and column scaling vector, the transition matrix can be repeatedly optimized and adjusted. This iterative process can continuously improve the model's performance, enabling the model to better capture and learn complex relationships between samples, thereby improving prediction accuracy. Specifically, determining the transition matrix based on the kernel matrix includes:

[0090] Obtain the row scaling vector corresponding to the row in the kernel matrix and the column scaling vector corresponding to the column in the kernel matrix;

[0091] The kernel matrix is ​​diagonally scaled using row scaling vectors and column scaling vectors to obtain the initial transition matrix;

[0092] Iteratively update the row scaling vector and column scaling vector to update the initial transition matrix, thus obtaining the transition matrix.

[0093] Diagonal scaling refers to the process of multiplying each element of a matrix by a non-negative diagonal matrix. The vector formed by the diagonal elements of this non-negative diagonal matrix is ​​the scaling vector. Row scaling vectors and column scaling vectors are used to scale the matrix by row and column, respectively. In practical applications, scaling vectors can be used to diagonally scale a matrix to maintain its stochastic properties and gradually approach the optimal solution during iteration. This allows for effective adjustment of the transition matrix at each step through the iteration process, minimizing transition loss.

[0094] For example, diagonally scaling the kernel matrix K yields the initial transition matrix P. λ The process can be expressed as the following formula: P λ =diag(v t )Kdiag(u t ), where the row scaling vector v t The column scaling vector u is used for row-wise multiplication with each element in the transition matrix K. t Used for column-wise multiplication with each element in the transition matrix K. and All are non-negative. `diag` refers to the operation of placing the elements of a vector on the main diagonal of a diagonal matrix, thus `diag(v...` t ) and diag(u t ) can be represented by row-scaled vectors v t and column scaling vector u tLet be the diagonal matrix of the elements on the diagonal. Therefore, the above formula can be expressed as scaling the elements of the kernel matrix by multiplying the diagonal matrices corresponding to the row scaling vectors and column scaling vectors by each element of the kernel matrix, respectively.

[0095] In this way, the row scaling vector v can be obtained in the initial state before the iterative update (i.e., iteration t=0). 0 and column scaling vector u 0 And substitute P λ =diag(v t )Kdiag(u t The initial transition matrix P under the initial state is calculated. λ The row scaling vector v is updated iteratively. t and column scaling vector u t In each iteration update process, the row scaling vector and column scaling vector updated in the current iteration are substituted into P. λ =diag(v t )Kdiag(u t This is used to update the initial transition matrix and obtain the intermediate transition matrix P for the current iteration. λ This process continues until the iteration termination condition is met (such as reaching a preset number of iterations or the intermediate transition matrix P of the last two iterations). λ (Convergence), the intermediate transition matrix P obtained from the last iteration update. λ This is the transition matrix.

[0096] In some implementations, the row scaling vector and column scaling vector in the initial state before the iterative update (i.e., iteration t=0) can be preset vectors such as all-1 vectors.

[0097] In some implementations, the row scaling vector and column scaling vector can be iteratively updated based on teacher prediction information, student prediction information, and the kernel matrix to more accurately adjust the transition matrix. This better reflects the complex relationships in the teacher and student prediction information, allowing the final transition matrix to more accurately capture the transition amounts between samples. Specifically, iteratively updating the row scaling vector and column scaling vector to update the initial transition matrix yields the transition matrix, including:

[0098] Based on teacher prediction information and kernel matrix, the row scaling vector is iteratively updated to obtain the updated row scaling vector;

[0099] Based on student prediction information and kernel matrix, the column scaling vector is iteratively updated to obtain the updated column scaling vector.

[0100] The initial transition matrix is ​​updated based on the updated row scaling vector and the updated column scaling vector to obtain the transition matrix.

[0101] For example, during the iteration process, the row scaling vector and column scaling vector from the previous iteration can be updated based on teacher prediction information, student prediction information, and the transition matrix to obtain the row scaling vector and column scaling vector for the current iteration (i.e., the updated row scaling vector and the updated column scaling vector). Specifically, the updated row scaling vector v in the current iteration... t and the updated column scaling vector u t It can be obtained through the following formula:

[0102] Among them, t i and s i These are teacher prediction information and student prediction information, respectively. i and s i The row scaling vector and column scaling vector only need to be calculated once during the iteration process, i.e., t. i and s i This remains unchanged during the iteration. K is the initial transition matrix for the current iteration round. T Let v be the transpose of the initial transition matrix K. t-1 and u t-1 Let v be the row scaling vector and column scaling vector of the previous iteration (i.e., iteration t-1). Therefore, we can obtain the row-weighted vector by multiplying the transpose of the initial transition matrix of the current iteration by the row scaling vector of the previous iteration, and the column-weighted vector by multiplying the initial transition matrix of the current iteration by the column scaling vector of the previous iteration. The row-weighted vector and column-weighted vector represent the weighted results of the row scaling vector and column scaling vector of the previous iteration by the initial transition matrix in the current iteration, respectively. Then, we divide the teacher's prediction information and the student's prediction information by the row-weighted vector and column-weighted vector, respectively, to obtain the updated row scaling vector and updated column scaling vector for the current iteration. This process is repeated, and the updated row scaling vector and updated column scaling vector are then used as the row scaling vector v for the next iteration. t-1 and column scaling vector u t-1 and substitute The updated row scaling vector v for this round is calculated in the calculation. t and the updated column scaling vector u t This process continues iteratively until the iteration termination condition is met (such as reaching a preset number of iterations or the convergence of the intermediate transition matrix obtained from the latest two iterations).

[0103] In some implementations, the update process of the transition matrix can be defined as the update process of the kernel matrix to provide a more compact matrix-based solution, reduce parameter storage requirements, and improve iterative update efficiency. Specifically, the transition matrix is ​​determined based on the kernel matrix, including:

[0104] Based on the sample information of multimedia samples, the kernel matrix is ​​iteratively updated to obtain the updated kernel matrix;

[0105] Use the updated kernel matrix as the transition matrix.

[0106] Here, sample information refers to information related to multimedia samples and the teacher model or learning model. Sample information includes, but is not limited to, the output (i.e., prediction information) after the teacher model or student model has processed the multimedia samples for prediction, or the importance or weight information of the multimedia samples, etc. For example, sample information includes teacher sample information and student sample information. Teacher sample information includes at least one of the teacher prediction information of multimedia samples and the teacher sample weights of the teacher model corresponding to the multimedia samples. Student sample information includes at least one of the student prediction information of multimedia samples and the student sample weights of the student model corresponding to the multimedia samples.

[0107] For example, the transition matrix P λ It can be defined as P λ = K′, where K′ is the updated kernel matrix. It can be calculated using the Gaussian kernel function. The kernel matrix K is the initial state before the iterative update (i.e., iteration t=0). 0 , Therefore, the kernel matrix can be updated based on the sample information of the multimedia samples to obtain the updated kernel matrix K′, based on P. λ =K′, and use the updated kernel matrix as the transition matrix.

[0108] In practical applications, when the kernel matrix is ​​directly used as the transition matrix, it can be reconstructed into an equivalent vector form. The iteration process is performed only on the kernel matrix K, improving the efficiency of the iterative update. For example, in some implementations, based on the sample information of multimedia samples, the kernel matrix is ​​iteratively updated to obtain the updated kernel matrix, including:

[0109] The first scaling weight is determined based on the kernel matrix and student sample information.

[0110] The intermediate kernel matrix is ​​obtained from the first scaling weight and the kernel matrix;

[0111] The second scaling weight is determined based on the intermediate kernel matrix and teacher sample information;

[0112] The target kernel matrix is ​​obtained from the second scaling weight and the intermediate kernel matrix;

[0113] The target kernel matrix is ​​used as the kernel matrix to iteratively update the kernel matrix, resulting in the updated kernel matrix.

[0114] The scaling weights refer to the weights used in updating the kernel matrix. The first scaling weight, determined based on the kernel matrix and teacher sample information, is used to calculate the intermediate kernel matrix. The second scaling weight, determined by combining the intermediate kernel matrix and student sample information, is used to calculate the target kernel matrix.

[0115] For example, K, calculated using the Gaussian kernel function, can be used as the kernel matrix K in the initial state before iterative update (i.e., iteration t=0). 0 And by the kernel matrix K 0 With teacher prediction information t i The first scaling weight is calculated, and the result is obtained by combining the first scaling weight with the kernel matrix K. 0 The intermediate kernel matrix is ​​calculated. From this intermediate kernel matrix With student prediction information i The second scaling weight is calculated, and the result is obtained from the second scaling weight and the intermediate kernel matrix. The target kernel matrix K for the current iteration (t iteration) is calculated. t And the target kernel matrix K t This serves as the intermediate transition matrix for the current iteration (i.e., the updated intermediate transition matrix). Similarly, in each iteration, the target kernel matrix K obtained in the previous iteration can be used as the intermediate transition matrix. t K serves as the kernel matrix for this iteration round. 0 And calculate the target kernel matrix K for this iteration round. t This serves as the intermediate transition matrix for that iteration round until the iteration termination condition is met (such as reaching a preset number of iterations or the transition matrix P obtained from the latest two iterations). λ (Convergence), the intermediate transition matrix P obtained from the last iteration update. λ This is the transition matrix.

[0116] For example, in some implementations, teacher sample information includes teacher prediction information, and student sample information includes student prediction information, and the kernel matrix can be iteratively updated based on the teacher prediction information and student prediction information.

[0117] For example, the target kernel matrix for the current iteration can be obtained using the following formula: Among them, the initial kernel matrix Let be a vector of length d consisting entirely of 1s.

[0118] in, This means that in the current iteration round (i.e., the t-th iteration), the kernel matrix K from the previous iteration (t-1) can be used first. t-1 Divided by student prediction information s i Obtain the first scaling weight for the current iteration round, and then multiply it by K. t-1 Obtain the intermediate kernel matrix of the current iteration round. In this process, an all-one vector 1 can be added. d With K t-1 Multiply to obtain K t-1 scalar K t-1 1 d At the same time, it can Convert to a diagonal matrix and obtain the inverse of that diagonal matrix. This will be used as the first scaling weight for the current iteration round.

[0119] This indicates that the intermediate kernel matrix of the current iteration round can be used first. Divided by teacher prediction information t i Obtain the second scaling weight for the current iteration round, and then multiply it by the second scaling weight for the current iteration round. Obtain the target kernel matrix K for the current iteration round. t In this process, an all-one vector 1 can be added. d and Multiply by the transpose of the matrix to obtain scalar of transpose matrix At the same time Convert to a diagonal matrix and obtain the inverse of that diagonal matrix. This will be used as the second scaling weight for the current iteration round.

[0120] For example, in some implementations, teacher sample information includes the teacher sample weights of the teacher model corresponding to the multimedia samples, and student sample information includes the student sample weights of the student model corresponding to the multimedia samples. The kernel matrix is ​​then iteratively updated based on these teacher and student sample weights. This ensures that the teacher and student sample information remain unchanged during the iterative update process, simplifying the iteration process and improving efficiency. Here, the teacher sample weight refers to the degree of importance the teacher model places on each multimedia sample, and the student sample weight refers to the degree of importance the student model places on each multimedia sample.

[0121] For example, the above formula can be simplified based on the sample weights corresponding to the teacher and student models to use a more compact matrix form solution for the iterative process, enabling distillation outside the d-dimensional space and improving the model's expressiveness and generalization ability. For instance, the target kernel matrix for the current iteration can be obtained using the following formula: in, w s and w t These represent the weights of each element in the knowledge distillation process for the student model and the teacher model (i.e., student sample weights and teacher sample weights), respectively. Let b be a vector of all ones. It should be noted that the weights of the teacher samples and the weights of the student samples remain constant during the iteration process.

[0122] in, This means that in the current iteration round (i.e., the t-th iteration), the kernel matrix K from the previous iteration (t-1) can be used first. t-1 Divided by student sample weight w s Obtain the first scaling weight for the current iteration. Then multiply the first scaling weight for the current iteration by K. t-1 Obtain the kernel matrix updated in the current iteration round. In this process, an all-one vector 1 can be added. b With K t-1 Multiply to obtain K t-1 scalar K t-1 1 b At the same time, it can Convert to a diagonal matrix and obtain the inverse of that diagonal matrix. This will be used as the first scaling weight for the current iteration round.

[0123] This indicates that the kernel matrix updated in the current iteration can be used first. Divided by teacher sample weight w t Obtain the second scaling weight for the current iteration round, and then multiply it by the second scaling weight for the current iteration round. Obtain the kernel matrix K for the current iteration round. t In this process, an all-one vector 1 can be added. b and Multiply by the transpose of the matrix to obtain scalar of transpose matrix At the same time Convert to a diagonal matrix and obtain the inverse of that diagonal matrix. This will be used as the second scaling weight for the current iteration round.

[0124] In some implementations, when the multimedia samples are uniformly distributed, teacher sample information and student sample information can be represented by teacher sample weights and student sample weights, respectively, to further simplify the iteration process and improve iteration efficiency.

[0125] For example, in the case where the multimedia samples are uniformly distributed, (That is, to represent a uniform distribution). Therefore, the above formula can be further simplified to obtain the kernel matrix for the current iteration using the following formula:

[0126] in, This means that in the current iteration round (i.e., the t-th iteration), the kernel matrix K from the previous iteration (t-1) can be used first. t-1 With all-1 vector 1 b and 1 b Multiplying by the transpose yields the first scaling weight for the current iteration. Then by the kernel matrix K t-1 Divided by the first scaling weight Obtain the intermediate kernel matrix of the current iteration round. Scaling weights can make each multimedia sample equivalent to the same weight.

[0127] This means that in the current iteration, we can first convert the all-1 vector to 1. b and 1 b The transpose of the kernel matrix K from the previous iteration (t-1) t-1 Multiplying them together yields the second scaling weight for the current iteration. Then from the intermediate kernel matrix Dividing by the second scaling weight of the current iteration yields the target kernel matrix K for the current iteration. t .

[0128] In some implementations, the transport cost between teacher and student prediction information for any pair of multimedia samples can be determined based on the transfer matrix and the inner product of the prediction difference information, thus obtaining the transfer loss between teacher and student prediction information. Specifically, the transfer loss obtained based on the transfer matrix and the prediction difference information includes:

[0129] The transfer loss is obtained by performing an inner product operation on the transfer matrix and the prediction difference information.

[0130] For example, it can be derived from the transition matrix P λ The inner product of the predicted difference information D yields the transfer loss. like Where λ is the weight of entropy regularization, and λ>0.

[0131] In some implementations, multiple loss training models can be combined to help the student model better learn and follow the knowledge of the teacher model, thereby improving the effectiveness of knowledge distillation. Specifically, the knowledge distillation loss also includes at least one of prediction loss and knowledge loss. Based on prediction discrepancy information, the knowledge distillation loss of the student model for the multimedia sample set is determined, and further includes:

[0132] The prediction loss is determined based on the difference between the sample labels of multimedia samples and the student prediction information.

[0133] And / or, based on the difference between teacher-predicted information and student-predicted information, determine knowledge loss.

[0134] Here, sample labels refer to the labels used when performing predictive processing on multimedia samples. For example, sample labels for multimedia samples can be manually labeled or automatically labeled using existing pre-trained models. These labels can be used for classification or labeling of multimedia samples. Different sample labels can be set for different multimedia samples based on the specific application scenario of the task, and this application embodiment does not impose any limitations on this. For example, for sample images, the sample label can be the category of the main object in the image, such as "cat" or "dog". For sample text, the sample label can be a classification label such as topic, sentiment, or language type, such as "happy" or "sad", or it can be a label for entity types in the text, such as "person's name", "place name", "organization name", etc.

[0135] The prediction loss is used to measure the difference between the predicted probability distribution (i.e., student prediction information) and the actual labels (i.e., sample labels). The prediction loss can include cross-entropy loss or mean squared error loss (hereinafter referred to as the first mean squared error loss). For example, the difference between the sample labels of multimedia samples and student prediction information can be calculated using the cross-entropy loss function or the mean squared error loss function to obtain the cross-entropy loss or the mean squared error loss.

[0136] Knowledge loss is a function used to measure the loss in transferring knowledge from the teacher model to the student model during knowledge distillation. Knowledge loss can include divergence loss or mean squared error loss. Divergence loss is a measure of the difference between two probability distributions (i.e., teacher predictions and student predictions). For example, one or more of KL divergence, RKL divergence, and JS divergence can be used to calculate the difference between teacher and student predictions to obtain the divergence loss. Alternatively, the mean squared error loss function can be used to calculate the difference between teacher and student predictions to obtain the mean squared error loss (hereinafter referred to as the second mean squared error loss).

[0137] In some implementations, the prediction loss is determined based on the difference between the sample labels of the same multimedia samples and the student's prediction information, and the knowledge loss is obtained based on the difference between the same teacher's prediction information and the student's prediction information. That is, the prediction loss and knowledge loss are sample-level losses. Therefore, by combining the non-sample-level transfer loss with the sample-level prediction and knowledge losses, and comprehensively considering the differences between the overall sample and individual samples, the accuracy and predictive ability of the trained student model can be further improved.

[0138] For example, KL divergence and RKL divergence can be quantified when students predict information s i Approaching Teacher Prediction Information t i The amount of information lost over time is used to obtain the divergence loss. Specifically, the KL divergence can be expressed as follows: The RKL divergence can be expressed by the following formula: Where d represents vector s i and t i The number of dimensions, i.e., the number of elements in the vector, where j represents the index of an element in the vector.

[0139] For example, since KL divergence and RKL divergence are asymmetric, inconsistencies arise when capturing differences. Therefore, JS divergence can be used to calculate the divergence loss. JS divergence, by combining KL and RKL, overcomes the asymmetry of KL divergence. Specifically, JS divergence can be expressed as the following formula: in,

[0140] In some implementations, when the teacher and student models are regression models, the cross-entropy loss and divergence loss are not applicable to regression processing. Thus, when the teacher and student models are regression models, the knowledge distillation loss also includes a first mean squared error loss or a second mean squared error loss.

[0141] In some implementations, when the knowledge distillation loss includes multiple losses, these multiple losses can be weighted and summed to obtain a total loss, which is then used to train the student model. For example, the total loss can be calculated using the following formula: Alternatively, when both the teacher and student models are regression models, the total loss can be calculated using the following formula: Where b is the total number of samples and i is the sample index. For cross-entropy loss, y i For sample labels, For divergence loss, In order to transfer losses, The first mean square error loss, The second mean squared error loss is represented by α and β, which are weights. The specific values ​​of α and β can be set according to the application scenario or actual needs. For example, the range of α and β is 0 to 1, and they can both be set to 0.5.

[0142] In some implementations, when the sample labels are one-hot labels (i.e., true labels in one-hot encoded form), the teacher model's prediction vectors (logits) are unavailable. In this case, the one-hot labels can be considered as the "hypothetical" prediction vectors of the one-hot teacher model. Due to the dominant zeros in the one-hot labels, traditional divergence measures (such as KL divergence) cannot handle this extreme case of divergence quantification. This reduces the effectiveness of traditional divergence measures (such as KL divergence). Therefore, the KL loss term in the total loss can be ignored, and the total loss can be calculated using the following formula:

[0143] 150. Based on knowledge distillation loss, train the student model to obtain the trained student model, which is then used to predict multimedia information.

[0144] For example, in this embodiment, the student model can be trained for multiple training cycles based on the knowledge distillation loss. A training cycle refers to the number of times the multimedia sample set is input into the teacher and student models and the model parameters are updated once. In each training cycle, the knowledge distillation loss for the current training cycle can be calculated based on the prediction difference information. Backpropagation and model parameter updates are then performed based on this loss until a preset training termination condition is met (such as reaching a preset number of iterations or model convergence). The student model obtained from the last training cycle is the trained student model.

[0145] For example, a kernel function can be calculated based on the predicted difference information in each training cycle. Based on the calculated kernel function, the row scaling vector and column scaling vector are updated through multiple iterations, or the kernel function is updated iteratively, to update the transition matrix, obtaining the updated transition matrix for the current training cycle. The transition loss for the current training cycle is then calculated based on the updated transition matrix. For example, when the transition matrix P... λ Defined as P λ When =K′, K′ can be K N That is, P λ =K N Transfer of losses It can be represented as Among them, K N Let represent the updated kernel matrix in the Nth training cycle. For each training cycle, the transfer loss of the current training cycle can be calculated based on the difference matrix between the updated kernel matrix obtained in the current training cycle (i.e., training cycle N) and the current training cycle.

[0146] The model training scheme provided in this application can be applied to various knowledge distillation scenarios. For example, taking a computer vision model or a language model as an example, a multimedia sample set is obtained, which includes multiple multimedia samples. For each multimedia sample, a teacher model and a student model are used to perform prediction processing on the multimedia sample to obtain prediction information for the multimedia sample. The prediction information for the multimedia sample includes the teacher prediction information of the teacher model for the multimedia sample and the student prediction information of the student model for the multimedia sample. Based on the difference between the teacher prediction information and the student prediction information of any multimedia sample, the prediction difference information of the multimedia sample set is determined. Based on the prediction difference information, the knowledge distillation loss of the student model on the multimedia sample set is determined. Based on the knowledge distillation loss, the student model is trained to obtain the trained student model, which is used to perform prediction processing on multimedia information.

[0147] As can be seen from the above, the embodiments of this application determine the overall recognition difference across the entire multimedia sample set based on the difference between the teacher's prediction information and the student's prediction information of any multimedia sample in the multimedia sample set. This approach is not limited to focusing solely on the difference between teacher and student prediction information within the same multimedia sample, i.e., it is not limited to the difference in prediction information at the multimedia sample level. It can more comprehensively measure the overall difference between teacher and student prediction information across the entire multimedia sample set, thus fully capturing the overall difference between the teacher and student models. Based on this difference information, a knowledge distillation loss is constructed. This approach is not limited to multimedia sample-level distillation but comprehensively captures the overall difference between teacher and student prediction information across the entire multimedia sample set, thereby improving the accuracy and predictive ability of the trained student model.

[0148] The method described in the above embodiments will be further described in detail below.

[0149] In this embodiment, the method of this application embodiment will be described in detail using the model training process applied to computer vision models or language models as an example.

[0150] like Figure 2a As shown, the specific process of a model training method is as follows:

[0151] 210. Obtain a multimedia sample set, which includes multiple multimedia samples.

[0152] For example, a training set (i.e., a set of multimedia samples) can be given, where any multimedia sample in the training set is x. i and its sample labels are The training set can be obtained from multiple open-source datasets, and the multimedia samples can be sample images or sample text. In this embodiment, the teacher model and student model can be models with various structures, such as a purely discriminative encoder, an encoder-decoder, a purely generative decoder, and so on.

[0153] 220. For each multimedia sample, the teacher model and student model are used to perform prediction processing on the multimedia sample to obtain the prediction information of the multimedia sample.

[0154] The prediction information includes teacher predictions for multimedia samples from the teacher model, and student predictions for multimedia samples from the student model. For example, ... Figure 2b The model training process shown allows the multimedia sample set to be input into the teacher model and student model respectively for prediction processing, in order to obtain teacher prediction information and student prediction information for each multimedia sample. Teacher model f T And student model f S The output logits (i.e., the prediction vector) are activated by the softmax function σ. τ After activation processing, the predicted probability distribution of the teacher model is obtained. The predicted probability distribution of the student model Specifically as follows: t i =σ τ (f T (x i )), s i =σ τ (f S (x i Where τ is the temperature parameter and d is the dimension of the output logits. The goal of knowledge distillation is to minimize the measured divergence J(t). i ,s i (To facilitate knowledge transfer)

[0155] 230. Based on the difference between the teacher prediction information and the student prediction information of any multimedia sample, determine the prediction difference information of the multimedia sample set.

[0156] For example, such as Figure 2b The model training process shown can calculate a difference matrix based on the difference between teacher and student predictions, and use this difference matrix as the prediction difference information for the multimedia sample set. Therefore, the loss can be calculated based on the prediction difference information.

[0157] In related technologies, knowledge distillation loss is often calculated using KL divergence, RKL divergence, and JS divergence to train student models; however, these methods have drawbacks.

[0158] KL divergence has three limitations: First, it is asymmetric, exhibiting J0... KL (t i ,s i )≠J KL (s i ,t i This introduces inconsistency because it violates the property of being a distance metric. Secondly, the student model optimized using KL loss attempts to average the multimodal distribution of the teacher model, ultimately leading to underfitting of these patterns—a problem known as the modality averaging problem. Therefore, the student model fails to capture all the key patterns in the data, ultimately impacting the performance of the trained student model. Finally, KL divergence corresponds to a non-smooth function, posing a challenge to optimization.

[0159] Like KL divergence, RKL divergence inherently exhibits asymmetry, leading to inconsistencies in capturing differences. Furthermore, students optimized by RKL loss tend to focus only on high-probability events in the teacher's distribution, a phenomenon known as mode collapse. Therefore, if the teacher sets the probability of an event to zero, the student is forced to do the same. This "zero-coercion" effect can be problematic because the student model lacks the ability to learn the full distribution of the teacher's model, resulting in poor performance.

[0160] While JS divergence overcomes the asymmetry limitation of KL divergence, it remains susceptible to non-smoothness, making optimization challenging. Furthermore, due to the insufficient penalty of JS loss for matching low-probability regions, the student model may overestimate the probability of rare events. JS (t i ,s i There is also the risk of gradient vanishing when the gradient degenerates to a constant on distributions with little or no overlap.

[0161] To address the shortcomings of related technologies, this application proposes a model training method based on Sinkhorn distance. This method is based on a relaxed form of optimal transport plan and entropy regularization, which considers the minimum cost of quality transport that transforms one probability into another. Specifically, it first defines the Wasserstein distance, which involves a transport polyhedron U(t) i ,s i A set consisting of all matrices satisfying the following constraints composition: in It is a vector of all 1s, where P is the transfer matrix (i.e., the transition matrix).

[0162] While the Sinkhorn distance can be directly applied to sample-level logits (i.e., predicted vector) matching (i.e., sample-level distillation), it doesn't fully utilize its perception of differences in distribution structure. Therefore, in this embodiment, the Sinkhorn distance metric is not limited to sample-level logits pairing. Since the Sinkhorn distance is a symmetric metric, its derivation from optimal transfer (OT) imposes explicit constraints on matching correctness. This means that, given a set A and a set B of logits outputs from teachers and students respectively, minimizing the overall Sinkhorn distance between A and B forces precise element-level matching between two outputs from the same sample. This property allows it to go beyond sample-level distillation and lays the foundation for subsequent processing.

[0163] Therefore, in this embodiment of the application, a cost matrix (i.e., a difference matrix) is given. The Wasserstein distance is: Where D m,n It is usually t i and s i The absolute difference between the m-th and n-th elements: D m,n =|t i(m) -s i(n) | By not being limited to logits pairing between samples, a “batch-based” cost matrix D (i.e., prediction variance information) is obtained.

[0164] In some implementations, l can be used p The norm is used to measure the pairwise difference between the i-th and j-th samples in a batch, thus obtaining the "batch-based" cost matrix. Entry D i,j :D i,j =||t i -s j || p .

[0165] Therefore, this application proposes a "batch" SinKD, which avoids the pitfalls of sample knowledge distillation (KD) that only applies to each teacher-student multimedia sample pair. Instead, it performs KD on the set of logits of both the teacher and student models. That is, the entire batch of multimedia samples containing b samples participates in the overall output logits of the model. and In the divergence measurement, these logits come from the prediction information of the teacher model and the student model for any multimedia sample pair, respectively. Therefore, by batch reconstruction, especially when d << b, the dimensionality of the "observation" space can be increased. In this way, even complex and implicit distribution geometry can be captured through low-dimensional observation. In the embodiments of this application, no additional modules are introduced or the output format specific to the NLP task is modified.

[0166] 240. The kernel matrix is ​​obtained by calculating the predicted difference information using the kernel function.

[0167] 250. Determine the transition matrix based on the kernel matrix.

[0168] For example, such as Figure 2b The model training process shown can iteratively update the transition matrix based on prediction difference information to obtain the final transition matrix. This iterative update process can be based on Sinkhorn normalization, a normalization method for probability distribution matrices, specifically used to ensure that the rows and columns of the matrix satisfy the requirements of the probability distribution. Sinkhorn normalization can be used to process cost matrices or distance matrices, ensuring that these matrices can represent effective probability distributions, thus applicable to optimization problems, probabilistic models in machine learning, and numerical computation tasks related to probability distributions. The Sinkhorn normalization method iteratively converges the matrix to a state that satisfies the probability distribution constraints.

[0169] In this embodiment of the application, to avoid the large amount of computation required to solve the OT problem, a Sinkhorn distance is proposed as a constrained optimization method for quickly approximating the Wasserstein distance. The Sinkhorn distance is defined as the OT plan P λ The inner product between the transition matrix and the cost matrix D: J SD (t i ,s i )= <P λ ,D>。 where λ>0 is the weight for entropy regularization. This transition matrix is ​​the optimal transport matrix in the OT problem.

[0170] OT plan P λ This can be obtained by minimizing the following objective function: -λh(P). Where h(P) is the entropy of the transition matrix P. The entropy term encourages a more dispersed transmission plan for optimization.

[0171] For example, in some implementations, it can be achieved by adjusting t i and s i The sample-by-sample Sinkhorn normalization method, which performs iterative updates, can obtain P. λThe original solution:

[0172] By normalizing the Sinkhorn distance for each sample, P λ The basic solution in t i and s i This is done through iterative updates. This represents element-wise division, where t represents the number of iterations. Two vectors... All are non-negative. Kernel matrix It is constructed by applying a Gaussian kernel to D and performing entropy regularization using weights λ: P λ Defined as: P λ =diag(v t )Kdiag(u t ).

[0173] For example, in some implementations, P is set before batch Sinkhorn distance normalization is proposed. λ Reconstruct into an equivalent vector form, where iteration is performed only on K: in exist Defined in Chinese.

[0174] In some implementations, to perform distillation outside of d-dimensional space, embodiments of this application propose a more compact matrix-form solution for batch normalization, wherein... Where w s and w t ... Under these conditions, for K t The update can be further simplified to:

[0175] For simplicity, irrelevant constants have been excluded from the above equations. By performing a predetermined number of iterations T within the current training period (e.g., training period N), the kernel matrix is ​​updated T times to obtain the updated kernel matrix, which serves as the updated transition matrix: P λ =K N ,K N This represents the updated kernel matrix in the Nth training cycle.

[0176] 260. Based on the transition matrix and prediction difference information, the transition loss between teacher prediction information and student prediction information is obtained.

[0177] For example, such as Figure 2b The model training process shown can be used to calculate the transition loss based on the transition matrix. The Sinkhorn loss (i.e., the transition loss) based on the Sinkhorn distance can be defined as follows:

[0178] 270. Determine the prediction loss based on the difference between the sample labels of multimedia samples and the student's prediction information, and determine the knowledge loss based on the difference between the teacher's prediction information and the student's prediction information.

[0179] 280. Based on knowledge distillation loss, a student model is trained to obtain a trained student model. The trained student model is used to predict multimedia information. Knowledge distillation loss includes transfer loss, prediction loss and knowledge loss.

[0180] For example, for each batch of b samples, cross-entropy loss can be used. KL loss and Sinkhorn loss Distillation is performed to obtain the total knowledge distillation loss: Where α and β are weights. And during the distillation process, because The second term in the equation can be considered a constant.

[0181] Therefore, in this embodiment, the student model can be trained for multiple training cycles based on the knowledge distillation loss. In each training cycle, the knowledge distillation loss of the current training cycle can be calculated based on the above formula, and backpropagation and model parameter update processes can be performed based on this loss until the preset training termination condition is reached (such as reaching a preset number of times or model convergence). The student model updated in the last training cycle is the trained student model.

[0182] To verify the effectiveness of the model training method provided in this application, the embodiments of this application have been implemented and validated on multiple open-source and public datasets and various architectures (pure discriminative encoder, encoder-decoder, and pure generative decoder) to demonstrate effectiveness that surpasses many existing methods.

[0183] Validation Experiment 1: Validation experiment on the BERT model architecture. This validation experiment will use BERT... base Using BERT6(S) as the teacher model (T) and BERT6(S) as the student model (S), the method provided in this application is compared with existing methods on the GLUE dataset. The verification results are as follows: Figure 2c Table 1 shows the experimental results of Experiment 1.

[0184] Table 1 compares the scores of the basic BERT model architecture, distillation methods based on task-specific representations such as distillation method 1 (PKD), distillation method 2 (TinyBERT), distillation method 3 (RKD), distillation method 4 (CKD), distillation method 5 (SFTN), distillation method 6 (TAKD), distillation method 7 (ProKT), distillation method 8 (MGSKD), distillation method 9 (MetaDistill), and distillation method 10 (AD-KD), distillation methods based on task-specific Logits (i.e., prediction vectors) such as distillation method 11 (VanillaKD), distillation method 12 (RCO), distillation method 13 (DML), distillation method 14 (PD), and distillation method 15 (ReAugKD), and the method provided in this application (i.e., SinKD) on various evaluation metrics. Evaluation metrics may include metric 1 (COLA(MCC)), metric 2 (SST-2(ACC)), metric 3 (MINLI-(m / mm)(ACC)), metric 4 (MRPC(F1)), metric 5 (RTE(ACC)), metric 6 (QNLI(ACC)), metric 7 (QQP(ACC)), and metric 8 (STS-B(Spear)), etc. Table 1 shows two averages calculated for multiple evaluation metrics; one average excludes MINLI-(M / MM) and STS-B scores, while the other excludes only MINLI-(M / MM) scores. As can be seen from Table 1, on the BERT model architecture, the method provided in this embodiment scores significantly higher than other existing methods on each evaluation metric.

[0185] Verification Experiment 2: This verification experiment uses only a portion of the loss term in the loss function of the method provided in the embodiments of this application. The loss function in the method provided in this application embodiment is then compared with the method provided in this application embodiment (i.e., SinKD). The verification results are as follows: Figure 2d Table 2 shows the experimental results of Experiment 2.

[0186] In Table 2, only some loss items will be used, such as The method compared the scores of various evaluation metrics with the method provided in the embodiments of this application (i.e., SinKD). Evaluation metrics may include metric 1 (COLA (MCC)), metric 2 (SST-2 (ACC)), metric 3 (MINLI-(m / mm)(ACC)), metric 4 (MRPC(F1)), metric 5 (RTE(ACC)), metric 6 (QNLI(ACC)), metric 7 (QQP(ACC)), and metric 8 (STS-B(Spear)), etc. As can be seen from Table 2, the method provided in the embodiments of this application scores significantly higher than other existing methods on each evaluation metric.

[0187] Verification Experiment 3: This verification experiment uses various existing difference measurement functions (divergence), such as RKL divergence, JS divergence, and TVD divergence, to replace the transfer loss in the method provided in the embodiments of this application, as in Method 1. Method 2 Method 3 and Method 4 Compare it with the method (i.e., SinKD) provided in the embodiments of this application. The verification results are as follows. Figure 2e Table 3 shows the experimental results of Experiment 3.

[0188] Table 3 compares the scores of various evaluation metrics obtained by replacing the transfer loss with RKL divergence, JS divergence, and TVD divergence, such as Method 1 (RKL), Method 2 (JS), and Method 3 (TVD), with the method (SinKD) provided in this application embodiment. Evaluation metrics may include complexity, metric 1 (COLA (MCC)), metric 2 (SST-2 (ACC)), metric 3 (MINLI-(m / mm)(ACC)), metric 4 (MRPC (F1)), metric 5 (RTE (ACC)), metric 6 (QNLI (ACC)), metric 7 (QQP (ACC)), and metric 8 (STS-B (Spear)). Here, complexity 0 represents asymptotic complexity, typically used to describe the time or space complexity of an algorithm. As can be seen from Table 3, the method provided in this application embodiment scores significantly higher than other existing methods on various evaluation metrics. Therefore, compared to various existing difference measurement functions (divergence), even when applied within the framework of the embodiments of this application, they are not as good as the divergence (i.e., transfer loss) defined based on Sinkhorn distance proposed in the embodiments of this application.

[0189] Verification Experiment 4: Verification experiments on various model architectures such as T0 and GPT. This verification experiment compares the method provided in the embodiments of this application with existing methods on various model architectures other than the BERT model. The verification results are as follows: Figure 2f Table 4 shows the experimental results of Experiment 4. Figure 2g Table 5 shows the experimental results of Experiment 4. Figure 2h Table 6 shows the experimental results of Experiment 4.

[0190] Table 4 will include T0 11B As the teacher model (T), T0 11B As a student model (S), the scores of the T0 basic model architecture, based on Method 1 (KL divergence), Method 2 (KL+RKL divergence), Method 3 (KL+JS divergence), and the method (KL+SinKD) provided in this application embodiment (KL+SinKD), were compared on various evaluation metrics such as metric 1 (RTE(ACC)) and metric 2 (CB). Table 4 shows that, on the T0 model architecture, the method provided in this application embodiment significantly outperforms other existing methods in all evaluation metrics.

[0191] Table 5 lists GPT-Neo 1.3B As the teacher model (T), GPT-Neo 125M As a student model (S).

[0192] Table 6 lists GPT-Neo 2.7B As the teacher model (T), GPT-Neo 1.3B As a student model (S), the average scores (mean ± standard deviation) of various evaluation metrics, such as metric 1 (RTE(ACC)) and metric 2 (CB), were compared on the SUPEKGLUE dataset, using the GPT basic model architecture, the GPT basic model architecture based on method 1 (KL divergence), method 2 (KL+RKL divergence), method 3 (KL+JS divergence), and the method (KL+SinKD) provided in this application embodiment. For the RTE task, the prompt templates GPT-3sryle, lfNLf crowdsource, and mHst be IrWe were used; for the CB task, the prompt templates GPT-3sryle, musf be tnue, and should assume were used. As shown in Tables 5 and 6, on the GPT model architecture, the method provided in this application embodiment significantly outperforms other existing methods in all evaluation metrics.

[0193] Verification Experiment 5: Verification Experiment for Fine-tuning Learning of One-hot Labels. This verification experiment compares the method provided in this application embodiment with other methods such as Method 1 (CE / MSE), Method 2 (TaiLr), Method 3 (MixCE), and Method 4 (CE / MSE + Sample-level SinKD) for the fine-tuning learning task of one-hot labels. The verification results are as follows: Figure 2iTable 7 shows the experimental results of Experiment 5. As can be seen from Table 7, the embodiments of this application are also effective for fine-tuning learning of one-hot labels. The method of the embodiments of this application even outperforms specialized one-hot label fine-tuning methods such as CE / MSE, TaiLr, and MixCE. Furthermore, the batch processing method of this approach, compared to sample-level CE / MSE + sample-level SinKD, also shows significantly better results.

[0194] Experiment 6 is a validation experiment on computer vision model architectures. This validation experiment is based on the CIFAR-100 test set and uses computer vision model architectures such as ResNet, VGG, and ShuffleNet. Specifically, it targets Teacher Model 1 (ResNet-56), Student Model 1 (ResNet-20), Teacher Model 2 (ResNet-110), Student Model 2 (ResNet-20), Teacher Model 3 (ResNet-110), Student Model 3 (ResNet-32), Teacher Model 4 (VGG-13), Student Model 4 (VGG-8), Teacher Model 5 (ResNet-32x4), Student Model 5 (ShuffleNetV1), and Teacher Model 6 (ResNet-32x4) and Student Model 6 (ShuffleNetV2). The method provided in this application's embodiments is compared with existing methods such as Method 1 (KD), Method 2 (FitNET), Method 3 (AT), Method 4 (SP), Method 5 (CC), Method 6 (VID), Method 7 (RKD), Method 8 (PKT), Method 9 (AB), Method 10 (FT), Method 11 (ProKT), Method 12 (CRD), and Method 13 (MetaDistil). The verification results are as follows: Figure 2j Table 8 shows the experimental results of Experiment 6. As can be seen from Tables 5 and 6, the method provided in this application embodiment scores significantly higher than other existing methods on various evaluation metrics in terms of computer vision model architecture. Therefore, the method provided in this application embodiment is also effective for distillation of visual tasks (image classification models). Clearly, the method provided in this application embodiment can be applied not only to language models (generative and discriminative) but also to the distillation of visual models (such as classification and detection models).

[0195] As can be seen from the above, the embodiments of this application can circumvent the shortcomings of existing knowledge distillation methods (such as methods based on distribution difference measurement functions such as KL divergence, RKL divergence, and JS divergence). The embodiments of this application are based on a measure of difference information applicable to any two distributions, and a knowledge distillation loss is constructed based on this difference information. Therefore, it is not limited to multimedia sample-level distillation, but comprehensively captures the overall difference between teacher and student prediction information across the entire multimedia sample set by comprehensively measuring the overall difference between the teacher and student models, thereby improving the accuracy and predictive ability of the trained student model. Simultaneously, the trained student model is applicable to knowledge distillation of various models such as computer vision models and large language models, exhibiting better generalization ability.

[0196] To better implement the above methods, this application also provides a model training device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0197] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the model training device specifically integrated into the server as an example.

[0198] For example, such as Figure 3 As shown, the model training device may include an acquisition unit 310, a prediction unit 320, a difference determination unit 330, a loss determination unit 340, and a training unit 350, as follows:

[0199] (I) Acquisition Unit 310

[0200] Used to obtain a multimedia sample set, which includes multiple multimedia samples.

[0201] (II) Prediction Unit 320

[0202] For each multimedia sample, the teacher model and student model are used to perform prediction processing on the multimedia sample to obtain the prediction information of the multimedia sample. The prediction information of the multimedia sample includes the teacher prediction information of the multimedia sample by the teacher model and the student prediction information of the multimedia sample by the student model.

[0203] (III) Difference Determination Unit 330

[0204] This is used to determine the prediction difference information of a multimedia sample set based on the difference between the teacher prediction information and the student prediction information of any multimedia sample.

[0205] (iv) Loss Determination Unit 340

[0206] Used to determine the knowledge distillation loss of the student model on the multimedia sample set based on the prediction difference information.

[0207] In some implementations, the knowledge distillation loss includes a transfer loss. The loss determination unit is specifically used to: calculate a kernel matrix from the prediction difference information using a kernel function, where the elements of the kernel matrix represent the similarity between the teacher's prediction information and the student's prediction information for any multimedia sample; determine a transfer matrix based on the kernel matrix, where the elements of the transfer matrix represent the transfer amount between the teacher's prediction information and the student's prediction information for any multimedia sample; and obtain the transfer loss between the teacher's prediction information and the student's prediction information based on the transfer matrix and the prediction difference information.

[0208] In some implementations, determining the transition matrix based on the kernel matrix includes: iteratively updating the kernel matrix based on the sample information of the multimedia samples to obtain the updated kernel matrix. The sample information includes teacher sample information and student sample information. The teacher sample information includes at least one of teacher prediction information of the multimedia samples and teacher sample weights of the teacher model corresponding to the multimedia samples. The student sample information includes at least one of student prediction information of the multimedia samples and student sample weights of the student model corresponding to the multimedia samples.

[0209] In some implementations, the kernel matrix is ​​iteratively updated based on the sample information of multimedia samples to obtain an updated kernel matrix, including: determining a first scaling weight based on the kernel matrix and student sample information; obtaining an intermediate kernel matrix based on the first scaling weight and the kernel matrix; determining a second scaling weight based on the intermediate kernel matrix and teacher sample information; obtaining a target kernel matrix based on the second scaling weight and the intermediate kernel matrix; and using the target kernel matrix as the kernel matrix to iteratively update the kernel matrix to obtain an updated kernel matrix.

[0210] In some implementations, determining the transition matrix based on the kernel matrix includes: obtaining the row scaling vectors corresponding to the rows in the kernel matrix and the column scaling vectors corresponding to the columns in the kernel matrix; diagonally scaling the kernel matrix using the row scaling vectors and column scaling vectors to obtain an initial transition matrix; and iteratively updating the row scaling vectors and column scaling vectors to update the initial transition matrix to obtain the final transition matrix.

[0211] In some implementations, the knowledge distillation loss also includes at least one of prediction loss and knowledge loss, and the loss determination unit can also be used to: determine the prediction loss based on the difference between the sample labels of the multimedia samples and the student prediction information; and / or, determine the knowledge loss based on the difference between the teacher prediction information and the student prediction information.

[0212] (V) Training Unit 350

[0213] This is used to train a student model based on knowledge distillation loss, resulting in a trained student model that is then used for predictive processing of multimedia information.

[0214] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0215] As shown above, the model training device in this embodiment includes an acquisition unit, a prediction unit, a difference determination unit, a loss determination unit, and a training unit. The acquisition unit acquires a multimedia sample set, which includes multiple multimedia samples. The prediction unit performs prediction processing on each multimedia sample using both a teacher model and a student model to obtain prediction information for the multimedia sample. This prediction information includes teacher prediction information from the teacher model and student prediction information from the student model. The difference determination unit determines the prediction difference information of the multimedia sample set based on the difference between the teacher prediction information and the student prediction information of any multimedia sample. The loss determination unit determines the knowledge distillation loss of the student model on the multimedia sample set based on the prediction difference information. The training unit trains the student model based on the knowledge distillation loss to obtain a trained student model, which is used to perform prediction processing on multimedia information.

[0216] Therefore, this application's embodiments determine the overall recognition difference across the entire multimedia sample set based on the difference between the teacher's prediction information and the student's prediction information of any multimedia sample within the set. This approach goes beyond focusing solely on the difference between teacher and student prediction information within the same multimedia sample; it is not limited to the difference at the multimedia sample level. Instead, it comprehensively measures the overall difference between teacher and student prediction information across the entire multimedia sample set, thus fully capturing the overall difference between the teacher and student models. A knowledge distillation loss is then constructed based on this difference information. This approach goes beyond multimedia sample-level distillation; by comprehensively measuring the overall difference between teacher and student prediction information across the entire multimedia sample set, it fully captures the overall difference between the teacher and student models, thereby improving the accuracy and predictive ability of the trained student model.

[0217] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0218] In some embodiments, the model training device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the model training method of this application.

[0219] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a schematic diagram of the server structure involved in an embodiment of this application. Specifically:

[0220] The server may include components such as a processor 410 with one or more processing cores, a memory 420 with one or more computer-readable storage media, a power supply 430, an input module 440, and a communication module 450. Those skilled in the art will understand that... Figure 4 The server architecture shown does not constitute a limitation on the server and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0221] Processor 410 is the control center of the server, connecting various parts of the server via various interfaces and lines. It performs various server functions and processes data by running or executing software programs and / or modules stored in memory 420, and by calling data stored in memory 420. In some embodiments, processor 410 may include one or more processing cores; in some embodiments, processor 410 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 410.

[0222] The memory 420 can be used to store software programs and modules. The processor 410 executes various functional applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 420 may also include a memory controller to provide the processor 410 with access to the memory 420.

[0223] The server also includes a power supply 430 that supplies power to the various components. In some embodiments, the power supply 430 can be logically connected to the processor 410 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 430 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0224] The server may also include an input module 440, which can be used to receive input numeric or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0225] The server may also include a communication module 450. In some embodiments, the communication module 450 may include a wireless module, through which the server can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 450 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0226] Although not shown, the server may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 410 in the server loads the executable files corresponding to the processes of one or more applications into the memory 420 according to the following instructions, and the processor 410 runs the applications stored in the memory 420 to realize various functions, as follows:

[0227] A multimedia sample set is obtained, comprising multiple multimedia samples. For each multimedia sample, prediction processing is performed using both a teacher model and a student model to obtain prediction information. This prediction information includes teacher predictions and student predictions. Based on the difference between the teacher and student predictions for any multimedia sample, prediction difference information for the multimedia sample set is determined. The knowledge distillation loss of the student model for the multimedia sample set is determined based on this prediction difference information. The student model is trained using this knowledge distillation loss to obtain the trained student model, which is then used for predictive processing of multimedia information.

[0228] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0229] As can be seen from the above, the embodiments of this application determine the overall recognition difference across the entire multimedia sample set based on the difference between the teacher's prediction information and the student's prediction information of any multimedia sample in the multimedia sample set. This approach is not limited to focusing solely on the difference between teacher and student prediction information within the same multimedia sample, i.e., it is not limited to the difference in prediction information at the multimedia sample level. It can more comprehensively measure the overall difference between teacher and student prediction information across the entire multimedia sample set, thus fully capturing the overall difference between the teacher and student models. Based on this difference information, a knowledge distillation loss is constructed. This approach is not limited to multimedia sample-level distillation but comprehensively captures the overall difference between teacher and student prediction information across the entire multimedia sample set, thereby improving the accuracy and predictive ability of the trained student model.

[0230] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0231] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the model training methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0232] A multimedia sample set is obtained, comprising multiple multimedia samples. For each multimedia sample, prediction processing is performed using both a teacher model and a student model to obtain prediction information. This prediction information includes teacher predictions and student predictions. Based on the difference between the teacher and student predictions for any multimedia sample, prediction difference information for the multimedia sample set is determined. The knowledge distillation loss of the student model for the multimedia sample set is determined based on this prediction difference information. The student model is trained using this knowledge distillation loss to obtain the trained student model, which is then used for predictive processing of multimedia information.

[0233] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0234] According to one aspect of this application, a computer program product or computer program is provided, comprising a computer program or instructions that, when executed by a processor, implement the steps of the methods provided in the various optional implementations of the above embodiments. The computer program / instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instructions from the computer-readable storage medium and executes the computer program / instructions, causing the electronic device to perform the methods provided in the various optional implementations of the above embodiments.

[0235] Since the instructions stored in the storage medium can execute the steps in any of the model training methods provided in the embodiments of this application, the beneficial effects that any of the model training methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0236] The foregoing has provided a detailed description of a model training method, apparatus, electronic device, storage medium, and program product provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a multimedia sample set comprising a plurality of multimedia samples; for each multimedia sample, performing prediction processing on the multimedia sample by a teacher model and a student model respectively to obtain prediction information of the multimedia sample, the prediction information of the multimedia sample comprising teacher prediction information of the multimedia sample by the teacher model and student prediction information of the multimedia sample by the student model; determining prediction difference information of the multimedia sample set according to a difference between the teacher prediction information of any multimedia sample and the student prediction information of any multimedia sample; determining a knowledge distillation loss of the student model for the multimedia sample set according to the prediction difference information; training the student model based on the knowledge distillation loss to obtain a trained student model, the trained student model being used for prediction processing on multimedia information. 2.The model training method of claim 1, wherein, The knowledge distillation loss comprises a transfer loss, and the determining of the knowledge distillation loss of the student model for the multimedia sample set according to the prediction difference information comprises: calculating a kernel matrix from the prediction difference information by a kernel function, an element of the kernel matrix representing a similarity between the teacher prediction information of any multimedia sample and the student prediction information of any multimedia sample; determining a transfer matrix according to the kernel matrix, an element of the transfer matrix representing a transfer amount between the teacher prediction information of any multimedia sample and the student prediction information of any multimedia sample; obtaining a transfer loss between the teacher prediction information and the student prediction information based on the transfer matrix and the prediction difference information. 3.The model training method of claim 2, wherein, The determining of the transfer matrix according to the kernel matrix comprises: iteratively updating the kernel matrix based on sample information of the multimedia sample to obtain an updated kernel matrix, the sample information comprising teacher sample information and student sample information, the teacher sample information comprising at least one of the teacher prediction information of the multimedia sample and a teacher sample weight of the multimedia sample corresponding to the teacher model, and the student sample information comprising at least one of the student prediction information of the multimedia sample and a student sample weight of the multimedia sample corresponding to the student model; using the updated kernel matrix as the transfer matrix. 4.The model training method of claim 3, wherein, The iteratively updating of the kernel matrix based on the sample information of the multimedia sample to obtain the updated kernel matrix comprises: determining a first scaling weight based on the kernel matrix and the student sample information; obtaining an intermediate kernel matrix from the first scaling weight and the kernel matrix; determining a second scaling weight based on the intermediate kernel matrix and the teacher sample information; obtaining a target kernel matrix from the second scaling weight and the intermediate kernel matrix; using the target kernel matrix as the kernel matrix to iteratively update the kernel matrix to obtain the updated kernel matrix. 5.The model training method of claim 3, wherein, The determining of the transfer matrix according to the kernel matrix comprises: obtaining a row scaling vector corresponding to a row in the kernel matrix and a column scaling vector corresponding to a column in the kernel matrix; scaling the core matrix diagonally by the row scaling vector and the column scaling vector to obtain an initial transition matrix; iteratively updating the row scaling vector and the column scaling vector to update the initial transition matrix to obtain the transition matrix. 6.The model training method of claim 2, wherein, The knowledge distillation loss further includes at least one of a prediction loss and a knowledge loss, and the determining the knowledge distillation loss of the student model on the set of multimedia samples according to the prediction difference information further includes: determining the prediction loss based on a difference between a sample label of the multimedia sample and the student prediction information; and / or determining the knowledge loss based on a difference between the teacher prediction information and the student prediction information.

7. A model training apparatus characterized by comprising: The method comprises: obtaining a set of multimedia samples, the set of multimedia samples including a plurality of multimedia samples; performing prediction processing on each of the multimedia samples by a teacher model and a student model to obtain prediction information of the multimedia sample, the prediction information of the multimedia sample including teacher prediction information of the teacher model on the multimedia sample and student prediction information of the student model on the multimedia sample; determining prediction difference information of the set of multimedia samples according to a difference between the teacher prediction information of any of the multimedia samples and the student prediction information of any of the multimedia samples; determining a knowledge distillation loss of the student model on the set of multimedia samples according to the prediction difference information; training the student model based on the knowledge distillation loss to obtain a trained student model, the trained student model being used for prediction processing on multimedia information.

8. An electronic device, comprising: The processor and the memory, the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the model training method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, the instructions are suitable for being loaded by the processor to execute the steps in the model training method of any one of claims 1-6.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps in the model training method of any one of claims 1-6.

Images