Model training method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a model training method and device, equipment, a storage medium and a program product, which are used for improving the performance effect of a target student model on the basis of reducing the parameter quantity of the student model. The method comprises the following steps: acquiring object information, and extracting an embedded feature vector of the object information; processing the embedded feature vector based on a teacher model to obtain first interlayer information, first matrix information and first prediction information; processing the embedded feature vector based on the initial student model to obtain second interlayer information, second matrix information and second prediction information; calculating first loss information based on the first interlayer information and the second interlayer information, calculating second loss information based on the first matrix information and the second matrix information, and calculating third loss information based on the first prediction information and the second prediction information; and performing updating processing on model parameters of the initial student model based on the target loss information to obtain a target student model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a method, apparatus, device, storage medium, and program product for model training. Background Art

[0002] In recent years, pre-trained language models have achieved remarkable success in various natural language tasks. However, large-scale pre-trained models are costly in terms of storage, memory, and computational time, which poses challenges to online services in real-world applications. Therefore, it is crucial to compress the model while maintaining its performance.

[0003] In related model compression schemes, knowledge distillation technology is used to transfer the knowledge of the teacher model to a smaller-scale student model, so as to perform knowledge distillation processing on each layer component of the model structure through a hierarchical distillation strategy. However, the knowledge information extracted by using traditional distillation methods is all coarse-grained, and in order to reduce the number of parameters of the student model, some layers in the student model are directly removed, which easily leads to the inability of the student model to align with the knowledge distribution of the teacher model due to layer mismatch, resulting in poor performance of the distilled target student model and inability to handle subsequent downstream tasks with high quality. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, device, storage medium, and program product for model training, which are used to improve the performance of the target student model on the basis of reducing the number of parameters of the student model and improve the processing quality of subsequent downstream tasks.

[0005] In a first aspect, the embodiments of the present application provide a method for model training. The method includes:

[0006] Obtain object information and extract the embedded feature vector of the object information;

[0007] Process the embedded feature vector based on the teacher model to obtain first inter-layer information, first matrix information, and first prediction information. The first inter-layer information is used to describe the self-attention distribution and hidden layer state of the teacher model, the first matrix information is used to describe the matrix situation of the multi-head attention module and the matrix situation of the feed-forward neural network (FNN) module of the teacher model, and the first prediction information is used to describe the model prediction situation of the teacher model for the object information;

[0008] Process the embedded feature vectors based on the initial student model to obtain second inter-layer information, second matrix information, and second prediction information. The second inter-layer information is used to describe the self-attention distribution and hidden layer state of the initial student model. The second matrix information is used to describe the matrix situations of the multi-head attention module and the FNN module of the initial student model. The second prediction information is used to describe the model prediction situation of the initial student model for object information. The initial student model is a neural network model obtained by performing singular value decomposition on the weight matrix of the student model to be optimized.

[0009] Calculate first loss information based on the first inter-layer information and the second inter-layer information, calculate second loss information based on the first matrix information and the second matrix information, and calculate third loss information based on the first prediction information and the second prediction information.

[0010] Update the model parameters of the initial student model based on the target loss information to obtain a target student model. The target student model is used to process downstream tasks related to object information. The target loss information is one or more of the first loss information, the second loss information, and the third loss information.

[0011] In a second aspect, an embodiment of the present application provides a model training device. The model training device includes:

[0012] An acquisition unit for acquiring object information;

[0013] An extraction unit for extracting the embedded feature vectors of the object information;

[0014] A processing unit for processing the embedded feature vectors based on the teacher model to obtain first inter-layer information, first matrix information, and first prediction information. The first inter-layer information is used to describe the self-attention distribution and hidden layer state of the teacher model. The first matrix information is used to describe the matrix situations of the multi-head attention module and the feed-forward neural network FNN module of the teacher model. The first prediction information is used to describe the model prediction situation of the teacher model for object information;

[0015] A processing unit for processing the embedded feature vectors based on the initial student model to obtain second inter-layer information, second matrix information, and second prediction information. The second inter-layer information is used to describe the self-attention distribution and hidden layer state of the initial student model. The second matrix information is used to describe the matrix situations of the multi-head attention module and the FNN module of the initial student model. The second prediction information is used to describe the model prediction situation of the initial student model for object information. The initial student model is a neural network model obtained by performing singular value decomposition on the weight matrix of the student model to be optimized.

[0016] A processing unit, configured to calculate first loss information based on first inter-layer information and second inter-layer information, calculate second loss information based on first matrix information and second matrix information, and calculate third loss information based on first prediction information and second prediction information;

[0017] An updating unit, configured to perform an updating process on model parameters of an initial student model based on target loss information to obtain a target student model, where the target student model is used to process downstream tasks related to object information, and the target loss information is one or more of the first loss information, the second loss information, and the third loss information.

[0018] In some optional examples, the obtaining unit is further configured to: before obtaining the target student model by performing an updating process on model parameters of the initial student model based on the target loss information, obtain the number of distillation layers and the number of first structural layers, where the number of distillation layers is used to describe the number of intermediate layers that need to be aligned in the teacher model and the initial student model, and the number of first structural layers is used to describe the number of Transformer structures in the teacher model and the initial student model;

[0019] The processing unit is specifically configured to determine the target loss information from the first loss information, the second loss information, and the third loss information based on a comparison result between the number of distillation layers and the number of first structural layers.

[0020] In some other optional examples, the processing unit is specifically configured to:

[0021] When the number of distillation layers is greater than or equal to zero and less than a first value, select the second loss information from the first loss information, the second loss information, and the third loss information as the target loss information, where the first value is obtained by 1 / N times the number of first structural layers, and N is an integer greater than or equal to 2.

[0022] In some other optional examples, the processing unit is specifically configured to:

[0023] When the number of distillation layers is greater than or equal to the first value and less than or equal to the number of first structural layers, select the first loss information and the second loss information from the first loss information, the second loss information, and the third loss information as the target loss information, where the first value is obtained by 1 / N times the number of first structural layers, and N is an integer greater than or equal to 2.

[0024] In some other optional examples, the processing unit is specifically configured to:

[0025] When the number of distillation layers is equal to a second value, select the first loss information, the second loss information, and the third loss information as the target loss information, where the second value is obtained by the sum of the number of first structural layers and a preset value.

[0026] In some other alternative examples, the first inter-layer information includes the self-attention distribution information and the hidden layer information of the teacher model, and the second inter-layer information includes the self-attention distribution information and the hidden layer information of the initial student model; the processing unit is specifically configured to:

[0027] Calculate the difference between the self-attention distribution information of the initial student model and the self-attention distribution information of the teacher model to obtain the first inter-layer sub-loss value;

[0028] Calculate the difference between the hidden layer information of the initial student model and the hidden layer information of the teacher model to obtain the second inter-layer sub-loss value;

[0029] Determine the first loss information based on the first inter-layer sub-loss value and the second inter-layer sub-loss value.

[0030] In some other alternative examples, the self-attention distribution information of the teacher model includes the first self-attention distribution information of each of the h attention heads, and the self-attention distribution information of the initial student model includes the second self-attention distribution information of each attention head, where h is an integer greater than or equal to 1; the processing unit is specifically configured to:

[0031] Calculate the KL divergence between the second self-attention distribution information of each attention head and the first self-attention distribution information of each attention head to obtain h KL divergence values;

[0032] Calculate the sum of the h KL divergence values to obtain the total divergence value;

[0033] Calculate the quotient of the total divergence value and the number of attention heads to obtain the first inter-layer sub-loss value.

[0034] In some other alternative examples, the processing unit is specifically configured to: calculate the mean square error between the hidden layer information of the initial student model and the hidden layer information of the teacher model to obtain the second inter-layer sub-loss value.

[0035] In some other alternative examples, the first matrix information includes the matrix information of the multi-head attention module and the matrix information of the FNN module of the teacher model, and the second matrix information includes the matrix information of the multi-head attention module and the matrix information of the FNN module of the initial student model; the processing unit is specifically configured to:

[0036] Calculate the difference between the matrix information of the multi-head attention module of the initial student model and the matrix information of the multi-head attention module of the teacher model to obtain the first matrix sub-loss value;

[0037] Calculate the difference between the matrix information of the FNN module of the initial student model and the matrix information of the FNN module of the teacher model to obtain the second matrix sub-loss value;

[0038] Determine the second loss information based on the first matrix sub-loss value and the second matrix sub-loss value.

[0039] In some other alternative examples, the matrix information of the multi-head attention module of the teacher model includes the first self-attention matrix of each of the h attention heads, and the self-attention distribution information of the initial student model includes the second self-attention matrix of each attention head, where h is an integer greater than or equal to 1; the processing unit is specifically configured to:

[0040] Calculate the difference between the matrix information of the multi-head attention module of the initial student model and the matrix information of the multi-head attention module of the teacher model to obtain the first matrix sub-loss value, including:

[0041] Calculate the mean square error between the second self-attention matrix of each attention head and the first self-attention matrix of each attention head to obtain h first mean square error values;

[0042] Calculate the sum of the h first mean square error values to obtain the total first mean square error;

[0043] Calculate the quotient of the total first mean square error and the number of attention heads to obtain the first matrix sub-loss value.

[0044] In some other alternative examples, the matrix information of the FNN module of the teacher model includes the mapping matrix of the first FNN module of each of the h attention heads, and the self-attention distribution information of the initial student model includes the mapping matrix of the second FNN module of each attention head, where h is an integer greater than or equal to 1; the processing unit is specifically configured to:

[0045] Calculate the mean square error between the mapping matrix of the second FNN module of each attention head and the mapping matrix of the first FNN module of each attention head to obtain h second mean square error values;

[0046] Calculate the sum of the h second mean square error values to obtain the total second mean square error;

[0047] Calculate the quotient of the total second mean square error and the number of attention heads to obtain the second matrix sub-loss value.

[0048] In some other alternative examples, the processing unit is specifically configured to: calculate the cross-entropy loss value between the first prediction information and the second prediction information to obtain the third loss information.

[0049] In some other alternative examples, the acquisition unit is specifically further configured to: obtain the weight matrix of the student model to be optimized before the initial student model processes the embedded feature vector to obtain the second inter-layer information, the second matrix information, and the second prediction information;

[0050] The processing unit is specifically configured to:

[0051] Perform singular value decomposition on the weight matrix of the student model to be optimized to obtain a first matrix and a second matrix. The number of column elements of the first matrix and the number of row elements of the second matrix are both the same as the rank of the weight matrix of the student model to be optimized;

[0052] Replace the weight matrix of the student model to be optimized with the first matrix and the second matrix to obtain an initial student model.

[0053] A third aspect of the embodiments of the present application provides a computer device, including: a memory, a memory. The memory is used to store program instructions. The processor is used to execute the program instructions in the memory to execute the method corresponding to the implementation manner of any of the above aspects.

[0054] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When it runs on a computer, it causes the computer to execute the method corresponding to the implementation manner of any of the above aspects.

[0055] A fifth aspect of the embodiments of the present application provides a computer program product containing instructions. When it runs on a computer or a processor, it causes the computer or the processor to execute the method corresponding to the implementation manner of any of the above to execute the above.

[0056] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

[0057] In the embodiments of the present application, the weight matrix of the student model to be optimized is processed by using singular value decomposition, so that the obtained initial student model can align the number of layers with the teacher model while reducing the number of parameters, eliminating the gap generated by the mapping between different layers. Further, the present application not only determines the matrix loss by introducing the difference between the multi-head attention model and the intra-layer matrix information (such as the first matrix information and the second matrix information) in the feed-forward neural network FNN module, and guides the training of the initial student model from a finer-grained perspective of the intra-layer matrix; but also introduces the difference between the self-attention distribution and the inter-layer information (such as the first inter-layer information and the second inter-layer information) of the hidden layer state to determine the inter-layer loss, so as to realize combining the fine-grained knowledge of the intra-layer matrix, the inter-layer information and the model prediction result to determine the target loss, improve the consistency of the distribution between the teacher model and the initial student model, and further improve the performance effect of the subsequent trained target student model and the processing quality of the subsequent downstream tasks. Description of the Drawings

[0058] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0059] Figure 1 Fig. shows a schematic diagram of an implementation environment provided by the present application;

[0060] Figure 2 Fig. shows a schematic diagram of a framework process of model training provided by an embodiment of the present application;

[0061] Figure 3 Fig. shows a flowchart of a method for model training provided by an embodiment of the present application;

[0062] Figure 4 Fig. shows a schematic diagram of singular value decomposition provided by an embodiment of the present application;

[0063] Figure 5 Fig. shows a schematic diagram of a framework for calculating the first loss information provided by an embodiment of the present application;

[0064] Figure 6 Fig. shows a schematic diagram of a framework for calculating the second loss information provided by an embodiment of the present application;

[0065] Figure 7 Fig. shows a schematic diagram of a framework for the loss information of the target of disadvantages provided by an embodiment of the present application;

[0066] Figure 8 Fig. shows a schematic flowchart of a method for data processing provided by an embodiment of the present application;

[0067] Figure 9 Fig. shows a schematic diagram of an embodiment of a model training device provided by an embodiment of the present application;

[0068] Figure 10 Fig. shows a schematic diagram of an embodiment of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0069] The embodiments of the present application provide a method, device, equipment, storage medium and program product for model training, which are used to improve the performance effect of the target student model on the basis of reducing the number of parameters of the student model and improve the processing quality of subsequent downstream tasks.

[0070] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0071] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0072] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the implementation of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0073] Model compression refers to reducing the size, computational workload, and storage requirements of a model through a series of technical means while maintaining model performance as much as possible. This is particularly important for deploying deep learning models on resource-constrained devices (such as mobile devices and embedded systems). Model compression technology aims to address the challenges of storage, computing, and energy consumption of large models, allowing models to run efficiently on edge devices.

[0074] In the related model compression scheme, it is proposed to transfer the knowledge of large and complex models to smaller neural network structures through knowledge distillation technology, so as to realize the knowledge distillation of each layer component of the Transformer structure through a layered distillation strategy. The described knowledge distillation (KD) is a model compression technology that aims to transfer the knowledge of a large and complex model (called the "teacher model") to a small and lightweight model (called the "student model"). Through knowledge distillation, the student model can achieve performance similar to that of the teacher model while maintaining a smaller scale.

[0075] However, the knowledge information of each layer extracted by traditional distillation methods is coarse-grained. For example, the self-attention weight is the interaction result of the query (Q) matrix and the key (K) matrix. This globality causes the information to be compressed into weights of a fixed dimension, losing local details. Moreover, to reduce the number of parameters in the student model, current distillation methods will directly remove some layers in the student model. For example, the number of layers of the transformer structure in the student model is directly reduced from 12 layers to 6 layers. This easily leads to the inability of the student model to align with the knowledge distribution of the teacher model due to the mismatch of the number of layers, resulting in poor performance of the distilled target student model and inability to handle subsequent downstream tasks with high quality.

[0076] To solve the above-mentioned technical problems, an embodiment of the present application provides a method for model training. Exemplarily, an embodiment of the present application also provides a method for data processing. The method for model training and the method for data processing can be applied to one or more of the following scenarios. For example, it can be applied to the scenario of lightweighting online service models. For example, for various service models currently on the line, the target student model distilled by the present application can save resources and speed up inference while maintaining the performance of the original service model. It can also be applied to the scenario of compressing large language models. For example, the target student model of the present application can be deployed on edge devices, so as to directly deploy and run large language models (LLMs) on edge devices, providing users with richer productivity and lower computing power cost limitations. For another example, it can also be applied to terminal devices such as mobile devices. Through the method for model training and the method for data processing of the present application, image recognition, speech recognition, text recognition and other applications deployed in the device can be processed, achieving faster inference speed. For another example, it can also be applied to scenarios such as Internet of Things devices, drones, and robots. Through the method for model training and the method for data processing of the present application, the energy consumption of the device can be reduced and the battery life of the device can be extended. For another example, it can also be applicable to scenarios such as edge computing and large-scale deployment, which are not specifically limited in the present application. Through the method for model training provided by the present application, the fine-grained knowledge of the intra-layer matrix, the inter-layer information, and the model prediction results can be combined to determine the target loss, so as to improve the consistency of the distribution between the teacher model and the initial student model, and further improve the performance of the subsequently trained target student model and the processing quality of subsequent downstream tasks.

[0077] Exemplarily, Figure 1 shows a schematic diagram of an implementation environment provided by the present application.

[0078] As Figure 1As shown, in this implementation environment, it may include a model training device and a model using device. Among them, the model training device can be a computer device such as a terminal device, a server, etc.

[0079] In the model training stage of this application, after obtaining the object information, the model training device can extract the embedded feature vector of the object information. Subsequently, the model training device calls the teacher model to complete the processing of the embedded feature vector, obtaining relevant inter-layer information, matrix information, and prediction information. In addition, the model training device also needs to perform singular value decomposition on the weight matrix of the student model to be optimized to obtain the initial student model. It should be noted that in this initial student model and the teacher model, the number of layers of the two models is the same. In this way, the model training device also calls the initial student model to process the embedded feature vector, thereby obtaining the corresponding inter-layer information, matrix information, and prediction information. Thus, by using the differences in inter-layer information, matrix information, and prediction information between the teacher model and the initial student model, the target loss difference of the initial student model is determined, and the adjustment process of the model parameters of the initial student model is completed, thereby training the target student model.

[0080] In addition, the obtained target student model after training can be deployed in the model using device. The model using device can be a terminal device or a server or other devices, which is not specifically limited in the embodiments of this application. When a task needs to be processed, after obtaining the target object data, the model using device can automatically implement the processing of the downstream task related to the target object data by calling the target student model, and obtain the processing result of the relevant downstream task. For example, the described downstream tasks include but are not limited to object classification, object recognition, etc.; and for another example, object recognition includes but is not limited to image recognition, text recognition, speech recognition, etc., which is not specifically limited in this application.

[0081] It should be noted that the described terminal device may include but is not limited to a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, a medical device, a multimedia playback device, etc. The described server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms, etc., which is not specifically limited in this application.

[0082] It should be understood that in actual applications, the model training method of this application can also be applicable to other thanFigure 1 Other usage scenarios outside the illustrated implementation environment are not specifically defined and described in the embodiments of this application.

[0083] Combined with Figure 1 , for how the model training device described in Figure 1 trains to obtain the target student model, Figure 2 FIG. shows a schematic framework flow diagram of model training provided by an embodiment of this application. As Figure 2 shown, during the process of training the model, the model training device needs to call the teacher model and the initial student model. Among them, the initial student model is obtained by performing singular value decomposition (SDV) on the weight matrix to be optimized. And, in the initial student model and the teacher model, the number of layers of the Transformer structure of the two models is the same, for example, both are L layers, and the value of L is an integer greater than or equal to 1.

[0084] In this way, after the model training device obtains the object information, it can extract the embedded feature vector of the object information. Thus, the embedded feature vector is used as the input of the teacher model and the input of the initial student model respectively, so as to process the embedded feature vector through the teacher model and the initial school model respectively.

[0085] For the teacher model side, the model training device calls the teacher model to process the embedded feature vector, and obtains relevant first inter-layer information, first matrix information, and first prediction information. Through the first inter-layer information, the self-attention distribution and hidden layer state of the teacher model can be reflected. For example, the self-attention distribution in the teacher model can be reflected by the interaction result between the query matrix and the key matrix in the teacher model, and the hidden layer state can be reflected by the hidden layer information of each layer of the Transformer structure in the teacher model. The described first matrix information can describe the matrix situation of the multi-head attention module and the matrix situation of the feedforward neural network (FNN) module in the teacher model. For example, the matrix situation of the multi-head attention model includes, but is not limited to, matrix situations such as the Q matrix and the value (V) matrix. The matrix situation of the mentioned FNN module also includes, but is not limited to, up projections and down projections, etc. The mentioned first prediction information is used to describe the model prediction situation of the teacher model for the object information, such as the classification probability determined by the classifier in the output layer, etc., which is not limited in this application.

[0086] On the side of the initial student model, the model training device invokes the initial student model to process the embedded feature vector, thereby obtaining corresponding second inter-layer information, second matrix information, and second prediction information. The described second inter-layer information is used to describe the self-attention distribution and hidden layer state of the initial student model. For example, the self-attention distribution in the initial student model can be reflected based on the interaction result between the query matrix and the key matrix in the initial student model, and the hidden layer state can be reflected by the hidden layer information of each layer of the Transformer structure in the initial student model. The second matrix information is used to describe the matrix situation of the multi-head attention module and the FNN module of the initial student model. For instance, the matrix situation of the multi-head attention model includes but is not limited to matrices such as Q, K, and V, and the matrix situation of the mentioned FNN module also includes but is not limited to the dimension-increasing mapping matrix and the dimension-reducing mapping matrix, etc. The mentioned second prediction information is used to describe the model prediction situation of the initial student model for the object information, such as the classification probability determined by the classifier in the output layer, etc., which is not limited in this application.

[0087] Thus, by combining the first inter-layer information, the first matrix information, and the first prediction information, the first loss information is determined using the inter-layer information difference between the first inter-layer information and the second inter-layer information; similarly, the second loss information is determined using the matrix information difference between the first matrix information and the second matrix information; and further, the third loss information is calculated using the prediction information difference between the first prediction information and the second prediction information. In this way, the target loss difference of the initial student model is determined by one of the first loss information, the second loss information, and the third loss information, thereby completing the adjustment process of the model parameters of the initial student model, and thus training the target student model.

[0088] It should be noted that the teacher model and the initial student model described above Figure 1 and Figure 2 can both be a Transformer structure including L layers, etc., which is not limited in this application. Moreover, each layer of the Transformer structure at least includes the aforementioned multi-head attention (MHA) module and FNN module.

[0089] Next, with reference to the accompanying drawings, a method for model training provided by an embodiment of the present application will be introduced first. Figure 3 A flowchart showing a method for model training provided by an embodiment of the present application is shown. As Figure 3 shown, the method for model training may include the following steps:

[0090] 301. Obtain object information and extract the embedded feature vector of the object information.

[0091] In one or more embodiments, object information can be understood as object data to be processed. For example, it can be object data to be classified, or object data of a type to be recognized, etc. The described object information includes, but is not limited to, text, images, videos, voices, etc., and is not limited in this application.

[0092] After obtaining the object information, feature extraction processing can be performed on the object information to extract the embedded feature vector of the object information. For example, a feature extraction model such as a CNN can be called, and the object information can be used as the input of the feature extraction model, so as to perform feature extraction processing on the object information through the feature extraction model to obtain the corresponding embedded feature vector.

[0093] 302. Process the embedded feature vector based on the teacher model to obtain the first inter-layer information, the first matrix information, and the first prediction information.

[0094] In one or more embodiments, after the embedded feature vector of the object information is extracted, the teacher model is called to process the embedded feature vector, so as to obtain the corresponding first inter-layer information, the first matrix information, and the first prediction information.

[0095] The described first inter-layer information can reflect the self-attention distribution and hidden layer state of the teacher model. Exemplarily, in the first inter-layer information of this application, it includes the self-attention distribution information and hidden layer information of the teacher model. Through the self-attention distribution information of the teacher model, the self-attention distribution of the teacher model can be characterized. Similarly, through the hidden layer information of the teacher model, the hidden layer state of the teacher model can be characterized.

[0096] As a schematic description, regarding how to calculate the self-attention distribution information of the teacher model through the teacher model, it can be done by using the embedded feature vector as the input of the multi-head attention module of the teacher model, calculating the similarity between the query matrix and the key matrix through each attention head in the multi-head attention module, so as to obtain the attention weight distribution of each attention head. In this way, then extract the attention weight matrix corresponding to the attention head from the attention weight distributions of each attention head, and analyze the self-attention weights in the attention weight matrices of each attention head, so as to obtain the first self-attention distribution information of each attention head. Thus, by integrating the first self-attention distribution information of these multiple attention heads, the self-attention distribution information of the entire teacher model can be obtained.

[0097] Regarding how to calculate the hidden layer information of the teacher model through the teacher model, it can be obtained by taking the embedded feature vector as the input of the Transformer structure of the teacher model and performing a linear transformation on the embedded feature vector through the weight matrix and bias term information in the teacher model, so as to obtain the hidden layer information of the teacher model. For example, the hidden layer information of the teacher model satisfies the following formula, such as Among them, represents the hidden layer information of the teacher model, W1 and W2 represent the weight matrix of the teacher model, x represents the embedded feature vector, and b1 T , b2 T represents the bias term information of the teacher model.

[0098] In addition, the described first matrix information is used to describe the matrix situation of the multi-head attention module of the teacher model and the matrix situation of the feed-forward neural network FNN module. Exemplarily, in the first matrix information of this application, it includes the matrix information of the multi-head attention module of the teacher model and the matrix information of the FNN module. Through the matrix information of the multi-head attention module of the teacher model, the matrix situation of the multi-head attention module of the teacher model can be characterized. Similarly, through the matrix information of the FNN module of the teacher model, the matrix situation of the FNN module of the teacher model can be characterized.

[0099] As another illustrative description, the first self-attention matrix of each attention head in the teacher model can be directly extracted from the parameters of the teacher model, such as including but not limited to one or more of the query matrix, key matrix, and value matrix. In this way, by combining the first self-attention matrices of multiple attention heads, the matrix information of the multi-head attention module of the teacher model can be obtained. Similarly, the mapping matrix of the first FNN module of each attention head in the teacher model can also be directly extracted from the parameters of the teacher model, such as including but not limited to one or more of the dimension-increasing mapping matrix and the dimension-reducing mapping matrix. Thus, by combining the mapping matrices of the first FNN modules of these multiple attention heads, the matrix information of the FNN module of the teacher model can be obtained. It should be noted that the described first FNN module can be understood as the FNN module included in each attention head of the teacher model.

[0100] In addition, the mentioned first prediction information is used to describe the model prediction situation of the teacher model for the object information. As a schematic description, after the embedded feature vector is input into the teacher model, it can first be processed by the multi-head attention module in the first-layer Transformer structure of the teacher model, and then the information output by the multi-head attention module is combined with the embedded feature vector as the input of the FNN module in the first-layer Transformer structure, and is processed by the FNN module of the first-layer Transformer structure, and used as the input of the next-layer Transformer structure. After being processed by the L-layer Transformer structure in this way, the first prediction information of the teacher model can be output. More specifically, the softmax calculation can be performed on the hidden layer information output by the last-layer Transformer structure, so as to convert and obtain the corresponding prediction probability, that is, the first prediction information is determined, such as Z = softmax(H), where H is the hidden layer information output by the last-layer Transformer structure.

[0101] 303. Process the embedded feature vector based on the initial student model to obtain the second inter-layer information, the second matrix information, and the second prediction information. The initial student model is a neural network model obtained by performing singular value decomposition on the weight matrix of the student model to be optimized.

[0102] In one or more embodiments, before using the initial student model to process the embedded feature vector, the present application also needs to perform singular value decomposition on the weight matrix of the student model, so as to ensure that the optimized initial student model has the same number of Transformer structure layers as the Transformer structure layers in the teacher model while significantly reducing the number of parameters, which is convenient for subsequent alignment processing of the inter-layer information. Exemplarily, specifically, before using the initial student model to process the embedded feature vector, first obtain the weight matrix of the student model to be optimized. Then, perform singular value decomposition on the weight matrix of the student model to be optimized to obtain a first matrix and a second matrix. The number of column elements of the described first matrix and the number of row elements of the second matrix are both the same as the rank of the weight matrix of the student model to be optimized. In this way, the weight matrix of the student model to be optimized is replaced with the first matrix and the second matrix to obtain the initial student model.

[0103] For example, Figure 4 shows a schematic diagram of singular value decomposition provided by the present application. Combining the foregoing Figure 2 shown framework schematic diagram, as Figure 4 shown, if the weight matrix of the student model to be optimized is W S , its W S ∈R n×m, after singular value decomposition, the equivalent form of this W can be obtained S is, such as W S ≈ A × B. Among them, W S can be understood as the weight matrix of all parameters in the multi-head attention module and the FNN module. A ∈ R n×r , B ∈ R r×m , and r is the rank of the weight matrix W S . Through this low-rank decomposition, the parameters of the student model can be reduced from n × m to k(n + m), greatly reducing the number of parameters of the student model. For example, the dimension of A is n × r, the dimension of B is r × m, and the value of r is less than the value of n. Suppose the general setting is r = 256, n = m = 768. In this way, after the above-mentioned singular value decomposition, the number of parameters of the original weight matrix W S is n × m = 768 × 768, while the number of parameters of A × B is n × r + r × m = 768 × 256 + 256 × 768. It should be noted that for different compression ratios, it can be achieved by changing the weight matrix W S to optimize and obtain different initial student models. In addition, from Figure 4 it can also be seen that after singular value decomposition, the number of layers of the Transformer structure of the initial student model can also be kept the same as that of the Transformer structure of the teacher model, such as 12 layers, 18 layers, etc., which is not limited in this application.

[0104] In this way, after optimizing and obtaining the initial student model, the extracted embedded feature vector can be used as the input of the initial student model. Through the initial student model to process the embedded feature vector, the corresponding second inter-layer information, second matrix information, and second prediction information can be obtained.

[0105] It should be noted that the second inter-layer information is used to describe the self-attention distribution and hidden layer state of the initial student model. Exemplarily, in the second inter-layer information of this application, it includes the self-attention distribution information and hidden layer information of the initial student model. Through the self-attention distribution information of the initial student model, the self-attention distribution of the initial student model can be characterized. Similarly, through the hidden layer information of the initial student model, the hidden layer state of the initial student model can be characterized.

[0106] As a schematic description, regarding how to calculate the self-attention distribution information of the initial student model through the initial student model, it can be achieved by using the embedded feature vector as the input of the multi-head attention module of the initial student model, calculating the similarity between the query matrix and the key matrix through each attention head in the multi-head attention module, so as to obtain the attention weight distribution of each attention head. In this way, the attention weight matrix corresponding to each attention head is extracted from the attention weight distributions of each attention head, and the self-attention weights in the attention weight matrices of each attention head are analyzed, so as to obtain the second self-attention distribution information of each attention head. Thus, by integrating the second self-attention distribution information of these multiple attention heads, the self-attention distribution information of the entire initial student model can be obtained.

[0107] Regarding how to calculate the hidden layer information of the initial student model through the initial student model, it can be achieved by using the embedded feature vector as the input of the Transformer structure of the initial student model, and performing a linear transformation on the embedded feature vector through the weight matrix and bias term information in the initial student model, so as to obtain the hidden layer information of the initial student model. For example, the hidden layer information of the initial student model satisfies the following formula, such as Among them, represents the hidden layer information of the initial student model, A1×B1 and A2×B2 represent the weight matrices of the initial student model, x represents the embedded feature vector, and b1 S , b2 S represent the bias term information of the initial student model.

[0108] In addition, the second matrix information is used to describe the matrix situation of the multi-head attention module and the matrix situation of the FNN module of the initial student model. Exemplarily, in the second matrix information of this application, it includes the matrix information of the multi-head attention module of the initial student model and the matrix information of the FNN module. Through the matrix information of the multi-head attention module of the initial student model, the matrix situation of the multi-head attention module of the initial student model can be characterized. Similarly, through the matrix information of the FNN module of the initial student model, the matrix situation of the FNN module of the initial student model can be characterized.

[0109] As another illustrative description, from the parameters of the initial student model, the second self-attention matrix of each attention head in the initial student model can be directly extracted, such as including but not limited to one or more of the query matrix, key matrix, and value matrix. In this way, by combining the second self-attention matrices of multiple attention heads, the matrix information of the multi-head attention module of the initial student model can be obtained. Similarly, from the parameters of the initial student model, the mapping matrix of the second FNN module of each attention head in the initial student model can be directly extracted, such as including but not limited to one or more of the dimension-increasing mapping matrix and dimension-decreasing mapping matrix. Thus, by integrating the mapping matrices of the second FNN modules of these multiple attention heads, the matrix information of the FNN module of the initial student model can be obtained. It should be noted that the described second FNN module can be understood as the FNN module included in each attention head in the initial student model.

[0110] In addition, the second prediction information is used to describe the model prediction situation of the initial student model for the object information. As an illustrative description, after the embedded feature vector is input into the initial student model, it can be processed successively through the multi-head attention module in the first-layer Transformer structure of the initial student model, and then the information output by the multi-head attention module is combined with the embedded feature vector as the input of the FNN module in the second-layer Transformer structure, and processed through the FNN module of the first-layer Transformer structure, and used as the input of the next-layer Transformer structure. After being processed through L-layer Transformer structures in this way, the second prediction information of the initial student model can be output. More specifically, the softmax calculation can be performed on the hidden layer information output by the last-layer Transformer structure, so as to convert and obtain the corresponding prediction probability, that is, the second prediction information is determined.

[0111] 304. Calculate the first loss information based on the first inter-layer information and the second inter-layer information, calculate the second loss information based on the first matrix information and the second matrix information, and calculate the third loss information based on the first prediction information and the second prediction information.

[0112] In one or more embodiments, after obtaining the first inter-layer information through the teacher model and the second inter-layer information through the initial student model, the difference between the first inter-layer information and the second inter-layer information can be calculated to obtain the first loss information. Through this first loss information, the difference between the intermediate layer information of the teacher model and the intermediate layer information of the initial student model can be understood. Similarly, after obtaining the first matrix information through the teacher model and the second matrix information through the initial student model, the difference between the first matrix information and the second matrix information can be calculated to obtain the second loss information. Through this second loss information, the finer-grained difference between the matrix information of the teacher model and the matrix information of the initial student model can be understood. In addition, after obtaining the first prediction information through the teacher model and the second prediction information through the initial student model, the difference between the first prediction information and the second prediction information can be calculated to obtain the third loss information. Through this third loss information, the difference between the information prediction of the teacher model and the information prediction of the initial student model can be understood.

[0113] 305. Update the model parameters of the initial student model based on the target loss information to obtain a target student model, where the target student model is used to process downstream tasks related to object information, and the target loss information is one or more of the first loss information, the second loss information, and the third loss information.

[0114] In one or more embodiments, after calculating the first loss information, the second loss information, and the third loss information, the target loss information can be determined from the first loss information, the second loss information, and the third loss information, and then the model parameters of the initial student model can be updated and adjusted using the target loss information, thereby training to obtain the target student model. In this way, after adjusting to obtain the target student model, during the model usage phase, the target student model can be called to process downstream tasks related to the object information.

[0115] In the embodiments of the present application, the weight matrix of the student model to be optimized is processed by singular value decomposition, so that the obtained initial student model can align the number of layers with that of the teacher model while reducing the number of parameters, eliminating the gap generated by the mapping between different layers. Further, the present application not only determines the matrix loss by introducing the difference between the multi-head attention model and the intra-layer matrix information (such as the first matrix information and the second matrix information) in the feed-forward neural network FNN module, and guides the training of the initial student model from a finer-grained perspective of the intra-layer matrix; but also introduces the difference between the self-attention distribution and the inter-layer information such as the hidden layer state (such as the first inter-layer information and the second inter-layer information) to determine the inter-layer loss, so as to realize combining the fine-grained knowledge of the intra-layer matrix, the inter-layer information and the model prediction result to determine the target loss, improve the distribution consistency between the teacher model and the initial student model, and further improve the performance effect of the target student model obtained by subsequent training, and improve the processing quality of subsequent downstream tasks.

[0116] In some other alternative embodiments, based on one or more of the embodiments described above, regarding how to calculate the first loss information based on the first inter-layer information and the second inter-layer information in step 304, the calculation process can be understood with reference to the following Figure 3 framework schematic diagram described. Figure 5

[0117] From the content described in the above steps 302 and 303, it can be seen that: the first inter-layer information includes the self-attention distribution information and the hidden layer information of the teacher model, and the second inter-layer information includes the self-attention distribution information and the hidden layer information of the initial student model. Moreover, through the self-attention distribution information, rich hierarchical language information can be captured.

[0118] Based on this, as Figure 5 shown, in the process of calculating the first loss information, the first inter-layer sub-loss value can be obtained by calculating the difference between the self-attention distribution information of the teacher model and the self-attention distribution information of the initial student model. In this way, different from the related solutions that align the intermediate features of different layers between the teacher model and the student model, the present application needs to eliminate the gap generated by the mapping between different layers through singular value decomposition. On this basis, by analyzing the difference between the self-attention distributions of the teacher model and the initial student model, the deficiencies of the initial student model in feature attention (such as ignoring key regions, over-focusing on noise, etc.) can be identified, and then the attention mechanism of the initial student model can be adjusted to make it closer to the teacher model.

[0119] ​As a schematic description, in the self-attention distribution information of the teacher model, it also includes the first self-attention distribution information of each of the h attention heads. The described h takes an integer value greater than or equal to 1. Similarly, in the self-attention distribution information of the initial student model, it also includes the second self-attention distribution information of each of these h attention heads. Thus, in the process of calculating the first inter-layer sub-loss value, the KL divergence between the first self-attention distribution information of each attention head and the first self-attention distribution information of each attention head can be calculated first to obtain h KL divergence values. Subsequently, the sum of the h KL divergence values is calculated to obtain the total divergence value. In this way, by solving the quotient between the total divergence value and the number of attention heads, the first inter-layer sub-loss value is obtained. It should be noted that the involved first inter-layer sub-loss value can be used to describe the distribution difference between the self-attention distribution of the teacher model and the self-attention distribution of the initial student model.

[0120] For example, the first inter-layer sub-loss value can satisfy the following formula, that is:

[0121] Among them, represents the second self-attention distribution information of the a-th attention head, that is, the self-attention distribution information of the a-th attention head in the initial student model; represents the first self-attention distribution information of the a-th attention head, that is, the self-attention distribution information of the a-th attention head in the teacher model; represents the second self-attention distribution information of the a-th attention head and the first self-attention distribution information The KL divergence between them, A h represents the number of attention heads, L attn represents the first inter-layer sub-loss value.

[0122] In addition, in addition to paying attention to the information differences of the multi-head attention model, the present application can also extract knowledge from the output of the Transformer structure. Exemplarily, in the process of calculating the first loss information, it is also necessary to calculate the difference between the hidden layer information of the teacher model and the hidden layer information of the initial student model to obtain the second inter-layer sub-loss value. As a schematic description, specifically, the mean square error between the hidden layer information of the initial student model and the hidden layer information of the teacher model can be calculated, and then this mean square error is used as the second inter-layer sub-loss value. It should be noted that the involved second inter-layer sub-loss value can describe the difference between the hidden layer state of the teacher model and the hidden layer state of the initial student model.

[0123] For example, the second inter-layer sub-loss value satisfies the following formula, that is: Among them, h represents the hidden layer information of the initial student model, d represents the hidden dimension of the initial student model, and |x| represents the length of the object information; h represents the hidden layer information of the teacher model, d represents the hidden dimension of the teacher model; MSE() represents the mean squared error function, L hidn represents the second inter-layer sub-loss value.

[0124] Thus, after calculating the first inter-layer sub-loss value and the second inter-layer sub-loss value, the first loss information can be determined using the first inter-layer sub-loss value and the second inter-layer sub-loss value. For example, by calculating the sum of the first inter-layer sub-loss value and the second inter-layer sub-loss value, and then taking the total inter-layer sub-loss value as the first loss information. For example, the first loss information Loss1 = L attn + L hidn . Or, alternatively, the first inter-layer sub-loss value and the second inter-layer sub-loss value can be weighted and summed, and then taking the total inter-layer sub-loss value obtained from the weighted sum as the first loss information, which is not limited in this application specifically.

[0125] In some other alternative embodiments, based on one or more of the embodiments described above, for how to calculate the second loss information based on the first matrix information and the second matrix information in step 304, the calculation process can be understood with reference to the following Figure 3 framework schematic diagram described. Figure 6

[0126] As can be seen from the content described in the above steps 302 and 303, since the first matrix information includes the matrix information of the multi-head attention module and the FNN module of the teacher model, and moreover, the second matrix information includes the matrix information of the multi-head attention module and the FNN module of the initial student model.

[0127] Based on this, in addition to considering the information differences output between layers, this application can also introduce the intra-layer matrix information in the self-attention module and the FNN module to guide the training of the initial student model. As Figure 6 shown, in the process of calculating the second loss information, the first matrix sub-loss value can be obtained by calculating the difference between the matrix information of the multi-head attention module of the teacher model and the matrix information of the multi-head attention module of the initial student model. In this way, the multi-head attention module enables the teacher model and the initial student model to capture different hierarchical information in the input embedded feature vectors. Thus, further analyzing the difference between the matrix information of the multi-head attention modules of the teacher model and the initial student model can guide the initial student model to focus more on the key features in the object information, improve the representation ability of the key features, and improve the speed and efficiency of knowledge distillation.

[0128] As a schematic description, in the matrix information of the multi-head attention module of the teacher model, it also includes the first self-attention matrix of each of the h attention heads. For example, the first self-attention matrix may include, but is not limited to, the query matrix, key matrix, value matrix, etc. involved in the transformer structures of each layer in the teacher model. Similarly, in the matrix information of the multi-head attention module of the initial student model, it also includes the second self-attention matrix of each of these h attention heads. For example, the second self-attention matrix may include, but is not limited to, the query matrix, key matrix, value matrix, etc. involved in the transformer structures of each layer in the initial student model. Thus, in the process of calculating the first matrix sub-loss value, the mean square error between the first self-attention matrix of each attention head and the second self-attention matrix of each attention head can be calculated first, obtaining h first mean square error values. Subsequently, the sum of these h first mean square error values is calculated to obtain the total first mean square error. In this way, by solving the quotient between the total first mean square error and the number of attention heads, the first matrix sub-loss value is obtained. It should be noted that the involved first matrix sub-loss value can be used to describe the difference between the matrix situation of the multi-head attention module of the teacher model and the matrix situation of the multi-head attention module of the initial student model. Through the above method, each attention head may focus on different aspects of the input. By analyzing the differences between the matrix information of the self-attention modules of each attention head in the teacher model and the initial student model, it is possible to achieve fine-grained alignment of the matrices of each attention head in the teacher model and the initial student model, enabling the student model to gradually approach the teacher model layer by layer; moreover, it can also make the initial student model pay more attention to the key features of the input information, improve the feature expression ability of the initial student model, and enhance the efficiency of knowledge distillation.

[0129] For example, the first matrix sub-loss value can satisfy the following formula, that is:

[0130]

[0131] Where represents the second self-attention matrix of the a-th attention head, that is, the self-attention matrix of the a-th attention head in the initial student model, and d represents the hidden dimension; represents the first self-attention matrix of the a-th attention head, that is, the self-attention matrix of the a-th attention head in the teacher model; represents the second self-attention matrix of the a-th attention head and the first self-attention matrix the mean square error between them, A h represents the number of attention heads, and L MHA represents the first matrix sub-loss value.

[0132] In addition, the FNN module in the Transformer structure further processes the output of the multi-head attention (MHA) module through a series of linear transformations. In this way, in addition to considering the matrix information differences involved in the multi-head attention module, the present application can also introduce the matrix information differences in the FNN module to guide the training of the initial student model. Based on this, as Figure 6 shown, in the process of calculating the second loss information, it is also necessary to obtain the second matrix sub-loss value by calculating the difference between the matrix information of the FNN module of the teacher model and the matrix information of the FNN module of the initial student model.

[0133] As a schematic description, in the matrix information of the FNN module of the teacher model, it also includes the mapping matrices of the first FNN modules of each of the h attention heads. For example, the mapping matrices of the first FNN modules include, but are not limited to, the dimension-increasing mapping matrix and the dimension-decreasing mapping matrix in the teacher model mentioned above. Similarly, in the matrix information of the FNN module of the initial student model, it also includes the mapping matrices of the second FNN modules of each of these h attention heads. For example, the mapping matrices of the second FNN modules include, but are not limited to, the dimension-increasing mapping matrix and the dimension-decreasing mapping matrix in the initial student model mentioned above. In this way, in the process of calculating the first matrix sub-loss value, the mean square error between the mapping matrix of the first FNN module of each attention head and the mapping matrix of the second FNN module of each attention head can be calculated first to obtain h second mean square error values. Subsequently, the sum of these h second mean square error values is calculated to obtain the total second mean square error. In this way, by solving the quotient between the total second mean square error and the number of attention heads, the second matrix sub-loss value is obtained. It should be noted that the involved second matrix sub-loss value can be used to describe the difference between the matrix situation of the FNN module of the teacher model and the matrix situation of the FNN module of the initial student model. Through the above method, each attention head may focus on different aspects of the input. By analyzing the differences between the matrix information of the FNN modules of each attention head in the teacher model and the initial student model, the initial student model can be guided to learn more complex feature representations, improving its generalization ability; moreover, it can quickly align the weight matrices of each layer of the FNN model, effectively transfer knowledge such as feature transformation and non-linear mapping in the teacher model to the initial student model, enabling the initial student model to learn a reasonable weight configuration faster and accelerating the model convergence speed.

[0134] For example, the second matrix sub-loss value can satisfy the following formula, that is:

[0135] where represents the mapping matrix of the second FNN module of the a-th attention head, that is, the mapping matrix of the a-th attention head in the initial student model, d iRepresents the feed-forward size of the model; Represents the mapping matrix of the first FNN module of the a-th attention head, that is, the mapping matrix of the a-th attention head in the teacher model; Represents the mapping matrix of the a-th attention head And The mean square error between, A h Represents the number of attention heads, L FNN Represents the second matrix sub-loss value.

[0136] Thus, after calculating the first matrix sub-loss value and the second matrix sub-loss value, the second loss information can be determined using the first matrix sub-loss value and the second matrix sub-loss value. For example, by calculating the sum of the first matrix sub-loss value and the second matrix sub-loss value, and then taking the sum of the matrix sub-loss values as the second loss information. For example, the second loss information Loss2 = L MHA + L FNN . Or, alternatively, the first matrix sub-loss value and the second matrix sub-loss value can be weighted and summed, and then the sum of the matrix sub-loss values obtained by the weighted sum is used as the second loss information, which is not limited in this application specifically. That is to say, through the second loss information, the training loss of the model in terms of the matrix can be understood.

[0137] In some other alternative embodiments, regarding how to calculate the third loss information based on the first prediction information and the second prediction information in step 304, it can be calculated by calculating the cross-entropy loss value between the first prediction information and the second prediction information, and then taking the cross-entropy loss value as the third loss information. As a schematic description, a temperature value can also be introduced to calculate the cross-entropy loss value between the first prediction information and the second prediction information. Thus, the cross-entropy loss value is used as the third loss information.

[0138] For example, the third loss information can satisfy the following formula, that is: Where, L model Represents the third loss information, CE() represents the cross-entropy loss function, t represents the temperature value; Z T Represents the first prediction information, that is, the prediction information output by the teacher model after passing through L layers of transformers; Z S Represents the second prediction information, that is, the prediction information output by the initial student model after passing through L layers of transformers.

[0139] In the above manner, the introduction of the temperature value can adjust the probability distribution of the model output, so that the soft label output by the teacher model contains more relative relationships between categories, rather than just the final prediction results. It also enables the initial student model to learn the teacher model's knowledge, such as the similarity between categories, by minimizing the cross entropy loss between the softened output of the teacher model, thereby improving the performance of the initial student model.

[0140] In some other optional embodiments, in the aforementioned Figure 3 Based on the one or more embodiments described above, the calculation process of how to determine the target loss information in step 305 can refer to the following Figure 7 The framework diagram described is understood.

[0141] like Figure 7 As shown, before using the target loss information to adjust the model parameters of the initial student model, it is also necessary to determine the number of intermediate layers that need to be aligned from the teacher model and the initial student model to obtain the number of distillation layers. In other words, the number of distillation layers can describe the total number of intermediate layers that need to be aligned between the teacher model and the initial student model. In addition, it is also necessary to determine the number of layers of the Transformer structure from the teacher model and the initial student model to obtain the first number of structure layers. In other words, the number of layers of the Transformer structure contained in the teacher model and the initial student model can be understood through the first number of structure layers.

[0142] Furthermore, by comparing the number of layers between the distillation layer and the first structure layer, and then using the comparison result between the distillation layer and the first structure layer, the target loss information is determined from the first loss information, the second loss information and the third loss information. For example, when only a shallow network is involved, the matrix information difference can be used to learn the surface information of the knowledge, so the corresponding matrix loss can also be used to train the model. In addition, when shallow networks and deep networks are involved, only using matrix loss to train the model cannot pay attention to the deeper information involved in the deep network, so the matrix loss can be used to train the model in the shallow network, and the inter-layer loss can be used to train the model in the deep network, so as to achieve distillation goals of different granularities. As an illustrative description, under different comparison results, the target loss information determined from the first loss information, the second loss information and the third loss information is also different, which can be understood by referring to the following cases 1 to 3.

[0143] Case 1: 0≤m<L / N

[0144] Exemplarily, the described m represents the number of distillation layers, L represents the number of the first structural layers (i.e., the number of layers of the Transformer structure), and N is an integer greater than or equal to 2. After obtaining the number of distillation layers and the number of the first structural layers, through comparison, if it is determined that the number of distillation layers falls within the first interval [0, L / N), that is, when the number of distillation layers is greater than or equal to zero and less than the first value, at this time, the second loss information can be selected from the first loss information, the second loss information, and the third loss information as the target loss information. That is to say, when the number of distillation layers is less than 1 / N of the number of layers of the Transformer structure, only the matrix loss can be used to achieve model distillation for a model with a smaller number of layers.

[0145] It should be noted that the described first value can be understood as 1 / N times the number of layers of the Transformer structure (i.e., the number of the first structural layers). The described second loss information is understood with reference to the foregoing Figure 6 content described, and will not be elaborated here.

[0146] Case 2: L / N ≤ m ≤ L

[0147] Exemplarily, parameters such as the described m, L, N, L / N, etc. can be understood with reference to the content described in the foregoing Case 1, and will not be elaborated here. After obtaining the number of distillation layers and the number of the first structural layers, through comparison, if it is determined that the number of distillation layers falls within the second interval [L / N, L], that is, when the number of distillation layers is greater than or equal to the first value and less than or equal to the number of layers of the Transformer structure (i.e., the number of the first structural layers), at this time, the first loss information and the second loss information can be selected from the first loss information, the second loss information, and the third loss information as the target loss information. That is to say, when the number of distillation layers is greater than or equal to 1 / N of the number of layers of the Transformer structure and less than the number of layers of the Transformer structure, it is necessary to use the matrix loss to achieve model distillation for a model with a smaller number of layers, and it is also necessary to use the inter-layer loss to achieve model distillation for a model with a larger number of layers.

[0148] For example, assuming L = 12 and N = 2, then, the matrix loss (i.e., the second loss information Loss2 = L MHA +L FNN ) can be used for model distillation within layers 1 - 6, while the inter-layer loss (i.e., the first loss information Loss1 = L attn +L hidn ) can be used for distillation within layers 6 - 12.

[0149] Case 3: m = L + preset value

[0150] Exemplarily, parameters such as m and L described above can be understood with reference to the content described in the foregoing Case 1, and will not be elaborated here. After obtaining the number of distillation layers and the number of the first structure layers, through comparison, if it is determined that the number of distillation layers is equal to the second value (i.e., the number of the first structure layers + a preset value), at this time, the first loss information, the second loss information, and the third loss information can be directly selected as the target loss information. It should be noted that the described second value is obtained by the sum of the number of the first structure layers and the preset value. The described preset value may take values including but not limited to: 1, 2, etc., and will not be specifically limited in this application.

[0151] For example, assuming L = 12, N = 2, and the preset value is 1, then, matrix loss (i.e., the second loss information Loss2 = L MHA +L FNN ) can be used for model distillation within layers 1 - 6, while inter-layer loss (i.e., the first loss information Loss1 = L attn +L hidn ) can be used for distillation within layers 6 - 12, and then model loss (i.e., the third loss information L model ) can be used for distillation at the 13th layer (i.e., 12 + 1).

[0152] In this way, the loss values used for the distillation losses of each layer in the teacher model and the initial student model are as follows:

[0153]

[0154] It should be noted that the described preset value may take values including but not limited to 1, 2, etc., and the value of N is also not limited.

[0155] In the above - mentioned way, compared with the traditional solution that uses fixed loss values for training in both high - level or low - level layers, in this application, matrix loss is used for distillation within 0 to L / N layers, inter - layer loss is used for distillation within L / N to L layers, and model prediction loss is used for distillation when greater than L layers. On the one hand, it can enable the initial student model to gradually approximate the teacher model and gradually improve the model performance; on the other hand, it can dynamically adjust the losses of different layers according to the training progress of the initial student model to achieve more refined distillation.

[0156] In some optional examples, after training the aforementioned target student model, the target student model can be used to infer downstream tasks related to target object data. Exemplarily, Figure 8 shows a schematic flowchart of a data - processing method provided by an embodiment of this application. As Figure 8 shown, the data - processing method at least includes the following steps:

[0157] 801. Obtain target object data.

[0158] In one or more embodiments, the described target object data includes, but is not limited to, text data, image data, video data, etc., and is not limited in this application.

[0159] 802. Process the downstream task related to the target object data based on the target student model.

[0160] In one or more embodiments, after obtaining the target object data, the target object data can be used as the input of the target student model, and the target student model is used to complete the processing of the downstream task related to the target object data. For example, the downstream task includes, but is not limited to, the object classification task for the target object data, identifying the object type of the target object data, predicting the sorting situation of the target object data, etc., and is not limited in this application.

[0161] It should be noted that for the target student model described here, its training process can be understood with reference to the content described in the foregoing Figures 3 to 7 and will not be elaborated here.

[0162] Exemplarily, taking the selection of BERT-base(109M) as the teacher model as an example, Table 1 below shows the comparison results of the average accuracy performance of this application and the traditional scheme on the General Language Understanding Evaluation Dataset (GLUE) and the Stanford Question Answering Dataset (SQuAD).

[0163] Table 1

[0164] Method Number of parameters GLUE SQuAD1.1 / SQuAD2.0 BERT-base 109M 82.1 76.0 / 69.5 TinyBERT (traditional solution) 67.0M 81.4 68.2 / 65.8 This application 67.0M 82.0 74.9 / 69.1

[0165] As shown in Table 1, it can be seen that compared with the traditional scheme with the same number of parameters of 67.0M, the distillation method proposed in this application achieves higher performance. Specifically, the distillation method proposed in this application achieves 99.8% of the performance of the teacher model BERT-base with only 61.4% of the number of parameters on the GLUE evaluation dataset, and its effect on the SQuAD evaluation dataset also steadily improves.

[0166] For another example, Table 2 shows the experimental results of applying the model training method of this application to the autoregressive architecture model.

[0167]

[0168]

[0169] As shown in Table 2 above, the method for model training of the present application was applied to the mainstream GPT-2 and LLaMA-3, and experiments were conducted. For GPT2-124M, the student model achieved 82% of the performance of the teacher model with 62% of the number of parameters. Under the same parameter settings, the method for model training of the present application outperformed other extreme methods, such as Supervisefine tuning (SFT), Knowledge Distillation (KD), and SeqKD, etc. For the number of parameters of LLaMA-3 reduced from 1.23B to 0.69B, the method for model training of the present application achieved the best results on all three instruction-following datasets.

[0170] The above mainly introduced the solution provided by the embodiments of the present application from the perspective of the method. It can be understood that in order to implement the above functions, it includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the modules and algorithm steps of each example described in the embodiments disclosed in the present application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0171] The embodiments of the present application can divide the device into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0172] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0173] The model training device in the embodiments of the present application will be described in detail below. Figure 9 It is a schematic diagram of an embodiment of the model training device provided in the embodiments of the present application. As Figure 9As shown, the model training device may include:

[0174] An acquisition unit 901, configured to acquire object information;

[0175] An extraction unit 902, configured to extract an embedded feature vector of the object information;

[0176] A processing unit 903, configured to process the embedded feature vector based on a teacher model to obtain first inter-layer information, first matrix information, and first prediction information. The first inter-layer information is used to describe the self-attention distribution and hidden layer state of the teacher model. The first matrix information is used to describe the matrix conditions of the multi-head attention module and the feed-forward neural network FNN module of the teacher model. The first prediction information is used to describe the model prediction situation of the teacher model for the object information;

[0177] The processing unit 903 is further configured to process the embedded feature vector based on an initial student model to obtain second inter-layer information, second matrix information, and second prediction information. The second inter-layer information is used to describe the self-attention distribution and hidden layer state of the initial student model. The second matrix information is used to describe the matrix conditions of the multi-head attention module and the FNN module of the initial student model. The second prediction information is used to describe the model prediction situation of the initial student model for the object information. The initial student model is a neural network model obtained by performing singular value decomposition on the weight matrix of the student model to be optimized;

[0178] The processing unit 903 is configured to calculate first loss information based on the first inter-layer information and the second inter-layer information, calculate second loss information based on the first matrix information and the second matrix information, and calculate third loss information based on the first prediction information and the second prediction information;

[0179] An update unit 904, configured to update the model parameters of the initial student model based on the target loss information to obtain a target student model. The target student model is used to process downstream tasks related to the object information. The target loss information is one or more of the first loss information, the second loss information, and the third loss information.

[0180] Based on one or more of the foregoing Figure 9 In another embodiment provided by the present application based on the illustrated one or more embodiments, before the acquisition unit 901 further updates the model parameters of the initial student model based on the target loss information to obtain a target student model, the acquisition unit 901 is further configured to acquire the number of distillation layers and the number of first structure layers. The number of distillation layers is used to describe the number of intermediate layers that need to be aligned in the teacher model and the initial student model. The number of first structure layers is used to describe the number of Transformer structures in the teacher model and the initial student model;

[0181] The processing unit 903 is specifically configured to determine target loss information from the first loss information, the second loss information, and the third loss information based on a comparison result between the number of distillation layers and the number of first structure layers.

[0182] In the above Figure 9 Based on one or more embodiments shown, in another embodiment provided by the present application, the processing unit 903 is specifically configured to:

[0183] When the number of distillation layers is greater than or equal to zero and less than the first value, the second loss information is selected from the first loss information, the second loss information and the third loss information as the target loss information. The first value is obtained by 1 / N times the number of first structure layers, where N is an integer greater than or equal to 2.

[0184] In the above Figure 9 Based on one or more embodiments shown, in another embodiment provided by the present application, the processing unit 903 is specifically configured to:

[0185] When the number of distillation layers is greater than or equal to the first value and the number of distillation layers is less than or equal to the first number of structural layers, the first loss information and the second loss information are selected from the first loss information, the second loss information and the third loss information as the target loss information. The first value is obtained by 1 / N times the first number of structural layers, where N is an integer greater than or equal to 2.

[0186] In the above Figure 9 Based on one or more embodiments shown, in another embodiment provided by the present application, the processing unit 903 is specifically configured to:

[0187] When the number of distillation layers is equal to the second value, the first loss information, the second loss information and the third loss information are selected as the target loss information, and the second value is obtained by the sum of the first structure layer number and the preset value.

[0188] In the above Figure 9 On the basis of one or more embodiments shown, in another embodiment provided by the present application, the first inter-layer information includes self-attention distribution information and hidden layer information of the teacher model, and the second inter-layer information includes self-attention distribution information and hidden layer information of the initial student model; the processing unit 903 is specifically used to:

[0189] Calculate the difference between the self-attention distribution information of the initial student model and the self-attention distribution information of the teacher model to obtain the first-layer inter-sub-loss value;

[0190] Calculate the difference between the hidden layer information of the initial student model and the hidden layer information of the teacher model to obtain the second-layer inter-sub-loss value;

[0191] Determine the first loss information based on the first inter-layer sub-loss value and the second inter-layer sub-loss value.

[0192] Based on one or more of the embodiments shown above, in another embodiment provided by the present application, the self-attention distribution information of the teacher model includes the first self-attention distribution information of each of the h attention heads, and the self-attention distribution information of the initial student model includes the second self-attention distribution information of each attention head, where h is an integer greater than or equal to 1; the processing unit 903 is specifically configured to: Figure 9 Calculate the KL divergence between the second self-attention distribution information of each attention head and the first self-attention distribution information of each attention head to obtain h KL divergence values;

[0193] Calculate the sum of the h KL divergence values to obtain the total divergence value;

[0194] Calculate the quotient between the total divergence value and the number of attention heads to obtain the first inter-layer sub-loss value.

[0195] Based on one or more of the embodiments shown above, in another embodiment provided by the present application, the processing unit 903 is specifically configured to: calculate the mean square error between the hidden layer information of the initial student model and the hidden layer information of the teacher model to obtain the second inter-layer sub-loss value.

[0196] Based on one or more of the embodiments shown above, in another embodiment provided by the present application, the first matrix information includes the matrix information of the multi-head attention module and the matrix information of the FNN module of the teacher model, and the second matrix information includes the matrix information of the multi-head attention module and the matrix information of the FNN module of the initial student model; the processing unit 903 is specifically configured to: Figure 9 Calculate the difference between the matrix information of the multi-head attention module of the initial student model and the matrix information of the multi-head attention module of the teacher model to obtain the first matrix sub-loss value;

[0197] Calculate the difference between the matrix information of the FNN module of the initial student model and the matrix information of the FNN module of the teacher model to obtain the second matrix sub-loss value; Figure 9 Based on the first matrix sub-loss value and the second matrix sub-loss value, determine the second loss information.

[0198] Calculate the difference between the matrix information of the multi-head attention module of the initial student model and the matrix information of the multi-head attention module of the teacher model to obtain the first matrix sub-loss value;

[0199] Calculate the difference between the matrix information of the FNN module of the initial student model and the matrix information of the FNN module of the teacher model to obtain the second matrix sub-loss value;

[0200] Based on the first matrix sub-loss value and the second matrix sub-loss value, determine the second loss information.

[0201] Based on one or more of the embodiments shown above, in another embodiment provided by the present application, Figure 9Based on one or more of the illustrated embodiments, in another embodiment provided by the present application, the matrix information of the multi-head attention module of the teacher model includes the first self-attention matrix of each of the h attention heads, and the self-attention distribution information of the initial student model includes the second self-attention matrix of each attention head, where h is an integer greater than or equal to 1; the processing unit 903 is specifically configured to:

[0202] Calculate the difference between the matrix information of the multi-head attention module of the initial student model and the matrix information of the multi-head attention module of the teacher model to obtain a first matrix sub-loss value, including:

[0203] Calculate the mean square error between the second self-attention matrix of each attention head and the first self-attention matrix of each attention head to obtain h first mean square error values;

[0204] Calculate the sum of the h first mean square error values to obtain the total first mean square error;

[0205] Calculate the quotient between the total first mean square error and the number of attention heads to obtain the first matrix sub-loss value.

[0206] In the above Figure 9 Based on one or more of the illustrated embodiments, in another embodiment provided by the present application, the matrix information of the FNN module of the teacher model includes the mapping matrix of the first FNN module of each of the h attention heads, and the self-attention distribution information of the initial student model includes the mapping matrix of the second FNN module of each attention head, where h is an integer greater than or equal to 1; the processing unit 903 is specifically configured to:

[0207] Calculate the mean square error between the mapping matrix of the second FNN module of each attention head and the mapping matrix of the first FNN module of each attention head to obtain h second mean square error values;

[0208] Calculate the sum of the h second mean square error values to obtain the total second mean square error;

[0209] Calculate the quotient between the total second mean square error and the number of attention heads to obtain the second matrix sub-loss value.

[0210] In the above Figure 9 Based on one or more of the illustrated embodiments, in another embodiment provided by the present application, the processing unit 903 is specifically configured to: calculate the cross-entropy loss value between the first prediction information and the second prediction information to obtain the third loss information.

[0211] In the above Figure 9Based on one or more of the illustrated embodiments, in another embodiment provided by the present application, the obtaining unit 901 is further specifically configured to: before processing the embedded feature vector based on the initial student model to obtain the second inter-layer information, the second matrix information, and the second prediction information, obtain the weight matrix of the student model to be optimized;

[0212] The processing unit 903 is specifically configured to:

[0213] Perform singular value decomposition on the weight matrix of the student model to be optimized to obtain a first matrix and a second matrix. The number of column elements of the first matrix and the number of row elements of the second matrix are both the same as the rank of the weight matrix of the student model to be optimized;

[0214] Replace the weight matrix of the student model to be optimized with the first matrix and the second matrix to obtain the initial student model.

[0215] The model training apparatus in the embodiments of the present application is described above from the perspective of modular functional entities. Below, the data processing device in the embodiments of the present application is described from the perspective of hardware processing. Figure 10 FIG. is a schematic structural diagram of a data processing device provided by an embodiment of the present application. This data processing device may vary greatly due to configuration or performance differences. For example, it includes, but is not limited to, the aforementioned model training device, model usage device, model training apparatus, and so on.

[0216] As Figure 10 shown, this data processing device may vary greatly due to configuration or performance differences. It may include one or more central processing units (CPUs) 322 (for example, one or more processors) and a memory 332, and one or more storage media 330 (for example, one or more mass storage devices) for storing application programs 342 or data 344. Among them, the memory 332 and the storage media 330 may be transient storage or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the data processing device. Further, the central processing unit 322 may be configured to communicate with the storage media 330 and execute a series of instruction operations in the storage media 330 on the data processing device. Exemplarily, the central processing unit 322 is used to execute the application program 342 stored in the storage media 330, thereby implementing the model training method and so on provided by the above embodiments of the present application.

[0217] The data processing device may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSD TM, and so on.

[0218] Exemplarily, Figure 10 the central processing unit 322 in Figures 3 to 8 may cause the data processing device to execute the methods in the corresponding method embodiments by calling the computer-executable instructions stored in the memory 332.

[0219] The steps performed by the data processing device in the foregoing embodiments may be based on the Figure 10 data processing device structure shown.

[0220] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program or instructions are stored. When the computer program or instructions are executed by a processor, the steps of the methods described in the foregoing embodiments are implemented.

[0221] An embodiment of the present application also provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the steps of the methods described in the foregoing embodiments are implemented.

[0222] In the foregoing embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product.

[0223] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above may refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0224] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point, the displayed or discussed couplings or direct couplings or communication connections to each other may be through some interfaces, and the indirect couplings or communication connections of devices or units may be in electrical, mechanical, or other forms.

[0225] The unit described as a separation component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed across multiple network units. Some or all of these units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] In addition, in each embodiment of this application, each functional unit can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0227] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of this application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs, etc., which can store program codes.

[0228] The computer program product includes one or more computer instructions. When the computer execution instructions are loaded and executed on a computer, the processes or functions according to the embodiments of this application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, an SSD), etc.

[0229] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: include: Acquire object information, and extract embedded feature vectors of the object information; Processing the embedded feature vector based on the teacher model to obtain first inter-layer information, first matrix information and first prediction information, wherein the first inter-layer information is used to describe the self-attention distribution and hidden layer state of the teacher model, the first matrix information is used to describe the matrix of the multi-head attention module and the matrix of the feedforward neural network FNN module of the teacher model, and the first prediction information is used to describe the model prediction of the teacher model for the object information; The embedded feature vector is processed based on the initial student model to obtain second inter-layer information, second matrix information and second prediction information, wherein the second inter-layer information is used to describe the self-attention distribution and hidden layer state of the initial student model, the second matrix information is used to describe the matrix of the multi-head attention module and the matrix of the FNN module of the initial student model, and the second prediction information is used to describe the model prediction of the initial student model for the object information, and the initial student model is a neural network model obtained by performing singular value decomposition processing on the weight matrix of the student model to be optimized; Calculating first loss information based on the first inter-layer information and the second inter-layer information, calculating second loss information based on the first matrix information and the second matrix information, and calculating third loss information based on the first prediction information and the second prediction information; The model parameters of the initial student model are updated based on the target loss information to obtain a target student model, wherein the target student model is used to process downstream tasks related to the object information, and the target loss information is one or more of the first loss information, the second loss information, and the third loss information.

2. The method according to claim 1, characterized in that: Before updating the model parameters of the initial student model based on the target loss information to obtain the target student model, the method further includes: Obtaining the number of distillation layers and the first structure layer number, wherein the distillation layer number is used to describe the number of intermediate layers that need to be aligned between the teacher model and the initial student model, and the first structure layer number is used to describe the number of layers of the Transformer structure in the teacher model and the initial student model; Based on a comparison result between the number of distillation layers and the first number of structure layers, the target loss information is determined from the first loss information, the second loss information, and the third loss information.

3. The method according to claim 2, characterized in that Based on a comparison result between the number of distillation layers and the first number of structure layers, determining target loss information from the first loss information, the second loss information, and the third loss information, comprising: When the number of distillation layers is greater than or equal to zero and the number of distillation layers is less than a first value, the second loss information is selected from the first loss information, the second loss information and the third loss information as the target loss information, and the first value is obtained by 1 / N times the number of first structure layers, where N is an integer greater than or equal to 2.

4. The method according to claim 2, characterized in that: Based on a comparison result between the number of distillation layers and the first number of structure layers, determining target loss information from the first loss information, the second loss information, and the third loss information, comprising: When the number of distillation layers is greater than or equal to the first value and the number of distillation layers is less than or equal to the first number of structural layers, the first loss information and the second loss information are selected from the first loss information, the second loss information and the third loss information as target loss information, and the first value is obtained by 1 / N times the first number of structural layers, where N is an integer greater than or equal to 2.

5. The method according to claim 2, characterized in that: Based on a comparison result between the number of distillation layers and the first number of structure layers, determining target loss information from the first loss information, the second loss information, and the third loss information, comprising: When the number of distillation layers is equal to a second value, the first loss information, the second loss information and the third loss information are selected as target loss information, and the second value is obtained by the sum of the first number of structure layers and a preset value.

6. The method according to any one of claims 1 to 5, characterized in that The first inter-layer information includes the self-attention distribution information and hidden layer information of the teacher model, and the second inter-layer information includes the self-attention distribution information and hidden layer information of the initial student model; Calculating first loss information based on the first inter-layer information and the second inter-layer information includes: Calculate the difference between the self-attention distribution information of the initial student model and the self-attention distribution information of the teacher model to obtain a first inter-layer sub-loss value; Calculate the difference between the hidden layer information of the initial student model and the hidden layer information of the teacher model to obtain a second inter-layer sub-loss value; The first loss information is determined based on the first inter-layer sub-loss value and the second inter-layer sub-loss value.

7. The method according to claim 6, characterized in that The self-attention distribution information of the teacher model includes the first self-attention distribution information of each of the h attention heads, and the self-attention distribution information of the initial student model includes the second self-attention distribution information of each of the attention heads, where h is an integer greater than or equal to 1; Calculating the difference between the self-attention distribution information of the initial student model and the self-attention distribution information of the teacher model to obtain the first inter-layer sub-loss value, including: Calculate the KL divergence between the second self-attention distribution information of each of the attention heads and the first self-attention distribution information of each of the attention heads to obtain h KL divergence values; Calculate the sum of h KL divergence values ​​to obtain the total divergence value; The quotient between the sum of the divergence values ​​and the number of attention heads is calculated to obtain the first inter-layer sub-loss value.

8. The method according to any one of claims 1 to 5, characterized in that The first matrix information includes matrix information of the multi-head attention module and the matrix information of the FNN module of the teacher model, and the second matrix information includes matrix information of the multi-head attention module and the matrix information of the FNN module of the initial student model; Calculating second loss information based on the first matrix information and the second matrix information includes: Calculate the difference between the matrix information of the multi-head attention module of the initial student model and the matrix information of the multi-head attention module of the teacher model to obtain a first matrix sub-loss value; Calculate the difference between the matrix information of the FNN module of the initial student model and the matrix information of the FNN module of the teacher model to obtain a second matrix sub-loss value; The second loss information is determined based on the first matrix sub-loss value and the second matrix sub-loss value.

9. The method according to claim 8, characterized in that The matrix information of the multi-head attention module of the teacher model includes a first self-attention matrix of each of the h attention heads, and the self-attention distribution information of the initial student model includes a second self-attention matrix of each of the attention heads, where h is an integer greater than or equal to 1; Calculating the difference between the matrix information of the multi-head attention module of the initial student model and the matrix information of the multi-head attention module of the teacher model to obtain a first matrix sub-loss value, including: Calculate the mean square error between the second self-attention matrix of each of the attention heads and the first self-attention matrix of each of the attention heads to obtain h first mean square error values; Calculate the sum of h first mean square error values ​​to obtain a first mean square error sum; A quotient of the first mean square error sum and the number of attention heads is calculated to obtain a first matrix sub-loss value.

10. The method according to claim 8, characterized in that The matrix information of the FNN module of the teacher model includes the mapping matrix of the first FNN module of each of the h attention heads, and the self-attention distribution information of the initial student model includes the mapping matrix of the second FNN module of each of the attention heads, where h is an integer greater than or equal to 1; Calculating the difference between the matrix information of the FNN module of the initial student model and the matrix information of the FNN module of the teacher model to obtain a second matrix sub-loss value includes: Calculate the mean square error between the mapping matrix of the second FNN module of each of the attention heads and the mapping matrix of the first FNN module of each of the attention heads to obtain h second mean square error values; Calculate the sum of h second mean square error values ​​to obtain a second mean square error sum; The quotient of the second mean square error sum and the number of attention heads is calculated to obtain a second matrix sub-loss value.

11. The method according to any one of claims 1 to 10, characterized in that Before processing the embedded feature vector based on the initial student model to obtain second inter-layer information, second matrix information and second prediction information, the method further includes: Obtaining a weight matrix of the student model to be optimized; Performing singular value decomposition processing on the weight matrix of the student model to be optimized to obtain a first matrix and a second matrix, wherein the number of column elements of the first matrix and the number of row elements of the second matrix are both the same as the rank of the weight matrix of the student model to be optimized; The first matrix and the second matrix are used to replace the weight matrix of the student model to be optimized to obtain the initial student model.

12. A model training device, characterized in that: include: An acquisition unit, used for acquiring object information; An extraction unit, used to extract an embedded feature vector of the object information; A processing unit, configured to process the embedded feature vector based on a teacher model to obtain first inter-layer information, first matrix information, and first prediction information, wherein the first inter-layer information is used to describe the self-attention distribution and hidden layer state of the teacher model, the first matrix information is used to describe the matrix of the multi-head attention module and the matrix of the feedforward neural network FNN module of the teacher model, and the first prediction information is used to describe the model prediction of the teacher model for the object information; The processing unit is used to process the embedded feature vector based on the initial student model to obtain second inter-layer information, second matrix information and second prediction information, wherein the second inter-layer information is used to describe the self-attention distribution and hidden layer state of the initial student model, the second matrix information is used to describe the matrix of the multi-head attention module and the matrix of the FNN module of the initial student model, and the second prediction information is used to describe the model prediction of the initial student model for the object information, and the initial student model is a neural network model obtained by performing singular value decomposition processing on the weight matrix of the student model to be optimized; The processing unit is configured to calculate first loss information based on the first inter-layer information and the second inter-layer information, calculate second loss information based on the first matrix information and the second matrix information, and calculate third loss information based on the first prediction information and the second prediction information; An updating unit is used to update the model parameters of the initial student model based on the target loss information to obtain a target student model, wherein the target student model is used to process downstream tasks related to the object information, and the target loss information is one or more of the first loss information, the second loss information and the third loss information.

13. A data processing device, characterized in that: including a processor and a memory; Wherein, the memory stores program instructions; The processor is used to execute the program instructions stored in the memory to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when executed on a computer device, cause the computer device to perform the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises instructions, which, when executed on a computer device, cause the computer device to perform the method according to any one of claims 1 to 11.

Images