Heterogeneous knowledge distillation method and device based on Transform attention mechanism

Abstract

The invention provides a heterogeneous knowledge distillation method and device based on a Transform attention mechanism, and the method comprises the steps: obtaining first sample data and a corresponding first label; inputting the first sample data into a teacher model to obtain a first prediction result corresponding to the first sample data and a first feature output by each first feature layer in the teacher model; each first feature is input into a Transform network connected with the corresponding feature layer, and a second feature output by each Transform network is obtained; inputting the first sample data into a student model to obtain a third feature output by each second feature layer in the student model; and training the student model according to the difference between the first prediction result and the first label and the difference between the second feature and the third feature to obtain a trained student model. Therefore, a Transform network is used between the feature layers of the teacher model and the student model, so that the student model can learn more important data features of the teacher model in a short time.

Description

Technical Field

[0001] The present application relates to the technical field of knowledge distillation, and in particular to a heterogeneous knowledge distillation method and device based on Transformer attention mechanism. Background Art

[0002] The purpose of knowledge distillation is to transfer the knowledge learned by a large model or an integration of multiple models to another lightweight model. Feature layer knowledge transfer is a method of knowledge distillation. Feature layer knowledge transfer refers to the use of the features of the middle layer of the teacher model as the target of the student model, thereby improving the representation ability and information content of the transferred knowledge and effectively improving the distillation training effect. However, when the layers or blocks of the teacher model and the student model do not completely correspond, how to implement feature layer knowledge distillation becomes an urgent problem to be solved. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of this application is to propose a heterogeneous knowledge distillation method based on the Transformer attention mechanism, so that when the layers or blocks of the teacher model and the student model cannot completely correspond, a Transformer network is used between the feature layers of the teacher model and the student model. The Transformer network can effectively capture the global information of the teacher model based on the attention mechanism, and pass the captured second feature to the student model, so that the student model can learn more important data features of the teacher model in a short time.

[0005] The second purpose of this application is to propose a heterogeneous knowledge distillation device based on the Transformer attention mechanism.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above objectives, the first embodiment of the present application proposes a heterogeneous knowledge distillation method based on the Transformer attention mechanism, including:

[0010] Acquire a first data set, wherein the first data set includes first sample data and a corresponding first label;

[0011] Inputting the first sample data into the teacher model to obtain a first prediction result corresponding to the first sample data and a first feature output by each first feature layer in the teacher model;

[0012] Input each of the first features into a Transformer network connected to a corresponding feature layer, to obtain a second feature output by each of the Transformer networks;

[0013] Inputting the first sample data into the student model to obtain the third feature output by each second feature layer in the student model;

[0014] The student model is trained according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain a trained student model.

[0015] To achieve the above objectives, the second embodiment of the present application proposes a heterogeneous knowledge distillation device based on the Transformer attention mechanism, including:

[0016] An acquisition module, configured to acquire a first data set, wherein the first data set includes first sample data and a corresponding first label;

[0017] A first processing module, used for inputting the first sample data into a teacher model to obtain a first prediction result corresponding to the first sample data and a first feature output by each first feature layer in the teacher model;

[0018] A second processing module, used for inputting each of the first features into a Transformer network connected to a corresponding feature layer, to obtain a second feature output by each of the Transformer networks;

[0019] A third processing module, used for inputting the first sample data into a student model to obtain a third feature output by each second feature layer in the student model;

[0020] The fourth processing module is used to train the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain a trained student model.

[0021] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including:

[0022] A processor, and a memory communicatively connected to the processor;

[0023] The memory stores computer-executable instructions;

[0024] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the embodiment of the first aspect.

[0025] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect embodiment.

[0026] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, a computer program, which implements the method described in the first aspect when executed by a processor.

[0027] The heterogeneous knowledge distillation method and device based on the Transformer attention mechanism provided in the present application inputs the first sample data into the teacher model, obtains the first prediction result corresponding to the first sample data and the first feature outputted by each first feature layer in the teacher model, inputs each first feature into the Transformer network connected to the corresponding feature layer, obtains the second feature outputted by each Transformer network, inputs the first sample data into the student model, obtains the third feature outputted by each second feature layer in the student model, and finally trains the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature, to obtain the trained student model. Thus, in the case where the layers or blocks of the teacher model and the student model cannot completely correspond, the Transformer network is used between the feature layers of the teacher model and the student model, and the Transformer network can effectively capture the global information of the teacher model based on the attention mechanism, and pass the captured second feature to the student model, so that the student model can learn more important data features of the teacher model in a short time.

[0028] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A schematic diagram of a process of heterogeneous knowledge distillation method based on Transformer attention mechanism provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the structure of a mentor-apprentice model provided in one embodiment of the present disclosure;

[0032] Figure 3 A flowchart of another heterogeneous knowledge distillation method based on the Transformer attention mechanism provided in an embodiment of the present application;

[0033] Figure 4 A flowchart of another heterogeneous knowledge distillation method based on the Transformer attention mechanism provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of a process for determining a second feature based on a Transformer network provided in one embodiment of the present disclosure;

[0035] Figure 6 A schematic diagram of the structure of a Transformer block provided in one embodiment of the present disclosure;

[0036] Figure 7 A schematic diagram of the structure of a heterogeneous knowledge distillation device based on the Transformer attention mechanism provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0038] The following describes the heterogeneous knowledge distillation method and device based on the Transformer attention mechanism of an embodiment of the present application with reference to the accompanying drawings.

[0039] Figure 1 A flow chart of a heterogeneous knowledge distillation method based on the Transformer attention mechanism provided in an embodiment of the present application.

[0040] like Figure 1 As shown in the figure, the heterogeneous knowledge distillation method based on the Transformer attention mechanism includes the following steps:

[0041] Step 101: Acquire a first data set, wherein the first data set includes first sample data and a corresponding first label.

[0042] Among them, the first data set is used to synchronously train the teacher model, Transformer network and student model to achieve online distillation.

[0043] In some embodiments, the sample data is at least one of text, voice, image, and video, which is not limited in the present disclosure.

[0044] For example, if the task type of the teacher model is face recognition, the first sample data may be an image containing a face, and the first label may be the area of ​​the face in the image.

[0045] Step 102: input the first sample data into the teacher model to obtain a first prediction result corresponding to the first sample data and a first feature output by each first feature layer in the teacher model.

[0046] Figure 2 A schematic diagram of a master-apprentice model provided in one embodiment of the present disclosure; Figure 2 As shown in the figure, the master-apprentice model includes a teacher model, a Transformer network, and a student model.

[0047] The teacher model includes multiple first feature layers, wherein the jth first feature layer is used to process the first feature output by the j-1th first feature layer to obtain the first feature output by the jth first feature layer, and the last feature layer is used to output the first prediction result.

[0048] Step 103, input each first feature into the Transformer network connected to the corresponding feature layer, and obtain the second feature output by each Transformer network.

[0049] like Figure 2 As shown, each first feature layer is connected to a Transformer network. The Transformer network is used to learn the important information in the first feature and pass the important information in the first feature to the student model.

[0050] like Figure 2 As shown, the first feature output by the j-th first feature layer in the student model can be input into the Transformer network connected to the j-th first feature layer to obtain the second feature.

[0051] Step 104, input the first sample data into the student model to obtain the third feature output by each second feature layer in the student model.

[0052] Among them, the sizes of each feature layer or block of the student model and the teacher model do not completely correspond.

[0053] like Figure 2 As shown, the teacher model may include multiple second feature layers, wherein the i-th second feature layer is used to process the third feature output by the i-1-th second feature layer to obtain the third feature output by the i-th second feature layer.

[0054] like Figure 2As shown in the figure, the second feature output by the Transformer network is input into the student model, so that the student model uses the second feature as the training target to learn the knowledge learned by the teacher model.

[0055] Step 105: Train the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain a trained student model.

[0056] In some embodiments, a loss function can be first determined according to the task type corresponding to the teacher model, and then a first loss value corresponding to the difference between the first prediction result and the first label can be determined based on the loss function. A second loss value can be determined according to the difference between the second feature and the third feature. The first loss value and the second loss value are fused to obtain a target loss value. Finally, based on the target loss value, the student model is trained to obtain a trained student model.

[0057] Different task types use different loss functions. For example, the task type can be a classification model, a detection model, a regression task, etc. The present disclosure does not limit the loss function used for each task type.

[0058] In some embodiments, a weight corresponding to the second loss value may be determined, and based on the weight corresponding to the second loss value, the first loss value and the second loss value may be fused to obtain a target loss value.

[0059] In some embodiments, based on the activation function, the fourth feature corresponding to each second feature and the fifth feature corresponding to each third feature can be determined, and then the Euclidean distance between the fourth feature corresponding to the second feature output by the Transformer network connected to the i-th first feature layer and the fifth feature corresponding to the third feature output by the i-th second feature layer is determined, and finally the sum of multiple Euclidean distances is determined as the second loss value.

[0060] Among them, the formula for determining the target loss value can be as follows:

[0061] loss = L task +γ*L distill (1)

[0062] L distill =d p (σ T (ρ(T, M)), ·σ S (S)) (2)

[0063] σ(X)=MAX(X,·0) ( 3 )

[0064]

[0065] Among them, loss is the target loss value, L task is the first loss value, L distill is the second loss value, γ is the weight, ρ(T, M) is the second feature, S is the third feature, σ(X) is the activation function, when X is ρ(T, M), σ T (ρ(T, M) is the fourth feature corresponding to the second feature. When X is S, σ S (S) is the fifth feature corresponding to the third feature, C is the total number of first sample data used in one iteration training, N is the total number of Transformer networks, t ij is the jth fourth feature corresponding to the i-th sample data, s ij is the jth fifth feature corresponding to the i-th sample data, (t ij -s ij ) 2 Represents the Euclidean distance between the j-th fourth feature and the j-th fifth feature corresponding to the i-th sample data.

[0066] In the disclosed embodiment, a unified loss function is designed for various types of tasks, which can be applied to various visual tasks such as detection, regression, classification, segmentation, etc. in general scenarios. In the global teacher and student model, the loss function takes into account both the task loss of the teacher model and the distance loss between the student model feature layer and the teacher model feature layer, and balances the distribution of tasks between the two through a weight coefficient. As a result, the student model parameters can be closer to the learning teacher model parameters, so that the accuracy of the teacher model can be quickly achieved.

[0067] In some embodiments, if the sizes of the second feature and the third feature do not match, a 1*1 convolutional layer can be connected after the Transformer network so that the size of the output second feature matches the size of the third feature output by the student model.

[0068] In this embodiment, the first sample data is input into the teacher model to obtain the first prediction result corresponding to the first sample data and the first feature output by each first feature layer in the teacher model, each first feature is respectively input into the Transformer network connected to the corresponding feature layer to obtain the second feature output by each Transformer network, the first sample data is input into the student model to obtain the third feature output by each second feature layer in the student model, and finally the student model is trained according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain the trained student model. Thus, in the case where the layers or blocks of the teacher model and the student model cannot completely correspond, the Transformer network is used between the feature layers of the teacher model and the student model, and the Transformer network can effectively capture the global information of the teacher model based on the attention mechanism, and pass the captured second feature to the student model, so that the student model can learn more important data features of the teacher model in a short time.

[0069] In the disclosed embodiment, a Transformer network is used between the feature layers of the teacher model and the student model, which can not only effectively capture the global information of the teacher model, but also use a self-attention mechanism for feature interaction and transmission, so that information at different locations can be always paid attention to when processing the features of the teacher model. And because the Transformer model has extremely high computational efficiency and parallel processing capabilities, the Transformer network can process the first feature in parallel, and in the training of large-scale data sets, it can enable the student model to learn more data features and knowledge of the teacher model in a short time.

[0070] Figure 3 The following is a flow chart of a heterogeneous knowledge distillation method based on the Transformer attention mechanism provided in the embodiment of the present application. Figure 3 As shown, the heterogeneous knowledge distillation method based on the Transformer attention mechanism can include the following steps:

[0071] Step 301: Acquire a first data set, wherein the first data set includes first sample data and a corresponding first label.

[0072] Step 302: input the first sample data into the teacher model to obtain the first prediction result corresponding to the first sample data and the first feature output by each first feature layer in the teacher model.

[0073] Step 303: input the first sample data into the student model to obtain the third feature output by each second feature layer in the student model.

[0074] Step 304: train the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain a trained student model.

[0075] Step 305: Train the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain a trained student model.

[0076] The specific implementation forms of step 304 to step 305 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0077] Step 306: Acquire a second data set, wherein the second data set includes second sample data and corresponding second labels.

[0078] In some embodiments, the second sample data may be at least one of text, voice, image, and video.

[0079] In some embodiments, the second data set may be the same as or different from the first data set. Alternatively, the first data set is partially the same as the second data set, or the second data set includes the first data set. This disclosure does not limit this.

[0080] Step 307: input the second sample data into the trained student model to obtain a second prediction result corresponding to the second sample data.

[0081] Step 308: train the trained student model according to the difference between the second prediction result and the second label to obtain a target student model.

[0082] In some embodiments, a loss function can be determined according to the task type of the student model, and then based on the loss function, a third loss value corresponding to the difference between the second prediction result and the second label is determined, and based on the third loss value, the trained student model is trained to obtain a target student model.

[0083] In the disclosed embodiment, after performing knowledge distillation on the teacher model to obtain the student model, the student model can be further trained based on the second data set, thereby further improving the performance of the obtained target student model.

[0084] Figure 4 The following is a flow chart of a heterogeneous knowledge distillation method based on the Transformer attention mechanism provided in the embodiment of the present application. Figure 4 As shown, the heterogeneous knowledge distillation method based on the Transformer attention mechanism can include the following steps:

[0085] Step 401: Acquire a first data set, wherein the first data set includes first sample data and a corresponding first label.

[0086] Step 402: input the first sample data into the teacher model to obtain the first prediction result corresponding to the first sample data and the first feature output by each first feature layer in the teacher model.

[0087] The specific implementation forms of step 401 to step 402 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0088] Step 403: Divide the first feature into a preset number of feature blocks.

[0089] For example, when the first sample data is an image, the first feature can be evenly divided into 9 feature blocks, so that the dimension of the first feature can be reduced, and each feature block can be processed in parallel using the Transformer network, thereby reducing the feature dimension and improving processing efficiency.

[0090] Step 404: Input a preset number of feature blocks into the Transformer network to obtain the sixth feature corresponding to each feature block.

[0091] In some embodiments, skip connections, such as long skip connections and short skip connections, may be added to the Transformer network. Thus, the skip connections can retrieve some information lost in the upper layers to provide a better context for the Transformer network, thereby making the features of the student model more similar to those of the teacher model, so that the teacher model can better transfer knowledge to the student model.

[0092] In some embodiments, the mth Transformer block in the Transformer network is associated with the m+1th Transformer block and the m+nth Transformer block. 1 Transformer blocks are connected, where n 1 is an integer greater than 1; and / or, the m+nth 2 Transformer blocks and the m+nth 2 +1 Transformer block and m+nth 3 Transformer blocks are connected, where n 2 、n 3 is an integer, and n 3 Greater than n 2 +1,n 3 Less than n 1 .

[0093] For example, m is 1, n 1 is 4, n 2 is 1, n 3 If it is 3, the first Transformer block is connected to the second Transformer block and the 5th Transformer block, and the second Transformer block is connected to the third Transformer block and the 4th Transformer block.

[0094] Step 405 , based on the position of each feature block in the first feature, the sixth features corresponding to each feature block are spliced ​​to obtain the seventh feature.

[0095] Step 406, convolve the seventh feature with the preset convolution block to obtain a second feature corresponding to the first feature.

[0096] The preset convolution block may be a 3*3 convolution block. In the disclosed embodiment, the seventh feature output by the Transformer network is convolved with the preset convolution block to obtain better visual quality.

[0097] Figure 5 A schematic diagram of a process for determining a second feature based on a Transformer network is provided in an embodiment of the present disclosure, such as Figure 5 As shown in the figure, the Transformer network may include an embedding layer, 5 Transformer blocks, a linear layer, and a 3*3 convolutional block.

[0098] like Figure 5As shown in the Transformer network, determining the second feature corresponding to the first feature includes: first dividing the first feature into 9 feature blocks, then inputting the 9 feature blocks into the embedding layer (EmbeddingLayer) of the Transformer network to obtain embedded features, then inputting the embedded features into the first Transformer block, then inputting the features output by the first Transformer block into the second Transformer block, inputting the features output by the second Transformer block into the third Transformer block, fusing the features output by the second Transformer block and the features output by the third Transformer block, inputting the features output by the fourth Transformer block and the features output by the first Transformer block, inputting the features output by the fifth Transformer block, and inputting the features output by the fifth Transformer block into the linear layer to obtain the sixth feature, splicing the sixth features corresponding to each feature block based on the position of each feature block in the first feature to obtain the seventh feature, and finally convolving the seventh feature with the 3*3 convolution block to obtain the second feature corresponding to the first feature.

[0099] Figure 6 A schematic diagram of the structure of a Transformer block provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown in the figure, the Transformer block includes a multi-head attention layer, an Add layer, a Norm layer, and a multilayer perceptron (MLP). Add represents a residual connection, which is used to prevent network degradation. Norm represents layer normalization, which is used to normalize the activation values ​​of each layer. The MLP layer is a feedforward neural network, usually used for classification and regression.

[0100] Step 407, input the first sample data into the student model to obtain the third feature output by each second feature layer in the student model.

[0101] Step 408: Train the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain a trained student model.

[0102] The specific implementation forms of step 407 and step 408 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0103] In the disclosed embodiment, the first feature is divided into a preset number of feature blocks, and then the preset number of feature blocks are input into the Transformer network to obtain the sixth feature corresponding to each feature block, and then based on the position of each feature block in the first feature, the sixth feature corresponding to each feature block is spliced ​​to obtain the seventh feature, and finally the seventh feature is convolved with the preset convolution block to obtain the second feature corresponding to the first feature. In this way, the dimension of the first feature can be reduced, and each feature block can be processed in parallel using the Transformer network, thereby improving the efficiency of determining the second feature.

[0104] In order to implement the above embodiments, the present application also proposes a heterogeneous knowledge distillation device based on the Transformer attention mechanism.

[0105] Figure 7 A schematic diagram of the structure of a heterogeneous knowledge distillation device based on the Transformer attention mechanism provided in an embodiment of the present application. Figure 7 As shown in the figure, the heterogeneous knowledge distillation device based on the Transformer attention mechanism includes:

[0106] An acquisition module 701 is used to acquire a first data set, wherein the first data set includes first sample data and a corresponding first label;

[0107] A first processing module 702 is used to input the first sample data into the teacher model to obtain a first prediction result corresponding to the first sample data and a first feature output by each first feature layer in the teacher model;

[0108] The second processing module 703 is used to input each first feature into the Transformer network connected to the corresponding feature layer to obtain the second feature output by each Transformer network;

[0109] The third processing module 704 is used to input the first sample data into the student model to obtain the third feature output by each second feature layer in the student model;

[0110] The fourth processing module 705 is used to train the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature, to obtain a trained student model.

[0111] Furthermore, in a possible implementation manner of the embodiment of the present application, a fifth processing module is also included, which is used to:

[0112] Acquire a second data set, wherein the second data set includes second sample data and corresponding second labels;

[0113] Inputting the second sample data into the trained student model to obtain a second prediction result corresponding to the second sample data;

[0114] According to the difference between the second prediction result and the second label, the trained student model is trained to obtain a target student model.

[0115] Furthermore, in a possible implementation of the embodiment of the present application, the fourth processing module 705 is configured to:

[0116] Determine the loss function based on the task type corresponding to the teacher model;

[0117] Based on the loss function, determining a first loss value corresponding to a difference between the first prediction result and the first label;

[0118] determining a second loss value according to a difference between the second feature and the third feature;

[0119] The first loss value and the second loss value are combined to obtain a target loss value;

[0120] Based on the target loss value, the student model is trained to obtain a trained student model.

[0121] Furthermore, in a possible implementation of the embodiment of the present application, the fourth processing module 705 is configured to:

[0122] Based on the activation function, determine the fourth feature corresponding to each second feature and the fifth feature corresponding to each third feature;

[0123] Determine the Euclidean distance between the fourth feature corresponding to the second feature output by the Transformer network connected to the j-th first feature layer and the fifth feature corresponding to the third feature output by the j-th second feature layer;

[0124] The sum of multiple Euclidean distances is determined as the second loss value.

[0125] Furthermore, in a possible implementation of the embodiment of the present application, the second processing module 703 is configured to:

[0126] Dividing the first feature into a preset number of feature blocks;

[0127] Input a preset number of feature blocks into the Transformer network to obtain the sixth feature corresponding to each feature block;

[0128] Based on the position of each feature block in the first feature, the sixth feature corresponding to each feature block is spliced ​​to obtain the seventh feature;

[0129] The seventh feature is convolved with the preset convolution block to obtain a second feature corresponding to the first feature.

[0130] Furthermore, in a possible implementation of the embodiment of the present application, the mth Transformer block in the Transformer network is connected to the m+1th Transformer block and the m+nth Transformer block. 1 Transformer blocks are connected, where n 1 is an integer greater than 1; and / or, the m+nth 2 Transformer blocks and the m+nth 2 +1 Transformer block and m+nth 3 Transformer blocks are connected, where n 2 、n 3 is an integer, and n 2 Less than n 3 , n 3 Less than n 1 .

[0131] Furthermore, in a possible implementation manner of the embodiment of the present application, the sample data is at least one of text, voice, image, and video.

[0132] It should be noted that the aforementioned explanation of the embodiment of the heterogeneous knowledge distillation method based on the Transformer attention mechanism is also applicable to the heterogeneous knowledge distillation device based on the Transformer attention mechanism of this embodiment, and will not be repeated here.

[0133] In this embodiment, the first sample data is input into the teacher model to obtain the first prediction result corresponding to the first sample data and the first feature output by each first feature layer in the teacher model, each first feature is respectively input into the Transformer network connected to the corresponding feature layer to obtain the second feature output by each Transformer network, the first sample data is input into the student model to obtain the third feature output by each second feature layer in the student model, and finally the student model is trained according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain the trained student model. Thus, in the case where the layers or blocks of the teacher model and the student model cannot completely correspond, the Transformer network is used between the feature layers of the teacher model and the student model, and the Transformer network can effectively capture the global information of the teacher model based on the attention mechanism, and pass the captured second feature to the student model, so that the student model can learn more important data features of the teacher model in a short time.

[0134] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0135] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0136] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0137] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0138] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0139] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0140] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0141] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0142] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0143] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0144] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0145] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0147] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A heterogeneous knowledge distillation method based on Transformer attention mechanism, characterized in that: The following steps are involved: Acquire a first data set, wherein the first data set includes first sample data and a corresponding first label; Inputting the first sample data into the teacher model to obtain a first prediction result corresponding to the first sample data and a first feature output by each first feature layer in the teacher model; Input each of the first features into a Transformer network connected to a corresponding feature layer, to obtain a second feature output by each of the Transformer networks; Inputting the first sample data into the student model to obtain the third feature output by each second feature layer in the student model; The student model is trained according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain a trained student model.

2. The method according to claim 1, characterized in that After training the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain the trained student model, the method further includes: Acquire a second data set, wherein the second data set includes second sample data and corresponding second labels; Inputting the second sample data into the trained student model to obtain a second prediction result corresponding to the second sample data; According to the difference between the second prediction result and the second label, the trained student model is trained to obtain a target student model.

3. The method according to claim 1, characterized in that The step of training the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature, to obtain the trained student model comprises: Determining a loss function according to the task type corresponding to the teacher model; Based on the loss function, determining a first loss value corresponding to a difference between the first prediction result and the first label; determining a second loss value according to a difference between the second feature and the third feature; The first loss value and the second loss value are combined to obtain a target loss value; Based on the target loss value, the student model is trained to obtain a trained student model.

4. The method according to claim 3, characterized in that The determining of the second loss value according to the difference between the second feature and the third feature includes: Based on the activation function, determine a fourth feature corresponding to each of the second features and a fifth feature corresponding to each of the third features; Determine the Euclidean distance between the fourth feature corresponding to the second feature output by the Transformer network connected to the j-th first feature layer and the fifth feature corresponding to the third feature output by the j-th second feature layer; The sum of the multiple Euclidean distances is determined as the second loss value.

5. The method according to claim 1, characterized in that The step of inputting each of the first features into a Transformer network connected to a corresponding feature layer to obtain a second feature output by each of the Transformer networks comprises: Dividing the first feature into a preset number of feature blocks; Inputting the preset number of feature blocks into the Transformer network to obtain a sixth feature corresponding to each feature block; Based on the position of each feature block in the first feature, the sixth features corresponding to each feature block are spliced ​​to obtain a seventh feature; Convolve the seventh feature with a preset convolution block to obtain a second feature corresponding to the first feature.

6. The method according to claim 1, characterized in that The mth Transformer block in the Transformer network is connected to the m+1th Transformer block and the m+n1th Transformer block, where n1 is an integer greater than 1; and / or, The m+n2th Transformer block in the Transformer network is connected to the m+n2+1th Transformer block and the m+n3th Transformer block, wherein n2 and n3 are integers, and n2 is less than n3, and n3 is less than n1.

7. The method according to any one of claims 1 to 6, characterized in that: The sample data is at least one of text, voice, image, and video.

8. A heterogeneous knowledge distillation device based on Transformer attention mechanism, characterized in that: include: An acquisition module, configured to acquire a first data set, wherein the first data set includes first sample data and a corresponding first label; A first processing module, used for inputting the first sample data into a teacher model to obtain a first prediction result corresponding to the first sample data and a first feature output by each first feature layer in the teacher model; A second processing module, used for inputting each of the first features into a Transformer network connected to a corresponding feature layer, to obtain a second feature output by each of the Transformer networks; A third processing module, used for inputting the first sample data into a student model to obtain a third feature output by each second feature layer in the student model; The fourth processing module is used to train the student model according to the difference between the first prediction result and the first label, and the difference between the second feature and the third feature to obtain a trained student model.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images