A transformer model processing method, readable storage medium and device

Abstract

The present invention provides a transformer model processing method, a readable storage medium, and a device. The transformer model processing method includes the following steps: during training, calculating a target layer based on the sub-layer weights of the transformer model, and deleting or retaining the target layer during the next training. The present invention automatically determines which sub-layers to delete or retain through forward propagation and backward propagation. Specifically, the weights obtained during the current training are used to calculate which sub-layers to delete or retain during the next training. This reduces the number of model parameters and implements a lightweight transformer model.

Description

Technical Field

[0001] The present invention belongs to the field of speech recognition technology, and specifically relates to a transformer model processing method, a readable storage medium and a device. Background Art

[0002] The Transformer model is an encoder-decoder architecture stacked six times. The encoder structure has two sublayers: a multi-head attention layer (MHA) and a feed-forward neural network (FFN). The decoder structure is similar to the encoder, consisting of three sublayers: two MHA sublayers and one FFN sublayer. The encoder and decoder structures each have their own functions. The encoder structure primarily extracts features from the input audio signal, extracting high-level features after six repetitions. The decoder's first MHA sublayer extracts features from the input character information. The second MHA sublayer matches the high-level audio signal features extracted by the encoder with the character information features. The FFN sublayer then further extracts these matched features. After six repetitions, the resulting features are used for model prediction.

[0003] The general process of the Transformer model for speech recognition is as follows:

[0004] The pre-processed audio features are first input into the word embedding module to obtain the feature tensor, and then the corresponding position information is obtained through the position encoding function. The feature tensor and the position information are added together, and the feature tensor with position information obtained by addition is the input of the encoder. The input will then pass through the MHA and FNN modules in sequence. After obtaining the output of each module, the residuals will be added and the layer will be normalized. This is the operation of a single encoder layer. After repeating the operation of the encoder layer n times, the output of the encoder is obtained, and this output will serve as part of the input of the second MHA in each decoder layer. The original input of the decoder is text information, and the operations that the input undergoes in the decoder are similar to those in the encoder. After obtaining the final output of the decoder, it will pass through the linear layer (Linear) and the softmax function in sequence to obtain the recognized word.

[0005] The Transformer model has excellent performance in speech recognition tasks and demonstrates extraordinary performance, but it still has the disadvantage of a large number of parameters during training. Summary of the Invention

[0006] In order to overcome the above technical defects, the present invention provides a transformer model processing method, which can reduce the number of parameters of the transformer model.

[0007] In order to solve the above problems, the present invention is implemented according to the following technical solutions:

[0008] A transformer model processing method includes the following steps:

[0009] During training, the target layer is calculated based on the sub-layer weights of the transformer model, and the target layer is deleted or retained during the next training.

[0010] As a further improvement of the present invention, the step of calculating the target layer according to the sub-layer weights of the transformer model and deleting or retaining the target layer includes:

[0011] Perform matrix operations on the two sub-layers in the Encoder structure of the Transformer model and the last two sub-layers in the Decoder structure based on the weights associated with the sub-layer outputs;

[0012] Select the minimum value in the matrix operation, take the layer where the minimum value is located as the target layer, obtain different target layers for each sub-layer, and delete the target layer during the next training.

[0013] As a further improvement of the present invention, the step of calculating the target layer according to the sub-layer weights of the transformer model and deleting or retaining the target layer includes:

[0014] In the first MHA sublayer of the Transformer model's decoder, matrix operations are performed based on the weights associated with the sublayer's outputs.

[0015] Select the maximum value in the matrix operation, and use the layer where the maximum value is located as the target layer. In the next training, retain the target layer and delete other sublayers.

[0016] As a further improvement of the present invention, during initial training, the first sub-layer is deleted or retained.

[0017] On the other hand, the present invention also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the transformer model processing method as described above.

[0018] In addition, the present invention also provides a device, characterized in that the computer device includes a processor and a memory, the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor and executes the above-mentioned transformer model processing method.

[0019] Compared with the prior art, the present invention has the following beneficial effects: the present invention automatically confirms the sub-layers that need to be deleted or retained through forward propagation and back propagation, that is, the weights obtained through this training are used to calculate the sub-layers that need to be deleted or retained in the next training, thereby reducing the number of parameters of the model and realizing a lightweight transformer model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0021] Figure 1 This is a flowchart of the transformer model processing method in Example 1. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0023] Name explanation:

[0024] Encoder: Encoder

[0025] Decoder: Decoder

[0026] MHA sublayer: multi-head attention sublayer;

[0027] FFN sublayer feedforward neural network sublayer

[0028] Example 1

[0029] This embodiment provides a transformer model processing method, such as Figure 1 As shown, the steps include:

[0030] S1. During training, the target layer is calculated based on the sub-layer weights of the transformer model. During the next training, the target layer is deleted or retained.

[0031] Furthermore, step S1 includes:

[0032] S11. Perform matrix operations on the two sub-layers in the encoder structure of the Transformer model and the last two sub-layers in the decoder structure, based on the weights associated with the sub-layer outputs.

[0033] The weight associated with the sub-layer output is the nearest weight W of the sub-layer output i ∈R d1*d2 , suppose the matrix u1∈R is a matrix whose values are all 1 1*d1 and u2∈R d2*1 , do matrix operations: n i =u1W i u2.

[0034] S12. Select the minimum value in the matrix operation, take the layer where the minimum value is located as the target layer, obtain different target layers for each sub-layer respectively, and delete the target layer in the next training.

[0035] Specifically, after matrix operation, 6 values are obtained (because there are 6 layers in total, there are 6 n i ), the 6 n i For comparison, the smallest n i The layer is the target layer L t . Obtain different L for each sub-layer respectively t , and then delete the target layer during the next training.

[0036] S13. In the first MHA sublayer of the Decoder of the Transformer model, perform matrix operations based on the weights associated with the sublayer output. For details about matrix operations in this step, please refer to step S11.

[0037] S14. Select the maximum value in the matrix operation, and take the layer where the maximum value is located as the target layer. In the next training, retain the target layer and delete other sub-layers.

[0038] Specifically, 6 n are calculated by step S13. i Then, the 6 n i For comparison, the largest n i The layer where it is located is the target of the first MHA sub-layer of the Decoder. Only one target layer L is retained in the next training. t , delete other sublayers.

[0039] In addition, during initial training, the default target layer L t =0, that is, delete or keep the first sublayer.

[0040] For the relevant steps of training in this embodiment, please refer to the prior art and will not be described in detail here.

[0041] In summary, this implementation uses forward and backpropagation to automatically adjust which sublayers to remove or retain. Specifically, the weights obtained during the current training are used to calculate which sublayers to remove or retain during the next training. For the two sublayers in the encoder structure and the last two sublayers in the decoder structure (the second MHA sublayer and the FFN sublayer), only one sublayer is removed at a time. For the first MHA sublayer in the decoder structure, only one sublayer is retained at a time, while the other five are removed. This reduces the number of model parameters, resulting in a lightweight transformer model.

[0042] Example 2

[0043] This embodiment provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the transformer model processing method of Example 1.

[0044] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk, etc. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM).

[0045] Example 3

[0046] This embodiment provides a device, which can be a computer device or a mobile terminal device, such as a mobile phone, a tablet computer, etc., including a processor and a memory, wherein the memory stores program code, and the processor executes the program code to execute the transformer model processing method of Example 1.

[0047] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0048] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A transformer model processing method, characterized in that: Including steps: During training, the target layer is calculated based on the sub-layer weights of the transformer model, and the target layer is deleted or retained in the next training; The transformer calculates the target layer based on the sub-layer weights of the transformer model. The steps to delete or retain the target layer include: In the two sub-layers of the Transformer model's encoder structure and the last two sub-layers of the decoder structure, matrix operations are performed based on the weights associated with the sub-layer outputs. The last two sub-layers are: the second MHA sub-layer and the FFN sub-layer. The weight associated with the sub-layer output is: the most recent weight of the sub-layer output. Select the minimum value in the matrix operation, take the layer where the minimum value is located as the target layer, obtain different target layers for each sub-layer, and delete the target layer during the next training.

2. The Transformer model processing method according to claim 1, characterized in that: According to the sub-layer weights of the transformer model, the target layer is calculated and the steps of deleting or retaining the target layer include: In the first MHA sublayer of the Transformer model's decoder, matrix operations are performed based on the weights associated with the sublayer's outputs. Select the maximum value in the matrix operation, and use the layer where the maximum value is located as the target layer. In the next training, retain the target layer and delete other sublayers.

3. The transformer model processing method according to claim 1, characterized in that During initial training, delete or keep the first sublayer.

4. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the transformer model processing method according to any one of claims 1 to 3.

5. A device, characterized in that The device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the transformer model processing method according to any one of claims 1 to 3.

Images