Improved Transform model and text translation method and system
By introducing an adaptive long and short-head attention mechanism module into the Transformer model, the input vectors are divided inequality, which solves the problems of information loss and feature overlap in the multi-head attention mechanism, and improves the accuracy and convergence speed of the model.
Patent Information
- Application Number
- CN202411369112.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-12-20
AI Technical Summary
The multi-head attention mechanism forces equally divides the input word vector dimensions in the Transformer model, resulting in the loss of information at high and low latitudes, and cannot guarantee non-overlapping and mutual opposite sex, which affects model optimization and accuracy.
The adaptive long and short head attention mechanism module is introduced to divide the dimensions of the input vector inequality, assign the dimensions of each attention head through formulas, and calculate the Attention separately, enhancing the diversity of feature information and the diversity of dimension expression.
The accuracy and convergence speed of the model are improved, information loss and feature overlap are avoided, and information extraction capabilities of the attention mechanism are enhanced.
Smart Images
Figure CN119167930A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of natural language processing, and in particular relates to an improved Transformer model, a text translation method and a system. Background Art
[0002] At present, all major classic models in the field of natural language processing are based on the Transformer model architecture, of which the multi-head attention mechanism module is a crucial module.
[0003] In 2014, the Google Mind team used the Attention mechanism on the RNN model for the first time to perform image classification. The attention mechanism is a way to mimic human attention and is used to focus the model on a specific part of the input sequence when processing sequence data. In the basic attention mechanism, for a given query and a set of key-value pairs, the similarity between the query and each key is calculated, and then the similarity is converted into weights. Finally, these weights are used to perform weighted summation of the values to generate an attention vector.
[0004] When Ashish Vaswani et al. proposed the Transformer architecture, they first introduced the multi-head attention module. Compared with the traditional attention mechanism, the multi-head attention mechanism divides the dimensions of the Q, K, and V vectors into n equal parts, and each part is calculated separately using the attention mechanism to achieve the purpose of extracting different feature information. However, although the multi-head attention mechanism avoids the singleness of the calculation of the attention vector, it divides the input word vector dimension into n parts, calculates Q, K, V and the attention vector separately for each part, extracts as much diverse feature information as possible, and pays attention to the subtle relationship between word vectors, which can make the model detection more accurate to a certain extent. However, it also ignores the diversity of information expressed by dimensions of different lengths, and splits the integrity of the information originally expressed by a certain length dimension. Forcing the dimensions to be equally divided will cause the loss of high (low) dimensional information, and it cannot guarantee the non-overlapping and mutually different validity of the features extracted by the n equally divided dimensions, which is not conducive to the optimization of the model itself. Summary of the invention
[0005] The present invention provides an improved Transformer model, a text translation method and a system to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:
[0006] An improved Transformer model, the improved Transformer model comprising an adaptive long-short head attention mechanism module, the adaptive long-short head attention mechanism module divides the dimension of an input vector into unequal parts.
[0007] Furthermore, the specific method of the adaptive long-short head attention mechanism module to divide the dimension of the input vector into unequal parts is:
[0008] For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula:
[0009] i∈[1,n]
[0010] if i≤n-1:
[0011]
[0012] if i=n:
[0013]
[0014] In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension;
[0015] For each attention head, calculate Attention separately:
[0016]
[0017] Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
[0018]
[0019] A text translation method based on an improved Transformer model, comprising:
[0020] Parse the Multi30k machine translation dataset and divide it into training set, validation set, and test set.
[0021] Perform word segmentation and word frequency statistics on English and German texts respectively;
[0022] According to the frequency of words, the word segmentation is encoded according to the index to build a word list;
[0023] Map the original character text into digital encoding according to the vocabulary;
[0024] Constructing an improved Transformer model, wherein the improved Transformer model includes an adaptive long-short head attention mechanism module, and the adaptive long-short head attention mechanism module divides the dimension of the input vector into unequal parts;
[0025] Training the improved Transformer model;
[0026] The text translation task is performed by using the improved Transformer model.
[0027] Furthermore, the specific method of the adaptive long-short head attention mechanism module in the constructed improved Transformer model to divide the dimension of the input vector into unequal parts is:
[0028] For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula:
[0029] i∈[1,n]
[0030] if i≤n-1:
[0031]
[0032] if i=n:
[0033]
[0034] In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension;
[0035] For each attention head, calculate Attention separately:
[0036]
[0037] Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
[0038]
[0039] Furthermore, the dataset is divided into training set, validation set, and test set according to the ratio of 7:2:1.
[0040] A text translation system based on an improved Transformer model, comprising:
[0041] The data partitioning module is used to parse the Multi30k machine translation dataset and divide it into training set, validation set, and test set.
[0042] A data processing module is used to perform word segmentation and word frequency statistics on English text and German text respectively;
[0043] A vocabulary building module is used to encode the word segments according to the index based on the word frequency to build a vocabulary;
[0044] A mapping module is used to map the original character text into a digital code according to the vocabulary;
[0045] A construction module is used to construct an improved Transformer model, wherein the improved Transformer model includes an adaptive long-short head attention mechanism module, and the adaptive long-short head attention mechanism module divides the dimension of the input vector into unequal parts;
[0046] The improved Transformer model is trained, and a text translation task is performed by using the trained improved Transformer model.
[0047] Furthermore, the specific method for the adaptive long-short head attention mechanism module in the improved Transformer model constructed by the building module to divide the dimension of the input vector into unequal parts is:
[0048] For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula:
[0049] i∈[1,n]
[0050] if i≤n-1:
[0051]
[0052] if i=n:
[0053]
[0054] In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension;
[0055] For each attention head, calculate Attention separately:
[0056]
[0057] Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
[0058]
[0059] Furthermore, the data set is divided into a training set, a validation set, and a test set according to a ratio of 7:2:1 through the data partitioning module.
[0060] The benefit of the present invention lies in the improved Transformer model, text translation method and system provided, which improve the Transformer model by replacing the multi-head attention mechanism module in the Transformer model with an adaptive long-short head attention mechanism module, giving multiple attention heads dimensions of different lengths, so that the model can extract feature information between dimensions of different lengths, thereby achieving the purpose of improving accuracy and accelerating convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0062] Figure 1 is a schematic diagram of an improved Transformer model of the present invention;
[0063] Figure 2 It is a schematic diagram of an adaptive long-short head attention mechanism module of an improved Transformer model of the present invention. DETAILED DESCRIPTION
[0064] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0065] like Figure 1 An improved Transformer model of the present application is shown, and the improved Transformer model includes an adaptive long-short head attention mechanism module. The adaptive long-short head attention mechanism (Adaptive Long-short headattention mechanism, ALS-HAM) module divides the dimension of the input vector into unequal parts.
[0066] Specifically, Figure 2 The figure shows a schematic diagram of the adaptive long-short head attention mechanism module of the present application. The specific method of the adaptive long-short head attention mechanism module of the present application to divide the dimension of the input vector into unequal parts is:
[0067] For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula:
[0068] i∈[1,n]
[0069] if i≤n-1:
[0070]
[0071] if i=n:
[0072]
[0073] In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension.
[0074] For each attention head, calculate Attention separately:
[0075]
[0076] Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
[0077]
[0078] For example, take the 8-head attention mechanism with d=512 and n=8 as an example; the original 8-head attention mechanism divides d into 8 parts, each with 64 dimensions. After the adaptive long-short head attention mechanism of this application is adopted, the division method is as follows:
[0079] Dimension of the first attention head:
[0080]
[0081] Dimension of the second attention head:
[0082]
[0083] The dimension of the third attention head:
[0084]
[0085] Dimension of the fourth attention head:
[0086]
[0087] The dimension of the fifth attention head:
[0088]
[0089] Dimension of the sixth attention head:
[0090]
[0091] Dimension of the seventh attention head:
[0092]
[0093] The dimension of the eighth attention head:
[0094]
[0095] At this time, the sum of the dimensions of all attention heads (128×2+64×2+32×2+16+48) is exactly equal to 512. The long-short head attention mechanism module proposed in this application abandons the traditional equal division mode, divides the multi-head dimensions of the attention mechanism from high dimension to low dimension, and enhances the diversity of attention extraction information and the diversity of information expression in different dimensions.
[0096] The present application also discloses a text translation method based on an improved Transformer model, comprising:
[0097] S1: Parse the Multi30k machine translation dataset and divide it into training set, validation set, and test set. Specifically, divide the dataset into training set, validation set, and test set according to the ratio of 7:2:1.
[0098] S2: Perform word segmentation and word frequency statistics on the English text and German text respectively.
[0099] S3: Encode the word segments according to the frequency and index them to build a vocabulary.
[0100] S4: Map the original character text into digital encoding according to the vocabulary.
[0101] S5: Build an improved Transformer model. The improved Transformer model includes an adaptive long-short head attention mechanism module. The adaptive long-short head attention mechanism module divides the dimensions of the input vector into unequal parts.
[0102] S6: Train the improved Transformer model.
[0103] S7: Performing a text translation task by improving the Transformer model. In this application, specifically, a German-English machine translation task is performed.
[0104] In the implementation manner of the present application, the specific method for the adaptive long-short head attention mechanism module in the constructed improved Transformer model to divide the dimension of the input vector into unequal parts is:
[0105] For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula:
[0106] i∈[1,n]
[0107] if i≤n-1:
[0108]
[0109] if i=n:
[0110]
[0111] In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension.
[0112] For each attention head, calculate Attention separately:
[0113]
[0114] Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
[0115]
[0116] The present application also discloses a text translation system based on an improved Transformer model, comprising: a data partitioning module, a data processing module, a vocabulary building module, a mapping module and a building module.
[0117] The data partitioning module is used to parse the Multi30k machine translation data set and divide it into a training set, a validation set, and a test set. Optionally, the data partitioning module divides the data set into a training set, a validation set, and a test set according to a ratio of 7:2:1. The data processing module is used to perform word segmentation and word frequency statistics on English text and German text respectively. The vocabulary construction module is used to encode the word segmentation according to the index according to the word frequency to construct a vocabulary. The mapping module is used to map the original character text into a digital encoding according to the vocabulary. The construction module is used to construct an improved Transformer model, and the improved Transformer model includes an adaptive long-short head attention mechanism module, and the adaptive long-short head attention mechanism module divides the dimension of the input vector into unequal parts. The improved Transformer model is trained, and the text translation task is performed by the trained improved Transformer model. In the present application, specifically, the German-English machine translation task is performed.
[0118] In the implementation manner of the present application, the specific method of dividing the dimension of the input vector into unequal parts by the adaptive long-short head attention mechanism module in the improved Transformer model constructed by the building module is:
[0119] For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula:
[0120] i∈[1,n]
[0121] if i≤n-1:
[0122]
[0123] if i=n:
[0124]
[0125] In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension.
[0126] For each attention head, calculate Attention separately:
[0127]
[0128] Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
[0129]
[0130] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.
Claims
1. An improved Transformer model, characterized in that: The improved Transformer model includes an adaptive long-short head attention mechanism module, which divides the dimension of the input vector into unequal parts.
2. The improved Transformer model according to claim 1, characterized in that The specific method of the adaptive long-short head attention mechanism module to divide the dimension of the input vector into unequal parts is: For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula: i∈[1,n] ifi≤n-1: ifi=n: In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension; For each attention head, calculate Attention separately: Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
3. A text translation method based on an improved Transformer model, characterized in that: Include: Parse the Multi30k machine translation dataset and divide it into training set, validation set, and test set. Perform word segmentation and word frequency statistics on English and German texts respectively; According to the frequency of words, the word segmentation is encoded according to the index to build a word list; Map the original character text into digital encoding according to the vocabulary; Constructing an improved Transformer model, wherein the improved Transformer model includes an adaptive long-short head attention mechanism module, and the adaptive long-short head attention mechanism module divides the dimension of the input vector into unequal parts; Training the improved Transformer model; The text translation task is performed by using the improved Transformer model.
4. The text translation method based on the improved Transformer model according to claim 3, characterized in that: The specific method of the adaptive long-short head attention mechanism module in the improved Transformer model to divide the dimensions of the input vector into unequal parts is: For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula: i∈[1,n] ifi≤n-1: ifi=n: In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension; For each attention head, calculate Attention separately: Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
5. The text translation method based on the improved Transformer model according to claim 4, characterized in that: The data set is divided into training set, validation set, and test set according to the ratio of 7:2:
1.
6. A text translation system based on an improved Transformer model, characterized in that: Include: The data partitioning module is used to parse the Multi30k machine translation dataset and divide it into training set, validation set, and test set. A data processing module is used to perform word segmentation and word frequency statistics on English text and German text respectively; A vocabulary building module is used to encode the word segments according to the frequency of the words and then build a vocabulary; A mapping module is used to map the original character text into a digital code according to the vocabulary; A construction module is used to construct an improved Transformer model, wherein the improved Transformer model includes an adaptive long-short head attention mechanism module, and the adaptive long-short head attention mechanism module divides the dimension of the input vector into unequal parts; The improved Transformer model is trained, and a text translation task is performed by using the trained improved Transformer model.
7. The text translation method based on the improved Transformer model according to claim 6, characterized in that: The specific method for the adaptive long-short head attention mechanism module in the improved Transformer model constructed by the building module to divide the dimension of the input vector into unequal parts is: For the input feature vector x(b,l,d), where batch, length, and dims represent the vector dimension, it is divided into an n-head attention mechanism, and the dimension of each attention head is divided according to the following formula: i∈[1,n] ifi≤n-1: ifi=n: In the above formula, i represents the i-th attention head, n represents the number of attention heads, represents the dimension of the i-th attention head, and d represents the vector dimension; For each attention head, calculate Attention separately: Then, the Attention of all heads is superimposed to obtain the final attention mechanism:
8. The text translation method based on the improved Transformer model according to claim 6, characterized in that: The data partitioning module is used to partition the data set into a training set, a validation set, and a test set according to a ratio of 7:2:1.