A semantic recognition method and system based on relative position encoding

Abstract

This invention proposes a semantic recognition method and system based on relative position encoding, comprising: acquiring a text sequence of text to be semantically recognized as training data; splitting the training data and inputting it in parallel into a multi-head attention layer; the multi-head attention layer assigning attention scores to each split data by calculating the relative position of the split data in the training data, and generating relative position encodings for each split data based on the attention scores; a recurrent neural network performing semantic recognition of each split data in parallel according to the relative position encodings of each split data; and aggregating the semantic recognition results corresponding to each split data to obtain the semantic recognition result.

Description

Technical Field

[0001] This invention relates to the field of semantic recognition technology, and in particular to a semantic recognition method and system based on relative position encoding. Background Technology

[0002] In any language, the position and order of words in a sentence are crucial. They are not only components of the sentence's grammatical structure but also essential concepts for expressing meaning. A word's position or order within a sentence can distort its meaning. For example, if a neural network model cannot distinguish between multiple occurrences of the same word in a sentence and mistakenly identifies them as identical, its output is likely to be incorrect. Introducing word order information helps in understanding the sentence's intended meaning; this word order information is represented by position encoding (Embedding, PE).

[0003] In traditional Recurrent Neural Networks (RNNs), words in an input sequence (such as a sentence) are processed one by one according to their position in the sequence, with the RNN processing one word at each time step. Although the representation vectors themselves (such as word embedding vectors) do not carry any positional information, because the state vectors at each time step are different, the RNN can still distinguish between consecutive words even when the same word appears at different time steps, thus producing different outputs. We can consider that the state vector implicitly contains positional information in the input sequence. In fact, it contains syntactic and semantic information between words input at past time steps, and this information has an effect on subsequent input words and outputs that is equivalent to the effect of positional information.

[0004] To avoid the training complexity caused by the recursive mechanism of RNNs, the Transformer architecture employs a multi-head attention mechanism. When a sequence enters the multi-head attention sequence, all words are processed in parallel, which greatly improves training speed. Without positional information, identical words in the sequence would have no syntactic or semantic difference for the multi-head attention (self-attention) module, producing the same output. Therefore, positional information for each word needs to be manually added to the input sequence. This positional information acts similarly to the time step in an RNN.

[0005] Because positional information is generated as a vector, or rather, an encoded form, it can be added to (or used in other operations) the embedding vector in the input sequence. Therefore, positional information is also called positional encoding or positional embedding vector. Note that each position in the input sequence has its own positional encoding, which is not a scalar but a multi-dimensional vector. As follows... Figure 1 As shown, there are generally two ways to add positional encoding: one is to add it between the embedding and self-attention. The positional encoding vector can be directly added element-wise to the word embedding vector of the input sequence before being input into the Self-Attention module; this can be called external positional encoding, or static positional encoding. The other is to add it inside self-attention. The positional encoding vector is generated through training, and when the Self-Attention module is trained, this positional vector is also trained. Before training ends, the value of this positional encoding is not fixed; therefore, this built-in positional encoding can also be called dynamic positional encoding. Summary of the Invention

[0006] Specifically, to address the aforementioned technical problems, this invention proposes a semantic recognition method based on relative position encoding, which includes:

[0007] Step 1: Construct a recurrent neural network including a multi-head attention layer. Obtain the text sequence of labeled semantic recognition results as training data. Split the training data and input it into the multi-head attention layer in parallel. The multi-head attention layer calculates the relative position of the split data in the training data, assigns attention scores to each split data, and generates the relative position code of each split data based on the attention scores corresponding to each split data.

[0008] Step 2: The recurrent neural network encodes the relative positions of each split data according to the data, performs parallel semantic recognition of each split data, aggregates the semantic recognition results of each split data, obtains the semantic recognition results of the training data, and generates a loss function by comparing it with the labeled semantic recognition results. The recurrent neural network is iteratively trained until the loss function converges or reaches the preset number of iterations, and the current recurrent neural network is saved as a semantic recognition model.

[0009] Step 3: Input the text sequence to be semantically recognized into the semantic recognition model to obtain the semantic recognition result of the text sequence to be semantically recognized.

[0010] The semantic recognition method based on relative position encoding, wherein the multi-head attention layer assigns attention scores to the split data through the following:

[0011] Step 11: Calculate Q, K, and A according to the following formula:

[0012] Q = X * W q →Q[n,T,h,64]

[0013] K = X * W k →K[n, T, h, 64]

[0014] A[T,T,64]select←posTable[2k+1,64]

[0015] Where Q is the query; K is the key; A is the relative position code; A is obtained by looking up the RPR pos_tab[2k+1, 64] weight matrix; X: [n, T, d a ], W q : [h*64, d a ], W k : [h*64, d a ], posTable: [2k+1, 64], n represents the batch size of the input data sequence, T represents the length of the data sequence, d a The dimension of the embedding vector of the data sequence is represented, h represents the number of heads in the multi-head attention layer, and 64 is the head length in the multi-head attention layer;

[0016] Step 12: Perform matrix multiplication on Q and K respectively, and add the matrix multiplication results to obtain the attention score: AttentionscoreE = Q*K + Q*A.

[0017] The semantic recognition method based on relative position encoding, wherein the calculation process of Q*K is as follows:

[0018] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0019]

[0020] In the formula, trans represents changing the order in which the tensor data is arranged.

[0021] 4. The semantic recognition method based on relative position encoding as described in claim 2 or 3, wherein the calculation process of Q*A is as follows:

[0022] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0023] Q*A: Q t [n,h,T,64]*posTable[2k+1,64]→[n*h,T,2k+1]select→[n,h,T,T]

[0024] In the formula, trans represents changing the order in which the tensor data is arranged.

[0025] This invention also proposes a semantic recognition system based on relative position encoding, which includes:

[0026] The initial module is used to construct a recurrent neural network including a multi-head attention layer. It obtains text sequences of labeled semantic recognition results as training data, splits the training data, and inputs them into the multi-head attention layer in parallel. The multi-head attention layer calculates the relative position of the split data in the training data, assigns attention scores to each split data, and generates the relative position encoding of each split data based on the attention scores corresponding to each split data.

[0027] The training module is used to enable the recurrent neural network to encode the relative positions of each split data, perform parallel semantic recognition of each split data, aggregate the semantic recognition results corresponding to each split data, obtain the semantic recognition results of the training data, and generate a loss function by comparing it with the labeled semantic recognition results. The recurrent neural network is iteratively trained until the loss function converges or reaches the preset number of iterations, and the current recurrent neural network is saved as a semantic recognition model.

[0028] The recognition module is used to input the text sequence to be semantically recognized into the semantic recognition model and obtain the semantic recognition result of the text sequence to be semantically recognized.

[0029] In the semantic recognition system based on relative position encoding, the multi-head attention layer assigns attention scores to the split data as follows:

[0030] The query module is used to obtain Q, K, and A according to the following formula:

[0031] Q = X * W a →Q[n, T, h, 64]

[0032] K = X * W k →K[n, T, h, 64]

[0033] A[T,T,64]select←posTable[2k+1,64]

[0034] Where Q is the query; K is the key; A is the relative position code; A is obtained by looking up the RPR pos_tab[2k+1, 64] weight matrix; X: [n, T, d a ], W q : [h*64, d a ], W k : [h*64, d a], posTable: [2k+1, 64], n represents the batch size of the input data sequence, T represents the length of the data sequence, d a The dimension of the embedding vector of the data sequence is represented, h represents the number of heads in the multi-head attention layer, and 64 is the head length in the multi-head attention layer;

[0035] The multiply-add module is used to perform matrix multiplication on Q and K respectively, and add the matrix multiplication results to obtain the attention score: AttentionscoreE=Q*K+Q*A.

[0036] In the semantic recognition system based on relative position encoding, the multiply-accumulate module is used to obtain Q*K according to the following formula:

[0037] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0038]

[0039] In the formula, trans represents changing the order in which the tensor data is arranged.

[0040] In the semantic recognition system based on relative position encoding, the multiply-accumulate module is used to obtain Q*A according to the following formula:

[0041] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0042] Q*A: Q t [n,h,T,64]*posTable[2k+1,64]→[n*h,T,2k+1]select→[n,h,T,T]

[0043] In the formula, trans represents changing the order in which the tensor data is arranged.

[0044] The present invention also proposes a storage medium for storing a program that executes any of the semantic recognition methods based on relative position encoding.

[0045] The present invention also proposes a client for any of the aforementioned semantic recognition systems based on relative position encoding.

[0046] As can be seen from the above solutions, the advantages of the present invention are:

[0047] 1. Reduce transpose by 2 (transpose before multiplying Q and A + transpose of the result of Q × A).

[0048] 2. The batchdot of [T,nh,64]*[T,T,64] is changed to [nh,T,64]*[2K+1,64], reducing the computational cost of matrix multiplication. Here, batch_dot is the terminology used in the TensorFlow framework, equivalent to batch_matmul, meaning a batch of matmuls.

[0049] In summary, this improves semantic recognition efficiency and recognition speed. Attached Figure Description

[0050] Figure 1 Insert a diagram for position encoding;

[0051] Figure 2 This illustrates the insertion position for relative position encoding in self-attention.

[0052] Figure 3 This is a diagram illustrating the relative positional relationships;

[0053] Figure 4 This is a schematic diagram of the RPR embedding lookup table;

[0054] Figure 5 This is a flowchart of the present invention. Detailed Implementation

[0055] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0056] The Transformer uses external static positional encoding, where the dimension of the encoding vector is d. model d model Determined by the parameters of the decoder model, and also the dimension of the embedding, the specific encoding formula for the position vector PE is as follows:

[0057]

[0058]

[0059] Where pos represents the position and i represents the dimension, 2i and 2i+1 are used to distinguish whether the dimension is an odd or even position. Odd and even positions are encoded differently. Then the word vector and the position vector are added together to obtain the final input for each word.

[0060] Relative position encoding:

[0061] 1. The Proposal of Relative Position Encoding

[0062] According to the definition of positional encoding, the attention score between the i-th word and the j-th word in a sequence is:

[0063]

[0064] Among them W q W k These are the query and key parameters added to each head by self-attention. and Is input x i and x j The word embedding vector, U i and U j It is the position vector of the i-th position and the j-th position.

[0065] The positional encoding method used by Transformers loses relative positional information during self-attention calculation and needs to be added back during the self-attention process. Specifically, a trainable parameter representing the relative position is added to both the attention score and weighted value calculations. This parameter is called Relative Position Representations (RPR). Note that RPRRembedding is shared between attention heads within the same layer, but the RPR may differ between different layers. The attention score and weighted value positions are shown below. Figure 2 As shown.

[0066] Among them, attention score ij and weighted value z i The calculation formula is as follows:

[0067]

[0068]

[0069] use x represents i and x j The relative position information depends only on the difference k between i and j. Specifically, as follows:

[0070]

[0071]

[0072] clip(x,k)=max(-k,min(k,x)) (5)

[0073] like Figure 3 As shown in the figure in the paper "Self-Attention with Relative Position Representations", the relative positional relationships in the sequence are represented quite vividly.

[0074] The maximum number of words is clipped to an absolute value k. Moving k words to the left, all words to the left of that value are 0; moving k words to the right, all words to the right of that value are k. This represents an index range of 2k+1. The RPR weight A[T,T,headsize] is derived from pos_tab[2*k+1,headsize]. For example, with K=3 and T=10, the RPR embedding lookup table is as follows: Figure 4 As shown.

[0075] 2. Optimization calculation of relative position:

[0076] Expanding equation (3) yields:

[0077]

[0078] The input tensors and their dimensions and shapes are as follows:

[0079] X: [n, T, d] a ], W q : [h*64, d a ], W k : [h*64, d a ] and posTable: [2k+1, 64], where n represents the batch of the input statement sequence, T represents the sentence length, and d a The dimension of the embedding vector is represented by , h represents the number of heads in the self-attention mechanism (head num), and 64 represents the head length (head_size). A is derived from the RPR pos_tab[2k+1, 64] weight matrix by looking up a table.

[0080] The calculation process is as follows:

[0081] Step 1: Calculate Q, K, and A. Q: query; K: keys; A: relative position encoding, obtained by querying and expanding the position-encoded embedding vector table; trans is short for transpose, which means changing the order of tensor data.

[0082] Q = X * W q →Q[n, T, h, 64]

[0083] K = X * W k →K[n, T, h, 64]

[0084] A[T,T,64]select←posTable[2k+1,64]

[0085] Arrows indicate the results of data operations, such as X and W. q Multiplying them yields Q with dimensions [n, T, h, 64]. Second step: two batch_matmuls

[0086]

[0087]

[0088] Step 3: Add the two results from step 2 together.

[0089] AttentionScoreE=Q*K+Q*A

[0090] Due to the maximum distance K truncation, there are many repeated elements in the RPR weights A, and the second term of equation (6) There is a lot of redundant computation, and the above calculation process involves multiple transposes. This solution proposes replacing the batch_matmul of Q and A in the second step with the batch_matmul of Q and the RPR pos_tab matrix. The second step of the above calculation is modified as follows:

[0091] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0092]

[0093] Q*A: Q t [n,h,T,64]*posTable[2k+1,64]→[n*h,T,2k+1]select→[n,h,T,T]

[0094] The present invention also proposes a relative position encoding system, comprising: an encoding module for constructing a recurrent neural network including a multi-head attention layer, acquiring a data sequence to be encoded in relative position, splitting the data sequence and inputting it in parallel into the multi-head attention layer, the multi-head attention layer calculating the relative position of the split data in the data sequence, assigning attention scores to each split data, and generating relative position codes for each split data based on the attention scores corresponding to each split data.

[0095] The relative position encoding method, wherein the multi-head attention layer assigns attention scores to the split data as follows:

[0096] Encoding module, used to calculate Q, K, A:

[0097] Q = X * W q →Q[n, T, h, 64]

[0098] K = X * W k →K[n, T, h, 64]

[0099] A[T,T,64]select←posTable[2k+1,64]

[0100] Where Q is the query; K is the key; A is the relative position code; A is obtained by looking up the RPR pos_tab[2k+1, 64] weight matrix; X: [n, T, d a ], W q : [h*64, d a ], W k : [h*64, d a ], posTable: [2k+1, 64], n represents the batch size of the input data sequence, T represents the length of the data sequence, d a The dimension of the embedding vector of the data sequence is represented, h represents the number of heads in the multi-head attention layer, and 64 is the head length in the multi-head attention layer;

[0101] The calculation module is used to perform matrix multiplication on Q and K respectively, and add the matrix multiplication results to obtain the attention score: AttentionscoreE=Q*K+Q*A.

[0102] The relative position encoding method, wherein the calculation module includes:

[0103] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0104]

[0105] In the formula, trans represents changing the order in which the tensor data is arranged.

[0106] The relative position encoding method, wherein the calculation module includes:

[0107] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0108] Q*A: Q t[n,h,T,64]*posTable[2k+1,64]→[n*h,T,2k+1]select→[n,h,T,T]

[0109] In the formula, trans represents changing the order in which the tensor data is arranged.

[0110] Specifically, such as Figure 5 As shown, this invention proposes a semantic recognition method based on relative position encoding, which includes:

[0111] Step 1: Construct a recurrent neural network including a multi-head attention layer. Obtain the text sequence of labeled semantic recognition results as training data. Split the training data and input it into the multi-head attention layer in parallel. The multi-head attention layer calculates the relative position of the split data in the training data, assigns attention scores to each split data, and generates the relative position code of each split data based on the attention scores corresponding to each split data.

[0112] Step 2: The recurrent neural network encodes the relative positions of each split data according to the data, performs parallel semantic recognition of each split data, aggregates the semantic recognition results of each split data, obtains the semantic recognition results of the training data, and generates a loss function by comparing it with the labeled semantic recognition results. The recurrent neural network is iteratively trained until the loss function converges or reaches the preset number of iterations, and the current recurrent neural network is saved as a semantic recognition model.

[0113] Step 3: Input the text sequence to be semantically recognized into the semantic recognition model to obtain the semantic recognition result of the text sequence to be semantically recognized.

[0114] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0115] This invention also proposes a semantic recognition system based on relative position encoding, which includes:

[0116] The initial module is used to construct a recurrent neural network including a multi-head attention layer. It obtains text sequences of labeled semantic recognition results as training data, splits the training data, and inputs them into the multi-head attention layer in parallel. The multi-head attention layer calculates the relative position of the split data in the training data, assigns attention scores to each split data, and generates the relative position encoding of each split data based on the attention scores corresponding to each split data.

[0117] The training module is used to enable the recurrent neural network to encode the relative positions of each split data, perform parallel semantic recognition of each split data, aggregate the semantic recognition results corresponding to each split data, obtain the semantic recognition results of the training data, and generate a loss function by comparing it with the labeled semantic recognition results. The recurrent neural network is iteratively trained until the loss function converges or reaches the preset number of iterations, and the current recurrent neural network is saved as a semantic recognition model.

[0118] The recognition module is used to input the text sequence to be semantically recognized into the semantic recognition model and obtain the semantic recognition result of the text sequence to be semantically recognized.

[0119] In the semantic recognition system based on relative position encoding, the multi-head attention layer assigns attention scores to the split data as follows:

[0120] The query module is used to obtain Q, K, and A according to the following formula:

[0121] Q = X * W q →Q[n, T, h, 64]

[0122] K = X * W k →K[n, T, h, 64]

[0123] A[T,T,64]select←posTable[2k+1,64]

[0124] Where Q is the query; K is the key; A is the relative position code; A is obtained by looking up the RPR pos_tab[2k+1, 64] weight matrix; X: [n, T, d a ], W q : [h*64, d a ], W k : [h*64, d a ], posTable: [2k+1, 64], n represents the batch size of the input data sequence, T represents the length of the data sequence, d a The dimension of the embedding vector of the data sequence is represented, h represents the number of heads in the multi-head attention layer, and 64 is the head length in the multi-head attention layer;

[0125] The multiply-add module is used to perform matrix multiplication on Q and K respectively, and add the matrix multiplication results to obtain the attention score: AttentionscoreE=Q*K+Q*A.

[0126] In the semantic recognition system based on relative position encoding, the multiply-accumulate module is used to obtain Q*K according to the following formula:

[0127] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0128]

[0129] In the formula, trans represents changing the order in which the tensor data is arranged.

[0130] In the semantic recognition system based on relative position encoding, the multiply-accumulate module is used to obtain Q*A according to the following formula:

[0131] Q[n, T, h, 64]trans→Q t [n, h, T, 64]

[0132] Q*A: Q t [n,h,T,64]*posTable[2k+1,64]→[n*h,T,2k+1]select→[n,h,T,T]

[0133] In the formula, trans represents changing the order in which the tensor data is arranged.

[0134] The present invention also proposes a storage medium for storing a program that executes any of the semantic recognition methods based on relative position encoding.

[0135] The present invention also proposes a client for any of the aforementioned semantic recognition systems based on relative position encoding.

Claims

1. A semantic recognition method based on relative position encoding, characterized in that, include: Step 1: Construct a recurrent neural network including a multi-head attention layer. Obtain the text sequence of labeled semantic recognition results as training data. Split the training data and input it into the multi-head attention layer in parallel. The multi-head attention layer calculates the relative position of the split data in the training data, assigns attention scores to each split data, and generates the relative position code of each split data based on the attention scores corresponding to each split data. Step 2: The recurrent neural network encodes the relative positions of each split data according to the data, performs parallel semantic recognition of each split data, aggregates the semantic recognition results of each split data, obtains the semantic recognition results of the training data, and generates a loss function by comparing it with the labeled semantic recognition results. The recurrent neural network is iteratively trained until the loss function converges or reaches the preset number of iterations, and the current recurrent neural network is saved as a semantic recognition model. Step 3: Input the text sequence to be semantically recognized into the semantic recognition model to obtain the semantic recognition result of the text sequence to be semantically recognized; The multi-head attention layer assigns attention scores to the split data as follows: Step 11: Calculate Q, K, and A according to the following formula: Q=X*W q →Q[n,T,h,64] K=X*W k →K[n,T,h,64] A[T,T,64]select←posTable[2k+1,64] Where Q is the query; K is the key; A is the relative position code; A is obtained by looking up the RPR pos_tab[2k+1,64] weight matrix, and X:[n,T,d] a ], W q :[h*64,d a ], W k :[h*64,d a ], posTable:[2k+1,64], n represents the batch size of the input data sequence, T represents the length of the data sequence, d a The dimension of the embedding vector of the data sequence is represented, h represents the number of heads in the multi-head attention layer, and 64 is the head length in the multi-head attention layer; Step 12: Perform matrix multiplication on Q and K respectively, and add the matrix multiplication results to obtain the attention score: AttentionscoreE = Q*K + Q*A; The calculation process for Q*K is as follows: Q[n,T,h,64]trans→Q t [n,h,T,64] In the formula, trans represents changing the order in which the tensor data is arranged; The calculation process for Q*A is as follows: Q[n,T,h,64]trans→Q t [n,h,T,64] Q*A:Q t [n,h,T,64]*posTable[2k+1,64]→[n*h,T,2k+1]select→[n,h,T,T] In the formula, trans represents changing the order in which the tensor data is arranged.

2. A semantic recognition system based on relative position encoding, characterized in that, include: The initial module is used to construct a recurrent neural network including a multi-head attention layer. It obtains text sequences of labeled semantic recognition results as training data, splits the training data, and inputs them into the multi-head attention layer in parallel. The multi-head attention layer calculates the relative position of the split data in the training data, assigns attention scores to each split data, and generates the relative position encoding of each split data based on the attention scores corresponding to each split data. The training module is used to enable the recurrent neural network to encode the relative positions of each split data, perform parallel semantic recognition of each split data, aggregate the semantic recognition results corresponding to each split data, obtain the semantic recognition results of the training data, and generate a loss function by comparing it with the labeled semantic recognition results. The recurrent neural network is iteratively trained until the loss function converges or reaches the preset number of iterations, and the current recurrent neural network is saved as a semantic recognition model. The recognition module is used to input the text sequence to be semantically recognized into the semantic recognition model and obtain the semantic recognition result of the text sequence to be semantically recognized. This multi-head attention layer assigns attention scores to the split data as follows: The query module is used to obtain Q, K, and A according to the following formula: Q=X*W q →Q[m,T,h,64] K=X*W k →K[n,T,h,64] A[T,T,64]select←posTable2k+1,64] Where Q is the query; K is the key; A is the relative position code; A is obtained by looking up the RPR pos_tab[2k+1,64] weight matrix, and X:[n,T,d] a ], W q :[h*64,d a ], W k :[h*64,d a ], posTable:[2k+1,64], n represents the batch size of the input data sequence, T represents the length of the data sequence, d a The dimension of the embedding vector of the data sequence is represented, h represents the number of heads in the multi-head attention layer, and 64 is the head length in the multi-head attention layer; The multiply-add module is used to perform matrix multiplication on Q and K respectively, and add the matrix multiplication results to obtain the attention score: AttentionscoreE = Q*K + Q*A; This multiply-accumulate module is used to obtain Q*K according to the following formula: Q[n,T,h,64]trans→Q t [n,h,T,64] In the formula, trans represents changing the order in which the tensor data is arranged; This multiply-accumulate module is used to obtain Q*A according to the following formula: Q[n,T,h,64]trans→Qt[n,h,T,64] Q*A:Q t [n,h,T,64]*posTable[2k+1,64]→[n*h,T,2k+1]select→[n,h,T,T] In the formula, trans represents changing the order in which the tensor data is arranged.

3. A storage medium for storing a program that executes the semantic recognition method based on relative position encoding as described in claim 1.

4. A client for the semantic recognition system based on relative position encoding as described in claim 2.

Images