A summary generation method based on Transformer

Abstract

The present application discloses a summary generation method based on Transformer, which relates to the technical field of summary generation, including: obtaining a text sequence; establishing a language model to obtain a word embedding matrix; inputting the word embedding matrix into a position encoding model for position encoding to obtain a word embedding matrix with position information added; constructing a Transformer model to generate a summary; the Transformer model includes an encoder and a decoder, the encoder encodes the semantic information of the input text sequence through a multi-layer attention mechanism; the decoder generates a summary sequence through a multi-layer attention mechanism; the decoder includes a self-attention layer, in which an occlusion vector is obtained through a fully connected network, and a dot product calculation is performed between the occlusion vector and the key value representation inside the decoder to obtain an adjusted attention weight distribution; a residual connection is established between the encoder and the decoder, and the semantic features output by the encoder are directly input to the decoder. In view of the problem of low accuracy in generating summaries in the prior art, the present application improves the accuracy of summaries.

Description

Technical Field

[0001] The present application relates to the technical field of abstract generation, and in particular to a method for abstract generation based on Transformer. Background Art

[0002] With the rapid development of the Internet, network information has exploded. Massive text data has made manual browsing and extraction of key information a bottleneck. In order to improve the efficiency of information acquisition, automatic text summarization technology has emerged. Text summarization is to compress and extract the original text to generate a short text that can represent the main content of the original text. It can be widely used in news headline generation, document writing, information retrieval and other fields.

[0003] Early text summarization methods were mainly based on extraction and abstraction, relying on manually constructed features and rules. The summaries generated by such methods are fragmented and incoherent. In recent years, sequence-to-sequence models have been gradually applied to text summarization tasks. Their encoder-decoder framework uses a neural network model to learn the intrinsic semantic representation of text end-to-end. However, when processing long texts, the encoder of existing sequence-to-sequence models cannot fully learn semantic information, and the summary generated by the decoder lacks key details.

[0004] In the related art, for example, Chinese patent document CN116501863A provides a method and device for generating text summaries based on natural language processing, including: using the linear layer in the pre-built Fastformer model to analyze and split the text, pre-encoding the query sequence, key sequence and value sequence based on the Fastformer model, importing the vocabulary embedding vector into the pre-built pointer generation network model, optimizing the attention probability distribution through the pre-built coverage vector, calculating the summary vocabulary probability of the target word according to the summary vocabulary probability model, copying or constructing vocabulary from the text based on the summary vocabulary probability model, and generating a text summary. However, this scheme uses the pre-built Fastformer model to encode the text, but as a simplified Transformer structure, Fastformer has relatively weak encoding capabilities and cannot fully learn the semantic information of the text. Therefore, the accuracy of the summary generated by this scheme needs to be further improved. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In response to the problem of low accuracy in summarization generation in the prior art, the present application provides a summary generation method based on Transformer, which performs semantic modeling on the input text through multi-layer attention and generates a semantically coherent text summary using the occluded self-attention mechanism.

[0007] 2. Technical solution

[0008] The purpose of this application is achieved through the following technical solutions.

[0009] The embodiment of the present specification provides a Transformer-based summary generation method, including: obtaining a text sequence; establishing a language model, performing word embedding representation on the input text sequence, and obtaining a word embedding matrix; inputting the word embedding matrix into a position encoding model for position encoding, and obtaining a word embedding matrix with position information added; constructing a Transformer model, taking the word embedding matrix with position information added as input, and using the Transformer model to generate a summary of the input text sequence; the Transformer model includes an encoder and a decoder, the encoder encodes the semantic information of the input text sequence through a multi-layer attention mechanism; the decoder generates a summary sequence through a multi-layer attention mechanism; the decoder includes a self-attention layer, in the self-attention layer, an occlusion vector is obtained through a fully connected network, and a dot product calculation is performed between the occlusion vector and the key-value representation inside the decoder to obtain an adjusted attention weight distribution; a residual connection is established between the encoder and the decoder, and the semantic features output by the encoder are directly input to the decoder to transmit semantic information; the decoder outputs the word symbol sequence of the summary in sequence according to the time step to form a final summary.

[0010] Among them, the text sequence refers to the text to be processed that is input into the model, which can generally be a single sentence or a chapter composed of multiple sentences. In the summary generation task, the text sequence is the source text for which the summary needs to be generated. Word embedding is a method of dense vector representation. By mapping each word in the text to a dense vector, semantically similar words are closer in the vector space. The word embedding matrix is ​​a matrix that organizes the word embedding vectors of all words in a text sequence. In this application, the text sequence refers to the input chapter, which needs to be summarized. The word embedding representation of the input text sequence is to convert each word in the text into a dense vector to form a word embedding matrix, which provides a digital text representation for the input layer of the sequence to sequence model. The word embedding matrix carries the semantic information of the text and provides a basis for subsequent models to learn the semantic features of the text.

[0011] Among them, position encoding is to add a code that represents the position information of each element in the sequence in the sequence. It provides the model with hints of the sequence order by adding position information to the word embedding, helping the model learn position-related features. In this application, the word embedding matrix is ​​input into the position encoding model for position encoding, which is to add the position information of the word in the text sequence to the word vector. Position encoding provides the model with word order information, helping the model learn the contextual relationship between words in the word sequence, as well as the relative position relationship in the sequence. The word embedding matrix after adding position encoding is used as the input of the Transformer model, which can enhance the model's perception of the sequential structure of the text sequence. The word embedding matrix with added position information finally represents the position information of the text symbol through position encoding, and provides sequential structure information for subsequent sequence modeling and summary generation.

[0012] Among them, the Transformer model is a sequence-to-sequence modeling network structure based on the attention mechanism. It models the sequence with an encoder-decoder architecture. In this application, a Transformer model is constructed as the core module for generating summaries. The Transformer model takes a word embedding matrix with position encoding as input. The encoder analyzes the semantics and contextual information of the input text sequence through a multi-head attention mechanism. The decoder also uses a multi-head attention mechanism to generate a coherent summary sequence. The Transformer's attention mechanism can effectively model long-range dependencies. Finally, the Transformer model completes the encoding-decoding process from the input text sequence to the output summary sequence.

[0013] Among them, the occlusion vector is a vector used in the self-attention mechanism of the decoder to adjust the attention weight and enhance the decoder's perception of semantic information. In the attention mechanism, the key value is the query and reference object for the input sequence to perform attention calculation. The key usually comes from the current input and the value comes from the entire input sequence. They calculate the attention weight through a dot product operation. In this application, in the self-attention layer of the decoder, an occlusion vector is generated by a fully connected network. The occlusion vector is dot-producted with the key-value representation inside the decoder to adjust the attention weight. The key-value representation comes from the intermediate state inside the decoder and contains semantic information. The introduction of the occlusion vector enhances the decoder's perception of semantic information. The adjusted attention weight distribution allows the decoder to focus more on key content. The combination of occlusion vectors and key-value representations improves the decoder's self-attention process and helps the decoder generate a more semantically coherent summary.

[0014] Among them, residual connection refers to the process of directly passing the output of the previous network to the input of the next network through SKIP Connection in a deep neural network to help gradient back propagation and information flow. Semantic features refer to the high-level feature representations of the encoder, which contain the semantic information of the source text. In this application, a residual connection is established between the encoder and the decoder. The output semantic features of the encoder are directly input to the decoder through the residual connection. The semantic features contain the semantic understanding of the source text, which is conducive to generating a semantically accurate summary. The residual connection directly provides the decoder with the semantic information extracted by the encoder. This can enhance the decoder's perception of the text semantics and generate a coherent and accurate summary.

[0015] Specifically, the decoder outputs the word symbol sequence of the summary in sequence according to the time step to form the final summary. The decoder adopts an automatic regression generation mechanism to output the target sequence in each time step. At each time step, the decoder generates the probability distribution of the next word through self-attention calculation based on the currently generated word sequence. The occluded self-attention mechanism is used to focus on key semantic information and generate related words. The word representation generated in the previous time step is concat to the input of the current time step to form the input of the decoder. Repeat this process until a special termination word is generated or the maximum length is reached to form a complete summary word sequence. The generated words of the decoder at each time step rely on the previous word sequence to form a coherent semantics and accurately reflect the semantics of the source text. Using the Teacher Forcing mechanism, the target word sequence can be used to train the generation ability of the decoder. Finally, the decoder sequentially generates an accurate and coherent summary word sequence to form a complete text summary.

[0016] Furthermore, obtaining a word embedding matrix includes: performing word segmentation processing on an input text sequence using a Jieba word segmentation algorithm to obtain a word symbol sequence; constructing a word embedding layer based on an attention mechanism, the word embedding layer includes a dictionary and a self-attention encoding sublayer, and the dictionary stores the word vector of each word; obtaining the word vector corresponding to the word symbol sequence according to the dictionary to form a word vector sequence; using the obtained word vector sequence as input, performing multi-head self-attention calculation using the self-attention encoding sublayer to obtain contextual semantic information of the word vector; the self-attention encoding sublayer uses a Gelu nonlinear activation function to perform a nonlinear transformation on the word vector sequence, and outputs a context-enhanced word vector sequence; constructing a word embedding matrix for the input text sequence according to the context-enhanced word vector sequence.

[0017] Among them, Jieba word segmentation algorithm is a Chinese word segmentation algorithm based on word frequency statistics, which can perform word segmentation on Chinese text and divide sentences into individual words. The word symbol sequence refers to the word order representation formed on the text after word segmentation, which is a linear sequence of word symbols, representing the linear word order of the text. In this application, Jieba word segmentation algorithm is used to segment the input text to obtain the word segmentation result. Jieba word segmentation can accurately segment Chinese sentences and obtain words. The word segmentation results are sorted to form a linear sequence of words, that is, a word symbol sequence. The word symbol sequence retains the word order information of the text. The word symbol sequence is used as the input of word embedding, and the text word order information is represented by vectorization. It provides input for subsequent word embedding and position encoding.

[0018] Among them, attention mechanism: a mechanism that enables a neural network to focus on relevant inputs, assigning different weights by calculating the correlation between inputs. Dictionary: a lookup table that stores the mapping relationship between each word in the text and its word vector. Self-attention encoding sublayer: a neural network layer that uses a self-attention mechanism to extract features and characterize the input sequence. In this application, a word embedding layer based on an attention mechanism is constructed to optimize the representation of word vectors. The dictionary stores the pre-trained word vectors of each word for search. The self-attention encoding sublayer analyzes the correlation between input word vectors based on the attention mechanism and integrates contextual information. Finally, the word vector enhanced by the self-attention mechanism is output to incorporate richer contextual semantic information. Enhanced word vectors are provided as input for subsequent semantic analysis and encoding.

[0019] Among them, multi-head self-attention is an attention mechanism used in Transformer, which learns different semantic components of text through multiple parallel head structures. Contextual semantic information refers to the semantic components related to surrounding words contained in the word vector. In this application, a multi-head self-attention calculation mechanism is used to encode the word vector sequence. Each head structure focuses on a kind of semantic information. The multi-head structure can learn different semantic components of the word vector. The representations of all head structures are spliced ​​and integrated to obtain comprehensive contextual semantic information. Contextual semantic information enriches the semantic expression ability of word vectors. It provides a more powerful input word vector representation for subsequent semantic modeling.

[0020] Among them, the Gelu nonlinear activation function is a smooth non-saturated activation function, and the formula is Gelu(x)=xΦ(x), where Φ(x) is the normal distribution function. Gelu can significantly improve the nonlinear expression ability of the model. In the present application, the self-attention encoding sublayer adopts the Gelu activation function for nonlinear transformation. The Gelu function increases the nonlinearity of word vector sequence modeling and enhances the model fitting ability. The smooth and non-saturated characteristics of Gelu enable word vectors to undergo more effective nonlinear transformations. The Gelu function acts on the output word vector of the self-attention sublayer. Through nonlinear transformation, the output word vector obtains a more advanced abstract representation. Finally, the context-enhanced word vector sequence is output, and the word vector expression ability is improved.

[0021] Specifically, according to the word vector sequence after context enhancement, the word embedding matrix of the input text sequence is constructed. The input text is segmented and vector searched to form an initial word vector sequence. The initial word vector sequence enters the self-attention encoding sublayer for encoding. The self-attention sublayer learns the contextual semantic information of the word vector through a multi-head self-attention mechanism. The output representations of different head structures are spliced ​​to obtain comprehensive contextual semantics. After nonlinear transformations such as Gelu, the word vector obtains a more advanced semantic abstract representation. The word vector sequence after context enhancement is obtained, and each word vector incorporates the semantics of the surrounding words. According to the word order after the input text is segmented, the context enhanced word vectors are sorted to form a matrix. The rows of the word vector matrix correspond to the word order of the input text, and the columns represent the word vector dimensions. The final word embedding matrix contains rich contextual semantic information. As the input of the sequence model, the word embedding matrix provides a powerful semantic feature representation.

[0022] Furthermore, the word symbol sequence is obtained, including: using a hidden Markov model to calculate the probability of a hidden state sequence corresponding to the input text sequence; searching the hidden state sequence through a Viterbi algorithm to obtain a hidden state sequence corresponding to the maximum probability; and obtaining the word symbol sequence based on the hidden state sequence corresponding to the obtained maximum probability.

[0023] Among them, the hidden Markov model is a statistical Markov model used to describe a Markov process containing hidden states. It assumes that the transition probability between visible states depends on a hidden Markov chain. Hidden state sequence probability: refers to the use of a hidden Markov model to calculate the joint probability of a hidden state sequence corresponding to a given observation sequence. In this application, a hidden Markov model is used to describe the correspondence between a text word sequence and a part-of-speech tag sequence. Calculate the joint probability of the hidden state sequence of the part-of-speech tag corresponding to the input text word sequence. The hidden Markov model can model the correspondence between the word sequence representation and the part-of-speech tag sequence through learning. The hidden state sequence probability reflects the probability of the part-of-speech tag sequence corresponding to a given word sequence under the hidden Markov model. Find the hidden state sequence with the highest probability, that is, the most likely part-of-speech tag sequence, as the word symbol sequence.

[0024] Among them, the Viterbi algorithm is a dynamic programming algorithm used to find the most likely hidden state sequence in a probabilistic graphical model. In this application, after the hidden Markov model calculates the probability of the hidden state sequence. The Viterbi algorithm is used to search for the sequence with the highest probability in the hidden state sequence space. The Viterbi algorithm gradually calculates the maximum probability of each hidden state at each moment through dynamic programming. And records the BACKPTR pointer, pointing to the maximum probability previous state at the previous moment. The maximum probability path of the entire sequence can be found through BACKPTR backtracking. The hidden state sequence corresponding to this path is the most likely hidden state sequence. The hidden state sequence with the highest probability is obtained as the output of the word symbol sequence.

[0025] Specifically, according to the hidden state sequence corresponding to the maximum probability obtained, the word symbol sequence is obtained, and a hidden Markov model is constructed to model the conversion probability of the input text word sequence to the part-of-speech tag sequence. The Viterbi algorithm is used to search the hidden state sequence space to find the hidden state sequence with the maximum probability. During the search process, the optimal path is recorded and the BACKPTR pointer is saved. Through BACKPTR backtracking, the entire maximum probability sequence is obtained. This maximum probability hidden state sequence is the best part-of-speech tag sequence corresponding to the input word sequence. According to the part-of-speech tag sequence, the type and boundary of the word symbol are determined. The part-of-speech tag sequence is converted into a word symbol sequence as the final output. The word symbol sequence accurately reflects the word symbol type and semantic structure of the input text. Provide structured word symbol input for subsequent word vectorization and position encoding.

[0026] Further, obtaining a hidden state sequence corresponding to the maximum probability includes: constructing a Viterbi model including a state set S, an observation set V, a state transition probability matrix A, and an observation probability matrix B, wherein the state set S represents a hidden state of the hidden Markov model;

[0027] Set the initial state probability δ0(i) and the previous state pointer variable According to time t, traverse all states i and calculate δ t (i) and δ t (i) = max[δ t-1 (j)×A ij ]×B i (V t ), Among them, A ij represents the transition probability from state i to state j, B i (V t ) represents the probability of observing Vt in state i, V t Represents the symbols in the text sequence; δ t-1 (j)×A ij represents the product of the probability of state j at the previous moment and the probability of transition to state i; δ t (i) represents the state s at time t i The maximum probability of Denotes δ t (i) The optimal preceding state with the largest correspondence; starting from the last moment, recursively from Backtracking out the state sequence; getting The largest state sequence is the hidden state sequence corresponding to the maximum probability.

[0028] Among them, the state set S: represents the set of hidden states in the hidden Markov model, each state corresponds to a hidden state at a moment, here refers to the set of part-of-speech tags. The observation set V: represents the set of observed states in the hidden Markov model, each observed state is generated by a hidden state mapping, here refers to the word set. The state transition probability matrix A: a matrix that records the transition probability between hidden states, each element represents the transition probability from state i to state j. The observation probability matrix B: a matrix that records the probability of hidden state generating observations, each element represents the probability of state i generating observation j. In this application, a Viterbi model is constructed, including the above-mentioned state set S, observation set V and probability matrices A and B. S represents the set of part-of-speech tags, and V represents the set of words. The A matrix represents the transition probability between part-of-speech tags. The B matrix represents the emission probability of part-of-speech generated words. The parameters of the A and B matrices are determined by learning text statistical information. The Viterbi model establishes a probabilistic correspondence between word sequence representation and part-of-speech tags. It provides a model basis for calculating the part-of-speech tags of input text.

[0029] Among them, the state probability δ0(i) represents the probability of each hidden state i at the initial time t = 0 in the Viterbi model. Indicates the previous state pointer of each state i in the Viterbi model, which is used to record the previous state of the maximum probability path. In this application, the state probability δ0(i) at the initial time t=0 is set, that is, the probability of each hidden state i at the initial time. A more reasonable initial probability distribution can be set based on corpus statistics. Set the previous state pointer variable 0, indicating that there is no previous state at the initial moment. δ0(i) represents the probability distribution of each initial hidden state. It is used to record the optimal state pointer of the previous sequence in the Viterbi search process. δ0(i) and Initialize the Viterbi search and provide necessary parameters for the search process.

[0030] Specifically, δ t (i) represents the maximum probability of state i at time t, which is the variable that the Viterbi algorithm needs to calculate. i (V t ) represents the probability of generating the current observation Vt in state i. t-1 (j) represents the maximum probability of state j at the previous time t-1. ij Represents the state transition probability, the probability of transitioning from state j to state i. δ t-1 (j)×A ij represents the joint probability of transitioning from the previous state j to the current state i. Maximize δ t-1 (j)×A ij , we can get the maximum probability path to state i. The maximum probability path and the observation probability B of state i are combined. i (V t ) to obtain the maximum probability δ of state i t (i). Record the previous state j corresponding to the maximum probability. Solve δ through dynamic programming iteration t (i) and Implement the maximum probability path search. Repeat the steps until the text sequence traversal is completed and the maximum probability path is obtained.

[0031] Specifically, starting from the last moment, recursively from The state sequence is traced back in the text sequence. When the text sequence traversal calculation is completed, the δ of the last moment T is obtained. t (i) and Initialize the state sequence s to an empty sequence []. From the last state δ t (i=argmax[δ t (i)], that is, the state with the highest probability at the last moment. Add i to the head of the state sequence, s'=[i]+s. Find the previous pointer Find the previous optimal state j. Add j to the head of the state sequence, s' = [j] + s. Repeat the recursive search for the previous optimal state to form a state sequence. The recursive process ends when the initial state is traced back. Finally, a complete optimal state sequence s from the initial state to the terminal state is formed. s is the most likely state sequence corresponding to the given text. By backtracking the previous pointer, the complete optimal state sequence can be recursively restored, that is, the maximum probability word symbol sequence required.

[0032] Specifically, get The largest state sequence is used as the hidden state sequence corresponding to the maximum probability. Under the Viterbi model architecture, the hidden state probability of the input text sequence is calculated. The maximum probability δ of each state i at each time t is solved iteratively through dynamic programming. t (i) Calculate δ at the same time t (i) The corresponding optimal preceding state is stored in middle. Records the previous state of the maximum probability path to the current state i. When all text sequences are traversed and calculated, they converge to the final moment T. In the above example, select the state i with the highest probability. Recursive backtracking Find the optimal state of the previous order until the initial state. In this way, the state sequence corresponding to the entire maximum probability path is restored from back to front. This state sequence is the most likely hidden state sequence of the given text. The maximum probability state sequence is used as the word symbol sequence predicted by the hidden Markov model. By obtaining The state sequence is backtracked to accurately solve the optimal word symbol sequence corresponding to the input text and complete the generation of the word symbol sequence.

[0033] Further, the hidden Markov model adopts a bidirectional hidden Markov model. Among them, the bidirectional hidden Markov model (Bidirectional Hidden Markov Model, BiHMM) is an extension of a hidden Markov model, which combines two hidden Markov models to associate the input sequence with the output sequence: a forward hidden Markov model, which can obtain the probability distribution of the forward input to the output; a reverse hidden Markov model, which can obtain the probability distribution of the reverse input to the output. Taking the probabilities of the two models as a joint distribution, the probability correspondence between the input and output sequences can be better characterized. In this application, a bidirectional hidden Markov model is used to obtain the mapping probability of the input word sequence to the part-of-speech tag sequence. The forward probability distribution of the forward HMM modeling word sequence. The reverse probability distribution of the reverse HMM modeling word sequence. Combining the probability distribution of the two models, the correspondence between the word sequence and the part-of-speech tag can be more accurately characterized. Thereby, a more accurate part-of-speech tag sequence, i.e., a word symbol sequence, can be generated. The bidirectional hidden Markov model can better utilize the bidirectional information of the input sequence and improve the tagging accuracy.

[0034] Furthermore, position coding is added to the word embedding representation, including: obtaining the position information of each word vector in the word embedding matrix E(i), the position information includes: absolute position pos, word vector dimension i and total dimension dmodel of the word vector; calculating the position coding PE(pos,i) of each word vector to obtain the position coding matrix PE(pos,i); adding the position coding matrix PE(pos,i) to the word embedding matrix E(i) element by element to obtain the word embedding matrix with position coding added.

[0035] Among them, the absolute position pos: represents the specific position index of the word vector in the text sequence, starting from 0. The word vector dimension i: represents a specific dimension in the word vector, and the value range is 0 to dmodel-1. The total dimension dmodel of the word vector: represents the dimensional size of the word vector, which is generally set to 512 or 1024, etc. In this application, the position information of each word vector in the word vector matrix E(i) is obtained. Including the absolute position pos of the word vector, that is, its index position in the text sequence. The word vector dimension i, with a value range of [0, dmodel-1], represents a specific dimension of the word vector. dmodel is the total dimensional size of the word vector. Position encoding needs to be characterized according to the absolute position and dimension of the word vector. A position feature is added to each dimension of each word vector to represent its serial number information in the text sequence. These position information will be used for position encoding, adding semantic information of the serial number.

[0036] Specifically, calculate the position encoding PE(pos,i) of each word vector, obtain the position encoding matrix PE(pos,i), define the position encoding function PE(pos,i), and pass in the absolute position pos and serial number i of the word vector. i ranges from 0 to dmodel-1, corresponding to the serial number dimension of the word vector. pos represents the position index of the word vector in the sequence. Encode the position serial number information through trigonometric functions. Add the position encoding PE(pos,i) to the corresponding position of the word vector matrix E(i). Get a new encoding matrix E'(i) = E(i) + PE(pos,i). E'(i) combines the semantic information of the word vector itself and the serial number information of the position encoding. Finally, the encoding-enhanced word embedding matrix E'(i) is obtained, which provides enhanced features for sequence modeling.

[0037] Furthermore, the position encoding PE(pos,i) of each word vector is calculated, including: constructing a position encoding model, the position encoding model includes a position encoding layer, and the position encoding layer is used to calculate the position mapping function; for each word vector, calculating the position mapping function Among them, W1 and W2 are the learning weight matrices of the position encoding layer, b1 and b2 represent the bias vectors of the position encoding layer, Gelu is the Gaussian activation function; α and β represent the position offset and scaling factor respectively; the position encoding layer includes a gating unit, and the position encoding output by the position encoding layer is regularized by the gating unit. The gating unit includes a sigmoid-activated gating vector g, which scales the position encoding to obtain the scaled position mapping function Where g is the gate vector, ranging from 0 to 1; calculate the position code:

[0038] Specifically, the position encoding model generates position encoding by learning the position mapping function, and constructs a position encoding model, which includes a position encoding layer. The input of the position encoding layer is the position information of the word vector [pos, i, α, β]. pos represents the absolute position of the word vector, and i represents the dimension number of the word vector. α and β are the position offset and scaling factor, which can be learned to adjust the position encoding. First, a linear transformation is performed through the fully connected layer W1 and the bias b1. Then it is transformed nonlinearly through the Gelu activation function. The final position encoding is obtained by passing through a fully connected layer W2 and the bias b2. By learning the parameters of W1 and W2, b1 and b2, the model can adaptively output the position encoding. The position encoding is applied to the corresponding word vector to obtain an enhanced word vector expression. The position encoding process based on deep learning is completed.

[0039] More specifically, Gelu is a Gaussian error linear unit activation function. The Gelu function curve is smooth and the derivative is not discontinuous. Compared with ReLU, Gelu can express richer nonlinear transformations. Gelu does not cause gradient diffusion or gradient disappearance. It enables the model to perform more expressive nonlinear encoding of position information. The smooth and non-saturated nature of Gelu can stably approximate complex mapping functions. Improve the modeling ability and robustness of position encoding.

[0040] More specifically, b1 and b2 are bias vectors in the position encoding layer. The role of the bias vector is to offset the result after the linear transformation. b1 acts on the result after the linear transformation W1×[pos,i,α,β]. b2 acts on the result after the Gelu activation and the quadratic linear transformation W2. The bias vector is generally initialized to a 0 vector and then updated during training. Through gradient descent, the bias vector is continuously updated to make the model output more accurate. b1 and b2 will learn the mapping deviation from the input feature to the position encoding. Make the position encoding more consistent with the law of the current training data. At the beginning of training, b1 and b2 can be initialized to constants, such as 0.1, to avoid deadneurons. During training, the gradient of the loss is calculated, and b1 and b2 are iteratively updated through the optimizer. Finally, a bias vector that can generate a good position encoding is learned.

[0041] Among them, the gating unit: a module in a neural network that performs adaptive regularization on feature maps. The gating vector: a vector in the gating unit that scales the input feature map, with a value between 0 and 1. Regularization: In deep learning, the activation of the network layer is properly normalized to make it more evenly distributed. In this application, the position encoding layer is followed by a gating unit. The gating unit contains a gating vector g, which obtains a 0-1 value through a sigmoid activation function. g is the raw output of the position encoding layer. Perform regularized scaling. Obtain a new gated position encoding The gating vector g of the gating unit can adaptively regularize the position encoding, reduce the influence of the position encoding on the word vector, make the distribution of the position encoding more uniform, and will not cause too much interference to the original semantics, thereby achieving effective regularization of the position encoding.

[0042] Specifically, calculate the position code PE(pos,i), is the regularized position mapping function output by the gated unit. Apply the sin activation function. The value range of the sin activation function is -1 to 1. Keep the numerical distribution of the position code within a reasonable range. The original semantics of the word vector will not be seriously disturbed due to the position code being too large. The sin function also increases the nonlinear expression ability of the position code. The codes at different positions are more distinctive. Add the position code PE(pos,i) to the word vector matrix. Provide auxiliary information of the position number for the word vector. Fuse semantic information and position information to enhance the representation ability of the word vector.

[0043] Furthermore, the position encoding model adopts the RESNET network structure. Among them, the RESNET network structure is a deep residual neural network structure, which introduces the concept of residual connection. A residual connection is added between two network layers to achieve output = input + network calculation. The residual connection realizes the residual learning of the output to the input, avoiding the gradient disappearance of the deep network. The entire network forms a stack of multiple residual blocks, making the network very deep but easy to optimize. In this application, the position encoding model adopts the RESNET network structure. A residual connection is added between every two linear layers. The output is equal to the residual of the input transformed by the linear layer. A deep network is realized, but the gradient disappearance is avoided. The network can gradually learn to generate position encodings with stronger expressive power. The deep RESNET structure enhances the modeling ability of position encoding. Make the position encoding more accurate and rich in the characterization of word order information.

[0044] Furthermore, an occlusion vector is obtained, including: a fully connected network including a fully connected layer, a nonlinear activation layer and a linear mapping layer; the fully connected layer inputs the key-value representation inside the decoder, and outputs the initial representation of the occlusion vector using a mask mechanism; the nonlinear activation layer uses a Swish activation function to perform nonlinear mapping on the initial representation of the occlusion vector to obtain a final representation of the occlusion vector; the linear mapping layer linearly maps the nonlinearly mapped occlusion vector through matrix-vector multiplication to obtain an occlusion vector with the same dimension as the key value inside the decoder.

[0045] Among them, the masking mechanism refers to a technology that uses a mask vector to mask or shield certain elements in a neural network. In this application, the decoder generates key and value representations through a self-attention mechanism. The key value of the decoder is input into the fully connected layer to obtain the initial occlusion vector. In the fully connected layer, a masking mechanism is adopted. A mask vector is set to mark the elements at certain positions of the decoder as 1. In the weighted operation of the fully connected layer, an element-level masking operation is performed. The masked elements are multiplied by 0 to shield them. After the masked occlusion vector, the elements at certain positions are shielded to 0. This prevents the decoder from using information at future moments to achieve autoregressive generation. The masking mechanism controls the self-attention range of the decoder to ensure its generation capability.

[0046] Among them, the Swish activation function is a self-gated nonlinear activation function, which is defined as: Swish(x)=x×sigmoid(x), where x is the input feature and the range of sigmoid(x) is between (0, 1). In this application, the fully connected layer outputs the initial representation of the occlusion vector. The initial representation is input into the Swish activation layer for nonlinear transformation. Swish performs self-gated scaling activation on each element. The expression of each dimension in the occlusion vector is made more nonlinear. The Swish function is smooth and can avoid gradient disappearance. Swish will not cause neurons to die completely, making the expression richer. The activated occlusion vector becomes more discriminative. Finally, the final expression of the occlusion vector is output. It is used to control the autoregressive generation process of the decoder. Swish enhances the nonlinearity of the occlusion vector representation and improves the quality of controlling autoregression.

[0047] Among them, matrix-vector multiplication refers to a method of operation using matrices and vectors for multiplication. In this application, the linear mapping layer performs a linear transformation on the nonlinearly activated occlusion vector. The form of matrix-vector multiplication is adopted. Set a matrix W and multiply it with the occlusion vector v to obtain Wv. The matrix W contains the weight parameters of the linear mapping. Through training, a suitable linear transformation matrix W can be learned. Matrix-vector multiplication realizes the linear mapping of the occlusion vector from low dimension to high dimension. Map the dimension to the same as the decoder key value representation. In this way, the occlusion vector can directly act on the key value representation in dimension. Realizing linear transformation through matrix-vector multiplication is a standard operation of the linear layer. Finally, an occlusion vector with the same size as the decoder key value is obtained.

[0048] Specifically, an occlusion vector with the same dimension as the decoder key value is obtained, and the key value representation vector inside the decoder is dk-dimensional. The occlusion vector output by the nonlinear activation layer is dv-dimensional. Define a linear mapping matrix W with a size of dkx×dv. The dimension of each row vector of the W matrix is ​​the same as the occlusion vector dv. Linear projection mapping is performed on the occlusion vector v: u=Wv. The dimension of u is dk-dimensional, which is the same as the decoder key value. By adjusting the output dimension dk of the matrix W, it can be adaptively mapped to the desired dimension. The W matrix is ​​used as a training parameter to learn a matching linear mapping. The occlusion vector v is transformed into u of the same size after being mapped by W. u can act directly on the key value representation of the decoder. Autoregressive occlusion control for the decoder is realized. The generation order and language fluency of the decoder are guaranteed. The matching conversion of the occlusion vector size is realized through the matrix projection of the linear mapping layer.

[0049] Furthermore, constructing an initial vector includes: constructing an initial mask vector mask0, where the length of mask0 is the length of the key value WK inside the decoder; according to the current decoding timing, setting the subsequent moment position of mask0 to 0 and the previous moment position to 1; obtaining a dependency syntactic analysis tree of the input text sequence; calculating the correlation weight matrix D between the word vector and the key value WK according to the dependency syntactic analysis tree; comparing each element in the matrix D with a threshold, adjusting the vector at the corresponding position according to the comparison result, and when the element is greater than the threshold, setting the corresponding position to 1, otherwise setting it to 0, to obtain a structural mask mask1; calculating the correlation attention weight vector Z between mask1 and the key value WK inside the decoder; expanding the attention weight vector Z to a vector Z' with the same dimension as mask1 through a broadcast mechanism; concatenating mask1 and Z' to obtain a mask vector mask1'; multiplying mask1' by the key value WK to obtain an initial representation of the occlusion vector.

[0050] Among them, the mask vector (MaskVector) is a vector used to mask or shield specific elements in a neural network. In this application, an initial mask vector mask0 is constructed. The length of mask0 is set to the length of the key value WK inside the decoder. Each element of the mask vector can be 0 or 1. 0 means shielding, and the key value at this position is invalid for the attention mechanism. 1 means retention, and the key value at this position is valid for the attention mechanism. Mask0 is all 1 at initialization, and no elements are masked. Then mask0 is input into a specific Mask module. The Mask module performs a mask operation on mask0 to produce an actual mask result. Finally, according to the mask result, the attention weights inside the decoder are selectively shielded. The decoder is controlled to rely only on the generated words for autoregressive generation. The mask vector realizes controllable shielding of the decoder's self-attention.

[0051] Specifically, according to the current decoding timing, the subsequent moment position of mask0 is set to 0, and the previous moment position is set to 1, and the current decoding timing t is defined, starting from 0. Mask0 is an all-1 vector, and its length is the time step length T of the decoder key-value matrix. Traverse mask0, and the index range is 0 to T-1. When the index i < t, it means the current and previous moments, mask0[i] = 1. When the index i ≥ t, it means the subsequent moment, mask0[i] = 0. In this way, a new mask vector can be obtained. The mask at the previous moment is 1, and the attention mechanism is valid for these positions. The mask at the subsequent moment is 0, and the attention mechanism is shielded for these positions. As the decoding timing t increases, attention is gradually allowed to more previous positions. The masking of future information is achieved, and the decoder only relies on the generated word symbols.

[0052] Among them, the dependency syntactic analysis tree (DependencyParse Tree) is a syntactic analysis method, which represents the sentence structure through the dependency relationship between sentence components. In this application, the input text sequence is subjected to dependency syntactic analysis to obtain a dependency syntactic tree. The dependency syntactic tree has words as nodes and dependencies as directed edges. The root of the tree is the core of the entire sentence, generally a verb or predicate. Each node has and only has one head node, representing the dependency relationship. The dependency syntactic tree structure expresses the semantic and grammatical relationship of the sentence. In the encoder, the dependency syntactic tree can be used to enhance the structured representation of word vectors. Combining word vectors and dependency tree representations, the encoder is provided with richer syntactic features. The dependency syntactic representation enhances the encoder's ability to model sentence structure. Help the decoder generate a language sequence that conforms to the grammatical structure.

[0053] Specifically, calculate the relevance weight matrix D according to the dependency syntactic analysis tree, and construct the word vector matrix E, where each row is a word vector. Construct the decoder key value matrix WK, with the dimension timestepsxkeydim. Initialize the relevance matrix D to 0, with a size of E rows x WK columns. According to the dependency syntactic tree, traverse the dependency relationship of each node word. If the word w i Dependency word w j , then find w i and w j The row indices i and j in matrix E. In D, D i For the position where the column index of WK is equal to j, add the relevance weight v. v can be a scalar 1 or the weight value of the dependency. Repeat for all dependencies in the tree and finally get the relevance weight matrix D of the word vector and the key value. D incorporates the dependency syntactic structure information and enhances the modeling ability of the encoder.

[0054] Specifically, the matrix D is compared with the threshold to generate the structural mask mask1, where each element in the D matrix represents the correlation weight between the word vector and the value of the decoder key. Set a correlation threshold τ, which can be pre-set or determined by the validation set. Traverse each position (i, j) of the matrix D. Compare D(i, j) with the threshold τ. If D(i, j)>τ, then mask1(i, j)=1. If D(i, j)≤τ, then mask1(i, j)=0. Mask1 is a 0-1 matrix with the same size as D. 1 in mask1 indicates a position with strong correlation. 0 in mask1 indicates a position with weak correlation. Finally, the structural mask mask1 is obtained. Mask1 retains the strongly correlated dependencies and shields the weakly correlated dependencies. It plays a filtering role in the dependency structure and enhances the structural representation. The structural mask mask1 is generated by threshold comparison to highlight the strongly correlated dependency syntactic structure.

[0055] More specifically, mask0 implements shielding of future information according to the time sequence to ensure the autoregressive generation of the decoder. Mask1 implements modeling of grammatical structure according to the dependency syntactic tree and provides structural knowledge. The two masking mechanisms work simultaneously to achieve dual control of generation position and grammatical constraints. Mask0 controls the sequential generation of the decoder, and mask1 guides the generation to conform to the grammatical structure. The double masking enhances the natural language generation capability of the decoder. The generated language conforms to both the time sequence and the syntactic structure. Mask0 ensures language fluency, and mask1 ensures language rationality. The two masks constrain together to generate higher quality language sequences. The double masking mechanism improves the encoder-decoder framework in both grammar and word order modeling.

[0056] Specifically, calculate the correlation attention weight vector Z between mask1 and the decoder key value WK, copy mask1 timestep times, flush with WK, and obtain the mask matrix M. Calculate the transposed matrix WK' of WK, with the dimension keydimxtimesteps. Calculate the matrix multiplication WK' to obtain the correlation matrix R. The R matrix combines the position mask mask1 and the decoder's internal key value representation. Normalize R rows to obtain the attention weight matrix A. Each row of A is an attention weight vector, which represents the attention distribution of the word to the decoder key value. Then multiply A with the initial mask mask0 matrix for masking. Obtain the final attention weight matrix Z that integrates the position and semantic structure information. Z will be used to weight the decoder's internal key value representation to achieve grammatical structure modeling. The attention weight matrix Z that integrates the position and grammatical structure constraints is obtained through matrix calculation.

[0057] Among them, the broadcasting mechanism is a mechanism that allows tensors of different shapes to be operated. In this application, the shape of the attention weight Z is (number of words, number of time steps). The shape of mask1 is (number of words, key vector dimension). Through broadcasting, Z can be automatically expanded to the same size as mask1. That is, Z copies the number of time steps on the key vector dimension. After broadcast expansion, Z and mask1 have the same shape and can perform element-level operations. Finally, the occlusion vector Z' that fuses the position and structure information is obtained. Z' occludes the internal representation of the decoder that is not grammatically independent.

[0058] Specifically, mask1 and Z' are concatenated to obtain the mask vector mask1', where mask1 is the structural mask matrix generated by the dependency syntax tree, and its size is (number of words, key vector dimension). Z' is expanded by broadcasting the attention weight vector to the same size as mask1. The concatenation operation [·] is defined to represent connecting tensors in the specified dimension. Perform the concatenation operation: mask1' = [mask1, Z'], and the concatenation mode is column-wise concatenation. That is, mask1 and Z' are tiled and concatenated on the column dimension. Mask1 provides grammatical structure constraints, and Z' provides attention weight constraints. Mask1' concatenates the two constraint representations together. Mask1' serves as a new occlusion vector, which has both grammatical and attention constraints. Mask1' is input into the occlusion module to control the attention process of the decoder. Syntactically structured attention occlusion generation is realized. Concatenate mask1 and Z' in the column direction to obtain a mask vector mask1' with dual constraints.

[0059] Specifically, mask1' is multiplied by the key value WK to obtain the initial representation of the occlusion vector. Mask1' is the concatenated dual-constrained mask vector with a dimension of (number of words, key dimension). WK is the key-value representation inside the decoder with a dimension of (time step, key dimension). Matrix multiplication is performed on mask1' and WK, mask1×WK. In the matrix multiplication, the word dimension of mask1' will be multiplied correspondingly with the time step dimension of WK. The initial representation of the occlusion vector is obtained with a dimension of (number of words, time step). Each element in the initial representation encodes the occlusion value of a given word at a given decoding moment. 1 means no occlusion, and 0 means complete occlusion. The dual constraints of grammatical structure and attention weight are encoded. Subsequently, the final occlusion vector is generated through nonlinear layers and linear transformations. Finally, the occlusion vector is applied to the internal representation of the decoder to achieve grammatically structured decoding.

[0060] 3. Beneficial effects

[0061] Compared with the prior art, the advantages of this application are:

[0062] (1) Through the multi-layer attention mechanism of the Transformer model, the key information in the input text sequence can be effectively captured, improving the quality and accuracy of summary generation. This global semantic modeling capability enables the model to better understand the semantic structure and relevance of the text when generating summaries;

[0063] (2) Introducing the position encoding model, embedding the position information into the word embedding matrix helps the model understand the word order relationship in the text and improves the coherence and accuracy of summary generation. Position encoding enables the model to better process the order information of the input text, thereby more accurately capturing the semantic features of the text;

[0064] (3) By establishing a residual connection between the encoder and the decoder, the semantic features extracted from the encoder can be directly transferred to the decoder, which enhances the decoder's ability to understand the input text and improves the effect of generating summaries. This transfer of semantic information enables the decoder to more accurately generate summaries related to the input text;

[0065] (4) The introduction of a bidirectional hidden Markov model can better capture the complex structure and potential information in the text sequence, improve the modeling ability of the input text, and thus improve the quality and accuracy of the generated summary. The application of this model can more comprehensively understand the semantics and context of the text, thereby generating a more accurate and comprehensive summary;

[0066] (5) The introduction of structural masks and occlusion vectors effectively limits the model’s attention range during the decoding process, allowing the model to pay more attention to information related to the currently generated token, improving the accuracy and fluency of the generated summary. This optimization can prevent the model from paying too much attention to irrelevant information, thereby generating a more refined and accurate summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0068] Figure 1 is an exemplary flow chart of a Transformer-based summary generation method according to some embodiments of this specification;

[0069] Figure 2 is an exemplary flow chart of generating a word embedding matrix according to some embodiments of this specification;

[0070] Figure 3 is an exemplary flow chart of generating position codes according to some embodiments of this specification;

[0071] Figure 4 is an exemplary flowchart of generating an initial occlusion vector according to some embodiments of the present specification. DETAILED DESCRIPTION

[0072] The method and system provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0073] Figure 1This is an exemplary flow chart of a Transformer-based summary generation method shown in some embodiments of the present specification, obtaining a text sequence; establishing a language model, performing word embedding representation on the input text sequence, and obtaining a word embedding matrix; inputting the word embedding matrix into a position encoding model for position encoding, and obtaining a word embedding matrix with position information added; constructing a Transformer model, taking the word embedding matrix with position information added as input, and using the Transformer model to generate a summary of the input text sequence; the Transformer model includes an encoder and a decoder, the encoder encodes the semantic information of the input text sequence through a multi-layer attention mechanism; the decoder generates a summary sequence through a multi-layer attention mechanism; the decoder includes a self-attention layer, in which an occlusion vector is obtained through a fully connected network, and a dot product calculation is performed between the occlusion vector and the key-value representation inside the decoder to obtain an adjusted attention weight distribution; a residual connection is established between the encoder and the decoder, and the semantic features output by the encoder are directly input to the decoder to transmit semantic information; the decoder outputs the word symbol sequence of the summary in sequence according to the time step to form a final summary.

[0074] Figure 2 This is an exemplary flow chart for generating a word embedding matrix according to some embodiments of the present specification, and obtaining the word embedding matrix includes: inputting a text sequence: "Machine learning is a branch of artificial intelligence." Using Jieba Chinese word segmentation, a word sequence is obtained: "Machine learning is a branch of artificial intelligence", importing the Jieba word segmentation module, defining the text sequence to be segmented, for example: "Machine learning is a branch of artificial intelligence", setting the word segmentation mode, such as the precise mode cut_all = False, calling the jieba.cut interface, passing in the text sequence for word segmentation, jieba.cut will use the dictionary and word frequency data, and use the forward maximum matching algorithm to segment the text, and convert the word segmentation results into a list storage, for example seg_list = list (jieba.cut (text)), traversing or connecting the word segmentation list to obtain the text of space segmentation: "Machine learning is a branch of artificial intelligence", outputting the word segmentation result, that is, obtaining the final word segmentation word sequence.

[0075] Construct a hidden Markov model and use the forward-backward algorithm to calculate the probability of the hidden state sequence. Define the observation sequence O = {O1, O2, O3, ..., O T}, for example, the word sequence obtained by word segmentation, define the state set S = {S1, S2, S3, ..., S N}, N is the number of hidden states, and the forward probability is initialized as α1(i) = πi×bi(O t ), πi is the initial state distribution probability, bi(O t ) generates O for state Sit Probability, forward iteration: for t = 2, ..., T: for all states i: α t (i) = [∑jα t-1 (j)×α ij ]bi(O t ), α ij is the transition probability from state i to state j. Similarly, the reverse iteration calculates β t (i), the reverse probability is pushed forward from the final state. Finally, at each time t, the probability of state i is: P(qt=Si|O)∝α t (i)β t (i), that is, the product of the forward probability and the reverse probability, is the probability of the hidden state. After traversing all states, the probability of the entire hidden state sequence can be obtained. In this way, the probabilities of all hidden states of the hidden Markov model at each moment are calculated through the forward-backward algorithm, and then the joint probability of the entire hidden state sequence is obtained.

[0076] Through the Viterbi algorithm, define the hidden state set S = {S1, S2, S3, S4, S5, S6}, define the observation sequence O = {O1, O2, O3, O4, O5, O6} (word sequence), initialize: δ1(i) = πi×bi(O1), Recursion: For t=2, ..., T: For all states i: δ t (i) = max[δ t-1 (j)×α ij ]×bi(Ot), Termination: P * =max[δ t (i)],T * = argmax[δ t (i)] Optimal path backtracking: t = T-1, T-2, ..., 1, The optimal hidden state sequence is obtained: S1, S2, S3, S4, S5, S6. The hidden Markov model is solved by the recursive and backtracking process of the Viterbi algorithm, and the optimal hidden state sequence S1, S2, S3, S4, S5, S6 is obtained.

[0077] Word sequence: "Machine learning is a branch of artificial intelligence", build a word embedding layer, including a word vector dictionary and a self-attention encoding layer. The dictionary contains: machine learning->x1, is->x2, artificial intelligence->x3, of->x4, branch->x5. Look up the dictionary to get the word vector: machine learning->x1, is->x2, artificial intelligence->x3, of->x4, branch->x5. Word vector: {x1,x2,x3,x4,x5}, prepare a word vector for each word, build a word vector dictionary, and then search for the corresponding word vector according to the input word sequence, and get the word vector sequence corresponding to the word sequence {x1,x2,x3,x4,x5}. The self-attention encoding layer performs multi-head self-attention calculations and obtains an enhanced word vector representation. Input the word vector sequence:

[0078] {x1,x2,x3,x4,x5}, in the self-attention encoding layer, the word vectors are linearly mapped to obtain query, key, and value vectors: q1,k1,v1=x1×WQ,x1×WK,x1×WV, q2,k2,v2=x2×WQ,x2×WK,x2×WV.

[0079] q3,k3,v3=x3×WQ,x3×WK,x3×WV, q4,k4,v4=x4×WQ,x4×WK,x4×WV,

[0080] q5,k5,v5=x5×WQ,x5×WK,x5×WV. Calculate the attention weight: a 12 =attention(q1,k2),a 13 =attention(q1,k3),a 14 =attention(q1,k4),a 15 =attention(q1,k5). 14 represents the attention weight of q1 to k4, a 15 Indicates the attention weight of q1 to k5. The attention weight indicates the degree of attention of q1 to other word vectors k2 to k5. The weighted sum can be performed based on the weight vector a1 and the value vector v to obtain the enhanced representation of q1. The attention value indicates the degree of attention of q1 to other word vectors. Based on the weight vector and the value vector, the enhanced representation is obtained:

[0081] x1'=∑ia 1i ×v i , for other word vectors x iRepeat the above operation. Finally, we get the enhanced word vector: [x1', x2', x3', x4', x5'], and construct the word embedding matrix: E = [x1', x2', x3', x4', x5']. Therefore, by performing multi-head self-attention calculation in the self-attention encoding layer, we get the enhanced word vector representation and construct the word embedding matrix E as the model input.

[0082] Figure 3 This is an exemplary flow chart for generating position coding according to some embodiments of the present specification, defining a word embedding matrix E, which includes word vectors of N words, each of which has a length of d, and constructing a position coding model PosEncoder, including a position coding layer PosEncoding, which is responsible for learning the parameters of the position mapping function: weight matrix, bias, etc. During the forward propagation of the model, the word embedding matrix E is passed into PosEncoder, and for each word vector in E, the position mapping function is calculated according to its position pos. The position mapping function first undergoes a nonlinear transformation, and then is multiplied by a gate vector to adjust the scale. The adjusted position mapping function acts as a sine function to obtain a position coding vector, which is concatenated with the original word vector to obtain a word vector with added position information, and finally a new word embedding matrix E_pos is formed as the encoder input, and the encoder learns the semantic feature representation of the input text based on E_pos.

[0083] The position encoding layer learns the position mapping function. The position encoding layer needs to learn the following parameters: weight matrix W1, size din×dmodel, weight matrix W2, size dmodel×dout, bias b1, size dmodel, bias b2, size dout. Where din represents the input dimension, dmodel is the intermediate representation dimension, and dout is the output dimension. i , whose position number is pos_i: traverse each dimension j of the word vector, j from 1 to d, take the position number pos_i as a column vector, and concatenate it with the subscript j. After an affine transformation, the intermediate representation is obtained: z = W1×[posi,j]+b1, and the nonlinear activation function GELU is applied to z to obtain the representation after nonlinear conversion. After another affine transformation, the output of the position mapping is obtained: phi = W2×GELU(z)+b2. Repeat this process, traverse all dimensions j, and obtain all outputs of the position mapping function. Finally, phi is multiplied by the gated vector to obtain the adjusted position encoding. By learning parameters such as W1, W2, b1, b2, etc., the calculation of the position mapping function is realized, thereby obtaining the position encoding representation.

[0084] Calculate position mapping function Input position number pos_i, representing word vector x i, traverse each dimension j of the word vector, j=1,...,d. Define the learnable parameters of the position encoding layer: weight matrix W1, size din×dmodel, weight matrix W2, size dmodel×dout, bias b1, size dmodel, bias b2, size dout. Concatenate pos_i and j to form the input, and perform a nonlinear transformation on z through affine transformation: z=W1[posi,j]+b1, for example, using the GELU activation function. Perform an affine transformation again to obtain the output of the position mapping: phi=W2×GELU(z)+b2, repeat the above process, traverse all j, and obtain all the position mapping outputs of the word vector, and finally Multiplying with the gating vector generates the positional encoding.

[0085] Use the gated unit to generate position encoding and obtain the position mapping function output The shape is (N, d). In the gated unit, the gated vector g is learned, and the shape is (N, d). g is obtained through a fully connected layer and the sigmoid activation function is applied. The sigmoid function maps the value of g to between 0 and 1. Multiply g element by element, that is, Each element of is scaled and adjusted, and the adjusted result is recorded as The shape remains unchanged and is still (N, d). Apply the sine function for nonlinear transformation and convert the position number into a specific encoding value to obtain the final position encoding result PE, which still has the shape of (N, d). PE fully incorporates the position information and can be concatenated with the word vector.

[0086] The position code is input into the encoder and passed to the decoder. The input word embedding matrix E contains N word vectors, each with a length of d. In the position coding module, the position coding matrix PE corresponding to the word vector sequence is generated. PE also contains N encoding vectors with a vector length of d. E and PE are concatenated row by row to obtain a new word vector matrix E' with position information added. The shape of E' becomes (N, 2d), which contains word vectors and position codes. The text encoder is defined with input E' and output semantic representation matrix H. H combines the semantic representation of word meaning information and position information. The text decoder is defined and H is used as the encoding information input. At the same time, the word vector part of E is also input into the decoder through residual connection. Based on the encoding semantics H, the decoder introduces the original word vector E for decoding.

[0087] Figure 4This is an exemplary flow chart for generating an initial occlusion vector according to some embodiments of the present specification, defining a fully connected network structure, including the following layers: a fully connected layer FC, a nonlinear activation layer Activation, a linear mapping layer Mapping, and the internal representation of the decoder includes a key value matrix WK, which has a shape of (N, d). WK is used as input and passed to the fully connected layer FC. FC outputs the initial representation Z0 of the occlusion vector through matrix-vector multiplication, and Z0 is further passed to the nonlinear activation layer Activation, such as ReLU. The Activation layer performs a nonlinear transformation on Z0 to obtain the activated representation Z1, which is passed to the final linear mapping layer Mapping. The Mapping layer performs a linear projection and outputs the final initial representation Z2 of the occlusion vector. In this way, the initial representation of the occlusion vector is obtained through the learning transformation of the fully connected network, and then adjusted in combination with the mask vector to generate the final occlusion vector. The initial mask vector is constructed, and the key value representation of the decoder is WK, whose length is N. The initial mask vector mask0 of all 1s is constructed, and the length is also set to N. Each element in mask0 is initialized to 1: mask0 = [1,1,.....1], and the current decoding time sequence t is obtained, where t is less than N. According to t, the positions of subsequent moments in mask0 are set to 0: mask0 = [1,1,.....1,0,0,......,0], where the first t positions remain 1 and the subsequent positions become 0. In this way, according to the current decoding time sequence t, the initial mask vector mask0 is constructed, and the leading part of mask0 is marked as 1 and can participate in the attention calculation. The subsequent part is marked as 0 and does not participate in the attention calculation.

[0088] Perform dependency syntactic analysis on the input text and obtain the analysis tree: the input text is converted into a word vector matrix E with a shape of (N, d), and the key value matrix of the decoder is WK, also with a shape of (N, d). Calculate the correlation weight matrix D of E and WK: Each value in D represents the correlation weight between word vectors. Based on D and the dependency tree, a structural mask vector can be constructed, and finally the occlusion vector can be learned. The correlation weight matrix D of the word vector and WK has been obtained. Define the threshold of the correlation weight, for example 0.5, and traverse each element d in D ij , if d ij If it is greater than the threshold, the corresponding position in mask1 is set to 1. If it is less than the threshold, the corresponding position is set to 0. In this way, the structural mask mask1 is constructed, and the attention weight vector Z of the correlation between mask1 and WK is calculated. It is to perform weighted summation on WK, and the weight comes from mask1. Each value in Z represents the attention weight of the word and the current decoded word. According to the distribution of Z, it can be clear which words should be paid attention to or not. Finally, Z participates in the generation of the occlusion vector to achieve structured occlusion.

[0089] By using the broadcast mechanism to expand the attention weights and construct the mask vector, an attention weight vector Z with a length of N has been obtained. Z represents the attention weight between the current decoded word and the input sequence representation. Z is repeated N times to obtain a new matrix Z', which has the same shape as mask1, that is, Z is expanded into a two-dimensional matrix Z' through the broadcast mechanism. Each row in Z' repeats the attention weight distribution of Z, and the structural mask matrix mask1 is defined, which is also an N×N matrix. Mask1 and Z' are concatenated element by element to obtain a new mask matrix mask1' with a length of 2N. Mask1' combines structural information and attention weight information. Mask1' will be multiplied with WK to generate the initial representation of the occlusion vector.

[0090] The mask vector is used to generate the occlusion vector. The mask vector mask1' has been constructed, which integrates the structural information and the attention weight information. The key value inside the decoder is represented as WK, which has the same dimension as mask1'. Mask1' is multiplied element by element with WK to obtain a vector as the initial representation of the occlusion vector. The initial representation is passed to a nonlinear activation layer, such as ReLU. The ReLU layer performs a nonlinear transformation on it to obtain the activated occlusion vector, which is then passed to a linear mapping layer for linear projection. The linear layer learns the final representation of the occlusion vector based on the mask vector and WK. The final occlusion vector fully integrates the structural information and can be used to adjust the attention. The occlusion vector is passed to the self-attention layer to adjust the attention distribution and achieve structured occlusion.

[0091] The initial representation of the occlusion vector is Z, whose shape is (N, dz), where N is the vector length and dz is the dimension. The Swish activation layer is defined to implement element-level nonlinear transformation. For each element z in Z i , use the Swish function to transform: z i Pass in the swish function to get the activated z i _swish, perform the same transformation on all elements in Z to obtain the activated matrix Z_nl. The shape of Z_nl remains unchanged, still (N, dz), and each element is nonlinearly mapped through the Swish function. Define a linear mapping layer, including a weight matrix W, whose shape is (dz, dk), where dz is the initial dimension of the occlusion vector, and dk is the same as the dimension of the decoder key value WK. The activation representation Z_nl of the occlusion vector has been obtained, and the shape is (N, dz). Matrix-vector multiplication is performed to achieve linear transformation: the shape of Z_final is (N, dk), which matches WK. Z_final is the final representation of the occlusion vector, which contains the dimensional information required by the decoder. Z_final is passed to the self-attention layer and used in combination with WK to generate the corresponding occlusion vector according to the decoder representation, thereby occluding irrelevant content and adjusting the attention distribution.

[0092] The encoder reads the input text and outputs a text feature representation matrix H. H incorporates the semantic information of the text. Define the encoder model structure, which includes a multi-layer self-attention network. Input a text sequence and convert it into a word vector sequence X. X is passed to the first layer of self-attention to obtain a semantic representation Z1. Z1 incorporates the local semantic information of the text sequence. Z1 is passed to the next layer of self-attention, and the semantics are extracted to obtain Z2. Each layer of self-attention learns the high-level semantic features of the text. Repeat this process, and the last layer outputs the semantic representation H of the text. H is a matrix, and each row represents the semantics of a word in the text. H integrates semantic information from each layer. The encoder learns the multi-granular semantic features of the text through the self-attention layer. Finally, the text semantic representation matrix H is output.

[0093] At the first moment, the decoder generates the first word y1 based on H. At the second moment, the decoder predicts the next word y2 based on H and y1. The encoder outputs the text semantic representation matrix H. When the decoder decodes for the first time, only the encoder outputs H. Define the multi-layer self-attention network structure of the decoder. At the first moment, H is input to the bottom layer of the decoder. Through the self-attention layer, the probability distribution of the first word y1 is generated. Random sampling according to the distribution obtains the first word y1. At the second moment, the bottom layer of the decoder receives H and y1 at the same time. y1 incorporates the generated semantic information. After the self-attention layer, the probability distribution of the next word y2 is predicted. Sampling again obtains y2. Through self-attention, the decoder considers the encoder features and the generated words at the same time, and gradually predicts and outputs the word at the next moment. The decoder gradually generates the word symbol at the next moment. When generating the current word, consider H and the previously generated word sequence. Repeat this process until a complete summary sequence is generated. The summary sequence is composed of word symbols generated by the decoder step by step. The final generated complete summary sequence is used as the result of the text summary. Output the summary word sequence generated by the decoder to complete the text summary task.

Claims

1. A summary generation method based on Transformer, comprising: Get the input text sequence for generating summary; Establish a language model, perform word embedding representation on the input text sequence, and obtain a word embedding matrix; Input the word embedding matrix into the pre-trained positional encoding model for position encoding to obtain a word embedding matrix with position information added; Build a Transformer model, take the word embedding matrix with position information as input, use the Transformer model to encode and decode the input text sequence, and generate a summary of the input text sequence; Add positional encoding to word embedding representation, including: Get word embedding matrix The position information of each word vector in the sequence is used to construct a position information matrix; the position information includes: the absolute position pos of the word vector in the sequence, the dimension index i of the word vector, and the total dimension dmodel of the word vector; For each word vector in the word embedding matrix, calculate the position encoding of each word vector , get the position encoding matrix ; The position encoding matrix With word embedding matrix Add element by element to get the word embedding matrix with position encoding added; Calculate the positional encoding of each word vector ,include: Construct a position encoding model to generate corresponding position encoding according to the position information of the word vector; wherein the position encoding model includes a position encoding layer, the position encoding layer adopts a feedforward neural network structure, and is used to learn the mapping function from position information to position encoding. The input of the position encoding layer is the obtained position information matrix, and the output is the corresponding position encoding matrix; For each position information vector in the position information matrix, the position encoding layer is used to calculate the position mapping function : in, and is the learning weight matrix of the position encoding layer, and represents the bias vector of the position encoding layer, and Gelu is the Gaussian activation function; and Represent the position offset and scaling factor respectively; The position encoding layer includes a gating unit. The position encoding output by the position encoding layer is regularized by the gating unit. The gating unit includes a sigmoid-activated gating vector g. The position encoding is scaled to obtain a scaled position mapping function. : Where g is the gate vector, ranging from 0 to 1; Calculate the positional encoding: 。 2. The Transformer-based summary generation method according to claim 1, characterized in that: The Transformer model consists of an encoder and a decoder. The encoder uses a multi-layer attention mechanism to encode the input word embedding matrix, extract the semantic information of the input text sequence, and obtain the encoded semantic features. The decoder uses a multi-layer attention mechanism to decode the encoded semantic features output by the encoder to generate a summary sequence; A self-attention layer is set in the decoder, and the occlusion vector is obtained through a fully connected network. The dot product calculation is performed between the occlusion vector and the key-value representation inside the decoder to obtain the adjusted attention weight distribution, which is used to adjust the attention calculation process of the decoder. A residual connection is established between the encoder and the decoder. The encoded semantic features output by the encoder are directly transferred to the decoder through the residual connection and fused with the intermediate representation of the decoder to transfer semantic information. The decoder decodes iteratively in time. At each time step, based on the decoder output and attention weight distribution at the current moment, it predicts the summary word symbol at the next moment, outputs the summary word symbol sequence in sequence, and concatenates the output summary word symbol sequence to form the final summary text.

3. The Transformer-based summary generation method according to claim 1, characterized in that: Get the word embedding matrix, including: Use Jieba word segmentation algorithm to segment the input text sequence to obtain a word symbol sequence; Construct a word embedding layer based on the attention mechanism. The word embedding layer includes a dictionary and a self-attention encoding sublayer. The dictionary stores the word vectors corresponding to each pre-trained word. According to the constructed dictionary, the corresponding word vector is obtained for each word in the obtained word symbol sequence to form a word vector sequence; The obtained word vector sequence is used as input, and the self-attention encoding sublayer is used to perform multi-head self-attention calculation to obtain the contextual semantic information of the word vector; The self-attention encoding sublayer uses the Gelu nonlinear activation function to perform nonlinear transformation on the word vector sequence obtained by multi-head attention calculation, and outputs the word vector sequence with contextual semantic enhancement; According to the context-enhanced word vector sequence, the word embedding matrix of the input text sequence is constructed.

4. The Transformer-based summary generation method according to claim 3, characterized in that: Get the word symbol sequence, including: The obtained input text sequence is modeled using a hidden Markov model, and the probability distribution of the hidden state sequence corresponding to the input text sequence is calculated; Decode the input text sequence using the Viterbi algorithm and search for the hidden state sequence corresponding to the maximum probability; According to the hidden state sequence corresponding to the maximum probability obtained, combined with the words in the input text sequence, a word symbol sequence is obtained.

5. The Transformer-based summary generation method according to claim 4, characterized in that: Obtain the hidden state sequence corresponding to the maximum probability, including: Construct a Viterbi model to search for the optimal hidden state sequence based on the hidden Markov model; wherein the Viterbi model includes a state set S, an observation set V, a state transition probability matrix A and an observation probability matrix B; Among them, the state set S represents all possible hidden states in the hidden Markov model, and the observation set V represents the words in the input text sequence; the state transition probability matrix A represents the transition probability between hidden states, and the observation probability matrix B represents the probability of each hidden state generating an observed state; Set the initial conditions of the Viterbi algorithm, which include the probabilities of each state at the initial moment and the previous state pointer variable ; For each time t, traverse all hidden states i and calculate the time to reach state i at time t and : in, represents the transition probability from state i to state j, represents the probability of observing Vt in state i, Represents symbols in a text sequence; Represents the product of the probability of state j at the previous moment and the probability of transition to state i; Indicates the state at time t The maximum probability of express The optimal preceding state of the maximum correspondence; After completing the traversal of all moments, starting from the last moment T, recursively Backtrack the entire optimal hidden state sequence; The optimal hidden state sequence is obtained as the hidden state sequence corresponding to the maximum probability.

6. The Transformer-based summary generation method according to claim 5, characterized in that: The hidden Markov model adopts a bidirectional hidden Markov model.

7. The Transformer-based summary generation method according to claim 1, characterized in that: The position encoding model adopts the RESNET network structure.

8. The Transformer-based summary generation method according to any one of claims 2 to 7, characterized in that: Get the occlusion vector, including: Construct a fully connected network to generate an occlusion vector according to the key-value representation inside the decoder; wherein the fully connected network includes a fully connected layer, a nonlinear activation layer, and a linear mapping layer; The key-value representation inside the decoder is used as the input of the fully connected layer, and the mask mechanism is used to output the initial representation of the occlusion vector; The nonlinear activation layer uses the Swish activation function to perform nonlinear mapping on the initial representation of the occlusion vector to obtain the final representation of the occlusion vector; The linear mapping layer linearly maps the nonlinearly mapped occlusion vector through matrix-vector multiplication to an occlusion vector with the same dimension as the key value inside the decoder.

9. The Transformer-based summary generation method according to claim 8, characterized in that: The mask mechanism is used to output the initial representation of the occlusion vector, including: Constructing the initial mask vector , The length is the key value inside the decoder Length; According to the current decoding timing, The subsequent time position is set to 0, and the previous time position is set to 1; Perform dependency syntactic analysis on the input text sequence to obtain a dependency syntactic analysis tree; According to the dependency syntactic analysis tree, calculate the correlation weight matrix D between each word vector in the input text sequence and the key value WK inside the decoder; Compare each element in the matrix D with the threshold, and adjust the vector at the corresponding position according to the comparison result. When the element is greater than the threshold, the corresponding position is set to 1, otherwise it is set to 0, and the structure mask is obtained. ; calculate The attention weight vector Z related to the key value WK inside the decoder; The attention weight vector Z is expanded through the broadcast mechanism to Vectors of the same dimension ; Will and Concatenate and get the mask vector ; Will With key value Multiply them together to get the initial representation of the occlusion vector.

Images