SSM-based model for generating long document abstract / keyword

Abstract

The invention relates to an SSM-based model for generating long document abstracts / keywords. The SSM-based model comprises an encoder, a decoder and an output vector module, the encoder comprises an SSM and a bit-by-bit feed-forward network. The SSM is used for converting an input text sequence through hidden state representation to generate an output text association sequence; the bit-by-bit feed-forward network is used for performing nonlinear transformation on the representation vector of each position in the text association sequence to generate code output; the decoder is of a multi-layer structure, and a target sequence is generated step by step according to codes output by the encoder and output of a previous layer of the decoder; and an output vector module reads a target sequence output by the decoder, and obtains a text output by the model based on the temperature coefficient and a Top-k sampling method. By introducing the SSM, the model can process longer texts, and meanwhile, the expenditure of computing resources is reduced. And then, in a decoder, a decoding strategy of Top-k and a temperature coefficient is introduced, so that the content of the generated abstract text is more accurate and closer to the theme.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a model based on SSM for generating long document summaries / keywords. Background Art

[0002] Currently, the Transformer model is often used to generate text summaries after text recognition. However, some types of documents, such as scientific documents, are usually long, involving complex structures and a large amount of information, while the input length of the Transformer model is usually limited. When processing longer scientific documents, the model may not be able to effectively capture global information, resulting in an incomplete summary. At the same time, scientific documents often contain a large number of professional terms and complex technical concepts. Even after large-scale pre-training, existing models may still not be able to fully understand the knowledge in a specific field. This may result in the generated summary being lacking in professionalism and accuracy.

[0003] The first problem is that it cannot handle long documents: 1. Insufficient global information capture: Existing Transformer models (such as BART and BERT) face input length limitations when processing long documents (such as BERT's default 512 tokens), so the model can usually only process part of the document and cannot fully capture all the key information in the document. Scientific documents are usually complex in structure and long in length. The single input limitation may cause the summary generation model to miss key paragraphs or important information, and the generated summary is relatively one-sided and lacks an overall overview of the document. This limitation is particularly prominent when processing long scientific documents, because it not only includes the main text, but also contains important information such as tables, images, and citations.

[0004] 2. Insufficient processing capabilities for complex structured documents: Scientific and technological documents often contain complex structures such as tables, formulas, and cross-paragraph references. The existing Transformer architecture is mainly targeted at linear texts and has difficulty identifying and processing these nonlinear or structured information. In particular, the existing model cannot well understand and integrate cross-paragraph relevance and reference information, resulting in the generated abstract lacking logic and integrity of information transmission. This may cause the abstract to miss important interconnected arguments, data, or conclusions in the document, affecting its comprehensive presentation of the main theme of the document.

[0005] The second problem is the high computational resource overhead: 1. The computational complexity of the self-attention mechanism is high: The Transformer model relies on the self-attention mechanism to capture long-distance dependencies in the sequence, and the computational complexity of this mechanism is O(n²), that is, as the length of the input sequence increases, the computational complexity of the model increases quadratically. For long texts such as scientific literature, the demand for computing resources will increase significantly. This not only causes the model to require a large amount of computing resources during training, but also makes the computational cost of the inference stage very high. Especially in actual application scenarios, if a large number of scientific literature needs to be summarized in real time or in batches, the computational overhead will become a bottleneck.

[0006] 2. Excessive memory consumption when processing long sequences: In the Transformer architecture, each word needs to be calculated with all other words in the sequence, which will cause the model's memory consumption to increase dramatically when processing long documents. Especially for long texts in scientific literature, the sequence length often exceeds the model's default processing capabilities, resulting in a rapid expansion of computing resource requirements. Existing hardware architectures may face memory overflow or performance degradation when processing these ultra-long sequences, greatly limiting the actual application scenarios of the model.

[0007] At present, in order to adapt to the input length limit of the model, the common processing method is to truncate or slide the window of the long document, but this method often leads to the loss of important information. The head and tail information of the document may be ignored, or the long-distance contextual connection cannot be captured, affecting the integrity of the summary. The model cannot process long sequences in parallel: Due to the global computing characteristics of the self-attention mechanism, the model must process the dependencies between each word in the sequence and all other words, resulting in limited computational parallelism, especially when the sequence is long, it cannot fully utilize the advantages of parallel computing, reducing computational efficiency. At the same time, for generative summary models (such as BART), not only does the training phase require a lot of resources, but the inference phase also has high overhead. When generating summaries, each step of the decoder depends on the previous output. This auto-regressive generation method further increases the computational cost, especially when generating long summaries, the time and resource consumption of inference increase significantly. Summary of the invention

[0008] In view of the technical problems existing in the prior art, the present invention provides a model based on SSM for generating long document summaries / keywords. The introduction of SSM can enable the model to process longer texts while reducing the overhead of computing resources.

[0009] According to a first aspect of the present invention, a model for generating long document summaries / keywords based on SSM is provided, comprising: an encoder, a decoder and an output vector module; The encoder includes: an SSM and a bit-by-bit feedforward network; The SSM is used to transform the input text sequence through the hidden state representation to generate an output text association sequence; the bit-by-bit feedforward network is used to perform a nonlinear transformation on the representation vector of each position in the text association sequence to generate a coded output; The decoder is a multi-layer structure, which gradually generates a target sequence according to the code output by the encoder and the output of the previous layer of the decoder; The output vector module reads the target sequence output by the decoder, and obtains the text output by the model based on the temperature coefficient and the Top-k sampling method.

[0010] Based on the above technical solution, the present invention can also make the following improvements.

[0011] Optionally, the formula of the SSM is: ; ; Among them, x[k] represents the hidden state of the current sequence, u[k] represents the input of the sequence, y[k] represents the output of the sequence, and Ad, Bd, Cd, and Dd are model parameters.

[0012] Optionally, the bitwise feedforward network includes two fully connected layers and a nonlinear activation function, expressed as: ; Where x is the input vector, and is the weight matrix, and is the bias term, and max represents the ReLU activation function.

[0013] Optionally, before the input text sequence / the output of the previous layer of the decoder enters the encoder / decoder, it is also subjected to a learnable position encoding to add position information.

[0014] Optionally, the decoder comprises: a bit-by-bit feedforward network and a multi-head attention module; The multi-head attention module includes: multi-head attention and masked multi-head attention; the code output by the encoder is output to the bit-by-bit feedforward network after passing through the multi-head attention, and the output of the previous layer of the decoder is output to the multi-head attention and the bit-by-bit feedforward network after passing through the masked multi-head attention; The bit-by-bit feedforward network performs nonlinear transformation on the representation vector of each position to obtain the target sequence of the decoder.

[0015] Optionally, the expression of the multi-head attention is: ; in, is the output of multi-head attention, Q is the query matrix, K is the key matrix, V is the value matrix, is the dimension of query and key, softmax represents the softmax function; The expression of the masked multi-head attention is: ; To mask the output of multi-head attention, mask represents the word used to mask future words.

[0016] Optionally, the processing of the output vector module includes: The target sequence output by the decoder is read, each word in the target sequence is converted into a probability distribution based on a temperature coefficient, top k words with the highest probability are selected from the target sequence based on a vocabulary of a model, and random sampling is performed among the top k words.

[0017] Optionally, the expression for converting each word in the target sequence into a probability distribution based on the temperature coefficient is: ; in, is the probability of the original generated i-th word, N is the total number of words, j is the index from 1 to N, T is the temperature coefficient, Represents the softmax probability of the i-th word.

[0018] Optionally, after the output vector module outputs the text, the step further includes: extracting keywords from the text based on a PositionRank keyword extraction algorithm.

[0019] Optionally, the process of extracting keywords from the text based on the PositionRank keyword extraction algorithm includes: Obtaining a hidden vector sequence generated by the decoder; The vector sequence is clustered using a K-means clustering algorithm, and a candidate keyword cluster is represented by a cluster center vector; Use cluster center vectors as nodes and use the similarity between nodes to establish edges; The final PageRank value of each node is obtained by iterative calculation, all nodes are sorted according to the final PageRank value, and a set number of nodes with the highest ranking are selected as keywords.

[0020] Optionally, the formula for obtaining the final PageRank value of each node by iterative calculation is: ; in, is the PageRank value of node i, d is the damping factor, N is the total number of nodes, M(i) is the set of other nodes pointing to node i, PR(j) is the PageRank value of node j, and L(j) is the number of edges starting from node j. The present invention provides a model based on SSM for generating long document summaries / keywords, introduces a state space model to replace the self-attention layer of the encoder, uses the recursive mechanism of SSM, makes each word element only depend on the state of the previous word element, reduces the computational complexity to O(n), greatly reduces the processing cost of long sequences, and improves the efficiency of summary generation.

[0021] The Transformer-based BART model is adopted, and the attention mechanism and end-to-end architecture are introduced to improve the feature extraction capability of the model while ensuring the logical coherence of the summary content.

[0022] A decoding strategy combining Top-k and temperature coefficient is adopted. Top-k limits the range of candidate words, avoids generating low-probability words, and ensures the compactness and relevance of the summary content. The temperature coefficient adjusts the randomness of the generated words, moderately introduces diversity, and balances the coherence and creativity of the generated content.

[0023] The PositionRank keyword extraction algorithm is introduced to extract keywords from the vectors generated by the BART decoder. This process combines the word vector representation, semantic similarity, and generation position of the generated text, thereby improving the accuracy and relevance of keyword extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic structural diagram of an embodiment of a model for generating long document summaries / keywords based on SSM provided by the present invention. DETAILED DESCRIPTION

[0025] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0026] Figure 1 A schematic diagram of the structure of an embodiment of a model for generating long document summaries / keywords based on SSM (Structured Space Model) provided by the present invention, such as Figure 1 As shown, the model includes: encoder, decoder and output vector modules.

[0027] The encoder includes: SSM and a bit-by-bit feed-forward network.

[0028] SSM is used to transform the input text sequence through hidden state representation to generate an output text association sequence; the bit-by-bit feedforward network is used to perform nonlinear transformation on the representation vector of each position in the text association sequence to generate the encoded output.

[0029] The decoder has a multi-layer structure, which gradually generates the target sequence based on the code output by the encoder and the output of the previous layer of the decoder.

[0030] The output vector module reads the target sequence output by the decoder and obtains the text output by the model based on the temperature coefficient and Top-k sampling method.

[0031] The present invention provides a model for generating long document summaries / keywords based on SSM. The introduction of SSM enables the model to process longer texts while reducing the cost of computing resources. Then, in the decoder, the Top-k and temperature coefficient decoding strategies are introduced to make the generated summary text content more accurate and closer to the topic.

[0032] Example 1 Embodiment 1 provided by the present invention is an embodiment of a model based on SSM for generating long document summaries / keywords provided by the present invention, combined with Figure 1 It can be seen that an embodiment of the model includes: an encoder, a decoder and an output vector module.

[0033] The encoder includes: SSM and a bit-by-bit feed-forward network.

[0034] SSM is used to transform the input text sequence through hidden state representation to generate an output text association sequence; the bit-by-bit feedforward network is used to perform nonlinear transformation on the representation vector of each position in the text association sequence to generate the encoded output.

[0035] In a possible implementation manner, the formula of SSM is: .

[0036] .

[0037] Among them, x[k] represents the hidden state of the current sequence, u[k] represents the input of the sequence, y[k] represents the output of the sequence, and Ad, Bd, Cd, and Dd are model parameters.

[0038] SSM describes the state representation of the text at each sorting position and predicts its next state based on the input. It transforms the input text sequence through the hidden state representation and finally generates the output text association sequence. This discretization makes SSM easier to implement and train on a computer.

[0039] In a possible implementation manner, the bitwise feedforward network includes two fully connected layers and a nonlinear activation function, expressed as: .

[0040] Where x is the input vector, and is the weight matrix, and is the bias term, and max represents the ReLU activation function.

[0041] The first layer of linear transformation (weight matrix W1) of the bit-by-bit feedforward network increases the input dimension, allowing the network to learn more information. The activation function (such as ReLU) introduces nonlinearity, allowing the network to capture complex features. The second layer of linear transformation (weight matrix W2) reduces the dimension back to the original input size. At the same time, in the bit-by-bit feedforward network, the processing of each position is independent, which can ensure that the local features are fully learned, and ultimately achieve computational parallelism, greatly improving the efficiency of training and reasoning, and enhancing the expressiveness of the model.

[0042] The decoder has a multi-layer structure, which gradually generates the target sequence based on the code output by the encoder and the output of the previous layer of the decoder.

[0043] In one possible embodiment, the decoder includes: a bit-by-bit feedforward network and a multi-head attention module.

[0044] The multi-head attention module includes: multi-head attention and masked multi-head attention; the encoding output by the encoder is output to the bit-by-bit feedforward network after passing through the multi-head attention, and the output of the previous layer of the decoder is output to the multi-head attention and bit-by-bit feedforward network after passing through the masked multi-head attention.

[0045] The bit-by-bit feedforward network performs a nonlinear transformation on the representation vector of each position to obtain the target sequence of the decoder.

[0046] In one possible implementation manner, the expression of multi-head attention is: .

[0047] in, is the output of multi-head attention, Q is the query matrix, K is the key matrix, V is the value matrix, is the dimension of query and key, and softmax represents the softmax function.

[0048] The expression of masked multi-head attention is: .

[0049] To mask the output of multi-head attention, mask represents the word used to mask future words.

[0050] Multi-head attention is a way to parallelize the attention mechanism, allowing the model to capture the dependencies between different positions in the sequence in multiple different subspaces, and allowing the model to calculate attention in parallel in multiple subspaces, thereby capturing more rich contextual information.

[0051] Multi-head attention linearly transforms the input to generate a query vector (Query), a key vector (Key), and a value vector (Value). Each head calculates the dot product of the query and the key, obtains the weight of each value vector, generates different attention distributions, concatenates the attention results of all heads, and generates the final output through a linear transformation; masked multi-head attention masks future time steps when calculating the dot product of the query and the key.

[0052] In the decoder, masked multi-head attention is used. Masked multi-head attention has the same structure as multi-head attention, but when calculating the dot product of the query and the key, future time steps are masked. Masked multi-head attention can ensure that the model generates text in sequence.

[0053] In one possible implementation manner, before the input text sequence / the output of the previous layer of the decoder enters the encoder / decoder, it is also subjected to a learnable position encoding to add position information.

[0054] Learnable positional encodings are different from fixed positional encodings in that they learn the positional encoding vector as part of the model parameters during training. They do not rely on predefined rules (such as sine and cosine functions), but directly learn the position information that best suits the task through data. Therefore, when dealing with complex sequence tasks, learnable positional encodings can adapt to the impact of different positions on the task, unlike fixed encodings that are limited to static position information.

[0055] In one possible embodiment, The output vector module reads the target sequence output by the decoder and obtains the text output by the model based on the temperature coefficient and Top-k sampling method.

[0056] In a possible implementation manner, the processing of the output vector module includes: Read the target sequence output by the decoder, convert each word in the target sequence into a probability distribution based on the temperature coefficient, select the top k words with the highest probability from the target sequence based on the vocabulary of the model, and perform random sampling among the top k words.

[0057] Top-k sampling is a method to control the generated content by limiting the range of candidate words when generating text. The main goal of Top-k sampling is to avoid generating words with too low probability and ensure the quality and coherence of the generated content. By adjusting the value of k, the output is restricted so that the generated text will not contain some low-probability, random words, avoid the generated content being too divergent, and the generated text is more in line with the actual context.

[0058] In a possible embodiment, the expression for converting each word in the target sequence into a probability distribution based on the temperature coefficient is: .

[0059] in, is the probability of the original generated i-th word, N is the total number of words, j is the index from 1 to N, T is the temperature coefficient, Represents the softmax probability of the i-th word.

[0060] The temperature coefficient acts on the probability distribution of words, adjusting the relative probabilities between words. When the temperature is high, the model will "smooth" the probability distribution, and the probability difference between words will become smaller. At this time, the model is more random, and even the probability of selecting low-probability words will increase, and the generated content will be more diverse and creative. When the temperature is low, the probability distribution of the model will be more sharp, the probability of selecting high-probability words will increase greatly, and low-probability words will hardly be selected, and the generated content will be more certain and coherent. When T=1, the generated result keeps the original probability distribution unchanged. When T<1, high-probability words become more prominent and the generation is more certain. When T>1, the probability distribution is flatter, and the model generation is more diverse and random.

[0061] Combine Top-k and temperature coefficient to achieve both diversity and coherence in the generated content. Top-k limits the range of candidate words to ensure that the generated words are not too divergent, while the temperature coefficient adjusts the randomness of the selection process to make the generated results more natural or creative.

[0062] In a possible implementation manner, after the output vector module outputs the text, the method further includes: extracting keywords from the text based on a PositionRank keyword extraction algorithm.

[0063] PositionRank is a graph-based unsupervised keyword extraction algorithm that improves the effect of keyword extraction by combining the position information and co-occurrence relationship of words.

[0064] In a possible implementation manner, the process of extracting keywords from a text based on the PositionRank keyword extraction algorithm includes: Obtain the hidden vector sequence generated by the decoder; in order to combine the advantages of the BART model, the embodiment of the present invention applies the PositionRank keyword extraction algorithm to the hidden vectors generated by the decoder, because these vectors have captured the semantic information of each generated position and can be directly operated in the vector space.

[0065] The K-means clustering algorithm is used to cluster the vector sequence, and the candidate keyword clusters are represented by cluster center vectors; each cluster can be regarded as a potential keyword region, corresponding to a group of semantically similar vectors, so that each candidate keyword can be represented by the cluster center vector. Then, the average similarity of the vectors in each cluster is calculated to determine the quality of the cluster as a candidate keyword, and the candidate keywords are sorted according to representativeness or cluster size, thus completing the cluster sorting of candidate keyword vectors.

[0066] The cluster center vectors are used as nodes of the graph, and the similarities between nodes are used to establish edges.

[0067] The weight of the edge reflects the semantic similarity between cluster centers. At the same time, the nodes corresponding to the vectors near the beginning of the sequence can be assigned higher initial scores, because in PositionRank, the initial weight of each word depends not only on the frequency of the word, but also on the position where it first appears in the text.

[0068] The final PageRank value of each node is obtained by iterative calculation, all nodes are sorted according to the final PageRank value, and a set number of nodes with the highest ranking are selected as keywords.

[0069] The formula for iterative calculation is: .

[0070] Where PR(i) is the PageRank value of node i. d is the damping factor, usually set to 0.85, to simulate the probability of random jumps to other nodes. N is the total number of nodes. M(i) is the set of other nodes pointing to node i. PR(j) is the PageRank value of node j. L(j) is the out-degree of node j, that is, the number of edges from node j.

[0071] The PageRank value of each node is calculated iteratively using the iterative calculation formula until the change in the PageRank value is less than a preset threshold such as 10-6, or the maximum number of iterations is reached. When converged, the PageRank value of each node represents its relative importance in the graph. Finally, all nodes are sorted according to the final PageRank value. Nodes with higher scores are considered more important, and the top several nodes are selected as keywords.

[0072] Unlike deep learning-based methods, the PositionRank keyword extraction algorithm is an unsupervised algorithm that does not require training data and can be used directly in various texts or vector spaces. At the same time, PositionRank adds prior information in the word extraction stage, and uses the position information and lexical relationships of words to further optimize input features, thereby improving the accuracy and robustness of the model in downstream tasks. The embodiment of the present invention combines PositionRank with BART to achieve more accurate text summarization and keyword extraction without increasing the computational overhead too much.

[0073] Finally, the PositionRank keyword extraction algorithm is used after the decoder to extract keywords from the output vector, reducing the computational overhead while ensuring the accuracy of the extraction results.

[0074] The embodiment of the present invention provides a model based on SSM for generating long document summaries / keywords, which increases the token length of text input and speeds up training and reasoning. Due to the introduction of the recursive mechanism of SSM, the computational cost of long sequence processing is greatly reduced. Compared with the traditional self-attention mechanism, it can more effectively process long-length inputs such as scientific and technological documents, and generate summaries faster. At the same time, through the attention mechanism introduced by Transform, the model can better capture complex feature patterns, and the generated summary information is more accurate and logical, which is suitable for summarizing complex scientific and technological document content. The combined use of decoding strategies based on Top-k and temperature coefficients ensures that the generated summary is not only accurate and compact in content, but also has a certain degree of flexibility, making the generated results more expressive and readable. At the same time, the PositionRank keyword extraction algorithm is introduced to extract keywords in the vector space of the decoder, which reduces the computational overhead while ensuring the accuracy of the extraction results.

[0075] The embodiment of the present invention provides a model for generating long document summaries / keywords based on SSM, and its beneficial effects include: The state space model is introduced to replace the self-attention layer of the encoder. The recursive mechanism of SSM is used to make each word only depend on the state of the previous word, reducing the computational complexity to O(n), greatly reducing the processing cost of long sequences and improving the efficiency of summary generation.

[0076] The Transformer-based BART model is adopted, and the attention mechanism and end-to-end architecture are introduced to improve the feature extraction capability of the model while ensuring the logical coherence of the summary content.

[0077] A decoding strategy combining Top-k and temperature coefficient is adopted. Top-k limits the range of candidate words, avoids generating low-probability words, and ensures the compactness and relevance of the summary content. The temperature coefficient adjusts the randomness of the generated words, moderately introduces diversity, and balances the coherence and creativity of the generated content.

[0078] The PositionRank keyword extraction algorithm is introduced to extract keywords from the vectors generated by the BART decoder. This process combines the word vector representation, semantic similarity, and generation position of the generated text, thereby improving the accuracy and relevance of keyword extraction.

[0079] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0084] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0085] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A model based on SSM for generating long document summaries / keywords, characterized in that: The model includes: an encoder, a decoder and an output vector module; The encoder includes: an SSM and a bit-by-bit feedforward network; The SSM is used to transform the input text sequence through the hidden state representation to generate an output text association sequence; the bit-by-bit feedforward network is used to perform a nonlinear transformation on the representation vector of each position in the text association sequence to generate a coded output; The decoder is a multi-layer structure, which gradually generates a target sequence according to the code output by the encoder and the output of the previous layer of the decoder; The output vector module reads the target sequence output by the decoder, and obtains the text output by the model based on the temperature coefficient and the Top-k sampling method.

2. The model according to claim 1, characterized in that The formula of the SSM is: ； ； Among them, x[k] represents the hidden state of the current sequence, u[k] represents the input of the sequence, y[k] represents the output of the sequence, and Ad, Bd, Cd, and Dd are model parameters.

3. The model according to claim 1, characterized in that The bitwise feedforward network contains two fully connected layers and a nonlinear activation function, expressed as: ； Where x is the input vector, and is the weight matrix, and is the bias term, and max represents the ReLU activation function.

4. The model according to claim 1, characterized in that Before the input text sequence / the output of the previous layer of the decoder enters the encoder / decoder, position information is added through a learnable position encoding.

5. The model according to claim 1, characterized in that The decoder includes: a bit-by-bit feedforward network and a multi-head attention module; The multi-head attention module includes: multi-head attention and masked multi-head attention; the code output by the encoder is output to the bit-by-bit feedforward network after passing through the multi-head attention, and the output of the previous layer of the decoder is output to the multi-head attention and the bit-by-bit feedforward network after passing through the masked multi-head attention; The bit-by-bit feedforward network performs nonlinear transformation on the representation vector of each position to obtain the target sequence of the decoder.

6. The model according to claim 5, characterized in that The expression of the multi-head attention is: ； in, is the output of multi-head attention, Q is the query matrix, K is the key matrix, V is the value matrix, is the dimension of query and key, softmax represents the softmax function; The expression of the masked multi-head attention is: ； To mask the output of multi-head attention, mask represents the word used to mask future words.

7. The model according to claim 1, characterized in that The processing process of the output vector module includes: The target sequence output by the decoder is read, each word in the target sequence is converted into a probability distribution based on a temperature coefficient, top k words with the highest probability are selected from the target sequence based on a vocabulary of a model, and random sampling is performed among the top k words.

8. The model according to claim 7, characterized in that The expression for converting each word in the target sequence into a probability distribution based on the temperature coefficient is: ； in, is the probability of the original generated i-th word, N is the total number of words, j is the index from 1 to N, T is the temperature coefficient, Represents the softmax probability of the i-th word.

9. The model according to claim 1, characterized in that After the output vector module outputs the text, the method further includes: extracting keywords from the text based on a PositionRank keyword extraction algorithm; The process of extracting keywords from the text based on the PositionRank keyword extraction algorithm includes: Obtaining a hidden vector sequence generated by the decoder; The vector sequence is clustered using a K-means clustering algorithm, and a candidate keyword cluster is represented by a cluster center vector; Use cluster center vectors as nodes and use the similarity between nodes to establish edges; The final PageRank value of each node is obtained by iterative calculation, all nodes are sorted according to the final PageRank value, and a set number of nodes with the highest ranking are selected as keywords.

10. The model according to claim 9, characterized in that The formula for obtaining the final PageRank value of each node through iterative calculation is: ； in, is the PageRank value of node i, d is the damping factor, N is the total number of nodes, M(i) is the set of other nodes pointing to node i, PR(j) is the PageRank value of node j, and L(j) is the number of edges starting from node j.

Images