Long text abstract generation method and device

Abstract

The invention provides a long text abstract generation method and device, and belongs to the technical field of natural text processing, the method comprises the following steps: using Euclidean norm to carry out importance sorting on Tokens and carrying out compression storage according to a sparse rate, so that global key information is completely reserved and memory occupation is obviously reduced; in the decoding stage, local attention scores and global attention scores are calculated in parallel, entropy differences are mapped into fusion weights through Sigmoid by combining temperature adjusting parameters, and dynamic balance of local details and long-distance dependence is achieved. The local key value pairs and the global key value pairs are subjected to weighted integration based on the fusion weight, a continuous semantic spectrum is formed in a single decoding layer, splicing breakage caused by traditional partitioning is eliminated, the problems of input limitation and semantic splitting are effectively relieved, the context length capable of being processed by a model is expanded under the condition that the calculation amount is not remarkably increased, and the method has the advantages of being simple in structure and convenient to operate. And local and global context information is adaptively fused, so that the accuracy and continuity of the abstract are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a long text abstract generation method and device. BACKGROUND

[0002] Text abstract generation is an important task in the field of natural language processing, which aims to compress lengthy original text into a concise and key information-containing abstract. This technology has important application value in news aggregation, public opinion monitoring, legal document processing, academic information retrieval, etc. In recent years, with the rise of pre-training language models, generative abstract methods based on the Transformer architecture have made significant progress in English long text tasks, such as BERT, BART, Longformer, BigBird, T5, etc.

[0003] However, such models still have some problems when processing Chinese long text abstracts, the main reasons including: language characteristic differences, Chinese has significant differences in grammar structure, word segmentation mechanism, and chapter organization method from English, and direct migration of English long text modeling technology often has limited effect; high training cost, training a pre-training model suitable for Chinese long text from scratch requires extremely high computing and storage resources, which is difficult for most research and development teams to bear; insufficient data resources, the length of existing Chinese abstract data sets is generally short, the average input length is much lower than the requirement of long text tasks, and the annotation quality is uneven, which is difficult to support the training and evaluation of high-quality long text generation models.

[0004] Researchers have carried out a lot of exploration in long text modeling, but the existing methods generally have the problems of limited context length, difficulty in maintaining coherence of cross-paragraph semantics, and high reasoning cost. Especially in Chinese tasks, limited by corpus structure and model adaptation, its performance in long-distance dependency capture and abstract condensation has not yet reached the ideal level. The block processing strategy divides long text into fragments for independent coding, and then concatenates the input decoder. This method expands the input capacity, but causes semantic fragmentation between fragments, and key information is lost in the concatenation process (such as cross-fragment reference resolution failure). There is also a sparse attention mechanism, which reduces computational complexity through local window, random attention, etc. However, such methods require customized pre-training and are difficult to adapt to Chinese language characteristics, and tend to ignore global key information when generating abstracts. SUMMARY

[0005] The present application provides a long text abstract generation method and device to solve the defects in the prior art and improve the context modeling capability of long text.

[0006] In a first aspect, the present application provides a long text abstract generation method, comprising: According to the preset preprocessing process, the context-related hidden vector representation of each token is extracted from the long Chinese text to be processed; Obtain the Euclidean norm of the context-related hidden vector representation, and selectively store tokens based on the Euclidean norm, a preset sparsity rate, and the total number of tokens to obtain a sparsely stored set of key tokens. During the decoding phase, the local attention score and global attention score are calculated in parallel for the current layer query vector; Based on the local attention score and the global attention score, the local entropy and the global entropy are determined, and a temperature adjustment parameter is introduced to map the entropy difference between the local entropy and the global entropy through the Sigmoid function to obtain the fusion weight; Based on the fusion weight, a weighted fusion is performed on the local key-value pairs and the global key-value pairs to obtain fused key-value pairs. The fused key-value pairs are then input into the decoder to generate a digest token sequence, which is used to output a Chinese long text digest.

[0007] Furthermore, the step of extracting the context-dependent hidden vector representation of each token from the long Chinese text to be processed according to a preset preprocessing procedure includes: Obtain the token sequence of the long Chinese text to be processed after word segmentation or sub-word segmentation; Using a sliding window with a preset format, the token sequence is divided into overlapping blocks to generate multiple overlapping text blocks; The multiple overlapping text blocks are input into a pre-trained Chinese BART encoder, and multi-layer Transformer encoding is performed block by block to obtain the context-related hidden vector representation of each token in the high-dimensional semantic space.

[0008] Furthermore, the text block includes a start position and an end position, and the start position and end position satisfy the following relationship: ; in, Starting position This is the end position.

[0009] Furthermore, the local attention score is calculated from the local key-value pairs of the current layer, and the global attention score is calculated from the query vector of the current layer and the global key-value pairs retrieved from the sparsely stored key token set.

[0010] Furthermore, determining the local entropy and global entropy based on the local attention score and the global attention score includes: Soft max normalization is performed on the local attention score and the global attention score respectively to obtain the local attention weight and the global attention weight; Calculate local and global entropy based on the information entropy formula; The information entropy formula is specifically as follows: ; in, These are the local attention weights or global attention weights after Softmax normalization. It is a very small positive number.

[0011] Furthermore, the introduction of temperature regulation parameters maps the entropy difference between the local and global entropies using the Sigmoid function to obtain the fusion weights, specifically including: ; in, This represents the Sigmoid function. For local entropy, For global entropy, This indicates the temperature control parameter.

[0012] Furthermore, based on the aforementioned fusion weights, a weighted fusion is performed on the local key-value pairs and the global key-value pairs to obtain the fused key-value pairs, as shown in the following formula: ; ; in, and For local key-value pairs, and The final calculated key-value pairs are global key-value pairs. and , For weight fusion.

[0013] Secondly, the present invention also provides a long text summarization generation apparatus, comprising: The preprocessing module is used to extract the context-related hidden vector representation of each token from the long Chinese text to be processed according to the preset preprocessing process. The sparse storage module is used to obtain the Euclidean norm of the context-related hidden vector representation, and selectively store tokens according to the Euclidean norm, the preset sparsity rate and the total number of tokens, to obtain the sparsely stored key token set. The attention calculation module is used to calculate the local attention score and global attention score in parallel on the current layer query vector during the decoding phase. The fusion weight calculation module is used to determine the local entropy and global entropy based on the local attention score and the global attention score, and to map the entropy difference between the local entropy and the global entropy through the Sigmoid function by introducing a temperature adjustment parameter to obtain the fusion weight; The entropy-guided attention fusion module is used to perform weighted fusion of local key-value pairs and global key-value pairs according to the fusion weights to obtain fused key-value pairs. The content generation module is used to input the fused key-value pairs into the decoder to generate a digest token sequence, so as to output a long Chinese text digest.

[0014] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the long text summarization methods described above.

[0015] This invention, without increasing model parameters, utilizes the Euclidean norm to prioritize tokens and compresses their storage according to sparsity, ensuring complete preservation of global key information while significantly reducing memory usage. During the decoding phase, local and global attention scores are calculated in parallel, and the entropy difference is mapped to a fusion weight via a Sigmoid function using a temperature adjustment parameter, achieving a dynamic balance between local details and long-distance dependencies. Based on this fusion weight, local and global key-value pairs are weighted and integrated, forming a continuous semantic spectrum within a single decoding layer, eliminating splicing breaks caused by traditional block segmentation. The final generated summary token sequence maintains factual consistency and coherent expression even when the input far exceeds a fixed window length, significantly reducing redundant computation and improving summary accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an optional long text summarization method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an optional text segmentation process provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of an optional process for sparse storage of a token set provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of an optional entropy-guided attention fusion process provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an optional electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0020] The following is combined Figures 1-3 This invention describes the long text summarization method and apparatus provided in embodiments of the present invention.

[0021] Figure 1 This is a flowchart illustrating the long text summarization method provided by the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps: Step 102: According to the preset preprocessing flow, extract the context-related hidden vector representation of each token from the long Chinese text to be processed; Step 104: Obtain the Euclidean norm of the context-related hidden vector representation, and selectively store the tokens according to the Euclidean norm, the preset sparsity rate and the total number of tokens to obtain the sparsely stored key token set. Step 106: During the decoding phase, calculate the local attention score and global attention score in parallel for the current layer query vector; Step 108: Based on the local attention score and the global attention score, determine the local entropy and the global entropy, and introduce a temperature adjustment parameter to map the entropy difference between the local entropy and the global entropy through the Sigmoid function to obtain the fusion weight; Step 110: Based on the fusion weight, perform weighted fusion on the local key-value pairs and the global key-value pairs to obtain fused key-value pairs; input the fused key-value pairs into the decoder to generate a digest token sequence to output a Chinese long text digest.

[0022] Regarding step 102: According to the preset preprocessing flow, extract the context-related hidden vector representation of each Token from the long Chinese text to be processed.

[0023] The long text sequence is sequentially fed into the encoder of the preset text segmentation processing program using overlapping sliding windows. The window size can be fixed along with the stride to ensure semantic overlap between adjacent segments. The hidden state tensor output by the last layer of the encoder is the "context-dependent hidden vector representation," and its dimension d is consistent with the model base so that all subsequent importance calculations can directly reuse this tensor without additional forward computation.

[0024] Optionally, the step of extracting the context-related hidden vector representation of each token from the Chinese long text to be processed according to the preset preprocessing process in the long text summarization method of this embodiment includes: obtaining the token sequence after word segmentation or sub-word segmentation of the Chinese long text to be processed; Using a sliding window with a preset format, the token sequence is divided into overlapping blocks to generate multiple overlapping text blocks; The multiple overlapping text blocks are input into a pre-trained Chinese BART encoder, and multi-layer Transformer encoding is performed block by block to obtain the context-related hidden vector representation of each token in the high-dimensional semantic space.

[0025] In this embodiment, Figure 2 This is a flowchart illustrating the text segmentation process provided by the present invention, as shown below. Figure 2 As shown, the preprocessing flow for long Chinese texts using text segmentation is as follows: First, the long Chinese text is input for preprocessing; then, the block processing process is executed, which divides the input long Chinese text into overlapping blocks using a sliding window with a fixed size and stride. Next, the block encoding process is executed, which inputs the constructed text blocks into the BART encoder to generate a context-dependent hidden vector representation of each token in the high-dimensional semantic space.

[0026] The text block includes a start position and an end position, and the start position and end position satisfy the following relationship: ; in, Starting position This is the end position.

[0027] Regarding step 108: Obtain the Euclidean norm of the context-related hidden vector representation, and selectively store tokens based on the Euclidean norm, the preset sparsity rate, and the total number of tokens to obtain a sparsely stored set of key tokens.

[0028] During sparse storage, the Euclidean norm of each context vector is calculated in real time as a quantification of the vector's saliency. A configurable circular buffer is maintained simultaneously to store the vectors with the highest norm online. The control unit dynamically calculates the retention limit based on a preset sparsity rate R: when the buffer reaches its limit, a new vector is replaced only if its norm is greater than the minimum value of existing vectors and passes consistency verification. This ensures that globally critical information remains in GPU memory, while GPU memory usage is linearly compressed with R. Discarded vectors are immediately released from GPU memory and are no longer accessed in subsequent attention calculations, forming a sparse representation of the "key token set".

[0029] For example, Figure 3 This is a schematic diagram of the process for sparse storage of a token set provided by the present invention, as shown below. Figure 3 As shown, the L2 norm of the vector corresponding to each Token is calculated for the context-dependent hidden vector representation. Based on the preset sparsity R and the total number of Tokens L, the number of Tokens to be retained is determined. The hidden vectors of the top n Tokens with the largest L2 norm are stored as key vectors in the global sparse memory, and the remaining Token vectors are discarded, thereby reducing memory overhead.

[0030] Regarding step 106: During the decoding phase, the local attention score and global attention score are calculated in parallel for the current layer query vector.

[0031] During the decoding phase, the local attention score and the global attention score are calculated in parallel for the current layer query vector. The local attention score is calculated from the local key-value pairs of the current layer, and the global attention score is calculated from the current layer query vector and the global key-value pairs retrieved from the sparsely stored key token set.

[0032] Figure 4 This is a schematic diagram of an entropy-guided attention fusion process according to the present invention, as shown below. Figure 4 As shown, the decoder unfolds two attention paths in parallel at each layer: one is the local path, which uses the query vector of the current layer... Based on, and randomly sampled key-value pairs within the same window , Calculate the dot product to obtain the local attention score; secondly, the global path, the previous layer query vector. With sparse storage area , Calculate the dot product to obtain the global attention score.

[0033] Regarding step 108: Based on the local attention score and the global attention score, determine the local entropy and the global entropy, and introduce a temperature adjustment parameter to map the entropy difference between the local entropy and the global entropy through the Sigmoid function to obtain the fusion weight.

[0034] Calculate the information entropy of the normalized attention distribution to obtain the local attention entropy. With global attention entropy The temperature regulation parameter T is injected in a learnable or manually set manner to scale the entropy difference. . After Sigmoid mapping, the output is a fusion weight α between 0 and 1. Its real-time value reflects the dynamic dependence of the current decoding position on local details and global context.

[0035] Optionally, Soft max normalization is performed on the local attention score and the global attention score respectively to obtain the local attention weight and the global attention weight; Calculate local and global entropy based on the information entropy formula; The information entropy formula is specifically as follows: ; in, These are the local attention weights or global attention weights after Softmax normalization. It is an extremely small positive number to prevent undefined logarithms or overflow problems in numerical computation, and to ensure the numerical stability of entropy calculation.

[0036] Optionally, the introduction of temperature regulation parameters maps the entropy difference between the local and global entropies using a Sigmoid function to obtain the fusion weights, specifically including: ; in, This represents the Sigmoid function. For local entropy, For global entropy, This indicates the temperature control parameter.

[0037] Regarding step 110: Based on the fusion weight, perform weighted fusion on the local key-value pairs and the global key-value pairs to obtain fused key-value pairs, and input the fused key-value pairs into the decoder to generate a digest token sequence to output a Chinese long text digest.

[0038] Perform linear interpolation on the K and V vectors according to the fusion weights: , Similarly, the fused key-value pairs are directly fed into the remaining branch of the multi-head attention network, and after passing through the feedforward network, the output hidden state of this layer is generated. The decoder executes the above process cyclically, gradually generating a sequence of digest tokens until the end symbol is triggered or the maximum length limit is reached, and finally concatenating them into a Chinese long text digest.

[0039] Based on the above embodiments, as an optional embodiment, the following specific example will further illustrate a long text summarization method of the present invention.

[0040] Step 1: Input the long Chinese text that needs to be processed; Step 2, block processing, is as follows: The input length from Step 1 is... input sequence Using a sliding window with a fixed size of 1024 tokens and a fixed stride of 512 tokens, the input long Chinese text is segmented into overlapping text blocks:

[0041] in, It is the number of blocks. and They are the first The start and end positions of each block, and due to text block overlap, therefore The overlap ratio is The overlap ratio here is .

[0042] Step 3, the block encoding process, is as follows: The input long Chinese text sequence is segmented into words or sub-words to obtain a discrete token sequence:

[0043] in, This represents the length of the input long Chinese text sequence. Each token is embedded and mapped to obtain its word vector representation:

[0044] in Represents a word embedding mapping function. This represents positional encoding, used to preserve the positional information of the sequence. The embedded sequence is fed into the multi-layer Transformer encoding module of the BART encoder, with each layer consisting of a multi-head self-attention mechanism and a feedforward network. The encoding process is as follows:

[0045]

[0046] in Indicates the number of coding layers. For the initial embedding vector, For the first Layer output. After all encoding layers, the context-dependent hidden vector representation of each token is obtained: .

[0047] like Figure 3 As shown, sparse storage includes the following steps: Step 4: Calculate the L2 norm of each token, as follows: High-dimensional semantic vector representation of a single token. ,in Representing vector dimension; through formula To calculate the L2 norm.

[0048] Step 5: Determine the token to be saved to global storage, specifically as follows: First, determine a sparsity coefficient. Based on the total number of the currently input token sequences It can be done through formula To calculate the number of tokens to retain, the input token sequence is arranged in descending order according to the L2 norm calculated in step 5. The tokens with the largest L2 norm are then selected. Each token is added to the global sparse storage; like Figure 4 As shown, entropy-guided attention fusion includes the following steps: Step 6: Calculate the attention scores for local and global attention. The local attention score is obtained by calculating the similarity between the current input query vector and the local key-value pairs of the current layer. For the decoder... Layer, query vector Key-value pairs from decoder hidden state Encoder state sampled randomly from the current layer:

[0049]

[0050]

[0051] Then through the formula

[0052] The local attention score is calculated. Simultaneously, the global attention key-value pairs retrieved from memory are used... You can then use the formula ; Calculate the global attention score According to the formula

[0053] We calculate the local attention weights and global attention weights separately. Then, we calculate the local attention entropy based on the local attention weights and global attention weights. and global attention entropy , through formula

[0054] To calculate the local entropy separately and global entropy Finally, the fusion weights are calculated, incorporating a temperature parameter. To adjust for the influence of entropy difference, the result is calculated using the Sigmoid function. Final fusion weights within the range The specific formula used is as follows:

[0055] in This represents the Sigmoid function. Based on the calculated fusion weights... The local key-value pairs and global key-value pairs are merged using a weighted average, where the weight of the local key-value pairs is... The weight of the global key-value pairs is The formula is as follows:

[0056]

[0057] in and For local key-value pairs, and The final calculated key-value pairs are global key-value pairs. and .

[0058] On the other hand, the present invention also provides a long text summarization generation device, the device comprising: a preprocessing module, used to extract the context-related hidden vector representation of each token from the Chinese long text to be processed according to a preset preprocessing process; The sparse storage module is used to obtain the Euclidean norm of the context-related hidden vector representation, and selectively store tokens according to the Euclidean norm, the preset sparsity rate and the total number of tokens, to obtain the sparsely stored key token set. The attention calculation module is used to calculate the local attention score and global attention score in parallel on the current layer query vector during the decoding phase. The fusion weight calculation module is used to determine the local entropy and global entropy based on the local attention score and the global attention score, and to map the entropy difference between the local entropy and the global entropy through the Sigmoid function by introducing a temperature adjustment parameter to obtain the fusion weight; The entropy-guided attention fusion module is used to perform weighted fusion of local key-value pairs and global key-value pairs according to the fusion weights to obtain fused key-value pairs. The content generation module is used to input the fused key-value pairs into the decoder to generate a digest token sequence, so as to output a long Chinese text digest.

[0059] It should be noted that the long text summarization generation device provided in this embodiment of the invention can execute the long text summarization generation method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0060] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can call logical instructions in the memory 530 to execute a long text summarization method. This method includes: extracting the context-dependent hidden vector representation of each token from the Chinese long text to be processed according to a preset preprocessing flow; obtaining the Euclidean norm of the context-dependent hidden vector representation, and selectively storing tokens according to the Euclidean norm, a preset sparsity rate, and the total number of tokens to obtain a sparsely stored key token set; in the decoding stage, parallel calculating local attention scores and global attention scores on the current layer query vector; determining local entropy and global entropy based on the local attention scores and global attention scores, and introducing a temperature adjustment parameter to map the entropy difference between the local entropy and global entropy using a Sigmoid function to obtain a fusion weight; performing weighted fusion on local key-value pairs and global key-value pairs according to the fusion weight to obtain fused key-value pairs, and inputting the fused key-value pairs into the decoder to generate a summary token sequence to output a Chinese long text summary.

[0061] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the long text summarization generation method provided in the above embodiments. The method includes: extracting the context-related hidden vector representation of each token from the Chinese long text to be processed according to a preset preprocessing flow; obtaining the Euclidean norm of the context-related hidden vector representation, and selectively storing tokens according to the Euclidean norm, a preset sparsity rate, and the total number of tokens to obtain a sparsely stored key token set; in the decoding stage, calculating the local attention score and the global attention score in parallel for the current layer query vector; determining the local entropy and the global entropy according to the local attention score and the global attention score, and introducing a temperature adjustment parameter to map the entropy difference between the local entropy and the global entropy through the Sigmoid function to obtain a fusion weight; performing weighted fusion on the local key-value pairs and the global key-value pairs according to the fusion weight to obtain fused key-value pairs, and inputting the fused key-value pairs into the decoder to generate a summary token sequence to output a Chinese long text summary.

[0063] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the long text summarization method provided in the above embodiments. The method includes: extracting the context-related hidden vector representation of each token from the Chinese long text to be processed according to a preset preprocessing flow; obtaining the Euclidean norm of the context-related hidden vector representation, and selectively storing tokens according to the Euclidean norm, a preset sparsity rate, and the total number of tokens to obtain a sparsely stored key token set; in the decoding stage, calculating the local attention score and the global attention score in parallel for the current layer query vector; determining the local entropy and the global entropy according to the local attention score and the global attention score, and introducing a temperature adjustment parameter to map the entropy difference between the local entropy and the global entropy through a Sigmoid function to obtain a fusion weight; performing a weighted fusion of the local key-value pairs and the global key-value pairs according to the fusion weight to obtain fused key-value pairs, and inputting the fused key-value pairs into the decoder to generate a summary token sequence to output a Chinese long text summary.

[0064] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0065] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating long text summaries, characterized in that, include: According to the preset preprocessing process, the context-related hidden vector representation of each token is extracted from the long Chinese text to be processed; Obtain the Euclidean norm of the context-related hidden vector representation, and selectively store tokens based on the Euclidean norm, a preset sparsity rate, and the total number of tokens to obtain a sparsely stored set of key tokens. During the decoding phase, the local attention score and global attention score are calculated in parallel for the current layer query vector; Based on the local attention score and the global attention score, the local entropy and the global entropy are determined, and a temperature adjustment parameter is introduced to map the entropy difference between the local entropy and the global entropy through the Sigmoid function to obtain the fusion weight; Based on the fusion weight, a weighted fusion is performed on the local key-value pairs and the global key-value pairs to obtain fused key-value pairs. The fused key-value pairs are then input into the decoder to generate a digest token sequence, which is used to output a Chinese long text digest.

2. The long text summarization method according to claim 1, characterized in that, The step of extracting the context-dependent hidden vector representation of each token from the long Chinese text to be processed according to the preset preprocessing flow includes: Obtain the token sequence of the long Chinese text to be processed after word segmentation or sub-word segmentation; Using a sliding window with a preset format, the token sequence is divided into overlapping blocks to generate multiple overlapping text blocks; The multiple overlapping text blocks are input into a pre-trained Chinese BART encoder, and multi-layer Transformer encoding is performed block by block to obtain the context-related hidden vector representation of each token in the high-dimensional semantic space.

3. The long text summarization method according to claim 2, characterized in that, The text block includes a start position and an end position, and the start position and end position satisfy the following relationship: ； in, Starting position This is the end position.

4. The long text summarization method according to claim 1, characterized in that, The local attention score is calculated from the local key-value pairs of the current layer, and the global attention score is calculated from the query vector of the current layer and the global key-value pairs retrieved from the sparsely stored set of key tokens.

5. The long text summarization method according to claim 1, characterized in that, The step of determining the local entropy and global entropy based on the local attention score and the global attention score includes: Soft max normalization is performed on the local attention score and the global attention score respectively to obtain the local attention weight and the global attention weight; Calculate local and global entropy based on the information entropy formula; The information entropy formula is specifically as follows: ； in, These are the local attention weights or global attention weights after Softmax normalization. It is a very small positive number.

6. The long text summarization method according to claim 1, characterized in that, The introduction of temperature regulation parameters maps the entropy difference between the local and global entropies using the Sigmoid function to obtain the fusion weight, specifically including: ； in, This represents the Sigmoid function. For local entropy, For global entropy, This indicates the temperature control parameter.

7. The long text summarization method according to claim 1, characterized in that, Based on the fusion weights, a weighted fusion is performed on the local key-value pairs and the global key-value pairs to obtain the fused key-value pairs. The specific formula is as follows: ； ； in, and For local key-value pairs, and The final calculated key-value pairs are global key-value pairs. and , For weight fusion.

8. A long text summarization generation device, characterized in that, include: The preprocessing module is used to extract the context-related hidden vector representation of each token from the long Chinese text to be processed according to the preset preprocessing process. The sparse storage module is used to obtain the Euclidean norm of the context-related hidden vector representation, and selectively store tokens according to the Euclidean norm, the preset sparsity rate and the total number of tokens, to obtain the sparsely stored key token set. The attention calculation module is used to calculate the local attention score and global attention score in parallel on the current layer query vector during the decoding phase. The fusion weight calculation module is used to determine the local entropy and global entropy based on the local attention score and the global attention score, and to map the entropy difference between the local entropy and the global entropy through the Sigmoid function by introducing a temperature adjustment parameter to obtain the fusion weight; The entropy-guided attention fusion module is used to perform weighted fusion of local key-value pairs and global key-value pairs according to the fusion weights to obtain fused key-value pairs. The content generation module is used to input the fused key-value pairs into the decoder to generate a digest token sequence, so as to output a long Chinese text digest.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the long text summarization generation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the long text summarization method as described in any one of claims 1 to 7.

Images