A long text abstract generation method and device

Abstract

The application provides a long text abstract generation method and device, and belongs to the technical field of natural text processing. The method comprises the following steps: using the Euclidean norm to sort the importance of Token and compressing and storing the Token according to the sparsity, so that the global key information can be completely retained and the memory occupation is significantly reduced; the local and global attention scores are calculated in parallel in the decoding stage, the entropy difference is mapped into the fusion weight through Sigmoid combined with the temperature regulation parameter, the dynamic balance of the local details and the long-distance dependence is realized, the local key-value pairs and the global key-value pairs are weighted and integrated based on the fusion weight, the continuous semantic spectrum is formed in a single decoding layer, the splicing fracture caused by the traditional block is eliminated, the problems of input limitation and semantic fragmentation are effectively relieved, the context length that can be processed by the model is expanded without significantly increasing the calculation amount, and the accuracy and coherence of the abstract are effectively improved through adaptive fusion of the local and global context information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a long text abstract generation method and device. BACKGROUND

[0002] Text abstract generation is an important task in the field of natural language processing, which aims to compress lengthy original text into a concise and key information-containing abstract. This technology has important application value in news aggregation, public opinion monitoring, legal document processing, academic information retrieval, etc. In recent years, with the rise of pre-training language models, generative abstract methods based on the Transformer architecture have made significant progress in English long text tasks, such as BERT, BART, Longformer, BigBird, T5, etc.

[0003] However, such models still have some problems when processing Chinese long text abstracts, the main reasons including: language characteristic differences, Chinese has significant differences in grammar structure, word segmentation mechanism, and chapter organization method from English, and direct migration of English long text modeling technology often has limited effect; high training cost, training a pre-training model suitable for Chinese long text from scratch requires extremely high computing and storage resources, which is difficult for most research and development teams to bear; insufficient data resources, the length of existing Chinese abstract data sets is generally shorter, the average input length is much lower than the requirement of long text tasks, and the annotation quality is uneven, which is difficult to support the training and evaluation of high-quality long text generation models.

[0004] Researchers have carried out a lot of exploration in long text modeling, but the existing methods generally have the problems of limited context length, difficulty in maintaining coherence of cross-paragraph semantics, and high reasoning cost. Especially in Chinese tasks, limited by corpus structure and model adaptation, its performance in long-distance dependency capture and abstract condensation has not yet reached the ideal level. The block processing strategy divides long text into fragments for independent coding, and then concatenates the input decoder. This method expands the input capacity, but causes semantic fragmentation between fragments, and key information is lost in the concatenation process (such as cross-fragment reference resolution failure). There is also a sparse attention mechanism, which reduces computational complexity through local window, random attention, etc. However, such methods require customized pre-training and are difficult to adapt to Chinese language characteristics, and tend to ignore global key information when generating abstracts. SUMMARY

[0005] The present application provides a long text abstract generation method and device to solve the defects in the prior art and improve the context modeling capability of long text.

[0006] In a first aspect, the present application provides a long text abstract generation method, comprising:

[0007] According to a preset preprocessing procedure, a context-related hidden vector representation of each Token is extracted from the to-be-processed Chinese long text;

[0008] A Euclidean norm of the context-related hidden vector representation is obtained, and according to the Euclidean norm, a preset sparsity rate and a total number of Tokens, a Token is selectively stored to obtain a set of key Tokens after sparse storage;

[0009] In the decoding stage, the local attention score and the global attention score are calculated in parallel for the current layer query vector;

[0010] According to the local attention score and the global attention score, the local entropy and the global entropy are determined, and the entropy difference between the local entropy and the global entropy is mapped by a Sigmoid function to obtain a fusion weight by introducing a temperature adjustment parameter;

[0011] According to the fusion weight, the local key-value pair and the global key-value pair are weighted and fused to obtain a fused key-value pair, and the fused key-value pair is input into a decoder to generate a summary Token sequence, so as to output a Chinese long text summary.

[0012] Further, the context-related hidden vector representation of each Token is extracted from the to-be-processed Chinese long text according to a preset preprocessing procedure, which comprises:

[0013] The Token sequence of the to-be-processed Chinese long text after word segmentation or subword segmentation is obtained;

[0014] A sliding window in a preset format is used to overlap and block the Token sequence to generate a plurality of overlapping text blocks;

[0015] The plurality of overlapping text blocks are respectively input into a pre-trained Chinese BART encoder, and multi-layer Transformer encoding is performed on each block to obtain a context-related hidden vector representation of each Token in a high-dimensional semantic space.

[0016] Further, the text block contains a start position and an end position, and the start position and the end position satisfy the following relationship:

[0017] ;

[0018] wherein, is the start position, is the end position.

[0019] Further, the local attention score is calculated from the local key-value pair of the current layer, and the global attention score is calculated from the query vector of the current layer and the global key-value pair retrieved from the set of key Tokens after sparse storage.

[0020] Further, the local entropy and the global entropy are determined according to the local attention score and the global attention score, including:

[0021] The Soft max normalization is performed on the local attention score and the global attention score respectively to obtain a local attention weight and a global attention weight;

[0022] The local entropy and the global entropy are calculated based on an information entropy formula;

[0023] The information entropy formula is specifically:

[0024] ;

[0025] Wherein, is the local attention weight or the global attention weight after the Soft max normalization, is a very small positive number.

[0026] Further, the entropy difference between the local entropy and the global entropy is mapped by a Sigmoid function to obtain a fusion weight by introducing a temperature adjustment parameter, including:

[0027] ;

[0028] Wherein, represents the Sigmoid function, is the local entropy, is the global entropy, represents the temperature adjustment parameter.

[0029] Further, the local key-value pair and the global key-value pair are weighted fused according to the fusion weight to obtain a fused key-value pair, and the specific formula is as follows:

[0030] ;

[0031] ;

[0032] Wherein, and are the local key-value pair, and are the global key-value pair, and finally the fused key-value pair and , is the fusion weight.

[0033] In a second aspect, the application further provides a long text abstract generation device, including:

[0034] The preprocessing module is used to extract the context-related hidden vector representation of each token from the long Chinese text to be processed according to the preset preprocessing process.

[0035] The sparse storage module is used to obtain the Euclidean norm of the context-related hidden vector representation, and selectively store tokens according to the Euclidean norm, the preset sparsity rate and the total number of tokens, to obtain the sparsely stored key token set.

[0036] The attention calculation module is used to calculate the local attention score and global attention score in parallel on the current layer query vector during the decoding phase.

[0037] The fusion weight calculation module is used to determine the local entropy and global entropy based on the local attention score and the global attention score, and to map the entropy difference between the local entropy and the global entropy through the Sigmoid function by introducing a temperature adjustment parameter to obtain the fusion weight;

[0038] The entropy-guided attention fusion module is used to perform weighted fusion of local key-value pairs and global key-value pairs according to the fusion weights to obtain fused key-value pairs.

[0039] The content generation module is used to input the fused key-value pairs into the decoder to generate a digest token sequence, so as to output a long Chinese text digest.

[0040] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the long text summarization methods described above.

[0041] This invention, without increasing model parameters, utilizes the Euclidean norm to prioritize tokens and compresses their storage according to sparsity, ensuring complete preservation of global key information while significantly reducing memory usage. During the decoding phase, local and global attention scores are calculated in parallel, and the entropy difference is mapped to a fusion weight via a Sigmoid function using a temperature adjustment parameter, achieving a dynamic balance between local details and long-distance dependencies. Based on this fusion weight, local and global key-value pairs are weighted and integrated, forming a continuous semantic spectrum within a single decoding layer, eliminating splicing breaks caused by traditional block segmentation. The final generated summary token sequence maintains factual consistency and coherent expression even when the input far exceeds a fixed window length, significantly reducing redundant computation and improving summary accuracy. Attached Figure Description

[0042] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0043] Figure 1 is a flowchart of an optional long text summary generation method provided by an embodiment of the present application;

[0044] Figure 2 is a flowchart of an optional text chunking processing provided by an embodiment of the present application,

[0045] Figure 3 is a flowchart of an optional sparse storage of Token set provided by an embodiment of the present application;

[0046] Figure 4 is a flowchart of an optional entropy-guided attention fusion provided by an embodiment of the present application;

[0047] Figure 5 is a structural diagram of an optional electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the following will combine the drawings in the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of protection of the present application.

[0049] It should be noted that in the description of the embodiments of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the sentence "comprises a" does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0050] The following will describe the long text summary generation method and device provided by the embodiments of the present application in combination with Figures 1-3

[0051] Figure 1 ​is a flowchart of a long text abstract generation method provided by the present application, as shown in Figure 1 including but not limited to the following steps:

[0052] Step 102: According to the preset preprocessing procedure, the context-related hidden vector representation of each Token is extracted from the to-be-processed Chinese long text;

[0053] Step 104: Obtain the Euclidean norm of the context-related hidden vector representation, and selectively store the Token according to the Euclidean norm, the preset sparsity and the total number of Tokens to obtain a sparse storage key Token set;

[0054] Step 106: In the decoding stage, the local attention score and the global attention score are calculated in parallel for the current layer query vector;

[0055] Step 108: According to the local attention score and the global attention score, the local entropy and the global entropy are determined, and the entropy difference between the local entropy and the global entropy is mapped by a Sigmoid function to obtain a fusion weight by introducing a temperature adjustment parameter;

[0056] Step 110: According to the fusion weight, the local key-value pair and the global key-value pair are weighted and fused to obtain a fused key-value pair; the fused key-value pair is input into the decoder to generate an abstract Token sequence, and the Chinese long text abstract is output.

[0057] Regarding step 102: According to the preset preprocessing procedure, the context-related hidden vector representation of each Token is extracted from the to-be-processed Chinese long text.

[0058] The long text sequence is fed into the encoder of the preset text blocking processing program in turn according to the overlapping sliding window, wherein the window size can be fixed with the stride to ensure that the adjacent segments have semantic overlap. The hidden state tensor output by the last layer of the encoder is the "context-related hidden vector representation", and the dimension d can be consistent with the model base to enable all subsequent importance calculations to directly reuse the tensor without additional forward calculation.

[0059] Optionally, in the long text abstract generation method, the context-related hidden vector representation of each Token is extracted from the to-be-processed Chinese long text according to the preset preprocessing procedure, which includes: obtaining the Token sequence after the to-be-processed Chinese long text is segmented by word or subword;

[0060] A sliding window with a preset format is used to overlap and block the Token sequence to generate a plurality of text blocks that overlap with each other;

[0061] The multiple overlapping text blocks are input into a pre-trained Chinese BART encoder, and multi-layer Transformer encoding is performed block by block to obtain the context-related hidden vector representation of each token in the high-dimensional semantic space.

[0062] In this embodiment, Figure 2 This is a flowchart illustrating the text segmentation process provided by the present invention, as shown below. Figure 2 As shown, the preprocessing flow for long Chinese texts using text segmentation is as follows: First, the long Chinese text is input for preprocessing; then, the block processing process is executed, which divides the input long Chinese text into overlapping blocks using a sliding window with a fixed size and stride. Next, the block encoding process is executed, which inputs the constructed text blocks into the BART encoder to generate a context-dependent hidden vector representation of each token in the high-dimensional semantic space.

[0063] The text block includes a start position and an end position, and the start position and end position satisfy the following relationship:

[0064] ;

[0065] in, Starting position This is the end position.

[0066] Regarding step 108: Obtain the Euclidean norm of the context-related hidden vector representation, and selectively store tokens based on the Euclidean norm, the preset sparsity rate, and the total number of tokens to obtain a sparsely stored set of key tokens.

[0067] During sparse storage, the Euclidean norm of each context vector is calculated in real time as a quantification of the vector's saliency. A configurable circular buffer is maintained simultaneously to store the vectors with the highest norm online. The control unit dynamically calculates the retention limit based on a preset sparsity rate R: when the buffer reaches its limit, a new vector is replaced only if its norm is greater than the minimum value of existing vectors and passes consistency verification. This ensures that globally critical information remains in GPU memory, while GPU memory usage is linearly compressed with R. Discarded vectors are immediately released from GPU memory and are no longer accessed in subsequent attention calculations, forming a sparse representation of the "key token set".

[0068] For example, Figure 3 This is a schematic diagram of the process for sparse storage of a token set provided by the present invention, as shown below. Figure 3As shown, the L2 norm of the vector corresponding to each Token is calculated for the context-dependent hidden vector representation, and the number of Tokens to be retained is determined according to the preset sparsity R and the total number of Tokens L. The hidden vectors of the first n Tokens with the largest L2 norm are stored as key vectors in the global sparse memory, and the remaining Token vectors are discarded, thereby reducing the memory overhead.

[0069] Regarding step 106: In the decoding stage, the local attention score and the global attention score are calculated in parallel for the current layer query vector.

[0070] In the decoding stage, the local attention score and the global attention score are calculated in parallel for the current layer query vector, wherein the local attention score is calculated from the current layer local key-value pair, and the global attention score is calculated from the current layer query vector and the global key-value pair retrieved from the key Token set after sparse storage.

[0071] Figure 4 is a flowchart of an entropy-guided attention fusion according to the present application, as shown in Figure 4 The decoder expands two attention paths in parallel for each layer, one is the local path, which calculates the dot product of the current layer query vector and the randomly sampled key-value pair , in the same window to obtain the local attention score; the other is the global path, which calculates the dot product of the previous layer query vector and the key , in the sparse storage area to obtain the global attention score.

[0072] Regarding step 108: According to the local attention score and the global attention score, the local entropy and the global entropy are determined, and a temperature adjustment parameter is introduced to map the entropy difference between the local entropy and the global entropy through a Sigmoid function to obtain a fusion weight.

[0073] The information entropy of the normalized attention distribution is calculated to obtain the local attention entropy and the global attention entropy ; the temperature adjustment parameter T is injected in a learnable or artificially set manner to scale the entropy difference . After Sigmoid mapping, the fusion weight a between 0 and 1 is output, and its real-time value reflects the dynamic dependence strength of the current decoding position on local details and global context.

[0074] Optionally, Soft max normalization is performed on the local attention score and the global attention score to obtain the local attention weight and the global attention weight;

[0075] Calculate local and global entropy based on the information entropy formula;

[0076] The information entropy formula is specifically as follows:

[0077] ;

[0078] in, These are the local attention weights or global attention weights after Softmax normalization. It is an extremely small positive number to prevent undefined logarithms or overflow problems in numerical computation, and to ensure the numerical stability of entropy calculation.

[0079] Optionally, the introduction of temperature regulation parameters maps the entropy difference between the local and global entropies using a Sigmoid function to obtain the fusion weights, specifically including:

[0080] ;

[0081] in, This represents the Sigmoid function. For local entropy, For global entropy, This indicates the temperature control parameter.

[0082] Regarding step 110: Based on the fusion weight, perform weighted fusion on the local key-value pairs and the global key-value pairs to obtain fused key-value pairs, and input the fused key-value pairs into the decoder to generate a digest token sequence to output a Chinese long text digest.

[0083] Perform linear interpolation on the K and V vectors according to the fusion weights: , Similarly, the fused key-value pairs are directly fed into the remaining branch of the multi-head attention network, and after passing through the feedforward network, the output hidden state of this layer is generated. The decoder executes the above process cyclically, gradually generating a sequence of digest tokens until the end symbol is triggered or the maximum length limit is reached, and finally concatenating them into a Chinese long text digest.

[0084] Based on the above embodiments, as an optional embodiment, the following specific example will further illustrate a long text summarization method of the present invention.

[0085] Step 1: Input the long Chinese text that needs to be processed;

[0086] Step 2, block processing, is as follows: The input length from Step 1 is... input sequence Using a sliding window with a fixed size of 1024 tokens and a fixed stride of 512 tokens, the input long Chinese text is segmented into overlapping text blocks:

[0087]

[0088] where, is the number of blocks, and are the start and end positions of the th block, respectively, and due to the overlapping of text blocks, the overlap ratio is Here, the overlap ratio is .

[0089] Step 3, Block Encoding Process, as follows: the input Chinese long text sequence is segmented by word or sub-word, obtaining a discrete Token sequence:

[0090]

[0091] where, denotes the length of the input long Chinese text sequence. Each Token is mapped by embedding to obtain a word vector representation:

[0092]

[0093] where denotes the word embedding mapping function, denotes the position encoding, which preserves the position information of the sequence. The embedded sequence is input into the multi-layer Transformer encoding module of the BART encoder, each layer consisting of multi-head self-attention mechanism and feedforward network. The encoding process is:

[0094]

[0095]

[0096] where denotes the number of encoding layers, is the initial embedding vector, is the output of the th layer. After passing through all the encoding layers, the context-dependent hidden vector representation of each Token is obtained: .

[0097] As shown in Figure 3 , sparse storage includes the following steps:

[0098] Step 4, Calculate the L2 norm of each Token, as follows: the high-dimensional semantic vector representation of a single Token is , where denotes the vector dimension; the L2 norm is calculated by the formula .

[0099] Step 5, determine the Token to be saved into the global storage, as follows: first determine a sparsity coefficient , according to the total number of Token sequence currently input , the number of Token to be retained can be calculated by the formula , arrange the input Token sequence in descending order according to the L2 norm calculated in step 5, and add the first Token with the largest L2 norm in the sequence to the global sparse storage;

[0100] As shown in Figure 4 , the entropy-guided attention fusion includes the following steps:

[0101] Step 6, calculate the attention scores of local attention and global attention, obtain the local attention score by calculating the similarity between the current input query vector and the local key-value pair of the current layer. For the layer of the decoder, the query vector comes from the decoder hidden state, and the key-value pair comes from the randomly sampled encoder state of the current layer:

[0102]

[0103]

[0104]

[0105] Then the local attention score is calculated by the formula

[0106]

[0107] At the same time, according to the global attention key-value pair retrieved from the memory , the global attention score can be calculated according to the formula

[0108] ;

[0109] . The local attention weight and the global attention weight are calculated according to the formula

[0110]

[0111] respectively. Then, the local attention entropy and the global attention entropy are calculated according to the local attention weight and the global attention weight, and the local attention entropy is calculated by the formula

[0112] ​

[0113] to calculate local entropy and global entropy respectively. Finally, the fusion weight is calculated, introducing a temperature parameter to adjust the influence of the entropy difference, and the result is calculated by the Sigmoid function to obtain the final fusion weight in the range of , the specific formula used is as follows

[0114]

[0115] where represents the Sigmoid function. According to the calculated fusion weight , the local key-value pair and the global key-value pair are fused in a weighted average manner, where the weight of the local key-value pair is , and the weight of the global key-value pair is , and the formula is as follows:

[0116]

[0117]

[0118] where and are local key-value pairs, and are global key-value pairs, and the final fused key-value pairs and are calculated.

[0119] On the other hand, the application also provides a long text summary generation device, which comprises: a preprocessing module for extracting the context-related hidden vector representation of each Token from the Chinese long text to be processed according to a preset preprocessing process;

[0120] a sparse storage module for obtaining the Euclidean norm of the context-related hidden vector representation, and selectively storing Tokens according to the Euclidean norm, a preset sparsity and the total number of Tokens to obtain a set of key Tokens after sparse storage;

[0121] an attention calculation module for calculating local attention scores and global attention scores in parallel for the current layer query vector in the decoding stage;

[0122] a fusion weight calculation module for determining local entropy and global entropy according to the local attention scores and global attention scores, and introducing a temperature adjustment parameter to map the entropy difference between the local entropy and the global entropy by a Sigmoid function to obtain a fusion weight;

[0123] An entropy-guided attention fusion module is configured to perform weighted fusion on the local key-value pair and the global key-value pair according to the fusion weight to obtain a fused key-value pair.

[0124] A content generation module is configured to input the fused key-value pair into a decoder to generate an abstract Token sequence, and output a Chinese long text abstract.

[0125] It should be noted that the long text abstract generation device provided in the embodiments of the present application can execute the long text abstract generation method described in any of the above embodiments in the specific operation, and the embodiments of the present application will not be repeated here.

[0126] Figure 5 is a structural schematic diagram of an electronic device provided by the present application, as Figure 5 shown, the electronic device can include a processor (processor) 510, a communication interface (communications interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, the memory 530 complete the communication among each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the long text abstract generation method, which includes: according to a preset preprocessing process, extracting the context-related hidden vector representation of each Token from the to-be-processed Chinese long text; obtaining the Euclidean norm of the context-related hidden vector representation, and selectively storing the Token according to the Euclidean norm, the preset sparsity and the total number of Tokens to obtain a set of sparsely stored key Tokens; in the decoding stage, the local attention score and the global attention score are calculated in parallel for the current layer query vector; according to the local attention score and the global attention score, the local entropy and the global entropy are determined, and a temperature adjustment parameter is introduced to map the entropy difference between the local entropy and the global entropy through a Sigmoid function to obtain a fusion weight; according to the fusion weight, weighted fusion is performed on the local key-value pair and the global key-value pair to obtain a fused key-value pair, and the fused key-value pair is input into a decoder to generate an abstract Token sequence, and output a Chinese long text abstract.

[0127] In addition, the logic instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0128] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the long text summary generation method provided by the above-mentioned embodiments, and the method comprises the following steps: according to a preset preprocessing process, extracting a context-related hidden vector representation of each Token from a to-be-processed Chinese long text; obtaining the Euclidean norm of the context-related hidden vector representation, and selectively storing the Token according to the Euclidean norm, a preset sparsity rate and a total number of Tokens, to obtain a sparse stored key Token set; in the decoding stage, calculating the local attention score and the global attention score in parallel for the current layer query vector; according to the local attention score and the global attention score, determining the local entropy and the global entropy, and introducing a temperature adjustment parameter to map the entropy difference of the local entropy and the global entropy through a Sigmoid function, to obtain a fusion weight; according to the fusion weight, performing weighted fusion on the local key-value pair and the global key-value pair, to obtain a fused key-value pair, and inputting the fused key-value pair into a decoder to generate a summary Token sequence, to output a Chinese long text summary.

[0129] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the long text summary generation method provided by the above embodiments, and the method comprises: extracting a context-related hidden vector representation of each Token from a to-be-processed Chinese long text according to a preset preprocessing procedure; obtaining a Euclidean norm of the context-related hidden vector representation, and selectively storing Tokens according to the Euclidean norm, a preset sparsity rate and a total number of Tokens to obtain a set of sparsely stored key Tokens; in a decoding stage, calculating a local attention score and a global attention score in parallel for a current layer query vector; determining a local entropy and a global entropy according to the local attention score and the global attention score, and introducing a temperature adjustment parameter to map an entropy difference between the local entropy and the global entropy through a Sigmoid function to obtain a fusion weight; performing weighted fusion on a local key-value pair and a global key-value pair according to the fusion weight to obtain a fused key-value pair, and inputting the fused key-value pair into a decoder to generate a summary Token sequence to output a Chinese long text summary.

[0130] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0131] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary universal hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0132] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A long text abstract generation method, characterized by, The method comprises the following steps: According to the preset preprocessing procedure, the context-related hidden vector representation of each Token in the to-be-processed Chinese long text is extracted; The Euclidean norm of the context-related hidden vector representation is obtained, and the Tokens are selectively stored according to the Euclidean norm, a preset sparsity rate and the total number of Tokens to obtain a set of key Tokens after sparse storage; In the decoding stage, the local attention score and the global attention score are calculated in parallel for the current layer query vector; According to the local attention score and the global attention score, the local entropy and the global entropy are determined, and a temperature adjustment parameter is introduced to map the entropy difference between the local entropy and the global entropy through a Sigmoid function to obtain a fusion weight; the determination of the local entropy and the global entropy according to the local attention score and the global attention score comprises the following steps: The Soft max normalization is performed on the local attention score and the global attention score respectively to obtain the local attention weight and the global attention weight; The local entropy and the global entropy are calculated based on an information entropy formula; The information entropy formula is as follows: ； wherein, is the local attention weight or the global attention weight normalized by Soft max, is a very small positive number; According to the fusion weight, the local key-value pair and the global key-value pair are weighted and fused to obtain a fused key-value pair, and the fused key-value pair is input into a decoder to generate a summary Token sequence, so as to output a Chinese long text summary.

2. The long text summarization method of claim 1, wherein, The method comprises the following steps: The Token sequence of the to-be-processed Chinese long text after word segmentation or subword segmentation is obtained; The Token sequence is divided into multiple overlapping text blocks by using a sliding window with a preset format; The multiple overlapping text blocks are input into a pre-trained Chinese BART encoder, and multi-layer Transformer encoding is performed on the text blocks to obtain the context-related hidden vector representation of each Token in a high-dimensional semantic space.

3. The long text summarization method of claim 2, wherein, The text block comprises a start position and an end position, and the start position and the end position satisfy the following relationship: ； wherein is the start position, is the end position.

4. The long text summarization method of claim 1, wherein, The local attention score is calculated from the local key-value pair of the current layer, and the global attention score is calculated from the global key-value pair retrieved from the set of key Tokens after sparse storage.

5. The long text summarization method according to claim 1, wherein, The local entropy and the global entropy are mapped through a Sigmoid function by introducing a temperature adjustment parameter to obtain a fusion weight, and the specific steps comprise the following steps: ； wherein, denotes a Sigmoid function, is a local entropy, is a global entropy, denotes a temperature regulation parameter.

6. The long text summarization method of claim 1, wherein, According to the fusion weight, the local key-value pair and the global key-value pair are weighted and fused to obtain a fused key-value pair, and the specific formula is as follows: ； ； wherein, and are local key-value pairs, and are global key-value pairs, resulting in fused key-value pairs and , are fusion weights.

7. A long text abstract generation apparatus characterized by comprising: The method comprises the following steps: A preprocessing module is configured to extract the context-related hidden vector representation of each Token in the to-be-processed Chinese long text according to a preset preprocessing procedure; A sparse storage module is configured to obtain the Euclidean norm of the context-related hidden vector representation, and selectively store the Tokens according to the Euclidean norm, a preset sparsity rate and the total number of Tokens to obtain a set of key Tokens after sparse storage; An attention calculation module is configured to calculate, in a decoding stage, local attention scores and global attention scores in parallel for a current layer query vector; A fusion weight calculation module is configured to determine local entropy and global entropy according to the local attention scores and the global attention scores, and introduce a temperature adjustment parameter to map an entropy difference between the local entropy and the global entropy through a Sigmoid function to obtain a fusion weight. The determination of the local entropy and the global entropy according to the local attention scores and the global attention scores includes: performing Soft max normalization on the local attention scores and the global attention scores respectively to obtain local attention weights and global attention weights; calculating the local entropy and the global entropy based on an information entropy formula; ； wherein, is the local attention weight or the global attention weight normalized by Soft max, is a very small positive number; The information entropy formula is specifically as follows: An entropy-guided attention fusion module is configured to perform weighted fusion on local key-value pairs and global key-value pairs according to the fusion weight to obtain fused key-value pairs.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, A content generation module is configured to input the fused key-value pairs into a decoder to generate a Token sequence of a summary and output a Chinese long text summary. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The processor executes the computer program to implement the steps of the long text summary generation method according to any one of claims 1 to 6. The computer program is executed by the processor to implement the steps of the long text summary generation method according to any one of claims 1 to 6.

Images