Abstraction abstract generation method based on dual-granularity guidance and entity replication

Abstract

The invention discloses an abstract abstract generation method based on dual-granularity guidance and entity replication, which uses an entity replication mechanism to allow a model to copy an entity of a source document when an abstract abstract is generated, so that key information can be effectively prevented from going wrong, and the correctness of the abstract entity is ensured; in order to prevent semantic deviation of the abstract, key entities and keywords are used as guidance information to standardize generation of the abstract abstract, so that the fact consistency of generation of the abstract abstract is improved, and the loyalty of the abstract abstract is ensured.

Description

Technical Field

[0001] This invention belongs to the field of automatic text summarization technology, specifically relating to an abstract summarization generation method based on dual-granularity guidance and entity replication. Background Technology

[0002] Text summarization aims to condense a document or set of documents into a concise summary while preserving the semantics of the original source text. Text summarization techniques are generally divided into extractive and abstractive methods. Extractive methods select important words or sentences from the input text and then combine them into a summary; abstractive methods use models to generate summaries that may contain new words or sentences. Compared to extractive methods, abstractive methods are more flexible and more likely to generate highly accurate summaries.

[0003] However, the flexibility of abstract summarization also brings more problems. Firstly, due to the high variability of source documents, it is easy to suffer from an incomplete vocabulary (OOV problem), especially for important entity information. Entities that should appear in the summary may be missing or contain other incorrect entities, significantly reducing the quality of the summary. Secondly, most existing abstract summarization models are designed based on an encoder-decoder architecture. Since the contextual semantics of the source document are learned by the encoder, the decoding generation relies entirely on the contextual semantics, which leads to reduced factual consistency in the generated summary.

[0004] In view of this, how to further improve the quality of abstract summary generation has become a pressing technical issue in this industry. Summary of the Invention

[0005] To achieve the above objectives, this invention aims to provide an abstract summary generation method based on dual-granularity guidance and entity replication. It uses a specially developed summary generation model, which is a framework trained using dual-granularity guidance and entity replication. By utilizing guidance information at two different granularity levels (keyword level and key entity level) during the generation process, and combining it with entity replication, a reliable abstract summary is generated.

[0006] This is mainly reflected in the fact that, in order to solve the problem of entity mismatch, the above-mentioned method of the present invention uses an entity copying mechanism, which allows the model to copy the entities of the source document when generating the abstract summary. This can effectively prevent errors in key information and ensure the correctness of the summary entities. In order to prevent semantic deviation of the summary, the above-mentioned method of the present invention uses key entities and keywords as guiding information to standardize the generation of the abstract summary, improve the factual consistency of the generated abstract summary, and ensure the fidelity of the abstract summary.

[0007] Specifically, we first extract entities (entity phrases) from the source document, which enables the model to replicate entities during the decoding phase, ensuring that the generated entities have the same semantics as the entities in the source document. Further, we treat the extracted entities as candidate entities and calculate their relevance in the source document to obtain key entities, which serve as guidance information at multiple granular levels. At the same time, we also identify key tokens in the source document, which serve as guidance information at a single granular level.

[0008] As is generally known, the aforementioned token refers to a basic or smallest unit of text in Natural Language Processing (NLP). It can typically be a word, a phrase, a punctuation mark, a character, etc., depending on the needs and methods of text processing. For example, in English, a token can be a word or a punctuation mark; in Chinese, characters or words are usually used as tokens. This enables the model to process and understand the text. For the model, the input text is broken down into individual tokens. A token is a digital representation, and each token is associated with a unique numeric ID. The model uses these IDs to distinguish different tokens. During training, the model learns how to map text to these numeric IDs in order to encode and decode new text.

[0009] Specifically, given a source text X = {x1, x2, ..., x...} n}, where x i Let represent the i-th token in the source text X, and n represent the total length of the source text X. The goal is to compress the source text X into a concise abstract summary text Y = {y1, y2, ..., yn}. m}, where y j Let represent the j-th token in the abstract summary text Y, and m represent the total length of the abstract summary text Y, while preserving the main idea of ​​the source text X. Therefore, the above method of the present invention specifically includes the following steps:

[0010] Step S1, Encoder-Decoder, specifically includes the following steps:

[0011] S11. Given that abstract summary texts widely use a sequence-to-sequence model architecture, an encoder is first used to encode the source text X to obtain the contextual hidden state representation H of the contextual semantic features of the source text X. e The formula is as follows:

[0012] H e =Encoder(X) (1)

[0013] In the formula, in h represents the hidden state representation of the i-th token in the encoder, where h is the size of the hidden state representation.

[0014] S12, Context-based Hidden State Representation H e And the generated text Y containing t-1 tokens ＜t ={y1, y2, ..., y t-1 The encoder-decoder attention mechanism uses the decoder to generate the hidden state representation of the t-th token in the t-th decoding stage. Finally, the output Y is predicted from vocabulary V. t The probability P of (i.e., outputting the token) is given by the following formula:

[0015]

[0016] In the formula, E∈R V×h Let V be the embedding matrix of the vocabulary V, and softmax be the activation function.

[0017] Step S2: Based on dual-granularity guidance using key tokens and key entities, given that semantics relying entirely on semantic features acquired by the encoder during the abstract summary generation stage can easily lead to semantic deviation and affect the quality of the generated abstract summary, this invention uses key tokens (keywords) and key entities (key entities) as guiding information to effectively enrich and control semantic information. This better ensures the consistency of the main idea between the generated abstract summary text Y and the source text X. It should be noted that the method for calculating key token and key entity information in this invention differs from directly selecting important key tokens and key entities from the source document using certain algorithms. Instead, this invention assigns a relevance probability distribution on the vocabulary V based on each token or entity information, specifically including the following steps:

[0018] S21. Obtain key token-level guidance information. Key tokens have a high correlation with source text X. Therefore, assign a probability distribution of relevance to vocabulary V to each token in source text X. First, use the encoder's contextual hidden state representation to learn the token-vocabulary similarity. The specific formula is as follows:

[0019] S t =gelu(H e W1)W2E (3)

[0020] In the formula, A token (i.e., x) in the source text X iThe context similarity between tokens in the vocabulary V and the word V, where n is the total number of tokens; W1 and W2 are the weight matrices to be learned, and H is the weight matrix to be learned. e For context-hidden state representation, E∈R V×h Let V be the embedding matrix of the vocabulary V, and GELU be the activation function.

[0021] S22. Combine Token-Vocabulary similarity with decoder cross-attention to generate guidance information. Specifically, in the t-th decoding stage, use the cross-attention weights of the last layer L of the decoder. The key token-vocabulary similarity is weighted and summed to obtain the decoder's key token-level guidance information. The specific formula is as follows:

[0022]

[0023] In the formula, For cross-attention weights, Y represents t The cross-attention distribution between the output token and the source token in the last layer L of the decoder, and This is the cross-attention weight of the i-th token;

[0024] S23. Solve for the matrix representation of key token-level guidance information. The specific formula is as follows:

[0025] F token =A d_token S t (5)

[0026] In the formula, F token ∈R n×|V| , f i token This represents the key token-level guidance information for step i; A d_token for Matrix form;

[0027] S24. Obtain key entities, each entity consists of multiple tokens {x} i , ..., x j If the entity is composed of}, then the hidden state of this entity is represented as Here, avg represents the average value function; the subscripts i and j represent any few of them, that is, the hidden state representation of this entity is obtained by averaging the hidden state representations of these tokens;

[0028] S25. Obtain key entity-level guidance information. First, use the Long Short-Term Memory (LSTM) model to obtain the contextual hidden state representation of the entity. The specific formula is as follows:

[0029] H e_entity =LSTM(R) (6)

[0030] In the formula, R = {e1, e2, ..., e} k}, where k represents the number of entities;

[0031] S26. Context-based Hidden State Representation H e_entity Let's learn about Entity-EntitiesSet similarity S e The formula is as follows:

[0032] S e =(gelu(H e_entity W3)W4E (7)

[0033] In the formula, Entity-EntitiesSet similarity for an entity, where k represents the number of entities; W3 and W4 are the weight matrices to be learned, and H... e_entity The context-hidden state representation of an entity, E∈R V×h Let V be the embedding matrix of the vocabulary V, and GELU be the activation function.

[0034] S27, Calculate the similarity between Entity-EntitiesSets. e Combined with the decoder's cross-attention, key entity-level guidance information is finally generated. Specifically, in the t-th decoding stage, the cross-attention weights of the last layer L of the decoder are used. Similarity S between Entity-EntitiesSet e By performing a weighted summation, the key entity-level guidance information for the decoder is finally obtained, as shown in the following formula:

[0035]

[0036] In the formula, Here, represents the cross-attention weights, indicating the distribution of cross-attention between the entity context and the last layer L of the decoder. This is the cross-attention weight of the i-th entity;

[0037] S28. Solve for the matrix representation of key entity-level guidance information. The specific formula is as follows:

[0038] F entity =A d_entity S e (9)

[0039] In the formula, F entity ∈R n×|V| , f i entity Represents the key entity-level guidance information for step i; A d_entity for Matrix form;

[0040] S29. Merge key token-level guidance information and key entity-level guidance information to obtain the corrected Y for predicting output from vocabulary V. t The probability of is given by the following formula:

[0041]

[0042] In the formula, Y t Y is the text containing the t-th token. ＜t The generated text contains t-1 tokens (excluding the t-th token), λ is a custom hyperparameter, E∈R V×h Let V be the embedding matrix of the vocabulary. f is the hidden state representation of the key token generated by the decoder in step t. t token f represents the key token-level guidance information for step t. t entity This represents the key entity-level guidance information at step t, where softmax is the activation function.

[0043] Step S3, entity copying, specifically includes the following steps:

[0044] S31. Calculate the logarithmic distribution vector of entities generated using the entity decoder, as shown in the following formula:

[0045]

[0046] In the formula, Let Q be the hidden state representation of the key token generated by the decoder at step t, and let Q and K be learnable weight matrices. The encoder hidden state representation of the j-th entity;

[0047] S32. To determine whether to copy or generate an entity in the t-th decoding stage, it is necessary to calculate the probabilities of copying and generating. Therefore, the hidden state of the decoder is mapped to singular values ​​as the copying probability, and the specific formula is as follows:

[0048]

[0049] In the formula, P t copy Let ∈[0,1] be the probability of replicating the entity at step t, then the probability of generating the entity is P. tgen =1-P t copy ; sigmoid is the activation function, and FFN represents a fully connected feedforward neural network. Let be the hidden state representation of the key token generated by the decoder at step t;

[0050] S33. Calculate the final generation probability: The specific formula is as follows:

[0051]

[0052] In the formula, This represents the logarithmic distribution vector of the entity generated using the decoder; This represents the logarithmic distribution vector of the token generated using the encoder;

[0053] Step S4: Use the cross-entropy loss function as the final loss function for the model. The specific formula is as follows:

[0054] Loss=Σ t CrossEntropyLoss(P t final , t t (14)

[0055] In the formula, CrossEntropyLoss is the cross-entropy loss function; t t The index of the t-th real tag token is predicted from the vocabulary V, i.e., t t ∈[0, |V|+|E|]; |V| represents the size of the vocabulary; |E| represents the number of entities.

[0056] Furthermore, the present invention also provides an electronic device comprising a processor and a memory; the memory for storing a computer program; and the processor for executing the program stored in the memory to implement the steps of the above method.

[0057] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0058] Compared to existing technologies, this invention primarily improves the thematic consistency of generated summaries through an entity replication mechanism and a dual-granularity guidance mechanism (entity-level guidance and word-level guidance information). Specifically, an entity-based replication mechanism is designed first, allowing the model to replicate entities from the source document in addition to the original token replication, thereby improving the quality of the summary in terms of entity meaning. Furthermore, to prevent semantic omissions and deviations in the generated summary, which could lead to inconsistencies between the generated summary and the facts of the source document, entity and word-level guidance information is proposed to guide the model's summary generation process, thereby controlling the semantics of the summary and ensuring it more closely approximates the thematic information of the source document. Experiments show that these two methods effectively improve the performance of the model's summary generation, and the generated summary is more closely aligned with the facts of the source document. Attached Figure Description

[0059] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings. In the drawings, unless otherwise specified, the same reference numerals throughout the various figures denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in the present invention to provide a further understanding of the invention, constitute a part of this application, and should not be considered as limiting the scope of the invention. Wherein:

[0060] Figure 1 This is a flowchart of the abstract summary generation method provided by the present invention;

[0061] Figure 2 This is a structural block diagram of the generation model used in the abstract summary generation method provided by the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and should not be construed as limiting the invention.

[0063] The following is in conjunction with the appendix Figure 1 and 2 This invention provides a method for generating abstract summaries based on dual-granularity guidance and entity replication. Given a source text X = {x1, x2, ..., x...} n}, where x i Let represent the i-th token in the source text X, and n represent the total length of the source text X. The goal is to compress the source text X into a concise abstract summary text Y = {y1, y2, ..., yn}. m}, where y jLet represent the j-th token in the abstract summary text Y, and m represent the total length of the abstract summary text Y, while retaining the main idea of ​​the source text X.

[0064] Therefore, the above-mentioned method of the present invention specifically includes the following steps:

[0065] Step S1, Encoder-Decoder, specifically includes the following steps:

[0066] S11. Given that abstract summary texts widely use a sequence-to-sequence model architecture, an encoder is first used to encode the source text X to obtain the contextual hidden state representation H of the contextual semantic features of the source text X. e The formula is as follows:

[0067] H e =Encoder(X) (1)

[0068] In the formula, in h represents the hidden state representation of the i-th token in the encoder, where h is the size of the hidden state representation.

[0069] S12, Context-based Hidden State Representation H e And the generated text Y containing t-1 tokens ＜t ={y1, y2, ..., y t-1 The encoder-decoder attention mechanism uses the decoder to generate the hidden state representation of the t-th token in the t-th decoding stage. Finally, the output Y is predicted from vocabulary V. t The probability P of (i.e., outputting the token) is given by the following formula:

[0070]

[0071] In the formula, E∈R V×h Let V be the embedding matrix of the vocabulary V, and softmax be the activation function.

[0072] Step S2: Based on dual-granularity guidance using key tokens and key entities, given that semantics relying entirely on semantic features acquired by the encoder during the abstract summary generation stage can easily lead to semantic deviation and affect the quality of the generated abstract summary, this invention uses key tokens (keywords) and key entities (key entities) as guiding information to effectively enrich and control semantic information. This better ensures the consistency of the main idea between the generated abstract summary text Y and the source text X. It should be noted that the method for calculating key token and key entity information in this invention differs from directly selecting important key tokens and key entities from the source document using certain algorithms. Instead, this invention assigns a relevance probability distribution on the vocabulary V based on each token or entity information, specifically including the following steps:

[0073] S21. Obtain key token-level guidance information. Key tokens have a high correlation with source text X. Therefore, assign a probability distribution of relevance to vocabulary V to each token in source text X. First, use the encoder's contextual hidden state representation to learn the token-vocabulary similarity. The specific formula is as follows:

[0074] S t =gelu(H e W1)W2E (3)

[0075] In the formula, A token (i.e., x) in the source text X i The context similarity between tokens in the vocabulary V and the word V, where n is the total number of tokens; W1 and W2 are the weight matrices to be learned, and H is the weight matrix to be learned. e For context-hidden state representation, E∈R V×h Let V be the embedding matrix of the vocabulary V, and GELU be the activation function.

[0076] S22. Combine Token-Vocabulary similarity with decoder cross-attention to generate guidance information. Specifically, in the t-th decoding stage, use the cross-attention weights of the last layer L of the decoder. The key token-vocabulary similarity is weighted and summed to obtain the decoder's key token-level guidance information. The specific formula is as follows:

[0077]

[0078] In the formula, For cross-attention weights, Y represents t The cross-attention distribution between the output token and the source token in the last layer L of the decoder, and This is the cross-attention weight of the i-th token;

[0079] S23. Solve for the matrix representation of key token-level guidance information. The specific formula is as follows:

[0080] F token =A d_token S t (5)

[0081] In the formula, F token ∈R n×|V| , f i token This represents the key token-level guidance information for step i; A d_token for Matrix form;

[0082] S24. Obtain key entities, each entity consists of multiple tokens {x} i , ..., x j If the entity is composed of}, then the hidden state of this entity is represented as Here, avg represents the average value function; the subscripts i and j represent any few of them, that is, the hidden state representation of this entity is obtained by averaging the hidden state representations of these tokens;

[0083] S25. Obtain key entity-level guidance information. First, use the Long Short-Term Memory (LSTM) model to obtain the contextual hidden state representation of the entity. The specific formula is as follows:

[0084] H e_entity =LSTM(R) (6)

[0085] In the formula, R = {e1, e2, ..., e} k}, where k represents the number of entities;

[0086] S26. Context-based Hidden State Representation H e_entity Let's learn about Entity-EntitiesSet similarity S e The formula is as follows:

[0087] S e =(gelu(H e_entity W3)W4E (7)

[0088] In the formula, Entity-EntitiesSet similarity for an entity, where k represents the number of entities; W3 and W4 are the weight matrices to be learned, and H... e_entity The context-hidden state representation of an entity, E∈R V×hLet V be the embedding matrix of the vocabulary V, and GELU be the activation function.

[0089] S27, Calculate the similarity between Entity-EntitiesSets. e Combined with the decoder's cross-attention, key entity-level guidance information is finally generated. Specifically, in the t-th decoding stage, the cross-attention weights of the last layer L of the decoder are used. Similarity S between Entity-EntitiesSet e By performing a weighted summation, the key entity-level guidance information for the decoder is finally obtained, as shown in the following formula:

[0090]

[0091] In the formula, Here, represents the cross-attention weights, indicating the distribution of cross-attention between the entity context and the last layer L of the decoder. This is the cross-attention weight of the i-th entity;

[0092] S28. Solve for the matrix representation of key entity-level guidance information. The specific formula is as follows:

[0093] F entity =A d_entity S e (9)

[0094] In the formula, F entity ∈R n×|V| , f i entity Represents the key entity-level guidance information for step i; A d_entity for Matrix form;

[0095] S29. Merge key token-level guidance information and key entity-level guidance information to obtain the corrected Y for predicting output from vocabulary V. t The probability of is given by the following formula:

[0096]

[0097] In the formula, Y t Y is the text containing the t-th token. ＜t The generated text contains t-1 tokens (excluding the t-th token), λ is a custom hyperparameter, E∈R V×h Let V be the embedding matrix of the vocabulary. f is the hidden state representation of the key token generated by the decoder in step t. t tokenf represents the key token-level guidance information for step t. t entity This represents the key entity-level guidance information at step t, where softmax is the activation function.

[0098] Step S3, entity copying, specifically includes the following steps:

[0099] S31. Calculate the logarithmic distribution vector of entities generated using the entity decoder, as shown in the following formula:

[0100]

[0101] In the formula, Let Q be the hidden state representation of the key token generated by the decoder at step t, and let Q and K be learnable weight matrices. The encoder hidden state representation of the j-th entity;

[0102] S32. To determine whether to copy or generate an entity in the t-th decoding stage, it is necessary to calculate the probabilities of copying and generating. Therefore, the hidden state of the decoder is mapped to singular values ​​as the copying probability, and the specific formula is as follows:

[0103]

[0104] In the formula, P t copy Let ∈[0,1] be the probability of replicating the entity at step t, then the probability of generating the entity is P. t gen =1-P t copy ; sigmoid is the activation function, and FFN represents a fully connected feedforward neural network. Let be the hidden state representation of the key token generated by the decoder at step t;

[0105] S33. Calculate the final generation probability: The specific formula is as follows:

[0106]

[0107] In the formula, This represents the logarithmic distribution vector of the entity generated using the decoder; This represents the logarithmic distribution vector of the token generated using the encoder;

[0108] Step S4: Use the cross-entropy loss function as the final loss function for the model. The specific formula is as follows:

[0109] Loss=Σ t CrossEntropyLoss(P tfinal , t t (14)

[0110] In the formula, CrossEntropyLoss is the cross-entropy loss function; t t The index of the t-th real tag token is predicted from the vocabulary V, i.e., t t ∈[0, |V|+|E|]; |V| represents the size of the vocabulary; |E| represents the number of entities.

[0111] In another embodiment, the present invention also provides an electronic device including a processor and a memory; the memory for storing a computer program; and the processor for executing the program stored in the memory to implement the steps of the above method.

[0112] In yet another embodiment, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0113] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. An abstract summary generation method based on dual-granularity guidance and entity replication, which is used to generate abstract summaries from source text X = {x1, x2, ..., x...} n Compress it into a concise abstract summary text Y = {y1, y2, ..., y...} m },in, n represents the total length of the source text X, and m represents the total length of the abstract summary text Y. The method is characterized by the following steps: Step S1, encoding-decoding, specifically includes the following steps: S11. Encode the source text X using an encoder to obtain the contextual hidden state representation H of the contextual semantic features of the source text X. e The formula is as follows: H e =Encoder(X) (1) In the formula, in h represents the hidden state representation of the i-th token in the encoder, where h is the size of the hidden state representation. S12, Context-based Hidden State Representation H e And the generated text Y containing t-1 tokens ＜t ={y1, y2, ..., y t-1 The encoder-decoder attention mechanism uses the decoder to generate the hidden state representation of the t-th token in the t-th decoding stage. Finally, the output Y is predicted from vocabulary V. t The probability P of is given by the following formula: In the formula, E∈R V×h Let V be the embedding matrix of the vocabulary V, and softmax be the activation function. Step S2, based on dual-granularity guidance of key tokens and key entities, specifically includes the following steps: S21. Obtain key token-level guidance information. First, use the encoder's contextual hidden state representation to learn the token-vocabulary similarity. The specific formula is as follows: S t =gelu(H e W1)W2E (3) In the formula, A token (i.e., x) in the source text X i The context similarity between tokens in the vocabulary V and the word V, where n is the total number of tokens; W1 and W2 are the weight matrices to be learned, and H is the weight matrix to be learned. e For context-hidden state representation, E∈R V×h Let V be the embedding matrix of the vocabulary V, and GELU be the activation function. S22. Combine Token-Vocabulary similarity with decoder cross-attention to generate guidance information. Specifically, in the t-th decoding stage, use the cross-attention weights of the last layer L of the decoder. The key token-vocabulary similarity is weighted and summed to obtain the decoder's key token-level guidance information. The specific formula is as follows: In the formula, For cross-attention weights, Y represents t The cross-attention distribution between the source token and the last layer L of the decoder, and This is the cross-attention weight of the i-th token; S23. Solve for the matrix representation of key token-level guidance information. The specific formula is as follows: F token =A d_token S t (5) In the formula, F token ∈R n×|V| , f i token This represents the key token-level guidance information for step i; A d_token for Matrix form; S24. Obtain key entities, each entity consists of multiple tokens {x} i , ..., x j If the entity is composed of}, then the hidden state of this entity is represented as Here, avg represents the average value function; the subscripts i and j represent any few of them, that is, the hidden state representation of this entity is obtained by averaging the hidden state representations of these tokens; S25. Obtain key entity-level guidance information. First, use the Long Short-Term Memory (LSTM) model to obtain the contextual hidden state representation of the entity. The specific formula is as follows: H e_entity =LSTM(R) (6) In the formula, R = {e1, e2, ..., e} k }, where k represents the number of entities; S26. Context-based Hidden State Representation H e_entity Let's learn about Entity-EntitiesSet similarity S e The formula is as follows: S e =(gelu(H e_entity W3)W4E (7) In the formula, Entity-EntitiesSet similarity for an entity, where k represents the number of entities; W3 and W4 are the weight matrices to be learned, and H... e_entity The context-hidden state representation of an entity, E∈R V ×h Let V be the embedding matrix of the vocabulary V, and GELU be the activation function. S27, Calculate the similarity between Entity-EntitiesSets. e Combined with the decoder's cross-attention, key entity-level guidance information is finally generated. Specifically, in the t-th decoding stage, the cross-attention weights of the last layer L of the decoder are used. Similarity S between Entity-EntitiesSet e By performing a weighted summation, the key entity-level guidance information for the decoder is finally obtained, as shown in the following formula: In the formula, Here, represents the cross-attention weights, indicating the distribution of cross-attention between the entity context and the last layer L of the decoder. This is the cross-attention weight of the i-th entity; S28. Solve for the matrix representation of key entity-level guidance information. The specific formula is as follows: F entity =A d_entity S e (9) In the formula, F entity ∈R n×|V| , f i entity Represents the key entity-level guidance information for step i; A d_entity for Matrix form; S29. Merge key token-level guidance information and key entity-level guidance information to obtain the corrected Y for predicting output from vocabulary V. t The probability of is given by the following formula: In the formula, Y t Y is the text containing the t-th token. ＜t The generated text contains t-1 tokens, λ is a custom hyperparameter, E∈R V×h Let V be the embedding matrix of the vocabulary. f is the hidden state representation of the key token generated by the decoder in step t. t token f represents the key token-level guidance information for step t. t entity This represents the key entity-level guidance information at step t, where softmax is the activation function. Step S3, entity copying, specifically includes the following steps: S31. Calculate the logarithmic distribution vector of entities generated using the entity decoder, as shown in the following formula: In the formula, Let Q be the hidden state representation of the key token generated by the decoder at step t, and let Q and K be learnable weight matrices. The encoder hidden state representation of the j-th entity; S32. To determine whether to copy or generate an entity in the t-th decoding stage, it is necessary to calculate the probabilities of copying and generating. Therefore, the hidden state of the decoder is mapped to singular values ​​as the copying probability, as shown in the following formula: In the formula, Let be the probability of replicating an entity at step t. Then the probability of generating an entity is... Sigmad is the activation function, and FFN represents a fully connected feedforward neural network. Let be the hidden state representation of the key token generated by the decoder at step t; S33. Calculate the final generation probability: The specific formula is as follows: In the formula, This represents the logarithmic distribution vector of the entity generated using the decoder; This represents the logarithmic distribution vector of the token generated using the encoder; Step S4: Use the cross-entropy loss function as the final loss function. The specific formula is as follows: In the formula, CrossEntropyLoss is the cross-entropy loss function; t t The index of the t-th real tag token is predicted from the vocabulary V, i.e., t t ∈[0, |V|+|E|]; |V| represents the size of the vocabulary; |E| represents the number of entities.

2. An electronic device, characterized in that, The electronic device includes a processor and a memory; the memory is used to store computer programs; the processor is used to execute the program stored in the memory to implement the steps of the abstract digest generation method of claim 1.

3. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the abstract digest generation method of claim 1.

Images