Text content summary generation method based on knowledge graph

Through the knowledge graph-based text content abstract generation method, and the entity enhancement and BERT model are used to solve the problem of professional text abstract extraction accuracy. The generated abstract is more accurate and cohesive, and is suitable for content abstract generation in multiple articles.

CN114358006BActive Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210018039.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-07
Publication Date
2024-11-08
Estimated Expiration
2042-01-07

AI Technical Summary

Technical Problem

The existing text summary model is difficult to accurately extract abstracts from texts with strong professionalism, resulting in ambiguity and lack of attraction in the generated abstracts.

Method used

Using a knowledge graph-based text content summary generation method, a more condensed text summary is generated through entity enhancement and BERT pre-trained model, combined with an encoder and a decoder. Specific steps include preprocessing, entity enhancement, encoding and decoding of the input text, using the knowledge graph network and the BERT model to calculate the entity similarity, and generating the final summary.

Benefits of technology

Through knowledge graph technology, the text is extracted and logical structure analysis is performed. The generated abstract is more accurate and cohesive, suitable for the generation of content abstracts of single articles and multiple articles, realizing automatic summary sorting of professional texts such as natural resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114358006B_ABST
    Figure CN114358006B_ABST
Patent Text Reader

Abstract

The text content summary generation method based on knowledge graph uses knowledge graph technology to extract knowledge nodes and analyze logical structures of longer text content, and uses graph semantic summarization technology to generate a more cohesive text content summary. This method can not only be used to generate content for a single article, but also can generate content summaries for multiple articles by combining classification and fusion technology, and can realize automatic summarization of various aspects of a natural resource.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of text content processing, and specifically relates to a method for generating a text content summary based on a knowledge graph. Background Art

[0002] With the development of deep learning related theories and technologies, text summarization technology, as an upstream task of natural language processing technology, compresses and summarizes the information of a specified document to provide a short and comprehensive summary. The core of single-document automatic summarization technology is to evaluate the core information in a single text, and based on technologies such as semantic role labeling technology and semantic dependency technology, deeply understand text units such as words and sentences, summarize key information, and generate the final summary.

[0003] However, existing text summarization technologies mainly focus on short texts, and the generated summaries are mostly a summary of the original text. Since there are multiple entity keywords in the text, if only the original text information is used, the summary model cannot correctly understand the meaning of the entity, resulting in the final generated summary being ambiguous and lacking in attractiveness. Summary of the invention

[0004] The technical problem to be solved by the present invention is that for highly professional texts such as natural resource survey documents, the existing summary model cannot accurately extract the summary in the text, and how to improve the accuracy of professional text summary extraction. Therefore, a text content summary generation method based on knowledge graph is proposed.

[0005] The text content summary generation method based on knowledge graph includes the following steps:

[0006] Step 1: preprocess the input text, segment it into sentences to get a set of sentences, and set the summary output format;

[0007] Step 2: Use the knowledge graph network to perform entity enhancement on the text based on entity concept sets and concept similarity;

[0008] Step 3: For the obtained sentence set, use the BERT pre-trained model to initialize the text sequence vector;

[0009] Step 4: Use the encoder to encode the text sentence and the entity-enhanced sentence, and concatenate the two through the decoder to generate a summary.

[0010] Furthermore, the step 1 includes the following process:

[0011] Input the original text S, refer to the summary A, use the sentence segmentation tool jieba to segment S, and get the sentence set L s, and perform word splitting operation on summary A to obtain the token combination of summary A, and add the start identifier A at the beginning and end respectively start and the end identifier A end , and control the length of summary A to be K characters;

[0012] Furthermore, the step 2 includes the following process:

[0013] For the set L s The sentence in l g , using the named entity recognition tool ltp, we get sentence l g The entity set E = {e1,e2,…e m}; Use the knowledge graph network to perform entity enhancement on the original text S and obtain the entity enhancement set D Inc , the specific process is as follows:

[0014] In the development domain knowledge graph G containing entities and entity concepts, if entity e i ∈G, then the concept set of the entity is obtained according to the graph

[0015] Calculate l using the text similarity function Sim g The similarity with each concept in the concept set, where text similarity is calculated as follows:

[0016]

[0017] Among them, sentence l g The vector is represented as l g ={x1,x2,…,x n}, d i The vector is represented by d i ={y1,y2,…,y n}, the vector representation of the text is extracted through the BERT pre-trained model;

[0018] Then, select the g The best matching entity e i A concept

[0019] Set the concept candidate threshold k if When the concept Join the entity enhancement concentration;

[0020] Repeat the above steps, traverse the entity set E, and get sentence l g The entity enhancement set D Inc .

[0021] Furthermore, step 3 includes the following process:

[0022] For the sentence set L s The sentence in l g , a single Chinese character is divided into a token set V of length M g ={v1,v2,…v M}, and use the common vocabulary provided by the Bert Chinese pre-training model to convert V g The single words in the word list are converted into IDs in the word list, and the original text sequence P = {p1, p2, ... p M}, where P = BertID (V g );

[0023] For the token set in summary A, use the BertID function to obtain a summary text sequence Q of length K = {q1, q2, ...q K}; For entity enhancement set D Inc The Token set in the Token set uses the BertID function to obtain an entity text sequence R with a length of N = {r1, r2, ... r N};

[0024] For the original text sequence P, summary text sequence Q, entity text sequence R, the Bert Chinese pre-training model BertEmb function is used. The BERTEmb function is the Encoder part in the Transformer architecture, which includes N layers of encoder calculation units stacked together. Each layer contains Mutil-HeadAttention, LayerNormalization, and Feed Forward. The original text sequence vector representation X = {x1, x2, ... x M}, the abstract text sequence vector representation Y = {y1, y2, ... y K}, entity text sequence vector representation Z = {z1,z2,…z N}.

[0025] Furthermore, step 4 includes the following process:

[0026] In the original text encoder, the original text sequence vector representation X is used as input, and the LSTM model is used to calculate the hidden layer state h at the t1 (1≤t1≤M) time step t1 , where h t1 =LSTM enc (x t1 ,h x1-1 ), x t1 is the input of time step t1, h t1-1is the hidden state of the previous time step; for a sequence input of length M, after a total of M time steps, the original hidden state sequence H = {h1,h2,…h M};

[0027] The model architecture of the entity encoder is the same as that of the original text encoder. It takes the entity text sequence vector representation Z as input and calculates the hidden state h' at the t2 (1≤t2≤K) time step t2 =LSTM enc (x t2 ,h' t2-1 ), and after N time steps, the entity part hidden state sequence H'={h'1,h'2,…h' N};

[0028] In the decoder, the hidden state sequence of the last time step of H and H' is used to initialize the initial state of the decoder s0 = ReLU (W f ·[h M ,h' N ]), where ReLU(x)=max(0,x), W f is the learning parameter;

[0029] In the decoder, the LSTM model is used to calculate the hidden state s at time step t (0≤t≤N) t =LSTM dec (y t-1 ,[s t-1 ,c t-1 ,c' t-1 ]), where s t-1 is the hidden state output of the previous time step, y t-1 is the decoder input of the previous time step, c t-1 ,c' t-1 is the context vector of the original text information and entity information at the previous time step;

[0030] For the current time step t, the decoder’s attention distribution is calculated using the original text information and entity information. The calculation method is as follows:

[0031]

[0032] a t =softmax(u t )

[0033]

[0034] a' t =softmax(u' t )

[0035] where a t , a' t is the attention distribution of original information and entity information at time step t, u t is the attention information of the hidden layer of the original information at time step t, where is the attention information of the i-th time step, h i is the hidden state of the original text at the i-th time step; u' t is the attention information of the hidden layer of entity information at time step t, where is the attention information of the jth time step, h' j is the hidden state of the entity at the jth time step; W h , W s , W' h , W' s , v T , v' T , b attn , b' attn is the learning parameter;

[0036] Using the attention distribution of the original text information and entity information at time step t, calculate the context vector of the current time step where a ti represents the attention representation of the original text at the i-th time, a' tj represents the attention representation of the entity at the jth time step;

[0037] During decoding, the parameter λ is used to control the ratio of original information and entity information, and the probability matrix P(w) of the words in the vocabulary is obtained:

[0038]

[0039] The current time step in is the model learning parameter, σ is the sigmoid function;

[0040] The loss function for model training is is the real character of the current time step, and the back propagation algorithm is used to update the learning parameters, that is, the function x'=BP(x), x is the learning parameter;

[0041] At this point, the n words with the highest output probability at time t are obtained. Through the beam-search algorithm, these n words are combined with the words at the previous time t-1 to generate the candidate summary set at time t. The beam-search algorithm stipulates that the size of beamsize is k. In the first time step, the k words with the largest current conditional probability are selected as the first word of the candidate output sequence. In each subsequent time step, based on the output sequence of the previous step, the k words with the largest conditional probability among all combinations are selected as the candidate output sequence at that time step. K candidates are always maintained. Finally, the best one is selected from the k candidates.

[0042] Repeat the above process until the output word is A end The identifier or generated digest length is K, and the final digest is obtained.

[0043] The beneficial effects achieved by the present invention are as follows: the present method uses the knowledge graph technology to extract knowledge nodes and analyze the logical structure of a long text content, and uses the graph semantic summarization technology to generate a more condensed text content summary. The method can not only be used to generate content for a single article, but also can realize the generation of content summaries for multiple articles by combining classification fusion technology, and can realize the automatic summarization of various aspects of a natural resource. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The figure is a flow chart of the overall structure of the method in the embodiment of the present invention.

[0045] Figure 2 This is a knowledge graph entity mapping flowchart in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0047] This embodiment proposes a method for generating a text content summary based on a knowledge graph. This embodiment is mainly aimed at natural resource texts. Generally, the description texts about natural resources (for example, the introductions to the "Yellow River" and "Yangtze River" in Baidu Encyclopedia) are long and contain many professional terms, making them difficult to read.

[0048] The following uses the original text and summary text of the Yellow River in Baidu Encyclopedia as an example to illustrate the specific implementation of the summary generation algorithm of this patent:

[0049] In step 100, for the input Yellow River original text S and Yellow River summary A, sentence segmentation tool (jieba) is used to segment S to obtain a sentence set L s , and add the start identifier A to the beginning and end of summary A respectively startand the end identifier A end , and control the length of summary A to be K characters.

[0050] Reference Figure 2 In step 201, for the set L s The sentence in l g , using the named entity recognition tool ltp, we get sentence l g The entity set E = {e1,e2,…e m}; Use the knowledge graph network to perform entity enhancement on the original text S and obtain the entity enhancement set D Inc , the specific process is as follows:

[0051] In step 202, in the development domain knowledge graph G (the graph contains entities and entity concepts), if entity e i ∈G, then the concept set of the entity can be obtained according to the graph For example, the entity concepts corresponding to the entity "Yellow River" include: China's second longest river, Taiwan male actor, Handan Yellow River old course, etc.

[0052] In step 203, the text similarity function Sim is used to calculate l g The similarity with each concept in the concept set, where text similarity is calculated as follows:

[0053]

[0054] Among them, sentence l g The vector is represented as l g ={x1,x2,…,x n}, d i The vector is represented by d i ={y1,y2,…,y n}, the vector representation of the text is extracted through the BERT pre-trained model.

[0055] Then, select the g The best matching entity e i A concept

[0056] And select g The best matching entity e i A concept For example, the concept that best matches the text about the Yellow River is: the second longest river in China.

[0057] In step 204, a concept candidate threshold k is set. When the concept Added to the entity enhancement set.

[0058] After traversing the entity set E, we get sentence l g The entity enhancement set D Inc .

[0059] In step 300, for the sentence set L s The sentence in l g , a single Chinese character is divided into a token set V of length M g ={v1,v2,…v M}, and use the common vocabulary provided by the Bert Chinese pre-training model to convert V g The single word in the word is converted into an ID in the vocabulary, and the conversion function BertID is used to obtain the text sequence P = {p1, p2, ... p M}, where P = BertID (V g ).

[0060] For the token set in summary A, use the BertID function to obtain a summary text sequence Q of length K = {q1, q2, ...q K}; For entity enhancement set D Inc The Token set in the Token set uses the BertID function to obtain an entity text sequence R with a length of N = {r1, r2, ... r N}.

[0061] For the original text sequence P, summary sequence Q, entity sequence R, the Bert Chinese pre-training model BertEmb function is used. The BERTEmb function is the Encoder part in the Transformer architecture, which includes N layers of encoder calculation units stacked together. Each layer contains Mutil-HeadAttention, LayerNormalization, and FeedForward. The original text sequence vector representation X = {x1, x2, ... x M}, the abstract text sequence vector representation Y = {y1, y2, ... y K}, entity text sequence vector representation Z = {z1,z2,…z N}.

[0062] In step 400, in the original text encoder, the original text sequence X is used as input, and the LSTM model is used to calculate the hidden layer state h at the t1 (1≤t1≤M) time step. t1 , where h t1 =LSTM enc (x t1 ,h t1-1 ), x t1 is the input of time step t1, ht1-1 is the hidden state of the previous time step. For a sequence input of length M, after a total of M time steps, the original hidden state sequence H = {h1,h2,…h M}.

[0063] The model architecture of the entity encoder is the same as that of the original text encoder. It takes the entity text sequence Z as input and calculates the hidden state h' at the t2 (1≤t2≤K) time step t2 =LSTM enc (x t2 ,h' t2-1 ), and after N time steps, the entity part hidden state sequence H'={h'1,h'2,…h' N}.

[0064] In the decoder, the hidden state sequence of the last time step of H and H' obtained in step 6 and step 7 is used to initialize the initial state of the decoder s0 = ReLU (W f ·[h M ,h' N ]), where ReLU(x)=max(0,x), W f To learn the parameters.

[0065] In the decoder, the LSTM model is used to calculate the hidden state s at time step t (0≤t≤N) t =LSTM dec (y t-1 ,[s t-1 ,c t-1 ,c' t-1 ]), where s t-1 is the hidden state output of the previous time step, y t-1 is the decoder input of the previous time step, c t-1 ,c' t-1 is the context vector of the original text information and entity information at the previous time step.

[0066] For the current time step t, the decoder’s attention distribution is calculated using the original text information and entity information. The calculation method is as follows:

[0067]

[0068] a t =softmax(u t )

[0069]

[0070] a' t =softmax(u't )

[0071] where a t , a' t is the attention distribution of original information and entity information at time step t, u t is the attention information of the hidden layer of the original information at time step t, where is the attention information of the i-th time step, h i is the hidden state of the original text at the i-th time step; u' t is the attention information of the hidden layer of entity information at time step t, where is the attention information of the jth time step, h' j is the hidden state of the entity at the jth time step; W h , W s , W' h , W' s , v T , v' T , b attn , b' attn To learn the parameters.

[0072] Using the attention distribution of the original text information and entity information at time step t, calculate the context vector of the current time step where a ti represents the attention representation of the original text at the i-th time, a' tj Represents the attention representation of the entity at the jth time step.

[0073] During decoding, λ is used to control the ratio of original information and entity information, and the probability matrix P(w) of words in the vocabulary is obtained:

[0074]

[0075] The current time step in is the model learning parameter, and σ is the sigmoid function.

[0076] The loss function for model training is is the real character of the current time step, and the back propagation algorithm is used to update the learning parameters, that is, the function x'=BP(x), x is the learning parameter.

[0077] Get the n words with the highest output probability at time t, and use the beam-search algorithm to combine these n words with the words at the previous t-1 time to generate a candidate summary set at time t, where the beam-search algorithm specifies that the size of beamsize is k. In the first time step, select the k words with the largest current conditional probability as the first word of the candidate output sequence. In each subsequent time step, based on the output sequence of the previous step, select the k words with the largest conditional probability among all combinations as the candidate output sequence at that time step. Always keep k candidates. Finally, pick the best one from the k candidates.

[0078] Repeat the above process until the output word is A end Identifier or generate a summary of the Yellow River original text introduction with a summary length of K.

[0079] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A text content summary generation method based on knowledge graph, characterized by: The method comprises the following steps: Step 1: preprocess the input text, segment it into sentences to get a set of sentences, and set the summary output format; Step 2: Use the knowledge graph network to perform entity enhancement on the text based on entity concept sets and concept similarity; Step 2 The process includes: For the set L s The sentence in l g , using the named entity recognition tool ltp, we get sentence l g The entity set E = {e1, e2, ...e m }; Use the knowledge graph network to perform entity enhancement on the original text S and obtain the entity enhancement set D Inc , the specific process is as follows: In the development domain knowledge graph G containing entities and entity concepts, if entity e i ∈G, then the concept set of the entity is obtained according to the graph Calculate l using the text similarity function Sim g The similarity with each concept in the concept set, where text similarity is calculated as follows: Among them, sentence l g The vector is represented as l g ={x1, x2, ..., x n }, d i The vector is represented by d i ={y1, y2, ..., y n }, the vector representation of the text is extracted through the BERT pre-trained model; Then, select the g The best matching entity e i A concept Set the concept candidate threshold k if When the concept Join the entity enhancement concentration; Repeat the above steps, traverse the entity set E, and get sentence l g The entity enhancement set D Inc ; Step 3: For the obtained sentence set, use the BERT pre-trained model to initialize the text sequence vector; Step 4: Encode the text sentence and the entity-enhanced sentence using the encoder, and concatenate the two using the decoder to generate a summary; The step 4 includes the following process: In the original text encoder, the original text sequence vector representation X is used as input, and the LSTM model is used to calculate the hidden layer state h at the t1 time step. t1 , 1≤t1≤M, where h t1 =LSTM enc (x t1 ,h t1-1 ), x t1 is the input of time step t1, h t1-1 is the hidden state of the previous time step; for a sequence input of length M, after a total of M time steps, the original hidden state sequence H = {h1, h2, ...h M }; The model architecture of the entity encoder is the same as that of the original text encoder. It takes the entity text sequence vector representation Z as input and calculates the hidden state h′ at the t2 time step t2 =LSTM enc (x t2 , h′ t2-1 ), 1≤t2≤K, and after N time steps, the entity part hidden state sequence H′={h′1, h′2, ...h′ N }; In the decoder, the hidden state sequence of the last time step of H and H′ is used to initialize the initial state of the decoder s0 = ReLU (W f ·[h M , h′ N ]), where ReLU(x)=max(0,x), W f is the learning parameter; In the decoder, the LSTM model is used to calculate the hidden state s at time step t t =LSTM dec (y t-1 ,[s t-1 , c t-1 , c′ t-1 ]), 0≤t≤N, where s t-1 is the hidden state output of the previous time step, y t-1 is the decoder input of the previous time step, c t-1 , c′ t-1 is the context vector of the original text information and entity information at the previous time step; For the current time step t, the decoder’s attention distribution is calculated using the original text information and entity information. The calculation method is as follows: a t =softmax(u t ) a′ t =softmax(u′ t ) where a t , a′ t is the attention distribution of original information and entity information at time step t, u t is the attention information of the hidden layer of the original information at time step t, where is the attention information of the i-th time step, h i is the hidden state of the original text at the i-th time step; u′ t is the attention information of the hidden layer of entity information at time step t, where is the attention information of the jth time step, h′ j is the hidden state of the entity at the jth time step; W h , W s , W′ h , W′ s , v T , v′ T , b attn , b′ attn is the learning parameter; Using the attention distribution of the original text information and entity information at time step t, calculate the context vector of the current time step where a ti represents the attention representation of the original text at the i-th time, a′ tj represents the attention representation of the entity at the jth time step; During decoding, the parameter λ is used to control the ratio of original information and entity information, and the probability matrix P(w) of the words in the vocabulary is obtained: The current time step in is the model learning parameter, σ is the sigmoid function; The loss function for model training is is the real character of the current time step, and the back propagation algorithm is used to update the learning parameters, that is, the function x′=BP(x), where x is the learning parameter; At this point, the n words with the highest output probability at time t are obtained. Through the beam-search algorithm, these n words are combined with the words at the previous time t-1 to generate a candidate summary set at time t. The beam-search algorithm stipulates that the size of beamsize is k. In the first time step, the k words with the largest current conditional probability are selected as the first word of the candidate output sequence. In each subsequent time step, based on the output sequence of the previous step, the k words with the largest conditional probability among all combinations are selected as the candidate output sequence at that time step. K candidates are always maintained. Finally, the best one is selected from the k candidates. Repeat the above process until the output word is A end The identifier or generated digest length is K, and the final digest is obtained.

2. The method for generating text content summaries based on knowledge graph according to claim 1, characterized in that: Step 1 The process includes: Input the original text S, refer to the summary A, use the sentence segmentation tool jieba to segment S, and get the sentence set L s , and perform word splitting operation on summary A to obtain the token combination of summary A, and add the start identifier A at the beginning and end respectively start and the end identifier A end , and control the length of summary A to be K characters.

3. The method for generating text content summaries based on knowledge graph according to claim 1, characterized in that: Step 3 The process includes: For the sentence set L s The sentence in l g , a single Chinese character is divided into a token set V of length M g = {v1, v2, ... v M }, and use the common vocabulary provided by the Bert Chinese pre-training model to convert V g The single words in the word list are converted into IDs in the word list, and the original text sequence P = {p1, p2, ... p M }, where P = BertID (V g ); For the token set in summary A, use the BertID function to obtain a summary text sequence Q of length K = {q1, q2, ...q K }; For entity enhancement set D Inc The Token set in the BertID function is used to obtain an entity text sequence R = {r1, r2, ...r N }; For the original text sequence P, summary text sequence Q, entity text sequence R, the Bert Chinese pre-training model BertEmb function is used to obtain the embedding vector representation of the encoder and decoder. The BERTEmb function is the Encoder part of the Transformer architecture, which includes N layers of encoder calculation units stacked together. Each layer contains Mutil-Head Attention, Layer Normalization, and Feed Forward. The original text sequence vector representation X = {x1, x2, ...x M }, abstract text sequence vector representation Y = {y1, y2, ...y K }, entity text sequence vector representation Z = {z1, z2, ...z N }.

Citation Information

Patent Citations

  • Text similarity calculation method and device, storage medium and electronic device

    CN110489751A

  • Method for generating guided text abstract based on Transformer

    CN111897949A

  • Abstract generation method based on planning mechanism and knowledge graph guidance

    CN112597296A

  • Automatic abstract extraction method and apparatus, and computer device and storage medium

    WO2019169719A1