A method and system for generating multi-document scientific literature abstracts by fusing rhetorical structures

Abstract

The application discloses a kind of fusion rhetoric structure's multi-document scientific literature abstract generation method and system, the present application includes determining rhetoric structure system, the relevant work section of paper sample is marked to construct training dataset and trains rhetoric structure classifier to establish the mapping relationship between relevant work section, rhetoric function;The relevant work section of target literature is marked with the rhetoric function using the rhetoric structure classifier trained, and the relevant work generation model based on Transformer is used, rhetoric structure information is fused, and the summary section of the input target literature and the summary section of the reference literature cited by target literature are used to output the relevant work section of target literature.The present application aims to solve the problem that the existing multi-document scientific literature abstract model is prone to rhetoric structure confusion, and the logic of the preceding and following text is not clear, to generate more accurate and coherent abstracts in the specific scenario of automatic generation of relevant work sections for multi-document scientific literature abstracts.

Description

Technical Field

[0001] This invention relates to automatic document summarization technology in the field of text processing, specifically to a method and system for generating summarization of multi-document scientific and technological literature that incorporates rhetorical structures. Background Technology

[0002] With the popularization of the internet and the rapid development of information technology, educational resources and data are characterized by diversified sources and heterogeneous forms. In addition to traditional textbooks, electronic lecture notes, online publications, and MOOCs (Massive Open Online Courses) have emerged on a large scale. This massive amount of multi-source, heterogeneous educational data will lead to information overload and learning disorientation. How to efficiently integrate these massive resources and data is a key scientific issue in the development of science and education innovation platforms. Multi-document scientific resource aggregation technology, with multi-document scientific literature summarization technology at its core, is a key technology for solving this scientific problem. The purpose of multi-document scientific literature summarization is to generate concise, condensed, and accurate summaries for a given single or multiple scientific documents, thereby alleviating the problem of information overload in human-computer collaborative learning.

[0003] Related work generation falls under the category of summarization. It refers to generating the content of related work sections of a document based on the abstracts of the target and reference documents. According to different summarization methods, multi-document scientific literature summaries can be divided into extractive summarization and abstractive summarization. Extractive summarization extracts ready-made sentences from the original text to form a summary. Since it only requires scoring the sentences in the original text, sorting them according to the scores, and then combining post-processing steps such as redundancy removal, reference substitution, and tense transformation, the summary is simple and fluent. However, it can also result in stiff summaries, disjointed sentences, and a lack of adherence to human-generated summarization patterns, often leading to poor readability. Conversely, abstractive summarization aims to generate summaries using new expressions, producing words and phrases not found in the original text. It features contextual coherence and strong readability, and has therefore become the mainstream approach in recent years.

[0004] Early generative summarization research primarily focused on sentence compression. Researchers found that professional summarizers typically use this method of compressing original sentences to generate summaries. Based on the Ziff-Davis corpus, an algorithm was designed to match manually written summaries with sentences in the original text. The results showed that 78% of the summary sentences were generated by editing the original text, and more than half of these sentences were obtained by compressing the original sentences, i.e., deleting parts of the text. Sentence compression research includes rule-based methods and machine learning-based methods. Rule-based methods use information including syntactic knowledge, contextual information, and word frequency to comprehensively determine which parts of the sentence can be deleted; machine learning-based methods use data-driven learning to learn which nodes in the syntax tree can be deleted, thus overcoming the limitations of rule-based methods.

[0005] In recent years, with the explosive development of deep learning, text generation technology has gradually matured and begun to be applied in the field of multi-document scientific literature summarization. Deep learning-based generative summarization methods employ an encoder-decoder framework. The encoder is responsible for encoding the input text sequence into a low-dimensional dense real-valued vector to extract text features, and the decoder decodes from this vector to generate the corresponding output sequence. Initially, researchers used recurrent neural networks (RNNs) or convolutional neural networks (CNNs) as encoders and RNNs as decoders. Later, with the introduction of the Transformer, its multi-head self-attention mechanism overcame the local dependencies of RNNs, achieving superior performance on many tasks. Current research on multi-document scientific literature summarization is also based on the Transformer encoder-decoder structure.

[0006] Previous studies on multi-document scientific literature summarization based on Transformer have been able to accurately model the content of scientific literature and the cross-relationships between multiple documents. However, the summaries they generate often suffer from problems such as chaotic rhetorical structure and unclear logical context. These problems greatly affect the effectiveness of the summaries and hinder the application of multi-document scientific and technological resource aggregation.

[0007] Therefore, how to overcome the problems of chaotic rhetorical structure and unclear logical context in the process of abstract generation, and generate more accurate and coherent abstracts, is a technical problem that needs to be solved urgently in multi-document scientific and technological literature abstracting. Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a method and system for generating multi-document scientific and technological literature abstracts that integrates rhetorical structures, in response to the above-mentioned problems in the existing multi-document scientific and technological literature abstract models. This invention aims to solve the problems of chaotic rhetorical structures and unclear logical context in existing multi-document scientific and technological literature abstract models, and to generate more accurate and coherent abstracts by automatically generating abstract chapters as a specific scenario for multi-document scientific and technological literature abstracts.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0010] A method for generating multi-document scientific and technological literature abstracts that integrates rhetorical structures includes:

[0011] S101, Rhetorical structure analysis is performed on the relevant working sections of the paper sample to determine the rhetorical structure system of the sentences, the rhetorical structure system including different rhetorical functions;

[0012] S102, based on the rhetorical structure system of sentences, the sentences of the relevant working chapters of the paper sample are annotated with rhetorical structure.

[0013] S103, construct a training dataset using the relevant working chapters of the paper samples obtained by rhetorical structure annotation, and use the training dataset to train a rhetorical structure classifier to establish the mapping relationship between the relevant working chapters and rhetorical functions of the paper.

[0014] S104, using a trained rhetorical structure classifier to annotate the rhetorical functions of relevant working chapters in the target document;

[0015] S105 utilizes a Transformer-based related work generation model, integrates rhetorical structure information, and outputs the related work sections of the target document based on the abstract sections of the target document and the abstract sections of the references cited by the target document.

[0016] Optionally, the rhetorical structure system of the sentence determined in step S101 is a two-level rhetorical structure. The first-level rhetorical structure includes some or all of the following six types: description of the topic, citation of references, establishing a connection between existing references and one's own work, description of one's own work, conveying other communicative functions, and types that are difficult to determine. The second-level rhetorical structure consists of the rhetorical functions under the first-level rhetorical structure, wherein: the rhetorical functions under the description of the topic include two types: general description of the topic and citation of currently known knowledge; the rhetorical functions under the citation of references include general description of the goals of multiple studies, general description of the methods of multiple studies, general description of the results of multiple studies, description of the goals of a single study, and description of the methods of a single study. There are seven types of rhetorical functions under the categories of description of law, description of the results of a single study, and other citation purposes. Six types of rhetorical functions are categorized under the category of establishing a connection between existing references and one's own work: explaining the relationship between the methods of the references and one's own work, explaining the relationship between the goals of the references and one's own work, explaining the relationship between the goals of the references and one's own work, expressing the shortcomings of the references, explaining the significance of the references, and other comments by the author. Four types of rhetorical functions are categorized under the category of describing one's own work: describing the goals of one's own work, describing the motivation of one's own work, describing the methods of one's own work, and describing the results of one's own work. Two types of rhetorical functions are categorized under the category of conveying other communicative functions: expressing signals of transfer and other functional statements. One type of rhetorical function is difficult to determine.

[0017] Optionally, in step S103, when training the rhetorical structure classifier to establish the mapping relationship between relevant working chapters and rhetorical functions in the paper, the model used for the rhetorical structure classifier is the pre-trained model SciBERT-base, and the loss function used when training the rhetorical structure classifier is the average binary cross-entropy loss function of the gold label.

[0018] Optionally, the relevant work generation model in step S105 is an encoder-decoder structure consisting of an encoder and a decoder connected together. The encoder includes a text encoder and a graph encoder. The text encoder is used to encode words in the abstracts of the target document and the references into word representations, respectively. The graph encoder is used to model the rhetorical structure information of the abstracts into the word representations to obtain graph-enhanced word representations. The rhetorical structure information of the abstracts is labeled using a rhetorical structure classifier, which includes five types: background, target, method, result, and others. The decoder is used to decode the graph-enhanced word representations to obtain the rhetorical planning sequence and generate the content of the relevant work sections of the target document.

[0019] Optionally, the text encoder is a Transformer-based encoder, which includes L1 cascaded text encoding layers with identical structures. Each text encoding layer includes a multi-head pooling layer (MHPool) and a feedforward neural network sublayer (FFN). Each text encoding layer encodes words in the abstract sections of the target document and references into word representations, including:

[0020] S201, firstly, the words in the target document and references are encoded through word embedding to obtain the word representations of the target document and references as follows: and Where i and j represent the j-th word of the i-th statement, and T and D represent the target document and references, respectively;

[0021] S202, For the obtained word representations, use a multi-head pooling layer (MHPool) to process them to obtain the sentence representations of the target documents and references. and Then, the statement representation is added to the word representation according to the following formula, and the enhanced word representation is obtained by using the feedforward neural network sublayer FFN as the final word representation:

[0022]

[0023] In the above formula, The word representation is enhanced for the abstract sections of the target document, and FNN is a feedforward neural network sublayer. The word representation of the abstract section of the target document. The statement representation of the target document; Enhanced word representation for the abstract sections of references. The words in the abstract section of the references are used to represent the text. The 'e' indicates that the statement or word representation in the reference is obtained through word embedding.

[0024] Optionally, the graph encoder is a Transformer-based graph encoder, and the step of modeling rhetorical structure information into word representations to obtain graph-enhanced word representations includes:

[0025] S301, Initialize the global nodes for constructing the rhetorical structure graph, which consists of nodes and edges, with the global nodes serving as the root nodes;

[0026] S302, initialize the nodes in the rhetoric structure graph except for the global node. The nodes except for the global node include words, rhetoric functions, target documents, and references. Among them, word nodes are leaf nodes and are randomly initialized. The representation of the rhetoric function is obtained by average pooling of the representations of all statements under the rhetoric function. The representation of the target document is obtained by average pooling of the representations of all statements within the target document. The representation of the reference is obtained by average pooling of the representations of all statements within the reference. Initialize the edges in the rhetoric structure graph. There are five types of edges: undirected edges between rhetoric functions and the words of the sentences they contain, undirected edges between documents and their sentence rhetoric functions, undirected edges between the same type of rhetoric functions in different documents, undirected edges between target documents and references, and undirected edges between global nodes and all other nodes.

[0027] S303, a Transformer-based graph encoder is used to encode the rhetorical structure graph into graph nodes. The Transformer-based graph encoder includes cascaded L2 graph encoding layers with identical structures. Each graph encoding layer includes a multi-head self-attention sublayer (MHSA) and a feedforward neural network sublayer (FFN). Each graph encoding layer models the rhetorical structure information into the word representation to obtain the graph-enhanced word representation, including: firstly, using the multi-head self-attention sublayer (MHSA) to obtain the target document node representation. The reference node representation is obtained using a multi-head pooling layer MHPool. Then represent the target document node. Adding to the word representation of the target document yields a graph-enhanced word representation of the target document; the reference node representation is then added. Adding the word representation to the references yields the word representation of the references with enhanced graphical information:

[0028]

[0029] In the above formula, The word representation is an augmented version of the abstract chapter graph information of the target document, with FNN being a feedforward neural network sublayer. Enhanced word representation for the abstract sections of the target document. Represented as a node for the target document; Enhanced word representation for the abstract sections of references. Enhanced word representation for the abstract sections of references. This is used to represent reference nodes.

[0030] Optionally, the decoder is a Transformer-based decoder, which includes L3 cascaded decoding layers with identical structures and a linear layer. Each decoding layer includes a multi-head self-attention sublayer (MHSA), a multi-head cross-attention sublayer (MHCA), and a feedforward neural network layer (FFN). Each decoding layer decodes the word representation enhanced with graph information to first extract the rhetorical planning sequence and generate the relevant working chapters of the target literature.

[0031] S401, firstly, the decoding state of the t-th word is calculated using the following formula through a multi-head self-attention sublayer (MHSA). Cross-attention features with target and reference words

[0032]

[0033] In the above formula, LayerNorm is the layer normalization operation, and MHCA represents the multi-head cross-attention sublayer. This indicates the current decoding state; Concat is a channel stacking operation. Word representations enhanced with section diagram information for the target document's abstract. Word representations enhanced with section diagram information for the abstract of references;

[0034] S402, decode the state of the t-th word. Cross-attention features with target and reference words After processing by the feedforward neural network layer FFN, the cross-attention features are then compared with those before processing by the feedforward neural network layer FFN. The summation, followed by layer normalization, serves as the cross-attention feature output by the decoding layer.

[0035] Let the cross-attention feature of the output of the last decoding layer be . The cross-attention features output by the last decoding layer The vector representations are fed into a linear layer, where the probability mapping from the vector representation to the word vocabulary is achieved according to the following formula:

[0036]

[0037] In the above formula, Let W be the probability mapping from a vector representation to a word vocabulary, where Softmax is the softmax activation function. g b is the weight matrix parameter of the linear layer. g The bias parameters for the linear layer; obtain the probability p that words from the relevant working chapters are directly copied from the input document. cAnd calculate the generation probability p of the t-th word based on the following formula. t :

[0038]

[0039] In the above formula, z t These are the weight learning parameters.

[0040] Optionally, the loss function used by the related work generation model during training in step S105 is expressed as follows:

[0041]

[0042] In the above formula, Represents the loss function. This refers to the content of the relevant working chapters obtained by combining the content of the rhetorical planning sequence and the relevant working chapters as classification labels; p t Let be the probability of generating the t-th word. The t-th word represents the content of the relevant working chapter obtained by splicing and combining the content of the rhetorical planning sequence and the relevant working chapter of the standard, and the rhetorical planning sequence is composed of rhetorical plans, each of which is a symbolic representation of a corresponding rhetorical function. The splicing and combining of the content of the rhetorical planning sequence and the relevant working chapter of the standard includes splicing using specified splicing characters.

[0043] Furthermore, this embodiment also provides a multi-document scientific and technological literature abstract generation system that integrates rhetorical structures, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the multi-document scientific and technological literature abstract generation method that integrates rhetorical structures.

[0044] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program, which is used to be programmed or configured by a microprocessor to execute the multi-document scientific literature abstract generation method with integrated rhetorical structure.

[0045] Compared with the prior art, the present invention has the following main advantages:

[0046] 1. The rhetorical structure classification system of related work chapters constructed by this invention provides a clear insight into the understanding and organization of related work content, and provides support for selecting better related work generation strategies.

[0047] 2. This invention introduces rhetorical structures at both the encoding and decoding ends for document modeling and summary generation, achieving better summary content selection and more coherent rhetorical structures, resulting in clearer and more logical summary output. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the structure of the related work generation model based on Transformer in an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of the rhetorical structure diagram in an embodiment of the present invention.

[0051] Figure 4 This is an example of a relevant working section in an embodiment of the present invention. Detailed Implementation

[0052] like Figure 1 As shown in this embodiment, the multi-document scientific literature abstract generation method integrating rhetorical structures includes:

[0053] S101, Rhetorical structure analysis is performed on the relevant working sections of the paper sample to determine the rhetorical structure system of the sentences, the rhetorical structure system including different rhetorical functions;

[0054] S102, based on the rhetorical structure system of sentences, the sentences of the relevant working chapters of the paper sample are annotated with rhetorical structure.

[0055] S103, construct a training dataset using the relevant working chapters of the paper samples obtained by rhetorical structure annotation, and use the training dataset to train a rhetorical structure classifier to establish the mapping relationship between the relevant working chapters and rhetorical functions of the paper.

[0056] S104, using a trained rhetorical structure classifier to annotate the rhetorical functions of relevant working chapters in the target document;

[0057] S105 utilizes a Transformer-based related work generation model, integrates rhetorical structure information, and outputs the related work sections of the target document based on the abstract sections of the target document and the abstract sections of the references cited by the target document.

[0058] Drawing on the two-level rhetorical structure of the CARS model (see Swales JM, Swales J. Genre analysis: English in academic and research settings [M]. Cambridge University Press, 1990), the rhetorical structure system (PSED) of the sentences determined in step S101 of this embodiment also adopts a two-level rhetorical structure. Specifically, through the analysis of 50 relevant working chapters, the two-level rhetorical structure of the rhetorical structure system in this embodiment is as follows:

[0059] The first-level rhetorical structure (Move) includes some or all of six types: description of the topic, citation of references, establishing a connection between existing references and the work itself, description of the work itself, conveying other communicative functions, and types that are difficult to define. The description of the topic (P) helps the reader understand the current state of knowledge about the topic being summarized and is generally used as the beginning of a paragraph. The citation of references (S) provides a clear and concise summary of the references. Furthermore, in this embodiment, the summary of references is further refined into objectives, methods, and results to explore whether different summarizing strategies differ depending on the aspects of the references mentioned in the abstract. The description of the work itself (D) covers the objectives, motivations, methods, and results of the work. These four first-level rhetorical structures (Moves) are essential for a well-structured related work section, forming the core of the rhetorical structure system of sentences in the related work section (PSED). However, it should be noted that not every relevant work section consists of all four moves mentioned above. Sometimes, authors may omit certain moves to achieve specific narrative purposes. Besides the four moves mentioned above, this embodiment also identifies two other types of moves: those conveying other communication functions, used to convey functions not closely related to the abstract task, including diverting the reader's attention to other references or other sections of the current paper; and those of difficult-to-determine type, used for moves whose category is difficult to determine due to unclear wording, omissions in text recognition processing, garbled characters, or other issues.

[0060] The second-level rhetorical structure (Step) consists of rhetorical functions under the first-level rhetorical structure, including: (1) the rhetorical functions under the description of the topic (P) include two types: general description of the topic and citation of currently known knowledge; (2) the rhetorical functions under the citation of references (S) include seven types: general description of the goals of multiple studies, general description of the methods of multiple studies, general description of the results of multiple studies, description of the goals of a single study, description of the methods of a single study, description of the results of a single study, and other citation purposes; and (3) the rhetorical function package under the connection between existing references and one's own work (E). The rhetorical functions under the description of one's own work (D) include six types: explaining the relationship between the methods of the references and one's own work, explaining the relationship between the goals of the references and one's own work, explaining the shortcomings of the references, explaining the significance of the references, and other comments by the author; (4) The rhetorical functions under the description of one's own work (D) include four types: describing the goals of one's own work, describing the motivation of one's own work, describing the methods of one's own work, and describing the results of one's own work; (5) The rhetorical functions under the function of conveying other communication include two types: expressing the signal of transfer and other functional statements; (6) The rhetorical functions under the difficult-to-determine type include one type of difficult-to-determine type.

[0061] In step S102 of this embodiment, when annotating sentences in relevant work sections of the paper samples using a sentence-based rhetorical structure system, two graduate students majoring in natural language processing were asked to annotate 240 randomly selected relevant works. This embodiment collected inconsistencies during the annotation process, then a third annotator was asked to perform independent annotation, and the final result was determined by majority voting. If the annotation results from the three annotators differed, a meeting was held to decide the final result. The Cohen Kappa coefficient for pairwise consistency between the two annotators was 0.73, indicating the reliability of the annotation results. In this embodiment, the annotated dataset of relevant works with rhetorical structure annotations is named the Relevant Works Rhetorical Structure Annotation Dataset RF-RWG, which contains 240 relevant works and a total of 1254 sentences.

[0062] In step S103 of this embodiment, when training the rhetorical structure classifier to establish the mapping relationship between relevant working chapters and rhetorical functions in the paper, the model used for the rhetorical structure classifier is the pre-trained model SciBERT-base (for details, see "I. Beltagy, K. Lo, A. Cohan, Scibert: A pretrained language model for scientific text, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3615-3620." translated as "Scibert: A Pretrained Language Model for Scientific Text", published at the 2019 Conference on Empirical Methods in Natural Language Processing). The loss function used when training the rhetorical structure classifier is the average binary cross-entropy loss function with gold labels. The training set contained 1044 sentences from 200 relevant works, with Adam as the optimizer. A grid search was performed on the validation set (210 sentences from 40 relevant works) to determine the optimal hyperparameters for the classification model. Ultimately, the best performance on the validation set was 70.1% accuracy, indicating that the trained rhetorical structure classifier can serve as a qualified rhetorical structure classifier for automatically annotating larger datasets of relevant works.

[0063] In step S104 of this embodiment, when the rhetorical function is labeled in the relevant work chapters of the target document using the trained rhetorical structure classifier, it is labeled as including five types: background, target, method, result and others. The rhetorical structure classifier trained in this paper is used to automatically label the source documents of the dataset generated by the relevant work with rhetorical structure.

[0064] like Figure 2As shown, the relevant work generation model in step S105 of this embodiment is an encoder-decoder structure consisting of an encoder and a decoder. The encoder includes a text encoder and a graph encoder. The text encoder is used to encode words in the abstracts of the target document and the references into word representations, respectively. The graph encoder is used to model the rhetorical structure information of the abstracts into the word representations to obtain graph-enhanced word representations. The rhetorical structure information of the abstracts is labeled using a rhetorical structure classifier, including five types: background, target, method, result, and others. The decoder is used to decode the graph-enhanced word representations to obtain the rhetorical planning sequence and generate the content of the relevant work sections of the target document. Figure 2 Chinese w n1 ~w nj These represent the j instances (words) output.

[0065] In this embodiment, the text encoder is a Transformer-based encoder, which includes L1 cascaded text encoding layers with identical structures. Each text encoding layer includes a multi-head pooling layer (MHPool) and a feedforward neural network sublayer (FFN). Each text encoding layer encodes words in the abstract sections of the target document and references into word representations, including:

[0066] S201, firstly, the words in the target document and references are encoded through word embedding to obtain the word representations of the target document and references as follows: and Where i and j represent the j-th word of the i-th sentence, T and D represent the target document and references, respectively, where the subscript e represents word embedding and the superscript L1 is the text encoding layer number;

[0067] S202. For the obtained word representations, the multi-head pooling layer MHPool (for implementation details, refer to "Liu Y, Lapata M. Hierarchical Transformers for Multi-Document Summarization[C] / / Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:5070-5081." translated as Hierarchical Transformers-based Multi-Document Summarization, published at the 2019 Annual Meeting of the Association for Computational Linguistics) is used to process the sentence representations of the target documents and references. and Then, the statement representation is added to the word representation according to the following formula, and the enhanced word representation is obtained by using the feedforward neural network sublayer FFN as the final word representation:

[0068]

[0069] In the above formula, The word representation is enhanced for the abstract sections of the target document, and FNN is a feedforward neural network sublayer. The word representation of the abstract section of the target document. The statement representation of the target document; Enhanced word representation for the abstract sections of references. The words in the abstract section of the references are used to represent the text. The 'e' indicates that the statement or word representation in the reference is obtained through word embedding.

[0070] The word representation obtained by the text encoder fails to consider the rhetorical structure information of the input. Therefore, this embodiment utilizes an additional graph encoder to model the rhetorical structure information into the word representation. In this embodiment, the graph encoder is a Transformer-based graph encoder. Modeling the rhetorical structure information into the word representation to obtain the graph-enhanced word representation includes:

[0071] S301, Initialize the global nodes for constructing the rhetorical structure graph, which consists of nodes and edges, with the global nodes serving as the root nodes, such as... Figure 3 As shown;

[0072] S302, Initialize nodes in the rhetorical structure graph excluding global nodes, such as Figure 3 As shown, nodes other than global nodes include words, rhetorical functions, target documents, and references. Word nodes are leaf nodes, and are randomly initialized. The representation of a rhetorical function is obtained by average pooling of all statement representations under that rhetorical function. The representation of a target document is obtained by average pooling of all statement representations within the target document. The representation of a reference is obtained by average pooling of all statement representations within the reference document. The edges in the rhetorical structure graph are initialized, and there are five types of edges: undirected edges between a rhetorical function and the word of its sentence, undirected edges between a document and its sentence rhetorical function, undirected edges between the same type of rhetorical function in different documents, undirected edges between the target document and the references, and undirected edges between global nodes and all other nodes.

[0073] The S303 employs a Transformer-based graph encoder (details can be found in "Koncel-Kedziorski, R., Bekal, D., Luan, Y., Lapata, M., Hajishirzi, H.: Text generation from knowledgegraphs with graph transformers. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume"). 1, pp.2284-2293 (2019), translated as Knowledge Graph to Text Generation Based on Graph Transformer, published at the 2019 Association for Computational Linguistics North America Conference, encodes rhetorical structure graphs using graph nodes. The Transformer-based graph encoder includes cascaded and structurally identical L2 graph encoding layers. Each graph encoding layer includes a multi-head self-attention sublayer (MHSA) and a feedforward neural network sublayer (FFN). Each graph encoding layer models rhetorical structure information into word representations to obtain graph-enhanced word representations. This includes: firstly, using the multi-head self-attention sublayer (MHSA) to obtain the target document node representation. The reference node representation is obtained using a multi-head pooling layer MHPool. Then represent the target document node. Adding to the word representation of the target document yields a graph-enhanced word representation of the target document; the reference node representation is then added. Adding the word representation to the references yields the word representation of the references with enhanced graphical information:

[0074]

[0075] In the above formula, The word representation is an augmented version of the abstract chapter graph information of the target document, with FNN being a feedforward neural network sublayer. Enhanced word representation for the abstract sections of the target document. Represented as a node for the target document; Enhanced word representation for the abstract sections of references. Enhanced word representation for the abstract sections of references. This is used to represent reference nodes.

[0076] To address the issue of chaotic rhetorical structure output by relevant work models, this embodiment proposes a method using rhetorical planning as an intermediate decoding step to better generate relevant work. In this embodiment, the decoder is a Transformer-based decoder, comprising L3 cascaded, structurally identical decoding layers and a linear layer. Each decoding layer includes a multi-head self-attention sublayer (MHSA), a multi-head cross-attention sublayer (MHCA), and a feedforward neural network layer (FFN). Each decoding layer decodes the rhetorical planning sequence from the graph-enhanced word representations and generates the relevant work section content of the target document, including:

[0077] S401, firstly, the decoding state of the t-th word is calculated using the following formula through a multi-head self-attention sublayer (MHSA). Cross-attention features with target and reference words

[0078]

[0079] In the above formula, LayerNorm is the layer normalization operation, and MHCA represents the multi-head cross-attention sublayer. This indicates the current decoding state; Concat is a channel stacking operation. Word representations enhanced with section diagram information for the target document's abstract. Word representations enhanced with section diagram information for the abstract of references;

[0080] S402, decode the state of the t-th word. Cross-attention features with target and reference words After processing by the feedforward neural network layer FFN, the cross-attention features are then compared with those before processing by the feedforward neural network layer FFN. The summation, followed by layer normalization, serves as the cross-attention feature output of the decoding layer, which can be expressed as:

[0081]

[0082] It should be noted that the left side of the above equation... The right side represents the cross-attention features output by this decoding layer. The cross-attention features output by the multi-head self-attention sublayer (MHSA) within this decoding layer are represented using the same notation, borrowed from the updated function expression during programming for ease of understanding. Let the cross-attention features output by the last decoding layer be denoted as... The cross-attention features output by the last decoding layer The vector representations are fed into a linear layer, where the probability mapping from the vector representation to the word vocabulary is achieved according to the following formula:

[0083]

[0084] In the above formula, Let W be the probability mapping from a vector representation to a word vocabulary, where Softmax is the softmax activation function. g b is the weight matrix parameter of the linear layer. g The bias parameters for the linear layer; obtain the probability p that words from the relevant working chapters are directly copied from the input document. c And calculate the generation probability p of the t-th word based on the following formula. t :

[0085]

[0086] In the above formula, z t represents the weight learning parameters. To alleviate the Out-Of-Vocabulary (OOV) problem during decoding (words in the summary are not in the vocabulary), this embodiment employs a Copy mechanism (for implementation details, refer to "See A, Liu PJ, Manning CD. Get To The Point: Summarization with Pointer-Generator Networks [C] / / Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017: 1073-1083.") to model the probability that words in the summary are directly copied from the input document, i.e., obtaining the probability p of words in the relevant work section being directly copied from the input document. c .

[0087] The loss function used during training of the related work generation model in step S105 of this embodiment is expressed as follows:

[0088]

[0089] In the above formula, Represents the loss function. This refers to the content of the relevant working chapters obtained by combining the content of the rhetorical planning sequence and the relevant working chapters as classification labels; p t Let be the probability of generating the t-th word. The t-th word represents the content of the relevant working chapter obtained by concatenating and combining the content of the rhetorical planning sequence and the relevant working chapter of the standard, wherein the rhetorical planning sequence is composed of rhetorical plans, each rhetorical plan being a symbolic representation of a corresponding rhetorical function, and the concatenation and combination of the content of the rhetorical planning sequence and the relevant working chapter of the standard includes concatenation using specified concatenation characters. For example... Figure 4 As shown, in this embodiment, each rhetorical plan is represented by a symbol corresponding to a rhetorical function.<Type_1> ~<Type_6> The content splicing and combination of relevant working chapters of the rhetorical planning sequence and standards includes splicing using the specified splicing character "|||". Figure 4 In the middle section, the examples in the relevant working chapters on the left can be mapped to the rhetorical functions in the middle section, and then sequentially mapped to the symbolic representations in the rhetorical planning on the right.<Type_1> ~<Type_6> The content of the relevant working chapters obtained by splicing and combining the content of the relevant working chapters of the rhetoric planning sequence and standards (the expanded content of the relevant working chapters) is as follows: Figure 4 As shown at the bottom. It needs to be explained that... Figure 4 The relevant sections in the documentation are English examples, and their specific content is beyond the scope of this application. To integrate rhetorical planning into decoding, this embodiment maps each rhetorical function to a special character. Then, a sequence of these special characters is used as the rhetorical plan and placed before the standard related work, serving as an extended related work sent to the model's decoding end for training. During model training, this embodiment sets a learning rate warmup, gradually increasing the learning rate in the first N steps, and gradually decreasing it as the number of training steps exceeds N. The total number of training steps for the model is N. S In this embodiment, the intermediate training nodes are saved every S training steps for subsequent validation and testing. Furthermore, the validation set data is used to select the best-performing node from several trained model nodes for testing. During model testing, the model parameters are fixed and not updated. The beam search width for decoding is set to B, meaning that when the model predicts each word, it saves the five candidates with the highest scores, which to some extent avoids the shortcomings of the greedy sampling method and improves the quality of sentence generation. The minimum generated summary length is set to L words. This embodiment also incorporates an N-gram blocking strategy to reduce the repetition rate of N-tuples in the model and decrease summary redundancy.

[0090] To verify the multi-document scientific literature abstract generation method integrating rhetorical structures in this embodiment, the generally accepted evaluation metric ROUGE (details can be found in "Lin CY, Hovy E. Manual and automatic evaluation of summaries[C] / / Proceedings of the ACL-02 Workshop on Automatic Summarization.2002:45-51," published at the Automatic Summarization Workshop of the 2002 Association for Computational Linguistics Annual Meeting, pp. 45-51) is used. ROUGE-1 and ROUGE-2 are used to evaluate the information content of the generated abstract, while ROUGE-L is used to evaluate the fluency of the abstract. Experimental conditions: One workstation with an NVIDIA GeForce TITANX GPU, operating system Ubuntu 16.04, based on the PyTorch platform. The dataset uses three related works to generate standard datasets: Multi_Xscience, TAD, and TAS2. Their training sets contain 30369 / 208255 / 107700 samples, respectively. Each sample includes several target documents and abstracts of their cited references as input, and the standard related works as output. The validation and test sets contain 5066 / 5000 / 5000 and 5093 / 5000 / 5000 samples, respectively. The final test results using ROUGE-1 and ROUGE-2 metrics are shown in Table 1.

[0091] Table 1: Test results using ROUGE-1 and ROUGE-2 indices.

[0092]

[0093] As shown in Table 1, the experimental results of the existing benchmark methods are as follows: the benchmark model's evaluation results on the automatic evaluation metrics ROUGE-1 / 2 / L on the Multi_Xscience, TAD, and TAS2 datasets are 34.11 / 6.76 / 30.63; 31.70 / 6.41 / 29.01; and 28.53 / 4.96 / 25.78, respectively. The results obtained using the multi-document scientific literature abstract generation method that integrates rhetorical structures in this embodiment are as follows: the evaluation results on ROUGE-1 / 2 / L on the three datasets are 36.94 / 8.05 / 32.41; 35.09 / 7.43 / 31.62; and 31.73 / 5.77 / 28.01, all of which are superior to the experimental results of the benchmark methods.

[0094] In summary, the multi-document scientific literature abstract generation method integrating rhetorical structure in this embodiment includes: analyzing the rhetorical structure of relevant working sections of a paper sample to determine the rhetorical structure system of sentences; training a rhetorical structure classifier using labeled data and using the trained classifier to annotate the rhetorical functions of relevant working sections of the target document; and then using a Transformer-based relevant work generation model to integrate rhetorical structure information. Based on the input abstract sections of the target document and the abstract sections of the references cited by the target document, the relevant working sections of the target document are output. By utilizing the rhetorical structure system of sentences to participate in the generation of relevant working sections of the target document, the method can effectively overcome the problems of chaotic rhetorical structure and unclear contextual logic that occur during the abstract generation process, resulting in more accurate and coherent abstracts for multi-document scientific literature. The rhetorical structure classification system of relevant working sections constructed by the multi-document scientific literature abstract generation method integrating rhetorical structure in this embodiment provides a clear insight into the understanding and organization of relevant work content, and provides support for selecting better relevant work generation strategies. This embodiment of the multi-document scientific literature abstract generation method integrates rhetorical structures. Rhetorical structures are introduced at both the encoding and decoding ends for document modeling and abstract generation, resulting in better abstract content selection and more coherent and logically clear abstract output.

[0095] Furthermore, this embodiment also provides a multi-document scientific and technological literature abstract generation system that integrates rhetorical structures, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the multi-document scientific and technological literature abstract generation method that integrates rhetorical structures.

[0096] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program, which is used to be programmed or configured by a microprocessor to execute the multi-document scientific literature abstract generation method with integrated rhetorical structure.

[0097] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create an implementation for the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating multi-document scientific and technological literature abstracts that integrates rhetorical structures, characterized in that, include: S101, Rhetorical structure analysis is performed on the relevant working sections of the paper sample to determine the rhetorical structure system of the sentences, the rhetorical structure system including different rhetorical functions; S102, based on the rhetorical structure system of sentences, the sentences of the relevant working chapters of the paper sample are annotated with rhetorical structure. S103, construct a training dataset using the relevant working chapters of the paper samples obtained by rhetorical structure annotation, and use the training dataset to train a rhetorical structure classifier to establish the mapping relationship between the relevant working chapters and rhetorical functions of the paper. S104, using a trained rhetorical structure classifier to annotate the rhetorical functions of relevant working chapters in the target document; S105, utilizing a Transformer-based relevance generation model, rhetorical structure information is fused, and based on the abstract sections of the target document and the abstract sections of the references cited by the target document, the relevant work sections of the target document are output. The relevance generation model is an encoder-decoder structure consisting of an encoder and a decoder. The encoder includes a text encoder and a graph encoder. The text encoder encodes words in the abstract sections of the target document and the references into word representations, respectively. The graph encoder models the rhetorical structure information of the abstract sections into word representations to obtain graph-enhanced word representations. The rhetorical structure information of the abstract sections is labeled using a rhetorical structure classifier, including five types: background, target, method, result, and others. The decoder decodes the graph-enhanced word representations to obtain a rhetorical planning sequence and generates the content of the relevant work sections of the target document. The rhetorical structure of the sentences determined in step S101 is a two-level rhetorical structure. The first-level rhetorical structure includes some or all of the following six types: description of the topic, citation of references, establishing a connection between existing references and one's own work, description of one's own work, conveying other communicative functions, and types that are difficult to determine. The second-level rhetorical structure consists of the rhetorical functions under the first-level rhetorical structure. Specifically: the rhetorical functions under the description of the topic include two types: general description of the topic and citation of currently known knowledge; the rhetorical functions under the citation of references include general description of the goals of multiple studies, general description of the methods of multiple studies, general description of the results of multiple studies, description of the goals of a single study, and description of the methods of a single study. There are seven types of rhetorical functions under the categories of description, description of the results of a single study, and other citation purposes. Six types of rhetorical functions are categorized under the category of establishing a connection between existing references and one's own work: explaining the relationship between the methods of the references and one's own work, explaining the relationship between the goals of the references and one's own work, explaining the relationship between the goals of the references and one's own work, expressing the shortcomings of the references, explaining the significance of the references, and other comments by the author. Four types of rhetorical functions are categorized under the category of describing one's own work: describing the goals of one's own work, describing the motivation of one's own work, describing the methods of one's own work, and describing the results of one's own work. Two types of rhetorical functions are categorized under the category of conveying other communicative functions: expressing signals of transfer and other functional statements. One type of rhetorical function is difficult to determine.

2. The method for generating multi-document scientific and technological literature abstracts by incorporating rhetorical structures according to claim 1, characterized in that, In step S103, when training the rhetorical structure classifier to establish the mapping relationship between relevant working chapters and rhetorical functions in the paper, the model used for the rhetorical structure classifier is the pre-trained model SciBERT-base, and the loss function used when training the rhetorical structure classifier is the average binary cross-entropy loss function of the gold label.

3. The method for generating multi-document scientific and technological literature abstracts by incorporating rhetorical structures according to claim 1, characterized in that, The text encoder is a Transformer-based encoder, which includes cascaded encoders with identical structures. Each text encoding layer consists of a multi-head pooling layer (MHPool) and a feedforward neural network sublayer (FFN). Each text encoding layer encodes words from the abstract sections of the target document and references into word representations, including: S201, firstly, the words in the target document and references are encoded through word embedding to obtain the word representations of the target document and references as follows: and Where i and j represent the first and second digits, respectively. i The first statement j One word, T and D These represent the target document and the references, respectively. S202, For the obtained word representations, use a multi-head pooling layer (MHPool) to process them to obtain the sentence representations of the target documents and references. and Then, the sentence representation is added to the word representation according to the following formula, and the enhanced word representation is obtained by using the feedforward neural network sublayer FFN as the final word representation: ， ， In the above formula, Enhanced word representation for the abstract sections of the target document. FNN For feedforward neural network sublayers, The word representation of the abstract section of the target document. The statement representation of the target document; Enhanced word representation for the abstract sections of references. The words in the abstract section of the references are used to represent the text. The 'e' indicates that the statement or word representation in the reference is obtained through word embedding.

4. The method for generating multi-document scientific and technological literature abstracts by incorporating rhetorical structures according to claim 3, characterized in that, The graph encoder is a Transformer-based graph encoder, and the step of modeling rhetorical structure information into word representations to obtain graph-enhanced word representations includes: S301, Initialize the global nodes for constructing the rhetorical structure graph, which consists of nodes and edges, with the global nodes serving as the root nodes; S302, initialize the nodes in the rhetoric structure graph except for the global node. The nodes except for the global node include words, rhetoric functions, target documents, and references. Among them, word nodes are leaf nodes and are randomly initialized. The representation of the rhetoric function is obtained by average pooling of the representations of all statements under the rhetoric function. The representation of the target document is obtained by average pooling of the representations of all statements within the target document. The representation of the reference is obtained by average pooling of the representations of all statements within the reference. Initialize the edges in the rhetoric structure graph. There are five types of edges: undirected edges between rhetoric functions and the words of the sentences they contain, undirected edges between documents and their sentence rhetoric functions, undirected edges between the same type of rhetoric functions in different documents, undirected edges between target documents and references, and undirected edges between global nodes and all other nodes. S303, a Transformer-based graph encoder is used to encode the rhetorical structure graph into graph nodes. The Transformer-based graph encoder includes cascaded, structurally identical components. Each graph encoding layer consists of a multi-head self-attention sublayer (MHSA) and a feedforward neural network sublayer (FFN). Each graph encoding layer models rhetorical structure information into word representations to obtain graph-enhanced word representations. This includes: firstly, using the MHSA sublayer to obtain the target document node representation. The reference node representation is obtained by using a multi-head pooling layer MHPool. Then, the target document node is represented. Adding to the word representation of the target document yields a graph-enhanced word representation of the target document; the reference node representation is then added. Adding the word representation to the references yields the word representation of the references with enhanced graphical information: ， ， In the above formula, Word representations enhanced with section diagram information for the target document's abstract. FNN For feedforward neural network sublayers, Enhanced word representation for the abstract sections of the target document. Represented as a node for the target document; Enhanced word representation for the abstract sections of references. Enhanced word representation for the abstract sections of references. This is used to represent reference nodes.

5. The method for generating multi-document scientific and technological literature abstracts by incorporating rhetorical structures according to claim 4, characterized in that, The decoder is a Transformer-based decoder, which includes cascaded decoders with identical structures. Each decoding layer consists of a multi-head self-attention sublayer (MHSA), a multi-head cross-attention sublayer (MHCA), and a feedforward neural network layer (FFN). Each decoding layer decodes the word representation enhanced with graph information to derive the rhetorical planning sequence and generates the target document. The relevant chapters on this work include: S401, firstly, the decoding state of the t-th word is calculated using the following formula through a multi-head self-attention sublayer (MHSA). Cross-attention features with target and reference words : ， In the above formula, For layer normalization operation, MHCA This indicates a multi-head cross-attention sublayer. This is the current decoding state. For channel stacking operations, Word representations enhanced with section diagram information for the target document's abstract. Word representations enhanced with section diagram information for the abstract of references; S402, decode the state of the t-th word. Cross-attention features with target and reference words After processing by the feedforward neural network layer FFN, the cross-attention features are then compared with those before processing by the feedforward neural network layer FFN. The summation, followed by layer normalization, serves as the cross-attention feature output by the decoding layer. Let the cross-attention feature of the output of the last decoding layer be . Then the cross-attention features output by the last decoding layer The vector representations are fed into a linear layer, where the probability mapping from the vector representation to the word vocabulary is achieved according to the following formula: In the above formula, This represents the probability mapping from vector representations to a word vocabulary. The softmax activation function is used. The parameters are the weight matrix of the linear layer. The bias parameters for the linear layer; the probability that words from the relevant working chapters are copied directly from the input document. And calculate the generation probability of the t-th word based on the following formula. : ， In the above formula, These are the weight learning parameters.

6. The method for generating multi-document scientific and technological literature abstracts by incorporating rhetorical structures according to claim 5, characterized in that, The loss function used during training of the related work generation model in step S105 is expressed as follows: ， In the above formula, Represents the loss function. This refers to the content of the relevant working chapters obtained by splicing and combining the content of the rhetorical planning sequence and the relevant working chapters that serve as classification labels; Let be the probability of generating the t-th word. The t-th word represents the content of the relevant working chapter obtained by splicing and combining the content of the rhetorical planning sequence and the relevant working chapter of the standard, and the rhetorical planning sequence is composed of rhetorical plans, each of which is a symbolic representation of a corresponding rhetorical function. The splicing and combining of the content of the rhetorical planning sequence and the relevant working chapter of the standard includes splicing using specified splicing characters.

7. A multi-document scientific literature abstract generation system integrating rhetorical structures, comprising an interconnected microprocessor and a memory, characterized in that, The microprocessor is programmed or configured to execute the multi-document scientific literature abstract generation method according to any one of claims 1 to 6, which incorporates rhetorical structures.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to execute the multi-document scientific literature abstract generation method with integrated rhetorical structure as described in any one of claims 1 to 6.

Images