Rhetoric structure analyzer, rhetoric structure analysis method and program
By employing a decoder-only LLM to generate prompts and construct rhetorical structure trees, the device enhances the performance of rhetorical structure analysis, addressing the limitations of encoder-only models and achieving superior results.
Patent Information
- Application Number
- JP2024016946
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-02-07
- Publication Date
- 2025-08-20
AI Technical Summary
Current rhetorical structure analysis technologies using encoder-only pre-trained language models struggle to achieve sufficient performance due to the significant differences in mechanisms between these models and decoder-only pre-trained large-scale language models (LLMs), despite LLMs having many more parameters which typically lead to better performance.
A rhetorical structure analysis device that utilizes a decoder-only pre-trained large-scale language model (LLM) to generate prompts for tree structure, kernel, and discourse relation estimation, replacing the conventional feature extraction unit, and employs a stack and queue to construct rhetorical structure trees through shift-and-reduce operations.
This approach significantly improves the performance of rhetorical structure analysis by leveraging the capabilities of LLMs, achieving higher accuracy and efficiency compared to conventional methods.
Smart Images
Figure 2025121515000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a rhetorical structure analysis device, a rhetorical structure analysis method, and a program. [Background technology]
[0002] [Rhetorical structure tree] In rhetorical structure analysis, a document is considered as a sequence of text units called EDUs (Elementary Discourse Units), which are smaller than sentences and equivalent to clauses, and is represented as a rhetorical structure tree with a recursive structure in which terminal nodes represent these and non-terminal nodes represent the span relationships formed by the EDUs (see Figure 1).The contents of EDUs e1 to e6 (from RST Discourse Treebank, WSJ 1100) in Figure 1 are as follows: e1:[Westinghouse Electric Corp. said] e2:[it will buy Shaw-Walker Co.] e3:[Terms weren't disclosed.] e4:[Shaw-Walker,] e5:[based in Muskegon, Mich.,] e6:[makes metal files and desks, and seating and office systems furniture.] Although trees are typically represented as multiary trees, they can also be represented as equivalent complete bi-ary trees. In Figure 1, the terminal nodes are EDUs (e), and non-terminal nodes represent the nuclearity of the left and right spans they govern, i.e., whether they are kernels (N) or satellites (S). If a node is NS, the left span is N and the right span is S. If a node is SN, the right span is N and the left span is S. In the NS and SN cases, a discourse relation label from S to N is assigned, while in the NN case, an equivalent discourse relation label is assigned, representing the parallel relationship between the two spans. In the example in Figure 1, e1 is a satellite of e2, and the discourse relation label between them is Attribution. Also, e3 is a satellite of the span consisting of e1 and e2, and the discourse relation label between them is Elaboration.
[0003] [Rhetorical structure analysis] The current technique for inferring a rhetorical tree structure given a sequence of documents, i.e., EDUs, is realized using neural networks. After obtaining span vectors using an encoder-only pre-trained language model, the neural network performs (1) tree structure inference (deciding whether to merge or split spans), (2) kernel inference (selecting NS, SN, or NN), and (3) relation label inference.
[0004] In other words, it consists of a feature extraction unit that converts text spans (EDU sequences) into vectors, and a classification unit that receives the vectors output by the feature extraction unit and performs tree structure estimation, kernel estimation, and discourse relation label estimation (see Figure 2). act is tree structure estimation, FFN nuc is nuclear estimation, FFN rel serves as the discourse relation label.
[0005] Figure 2 shows the shift-and-reduce method for rhetorical structure analysis. In the shift-and-reduce method, a tree is constructed by repeating the following shift and reduce operations (actions) using a stack and a queue. Shift: Push the EDU at the top of the queue that stores the given EDU sequence onto the stack. Reduce: Combine the top two elements of the stack (either single EDUs or trees) to create a tree (making the two elements siblings) and push it onto the stack.
[0006] In addition, kernelity and discourse relation labels are also estimated during reduction. [Prior art documents] [Non-patent literature]
[0007] [Non-Patent Document 1] Naoki Kobayashi and Tsutomu Hirao and Hidetaka Kamigaito and Manabu Okumura and Masaaki Nagata, "A Simple and Strong Baseline for End-to-End Neural RST-style Discourse Parsing", Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6754 - 6766 December Summary of the Invention [Problem to be solved by the invention]
[0008] The current development of pre-trained language models is shifting focus from encoder-only to decoder-only models, and their scale is accelerating. While most encoder-only pre-trained language models have only a few million parameters, decoder-only pre-trained large-scale language models (hereafter referred to as "LLMs") often have over billions, and some even exceed tens of billions, of parameters. Generally, when using pre-trained models to solve natural language processing tasks, the more parameters there are, the better performance can be expected. However, there are significant differences in the mechanisms between encoder-only pre-trained language models and LLMs, and simply replacing the pre-trained language models in the feature extraction section of existing rhetorical structure analysis technologies with LLMs will not achieve sufficient performance.
[0009] The present invention has been made in view of the above points, and has as its object to improve the performance of rhetorical structure analysis. [Means for solving the problem]
[0010] In order to solve the above problem, the rhetorical structure analysis device has a query unit configured to generate a prompt for rhetorical structure analysis of an input document and input the prompt to an LLM, and a rhetorical structure tree construction unit configured to construct a rhetorical structure tree based on the output from the LLM. [Effects of the Invention]
[0011] The performance of rhetorical structure analysis can be improved. [Brief explanation of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a rhetorical structure tree. [Figure 2] FIG. 10 is a diagram for explaining estimation of a rhetorical tree structure using a neural network. [Figure 3] 1 is a diagram illustrating an example of the hardware configuration of a rhetorical structure analysis device 10 according to an embodiment of the present invention. [Figure 4] 1 is a diagram illustrating an example of a functional configuration of a rhetorical structure analysis device 10 according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing an example of a prompt when deciding on an action. [Figure 6] FIG. 10 shows an example of a prompt for determining a nuclear label. [Figure 7] FIG. 10 illustrates an example of a prompt for a discourse relation label. [Figure 8] FIG. 10 is a diagram showing an example of an output from the LLM 13 in response to a prompt generated in this embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a prompt when deciding on an action in top-down analysis. DETAILED DESCRIPTION OF THE INVENTION
[0013] In this embodiment, to address the above-mentioned issues, the feature extraction unit used in the conventional technology is eliminated, and instead a pre-trained large-scale language model (hereinafter referred to as "LLM") consisting only of a decoder is used. By repeatedly inputting prompts to the LLM, (1) tree structure estimation, (2) kernel estimation, and (3) discourse relation label estimation are performed on the document given as input.
[0014] An embodiment of the present invention will be described below with reference to the drawings. Fig. 3 is a diagram showing an example of the hardware configuration of a rhetorical structure analysis device 10 according to an embodiment of the present invention. The rhetorical structure analysis device 10 in Fig. 3 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.
[0015] The program that realizes the processing in the rhetorical structure analysis device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.
[0016] When an instruction to start the program is received, the memory device 103 reads out and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the rhetorical structure analysis device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.
[0017] Fig. 4 is a diagram showing an example of the functional configuration of a rhetorical structure analysis device 10 according to an embodiment of the present invention. In Fig. 4, the rhetorical structure analysis device 10 includes a rhetorical structure tree construction unit 11, a query unit 12, and an LLM 13. Each of these units is realized by a process executed by a processor 104 of one or more programs installed in the rhetorical structure analysis device 10.
[0018] The query unit 12 repeatedly generates prompts for the LLM 13 and inputs the prompts, and the rhetorical structure tree construction unit 11 constructs a rhetorical structure tree based on the output from the LLM 13. The LLM 13 is the LLM described above.
[0019] Here, the shift-and-reduce method is used as an example. Therefore, in this embodiment, the stack and queue are examples of data structures that store elements corresponding to the EDUs that make up the input document or elements corresponding to a tree structure based on the EDUs for rhetorical structure analysis. However, as will be described later, this embodiment is not limited to the shift-and-reduce method, and can be widely applied to general upward and downward analysis methods.
[0020] The rhetorical structure tree construction unit 11 has a stack and a queue, and passes their contents to the query unit 12. The query unit 12 generates a prompt based on the contents and inputs the prompt to the LLM 13. The rhetorical structure tree construction unit 11 receives an action (either a "shift" or "reduce" operation) output by the LLM 13 and updates the stack and queue according to the action. Note that "shift" indicates that a shift operation should be performed, and "reduce" indicates that a reduce operation should be performed. The stack is initially empty, and the queue is initially stored with all EDUs (Elementary Discourse Units) that make up the input document. An EDU is a text unit equivalent to a clause, smaller than a sentence. In other words, the input document is formed by a sequence of text units equivalent to EDUs. Note that the stack cannot be reduced unless two or more pieces of data are stored on it, so the default is to shift twice from the initial state. The final state is an empty queue with only one piece of data stored in the stack.
[0021] Furthermore, if the action is "reduce," the rhetorical structure tree construction unit 11 passes the state (contents) of the stack and queue to the query unit 12. The query unit 12 generates a prompt and inputs the prompt to the LLM 13. The rhetorical structure tree construction unit 11 receives the output (coreness label) of the LLM 13 and passes the received coreness label in addition to the state of the stack and queue to the query unit 12. The query unit 12 generates a prompt and inputs the prompt to the LLM 13. The rhetorical structure tree construction unit 11 receives the output (discourse relation label) of the LLM 13.
[0022] The rhetorical structure tree construction unit 11 and the query unit 12 will be described in detail below.
[0023] [Rhetorical structure tree construction part 11] The rhetorical structure tree construction unit 11 passes the state of the stack and queue to the inquiry unit 12, and constructs a tree structure and labels the kernelity and discourse relations according to the results obtained through the LLM 13.
[0024] If the output of LLM 13 is "shift," the rhetorical structure tree builder 11 places the EDU at the top of the queue on top of the stack and proceeds to the next step. If the output of LLM 13 is "reduce," the rhetorical structure tree builder 11 generates a tree for the top two elements of the stack (subtrees, or more precisely, text spans governed by subtrees, or single EDUs) with the second element as the left child and the top element as the right child, and places it on top of the stack. At the same time, it determines the left and right kernelity and discourse relation labels of the generated tree and proceeds to the next step.
[0025] For example, if the second element of the stack is a subtree with e2 as its left child and e3 as its right child, and the top element of the stack is e4, the rhetorical structure tree construction unit 11 performs a reduction operation to create a subtree with e2 and e3 as its left child and e4 as its right child at the top of the stack. At this time, since the output from the LLM 13 is "reduction," the rhetorical structure tree construction unit 11 passes the second element of the stack and the top element of the stack immediately before the reduction operation to the query unit 12, and obtains the nuclearity labels for each element from the LLM 13. Next, the rhetorical structure tree construction unit 11 passes the second element of the stack and the top element of the stack immediately before the reduction operation, as well as the nuclearity labels obtained for each element, to the query unit 12, and obtains the discourse relation labels from the LLM 13. The rhetorical structure tree construction unit 11 assigns these labels to the constructed subtree.
[0026] [Inquiry Section 12] The query unit 12 generates a prompt based on the information received from the rhetorical structure tree builder 11 and inputs it to the LLM 13 as follows: The query unit 12 passes the result (output from the LLM 13) to the rhetorical structure tree builder 11.
[0027] <When deciding on an action> The case of determining an action refers to the case where the LLM 13 outputs an action, and is the case where the query unit 12 receives the state of the stack and queue from the rhetorical structure tree construction unit 11.
[0028] In this case, the query unit 12 generates a prompt to query operations on the stack and queue, as shown in Figure 5, using the text corresponding to the second element of the stack and the text corresponding to the top element of the stack, and passes (inputs) the prompt to the LLM 13.
[0029] In FIG. 5, the part enclosed in {} is replaced with the data represented by the character string within the {} based on the state of the stack and queue.
[0030] <In the case of determining nuclear label> The case of determining the nuclearity label refers to the case where the LLM 13 outputs the nuclearity of the second element in the stack just before the reduction operation and the element at the top of the stack, and the case where the query unit 12 receives the second element in the stack just before the reduction operation and the element at the top of the stack from the rhetorical structure tree construction unit 11.
[0031] In this case, the query unit 12 generates a prompt to query the coreness of the element that is the target of the reduction operation, as shown in Figure 6, using the text corresponding to the second element in the stack and the text corresponding to the element at the top of the stack, and passes (inputs) the prompt to the LLM 13.
[0032] In FIG. 6, the part enclosed in {} is replaced with the data represented by the character string within the {} based on the state of the stack and queue.
[0033] <In the case of determining discourse relation labels> The case of discourse relation labels refers to the case where LLM 13 outputs discourse relation labels for the second element and the top element of the stack just before performing the reduction operation, and where query unit 12 receives from rhetorical structure tree construction unit 11 the second element and the top element of the stack just before performing the reduction operation, as well as the nuclearity labels of each element.
[0034] In this case, the query unit 12 generates a prompt to query the discourse relationship of the element associated with the core label, as shown in Figure 7, using the text corresponding to the second element in the stack, the text corresponding to the element at the top of the stack, and the core label for each element, and passes (inputs) the prompt to the LLM 13.
[0035] In Figure 7, the part enclosed in {} is replaced with the data represented by the character string in {} based on the state of the stack and queue, or the kernel label. Also, "relation r" (r is 1 to n) is replaced with n candidates for discourse relation labels.
[0036] A fine-tuned LLM13 is used to ensure appropriate output for the above prompts. When fine-tuning the LLM13, models for determining action, nuclear label, and discourse relation labels are trained using separate LLM13s. In the case of zero-shot or few-shot training, a single LLM13 model is used.
[0037] Figure 8 shows an example of output from the LLM 13 in response to a prompt generated in this embodiment. Figure 8 corresponds to the case where the LLM 13 outputs a reduction when a subtree consisting of e1 and e2 is stored second in the stack, e3 is stored at the top of the stack, and e4 is stored at the beginning of the queue. Figure 8 also shows an example of updating the rhetorical structure tree in response to the output from the LLM 13. In Figure 8, the LLM act is an LLM13 fine-tuned for action decisions. nuc is LLM13 fine-tuned for the determination of nuclear labels. rel is LLM13 fine-tuned for determining discourse relation labels.
[0038] This embodiment can be applied not only to the shift-and-reduce method but also to top-down parsing. Top-down parsing can be achieved by taking a sequence of EDUs for the entire document and recursively dividing it into two parts. Therefore, the prompt for determining the shift-and-reduce action (the prompt for determining the action) can be changed to the prompt shown in FIG. 9.
[0039] As described above, according to this embodiment, the following effects can be obtained.
[0040] Basically, the more parameters a language model has, the higher its performance in subsequent tasks. Previous rhetorical structure analysis technologies have utilized encoder-only pre-trained language models. However, currently, encoder-only pre-trained language models have only a few million parameters, while LLMs can have more than tens of billions. This embodiment is a rhetorical structure analysis technology that takes advantage of the capabilities of LLMs, enabling higher performance analysis than conventional technologies. In other words, the performance of rhetorical structure analysis can be improved.
[0041] Since the present invention is a method for constructing a constituent tree, it can be applied not only to rhetorical structure analysis but also to so-called syntactic analysis (phrase structure analysis).
[0042] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]
[0043] 10 Rhetorical structure analysis device 11 Rhetorical structure tree construction part 12 Inquiry Department 13 LLM 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory Device 104 processors 105 Interface Device B Bus
Claims
1. a query unit configured to generate prompts for rhetorical analysis of an input document and input the prompts to the LLM; a rhetorical structure tree builder configured to build a rhetorical structure tree based on output from the LLM; A rhetorical structure analysis device comprising:
2. the query unit is configured to generate a first prompt for querying an operation on a data structure that stores, for rhetorical structure analysis, an element corresponding to a text unit in a sequence of text units forming the document or an element corresponding to a tree structure based on the text units; a second prompt for querying the coreness of an element that is the target of the operation in the data structure; or a third prompt for querying the discourse relationship of an element related to the coreness in the data structure; The rhetorical structure tree construction unit is configured to perform an operation on the data structure according to an output from the LLM that received the first prompt, and to provide the kernel output from the LLM that received the second prompt and the discourse relation output from the LLM that received the third prompt to the rhetorical structure tree.
2. The rhetorical structure analysis device according to claim 1, wherein:
3. the data structure is a queue and stack for rhetorical structure analysis by shift-and-reduce; The inquiry unit is configured to generate the first prompt based on the queue and the stack, and when an operation output from an LLM that inputs the first prompt indicates a reduction, generate the second prompt based on the stack immediately before the reduction is performed, and generate the third prompt based on an operation output from an LLM that inputs the second prompt and the stack immediately before the reduction is performed.
3. The rhetorical structure analysis device according to claim 2.
4. a query procedure that generates prompts for rhetorical analysis of an input document and inputs the prompts to the LLM; a rhetorical structure tree construction procedure that constructs a rhetorical structure tree based on the output from the LLM; A rhetorical structure analysis method characterized in that the above is executed by a computer.
5. a query procedure that generates prompts for rhetorical analysis of an input document and inputs the prompts to the LLM; a rhetorical structure tree construction procedure that constructs a rhetorical structure tree based on the output from the LLM; A program characterized by causing a computer to execute the above.