Rhetorical structure analysis learning device, system, rhetorical structure analysis learning method, and program

Abstract

To provide a rhetorical structure analysis learning apparatus configured to improve the quality of parameter estimation by effectively utilizing a small quantity of training data in a parameter learning method of rhetorical structure analysis that automatically estimates structure of a document, and provide a learning method, and a program.SOLUTION: In a system comprising a rhetorical structure analysis learning apparatus 1 and a rhetorical structure estimation apparatus, the rhetorical structure analysis learning apparatus which performs learning of rhetorical structure analysis includes: a generation unit 10 which generates pieces of mini patch data constituting a minimum unit using a subtree of a tree structure obtained from a document of training data; and an estimation unit 20 which averages a predetermined number of top parameters, out of the parameters generated by each mini patch data, having high evaluation scores that indicate matching degrees between a subtree obtained by estimating the tree structure based on the parameters and a subtree of a tree structure indicated by ground truth data, to generate a final parameter.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to the field of natural language processing, which processes language using a computer, and in particular to a parameter learning method for rhetorical structure analysis that automatically infers the structure of a document. [Background technology]

[0002] In rhetorical structure analysis, a document is considered as a sequence of text units called Elementary Discourse Units (EDUs), which are smaller than sentences and equivalent to clauses. These are then represented as a tree formed by terminal nodes and EDUs (a tree with a recursive structure in which non-terminal nodes represent span relationships) (see Figure 8). In Figure 8, the terminal node is EDU (e). A non-terminal node represents the nuclear nature (subordinate relationship) of the span it governs, that is, whether it is a nucleus (N) or a satellite (S). A nucleus (N) is a text region that represents more central information, and a satellite (S) is a text region that supplements the nucleus.

[0003] A relationship label is assigned to the edge connecting two non-terminal nodes. When the two non-terminal nodes are a pair of S and N, the edge on the S side is assigned a relationship label such as Elaboration (18 types in total), and the edge on the N side is assigned the default label span. Note that span is the default label assigned to the N side, so it is not considered a relationship label. Also, when the two non-terminal nodes are a pair of N and N, a relationship label such as Same-Unit is assigned to both edges.

[0004] The technique of inferring the rhetorical tree structure for a given sequence of documents, i.e., EDUs, is currently implemented using neural networks. For example, after obtaining span vectors using a pre-trained language model, the neural network performs (1) tree structure inference (deciding whether to join or split spans), (2) kernelness inference (selecting either NS, SN, or NN), and (3) relation label inference (selecting one of 18 relations).

[0005] Note that most current rhetorical structure analysis methods are top-down, that is, they estimate trees by dividing spans. For example, the following explanation of Non-Patent Document 1 will be based on a top-down analyzer.

[0006] Next, Figure 9 shows an overview of a typical top-down analyzer. The top-down analyzer obtains a vector u of spans consisting of consecutive EDUs via the word embedding vector w, and inputs this vector u into a feedforward neural network to obtain a vector h. The top-down analyzer then inputs vector u into a bi-fin layer to estimate the tree structure, kernelity, and relation labels.

[0007] In this case, the span consisting of the i-th EDU to the j-th EDU is

[0008]

number

[0009]

number

[0010]

number

[0011]

number

[0012]

number

[0013]

number

[0014]

number

[0015]

number

[0016]

number

[0017] Finally, the kth EDU that divides the span consisting of the ith EDU and the jth EDU is determined using the following (Equation 5).

[0018]

number

[0019]

number

[0020]

number

[0021]

number

[0022]

number

[0023]

number

[0024] [Non-Patent Document 1] Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-down rst parsing utilizing granularity levels in documents. In Proceedings of the 2020 Conference on Artificial Intelligence for the American, pages 8099-8106 Summary of the Invention [Problem to be solved by the invention]

[0025] Generally, neural network learning, i.e., parameter estimation, requires a sufficient amount of training data. However, annotation for rhetorical structure analysis is costly, making it difficult to prepare large-scale training data.

[0026] The present invention has been made in view of the above points, and has as its object to improve the quality of parameter estimation by effectively utilizing a small amount of training data. [Means for solving the problem]

[0027] In order to solve the above problem, the invention according to claim 1 is a method for analyzing rhetorical structures. parameters study do This rhetorical structure analysis learning device has: a generation unit that generates each mini-batch data that constitutes the smallest unit by a subtree in a tree structure obtained from documents of training data; and an estimation unit that generates final parameters by averaging a predetermined number of parameters that have the highest evaluation scores, among the parameters generated by each mini-batch data, and indicate the degree of agreement between the subtree obtained by estimating the tree structure based on the respective parameters and the subtree of the tree structure indicated by the correct answer data. [Effects of the Invention]

[0028] As described above, according to the present invention, the smallest unit of training data is constructed not as the entire tree structure obtained from a document, but as a subtree in the tree structure, thereby increasing the number of parameter updates, and further averaging the parameters saved for each epoch during learning, thereby achieving the effect of making effective use of a smaller amount of training data than in the past and improving the quality of parameter estimation. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a diagram illustrating the overall configuration of a system constructed using a rhetorical structure analysis learning device and a rhetorical structure estimation device according to an embodiment. [Figure 2] 1 is a hardware configuration diagram of a rhetorical structure analysis learning device according to an embodiment. [Figure 3] 1 is a functional configuration diagram of a rhetorical structure analysis learning device according to an embodiment. [Figure 4] FIG. 2 is a detailed functional block diagram of a generation unit according to the embodiment. [Figure 5]FIG. 3 is a detailed functional block diagram of an estimation unit according to the embodiment. [Figure 6] 10 is a flowchart showing the processing of a generation unit of the rhetorical structure analysis learning device. [Figure 7] 10 is a flowchart showing the processing of an estimation unit of the rhetorical structure analysis learning device. [Figure 8] A diagram showing a tree with a recursive structure in rhetorical structure analysis. [Figure 9] FIG. 1 shows an overview of a general top-down analyzer. DETAILED DESCRIPTION OF THE INVENTION

[0030] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0031] [System configuration of the embodiment] First, an outline of the configuration of the system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of a system constructed using a rhetorical structure analysis learning device and a rhetorical structure estimation device according to this embodiment.

[0032] As shown in FIG. 1, the system is constructed by a rhetorical structure analysis learning device 1 and a rhetorical structure estimation device 2.

[0033] Furthermore, the rhetorical structure analysis learning device 1 and the rhetorical structure estimation device 2 can communicate with each other via a communication network such as the Internet or a LAN (Local Area Network). The communication network may be connected wirelessly or by wire.

[0034] The rhetorical structure analysis learning device 1 and the rhetorical structure estimation device 2 are configured by one or more computers.

[0035] The rhetorical structure analysis learning device 1 transmits each parameter generated by dividing the training data to the rhetorical structure estimation device 2. The rhetorical structure analysis learning device 1 transmits correct answer data in advance to the rhetorical structure estimation device 2. The correct answer data includes multiple pairs of text data representing documents and correct answer trees corresponding to this text data, and is called "development data" in the field of rhetorical structure analysis.

[0036] The rhetorical structure estimation device 2 generates an evaluation score indicating the degree of match between the subtree obtained by estimating the tree structure based on the parameters acquired from the rhetorical structure analysis learning device 1 and the subtree of the tree structure indicated by the correct answer data, and sends the evaluation score to the rhetorical structure analysis learning device 1. The evaluation score is indicated as a value between 0 and 1, with "0" indicating a complete mismatch and "1" indicating a complete match, for example.

[0037] [Hardware configuration of the rhetorical structure analysis learning device] Next, the hardware configuration of the rhetorical structure analysis learning device 1 will be described using Fig. 2. Fig. 2 is a hardware configuration diagram of the rhetorical structure analysis learning device according to the embodiment. Note that the hardware configuration of the rhetorical structure estimation device 2 is similar to that of the rhetorical structure analysis learning device 1, and therefore a description thereof will be omitted.

[0038] 2, the rhetorical structure analysis learning device 1 includes a processor 101, a memory 102, an auxiliary storage device 103, a connection device 104, a communication device 105, and a drive device 106. The hardware components constituting the rhetorical structure analysis learning device 1 are interconnected via a bus 107.

[0039] The processor 101 serves as a control unit that controls the entire rhetorical structure analysis learning device 1, and includes various computing devices such as a CPU (Central Processing Unit). The processor 101 reads and executes various programs onto the memory 102. The processor 101 may also include a GPGPU (General-purpose computing on graphics processing units).

[0040] The memory 102 has a main storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 101 and the memory 102 form a so-called computer, and the processor 101 executes various programs read onto the memory 102, thereby enabling the computer to realize various functions.

[0041] The auxiliary storage device 103 stores various programs and various information used when the processor 101 executes the various programs.

[0042] The connection device 104 is a connection device that connects an external device (for example, a display device 110, an operation device 111) to the rhetorical structure analysis learning device 1.

[0043] The communication device 105 is a communication device for transmitting and receiving various types of information to and from other devices.

[0044] The drive device 106 is a device for loading a (non-transitory) recording medium 130. The recording medium 130 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, or a magneto-optical disk. The recording medium 130 may also include semiconductor memory that records information electrically, such as a ROM (Read Only Memory) or flash memory.

[0045] The various programs to be installed in the auxiliary storage device 103 are installed, for example, by setting the distributed recording medium 130 in the drive device 106 and reading the various programs recorded on the recording medium 130 by the drive device 106. Alternatively, the various programs to be installed in the auxiliary storage device 103 may be installed by being downloaded from a network via the communication device 105.

[0046] [Functional configuration of the rhetorical structure analysis learning device] Next, the functional configuration of the rhetorical structure analysis learning device 1 will be described with reference to Figures 3 to 5. Figure 3 is a functional configuration diagram of the rhetorical structure analysis learning device according to the embodiment.

[0047] 3, the rhetorical structure analysis learning device 1 has an input unit 5, a generation unit 10, an estimation unit 20, and an output unit 24. Each of these units is a function realized by instructions from the processor 101 in FIG. 2 based on a program.

[0048] Furthermore, the memory 102 or the auxiliary storage device 103 in FIG. 2 stores a pre-trained language model m.

[0049] The input unit 5 inputs data from the user and data sent from the rhetorical structure estimation device 2. The output unit 24 transmits the correct answer data and each parameter to the rhetorical structure estimation device 2.

[0050] <Functional configuration of the generation unit> The generation unit 10 receives raw training data and generates mini-batch data, which is vector data that can be used for training a neural network. To this end, the generation unit 10 includes a training example generation unit 11, a span vector generation unit 12, and a mini-batch generation unit 13.

[0051] Of these, the learning example generation unit 11 receives (raw) training data from the input unit 5 and generates span division example data by converting it into span division example data with the span division of parent-child nodes in a tree structure as the smallest unit.

[0052] The span vector generation unit 12 generates vectors by converting the span division example data generated by the training example generation unit 11 into vectors.

[0053] The mini-batch generation unit 13 generates training data (mini-batch data) by dividing the vectors generated by the span vector generation unit 12 into a predetermined number of random p (N / p) blocks.

[0054] The processes of the training example generation unit 11, span vector generation unit 12, and mini-batch generation unit 13 will be described in detail later.

[0055] <Functional configuration of the estimation unit> The estimation unit 20 receives training data divided into N / p blocks as mini-batch data from the generation unit 10, and generates final parameters by optimizing (updating) the parameters until a specified maximum number of epochs is reached (or until the evaluation score converges to a certain value). Here, parameters are saved for each epoch, and the final parameters, which are the ultimate goal, are generated by averaging the K parameters with the highest evaluation scores. For this purpose, the estimation unit 20 includes a parameter optimization unit 21, a correct answer data evaluation unit 22, and a parameter averaging unit 23.

[0056] Of these, the parameter optimization unit 21 receives mini-batch data generated by dividing raw training data from the mini-batch generation unit 13, and optimizes (updates) the parameters until a specified maximum number of epochs is reached (or until the evaluation score converges to a certain value). Here, an epoch means that all N / p blocks are used to update the parameters, that is, N / p update repetitions are performed. Existing techniques such as AdamW are used to update the parameters.

[0057] The correct data evaluation unit 22 transmits correct data to the rhetorical structure estimation device 2 in advance via the output unit 24. Furthermore, the correct data evaluation unit 22 transmits the parameters acquired from the parameter optimization unit 21 to the rhetorical structure estimation device 2 via the output unit 24 each time the parameters are acquired. Furthermore, the correct data evaluation unit 22 receives the parameter evaluation score for each of the transmitted parameters from the rhetorical structure estimation device 2 via the input unit 5.

[0058] The parameter averaging unit 23 acquires pairs of each epoch number and each evaluation score from the supervised data evaluation unit 22, sorts the tuples of the epoch number and the evaluation score in descending order of the evaluation score, and determines (identifies) the top K evaluation scores. The parameter averaging unit 23 also acquires a set of K parameters stored in the parameter optimization unit 21. Furthermore, the parameter averaging unit 23 averages the set of K parameters to generate final parameters.

[0059] The processing of the parameter optimization unit 21, the correct data evaluation unit 22, and the parameter averaging unit 23 will be explained in detail later.

[0060] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described in detail with reference to FIGS.

[0061] <Generation process> First, the processing of the generation unit 10 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the processing of the generation unit of the rhetorical structure analysis learning device.

[0062] 6, the learning example generation unit 11 generates span division example data (S11) that represents the parent-child relationships of span division as shown below, based on all raw training data acquired via the input unit 5. For example, taking the tree shown in FIG. 8 as an example, the span division example data is shown as follows: · SPN(e1,e6) → N:span:SPN(e1,e3),S:Elab:SPN(e4,e6) · SPN(e1,e3) → N:span:SPN(e1,e2),S:Elab:SPN(e3,e3) · SPN(e1,e2) → S:Attrib:SPN(e1,e1),N:span:SPN(e2,e2) · SPN(e4,e6) → N:Same-Unit:SPN(e4,e5),N:Same-Unit:SPN(e6,e6) · SPN(e4,e5) → N:span:SPN(e4,e4),S:Elab:SPN(e6,e5) In addition, SPN(e i ,e j ) represents the span consisting of the i-th to j-th EDUs. The label span is the default label assigned to the N-side in the case of NS and SN kernels. Each of the above examples (each row) is the smallest unit that makes up a mini-batch.

[0063] Next, the span vector generation unit 12 receives from the training example generation unit 11 a span SPN(e i ,e j ) and converts the span division example data into a vector representation to generate a vector u (S12). The vector u is the leftmost word w of the i-th EDU. b(i) and the rightmost word w of the jth EDU e(j) The embedding vector of a word w is obtained using a pre-trained language model m. The dimension of the vector depends on the pre-trained language model m used.

[0064] Next, the mini-batch generation unit 13 receives all span division example data expressed as vectors from the span vector generation unit 12 and generates mini-batch data by dividing the data into random blocks of p pieces (S13). Here, if the number of span division examples obtained in the entire document is N, then N / p blocks are created.

[0065] Note that, as in the past, when the smallest unit of mini-batch construction is a tree of all documents, if the number of documents is M, even if the batch size is 1, the total number of blocks is only M / 1. This depends on the value of p, but since M < N, the number of blocks in a mini-batch constructed using documents as the smallest unit is smaller than N / p. Therefore, the number of iterations per epoch required for parameter updates also becomes smaller, which may result in poor updates. Unlike the conventional approach, which treats the smallest unit of training data as a tree for all documents, i.e., all divisions from the span representing the entire document to the division into single EDUs (leaves), as a single unit, in this embodiment, the smallest unit is changed to a subtree, resulting in a larger number of training data than the number of documents. Therefore, selecting a small value for p increases the number of blocks, which increases the number of parameter updates, and therefore machine learning is expected to be successful.

[0066] The above example is for constructing a tree in a top-down manner, but in a bottom-up manner, an example of span joining rather than span division can be used.

[0067] As a result of the above, the generating unit 10 generates training data divided into N / p blocks as mini-batch data.

[0068] <Processing of the estimation part> Next, the processing of the estimation unit 20 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the processing of the estimation unit of the rhetorical structure analysis learning device.

[0069] First, the correct answer data evaluation unit 22 transmits the correct answer data to the rhetorical structure estimation device 2 in advance via the output unit 24 (S21).

[0070] Next, the parameter optimization unit 21 initializes the parameters (S22). The parameter optimization unit 21 updates (optimizes) the parameters based on the training data divided into N / p blocks so as to minimize the loss score (Loss), and stores the parameters in the memory 102 or the like for each epoch (S23). Here, the loss score (Loss), which is the objective function for updating the parameters, is used to minimize the following (Equation 8).

[0071]

number

[0072] Next, the supervised data evaluation unit 22 determines whether the specified maximum number of epochs has been reached or whether the evaluation score has converged to a certain value (S26). The determination of convergence is made when the difference between the evaluation score of the current epoch and the evaluation score of the previous epoch reaches a certain value.

[0073]

number

[0074] If the specified maximum number of epochs has not been reached, or if the evaluation score has not converged to a certain value (S26; NO), the process returns to S23.

[0075] On the other hand, if the specified maximum number of epochs (for example, 10) is reached or the evaluation scores have converged to a certain value (S26; YES), the parameter averaging unit 23 acquires pairs of each epoch number and each evaluation score from the supervised data evaluation unit 22 (S27).The parameter averaging unit 23 then sorts the tuples of the epoch number and evaluation score in descending order of the evaluation score, and determines the top K evaluation scores (S28).

[0076] Next, the parameter averaging unit 23 acquires a set of K parameters stored in the parameter optimizing unit 21 (S29). Then, the parameter averaging unit 23 averages the set of K parameters by adding up the values of the parameters related to the top K evaluation scores and dividing by K to generate final parameters (S30).

[0077] [Major Effects of the Embodiments] As described above, according to this embodiment, when mini-batch training a neural network, the basic unit of the mini-batch is not the entire tree obtained from a document, but a subtree of that tree, i.e., a parent-child tree representing span division, and by averaging K parameters with high evaluation scores in the ground truth data, it is possible to expect improved performance in parameter estimation. This has the effect of making effective use of a small amount of training data and improving the quality of parameter estimation.

[0078] 〔supplement〕 The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

[0079] The input unit 5, generation unit 10, estimation unit 20, and output unit 24 can be realized by a computer and a program as described above, but this program can also be recorded on a (non-temporary) recording medium and provided, or provided via a network such as the Internet. [Explanation of symbols]

[0080] 1 Rhetorical Structure Analysis Learning Device 2 Rhetorical structure estimation device 5 Input section 10 Generation part 11 Learning example generation unit 12 Span Vector Generator 13 Mini-batch generation part 20 Estimation part 21 Parameter Optimization Unit 22 Correct Data Evaluation Unit 23 Parameter averaging section 24 Output section

Claims

1. A rhetorical structure analysis learning device that learns parameters of rhetorical structure analysis, comprising: a generation unit that generates mini-batch data each constituting a minimum unit by a subtree in a tree structure obtained from documents of training data; an estimation unit that generates final parameters by averaging a predetermined number of parameters that have the highest evaluation scores indicating the degree of agreement between a subtree obtained by estimating a tree structure based on each parameter and a subtree of the tree structure indicated by the correct answer data, among the parameters generated by each mini-batch data; A rhetorical structure analysis learning device having the above.

2. The rhetorical structure analysis learning device according to claim 1 , wherein the generating unit generates each of the mini-batch data by span division, with a minimum unit being a span division of a parent-child node of the tree structure as the subtree.

3. The estimation unit a parameter optimization unit that optimizes and stores the parameters based on the mini-batch data; a correct answer data evaluation unit that performs processing to transmit the parameters acquired from the parameter optimization unit to a rhetorical structure estimation device that calculates the evaluation score indicating the degree of agreement between a subtree obtained by estimating a tree structure based on the parameters and a subtree of the tree structure indicated by the correct answer data, and acquires the evaluation scores transmitted by the rhetorical structure estimation device in response to the processing; a parameter averaging unit that determines a predetermined number of the top evaluation scores from the evaluation scores, obtains a set of parameters related to the predetermined number of top evaluation scores from the parameters stored by the parameter optimization unit, and averages the set of parameters to generate the final parameters; 3. The rhetorical structure analysis learning device according to claim 1, further comprising:

4. The rhetorical structure analysis learning device of claim 3, wherein the correct answer data evaluation unit terminates the process of transmitting the parameters to the rhetorical structure estimation device when the number of times the evaluation score is obtained reaches a maximum number of epochs or when the evaluation score converges to a certain value.

5. The rhetorical structure analysis learning device according to any one of claims 1 to 4, a rhetorical structure estimation device that calculates an evaluation score indicating the degree of agreement between a subtree obtained by estimating a tree structure based on each parameter and a subtree of the tree structure indicated by the correct answer data; A system having:

6. A rhetorical structure analysis learning method executed by a rhetorical structure analysis learning device that learns parameters of rhetorical structure analysis, comprising: Generate each mini-batch data consisting of a minimum unit by a subtree in a tree structure obtained from the documents of the training data, Among the parameters generated from each mini-batch data, a predetermined number of parameters with the highest evaluation scores indicating the matching rate between a subtree obtained by estimating a tree structure based on the parameters and a subtree of the tree structure indicated by the correct answer data are averaged to generate final parameters. How to study rhetorical structure analysis.

7. A program causing a computer to execute the method according to claim 6.

Images