Long text division device, system, long text division method, and program

Abstract

The purpose of the present disclosure is to make it possible to: obtain high-quality training data when splitting a long sentence having a complex structure into a plurality of simple sentences semantically equivalent to the long sentence; and improve the quality of splitting a long sentence into simple sentences.　Therefore, the present disclosure pertains to a long sentence splitting apparatus for splitting a long sentence into a plurality of simple sentences equivalent to the long sentence. The long sentence splitting apparatus has: a training data filtering instrument for outputting predetermined training data obtained by deleting an erroneous input / output pair through filtering training data by the use of an entailment relationship recognizer; an encoder for encoding the predetermined training data outputted by the training data filtering instrument; and a decoder for decoding the predetermined training data encoded by the encoder.

Description

[Technical Field]

[0001] The present disclosure relates to a natural language processing technology that processes language using a computer, and in particular to a method for dividing a long sentence with a complex structure into multiple semantically equivalent simple sentences. [Background technology]

[0002] The technique of dividing a long sentence with a complex structure into multiple simple sentences that are semantically equivalent is called split and rephrase (Non-Patent Document 1).

[0003] An example of dividing a long sentence into multiple simple sentences, as described in Non-Patent Document 1, is shown below. (A) John Clancy is a labor politician who leads Birmingham, where architect John Madin, who designed 103 Colmore Row, was born. (B) Labor politician, John Clancy is the leader of Birmingham. (C) John Madin was born in this city. (D) He was the architect of 103 Colmore Row. In this example, a long sentence (A) with a complex structure is divided into multiple simple sentences (B)-(D) with equivalent meaning. This technology is mainly realized by an encoder-decoder model using neural networks.

[0004] In fact, in Non-Patent Document 1, the encoder-decoder model receives a complex long sentence (word sequence) as input, the encoder vectorizes it, and the decoder outputs a sequence of simple sentences as word sequences from the vectors. In this case, a special symbol is used to separate the simple sentences.

[0005]

number

[0006] Current encoder-decoder models have improved in performance by moving from Long Short Term Memory (LSTM)-based to transformer-based methods. However, the output of these models (simple sentence sequences in this case) has problems with faithfulness to the long sentences they input. In other words, the content of the output simple sentences may be inconsistent with the content of the long sentences they input.

[0007] In the split and rephrase task, there are two possible reasons why the content of the output sentence may be inconsistent with the content of the input sentence: (1) the training data contains input and output pairs that contain inconsistencies, and (2) the output sentence in the encoder-decoder model depends only on the internal state of the encoder, so the input sentence itself is not taken into account.

[0008] Regarding (1), training an encoder-decoder model requires a large amount of data. However, since it is difficult to manually create such a large amount of data, the technologies disclosed in Non-Patent Documents 1 and 2 address this issue by automatically generating training data. Specifically, the technology in Non-Patent Document 1 uses rules to select suitable data for split and rephrase from knowledge graphs and data representing them as sentences. In addition, the technology in Non-Patent Document 2 identifies sentences split into multiple sentences from Wikipedia's editing history and uses them as training data. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Shashi Narayan, Claire Gardent, Shay B. Cohen, and Anastasia Shimorina. 2017. Split and rephrase. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 606-616, [Non-patent document 2] Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. 2018. Learning to split and rephrase from Wikipedia edit history. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 732-737. Summary of the Invention [Problem to be solved by the invention]

[0010] However, with regard to (1) above, the automatically generated training data contains incorrect sentence segmentation, which causes the encoder-decoder model to generate sentences that differ in meaning from the input sentences when learning from the training data.

[0011] Regarding (2) above, the decoder generates a sequence of simple sentences from the internal state of the encoder. During this generation, the input sentence itself is not explicitly used. This leads to the problem that simple sentences whose meaning differs from the input sentence are output.

[0012] The present invention has been made in view of the above points, and aims to obtain high-quality training data and improve the quality of segmenting long sentences into single sentences. [Means for solving the problem]

[0013] In order to achieve the above object, the invention according to claim 1 is a long sentence segmentation device that segments a long sentence into a plurality of simple sentences equivalent to the long sentence, using an entailment recognizer: A pair of an input sentence and multiple simple sentences corresponding to the input sentence. By filtering the training data, It is determined whether the input sentence implies each simple sentence, and a simple sentence is generated by combining only the input sentence and the simple sentence that is determined to imply the input sentence. a training data filter that outputs predetermined training data; and a filter that filters the predetermined training data output by the training data filter. Sequentially encoding and generate the hidden state and an encoder for Generate was Based on the hidden state Decoding the predetermined training data and output a sequence of simple sentences a decoder for the training data filter generates tuples each consisting of an input sentence related to the training data and a simple sentence corresponding to the input sentence, determines whether the input sentence entails all of the simple sentences using the entailment recognizer, and outputs only tuples for which it is determined that the input sentence entails the simple sentence as the predetermined training data. It is a long text division device. [Effects of the Invention]

[0014] As described above, the present invention has the effect of improving the quality of dividing a long sentence into single sentences. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a functional configuration diagram of a long sentence segmentation device according to an embodiment; [Figure 2] FIG. 2 is a functional configuration diagram of the learning device according to the embodiment. [Figure 3] FIG. 1 is a diagram showing a data flow in a system having a long sentence segmentation device and a learning device. [Figure 4] 1 is a diagram illustrating the hardware configuration of a long sentence segmentation device and a learning device according to an embodiment. [Figure 5] 10 is a flowchart showing a parameter learning process. DETAILED DESCRIPTION OF THE INVENTION

[0016] [Functional configuration of long sentence segmentation device and learning device] <Outline of the functions of the long-sentence segmentation device and learning device> First, the outline of the functions of the long text segmentation device according to the embodiment will be described using FIGS. 1 and 2. FIG. 1 is a functional configuration diagram of the long text segmentation device according to the embodiment. FIG. 2 is a functional configuration diagram of the learning device according to the embodiment.

[0017] As shown in FIG. 1, the long text segmentation device 1 has a training data filter 2, an encoder 3, a decoder 4, and a reproducer 5, and learns a neural network for the input training data. Each of these units is a function realized by an instruction from the processor 101 in FIG. 4 described later based on a program.

[0018] In the long text segmentation device 1, the training data filter 2 receives training data from the outside, and for the input sentence x i and the corresponding single sentence y i,1 ,..., y i,k generates tuples of <x i , y i,1 ,..., <x i , y i,k >, and performs implication relation recognition on all the tuples to determine whether x i implies y i,j . Here, only those determined to imply y for all tuples are used as the training data of the neural network. In the subsequent processing, the single sentences y i,1 ,..., y i,k are treated as a single series of data connected by special symbols

[0019]

Number

[0020] The encoder 3 receives n pieces of predetermined training data {x, y} passed through the training data filter 2, n performs sequential encoding, and passes the hidden state h X to the attention layer 42 of the decoder 4.

[0021] The decoder 4 receives y<t (the word sequence from the beginning to t - 1 of y) from the outside and the hidden state h from the encoder 3X receives

[0022]

Number

[0023]

Number

[0024] The reproducer 5 receives x<t (the word sequence from the beginning of x to t - 1) from the outside and h from the decoder 4 Y and

[0025]

Number

[0026] Thus, the learning device 6 optimizes the parameters of the neural network for all the training data that have passed through the training data filter 2 through the above encoder 3, decoder 4, and reproducer 5.

[0027] <Detailed description of the function of the long text segmentation device> Subsequently, the function of the long text segmentation device according to the embodiment will be described in detail with reference to FIG. 1.

[0028] (Training data filter) The training data filter 2 has an implication relationship recognizer 21, extracts only the highly reliable pairs from the pairs of an input sentence (long text) and a single sentence given from the outside, and makes them available for training the neural network. Note that the implication relationship recognizer 21 has a function of determining whether an implication relationship holds between two sentences.

[0029] ] Here, the given m pieces of training data are

[0030]

Number

[0031]

number

[0032] It should be noted that existing technology may be used for the entailment recognizer 21. For example, the entailment recognizer 21 may be realized using a pre-trained language model (BERT: Bidirectional Encoder Representations from Transformers), or a pre-trained model trained for entailment recognition may be used.

[0033] (encoder) The encoder 3, decoder 4, and reconstructor 5 are all based on a transformer model. The encoder 3 in the transformer model has a self-attention layer 31 and a feedforward layer 32. First, an input sentence x is input to the self-attention layer 31 and converted into a sequence of vectors. That is,

[0034]

number

[0035]

number

[0036] (Decoder) The decoder 4 includes a self-attention layer 41, an attention layer 42, a feedforward layer 43, and an output layer 44.

[0037] The self-attention layer 41 receives a sequence of simple sentences y before time t and converts them into a vector sequence.

[0038]

number

[0039]

number

[0040]

number

[0041] The self-attention layer 51 receives the sequence of input sentences x before time t and converts them into a vector sequence.

[0042]

number

[0043]

number

[0044]

number

[0045] <Detailed explanation of the learning device functions> Next, the functions of the learning device according to the embodiment will be described in detail with reference to FIG.

[0046] As shown in Fig. 2, the learning device 6 has a sampling unit 7 and a parameter optimization unit 8. These units are functions realized by instructions from a processor 101 in Fig. 4, which will be described later, based on a program. The learning device 6 also learns the parameters of the encoder 3, decoder 4, and reproducer 5 in the long sentence segmentation device 1.

[0047] In the learning device 6, the sampling unit 7 samples data to be used for one training session from all the training data. Note that the all the training data here refers to all the predetermined training data obtained through the training data filter unit 2. Then, through the encoder 3, decoder 4, and reproducing unit 5,

[0048]

number

[0049]

number

[0050]

number

[0051] The exchange of data (parameters) between the above-mentioned long sentence segmentation device 1 and learning device 6 is shown in Figure 3. Figure 3 is a diagram showing the flow of data in a system having a long sentence segmentation device and a learning device.

[0052] The objective function for optimizing each parameter of the neural network (encoder 3, decoder 4, reproducer 5) used in the parameter optimization unit 8 is given by the following (Equation 3).

[0053]

number

[0054]

number

[0055] [Hardware configuration of long sentence segmentation device and learning device] Next, the hardware configuration of the long sentence segmentation device 1 will be described with reference to Fig. 4. Fig. 4 is a hardware configuration diagram of the long sentence segmentation device according to the embodiment. Note that the hardware configuration of the learning device 6 is the same as that of the long sentence segmentation device 1, and therefore the description thereof will be omitted.

[0056] 4, the long sentence segmentation device 1 includes a processor 101, a memory 102, an auxiliary storage device 103, a connection device 104, a communication device 105, and a drive device 106. The hardware components constituting the long sentence segmentation device 1 are connected to each other via a bus 107.

[0057] The processor 101 serves as a control unit that controls the entire long sentence segmentation device 1, and includes various arithmetic devices such as a CPU (Central Processing Unit). The processor 101 reads various programs onto a memory 102 and executes them. The processor 101 may include a GPGPU (General-purpose computing on graphics processing units).

[0058] The memory 102 has a main storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 101 and the memory 102 form a so-called computer, and the processor 101 executes various programs read onto the memory 102, thereby enabling the computer to realize various functions.

[0059] The auxiliary storage device 103 stores various programs and various information used when the processor 101 executes the various programs.

[0060] The connection device 104 is a connection device that connects the long sentence segmentation device 1 with an external device (for example, the display device 110, the operation device 111).

[0061] The communication device 105 is a communication device for transmitting and receiving various types of information to and from other devices.

[0062] The drive device 106 is a device for loading a (non-transitory) recording medium 130. The recording medium 130 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, or a magneto-optical disk. The recording medium 130 may also include semiconductor memory that records information electrically, such as a ROM (Read Only Memory) or flash memory.

[0063] The various programs to be installed in the auxiliary storage device 103 are installed, for example, by setting the distributed recording medium 130 in the drive device 106 and reading the various programs recorded on the recording medium 130 by the drive device 106. Alternatively, the various programs to be installed in the auxiliary storage device 103 may be installed by being downloaded from a network via the communication device 105.

[0064] [Parameter learning process] Next, the parameter learning process of this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the parameter learning process.

[0065] First, the training data filter 2 of the long sentence segmentation device 1 filters each training data to obtain predetermined training data (S11), and the sampling unit 7 of the learning device 6 samples the predetermined training data (S12).

[0066] Next, the encoder 3 of the long text segmentation device 1 encodes the predetermined training data (S13). In this case, the encoder 3 vectorizes the long text related to the predetermined training data.

[0067] Next, the decoder 4 decodes the predetermined training data (S14). In this case, the decoder 4 outputs a simple sentence sequence composed of word sequences obtained by vectorization by the encoder 3.

[0068] Next, the reproducing unit 5 reproduces the specific long sentence using the simple sentence sequence output by the decoder 4.

[0069] Next, the parameter optimization unit 8 of the learning device 6 calculates the error between the specific long sentence and the long sentence related to the input sentence related to the original training data (S16). Specifically, the parameter optimization unit 8 calculates the error between the input sentence x and the generated

[0070]

number

[0071]

number

[0072] Next, if the error has not converged to a constant value (S17: NO), the parameter optimization unit 8 updates the parameters (S18). On the other hand, if the error has converged to a constant value (S17: YES), the parameter optimization unit 8 saves each parameter for the encoder 3, decoder 4, and reproducer 5 (S19).

[0073] [Effects of the embodiment] As described above, according to this embodiment, erroneous input / output pairs are deleted by filtering (cleaning) the training data using the entailment recognizer 21 of the training data filter 2. In this case, the training data filter 2 utilizes the entailment relationship between long sentences and simple sentences. In this way, high-quality training data can be obtained, which has the effect of improving the quality of segmenting long sentences into simple sentences.

[0074] Furthermore, according to this embodiment, by adding the reproducing unit 5, reproducing the input when generating an output from the internal state of the decoder 4 is incorporated into the machine learning of the neural network. In this case, the reproducing unit 5 reproduces a specific long sentence so that the parameter optimization unit 8 can use the error between the reproduced long sentence and the original input sentence (long sentence) for machine learning. In this way, the addition of the function of reproducing the long sentence, which is the input data, has the effect of reducing the inconsistency between the meaning of the simple sentence, which is the output data, and the input sentence.

[0075] 〔supplement〕 The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

[0076] The functional configurations of the long sentence segmentation device 1 and the learning device 6 can be realized by a computer and a program as described above, but this program can also be recorded on a (non-temporary) recording medium and provided, or provided via a network such as the Internet. [Explanation of symbols]

[0077] 1 Long text division device 2. Training Data Filter 3 encoder 4 Decoder 5 Reproducer 6 Learning Device 7 Sampling section 8 Parameter Optimization Section 21 Entailment Recognizer 31 Self-attention layer 32 Feedforward Layer 41 Self-attention layer 42 Caution layer 43 Feedforward Layer 44 Output layer 51 Self-attention layer 52 Caution layer 53 Feedforward Layer 54 Output layer

Claims

1. A long sentence dividing device that divides a long sentence into a plurality of simple sentences equivalent to the long sentence, a training data filter that uses the entailment recognizer to filter training data consisting of pairs of an input sentence and a plurality of simple sentences corresponding to the input sentence, thereby determining whether the input sentence entails each simple sentence, and outputs predetermined training data consisting only of pairs of input sentences and simple sentences that are determined to entail the input sentence; an encoder that sequentially encodes the predetermined training data output by the training data filter to generate hidden states; a decoder that decodes the predetermined training data based on the hidden state generated by the encoder and outputs a sequence of simple sentences; and The training data filter generates tuples consisting of an input sentence related to the training data and a simple sentence corresponding to the input sentence, determines whether the input sentence implies all of the simple sentences using the entailment relation recognizer, and outputs only tuples for which it is determined that the input sentence implies the simple sentence as the specified training data.

2. 2. The long-sentence segmentation device according to claim 1, wherein the entailment recognizer is realized by BERT.

3. A long sentence segmentation device according to claim 1 or 2, comprising a reproducor that reproduces a long sentence that is an input sentence related to the predetermined training data based on the simple sentence sequence output by the decoder, and the reproducor outputs a specific long sentence that has been reproduced in order to calculate an error with the input sentence; a learning device that updates each parameter of the encoder, the decoder, and the reproducer by calculating the error between a long sentence that is an input sentence related to the predetermined training data and the specific long sentence reproduced by the reproducer; A system having:

4. A long sentence segmentation method executed by a long sentence segmentation device that segments a long sentence into a plurality of simple sentences equivalent to the long sentence, comprising: using an entailment recognizer to filter training data consisting of pairs of an input sentence and a plurality of simple sentences corresponding to the input sentence, thereby determining whether the input sentence entails each simple sentence, and outputting predetermined training data consisting only of pairs of input sentences and simple sentences that are determined to entail each other; Sequentially encoding the output predetermined training data to generate hidden states; Decoding the predetermined training data based on the generated hidden state to output a simple sentence sequence; When outputting the predetermined training data, a tuple consisting of an input sentence related to the training data and a simple sentence corresponding to the input sentence is generated, and whether or not the input sentence entails all of the simple sentences is determined by the entailment recognizer, and only the tuple for which it is determined that the input sentence entails the simple sentence is output as the predetermined training data. How to divide long texts.

5. A program causing a computer to execute the method according to claim 4.

Images