Language processing device, language processing method, learning method, and program

Abstract

When using a plurality of speech recognition results (recognition hypotheses) in a language processing device, the present invention achieves context-sensitive word selection, word selection criteria not dependent on reliability or frequency of occurrence, and hypothesis selection optimized for language processing purposes such as summarization and translation. The language processing device for this purpose comprises a plurality of vector representation units, a first encoding unit, an integration unit, and a decoding unit. The plurality of vector representation units convert a plurality of input texts into a plurality of first vector sequences. The first encoding unit converts the plurality of first vector sequences into a plurality of second vector sequences. The integration unit integrates the plurality of second vector sequences to generate a single integrated vector sequence. The decoding unit generates a language processing result on the basis of the integrated vector sequence.

Description

[Technical Field]

[0001] The disclosed technology relates to language processing technologies such as summarization and translation, which are linked to speech recognition. [Background technology]

[0002] Applications such as speech translation, voice dialogue, and speech summarization are realized through a combination of speech recognition devices and language processing devices. Figure 1 shows an example of the application. The input speech is transcribed into text by the speech recognition device 101, and the transcribed text is converted into a summary or translation by the language processing device 102 to generate output. However, because speech recognition devices are not 100% accurate, language processing devices are affected by the erroneous output of speech recognition devices and have the problem of not being able to produce correct results. To address this problem, there is a method to reduce the impact of errors in speech recognition devices on language processing devices by processing a single input audio with multiple speech recognition devices, obtaining recognition results from each speech recognition device as recognition candidates (recognition hypotheses), and processing the text that integrates these multiple recognition hypotheses (Non-Patent Literature 1). This is illustrated in Figure 2.

[0003] A single input voice is input to, for example, three speech recognition devices A(201), B(202), and C(203), and the speech recognition results are obtained as texts A, B, and C. The integration device 204 breaks down each text into words and estimates corresponding words between the texts using the utterance time and order of the words as clues. Once the corresponding words are determined, a score is assigned to each corresponding word group based on which word was used most frequently (voting) and the confidence level of each speech recognition device regarding that word, and the word with the highest score is selected. The integrating device connects the words with the highest scores and outputs the combined text R, which is a combination of hypothetical texts A, B, and C, to the language processing device 102. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jonathan G. Fiscus, "A POST-PROCESSING SYSTEM TO YIELD REDUCED WORD ERROR RATES: RECOGNIZER OUTPUT VOTING ERROR REDUCTION(ROVER)" [Overview of the project] [Problems that the invention aims to solve]

[0005] By using multiple speech recognition hypotheses, it is possible to compensate for recognition errors in one hypothesis by using the other hypotheses to arrive at the correct answer. However, the following challenges generally exist in this process.

[0006] (1) Multiple recognition results consist of sequences of multiple words, but word alignment is not performed, and the words output at the same time t do not necessarily represent the same sound across sequences. This is because word insertions and deletions occur due to misrecognition. Therefore, it is necessary to first identify the occurrence of insertions and deletions, correct them, and then perform word alignment.

[0007] (2) After resolving the correspondence between words, it is necessary to select which of the multiple recognition results are incorrect and which are correct. There are various methods for determining the criteria for this, but the most common methods are using the confidence level of the speech recognition or comparing it with other candidates and deciding by voting. However, each of these methods, and all of them, have the following challenges. (2-1) In methods that use the confidence level of speech recognition, the speech recognition device may make mistakes with confidence, and therefore, even in cases of misrecognition, a high confidence level may be assigned. As a result, it is difficult to make the correct selection by comparing the confidence levels of multiple hypotheses. This is especially difficult when comparing different speech recognition devices. (2-2) In the method of comparing hypotheses and deciding candidates by voting, voting is performed at each time t and the most frequently occurring word is adopted. The problem with this method is that it cannot consider similar words. Words that are similar but not exactly the same are counted as separate words. (2-3) A common issue with both methods is that they do not consider the surrounding words when selecting words, and therefore cannot consider the naturalness of the sentence when selecting words.

[0008] (3) When hypothesis integration is performed outside the language processing device, the integration is performed independently of the language processing device, making it impossible to perform hypothesis integration that is suitable for processing by the language processing device. For example, the language processing device 102 in Figure 2 has a learning mechanism inside, but the learning is confined to the language processing device, and it is difficult to perform learning in conjunction with the external integration device 204.

[0009] Non-patent document 1 attempts to solve the word alignment problem (1) among the above problems, but does not disclose problems (2) and (3), namely, consideration of surrounding words, word selection criteria other than reliability and frequency of occurrence, and hypothesis integration that takes into account the optimization of subsequent language processing. [Means for solving the problem]

[0010] To solve the above problems, the language processing device relating to the disclosed technology includes a plurality of vector representation units, a first encoding unit, a unification unit, and a decoding unit. Multiple vector representation units convert multiple input texts into multiple first vector sequences. The first encoding unit converts the multiple first vector sequences into multiple second vector sequences. The unification unit integrates the multiple second vector sequences to generate a single unified vector sequence. The decoding unit generates the language processing result based on the unified vector sequence. The unification unit may further include an alignment unit that aligns the second vector sequence and generates an aligned vector sequence, and an integration unit that integrates the aligned vector sequence into the unified vector sequence. [Effects of the Invention]

[0011] According to the disclosed technology, the following effects can be obtained. (1) Contextual consideration: Through encoding, input words are replaced with vector representations that take into account the surrounding words. Hypothesis integration is performed based on these vector representations, allowing for hypothesis integration that takes context into account. (2) Word selection criteria: Hypothesis integration does not require the use of statistical information from speech recognition devices, such as the reliability of speech recognition, because it takes into account linguistic naturalness and similarity. Therefore, it is possible to integrate the output results of various speech recognition devices. (3) Optimization for language processing units: The encoding unit and hypothesis integration unit are built into the language processing unit and include learnable weights. Therefore, hypothesis integration is updated through learning and optimized to maximize the accuracy of language processing. (4) In addition to the above, the hypothesis integration unit of the disclosure technology is a mapping that outputs the representative sequence of the received hypothesis sequence as is at the start of training, so that training can be started without degrading the performance of large-scale pre-trained models (BERT), etc. [Brief explanation of the drawing]

[0012] [Figure 1] This diagram illustrates an example of a language processing unit performing summarization and translation processing based on a single speech recognition result. [Figure 2] This diagram illustrates an example of integrating multiple speech recognition results in the preceding stage of a language processing unit in order to utilize multiple speech recognition results. [Figure 3] A diagram illustrating the functional block of the first embodiment of the disclosed technology. [Figure 4] A diagram illustrating the details of the alignment and integration unit of the first embodiment. [Figure 5] A diagram illustrating the language processing flow according to the first embodiment. [Figure 6] A diagram illustrating the encoding of a hypothetical vector sequence. [Figure 7] A diagram illustrating the alignment of encoded hypothesis vector sequences. [Figure 8] A diagram illustrating the integration of aligned hypothesis vector sequences. [Figure 9] A diagram illustrating the learning flow of the language processing device according to the first embodiment. [Figure 10] A diagram showing the correspondence between a conventional language processing device and a language processing device according to the first embodiment. [Figure 11] A diagram showing an example configuration of a computer operating as a language processing unit. [Modes for carrying out the invention]

[0013] The embodiments of the disclosed technology will be described in detail below. Components with the same function will be numbered identically, and redundant explanations will be omitted. Since the disclosed technology extends conventional language processing devices that use an Encoder-Decoder with attention model, we will first explain conventional language processing devices.

[0014] Figure 1 shows an example of the configuration of a conventional language processing device. The language processing device is built using deep learning, and each hypothesis (speech recognition result) is represented as a vector within the device. The language processing device 102 receives the recognition result from the speech recognition device 101 as text and converts the received text into a first word vector sequence using word embedding or the like. Here, word embedding refers to mapping the unique ID (scalar) assigned to each word based on a dictionary to a vector space, so that the relationships between different characters and words, such as their meaning and similarity, can be calculated in the vector space. The language processing unit 102 encodes the first word vector sequence and converts it into a second word vector sequence. Here, the first word vector sequence is a vector sequence that independently maps each word ID and has no dependencies on other words in the vector sequence. Then, the second word vector sequence is obtained by mapping each vector in the first word sequence to the other vector sequences, taking into account the relationships between them. The language processing device 102 further generates a third word vector sequence by applying cross-attention processing to the second word vector sequence. Here, cross-attention processing for a word in the second word vector sequence means, for example, if the language processing is "translation," extracting word information that should be paid attention to and used in order to correctly translate that word from all the words in the second word vector sequence. The decoding unit decodes the third word vector sequence to obtain the desired output (such as a summary or translation). Here, decoding the third word vector sequence means, for example, in the case of language processing for translation, calculating the current third word vector sequence using the features of past third word vector sequences, attention, and second word vector sequences, and generating text in the target language.

[0015] Disclosure technology extends conventional language processing units to handle multiple speech recognition results. Conventional language processing units could only process one hypothesis at a time, meaning they had no choice but to accept and process incorrect recognition results. However, disclosure technology processes multiple recognition results and integrates them internally, allowing for the selection of the most relevant results. Furthermore, this selection process is optimized from the perspective of maximizing the accuracy of the language processing unit.

[0016] [First Embodiment] Figure 3 is a functional block diagram of a language processing device according to the first embodiment of the disclosed technology, and Figure 4 is a functional block diagram showing details of the alignment and integration unit in Figure 3. Figure 5 is a flowchart illustrating the operation of the language processing device. The first embodiment will be described below, primarily with reference to Figures 3, 4, and 5.

[0017] In Figure 3, speech recognition devices 301 to 303 each process the same input speech and supply the speech recognition result (recognition hypothesis) as text to the language processing device 310. Therefore, N texts are input to the language processing device. The language processing unit 310 is equipped with N vector representation units 304, which convert N input texts into vector representations and generate N hypothetical vector sequences. Each hypothetical vector sequence is a sequence of multiple word vectors. The language processing device 310 is equipped with N first encoding units, which add arbitrary position information dependent on the language processing model to the output of the vector representation unit 304 and perform encoding, and output the encoded hypothetical vector sequence to the sorting and integrating unit 306.

[0018] Figure 4 shows the details of the alignment and integration unit, which consists of a hypothesis alignment unit and a hypothesis integration unit, both equipped with attention mechanisms. First, alignment (described later) is performed on N encoded hypothesis vector sequences in the hypothesis sorting unit 401. Then, the N aligned hypothesis vector sequences are integrated into a single vector sequence (described later) in the hypothesis integration unit 402.

[0019] The language processing unit 310 applies additional encoding to the integrated vector sequence output by the hypothesis integration unit using the second encoding unit. The cross-attention unit 308 applies attention processing to the integrated vector sequence according to the purpose of the language processing unit to generate a feature vector. The decoding unit 309 decodes the feature vector received from the cross-attention unit 308 and generates the desired output (such as a summary or translation).

[0020] Figure 5 is a flowchart of language processing by the language processing device 310 of the first embodiment. The language processing device 310 receives the text of the recognition result (recognition hypothesis) T from the speech recognition device. 1 From T N The hypothesis to be obtained is any N text sequences, regardless of the generation method, such as the N best hypothesis or the outputs of N speech recognitions. Note that the nth text T n M n It shall consist of these words.

[0021] The language processing device 301 uses the vector representation unit 304 to represent the hypothesis vector sequence S n by means of the distributed representation of words and the like as a vector. n is generated (step S502). S n is a sequence of Mn word vectors. S n =\[s n 1, s n 2, ..., s n Mn \] The language processing device 301 encodes S n in the first encoding unit to generate the encoded hypothesis vector sequence E n (step S503). At this time, each hypothesis is encoded independently, and the relationships between hypotheses are not considered. Since the number of words constituting S n generally varies between hypotheses, when encoding, zero vectors are padded at the end of the vector sequence to make it the same length as the longest encoded hypothesis vector sequence. Fig. 6 shows an example when E 1 is the longest hypothesis.

[0022] <Hypothesis alignment> Next, the language processing device 310 aligns the time information of each hypothesis in the hypothesis alignment unit 401 (step S504). This will be described while referring to Fig. 7 as well. The hypothesis alignment unit 401 includes an attention unit and performs alignment of other hypotheses with respect to the representative hypothesis. When the first hypothesis E 1 is used as the representative hypothesis, the aligned vector sequence ~E n related to any n-th hypothesis is derived as follows. ~E n =softmax((E 1 W Q )(E n W K ))E T W n W V ~E 1 =[ ~e 1 1, ~e 1 2,..., ~e 1 M1 ​ ~E 2 =[ ~e 2 1, ~e 2 2,..., ~e 2 M1 ] ... ~E n =[ ~e n 1, ~e n 2,..., ~e n M1 ] ... ~E N =[ ~e N 1, ~e N 2,..., ~e N M1 ] Here, W Q , W K , W V These represent the linear mapping weights for Query, Key, and Value in the attention, respectively, with the same number of input and output dimensions. T is the transpose symbol. In this alignment process, the similarity / relevance between the nth hypothesis and the representative hypothesis (1st) is measured using the inner product for each word vector, and the nth hypothesis is rearranged relative to the 1st hypothesis sequence. In addition, this alignment process also transfers corresponding word information to positions that were 0 vectors in the encoded hypothesis vector sequence.

[0023] <Integration of hypotheses> After aligning the hypotheses to obtain the time information for each hypothesis, the language processing device 310 uses the hypothesis integration unit 402 to perform ~E 1 From ~E N These are integrated into one (step S505). This process will be explained with reference to Figure 8. First, ~E 1 From ~E N Decompose it in the time direction to generate the following sequence of equivalence hypothesis vectors. C1=[~e 1 1, ~e 2 1, ..., ~e N 1] C2=[~e 1 2, ~e 2 2, ..., ~eN 2] ··· C m =[~e 1 m , ~e 2 m ,…, ~e N m ··· C M1 =[~e 1 M1 , ~e 2 M1 ,…, ~e N M1 That is, C m is the set of the m-th word vectors of each aligned vector sequence. First, for each word in C m , using attention, calculate the inner product with the m-th word e 1 m of the encoded vector sequence of the representative hypothesis, and process the calculation result with the softmax function to obtain the N-dimensional vector a m . a m =softmax((e 1 m )) T W Q C m ) Then, weight-average C m with each element of a m to obtain the integrated vector e att m of the m-th word. e att m =a m C m T

[0024] The language processing device 310 performs additional encoding (second encoding) on the word vector sequence integrated in this way (step S506), performs cross-attention processing on the encoding result (step S507), and decodes the cross-attention processing output to generate a final language processing result (such as summarization or translation, etc.) (step S508). ​​

[0025] [Learning of the language processing device] W of the language processing device of the first embodiment Q W K W V are learnable parameters, and when operating as a language processing device, parameters optimized by learning are set. As illustrated in FIG. 10, in the first embodiment, the encoding step of the existing Encoder-Decoder with attention model including a vector representation step (S103), an encoding step (S104), a cross-attention processing step (S105), and a decoding step (S106) is divided into the first half (first encoding) and the second half (second encoding), the processing up to the first encoding is parallelized, the outputs of the parallel processing are integrated into one in the alignment integration step, and it can be configured to be connected to the language processing after the second encoding. That is, in the learning of the language processing device of the first embodiment, for W Q W K W V other parameters, those of an existing pre-trained language processing device can be used. And, when starting learning with the initial values of W Q W K W V being the identity matrix, the output of the alignment integration unit at the initial stage of learning is E 1 =[e 1 1,...,e 1 M1 , which is very close. That is, the insertion of the alignment integration unit does not affect the existing language processing in the initial state, so learning can be started without degrading the performance of a large-scale pre-trained model such as BERT.

[0026] FIG. 9 is a flowchart of the learning procedure of the language processing device according to the first embodiment. It will be described while also using FIG. 3. Learning is performed using a set of learning data, which is a plurality of speech recognition results related to the same speech, and one teacher data, which is the correct answer for language processing. First, the first encoding unit 305 and the second encoding unit 307 are set to the trained model of a conventional language processing device (a language processing device that processes a single speech recognition result) as initial values. W of the sorting and integrating unit 306 Q , W K , W V The identity matrix is ​​set as the initial value (step S901). Next, the training data is processed by the language processing unit 310, and the results are obtained (step S902). Next, the difference between the language processing result and the training data is calculated (step S903). Next, it is determined whether the learning termination conditions have been met (step S904). The learning termination conditions are determined by whether the difference between the language processing result and the training data falls below a predetermined value, or whether a specified number of parameter adjustments have been performed. If step S904 is Yes, the learning process ends. If No, all parameters are adjusted to minimize the calculated difference (step S905), and steps S902 through 904 are executed again.

[0027] [Programs, recording media] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 11, and then causing the control unit 2010, input unit 2030, output unit 2040, etc. to operate.

[0028] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0029] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0030] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the above processing may be executed by a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring programs from the server computer to this computer. In this configuration, the program includes information used for processing by an electronic computing device that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the processing of the computer, etc.).

[0031] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

Claims

1. A language processing device that generates a vector sequence from text, in which each word constituting the text is represented as a vector, and decodes the vector sequence to generate the result of language processing, Multiple vector representation units that convert multiple texts into multiple first vector sequences, A first encoding unit that converts a plurality of the first vector sequences into a plurality of second vector sequences, A unification unit that integrates multiple aforementioned second vector sequences to generate a single unified vector sequence, A decoding unit that generates language processing results based on the aforementioned integrated vector sequence, A language processing device that includes a language processing unit.

2. A language processing device according to claim 1, The unification unit includes an alignment unit that aligns the second vector sequence and generates an aligned vector sequence, and an integration unit that integrates the aligned vector sequence into the unified vector sequence. A language processing device that includes a language processing unit.

3. A language processing device according to claim 2, The aligned vector sequence is obtained by the alignment unit determining a representative vector sequence from a plurality of second vector sequences, measuring the relationship between the representative vector sequence and each of the second vector sequences, and rearranging each of the second vector sequences based on the representative vector sequence. Language processing device.

4. A language processing device according to claim 3, The integrated vector sequence is a set obtained by the integration unit measuring the relationship between the m-th word of the representative vector sequence (where m is an integer from 1 to the number of words constituting the aligned vector sequence) and the m-th word group of the aligned vector sequence, and then weighting and adding each vector of the m-th word group based on the measurement results. Language processing device.

5. A language processing method that generates a vector sequence from text, in which each word constituting the text is represented as a vector, and decodes the vector sequence to generate the result of language processing, The process involves multiple vector representation units converting multiple texts into multiple first vector sequences, The first encoding unit performs the steps of converting a plurality of first vector sequences into a plurality of second vector sequences, The unification unit performs the steps of integrating multiple second vector sequences to generate a single unified vector sequence, The decoding unit generates a language processing result based on the integrated vector sequence, A language processing method that includes this.

6. A language processing method according to claim 5, The unification unit comprises an alignment unit and an integration unit. The alignment unit performs the alignment of the second vector sequence and generates an aligned vector sequence. The integration unit performs the steps of integrating the aligned vector sequence into the integrated vector sequence, A language processing method that includes this.

7. A method for learning deep learning parameters for a language processing device, wherein a vector representation unit generates multiple first vector sequences from multiple speech recognition result texts relating to the same speech, an encoding unit converts the multiple first vector sequences into multiple second vector sequences that take context into account, and an integration unit integrates the multiple second vector sequences using an attention mechanism, and generates language processing results from the integrated vector sequences, The steps include setting the initial values ​​of the parameters of the encoding unit to a trained model, The steps include setting the identity matrix as the initial value of the mapping matrix of the attention mechanism of the integration unit, Learning methods that include this.

8. A program for causing a computer to function as a language processing device according to any one of claims 1 to 4.

Images