Speech language processing device and program

Abstract

To provide a speech language processing device capable of accepting input sentences via speech for performing various language processing tasks.SOLUTION: The speech language processing device is equipped with an encoder unit and a large-scale language model processing unit. The encoder unit performs processing to encode speech and outputs encoded speech information. The large-scale language model processing unit outputs an output sentence based on an input task prompt and an input sentence. The large-scale language model processing unit is equipped with a masked multi-head attention unit, a cross-attention unit that performs multi-head attention processing by causing the encoded speech to represent the input sentence, and a feedforward network unit. A mechanism is provided to multiply the output from the cross-attention unit by a predetermined value between 0 and 1 inclusive.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a speech language processing device and a program. [Background technology]

[0002] Research is being conducted on the use of large-scale language models capable of machine learning to process various tasks based on input language (text, etc.).

[0003] Non-Patent Document 1 describes a large-scale language model GPT3 that performs pre-training for the next word prediction task and can solve various tasks.

[0004] In the technology described in Non-Patent Document 2, a large-scale language model is introduced into a speech recognition decoder. Then, prior information about the speech is input to the speech recognition decoder as a prompt to improve accuracy.

[0005] The technology described in Non-Patent Document 3 proposes a multimodal generative model, AudioPaLM, using the decoder model Palm-2 (a large-scale language model using only the decoder part). Specifically, the input embedding layer of Palm-2 is extended for speech input, and the speech input is tokenized and used as input to the large-scale language model.

[0006] Non-Patent Document 4 describes a technology in which humans provide feedback to a large-scale language model to generate text that is in line with human intent.

[0007] The technology described in Non-Patent Document 5 proposes a vision language model called Flamingo, which uses a gating mechanism in the language model to introduce a visual encoder. This technology enables text generation using image modal information passed from the image encoder. [Prior art documents]

Non-Patent Literature

[0008]

Non-Patent Literature 1

Non-Patent Literature 2

Non-Patent Literature 3

Fashion 4

[0009] However, the prior art has problems in allowing various tasks to be performed by inputting spoken language.

[0010] When using conventional technology, for example, by combining a speech recognition processing system with a system for processing subsequent tasks (e.g., machine dialogue, machine translation, etc.), applications such as speech dialogue and speech translation can be realized. However, when conventional technology is used as is, there is a problem that combining multiple models causes errors to propagate, resulting in a deterioration in processing accuracy. In addition, once spoken language is converted into text through speech recognition processing, important information contained in the original speech (such as prosody, distinctions between male and female voices, and distinctions between adult and child voices) is lost, making it impossible to use for subsequent tasks.

[0011] Furthermore, training a deep learning model requires a large amount of training data (input data and correct answer data) to solve a desired task based on voice input.

[0012] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a speech language processing device and program that can accept spoken input sentences in order to perform various language processing tasks. [Means for solving the problem]

[0013] [1] In order to solve the above-mentioned problems, a speech language processing apparatus according to one aspect of the present invention includes an encoder unit that performs a process of encoding speech and outputs encoded speech information, and a large-scale language model processing unit that outputs an output sentence based on an input task prompt and an input sentence, wherein the large-scale language model processing unit receives an output from the masked multi-head attention unit as input Q (query), receives the encoded speech information output from the encoder unit as V (value) and K (key), and operates the encoded speech to represent the input sentence, thereby generating a multi-head attention sentence. a first gate unit that multiplies the output from the cross-attention unit by a first predetermined value greater than or equal to 0 and less than or equal to 1 and outputs the result; a first adder unit that adds the output from the first gate unit and the output from the masked multi-head attention unit and outputs the result; a feedforward network unit that uses the output from the first adder unit as input to process a feedforward network and outputs the result of the processing; a second gate unit that multiplies the output from the feedforward network unit by a second predetermined value greater than or equal to 0 and less than or equal to 1 and outputs the result; and a second adder unit that adds the output from the second gate unit and the output from the first adder and outputs the result, wherein each of the first predetermined value and the second predetermined value is a value obtained by learning using learning data with an initial value of 0.

[0014] [2] In another aspect of the present invention, in the speech language processing device of [1] above, the large-scale language model processing unit is trained in advance using training data provided as a set of the task prompt, the text data of the input sentence, and the correct answer to the text data of the output sentence, while not inputting the encoded speech information as V (value) and K (key) of the cross-attention unit.

[0015] [3] In another aspect of the present invention, in the speech language processing device of [2] above, the encoder unit and the large-scale language model processing unit are trained using training data provided as a set of speech corresponding to an input sentence, the task prompt, a correct text for the input sentence, and a correct text for the output sentence.

[0016] [4] Also, one aspect of the present invention includes an encoder unit that performs a process of encoding speech and outputs encoded speech information, and a large-scale language model processing unit that outputs an output sentence based on an input task prompt and an input sentence, the large-scale language model processing unit including a masked multi-head attention unit that performs multi-head attention processing based on the task prompt and a past output from the large-scale language model processing unit, and a cross-attention unit that receives the output from the masked multi-head attention unit as a query (Q) and the encoded speech information output from the encoder unit as a value (V) and a key (K), and performs multi-head attention processing by making the encoded speech represent the input sentence. a first gate unit that multiplies the output from the cross-attention unit by a first predetermined value greater than or equal to 0 and less than 1 and outputs the result; a first adder unit that adds the output from the first gate unit and the output from the masked multi-head attention unit and outputs the result; a feedforward network unit that uses the output from the first adder unit as input to process a feedforward network and outputs the result of the processing; a second gate unit that multiplies the output from the feedforward network unit by a second predetermined value greater than or equal to 0 and less than 1 and outputs the result; and a second adder unit that adds the output from the second gate unit and the output from the first adder unit and outputs the result, wherein each of the first predetermined value and the second predetermined value is a value obtained by learning using training data with an initial value of 0. [Effects of the Invention]

[0017] According to the present invention, it is possible to use spoken language as input, generate output sentences in response to task prompts, and output the output sentences. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a speech language processing apparatus according to an embodiment of the present invention; [Figure 2] 2 is a schematic diagram showing one form for realizing the functions of an encoder unit and a large-scale language model processing unit included in the speech language processing apparatus according to the embodiment. FIG. [Figure 3] FIG. 2 is a block diagram showing the configuration of a gating mechanism in a decoder (large-scale language model processing unit) according to the embodiment. [Figure 4] FIG. 10 is a schematic diagram showing the flow of pre-learning using only text in the large-scale language model processing unit according to the embodiment. [Figure 5] 2 is a schematic diagram showing a flow of learning by pairs of speech and text in the speech language processing apparatus according to the embodiment. FIG. [Figure 6] 2 is a schematic diagram showing a processing flow at the time of inference in the spoken language processing apparatus according to the embodiment. FIG. [Figure 7] FIG. 10 is a schematic diagram for explaining in more detail the flow of processing during inference in the large-scale language model processing unit according to the embodiment. [Figure 8] 10 is a flowchart showing the procedure of an inference process by a trained speech language processing apparatus according to the embodiment. [Figure 9] FIG. 2 is a block diagram showing an example of the internal configuration of the speech language processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] Next, an embodiment of the present invention will be described with reference to the drawings.

[0020] This embodiment relates to an apparatus and a program for performing language processing such as dialogue and translation using speech by utilizing large language models (LLM). In this embodiment, speech input is enabled by introducing an encoder into a large language model configured using a Transformer decoder. This embodiment performs a desired task, such as dialogue or natural language translation. This embodiment executes task processing taking into account prosodic expressions and the like contained in the input speech.

[0021] FIG. 1 is a block diagram showing a schematic functional configuration of a speech language processing device according to this embodiment. As shown in the figure, the speech language processing device 1 includes a speech input unit 21, an encoder unit 22, a large-scale language model processing unit 23, and an output unit 24. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit also has a storage means as needed. The storage means is, for example, a memory for storing program variables or allocated by the execution of the program. Non-volatile storage means such as a magnetic hard disk drive or a solid-state drive (SSD) may also be used as needed. At least some of the functions of each functional unit may also be realized as a dedicated electronic circuit rather than a program. The functions of each unit are as follows:

[0022] The voice input unit 21 acquires voice and passes the voice in an appropriate format to the encoder unit 22. For example, the voice input unit 21 converts a voice waveform into acoustic feature data and passes the data to the encoder unit 22.

[0023] The encoder unit 22 performs a process of encoding the speech passed from the speech input unit 21. The encoder unit 22 passes the encoded speech information resulting from the process to the large-scale language model processing unit 23. In other words, the encoder unit 22 performs a process of encoding the speech and outputs the encoded speech information.

[0024] The encoder unit 22 is configured to perform processing that depends on internal parameters. The encoder unit 22 is capable of performing machine learning using training data. In other words, the values ​​of multiple parameters that the encoder unit 22 internally holds can be updated and optimized through learning using the training data.

[0025] The large-scale language model processing unit 23 is also called a "decoder unit." The large-scale language model processing unit 23 outputs the next word of the output sentence (speech recognition result + answer sentence) based on the information (encoded speech) passed from the encoder unit 22, the task prompt, and past output from the large-scale language model processing unit 23. In other words, the large-scale language model processing unit 23 outputs an output sentence based on the task prompt and the input sentence, based on the input task prompt and the input sentence.

[0026] The large-scale language model processing unit 23 is configured to perform processing that depends on parameters stored therein. The large-scale language model processing unit 23 is capable of performing machine learning using training data. In other words, the values ​​of multiple parameters stored therein can be updated and optimized by learning using the training data.

[0027] The large-scale language model processing unit 23 may be, for example, a language model such as a transformer decoder. Note that a large-scale language model that has been pre-trained may also be used as the large-scale language model processing unit 23. In this case, it is preferable to use a large-scale language model that has been tuned for dialogue and that has undergone RLHF (Reinforcement Learning from Human Feedback). Note that a sufficiently trained large-scale language model with a large number of parameters can flexibly provide task prompts. For example, a task prompt in the form of "Please have a dialogue" may be provided.

[0028] The large-scale language model processing unit 23 may use a pre-trained large-scale language model, or may be trained independently. The pre-training of the large-scale language model processing unit 23 can be performed using only text. The large-scale language model processing unit 23 processes a task according to an input task prompt. Here, the task may be, for example, a dialogue using sentences, a translation of a sentence, a summary of a sentence, etc. (not limited to the examples given here). When the large-scale language model processing unit 23 is pre-trained, a large amount of language resources such as a dialogue corpus or a translation corpus is used depending on the task to be solved.

[0029] The output unit 24 is a text. The output unit 24 extracts only the task answer sentence from the text (speech recognition result+task answer sentence) output from the large-scale language model processing unit 23, and outputs it to the outside as final text.

[0030] FIG. 2 is a schematic diagram showing one form that forms the basis for realizing the functions of the encoder unit 22 and the large-scale language model processing unit 23 of the speech language processing apparatus 1. The configuration shown in the figure is that of a transformer. The encoder unit 22 and the large-scale language model processing unit 23 can be realized by applying a transformer. The configuration shown in the figure includes an encoder 221 and a decoder 231. The encoder 221 corresponds to the encoder unit 22 in FIG. 1. The decoder 231 corresponds to the large-scale language model processing unit 23 (decoder unit) in FIG. 1. Note that the transformer itself belongs to existing technology. The decoder 231 of this embodiment has a configuration that will be described later with reference to FIG. 3.

[0031] 2, the encoder 221 includes a multi-head attention unit 2211 and a feed-forward network unit 2212. The decoder 231 includes a masked multi-head attention unit 2311, a multi-head attention unit 2312 (cross attention, multi-head attention), and a feed-forward network unit 2313.

[0032] The decoder 231 shown in Figure 2 has the same structure as a transformer decoder. The masked multi-head attention unit 2311 performs self-attention between past outputs from the decoder 231. The multi-head attention unit 2312 (cross-attention) performs cross-attention processing between the output from the encoder 221 and the decoder input (input to the decoder 231) that has undergone self-attention by the masked multi-head attention unit 2311.

[0033] Input data (in this embodiment, audio data representing a spoken language) is input to the encoder 221 after undergoing input embedding and positional encoding addition. The output from the encoder 221 is encoded audio information. The output from the encoder 221 is passed to the multi-head attention unit 2312 of the decoder 231. The input data to the decoder 231 is data obtained by shifting the output from the decoder 231 itself. The shifted output from the decoder 231 is input to the masked multi-head attention unit 2311 of the decoder 231 after undergoing output embedding and positional encoding addition. The masked multi-head attention unit 2311 performs multi-head attention processing based on the task prompt and past output from the decoder 231 (large-scale language model processing unit 23). The output from the masked multi-head attention unit 2311 is input to Q (Query) of the multi-head attention unit 2312 (cross attention). Meanwhile, the output from the encoder 221 is input to V (Value) and K (Key) of the multi-head attention unit 2312 (cross attention). The multi-head attention unit 2312 (cross attention unit) receives the output from the masked multi-head attention unit 2311 as Q (Query) and the encoded speech information output from the encoder 221 (encoder unit 22) as V (Value) and K (Key), and performs multi-head attention processing by making the encoded speech represent the input sentence. The output from the multi-head attention unit 2312 (cross attention) is processed by the feedforward network unit 2313 and then output. The output from the decoder 231 is output after undergoing linear transformation processing (Linear) and softmax function processing (Softmax). The output is a probability value for each word (token) that is part of the output word sequence (token sequence).

[0034] That is, the decoder 231 generates a past text output (a past sequence of words y1, y2, . . . , y t-1 ) and the speech information encoded by the encoder 221, the next word (token) y in the output text is t Output.

[0035] The decoder 231 of this embodiment is equipped with a gating mechanism. When performing learning using pairs of speech and text or performing inference processing, the decoder 231 uses the gating mechanism in the multi-head attention unit 2312 (cross-attention unit) and the feedforward network unit 2313. The gating mechanism of this embodiment will be described next.

[0036] It should be noted that the decoder 231 does not need to use a gating mechanism when pre-training is performed using only text.

[0037] 3 is a block diagram showing the configuration of one layer of the decoder 231 of this embodiment. The decoder 231 of this embodiment includes a gating mechanism. As shown in the figure, the decoder 231 is configured to include a pre-trained self-attention unit 23121, a pre-trained feedforward network unit 23122, a cross-attention unit 23123, a gate unit 23124, an adder 23131, a feedforward network unit 23125, a gate unit 23126, and an adder 23132.

[0038] The pre-trained self-attention unit 23121 shown in FIG. 3 corresponds to the masked multi-head attention unit 2311 in FIG. 2. Furthermore, the pre-trained feedforward network unit 23122 shown in FIG. 3 corresponds to the feedforward network unit 2313 in FIG. 2. The pre-trained self-attention unit 23121 and the pre-trained feedforward network unit 23122 have been pre-trained in pre-training using only the decoder 231. In this pre-training, the encoder 221 and the multi-head attention unit 2312 shown in FIG. 2 are not used. The configuration of the cross-attention unit 23123 itself shown in FIG. 3 is the same as the multi-head attention unit 2312 shown in FIG. 2. Furthermore, the configuration of the feedforward network unit 23125 itself shown in FIG. 3 is the same as the feedforward network unit 2313 shown in FIG. 2.

[0039] The pre-trained self-attention unit 23121 performs self-attention processing using the past output sentences (sequences of words previously output) of the decoder 231 as inputs of Q (query), V (value), and K (key).

[0040] The pre-trained feedforward network unit 23122 performs feedforward network processing based on the output from the pre-trained self-attention unit 23121. Note that the values ​​of the internal parameters of the pre-trained feedforward network unit 23122 have been pre-trained.

[0041] The cross-attention unit 23123 receives the output from the pre-trained feedforward network unit 23122 as Q (query) and receives the data (encoded speech information) output from the encoder 221 as V (value) and K (key) to perform cross-attention processing. Note that the output from the pre-trained feedforward network unit 23122 is data based on the output from the pre-trained self-attention unit 23121, as described above. That is, the cross-attention unit 23123 receives the data based on the output from the pre-trained self-attention unit 23121 as Q (query). The cross-attention unit 23123 performs multi-head attention processing by making the encoded speech represent the input sentence.

[0042] The gate unit 23124 (for convenience, also referred to as the "first gate unit") multiplies the output from the cross-attention unit 23123 by a value tanh(g1) based on the gate parameter g1 and outputs the result. tanh() is a hyperbolic tangent function. Here, the gate parameter g1 is a learnable parameter. Specifically, the gate parameter g1 is a parameter determined by learning with 0 as its initial value. In other words, the gate parameter g1 is a parameter that is learned with 0 as its initial value so as to gradually utilize the encoder output. In other words, the value multiplied in the gate unit 23124 is a value greater than or equal to 0 and less than or equal to 1, and is also referred to as the "first predetermined value."

[0043] The adder 23131 (also called the "first adder") adds the output from the gate unit 23124 and a value based on the output from the pre-trained self-attention unit 23121, and outputs the result. The output from the adder 23131 is passed to the feedforward network unit 23125.

[0044] The feedforward network unit 23125 receives the output from the adder 23131 (first adder) as an input value and performs feedforward network processing.

[0045] The gate unit 23126 (also referred to as the "second gate unit" for convenience) multiplies the output from the feedforward network unit 23125 by a value tanh(g2) based on the gate parameter g2, and outputs the result. Here, the gate parameter g2 is a learnable parameter. Specifically, the gate parameter g2 is a parameter that is determined by learning, with 0 as its initial value. In other words, the gate parameter g2 is a parameter that is learned so as to gradually utilize the encoder output, with 0 as its initial value. In other words, the value multiplied in the gate unit 23126 is a value greater than or equal to 0 and less than or equal to 1, and is also referred to as the "second predetermined value."

[0046] Adder 23132 (also called "second adder") adds the output from gate 23126 and the output from adder 23131 (first adder), and outputs the result.

[0047] Note that for each of the gate parameters g1 and g2, two types of gate parameters are prepared, one for the speech recognition task and one for the subsequent task, so that they can be trained separately. That is, the gate parameter g1 actually has two types: a gate parameter g11 for the speech recognition task and a gate parameter g12 for the subsequent task. Furthermore, the gate parameter g2 actually has two types: a gate parameter g21 for the speech recognition task and a gate parameter g22 for the subsequent task. By dividing the gate parameters in this way, it is possible to change the extent to which the output from the encoder is used for the speech recognition task and the subsequent task, respectively.

[0048] By using the gating mechanism described above, the large-scale language model processing unit 23 in this embodiment can perform training for a speech recognition task while preventing the forgetting of the capabilities acquired through pre-training performed using only text. Note that the pre-training performed using only text and the training for a speech recognition task (training performed using a pair of input speech and text) will be described later.

[0049] Next, we will explain the learning process and the inference process of the speech language processing device 1. The speech language processing device 1 can perform processes in the following modes [1], [2], and [3]. For example, a control unit (not shown) in the speech language processing device 1 may control which mode it operates in. [1] Preliminary learning using textbooks only [2] Learning by pairing audio and text [3] Reasoning

[0050] [1] Preliminary learning using textbooks only First, pre-training using only text will be described. Specifically, the large-scale language model processing unit 23 is trained using training data consisting only of text. In this pre-training, only the large-scale language model processing unit 23 (decoder unit) is trained. Note that the speech language processing apparatus 1 may be configured using a large-scale language model that has already been pre-trained, that is, by importing pre-trained model parameter values ​​from an external source.

[0051] 4 is a schematic diagram showing the flow of pre-learning using only text in the large-scale language model processing unit 23. As shown in the figure, the data input to the large-scale language model processing unit 23 in pre-learning using only text is data combining a task prompt and text. The large-scale language model processing unit 23 calculates and outputs the next word in the output text based on the input data. Based on the error (loss) between the text output from the large-scale language model processing unit 23 (the next word in the text) and the correct text (the next word in the text), the values ​​of the internal parameters of the large-scale language model processing unit 23 are adjusted (updated) by backpropagation.

[0052] When pre-training is performed using only text, the large-scale language model processing unit 23 does not use the cross-attention unit 23123 (Figure 3), unlike a normal encoder-decoder type model.

[0053] The input data format for text-only pre-training is "task prompt [SOS] task input sentence, task answer sentence [EOS]." The task prompt is a word that indicates the type of task. For small language models, a predetermined task prompt is required. For sufficiently large language models, the task prompt does not need to be predetermined. For example, for a task translating a Japanese sentence into English, the task prompt can be a free natural language expression such as "Please translate into English" or "Please translate into English." [SOS] is a special token that means "start of sentence." [EOS] is a special token that means "end of sentence." In other words, the input data in the above format includes information on the task type, task input sentence, and task answer sentence. The task type can be language translation, a sentence summary, a response to a question, etc.

[0054] Examples of input data for pre-learning using only text are as follows. The first example is "Translation [SOS] Hello, Good afternoon [EOS]." In this example, "Translation" is the task prompt. This prompt instructs the translation of the input sentence. In this example, "Hello" is the task input sentence, and "Good afternoon" is the task answer sentence (translation sentence) corresponding to the task input sentence. The second example is "Dialogue [SOS] Hello, Hello, It's Nice Weather, isn't it? [EOS]." In this example, "Dialogue" is the task prompt. This prompt instructs the generation of an answer sentence (a sentence that forms a dialogue) that responds to the input sentence. In this example, "Hello" is the task input sentence, and "Hello, It's Nice Weather, isn't it?" is the task answer sentence (a response sentence in a dialogue) corresponding to the task input sentence.

[0055] The large-scale language model processing unit 23 may be trained to be able to handle multiple tasks, or may be trained to be able to handle only a specific task. In the case of a sufficiently large language model, the capabilities of the large-scale language model processing unit 23 do not necessarily need to be specialized for a specific task. Note that an existing large-scale language model (LLM) may be utilized, and the model may be fine-tuned to be able to handle a specific task.

[0056] In the case of pre-training using only text, training is performed on the decoder 231 side in Figure 2. In pre-training using only text, training is not performed on the encoder 221. Furthermore, because the encoder 221 is not used, input values ​​to V and K of the multi-head attention unit 2312 (cross attention) are not supplied from the encoder 221 side. In other words, in the case of pre-training using only text, the output from the masked multi-head attention unit 2311 is not passed to the multi-head attention unit 2312 (cross attention), but is passed directly only to the subsequent Add&Norm (addition and normalization) layer.

[0057] In other words, the large-scale language model processing unit 23 is trained in advance using training data provided as a set of a task prompt, text data of an input sentence, and a correct answer to the text data of an output sentence, while avoiding inputting encoded speech information as V (value) and K (key) of the cross-attention unit (23123).

[0058] [2] Learning by pairing audio and text Next, learning using pairs of speech and text will be described. In this learning, the encoder unit 22 and the large-scale language model processing unit 23 are trained using learning data provided as pairs of speech and text. Specifically, the large-scale language model processing unit 23, which has completed pre-learning using only text as described above in [1], is incorporated into an encoder-decoder type end-to-end speech recognition system, and processing is performed using a task prompt. In other words, the task prompt, the output from the encoder unit 22, and past outputs from the large-scale language model processing unit 23 are input to the large-scale language model processing unit 23. Then, the text of the speech recognition result + task answer is output.

[0059] 5 is a schematic diagram showing the flow of learning using pairs of speech and text in the speech language processing device 1. Here, the learning data is a set of speech data, a task prompt, and a correct answer in the output text. The correct answer in the output text is a text in which a transcription of the speech and a task answer are concatenated.

[0060] As shown in the figure, speech data, which is part of the training data, is input to the encoder unit 22. The encoder unit 22 processes the input speech using the internal parameter values ​​at that time and calculates an output. The output from the encoder unit 22 is encoded speech information. The output from the encoder unit 22 is passed to the large-scale language model processing unit 23. A task prompt, which is part of the training data, is also passed to the large-scale language model processing unit 23. The large-scale language model processing unit 23 outputs the next word in sequence based on the task prompt, the output from the encoder unit 22, and past outputs from the large-scale language model processing unit 23 itself. The large-scale language model processing unit 23 performs calculations using the internal parameters at that time. The string of words output from the large-scale language model processing unit 23 is the text of the speech recognition result and the text of the task answer sentence. Based on the error (loss) between the text (speech recognition result + task answer sentence) output from the large-scale language model processing unit 23 and the correct text, the internal parameter values ​​of the large-scale language model processing unit 23 and the encoder unit 22 are adjusted (updated) using the error backpropagation method.

[0061] An example of training data for speech-text pair training is as follows:

[0062] In the first example of training data, the task prompt is "English translation." This is a prompt that instructs the user to translate into English as a task. The text (correct answer text) in this first example is something like "This is a pen. This is a pen [EOS]." [EOS] is a special token that indicates the end of a sentence. In this first example, the audio data input to the encoder unit 22 corresponds to the audio when "This is a pen." is read aloud.

[0063] In the second example of the training data, the task prompt is "Dialogue." This is a prompt that instructs the task of calculating a response sentence that is valid as a dialogue in response to an input sentence. The text (correct answer text) in this second example is "This is a pen. You have a pen, don't you? [EOS]." As mentioned above, [EOS] is a special token. Also, in this second example, the voice data input to the encoder unit 22 corresponds to the voice when "This is a pen." is read out loud.

[0064] By performing the learning using pairs of speech and text described here, the large-scale language model processing unit 23 acquires speech recognition capabilities while utilizing the text generation capabilities (answer sentence generation capabilities) that it had acquired through pre-learning using only text.

[0065] That is, after speech recognition, the large-scale language model processing unit 23 generates text according to the given task prompt.

[0066] In other words, the encoder unit 22 and the large-scale language model processing unit 23 have been machine-learned using training data provided as a set of speech corresponding to the input sentence, a task prompt, a correct text for the input sentence, and a correct text for the output sentence.

[0067] The speech recognition process is expressed by the following equation: Voice recognition: P(y t | voice, prompt, y 1~t-1 ) In the above formula, y 1~t-1 is the past recognition result, and prompt is the task type represented by the task prompt.

[0068] Furthermore, the processing of tasks subsequent to speech recognition is expressed by the following equation. Task processing: P(y´ t | voice, prompt, y 1~t-1 ,y´ 1~t-1 ) In the above formula, y 1~t is the speech recognition result, and y´ 1~t-1 is the result of the answer text for that task.

[0069] That is, the spoken language processing apparatus 1 according to this embodiment enables input of a sentence by voice and can generate a task answer based on the voice.

[0070] [3] Reasoning Next, the processing during inference will be described. During inference, calculations are performed using the trained encoder unit 22 and the large-scale language model processing unit 23. That is, calculations are performed using internal parameters optimized by training.

[0071] FIG. 6 is a schematic diagram showing the processing flow during inference in the spoken language processing device 1. As shown in the figure, unknown speech data to be inferred is input to the encoder unit 22. The encoder unit 22 processes the input speech using trained internal parameter values ​​and calculates an output. The output from the encoder unit 22 is encoded speech information. The output from the encoder unit 22 is passed to the large-scale language model processing unit 23. A task prompt is also input to the large-scale language model processing unit 23. The large-scale language model processing unit 23 sequentially outputs the next word based on this task prompt, the output from the encoder unit 22, and past outputs from the large-scale language model processing unit 23 itself. The large-scale language model processing unit 23 performs calculations using internal parameters optimized by training. The string of words output from the large-scale language model processing unit 23 is the text of the speech recognition result and the text of the task answer sentence. The output unit 24 extracts only the task answer sentence from the text (speech recognition result+task answer sentence) output from the large-scale language model processing unit 23, and outputs it to the outside as final text.

[0072] 7 is a schematic diagram for explaining in more detail the processing flow of the large-scale language model processing unit 23 during inference processing. In the diagram, for convenience, the words (tokens) input and output by the large-scale language model processing unit 23 are denoted by symbols T1, T2, ..., T12. A series of information obtained by encoding speech is sequentially passed from the encoder unit 22 to the large-scale language model processing unit 23. In this example, the speech input to the encoder unit 22 is the speech of the utterance "This is a pen."

[0073] In this situation, the large-scale language model processing unit 23 receives a task prompt. The illustrated "English translation" (T1) is a task prompt that indicates that the input sentence should be translated into English. Following this task prompt, the large-scale language model processing unit 23 receives input of a special token, [SOS] (start of sentence) (T2). In response to this, the large-scale language model processing unit 23 outputs "kore" (T3). This "kore" (T3) is the first word in the speech recognition result. Next, the large-scale language model processing unit 23 inputs "kore" (T3) output by itself. By performing a calculation based on this, the large-scale language model processing unit 23 outputs "wa" (T4). This "wa" (T4) is the word following "kore" (T3) in the speech recognition result. Next, the large-scale language model processing unit 23 inputs the word string "kore" (T3)-"wa" (T4) output by itself. By performing a calculation based on this, the large-scale language model processing unit 23 outputs "pen" (T5). This "pen" (T5) is the next word after "kore" (T3)-"wa" (T4) in the speech recognition result. Next, the large-scale language model processing unit 23 inputs the word sequence "kore" (T3)-"wa" (T4)-"pen" (T5) output by the large-scale language model processing unit 23 itself. This continues in the same manner. As shown in the figure, the large-scale language model processing unit 23 sequentially outputs "kore" (T3)-"wa" (T4)-"pen" (T5)-"desu" (T6)-"." (period) (T7)-"This" (T8)-"is" (T9)-"a" (T10)-"pen" (T11)-[EOS] (T12). The last output [EOS] (T12) is a special token indicating the end of the sentence. In other words, when [EOS] (T12) is output, the output from the large-scale language model processing unit 23 ends.

[0074] In the above example, among the strings output by the large-scale language model processing unit 23, "kore" (T3)-"wa" (T4)-"pen" (T5)-"desu" (T6)-"." (period) (T7) corresponds to the speech recognition result. Furthermore, "This" (T8)-"is" (T9)-"a" (T10)-"pen" (T11) corresponds to the task response sentence. Since the task executed here was English translation, "This" (T8)-"is" (T9)-"a" (T10)-"pen" (T11) corresponds to the English translation of the speech-input sentence. It is also possible to output a response sentence other than a translation in response to a task prompt. Tasks other than translation include, but are not limited to, dialogue and summarization.

[0075] The output unit 24 can arrange the string of words output by the large-scale language model processing unit 23 that correspond to the answer sentence, and output the text "This is a pen" (task answer sentence, English translation result sentence) to the outside.

[0076] In this way, the trained speech language processing apparatus 1 can calculate a task answer sentence in response to an input sentence that has been input by voice.

[0077] 8 is a flowchart showing the procedure of inference processing by the trained speech language processing apparatus 1. The procedure will be explained below with reference to this flowchart.

[0078] First, in step S1, the speech input unit 21 inputs speech data to the encoder unit 22. The speech data is, for example, vector data of acoustic features at a predetermined time. The encoder unit 22 calculates an output value based on the input speech data using internal parameter values. The output from the encoder unit 22 is a vector obtained by encoding the speech data. The output from the encoder unit 22 is passed to the large-scale language model processing unit 23. Specifically, the output from the encoder unit 22 becomes the input values ​​to V and K of the multi-head attention unit 2312 shown in FIG. 2.

[0079] Next, in step S2, the output from the encoder unit 22, the task prompt, and past output (past word strings) from the large-scale language model processing unit 23 are input to the large-scale language model processing unit 23. Based on these inputs, the large-scale language model processing unit 23 uses internal parameters to calculate values ​​to be output.

[0080] Next, in step S3, the large-scale language model processing unit 23 outputs the next word. The next word is the word that follows the string of previous words input in step S2. The string of these output words forms a text that concatenates the speech recognition result and the task answer sentence.

[0081] Next, in step S4, the output unit 24 outputs the text of the task answer sentence.

[0082] Next, in step S5, the speech language processing device 1 determines whether or not the processing of all the speech data has been completed. If the processing of all the speech data has been completed (step S5: YES), the entire processing of this flowchart is terminated. If the processing of all the speech data has not been completed (step S5: NO), the process returns to step S1 to input and process the next speech data.

[0083] As described above, when the processing of all speeches is completed, the inference processing by the spoken language processing apparatus 1 is completed.

[0084] FIG. 9 is a block diagram showing an example of the internal configuration of the speech language processing device 1 of this embodiment. The speech language processing device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902, etc. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0085] At least some of the functions of the speech language processing device 1 in this embodiment can be realized by a computer and a program. In this case, the functions can be realized by recording a program for realizing the functions on a computer-readable recording medium and loading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. The program may also be designed to realize some of the functions described above, or may be capable of realizing the functions described above in combination with a program already stored in the computer system.

[0086] As described above, the speech language processing device 1 of this embodiment makes it possible to obtain a solution to a task in response to a sentence input by voice (input sentence). Furthermore, because the sentence is input by voice, the encoder unit 22 and the large-scale language model processing unit 23 perform processing while utilizing information such as prosody contained in the speech. In other words, the speech language processing device 1 of this embodiment can perform processing based on a wealth of information compared to processing based on an input sentence that is text only.

[0087] As an example, in a dialogue task, it is possible to output different answer sentences depending on the prosody, etc., even for the same input sentence. Specifically, it is possible to output different answer sentences when the input is made in a voice with prosody corresponding to the emotion of anger and when the input is made in a voice with prosody corresponding to the emotion of frankness. For example, when the voice input "I'm tired" is made with prosody corresponding to anger, a reply sentence of polite dialogue (e.g., "Please take a good rest") can be output, and when the voice input "I'm tired" is made with a voice with prosody corresponding to frankness, a reply sentence of friendly dialogue (e.g., "Please let me know if there's anything you'd like to talk about") can be output.

[0088] As another example, in the case of a translation task, it is possible to output different answer sentences (here, translated sentences) depending on the prosody, etc., even for the same input sentence. Specifically, for example, if a child's voice inputs the Japanese phrase "Mom where?", the translated sentence (English translation) "Mom...where?" can be output, and if an adult's voice inputs the Japanese phrase "Mom where?", the translated sentence (English translation) "Where is my mother?" can be output.

[0089] In this case, the capabilities of the large-scale language model processing unit 23 can be used as is without being impaired.

[0090] As described above, according to this embodiment, the speech language processing apparatus 1 that has been properly trained can solve downstream tasks as an inference process based on speech input while also utilizing information (non-linguistic information) such as prosody contained in the speech. In other words, the speech language processing apparatus 1 can generate a response sentence that also depends on prosody and the like. As an example, the speech language processing apparatus 1 can generate a response sentence (reply, etc.) customized for an individual.

[0091] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0092] The present invention can be used as, for example, a translation device, a response device, a dialogue device, a summarization device, or other language processing device that processes based on spoken language, but the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0093] 1 Speech and language processing device 21 Audio input section 22 Encoder section 23 Large-scale language model processing section (decoder section) 24 Output section 221 Encoder 231 decoder 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus 2211 Multi-head Attention Department 2212 Feedforward Network Section 2311 Multi-head attention unit with mask 2312 Multi-head attention section (cross attention) 2313 Feedforward Network Section 23121 Pre-trained self-attention section 23122 Pre-trained feedforward network part 23123 Cross Attention Department 23124 Gate section 23125 Feedforward Network Section 23126 Gate section 23131,23132 Addition section

Claims

1. an encoder unit that performs a process of encoding audio and outputs encoded audio information; a large-scale language model processing unit that outputs an output sentence based on an input task prompt and an input sentence, based on the input task prompt and the input sentence; Equipped with The large-scale language model processing unit a masked multi-head attention unit that processes multi-head attention based on the task prompt and past outputs from the large-scale language model processing unit; a cross-attention unit that receives the output from the masked multi-head attention unit as a query (Q) and the encoded speech information output from the encoder unit as a value (V) and a key (K), and performs multi-head attention processing by making the encoded speech represent the input sentence; a first gate unit that multiplies an output from the cross-attention unit by a first predetermined value that is equal to or greater than 0 and equal to or less than 1, and outputs the result; a first adder that adds and outputs the output from the first gate unit and the output from the masked multi-head attention unit; a feedforward network unit that receives the output from the first adder unit as an input, processes the feedforward network, and outputs the result of the processing; a second gate unit that multiplies an output from the feedforward network unit by a second predetermined value that is equal to or greater than 0 and equal to or less than 1, and outputs the result; a second adder that adds the output from the second gate unit and the output from the first adder and outputs the result; Equipped with Each of the first predetermined value and the second predetermined value is a value determined by learning using learning data with an initial value set to 0. Speech language processor.

2. The large-scale language model processing unit The system is trained in advance using training data provided as a set of the task prompt, the text data of the input sentence, and the correct answer to the text data of the output sentence, while not inputting the encoded voice information as V (value) and K (key) of the cross-attention section. The speech language processing device according to claim 1 .

3. The encoder unit and the large-scale language model processing unit The system is trained using training data provided as a set of a speech corresponding to an input sentence, the task prompt, a textual correct answer for the input sentence, and a textual correct answer for the output sentence. The speech language processing device according to claim 2 .

4. an encoder unit that performs a process of encoding audio and outputs encoded audio information; a large-scale language model processing unit that outputs an output sentence based on an input task prompt and an input sentence, based on the input task prompt and the input sentence; Equipped with The large-scale language model processing unit a masked multi-head attention unit that processes multi-head attention based on the task prompt and past outputs from the large-scale language model processing unit; a cross-attention unit that receives the output from the masked multi-head attention unit as a query (Q) and the encoded speech information output from the encoder unit as a value (V) and a key (K), and performs multi-head attention processing by making the encoded speech represent the input sentence; a first gate unit that multiplies an output from the cross-attention unit by a first predetermined value that is equal to or greater than 0 and equal to or less than 1, and outputs the result; a first adder that adds and outputs the output from the first gate unit and the output from the masked multi-head attention unit; a feedforward network unit that receives the output from the first adder unit as an input, processes the feedforward network, and outputs the result of the processing; a second gate unit that multiplies an output from the feedforward network unit by a second predetermined value that is equal to or greater than 0 and equal to or less than 1, and outputs the result; a second adder that adds the output from the second gate unit and the output from the first adder and outputs the result; Equipped with Each of the first predetermined value and the second predetermined value is a value determined by learning using learning data with an initial value set to 0. A program that causes a computer to function as a speech language processor.

Images