Speech dialogue device, speech dialogue method, and program

Abstract

To provide a speech synthesis technique that can realize delicate pitch variation in a system utterance.SOLUTION: A speech interaction device includes: a word series generation part which generates a word series of a user utterance from a speech of the user utterance; a prosodic information series generation part which generates a series of prosodic information (hereinafter, referred to asa prosodic information series) for each of words of the user utterance from the speech of the user utterance and the word series; a system utterance generation part which generates a word series and a prosodic control signal series of the system utterance from the word series and prosodic information series of the user utterance through a system utterance model constituted as a neural network which inputs the word series and prosodic information series of the user utterance and outputs a word series of the system utterance and a series of a prosodic control signal of each word (hereinafter referred to as a prosodic control signal series); and a speech synthesis part which synthesizes a speech of the system utterance from the word series of the system utterance and the prosodic control signal series.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice dialogue technology. [Background technology]

[0002] Speech dialogue technology is a technology for generating speech when a system such as a robot is allowed to have a speech dialogue with a human. In order to generate natural speech as a dialogue, there are methods for controlling features such as the average power of the speech, the range of prosodic features, and the speech rate based on the history of past dialogues. For example, the technology described in Non-Patent Document 1 controls prosodic features at the sentence level. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] S. Fuscone et al., “Neural representations of dialogical history for improving upcoming turn acoustic parameters prediction,” in Proc. INTERSPEECH, 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology in Non-Patent Document 1 only controls rough features at the utterance level, such as features at the sentence level, and therefore has the problem of being unable to reproduce the subtle pitch changes heard in real human conversations.

[0005] Therefore, an object of the present invention is to provide a voice synthesis technique that realizes fine pitch changes in system utterances. [Means for solving the problem]

[0006] One aspect of the present invention includes a word sequence generation unit that generates a word sequence of a user utterance from a voice of the user utterance, a prosodic information sequence generation unit that generates a sequence of prosodic information for each word of the user utterance (hereinafter referred to as a prosodic information sequence) from the voice of the user utterance and the word sequence, a system utterance generation unit that generates a word sequence and a prosodic control signal sequence of a system utterance from the word sequence and prosodic information sequence of the user utterance using a system utterance generation model configured as a neural network that receives as input the word sequence and prosodic information sequence of the user utterance and outputs a word sequence of a system utterance and a sequence of a prosodic control signal for each word (hereinafter referred to as a prosodic control signal sequence), and a speech synthesis unit that synthesizes the speech of the system utterance from the word sequence and prosodic control signal sequence of the system utterance. Effect of the Invention

[0007] According to the present invention, it is possible to realize fine pitch changes in system utterances. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a voice dialogue device 100. FIG. [Diagram 2] 4 is a flowchart showing an operation of the voice dialogue device 100. [Diagram 3] FIG. 2 is a diagram illustrating a configuration of a system utterance generation model. [Figure 4] FIG. 2 is a diagram illustrating an example of a system utterance generation model. [Diagram 5] FIG. 2 is a diagram illustrating a configuration of a system utterance generation model. [Figure 6] FIG. 2 is a diagram illustrating an example of a system utterance generation model. [Figure 7] FIG. 2 is a diagram illustrating an example of a functional configuration of a computer that realizes each device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail. Note that components having the same functions are given the same reference numbers and duplicated explanations will be omitted.

[0010] Before describing each embodiment, the notation used in this specification will be described.

[0011] ^ (caret) represents a superscript. For example, x y^z Yes z is a superscript to x, and x y^z Yes z is a subscript to x. Also, _ (underscore) represents a subscript. For example, x y_z Yes z is a superscript to x, and x y_z Yes z is a subscript on x.

[0012] The superscripts "^" and "~" such as ^x and ~x for a certain letter x should actually be written directly above the "x", but due to the constraints of the description in the specification, they are written as ^x and ~x.

[0013] First Embodiment The voice dialogue device 100 receives a user's speech and synthesizes a system's speech using a system's speech generation model. Here, the system's speech generation model is a function that receives a user's speech word sequence and a prosodic information sequence for each word (hereinafter referred to as a prosodic information sequence) and outputs a system's speech word sequence and a prosodic control signal sequence for each word (hereinafter referred to as a prosodic control signal sequence), and can be configured as a neural network, for example.

[0014] The voice dialogue device 100 will be described below with reference to Figs. 1 and 2. Fig. 1 is a block diagram showing the configuration of the voice dialogue device 100. Fig. 2 is a flowchart showing the operation of the voice dialogue device 100. As shown in Fig. 1, the voice dialogue device 100 includes a word sequence generation unit 110, a prosodic information sequence generation unit 120, a system utterance generation unit 130, a voice synthesis unit 140, and a recording unit 190. The recording unit 190 is a component that appropriately records information required for processing by the voice dialogue device 100. The recording unit 190 records, for example, parameters of a system utterance generation model in advance.

[0015] The operation of the voice dialogue device 100 will be described with reference to FIG.

[0016] In S110, the word sequence generation unit 110 receives a user's speech and generates and outputs a word sequence of the user's speech from the user's speech. The word sequence generation unit 110 generates a word sequence, which is a sequence of words included in the user's speech, using, for example, a conventional speech recognition technology. If the result of recognizing the user's speech is not divided into words, the word sequence may be divided into words using, for example, morphological analysis.

[0017] In S120, the prosodic information sequence generation unit 120 receives the user's speech and the word sequence of the user's speech generated in S110 as input, and generates and outputs a prosodic information sequence of the user's speech from the user's speech and the word sequence. For example, an average of logarithm F0 for each word (hereinafter referred to as average logarithm F0) can be used as the prosodic information. The prosodic information sequence generation unit 120 generates an average logarithm F0 sequence of the user's speech using, for example, a conventional acoustic feature extraction technique. Specifically, the prosodic information sequence generation unit 120 performs word alignment for the user's speech based on the word sequence of the user's speech generated in S110, that is, temporal correspondence between speech and words, generates a logarithm F0 sequence by logarithmic transformation from the fundamental frequency F0 sequence extracted from the user's speech, and calculates the average logarithm F0 for each word based on the time information obtained by the word alignment, thereby generating an average logarithm F0 sequence of the user's speech.

[0018] In S130, the system utterance generation unit 130 receives the word sequence of the user utterance generated in S110 and the prosodic information sequence of the user utterance generated in S120 as input, and generates and outputs a word sequence of the system utterance and a prosodic control signal sequence from the word sequence and prosodic information sequence of the user utterance using a system utterance generation model. When the prosodic information is the average logarithm F0, for example, the difference of the average logarithm F0 (hereinafter referred to as differential F0 context) can be used as the prosodic control signal.

[0019] When the system utterance generation model is configured as a neural network, it may include a pair of encoders and decoders as shown in FIG. 3. The encoder receives a word sequence of a user utterance and an average logarithmic F0 sequence as input, and outputs a latent variable. Here, the latent variable is expressed as a vector of a predetermined dimension. On the other hand, the decoder receives a latent variable as input, and outputs a word sequence of a system utterance and a differential F0 context sequence. FIG. 4 is a diagram showing an example of a system utterance generation model configured as a neural network including a pair of encoders and decoders. FIG. 4 shows an example in which a word sequence of a user utterance, “I won the game”, is input to the system utterance generation model, and a word sequence of a system utterance, “you did it”, is output. Note that LF0 represents prosodic information for each word, and ΔLF0 represents a prosodic control signal for each word. As shown in FIG. 4, the system utterance generation model may include a pair of encoders and decoders equipped with an attention layer.

[0020] In addition, when the system utterance generation model is configured as a neural network, it may include two sets of an encoder and a decoder (hereinafter, referred to as a first encoder and a first decoder, and a second encoder and a second decoder, respectively) as shown in FIG. 5. The first encoder receives a word sequence of a user utterance as an input and outputs a first latent variable, and the first decoder receives the first latent variable and outputs a word sequence of a system utterance. The second encoder receives a word sequence of a user utterance and an average logarithmic F0 sequence, which are inputs of the first encoder, as an input and outputs a second latent variable, and the second decoder receives the second latent variable and a word sequence of a system utterance output by the first decoder, and outputs a differential F0 context sequence of the system utterance. FIG. 6 is a diagram showing an example of a system utterance generation model configured as a neural network including two sets of an encoder and a decoder. As in FIG. 4, FIG. 6 shows an example in which a word sequence of a user utterance, "I won the game", is input to the system utterance generation model and a word sequence of a system utterance, "you did it", is output. As shown in FIG. 6, the system speech generation model may include two sets of encoders and decoders, each with an attention layer.

[0021] When the system utterance generation model is configured as a neural network, the system utterance generation model is trained in advance using training data generated based on speech in a dialogue between people. When the dialogue between people is a two-party dialogue, the preceding utterance corresponds to the user's utterance, the subsequent utterance corresponds to the system's utterance, and the training data is a set of a word sequence and an average logarithm F0 sequence of the preceding utterance, and a word sequence and a differential F0 context sequence of the subsequent utterance. Here, the differential F0 context is the difference between the average logarithm F0 obtained from the actual utterance and the average logarithm F0 obtained from a synthetic voice generated by the conventional technology and having the same speech content as the actual utterance. Note that instead of a two-party dialogue, it is also possible to train using a dialogue between multiple speakers. In this case, it is advisable to include the conventional technology that generates synthetic voice as a speaker and normalize the F0 of each speaker by the average value for each speaker.

[0022] In S140, the speech synthesis unit 140 receives the word sequence of the system utterance generated in S130 and the prosodic control signal sequence, synthesizes and outputs a speech of the system utterance from the word sequence of the system utterance and the prosodic control signal sequence. When the prosodic control signal is a differential F0 context, the speech synthesis unit 140 can use, for example, a speech synthesis technique using a differential acoustic model described in Reference Non-Patent Document 1.

[0023] (Reference Non-Patent Document 1: S. Yamada et al., “A Study on Tailor-Made Speech Synthesis Based on Deep Neural Networks,” in Proc. IIH-MSP, pp.159-166, 2016.) According to an embodiment of the present invention, it is possible to realize minute pitch changes in system speech. In particular, by learning word-level pitch changes using human dialogue, it is possible to synthesize natural speech that takes into account phenomena seen in human dialogue (e.g., emphasis or synchronization of specific words).

[0024] <Additional Notes> 7 is a diagram showing an example of a functional configuration of a computer 2000 that realizes each of the above-mentioned devices. The processing in each of the above-mentioned devices can be implemented by having a recording unit 2020 load a program for causing the computer 2000 to function as each of the above-mentioned devices, and having the control unit 2010, the input unit 2030, the output unit 2040, and the like operate.

[0025] The device of the present invention has, as a single hardware entity, an input section to which a keyboard or the like can be connected, an output section to which a liquid crystal display or the like can be connected, a communication section to which a communication device (e.g., a communication cable) capable of communicating with the outside of the hardware entity can be connected, a CPU (which may also have a central processing unit, cache memory, registers, etc.), memories such as RAM and ROM, an external storage device such as a hard disk, and buses connecting the input section, output section, communication section, CPU, RAM, ROM, and external storage device so that data can be exchanged between them. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing a recording medium such as a CD-ROM. A physical entity equipped with such hardware resources is, for example, a general-purpose computer.

[0026] The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required in the processing of these programs (not limited to the external storage device, but for example the programs may be stored in a ROM, which is a read-only storage device). Data obtained by the processing of these programs is stored appropriately in the RAM, the external storage device, etc.

[0027] In the hardware entity, each program stored in an external storage device (or ROM, etc.) and the data required for processing each program are loaded into memory as necessary, and interpreted, executed, and processed by the CPU as appropriate, so that the CPU realizes a predetermined function (each component represented as the above, "unit," "means," etc.).

[0028] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention. In addition, the processes described in the above-described embodiment are not limited to being executed in chronological order according to the order described, but may be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as necessary.

[0029] As described above, when the processing functions of the hardware entities (the devices of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entities should have are described by a program. Then, by executing this program on a computer, the processing functions of the hardware entities are realized on the computer.

[0030] The program describing the processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium. Specifically, for example, a hard disk drive, a flexible disk, a magnetic tape, or the like can be used as the magnetic recording device, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), or the like can be used as the optical disk, an MO (Magneto-Optical disc), or the like can be used as the magneto-optical recording medium, and an EEP-ROM (Electronically Erasable and Programmable-Read Only Memory) or the like can be used as the semiconductor memory.

[0031] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.

[0032] A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its own storage device and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. The server computer may not transfer the program to the computer, but may execute the above-mentioned process by a so-called ASP (Application Service Provider) type service that realizes the processing function only by issuing an execution instruction and obtaining the result. Note that the program in this embodiment includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has a nature that specifies the processing of the computer, etc.).

[0033] In addition, in this embodiment, a hardware entity is configured by executing a specific program on a computer, but at least a part of the processing contents may be realized by hardware.

[0034] The foregoing description of the embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings. The embodiments have been chosen and depicted to provide a best illustration of the principles of the invention and to enable those skilled in the art to utilize the invention in various embodiments and with various modifications as may be suitable for the practical use contemplated. All such modifications and variations are within the scope of the invention as defined by the appended claims interpreted in accordance with the breadth to which they are fairly, legally, and equitably entitled.

Claims

1. a word sequence generation unit that generates a word sequence of a user utterance from a voice of the user utterance; a prosodic information sequence generating unit for generating a sequence of prosodic information for each word of the user utterance (hereinafter referred to as a prosodic information sequence) from the user utterance and a word sequence; a system utterance generation unit that receives as input a word sequence and a prosodic information sequence of a user utterance, and generates a word sequence of a system utterance and a prosodic control signal sequence from the word sequence and prosodic information sequence of the user utterance, using a system utterance generation model configured as a neural network that outputs a word sequence of a system utterance and a prosodic control signal sequence for each word (hereinafter referred to as a prosodic control signal sequence); a speech synthesis unit that synthesizes a speech of the system utterance from a word sequence of the system utterance and a prosody control signal sequence; Including, the system utterance generation model is trained using training data generated based on speech in a two-party dialogue between humans; The preceding utterance and the subsequent utterance in the two-party dialogue are respectively a user utterance and a system utterance, and the learning data is a set of a word sequence and a prosodic information sequence of the preceding utterance, and a word sequence and a prosodic control signal sequence of the subsequent utterance. Speech dialogue device.

2. 2. The speech dialogue device according to claim 1, the prosodic information is a mean logarithm F0, The prosodic control signal is a differential F0 context 1. A voice dialogue device comprising:

3. 3. The speech dialogue device according to claim 2, The neural network includes a pair of an encoder and a decoder; The encoder receives a word sequence of a user utterance and an average logarithmic F0 sequence as input, and outputs a latent variable. The decoder receives the latent variables as input and outputs a word sequence and a differential F0 context sequence of the system utterance.

1. A voice dialogue device comprising:

4. 3. The speech dialogue device according to claim 2, The neural network includes two sets of an encoder and a decoder (hereinafter, referred to as a first encoder and a first decoder, and a second encoder and a second decoder, respectively), The first encoder receives a word sequence of a user utterance as input and outputs a first latent variable; the first decoder receives the first latent variable as an input and outputs a word sequence of a system utterance; The second encoder receives the word sequence of the user utterance and the average logarithmic F0 sequence as input, and outputs a second latent variable; The second decoder receives the second latent variables and a word sequence of the system utterance as input, and outputs a differential F0 context sequence of the system utterance.

1. A voice dialogue device comprising:

5. a word sequence generating step of generating a word sequence of a user utterance from a voice of the user utterance by the speech dialogue device; a prosodic information sequence generating step of the speech dialogue apparatus generating a prosodic information sequence for each word of the user utterance (hereinafter referred to as a prosodic information sequence) from the voice of the user utterance and a word sequence; a system utterance generating step in which the speech dialogue apparatus generates a word sequence of a system utterance and a prosodic control signal sequence from the word sequence of the user utterance and the prosodic information sequence by using a system utterance generation model configured as a neural network that receives as input a word sequence and a prosodic information sequence of a user utterance and outputs a word sequence of a system utterance and a prosodic control signal sequence for each word (hereinafter referred to as a prosodic control signal sequence); a speech synthesis step of synthesizing a speech of the system utterance from a word sequence of the system utterance and a prosodic control signal sequence by the speech dialogue device; Including, the system utterance generation model is trained using training data generated based on speech in a two-party dialogue between humans; The preceding utterance and the subsequent utterance in the two-party dialogue are respectively a user utterance and a system utterance, and the learning data is a set of a word sequence and a prosodic information sequence of the preceding utterance, and a word sequence and a prosodic control signal sequence of the subsequent utterance. Voice interaction method.

6. 5. A program for causing a computer to function as the speech dialogue device according to claim 1.

Images