Speech synthesis device, speech synthesis method, and speech synthesis program

Abstract

A voice synthesis device according to the present disclosure includes a voice data acquisition unit, an utterance information acquisition unit, a book information acquisition unit, and a model learning unit. The utterance information acquisition unit acquires utterance information relating to an object to be uttered that is text included in a first book. The book information acquisition unit acquires image information relating to an image included in the first book. The voice data acquisition unit acquires voice data corresponding to the object to be uttered. The model learning unit generates a voice synthesis model for reading out a second book including text associated with the image on the basis of the utterance information acquired by the utterance information acquisition unit, the image information acquired by the book information acquisition unit, and the voice data acquired by the voice data acquisition unit.

Description

[Technical field]

[0001] The present disclosure relates to a voice synthesis device, a voice synthesis method, and a voice synthesis program. [Background technology]

[0002] In recent years, speech synthesis techniques based on DNNs (Deep Neural Networks) have been proposed in the field of speech synthesis. It is known that speech synthesis techniques based on DNNs can generate synthetic speech of higher quality than synthetic speech obtained by conventional methods (see Non-Patent Documents 1 and 2 below). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Zen, Heiga, Andrew Senior, and Mike Schuster. "STATISTICAL PARAMETRIC SPEECH SYNTHESIS USING DEEP NEURAL NETWORKS." Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013. [Non-Patent Document 2] Shen, Jonathan, et al. "NATURAL TTS SYNTHESIS BY CONDITIONING WAVENET ON MEL SPECTROGRAM PREDICTIONS." 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above-mentioned prior art, it may be difficult to read a book including images using natural synthetic voice.

[0005] For example, a book containing images is a picture book. Compared to a reading voice provided by a narrator reading a picture book, the above prior art has a difference in naturalness of intonation, etc. One factor of this difference is that the above prior art generates synthetic speech from linguistic information such as pronunciation and accent obtained from the text of the picture book.

[0006] When a narrator reads a picture book, the narrator uses a variety of information to speak, including not only linguistic information but also visual information obtained from illustrations (e.g., descriptions of characters and backgrounds) and the characters' feelings inferred from the long-term context.

[0007] Therefore, the present disclosure proposes a voice synthesis device, a voice synthesis method, and a voice synthesis program that are capable of reading a book including images in a natural synthetic voice. [Means for solving the problem]

[0008] In one aspect of the present disclosure, a speech synthesis device includes an acquisition unit that acquires speech information related to a speech target, which is text included in a first book, image information related to an image included in the first book, and speech data corresponding to the speech target, and a generation unit that generates a speech synthesis model for reading aloud a second book including text associated with an image based on the speech information, image information, and speech data acquired by the acquisition unit. Effect of the Invention

[0009] A speech synthesizer according to one or more embodiments of the present disclosure can read books that include images in a natural synthetic voice. [Brief description of the drawings]

[0010] [Figure 1]FIG. 1 is a block diagram of an example environment for speech synthesis. [Diagram 2] FIG. 2 illustrates an example of the structure of a speech synthesis model according to the present disclosure. [Figure 3A] FIG. 3A shows an overview of the speech synthesis process according to the present disclosure. [Figure 3B] FIG. 3B shows an overview of the speech synthesis process according to the present disclosure. [Figure 3C] FIG. 3C shows an overview of the speech synthesis process according to the present disclosure. [Figure 3D] FIG. 3D shows an overview of the speech synthesis process according to the present disclosure. [Figure 4] FIG. 4 is a block diagram of an example of the configuration of a voice synthesis device according to the present disclosure. [Diagram 5] FIG. 5 illustrates an example of utterance information according to the present disclosure. [Figure 6] FIG. 6 shows an example of book information according to the present disclosure. [Figure 7] FIG. 7 illustrates an example of training a speech synthesis model according to the present disclosure. [Figure 8] FIG. 8 illustrates an example of speech synthesis according to the present disclosure. [Figure 9] FIG. 9 is a flowchart illustrating an example of a process for generating a voice synthesis model. [Figure 10] FIG. 10 shows an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Several embodiments are described in detail below with reference to the drawings. However, the present invention is not limited to these several embodiments. The features of various embodiments can be combined in various ways, provided that the features are not mutually contradictory. The same elements are indicated by the same reference numerals, and duplicated descriptions are omitted.

[0012] [1. Environment for speech synthesis] First, an environment for speech synthesis according to the present disclosure will be described with reference to FIG.

[0013] 1 is a block diagram of an example of an environment for speech synthesis, environment 1. As shown in FIG. 1, environment 1 includes a speech synthesizer 100, a network 200, and a user device 300.

[0014] The speech synthesis device 100 is a device that performs one or more speech synthesis processes. The one or more speech synthesis processes include a process of generating a speech synthesis model and a process of generating synthetic speech using the generated speech synthesis model. An overview of the speech synthesis process according to the present disclosure will be described in the next section.

[0015] The speech synthesizer 100 is a data processing device such as a server. An example of the configuration of the speech synthesizer 100 will be described in Section 4.

[0016] The network 200 is, for example, a local area network (LAN), a wide area network (WAN), the Internet, etc. The network 200 connects the voice synthesizer 100 and the user device 300.

[0017] The user device 300 is a data processing device such as a client device. When a user wants to obtain a voice synthesis model, the user device 300 provides training data for the voice synthesis model to the voice synthesis device 100. The generated voice synthesis model is then provided to the user device 300 from the voice synthesis device 100.

[0018] Furthermore, when a user wishes to convert a book (e.g., an electronic book) into an audiobook, data relating to the book is provided to the voice synthesizer 100. In this case, a synthetic voice reading the book is provided to the user device 300 from the voice synthesizer 100.

[0019] [2. Structure of the speech synthesis model] Next, an example of the structure of a voice synthesis model according to the present disclosure will be described with reference to FIG.

[0020] 2 shows a model structure 10, which is an example of a model structure of a voice synthesis model according to the present disclosure. The voice synthesis model according to the present disclosure is implemented, for example, by a neural network. The model structure 10 is shown as a neural network structure for voice synthesis according to the present disclosure.

[0021] Neural networks have been used conventionally to implement speech synthesis models. A conventional neural network for speech synthesis has one input, which is a language vector obtained from text information contained in a book (see Non-Patent Document 2 above).

[0022] In contrast, the model structure 10 in Fig. 2 has two inputs: an input layer 11 and a visual information extraction layer 12, which are major differences between the neural network structure for speech synthesis according to the present disclosure and the neural network structure for speech synthesis according to the conventional disclosure.

[0023] The first input of the model structure 10 is a language vector, similar to a conventional neural network structure for speech synthesis. In the example of FIG. 2, the language vector is obtained by vectorizing the speech information extracted from the picture book 13. The speech information is information on the target of the speech. The target of the speech in the picture book 13 is a sentence contained in the picture book 13.

[0024] The second input of the model structure 10 is a visual feature vector, which is not present in conventional neural network structures for speech synthesis. In the example of FIG. 2, the visual feature vector is obtained by vectorizing illustration image information 14 extracted from a picture book 13. The illustration image information 14 is information on an image of an illustration. The image of an illustration in the picture book 13 is a picture included in the picture book 13.

[0025] The output of the visual information extraction layer 12 is, for example, input to a decoder layer (solid arrow), or may be input to an encoder layer (dashed arrow) depending on the implementation of the neural network.

[0026] [3. Overview of speech synthesis processing] Next, an overview of a voice synthesis process according to the present disclosure will be described with reference to Figures 3A, 3B, 3C, and 3D. The voice synthesis process described in this section includes a process of generating a neural network for voice synthesis, and this neural network has the model structure 10 described above with reference to Figure 2. Note that this overview is not intended to limit the present invention or the embodiments described in the following sections.

[0027] 3A, 3B, 3C and 3D collectively show an overview 20 of a speech synthesis process according to the present disclosure. The overview 20 includes eight steps.

[0028] 3A, in step S1, the speech synthesizer 100 of FIG. 1 acquires a speech signal 22 of a picture book 21 being read aloud.

[0029] In step S2, the voice synthesizer 100 generates voice data 23 from the voice signal 22. The voice data 23 includes voice parameters (for example, fundamental frequency) and spectral parameters (for example, mel spectrogram) of the voice signal 22.

[0030] 3B, in step S3, the speech synthesizer 100 extracts the speech information 24 from the picture book 21. In the example of Fig. 3B, a certain page of the picture book 21 includes the sentence "Good morning." Therefore, the speech information 24 includes the character string "Good morning."

[0031] In step S4, the speech synthesizer 100 extracts the illustration image information 25 from the picture book 21. In the example of Fig. 3B, the page including the above sentence includes a picture of the sun. Therefore, the illustration image information 25 includes an image of the sun.

[0032] 3C, in step S5, the speech synthesizer 100 vectorizes the speech information 24 and the illustration image information 25. In the example of FIG. 3C, the speech synthesizer 100 converts the speech information 24 into a language vector 26. The speech synthesizer 100 also converts the illustration image information 25 into a visual feature vector 27.

[0033] In step S6, the speech synthesizer 100 trains a neural network for speech synthesis. The speech synthesizer 100 uses the language vector 26 and the visual feature vector 27 obtained in step S5 as input of training data. The speech synthesizer 100 also uses the speech data 23 obtained in step S2 as output of training data. As a result, the speech synthesizer 100 generates a speech synthesis model 28.

[0034] 3D, in step S7, the speech synthesizer 100 generates a language vector 26a and a visual feature vector 27a from the picture book 21a that is the target of speech synthesis. The picture book 21a is an unknown picture book different from the picture book 21.

[0035] In step S8, the speech synthesis device 100 generates synthetic speech for reading the picture book 21a. First, the speech synthesis device 100 inputs the language vector 26a and the visual feature vector 27a to the speech synthesis model 28 to acquire speech features. Then, the speech synthesis device 100 generates a speech waveform from the speech features to generate synthetic speech.

[0036] As described above, the speech synthesis device 100 utilizes illustration image information 25 in speech synthesis for books such as picture books. Conventional speech synthesis techniques use linguistic information such as reading and accent as input to a neural network for speech synthesis. In contrast, the speech synthesis device 100 also utilizes visual information obtained from books such as picture books as input to a neural network for speech synthesis. Therefore, the speech synthesis device 100 can generate synthetic speech by taking into account information contained in illustrations.

[0037] 4. Configuration of the speech synthesis device Next, an example of the configuration of the voice synthesizer 100 will be described with reference to FIG.

[0038] Fig. 4 is a block diagram of a voice synthesizer 100 which is an example of the configuration of a voice synthesizer according to the present disclosure. As shown in Fig. 4, the voice synthesizer 100 includes a communication unit 110, a control unit 120, and a storage unit 130. The voice synthesizer 100 may include an input unit (e.g., a keyboard, a mouse) that receives input from an administrator of the voice synthesizer 100. The voice synthesizer 100 may also include an output unit (e.g., a liquid crystal display, an organic EL (Electro Luminescence) display) that displays information to the administrator of the voice synthesizer 100.

[0039] [4-1. Communication unit 110] The communication unit 110 is implemented by, for example, a network interface card (NIC). The communication unit 110 is connected to the network 200 by wire or wirelessly. The communication unit 110 can transmit and receive information to and from the user device 300 via the network 200.

[0040] 4-2. Control Unit 120 The control unit 120 is a controller. The control unit 120 is implemented by one or more processors (e.g., a central processing unit (CPU), a micro processing unit (MPU)) that use a random access memory (RAM) as a working area and execute various programs stored in a storage device of the voice synthesizer 100. The control unit 120 may also be implemented by an integrated circuit such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a general purpose graphic processing unit (GPGPU).

[0041] 4, the control unit 120 includes a voice data acquisition unit 121, an utterance information acquisition unit 122, a book information acquisition unit 123, a vector representation acquisition unit 124, a visual feature extraction unit 125, a model learning unit 126, and a voice synthesis unit 127. One or more processors of the voice synthesizer 100 can implement each control unit by executing instructions stored in one or more memories of the voice synthesizer 100. The data processing performed by each control unit is an example, and each control unit (e.g., a model learning unit) may perform data processing described in relation to other control units (e.g., a model learning unit).

[0042] The voice data acquisition unit 121, the speech information acquisition unit 122, and the book information acquisition unit 123 are examples of an "acquisition unit." The vector expression acquisition unit 124 is an example of a "first conversion unit." The visual feature extraction unit 125 is an example of a "second conversion unit." The model learning unit 126 is an example of a "generation unit."

[0043] [4-2-1. Voice data acquisition unit 121] The voice data acquisition unit 121 acquires voice data corresponding to a speech target in a book. The speech target is text included in the book. Examples of books include picture books and picture-story shows. For example, the speech target is text included in a specific page of the book. This text is associated with an image included in the specific page.

[0044] The voice data includes pre-recorded voices for use in training a voice synthesis model. The voice data holds voices including the speech of a narrator who reads out text included in book information (i.e., text included in a book) described below. The voice data is obtained by performing signal processing on a voice signal emitted by the narrator. The voice data holds voice parameters (e.g., pitch parameters such as fundamental frequency) and spectral parameters (e.g., mel spectrogram, cepstrum, mel cepstrum).

[0045] The voice data acquisition unit 121 can receive voice data from the user device 300. The voice data acquisition unit 121 can store the received voice data in the storage unit 130. The voice data acquisition unit 121 can acquire the voice data from the storage unit 130.

[0046] 4-2-2. Speech information acquisition unit 122 The speech information acquisition unit 122 acquires speech information related to the speech target. The speech information corresponds to the voice data acquired by the voice data acquisition unit 121. The speech information includes text information included in book information described later. The text information indicates the text included in the book.

[0047] As described below, the speech information may include information indicating the accent, part of speech, start time of a phoneme, or end time of a phoneme of the target speech.

[0048] The speech information includes pronunciation information assigned to each utterance in the audio data. This speech information is assigned to each utterance in the audio data acquired by the audio data acquisition unit 121. The speech information may include at least text information included in the book information described later.

[0049] The speech information added to the voice data may include information other than text information. For example, the speech information may include accent information (accent type, accent phrase length), part of speech information, and information on the start time or end time of each phoneme (phoneme segmentation information). The start time and end time are the elapsed time when the start point of each utterance is 0 [seconds].

[0050] FIG. 5 shows utterance information 30, which is an example of utterance information according to the present disclosure. As shown in FIG. 5, the utterance information 30 includes a character string "Good morning." In addition, an illustration number included in the book information described below is assigned to each utterance. In the example of FIG. 5, the utterance "O", the utterance "O", the utterance "HA", the utterance "YO", and the utterance "U" correspond to the illustration number "1". Each utterance is associated with a corresponding illustration number.

[0051] The illustration number is included in the book information, which will be described later, and indicates the correspondence between the speech information and the illustration. A unique ID (identifier), such as a number, is assigned to each illustration.

[0052] Returning to Fig. 4, the utterance information acquisition unit 122 can receive utterance information from the user device 300. The utterance information acquisition unit 122 can store the received utterance information in the storage unit 130. The utterance information acquisition unit 122 can acquire the utterance information from the storage unit 130.

[0053] [4-2-3. Book information acquisition unit 123] The book information acquisition unit 123 acquires various information related to a book. The book information includes text contained in the book. The book information also includes image information related to images contained in the book.

[0054] FIG. 6 shows book information 40, which is an example of book information according to the present disclosure. As shown in FIG. 5, the book information includes text information and illustration image information. The text information may be information required to create the above-mentioned voice data. The text information indicates, for example, a character string to be spoken in a picture book or a picture story. The illustration image information includes an image of an illustration corresponding to the text information.

[0055] Returning to Fig. 4, the book information acquisition unit 123 can receive book information from the user device 300. The book information acquisition unit 123 can store the received book information in the storage unit 130. The book information acquisition unit 123 can acquire the book information from the storage unit 130.

[0056] [4-2-4. Vector Expression Acquisition Unit 124] The vector expression acquisition unit 124 converts the speech information into a language vector that indicates linguistic information of the target of speech. The vector expression acquisition unit 124 acquires the language vector by converting the speech information into an expression (numerical expression) that can be used by the model learning unit 126 described later.

[0057] When text information (characters) is used as speech information, the one-hot representation is used to convert the speech information into a language vector. The number of dimensions of the vector of the one-hot representation is the number N of characters included in the speech information. The value of the dimension corresponding to the input character is "1", and the value of the dimension not corresponding to the input character is "0". As an example, when the value of the first dimension is "1" and the value of the dimension other than the first dimension is "0", the vector of the one-hot representation may correspond to the character "a". Similarly, when the value of the second dimension is "1" and the value of the dimension other than the second dimension is "0", the vector of the one-hot representation may correspond to the character "i".

[0058] When phonemes and accents are used as the speech information, the vector expression acquisition unit 124 converts the phonemes and accents into numerical vectors in the same manner as in the above-mentioned Non-Patent Document 1. When characters are used as the speech information, the vector expression acquisition unit 124 applies text analysis to the speech information. The vector expression acquisition unit 124 can use the phoneme and accent information obtained from the text analysis. Therefore, the vector expression acquisition unit 124 can convert the phonemes and accents into numerical vectors in the same manner as in the above-mentioned Non-Patent Document 1.

[0059] [4-2-5. Visual feature extraction unit 125] The visual feature extraction unit 125 can extract visual features from illustration image information included in the book information. The visual feature extraction unit 125 converts the image information into a visual feature vector indicating the visual features of the image included in the book. For example, the visual feature extraction unit 125 acquires the visual feature vector by converting the illustration image information included in the book information into a vector expression that can be used by the model learning unit 126 described later.

[0060] The visual feature extraction unit 125 outputs visual feature vectors from the illustration image information to be used as input to a neural network for speech synthesis.

[0061] A neural network for image recognition, which has been trained in advance from a large amount of image data, is used to convert the illustration image information into a visual feature vector. When the illustration image information is converted into a visual feature vector, the visual feature extraction unit 125 performs forward propagation processing from the illustration image information input to the neural network (see "Hu, Jie, Li Shen, and Gang Sun. "Squeeze-and-Excitation Networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.").

[0062] The visual feature extraction unit 125 finally acquires the information of the output layer, and outputs the information of the output layer as a visual feature vector.

[0063] The output visual feature information vector may be information other than the information of the output layer. The visual feature extraction unit 125 may use the output of any intermediate layer (Bottleneck layer) as the visual feature information vector. By using such a neural network for image recognition that has been trained in advance, the visual feature extraction unit 125 can obtain a vector that reflects information such as characters and backgrounds included in the illustration image information.

[0064] [4-2-6. Model learning unit 126] The model learning unit 126 generates a voice synthesis model based on the utterance information acquired by the utterance information acquisition unit 122, the image information acquired by the book information acquisition unit 123, and the voice data acquired by the voice data acquisition unit 121. To generate the voice synthesis model, the model learning unit 126 uses learning data including voice data associated with language vectors and visual feature vectors.

[0065] 7 shows learning 50, which is an example of learning of a voice synthesis model according to the present disclosure. In learning 50, the model learning unit 126 uses voice data, speech information, and illustration image information included in book information to learn a voice synthesis model (e.g., a neural network for voice synthesis). Learning 50 shows the flow of various data used to learn the voice synthesis model.

[0066] 7, the model learning unit 126 uses the speech data, the language vector acquired by the vector expression acquisition unit 124, and the visual feature vector acquired by the visual feature extraction unit 125 to learn a neural network for speech synthesis that estimates speech parameters from the language vector and the visual feature vector. The model learning unit 126 can use a learning algorithm similar to that in the above-mentioned Non-Patent Document 2.

[0067] The model learning unit 126 can use various neural network structures. For example, the model learning unit 126 can use not only a normal MLP (Multilayer Perceptron), but also neural networks such as an RNN (Recurrent Neural Network), an RNN-LSTM (Long Short Term Memory), a CNN (Convolutional Neural Network), and a Transformer, or a combination of these neural networks.

[0068] Returning to FIG. 4, the model learning unit 126 can store the generated voice synthesis model in the storage unit 130.

[0069] As described above, the model learning unit 126 uses the visual feature vector acquired by the visual feature extraction unit 125 in addition to the language vector used in a conventional neural network for voice synthesis. The visual information vector is acquired from illustration image information extracted from a book such as a picture book. As a result, the model learning unit 126 can train the neural network for voice synthesis by taking into consideration the appearance, facial expression, or background (e.g., scenery, weather, etc.) information of the character included in the illustration image information. The voice synthesis model generated by the model learning unit 126 enables the generation of synthetic voice with natural intonation.

[0070] [4-2-7. Speech synthesis unit 127] Returning to FIG. 4, the voice synthesis unit 127 uses the voice synthesis model generated by the model learning unit 126 to generate synthetic voice.

[0071] For example, the speech synthesis unit 127 acquires a speech synthesis model from the storage unit 130. The speech synthesis unit 127 also acquires a language vector and a visual feature vector from an unknown book. The speech synthesis unit 127 then inputs the acquired language vector and visual feature vector into the speech synthesis model to acquire speech features. The speech synthesis unit 127 generates a speech waveform from the acquired speech features to generate synthetic speech.

[0072] FIG. 8 shows a speech synthesis 60, which is an example of speech synthesis according to the present disclosure. A speech synthesis unit 127 generates synthetic speech from text contained in a picture book or a picture story that is the subject of speech synthesis, and illustration image information corresponding to the picture book or the picture story. The difference between speech synthesis 60 and the algorithm of Non-Patent Document 2 above is that speech synthesis unit 127 uses a visual feature vector, which is information other than a language vector, as an input to the speech synthesis model. The visual feature vector is obtained from a visual feature extraction unit 125. Speech synthesis 60 shows the flow of various data used to generate synthetic speech.

[0073] As shown in FIG. 8, the speech synthesis unit 127 applies text analysis to the input text to acquire information corresponding to speech information. The vector expression acquisition unit 124 converts the acquired speech information into a language vector. The visual feature extraction unit 125 converts illustration image information corresponding to the input text into a visual feature vector. The speech synthesis unit 127 inputs the language vector and the visual feature vector into a speech synthesis model generated by the model learning unit 126. Then, the speech feature is output by forward propagation. The speech synthesis unit 127 acquires synthetic speech by generating a speech waveform from the speech feature.

[0074] Prior to generating the speech waveform, the speech synthesis unit 127 may use an MLPG (Maximum Likelihood Generation) algorithm to obtain a speech parameter sequence smoothed in the time direction (see Masuko et al., “HMM-Based Speech Synthesis Using Dynamic Features,” IEICE Transactions on Computer Science, vol. J79-D-II, no. 12, pp. 2184-2190, Dec. 1996). To generate the speech waveform, the speech synthesis unit 127 may use a method of generating speech waveforms by signal processing (see Imai et al., “Mel-Log Spectral Approximation (MLSA) Filter for Speech Synthesis,” IEICE Transactions on Computer Science, Vol. J66-A, No. 2, pp. 122-129, Feb. 1983). In addition, the voice synthesis unit 127 may use a method of generating voice waveforms using a neural network (see "Oord, Aaron van den, et al. "WAVENET: A GENERATIVE MODEL FOR RAW AUDIO." arXiv preprint arXiv:1609.03499 (2016)").

[0075] [4-3. Storage section 130] Returning to FIG. 4, the storage unit 130 is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 130 includes voice data 131, speech information 132, book information 133, and a voice synthesis model 134. The voice data 131 is, for example, voice data acquired by the voice data acquisition unit 121. The speech information 132 is, for example, speech information acquired by the speech information acquisition unit 122. The book information 133 is book information acquired by the book information acquisition unit 123. The voice synthesis model 134 is, for example, a voice synthesis model generated by the model learning unit 126.

[0076] 5. Flowchart of voice synthesis process Next, a flowchart of an example of a voice synthesis process according to the present disclosure will be described with reference to Fig. 9. The example of the voice synthesis process includes a process for generating a voice synthesis model. The process for generating the voice synthesis model is performed by, for example, the voice synthesis device 100 of Fig. 1.

[0077] FIG. 9 is a flow chart showing process P100, which is an example of a process for generating a voice synthesis model.

[0078] As shown in FIG. 9, first, the utterance information acquiring unit 122 of the speech synthesizer 100 acquires the text contained in the book (step S101).

[0079] Next, the book information acquisition unit 123 of the speech synthesizer 100 acquires images that are included in the book and associated with the acquired text (step S102).

[0080] Next, the voice data acquisition unit 121 of the voice synthesizer 100 acquires a voice signal corresponding to the text acquired by the utterance information acquisition unit 122 (step S103).

[0081] Next, the model learning unit 126 of the speech synthesizer 100 generates a model for converting the text associated with the image into a speech signal based on the text acquired by the speech information acquisition unit 122, the image acquired by the book information acquisition unit 123, and the speech signal of the speech data acquisition unit 121 (step S104). For example, the generated model can convert the text associated with the image into a speech feature. The speech synthesis unit 127 of the speech synthesizer 100 can convert the generated speech feature into a speech signal.

[0082] 6. Effects As described above, the speech synthesizer 100 utilizes not only linguistic information obtained from the text of a book such as a picture book when the book is read aloud, but also visual information obtained from illustrations in the book. As a result, the speech synthesizer 100 can generate a synthetic speech that naturally reads a book such as a picture book.

[0083] [7. Other] A portion of a process described as being performed automatically may be performed manually. Alternatively, all or a portion of a process described as being performed manually may be performed automatically in a known manner. Furthermore, information including the procedure, specific names, various data, and parameters of the processes shown in this specification and drawings may be changed arbitrarily unless otherwise specified. For example, various information shown in each drawing is not limited to the illustrated information.

[0084] The components of the illustrated devices are intended to conceptually illustrate the functions of the devices. The components are not necessarily physically configured as shown in the drawings. In other words, the specific form of the distributed or integrated devices is not limited to the form of the system and device shown in the drawings. All or part of the devices may be functionally or physically distributed or integrated depending on various loads and usage conditions.

[0085] [8. Hardware Configuration] 10 is a diagram showing an example of a hardware configuration of a computer 1000. The systems and methods described in this specification are implemented by, for example, the computer 1000 shown in FIG.

[0086] 10 shows an example of a computer in which a program is executed to implement a speech synthesizer 100. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0087] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0088] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the voice synthesizer 100 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the voice synthesizer 100 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0089] The hard disk drive 1090 can store a voice synthesis program for the voice synthesis process. The voice synthesis program can also be created as a program product that, when executed, performs one or more of the methods described above.

[0090] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes the program.

[0091] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN, WAN, etc.). Then, the program module 1093 and the program data 1094 may be read by the CPU 1020 from the other computer via the network interface 1070.

[0092] 9. Summary of the embodiment As described above, the speech synthesis device 100 according to the present disclosure includes the speech data acquisition unit 121, the speech information acquisition unit 122, the book information acquisition unit 123, and the model learning unit 126. In at least one embodiment, the speech information acquisition unit 122 acquires speech information related to a speech target, which is a text included in a first book, the book information acquisition unit 123 acquires image information related to an image included in the first book, and the speech data acquisition unit 121 acquires speech data corresponding to the speech target. In at least one embodiment, the model learning unit 126 generates a speech synthesis model for reading aloud a second book including text associated with an image, based on the speech information acquired by the speech information acquisition unit 122, the image information acquired by the book information acquisition unit 123, and the speech data acquired by the speech data acquisition unit 121.

[0093] In some embodiments, the book information acquiring unit 123 acquires, as the image information, information about an image that is included on a specific page of the first book and that is associated with text included on the specific page.

[0094] In some embodiments, the audio data acquisition unit 121 acquires, as the audio data, data of audio reading out text that is included on a specific page of the first book and that is associated with an image included on the specific page.

[0095] In some embodiments, the speech information acquisition unit 122 acquires speech information indicating at least one of an accent, a part of speech, a start time of a phoneme, or an end time of a phoneme of a spoken subject.

[0096] As described above, the speech synthesis device 100 according to the present disclosure includes the vector representation acquisition unit 124 and the visual feature extraction unit 125. In at least one embodiment, the vector representation acquisition unit 124 converts the speech information into a language vector indicating linguistic information of the speech target. In at least one embodiment, the visual feature extraction unit 125 converts the image information into a visual feature vector indicating visual features of an image included in the first book. In some embodiments, the model training unit 126 generates a speech synthesis model using training data including speech data associated with the language vector and the visual feature vector.

[0097] Although various embodiments have been described in detail herein with reference to the drawings, these embodiments are merely examples and are not intended to limit the present invention to these embodiments. The features described herein can be realized in various ways, including various modifications and improvements based on the knowledge of those skilled in the art.

[0098] Furthermore, the above-mentioned "module (-er suffix, -or suffix)" can be read as a unit, a means, a circuit, etc. For example, a communication module, a control module, and a storage module can be read as a communication unit, a control unit, and a storage unit, respectively. Furthermore, each control unit in the control unit 120 (e.g., a model learner) can also be read as a model learning unit. [Explanation of symbols]

[0099] 1 Environment 100 Voice Synthesizer 110 Communications Department 120 Control section 121 Voice data acquisition unit 122 Speech information acquisition unit 123 Book Information Acquisition Department 124 Vector Representation Acquisition Unit 125 Visual Feature Extraction Unit 126 Model Learning Department 127 Voice synthesis unit 130 Storage section 131 Audio Data 132 Speech information 133 Book Information 134 Speech synthesis model 200 Network 300 User device

Claims

1. An acquisition unit that acquires speech information about a speech target, which is text included in data about a book, image information about images included in the data about the book, and voice data corresponding to the speech target; a generation unit that generates a voice synthesis model for reading aloud text associated with an image based on the speech information, image information, and voice data acquired by the acquisition unit; A speech synthesis device comprising:

2. The acquisition unit acquires, as the image information, information regarding an image included in a specific page of the book and associated with text included in the specific page. The speech synthesis device according to claim 1 .

3. The acquisition unit acquires, as the voice data, data of a voice reading out a text that is included on a specific page of the book and that is associated with an image included on the specific page. The speech synthesis device according to claim 1 .

4. The acquisition unit acquires the speech information indicating at least one of an accent, a part of speech, a start time of a phoneme, or an end time of a phoneme of the target speech. The speech synthesis device according to claim 1 .

5. a first conversion unit that converts the speech information into a language vector that indicates linguistic information of the target of speech; a second conversion unit that converts the image information into a visual feature vector that indicates visual features of an image included in the data related to the book, The generation unit generates the speech synthesis model using training data including the speech data associated with the language vector and the visual feature vector. The speech synthesis device according to claim 1 .

6. 1. A computer implemented method for speech synthesis, comprising: An acquisition step of acquiring speech information on a speech target, which is text included in data on a book, image information on an image included in the data on the book, and voice data corresponding to the speech target; a generating step of generating a voice synthesis model for reading aloud text associated with an image based on the speech information, image information, and voice data acquired by the acquiring step; A speech synthesis method comprising:

7. An acquisition step of acquiring speech information on a speech target, the speech information being text included in data on a book, image information on images included in the data on the book, and voice data corresponding to the speech target; a generation step of generating a speech synthesis model for reading aloud text associated with an image based on the speech information, image information, and voice data acquired by the acquisition step; A speech synthesis program that causes a computer to execute the following:

8. A first acquisition unit that acquires utterance information related to an utterance target, the utterance information being a text included in data related to a book; a second acquisition unit configured to acquire image information related to an image, the image information corresponding to the text included in the data related to the book; an acquisition unit that acquires synthetic speech corresponding to the speech target by inputting the speech information acquired by the first acquisition unit and the image information acquired by the second acquisition unit into a speech synthesis model for reading aloud text associated with an image; A speech synthesis device comprising:

Images