Information processing device, information processing method, program, and information processing system
The information processing system addresses the challenge of selecting a suitable voice for content by using a speaker search unit and content analysis to match voice quality with the content's topic and genre, enhancing the accuracy and efficiency of voice selection in speech synthesis.
Patent Information
- Application Number
- PCT/JP2024/018855
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-05-22
- Publication Date
- 2024-12-05
AI Technical Summary
Existing speech synthesis technologies face challenges in selecting the most suitable voice for content, especially when the number of available TTS speakers is large, as users find it time-consuming to choose a voice that matches the content's genre and tone, and current methods are insufficient for suggesting voices based on content type beyond spoken words.
An information processing system that includes a speaker search unit to select speaker data based on the topic and genre of text data, using a content analysis server to generate topic models and TTS reference data, and a speech synthesis server to generate synthesized speech data with a voice quality matching the content, allowing for more accurate voice selection.
The system effectively provides users with a voice suitable for the content by analyzing the topic and genre, reducing the time and effort in selecting a voice and improving the relevance of voice quality to the content's type.
Smart Images

Figure JP2024018855_05122024_PF_FP_ABST
Abstract
Description
Information processing device, information processing method, program, and information processing system
[0001] The present technology relates to an information processing device, an information processing method, a program, and an information processing system, and relates to speech synthesis processing used for, for example, narration, dialogue, etc. of content.
[0002] In recent years, an increasing number of content distribution services use text-to-speech (TTS) narration. The speech synthesis software and speech synthesis services used offer multiple TTS speakers, allowing users to select a voice quality that best suits their content. Patent Document 1 below discloses a technology that selects a male or female voice by tallying the frequency with which each word used in a sentence to be read aloud appears in male and female sentences.
[0003] Japanese Patent Application Publication No. 11-296193
[0004] If the number of TTS speakers provided is small, users can listen to sample voices from all TTS speakers to select a voice that best suits their content, but this means fewer options. On the other hand, if more TTS speakers are provided, it will take time for users to listen to sample voices from all speakers. Also, if there are voices with similar qualities, users may have trouble deciding which voice best suits their content.
[0005] To make it easier for users to select a voice, there are methods that differentiate between voices based on the high and low frequency of the TTS speaker's voice, the gender or age of the TTS speaker, and the image of the TTS speaker's voice (e.g., "mature voice" or "energetic voice"). However, even if a speaker is selected based on, for example, a "male with a medium voice frequency," the desired voice quality is likely to differ depending on whether the content is a documentary program or a sports program. Furthermore, the technology in Patent Document 1 limits the text to be read aloud to "spoken language." Therefore, it is still insufficient to suggest voices that correspond to the content.
[0006] Therefore, this technology enables users to more accurately provide audio that is suitable for the content from among the available audio options.
[0007] The information processing device according to the present technology includes a speaker search unit that searches for and selects speaker data from among a plurality of speaker data based on the topic of text data, and a voice synthesis unit that generates synthetic voice data for the text data using the speaker data selected by the speaker search unit. The speaker data having a voice quality according to the topic (topic or genre) of the content of the text data is selected, and a voice reading the text data is synthesized using the speaker data.
[0008] 1 is an explanatory diagram of a system configuration according to an embodiment of the present technology; FIG. 2 is an explanatory diagram of a functional configuration of an information terminal according to an embodiment; FIG. 3 is an explanatory diagram of a functional configuration of a speech synthesis server according to an embodiment; FIG. 4 is an explanatory diagram of a functional configuration of a content analysis server according to an embodiment; FIG. 5 is an explanatory diagram of a functional configuration of a database according to an embodiment; FIG. 6 is an explanatory diagram of a process of a TTS speaker suggestion system according to an embodiment; FIG. 7 is an explanatory diagram of an example of a topic used in an embodiment; FIG. 8 is an explanatory diagram of a learning process of voice features according to an embodiment; FIG. 9 is a flowchart of a topic model generation process according to an embodiment; FIG. 10 is a flowchart of a TTS reference data generation process according to an embodiment; FIG. 11 is a flowchart of a speech synthesis server process according to an embodiment; FIG. 12 is a flowchart of a database process according to an embodiment; FIG. 13 is an explanatory diagram of a display example on an information terminal according to an embodiment; FIG. 14 is a flowchart of a timestamp setting according to a change in speech features according to an embodiment; FIG. 15 is a flowchart of a TTS reference data generation process with reference to a timestamp according to an embodiment; FIG. 16 is a flowchart of a process of selecting multiple TTS reference data in a database according to an embodiment; FIG. 17 is an explanatory diagram of the number of reference data according to distribution according to an embodiment; FIG. 18 is an explanatory diagram of a display example on an information terminal according to an embodiment; FIG. 19 is a block diagram of an information processing device according to an embodiment;
[0009] The following describes the embodiments in the following order: <1. System configuration> <2. Operation of the TTS speaker suggestion system> <3. Processing of the content analysis server> <4. Processing of the speech synthesis server> <5. Processing of the database> <6. Display on the information terminal> <7. Handling when multiple speakers exist in the reference content> <8. Proposal of multiple speaker candidates> <9. Configuration of the information processing device> <10. Summary and modifications>
[0010] 1 shows an example of the configuration of a TTS speaker suggestion system 1 according to an embodiment. The TTS speaker suggestion system 1 is a system that suggests to a user 10 appropriate TTS speakers for the text of content CT1 being produced on an information terminal 100, for example.
[0011] In the TTS speaker suggestion system 1, an information terminal 100, a speech synthesis server 200, a content analysis server 300, a database 400, and a content provider 500 are connected via a network 600.
[0012] The user 10 operates the information terminal 100 to create the content CT1. In this disclosure, "content CT1" refers to a work that primarily uses audio, regardless of whether it includes video (moving images or still images). The information terminal 100 has the functions shown in FIG. 2 for content creation.
[0013] The content creation application 110 is a function for creating and editing the content CT1 in response to operations by the user 10. The display 120 is a function for displaying images to the user 10. The character input unit 130 is an input function for the user 10 to input text data into the content CT1, for example. The speaker 140 is a function for outputting audio to the user 10. The network communication unit 150 is a function for communicating with other devices via the network 600.
[0014] This information terminal 100 is connected to a network 600 such as the Internet via a network communication unit 150, and can obtain synthetic voice data AD by transmitting text data TD to a voice synthesis server 200. The transmitted text data TD is input by a user 10 during the production process by a content production application 110. For example, the user 10 inputs the text data TD using a character input unit 130.
[0015] The speech synthesis server 200 has the role of converting text data TD transmitted from the information terminal 100 into synthetic speech data AD. To this end, the speech synthesis server 200 includes a text-to-phoneme symbol conversion unit 210, a reference data acquisition unit 220, a speaker search unit 230, a retained speaker data unit 240, a speech synthesis unit 250, and a network communication unit 260, as shown in FIG.
[0016] The voice synthesis server 200 is connected to the network 600 via the network communication unit 260. The voice synthesis server 200 receives the text data TD transmitted from the information terminal 100.
[0017] The text-to-phoneme symbol converter 210 in the speech synthesis server 200 has a function for converting text data TD into phoneme data. A phoneme is a symbol that represents each individual voice sound. Using this function, the speech synthesis server 200 generates phoneme data corresponding to the text data TD transmitted from the information terminal 100.
[0018] The reference data acquisition unit 220 has a function of acquiring TTS reference data RD from the database 400. The TTS reference data RD is data used to generate synthetic speech data AD that matches the content CT1. The reference data acquisition unit 220 transmits the text data TD transmitted from the information terminal 100 to the database 400, and requests the database 400 for TTS reference data RD that matches the content CT1.
[0019] The speaker search unit 230 is a function that uses the speech feature RDX in the TTS reference data RD transmitted from the database 400 to search for speaker data held by the speech synthesis server 200 that has a voice similar to the speech feature RDX (similar speech feature).
[0020] The retained speaker data unit 240 has a function for storing speaker data searched by the speaker search unit 230. The speaker data includes a speaker ID that identifies each speaker and voice quality data associated with the speaker ID. The speech synthesis server 200 selects one of the multiple speaker data stored in the retained speaker data unit 240 and proposes it to the user 10.
[0021] The speech synthesis unit 250 has a function of generating synthetic speech data AD from text data TD. The speech synthesis unit 250 generates synthetic speech data AD corresponding to the text data TD based on the speaker data selected by the speaker search unit 230 and the phoneme data obtained by the text-to-phoneme conversion unit 210. The text-to-phoneme conversion unit 210 is also called grapheme-to-phoneme conversion (G2P). The speech synthesis unit 250 is generally composed of components called a "synthesizer" and a "vocoder" (not shown).
[0022] The voice synthesis server 200 sends the generated synthetic voice data AD to the network 600 using the network communication unit 260. The synthetic voice data AD is then transmitted to the network communication unit 150 of the information terminal 100 and is used by the content production application 110.
[0023] The content analysis server 300 shown in Figure 1 analyzes content CT2 provided by the content provider 500. Content CT2 is content that is generally distributed or broadcast, and is referred to as "reference content CT2" to distinguish it from content CT1. This reference content CT2 also refers to a work with recorded audio, with or without video. Analysis of the reference content CT2 by the content analysis server 300 generates TTS reference data RD, and the speech synthesis server 200 uses the TTS reference data RD during speech synthesis.
[0024] The purpose of this embodiment is to provide a voice quality that is suitable for the text data TD entered by the user 10, that is, a synthesized voice that is suitable for the content CT1. To achieve this, the role of the content analysis server 300 is to prepare information about a voice that is suitable for reading out the text data TD.
[0025] As shown in FIG. 4, the content analysis server 300 includes a content acquisition unit 310, a voice extraction unit 320, a voice recognition unit 330, a storage unit 340, a topic analysis unit 350, a voice feature acquisition unit 360, and a network communication unit 370.
[0026] The content analysis server 300 is capable of communicating with the content provider 500, the database 400, and the like via the network communication unit 370.
[0027] The content acquisition unit 310 has a function of acquiring reference content CT2 to be analyzed. The content acquisition unit 310 transmits an acquisition request RQ to the content provider 500 via the network communication unit 370 to acquire various reference content CT2. For example, the reference content CT2 may be downloaded from a public domain internet radio station. Alternatively, this can be achieved by individually requesting the content provider 500 to provide a content acquisition API (Application Programming Interface).
[0028] The audio extraction unit 320 extracts audio from the reference content CT2 acquired from the content provider 500. When the reference content CT2 is a moving image, audio extraction can be performed using existing software.
[0029] The voice recognition unit 330 converts the voice of the reference content CT2 extracted by the voice extraction unit 320 into text data, that is, an utterance TR. The storage unit 340 stores the utterance TR.
[0030] The topic analysis unit 350 determines a topic vector RDT for the text data transcribed by the speech recognition unit 330. The topic vector RDT is information that expresses the topic of the content (text data). Therefore, it can be said to be information that serves as an index for topic classification. Note that a topic is a term that indicates the content of the text data, such as the topic or genre. For example, in the case of movies, topics and topic genres such as topics about a specific movie, topics about the movie industry as a whole, and genres such as "action movies" and "comedy movies" are collectively called topics.
[0031] The audio feature acquisition unit 360 converts the audio data of the reference content CT2 extracted by the audio extraction unit 320 into an audio feature RDX called an "x-vector."
[0032] Using the above functions, the content analysis server 300 can obtain the following for the reference content CT2: topic vector RDT, speech features RDX, and content URL (RDU). This information is compiled into one TTS reference data RD, which is then stored in the database 400 via the network 600.
[0033] The database 400 receives a query from the speech synthesis server 200 and searches for speech features of a voice quality suitable for reading out the text data TD. The search uses the TTS reference data RD analyzed by the content analysis server 300.
[0034] 5, the database 400 includes a storage unit 410, a topic analysis unit 420, a topic similarity analysis unit 430, and a network communication unit 440. The database 400 is capable of communicating with the content analysis server 300, the speech synthesis server 200, etc. via the network communication unit 440.
[0035] The storage unit 410 stores TTS reference data RD for various reference content CT2. The figure shows that N pieces of TTS reference data RD are stored as TTS reference data RD-1 to TTS reference data RD-N. Each piece of TTS reference data RD includes a topic vector RDT, speech features RDX, and the URL (RDU) of the reference content CT2.
[0036] The topic analysis unit 420 has a function of converting the text data TD transmitted from the speech synthesis server 200 into a topic vector TV. The topic similarity analysis unit 430 has a function of searching for the TTS reference data RD with the highest similarity among the topic vector TV and the topic vector RDT of the TTS reference data RD stored in the storage unit 410.
[0037] The content provider 500 shown in Fig. 1 is a provider that provides reference content CT2. For example, a video posting site corresponds to the content provider 500. The content provider 500 is not limited to one, and may be made up of multiple companies or services. Furthermore, the reference content CT2 provided is not limited to video, and may be audio-only content provided by Internet radio.
[0038] 2. Operation of the TTS speaker suggestion system The following describes the operation of the TTS speaker suggestion system 1 configured as above. First, the purpose of the operation of the TTS speaker suggestion system 1 according to the embodiment will be described.
[0039] The TTS speaker suggestion system 1 operates to provide a user 10 who is creating content CT1 with a voice quality that matches the content CT1. For the narration and dialogue of content CT1, a voice that matches the content of that content CT1 must be adopted. Providing a wide variety of voice qualities for the text of content CT1 would expand the options available to the user 10, but the selection process itself would be time-consuming. Furthermore, even if the user 10 is able to select based on the high or low frequency of the voice, the gender or age of the speaker, or the image of the voice, the voice may not necessarily match the content of content CT1.
[0040] Therefore, the TTS speaker suggestion system 1 of this embodiment generates synthetic speech with a voice quality similar to that of the narrator or actor appearing in the reference content CT2 provided by the content provider 500, thereby suggesting to the user 10 a TTS speaker suitable for the content CT1 under production. By reading the narration and lines of the content CT1 in a voice similar to that of the speaker appearing in the reference content CT2 that is widely listened to, it is expected that the viewer will not feel uncomfortable with the atmosphere of the synthetic speech voice.
[0041] 6, the flow of operations of each device in the TTS speaker suggestion system 1 will be explained. First, as a prerequisite process for suggesting TTS speakers to the user 10, the content analysis server 300 generates a topic model TM and TTS reference data RD and stores them in the database 400 in process ST1.
[0042] The speech synthesis server 200 requires TTS reference data RD to generate synthetic speech data AD by a speaker appropriate for the genre and topic of the content CT1. Therefore, the content analysis server 300 analyzes the reference content CT2 and prepares the TTS reference data RD in the database 400. This process ST1 only needs to be performed sequentially and continuously for various reference content CT2. In other words, it does not need to be synchronized with process ST2 for suggesting speakers to the user 10 (information terminal 100).
[0043] In process ST1, the content analysis server 300 sends an acquisition request RQ to the content provider 500. In response to this, the content provider 500 transmits reference content CT2 to the content analysis server 300.
[0044] After acquiring the reference content CT2, the content analysis server 300 analyzes the acquired reference content CT2 and generates a topic model TM and TTS reference data RD, which are then transmitted to the database 400 for storage.
[0045] The content analysis server 300 monitors the distribution of reference content CT2 provided by the content provider 500 and can perform the above process ST1 as appropriate to generate TTS reference data RD. By focusing on analyzing popular reference content CT2, it is expected that TTS reference data RD can be obtained that will suggest to the user 10 a voice quality that will be accepted by a wider range of people.
[0046] Next, the process ST2 will be described. The process ST2 is executed in response to the user 10 transmitting the text data TD from the information terminal 100.
[0047] The information terminal 100 transmits text data TD to the speech synthesis server 200. The text data TD is, for example, data of sentences such as narration or dialogue of the content CT1 that the user 10 is creating.
[0048] The speech synthesis server 200 transmits the received text data TD to the database 400 .
[0049] The database 400 performs topic analysis and speech feature search on the received text data TD to select one or more TTS reference data RD suitable for the text data TD. The database 400 then transmits the TTS reference data RD obtained by these processes to the speech synthesis server 200.
[0050] The speech synthesis server 200 uses the speech feature RDX in the TTS reference data RD received from the database 400 to compare it with the speech feature of the retained speakers stored in the retained speaker data section 240 within the speech synthesis server 200, and selects the speaker data with the most similar voice quality.
[0051] The speech synthesis server 200 then performs speech synthesis processing on the text data TD using the speech synthesis model of the speaker data to generate synthetic speech data AD. The speech synthesis server 200 transmits the synthetic speech data AD to the information terminal 100.
[0052] In the information terminal 100, the user 10 uses the content production application 110 to incorporate the received synthetic voice data AD into the content CT1 being produced.
[0053] As described above, the TTS speaker suggestion system 1 suggests to the user 10 synthetic speech data AD with a voice quality suitable for the narration, lines, etc. of the content CT1.
[0054] Here, topic analysis and speech features will be explained. First, topic analysis will be explained. Fig. 7 shows information (defined as a topic model) generated based on some reference content CT2.
[0055] In the example of Figure 7, for simplicity, the genre classification is performed using two topics #1 and #2. Latent Dirichlet Allocation (LDA) was used for the genre classification. Topic #1 has a high probability of occurrence of words related to "movies." It can be seen that topic #2 has the appearance of words related to "smartphones." In this example, the words with the highest occurrence probability are displayed in groups of 10 in order of occurrence probability. This topic model is used to find out which topic a certain utterance belongs to.
[0056] For example, suppose the utterance is "I went to the movies last weekend." If you split this utterance into words, you will get the following word sequence: If you keep only the nouns in this word sequence, you will get a noun word sequence.
[0057] Sentence word sequence = ['last week', 'end', 'wa', 'movie', 'o', 'watch', 'te', 'ki', 'mashita', 'ta'] Noun word sequence = ['last week', 'movie']
[0058] When this noun word string is subjected to topic analysis using the LDA method, the following topic vector is obtained: Here, a topic vector is defined as representing the probability distribution of which topic a certain spoken text is classified into.
[0059] Topic #1: 0.8, Topic #2: 0.2 Topic vector = [0.8, 0.2] This means that the above utterance has a high probability of occurrence related to topic #1, and it can be determined that the topic of the utterance is in line with topic #1. Such a topic model is generated by the content analysis server 300 and used in processing in the database 400.
[0060] Next, we will explain voice features. The voice quality of the speaker appearing in the reference content CT2 and the voice quality of the speaker model stored in the speech synthesis server 200 are compared using features called "x-vectors." "x-vectors" are also called "deep speaker embedding," and are a type of "speaker identification technology" that uses deep learning technology. Typically, x-vectors are 512-dimensional vectors.
[0061] We will now explain how to use x-vector. Figure 8 shows the learning process of x-vector. The voices of multiple speakers, including A, B, and C, are input into the neural network. For each voice, voice features are extracted for each unit time interval from t=1 to t=T.
[0062] These features are then max pooled. Max pooling is a process in which a window of a certain size is slid over the input data and the maximum value is selected within that window, which allows the input data to be reduced.
[0063] The max-pooled features are then used in the classification unit to determine the speaker class and train the speaker. A portion of the classification unit obtained through this training can be used as an x-vector. For example, the x-vector obtained when inputting person A's voice can be thought of as a kind of person A's voiceprint.
[0064] This neural network can also be applied to speakers other than those used in training. Therefore, by inputting the voices of the narrator or performers in the reference content CT2 into this neural network, the x-vectors of the speakers of each content can be obtained. Similarly, by inputting the voices of the speakers stored in the speech synthesis server 200 into this neural network, the x-vectors of each speaker can be obtained. By comparing the x-vectors of two speakers, it is possible to determine whether their voices are similar. Voice similarity can be determined, for example, by using the cosine similarity between vectors or matching using PLDA (probabilistic linear discriminant analysis).
[0065] 3. Analysis Processing of Content Analysis Server The processing of the content analysis server 300 in process ST1 in Fig. 6 will be described. The processing of the content analysis server 300 has two model creation phases. First, there is a phase in which a topic model is created. Next, there is a phase in which it is determined to which topic each piece of reference content CT2 corresponds, and TTS reference data RD is created.
[0066] The topic model creation phase will now be described. The content analysis server 300 collects multiple pieces of reference content CT2 from the content provider 500 and creates a topic model from the spoken text in the reference content CT2 using the LDA method. The purpose of this is to categorize the reference content CT2 provided by the content provider 500 into genres.
[0067] The following description will be made using Figures 4 and 9. Note that Figure 9 is a flowchart of the processing performed by the processor serving as the content analysis server 300, in which the processing to be performed is indicated by a solid-line box, and for ease of understanding, data input or output to the processing is indicated by letters or symbols in parentheses. The storage unit 340 and database 400 where the data is stored are also shown. Step numbers are assigned to the processing performed by the processor serving as the content analysis server 300. This flowchart description format is also used in Figures 10, 11, 12, 14, 15, and 16, which will be described later.
[0068] The content analysis server 300 acquires the reference content CT2 from the content provider 500 by the content acquisition unit 310. In step S101, the audio extraction unit 320 performs an audio data extraction process on the acquired reference content CT2.
[0069] The extracted voice data is subjected to voice recognition processing by the voice recognition unit 330 in step S102 and converted into a spoken sentence TR, which is text data. The spoken sentence TR is stored in the storage unit 340 in step S103.
[0070] The processes of steps S101, S102, and S103 enclosed by dashed lines are repeatedly performed for multiple pieces of reference content CT2. Then, the processes of steps S101, S102, and S103 are performed for a predetermined number (M pieces) of reference content CT2, and when the number of utterance sentences TR stored in the storage unit 340 reaches M, topic analysis is performed by the topic analysis unit 350 in step S110 using the M stored utterance sentences TR. As a result, a topic model TM is obtained.
[0071] The generated topic model TM is transmitted to the database 400 via the network communication unit 370 in step S111.
[0072] Next, we will explain the TTS reference data RD creation phase. The purpose of this phase is to create TTS reference data RD that indicates what topic vectors a particular piece of reference content CT2 has, what characteristics the voice used in the reference content CT2 has, and which URL (Uniform Resource Locator) it is located at.
[0073] This will be described with reference to Figures 4 and 10. In the content analysis server 300, the audio extraction unit 320 extracts audio data from the reference content CT2 acquired by the content acquisition unit 310 in step S121.
[0074] The extracted speech data is subjected to speech recognition processing by the speech recognition unit 330 in step S123, and an utterance sentence TR is obtained. The utterance sentence TR is then topic analyzed by the topic analysis unit 350 in step S124, and a topic vector RDT is obtained. The topic vector RDT is a vector representing the probability that the utterance sentence TR belongs to a particular topic. In this topic analysis processing, the topic model TM obtained in the topic model creation phase of FIG. 9 is used.
[0075] The speech data extracted in step S121 is also used in step S125, and the speech feature acquisition unit 360 calculates speech features RDX.
[0076] In step S126, the content analysis server 300 combines the topic vector RDT calculated as described above, the speech feature RDX, and the URL (RDU) of this reference content CT2 to form one TTS reference data RD, and transmits this to the database 400. As a result, the TTS reference data RD is additionally stored in the database 400. Note that in the explanation here, the utterance sentence TR is extracted again from the reference content CT2 in step S123, but the utterance sentence TR generated in step S102 of the topic model creation phase in Figure 9 may also be cached and used.
[0077] 4. Processing of the Speech Synthesis Server Next, the processing of the speech synthesis server 200, particularly how the TTS reference data RD is used during speech synthesis, will be described with reference to FIGS.
[0078] The user 10 inputs text data TD to be synthesized using a content creation application 110 used in the information terminal 100. The text data TD is so-called natural sentences and does not need to be special sentences prepared for the TTS speaker suggestion system 1. Therefore, the user 10 does not need to learn a special writing method in order to use the TTS speaker suggestion system 1.
[0079] Upon receiving the text data TD, the speech synthesis server 200 performs the processing shown in Fig. 11. Upon receiving the text data TD, the speech synthesis server 200 performs processing in step S201 in which the text-to-phoneme symbol converter 210 converts the natural sentence into phoneme data for speech synthesis.
[0080] In step S202, the speech synthesis server 200 performs a process of acquiring TTS reference data RD using the reference data acquisition unit 220. Specifically, the speech synthesis server 200 transmits the text data TD to the database 400 and requests a search for TTS reference data RD. In response, the database 400 receives the text data TD as input, selects the TTS reference data RD stored in the database 400 whose topic vector RDT is closest to the topic vector of the text data TD, and transmits the selected TTS reference data RD to the speech synthesis server 200. The processing of the database 400 will be described later. The speech synthesis server 200 receives the TTS reference data RD selected in this manner from the database 400, i.e., the reference data corresponding to the text data TD.
[0081] The speech synthesis server 200 receives the TTS reference data RD from the database 400 as the relevant reference data, and in step S203 performs a speaker search using the speaker search unit 230. In this case, the speech synthesis server 200 uses the speech feature RDX included in the TTS reference data RD to calculate speakers with similar voice qualities from among the speaker models stored in the retained speaker data unit 240. Specifically, speakers with similar voice qualities can be obtained by calculating the cosine similarity between the speech feature RDX sent from the database 400 and the speech feature of the retained speaker model. This process makes it possible to derive the speaker ID of the speaker data that is most appropriate for the topic from among the speaker models stored in the speech synthesis server 200.
[0082] The speech synthesis server 200 obtains synthetic speech data AD by inputting the phoneme data obtained in step S201 and the speaker ID obtained in step S203 to the speech synthesis unit 250. Then, in step S205, the speech synthesis server 200 transmits the synthetic speech data AD, the speaker ID, and the reference URL to the information terminal 100 via the network communication unit 260. The reference URL is the URL (RDU) of the reference content CT2 included in the TTS reference data RD obtained in step S202.
[0083] 5. Database Processing The processing of the database 400 will be described with reference to Fig. 5 and Fig. 12. This is processing in response to the search request from the speech synthesis server 200 in step S202 of Fig. 11 described above.
[0084] When the database 400 receives the text data TD from the speech synthesis server 200, the topic analysis unit 420 performs topic analysis in step S211 of Fig. 12. In this topic analysis, the topic analysis of the text data TD is performed using the topic model TM stored in the storage unit 410, and a topic vector TV is generated.
[0085] Next, in step S212, the database 400 performs a topic search using the topic similarity analysis unit 430. This is a process of searching for a topic vector RDT similar to the topic vector TV. Specifically, this process searches for TTS reference data RD (RD-1...RD-N) generated from various reference content CT2 stored in the storage unit 410, whose topic vector RDT is similar to the topic vector TV of the text data TD.
[0086] Similar topic vectors mean that the content is in the same genre or has a similar topic. In other words, searching for TTS reference data RD with similar topic vectors is equivalent to searching for TTS reference data RD generated based on the content CT1 created by the user 10 and the reference content CT2 that has a similar genre or topic.
[0087] Cosine similarity can be used to search for a topic vector RDT similar to the topic vector TV. This allows the TTS reference data RD with the topic vector RDT most similar to the topic vector TV to be selected as the optimal topic. The database 400 then transmits the TTS reference data RD obtained as the optimal topic to the speech synthesis server 200 as the corresponding reference data for the current text data TD.
[0088] It is also possible to set a threshold value for the similarity, and if the similarity does not exceed the threshold value, send a plurality of TTS reference data RD to the speech synthesis server 200 .
[0089] 6. Display on Information Terminal An example of a display that is presented to the user 10 on the information terminal 100 as a result of the above processing will be described.
[0090] The data that the information terminal 100 receives from the speech synthesis server 200 are the synthetic speech data AD and the URL of the reference content CT2. The URL of the reference content CT2 is provided to show the user 10 information about the content of the speaker's voice that was used as a reference for generating the synthetic speech.
[0091] 13 shows an example of a display on the display 120 of the information terminal 100. On the display 120, a text box 31, a speaker ID 32, a synthesis start button 33, a play button 34, and a reference URL 35 are displayed.
[0092] The text box 31 is a box for inputting text data TD. The speaker ID 32 is the speaker ID selected by the speech synthesis server 200 in step S203 of FIG. 11. The synthesis start button 33 is an operator for instructing the start of speech synthesis processing. The play button 34 is an operator for playing back the synthesized speech. The reference URL 35 is the URL of reference content CT2 that referenced the voice, and is displayed, for example, in the form of a link to the reference content CT2.
[0093] On this screen, the user 10 can operate the play button 34 to listen to the voice of the speaker with the speaker ID 32 suggested by the speech synthesis server 200. Furthermore, by operating the reference URL 35, the user 10 can play the reference content CT2 that the speech synthesis server 200 used as a reference when selecting the speaker ID 32, and listen to the voice of the narration, etc. Therefore, the user 10 can not only listen to the voice of the text read aloud by the speaker ID suggested by the speech synthesis server 200, but also listen to the voice of the reference content CT2 that is similar in genre, etc. to the content CT1 being produced for the selection of that speaker ID.
[0094] 7. Handling Cases Where Reference Content Includes Multiple Speakers Up to this point, the explanation has been given on the assumption that each piece of reference content CT2 features only one speaker. However, in actual reference content CT2, multiple speakers usually appear. For example, in television programs, different announcers are often in charge of different genres, such as reports from the field, weather forecasts, traffic information, and sports information.
[0095] We have explained x-vectors, which are features that represent voices, as audio features. By using x-vectors, it is possible to detect when a speaker changes within the reference content CT2. This is a technique called "speaker diarization," and by using this technique, it is possible to handle cases where there are multiple speakers in a single piece of content.
[0096] This will be explained using Figure 14. In step S131, the content analysis server 300 performs audio extraction using the audio extraction unit 320 for the reference content CT2, thereby acquiring audio data. Next, in step S132, the audio feature acquisition unit 360 extracts audio features RDX every certain unit of time, for example, every 30 seconds. In step S133, feature change detection processing is performed on the audio features RDX for each unit of time, and changes equal to or greater than a threshold value are monitored.
[0097] Cosine similarity can be used to detect changes in feature quantities. If there is a change that exceeds a threshold, it means that the speaker has changed, so a timestamp is recorded and stored in timestamp database 341. Timestamp database 341 is prepared using, for example, a portion of the area in storage unit 340.
[0098] For example, the content analysis server 300 monitors changes in the speech feature RDX for the reference content CT2 and stores the change points. This makes it possible to handle reference content CT2 featuring multiple speakers with a configuration equivalent to the system described in the TTS reference data creation phase in FIG.
[0099] For example, Fig. 15 shows the process of the TTS reference data creation phase similar to Fig. 10. Note that the same processes as those in Fig. 10 are given the same step numbers and their explanations will be omitted.
[0100] 15, in step S121A, audio is extracted for each section determined by the timestamp for the reference content CT2 using the timestamp database 341. The extracted audio data is then processed in the same manner as in FIG. 10 to generate TTS reference data RD.
[0101] 8. Proposal of Multiple Speaker Candidates In the explanation so far, it has been explained that one speaker best suited to the content CT1 is proposed based on the words spoken in the content CT1 (text data TD). However, it is conceivable that some users 10 may wish to try other speakers as options. Assuming such a user 10, an example of a process for providing multiple speakers will be explained using FIG. 16. Note that in FIG. 16, the same processes as in FIG. 12 are assigned the same step numbers to avoid redundant explanation.
[0102] In the previous example shown in Figure 12, the topic similarity analysis unit 430 retrieves only one TTS reference data RD with the most similar topic vector in step S212. In step S212A of Figure 16, in order to increase the number of voices to be proposed, the topic similarity analysis unit 430 retrieves multiple TTS reference data RDs, selecting them in descending order of cosine similarity. In the figure, ten TTS reference data RDs are selected, and ten TTS reference data RDs (RD#1 to RD#10) are shown.
[0103] The TTS reference data RD#1 having the most similar topic vector RDT is designated as the "optimal speaker" data, and the data in descending order of similarity are designated as reference data #2, . . . , #10.
[0104] Here, we will explain cosine similarity again. In the previous explanation of speech features, we mentioned that x-vectors can be used as a speaker's voiceprint. Cosine similarity is used to check for similar voices. In FIG. 11, we described how the speech synthesis server 200 obtains TTS reference data RD based on text data TD transmitted from the information terminal 100. The TTS reference data RD includes a speech feature RDX, which is an x-vector. Because this speech feature RDX is a vector, it is possible to calculate the similarity of voice quality by calculating the cosine similarity with other speech features.
[0105] For example, the cosine similarity between two vectors a and b can be expressed by (Equation 1), and ranges from "-1" to "1".
[0106]
[0107] When the cosine similarity is "1", the angle between them is 0 degrees, and the vectors are in the same direction. This means that the voice qualities are completely similar. When the cosine similarity is "0", the angle between them is 90 degrees, and the vectors are in orthogonal directions. This means that there is no relationship between whether the voice qualities are similar or dissimilar. When the cosine similarity is "-1", the angle between them is 180 degrees, and the vectors are in the opposite direction. This means that the voice qualities are completely dissimilar.
[0108] In the process of FIG. 16, the database 400 performs the similarity evaluation process in step S220 using a plurality of (for example, 10) TTS reference data RD (RD#1 to RD#10).
[0109] In this similarity evaluation process, the speech feature RDX of the TTS reference data RD#1 of the optimal speaker is used as the "reference feature" and the cosine similarity with each speech feature RDX contained in the TTS reference data RD#2 to RD#10 is calculated.
[0110] In this case, TTS reference data RD having speech features RDX whose cosine similarity with the reference features is close to "0" is information on a speaker whose features are not dissimilar to those of the optimal speaker. Such TTS reference data RD#x is called an "orthogonal speaker."
[0111] TTS reference data RD having speech features RDX whose cosine similarity with the reference features is close to "-1" is information about a speaker whose features are dissimilar to those of the optimal speaker. Such TTS reference data RD#y is called a "reverse speaker."
[0112] The optimal speaker's TTS reference data RD#1, the orthogonal speaker's TTS reference data RD#x, and the opposite speaker's TTS reference data RD#y are collected into one data group and transmitted to the speech synthesis server 200.
[0113] In step S203 of Fig. 11, the speech synthesis server 200 searches for speaker data similar to the TTS reference data RD among the speaker data stored in the stored speaker data unit 240. In this case, speaker data similar to each of the TTS reference data RD#1, RD#x, and RD#y is searched for. Therefore, a speaker ID similar to the optimal speaker, a speaker ID similar to the orthogonal speaker, and a speaker ID similar to the reverse speaker are obtained. As a result, three voices with different voice qualities are proposed to the user 10.
[0114] In the above example, two speech features RDX with cosine similarities close to "0" and "-1" are calculated, but it is also possible to calculate multiple speech features with cosine similarities ranging from "-1" to "1." In this way, proposing to the user 10 reference data with cosine similarities that are orthogonal or opposite in direction from among reference data with high topic vector similarities means searching for reference content CT2 with a similar topic but different speaker voice qualities, and proposing to the user 10 voice qualities similar to those reference content CT2.
[0115] It is also possible to change the number of speakers suggested to the user 10 depending on the variance of the speech features. The following (Equation 2) is a matrix of the cosine similarity between the reference feature (the speech feature RDX of the optimal speaker) and the speech feature RDX of other TTS reference data RD.
[0116]
[0117] Here, s1,i represents the cosine similarity between the reference feature and the i-th feature. The variance of this matrix σ 2 is shown in (Equation 3), where μ is the average value.
[0118]
[0119] By calculating the variance, it is possible to obtain a rough indication of whether the proposed speakers converge to one speech feature or whether the features are scattered among multiple speakers. Here, n is the number of TTS reference data RDs for which similarity is evaluated. In the case of (Equation 2), n = 9. If 10 TTS reference data RDs (RD#1 to RD#10) are initially selected, the similarity of TTS reference data RD#2 to RD#10 is evaluated with respect to TTS reference data RD#1, so n = 9.
[0120] For example, if the variance is close to zero, the voice qualities of the speakers of that topic are roughly similar. In that case, it is easier to obtain the desired number of speakers by increasing the number of reference data for evaluating similarity rather than the top 10 topic vector similarities. For example, the top 20. Conversely, if the variance is sufficiently large, the number of TTS reference data RD to be evaluated does not need to be very large. This is expressed mathematically in (Equation 4), which uses the variance as the denominator. y is the number of TTS reference data RD to be evaluated.
[0121]
[0122] Figure 17 is a graph of equation 4. It can be seen that the number of reference data to be evaluated changes depending on the variance value (V). When the variance is close to zero, 30 TTS reference data RDs are used to examine the variation in voice quality. On the other hand, when the variance is large, a wide variety of voice quality can be obtained from within 10 TTS reference data RDs.
[0123] By doing the above, it is possible to suggest to the user 10 multiple speakers with similar topics but with variations in the speaker's voice quality. In this case, it is possible to display, for example, a screen such as that shown in Figure 18 on the display 120 of the information terminal 100.
[0124] The topic shown is "soccer" as an example. As in Fig. 13, a text box 31, a speaker ID 32, a synthesis start button 33, a play button 34, and a reference URL 35 are displayed on the display 120. However, the play button 34, the speaker ID 32, and the reference URL 35 are displayed as a synthetic speech list 36. In other words, a list of multiple speakers who are candidates for use in the content CT1 is displayed.
[0125] The synthesized speech list 36 displays speakers in descending order of cosine similarity of their topic vectors. As described above, these speakers have different voice qualities, including optimal speakers, orthogonal speakers, and backward speakers. Each speaker has the topic "soccer," which matches the topic of the text information entered in the text box 31. When selecting a speaker ID, a link to reference content CT2 featuring speakers with a variety of voice qualities is displayed as a reference URL 35.
[0126] 18, five speakers are presented, and the user 10 can play back the voice of each speaker by pressing the play button 34. The user 10 can also confirm the voice of the reference content CT2 that was referenced to select the speaker ID by operating the reference URL 35.
[0127] Here, let us consider the advantage of proposing multiple speaker candidates for one piece of content CT1. For example, let us consider a case where this technology is applied to video content such as online lessons for elementary school students.
[0128] When there are subjects such as Japanese, mathematics, science, and social studies, rather than performing a topic analysis on the text data TD for these four subjects together, a topic analysis for each subject can be performed to obtain a voice that is characteristic of each teacher. When only the topic vectors of the text data TD are considered, for example, the voices of the Japanese and social studies teachers may be similar. The user 10 may wish to change the speaker for each subject to avoid monotony.
[0129] In such cases, a topic search is performed using the topic vector TV for Japanese language and society, and orthogonal speakers and reverse speakers are selected from the reference data with the highest cosine similarity, thereby avoiding duplication of similar voices.
[0130] It is also possible to automatically divide a long piece of text into multiple topics and change the speaker. For example, even in video content for Japanese language classes, there are various topics for the unit, such as "novels," "criticism," and "poetry." For example, you want the reader to read "poetry" with emotion. In such cases, by performing topic analysis on a paragraph-by-paragraph basis, it becomes possible to suggest a speaker for a certain group. Paragraph detection can be done by detecting indents and blank lines.
[0131] 19 illustrates an example of the configuration of an information processing device 70 that can be used as the speech synthesis server 200, the content analysis server 300, the database 400, and the information terminal 100 in the above-described TTS speaker suggestion system 1. The information processing device 70 can be configured as, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like.
[0132] 19 executes various processes in accordance with programs stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate. In the CPU 71, the programs enable the CPU 71 to realize functions for performing various controls and calculations shown in FIGS. 2, 3, 4, and 5.
[0133] In addition, a processor other than the CPU 71 may also be provided, such as a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), or an AI (Artificial Intelligence) processor.
[0134] The CPU 71, ROM 72, RAM 73, and nonvolatile memory unit 74 are interconnected via a bus 83. The bus 83 is also connected to an input / output interface 75.
[0135] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. The input unit 76 detects an operation by the user 10, and a signal corresponding to the input operation is interpreted by the CPU 71.
[0136] The input / output interface 75 is also connected, either integrally or separately, to a display unit 77 such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 78 such as a speaker.
[0137] The display unit 77 performs various displays as a user interface. The display unit 77 is configured, for example, by a display device provided in the housing of the information processing device 70 or a separate display device connected to the information processing device 70. The display unit 77 displays various images on the display screen based on instructions from the CPU 71. The display unit 77 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 71.
[0138] The input / output interface 75 may be connected to a storage unit 79 configured with a solid state drive (SSD) or a hard disk drive (HDD), or a communication unit 80 configured with a modem or the like. The storage unit 79 can be used to store various data. Databases can also be constructed in the storage unit 79. For example, the retained speaker data unit 240 of the speech synthesis server 200, the storage unit 340 of the content analysis server 300, the storage unit 410 of the database 400, etc. can be configured using the storage unit 79.
[0139] The communication unit 80 performs communication processing via the network 600. For example, the network communication unit 150 of the information terminal 100, the network communication unit 260 of the speech synthesis server 200, the network communication unit 370 of the content analysis server 300, and the network communication unit 440 of the database 400 can be configured using the communication unit 80.
[0140] A drive 82 is also connected to the input / output interface 75 as needed, and a removable recording medium 81 such as a flash memory, a memory card, a magnetic disk, an optical disk, or a magneto-optical disk is appropriately attached. The drive 82 allows data files such as image files and various computer programs to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the audio output unit 78. In addition, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as needed.
[0141] In this information processing device 70, software can be installed via network communication by the communication unit 80 or via a removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.
[0142] The information terminal 100, the voice synthesis server 200, the content analysis server 300, and the database 400 can be configured using such an information processing device 70. The configuration of the information terminal 100 shown in Fig. 2, the configuration of the voice synthesis server 200 shown in Fig. 3, the configuration of the content analysis server 300 shown in Fig. 4, and the configuration of the database 400 shown in Fig. 5 can be realized by the hardware configuration of the information processing device 70 shown in Fig. 19 and software installed therein.
[0143] 10. Summary and Modifications The above embodiment provides the following effects.
[0144] The speech synthesis server 200 of this embodiment includes a speaker search unit 230 that searches among multiple pieces of speaker data based on the topic of the text data TD to select speaker data, and a speech synthesis unit 250 that generates synthetic speech for the text data TD using the speaker data selected by the speaker search unit 230 (see FIG. 3 ). That is, speaker data with a voice quality corresponding to the topic (topic or genre) of the text data TD is selected, and the speech for reading the text data TD is synthesized using that speaker data. This makes it possible to provide the user 10 with synthetic speech data AD using speaker data with a voice quality that matches the topic of the text data TD of the content CT1, rather than providing synthetic speech data AD with various voice qualities for the text data TD indiscriminately.
[0145] In the speech synthesis server 200 of the embodiment, the speaker search unit 230 acquires TTS reference data RD selected based on the topic of the text data TD, and selects speaker data from among multiple speaker data based on the similarity between the speech features indicated in the TTS reference data RD and the speech features of the speaker data (see FIG. 11 ). By acquiring the TTS reference data RD, information on speech features RDX that are generally considered to match the topic of the text data TD can be obtained. Therefore, the speech synthesis server 200 can select speaker data appropriate for the topic of the text data TD to be processed from the speaker data stored in the stored speaker data unit 240.
[0146] In the embodiment, the speech synthesis server 200 transmits text data TD to an external database 400 via the reference data acquisition unit 220 and receives TTS reference data RD from the database 400 (see FIGS. 6 and 11 ). By acquiring the TTS reference data RD from the database 400, the speech synthesis server 200 does not need to store a large number of TTS reference data RD. The speech synthesis server 200 can select appropriate speaker data from the speaker data it holds based on the speech feature values RDX of the TTS reference data RD corresponding to the text data TD to be processed. In other words, appropriate speaker data can be selected from the speaker data stored in the speech synthesis server 200 without performing processes such as saving, adding, and managing the TTS reference data RD.
[0147] In this embodiment, the TTS reference data RD includes a topic vector RDT that serves as an index for classifying the topic of the reference content CT2 (an index for determining similarity / dissimilarity) (see FIG. 5). This allows the database 400 to compare the topic vector RDT of the TTS reference data RD with the topic vector TV of the text data TD to be processed, received from the speech synthesis server 200, and select TTS reference data RD that corresponds to the topic of the text data TD. This allows the selection of TTS reference data RD that is appropriate for the topic of the text data TD to be processed.
[0148] The TTS reference data RD also includes speech features obtained by extracting features from the speech data of the reference content CT2 (see FIG. 5). By including the speech features RDX in the TTS reference data RD, the speech synthesis server 200 can select speaker data with a voice quality similar to the speech features RDX, which becomes speaker data with a voice quality that matches the topic of the text data TD.
[0149] The TTS reference data RD also includes information indicating the reference content CT2 used to create it (see FIG. 5). For example, the information indicating the reference content CT2 includes the content's URL (RDU). This enables a user interface that allows the user 10 to view the reference content CT2, as shown in FIGS. 13 and 18. The user 10 can use this information as a reference for content creation by knowing which reference content CT2 was used to select the synthesized speech by the speech synthesis server 200.
[0150] The speech synthesis server 200 of this embodiment includes a network communication unit 260 that transmits information about the speaker data selected by the speaker search unit 230 and the synthetic speech generated by the speech synthesis unit 250 to the information terminal 100 that is the sender of the text data (see FIG. 3). As described in FIG. 11, the speech synthesis server 200 transmits the speaker ID and synthetic speech data AD to the information terminal 100 via the network communication unit 260. This allows a service to be provided in which synthetic speech suited to content CT1 is suggested to the user 10 on a display screen such as that shown in FIG. 13 or 18.
[0151] In the speech synthesis server 200 of this embodiment, the network communication unit 260 performs processing to transmit information about the reference content CT2 contained in the TTS reference data RD to the information terminal 100 that is the sender of the text data TD (see FIG. 11). This allows the reference URL 35 to be displayed on a display screen such as that shown in FIG. 13 or 18, providing a path for the user 10 to view the reference content CT2.
[0152] In the embodiment, the speaker search unit 230 of the speech synthesis server 200 selects speaker data for each of a plurality of TTS reference data RD selected based on the topic of the text data TD based on the similarity between the speech features indicated in the TTS reference data RD and the speech features of the stored speaker data. By acquiring a plurality of TTS reference data RD (e.g., TTS reference data RD#1, RD#x, and RD#y in FIG. 16 ), the speech synthesis server 200 can obtain a plurality of pieces of speech feature information RDX that match the topic of the text data TD. Therefore, by selecting speaker data from the speaker data stored in the stored speaker data unit 240 based on each of the TTS reference data RD#1, RD#x, and RD#y, the speech synthesis server 200 can select a plurality of speaker data that match the topic of the content CT1 and suggest the voice quality of each to the user 10.
[0153] In the embodiment, the network communication unit 260 transmits information about speaker data selected based on multiple TTS reference data RD as information to be displayed as a list on the information terminal 100. For example, speaker IDs 32 and the like may be displayed as a list in the synthesized speech list 36 of Fig. 18. This allows the user 10 to preview multiple voice qualities that are selected as candidates based on the condition that they match the topic of the content CT1.
[0154] The plurality of TTS reference data RD are selected from among the plurality of TTS reference data RD selected in descending order of similarity to the topic of the text data TD. An example has been given in which the topic of the original reference content CT2 includes a first reference data item that is determined to have the highest similarity to the topic of the text data TD and one or more second reference data items selected based on a similarity evaluation of the first reference data item. For example, as shown in FIG. 16 , a plurality of TTS reference data RD (RD#1 to RD#10) are selected. Among them, the first reference data item (TTS reference data RD#1) and one or more second reference data items (TTS reference data RD#x, RD#y) that are determined to have the highest similarity are selected. This allows the selection of TTS reference data RD with a variety of voice qualities that match the topic using a similarity evaluation method.
[0155] In this case, the second reference data is reference data whose cosine similarity is evaluated as being orthogonal or opposite to the speech features of the first reference data. This results in the multiple TTS reference data RD#1, RD#x, and RD#y having speech features RDX with dissimilar voice qualities. Based on this, the speech synthesis server 200 can provide the user 10 with speaker data with variations of different voice qualities.
[0156] Furthermore, the number of multiple reference data selected in descending order of similarity to the topic of text data TD is set according to the variance of the speech features (see, for example, Figures 16 and 17). This allows for the number of TTS reference data RD initially selected in descending order of similarity to be increased to increase the variance, if the multiple TTS reference data RD have roughly similar voice qualities. This prevents the TTS reference data RD#x and RD#y selected based on the similarity evaluation from having a voice quality similar to that of TTS reference data RD#1. In other words, controlling the parameter for selecting TTS reference data RD according to the variance allows for a wide variety of voice qualities to be ultimately proposed to user 10.
[0157] The TTS speaker suggestion system 1 of this embodiment includes a content analysis device (content analysis server 300) that analyzes reference content CT2 to generate TTS reference data RD containing information about the topic and information about speech characteristics. The TTS speaker suggestion system 1 also includes a database 400 that stores the TTS reference data RD generated by the content analysis server 300 and selects TTS reference data RD based on the topic of text data TD. The TTS speaker suggestion system 1 further includes a speech synthesis server 200, which includes the speaker search unit 230 and speech synthesis unit 250 described above. In this TTS speaker suggestion system 1, the content analysis server 300 and database 400 generate and store TTS reference data RD based on various reference content CT2. Using these information resources, the speech synthesis server 200 can provide synthesized speech that matches the text data TD received from the information terminal 100. The more substantial the quality and quantity of the TTS reference data RD, the more the speech synthesis server 200 can provide the user 10 with synthesized speech with a voice quality that more closely matches the text data TD.
[0158] The program of the embodiment is a program that causes, for example, a CPU, a DSP (digital signal processor), an AI processor, or an information processing device 70 that includes these to execute the processing shown in Fig. 11. That is, the program of the embodiment is a program that causes the information processing device to execute a speaker search process that selects speaker data from a plurality of speaker data by performing a search based on the topic of the text data TD, and a speech synthesis process that generates synthetic speech of the text data TD using the speaker data selected in the speaker search process.
[0159] By using such a program, an information processing device serving as the speech synthesis server 200 of the embodiment can be realized in, for example, a computer device, a mobile terminal device, or other device capable of executing information processing.
[0160] Such a program can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such a program can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
[0161] Furthermore, such a program is suitable for widely providing the information processing device 70 that constitutes the speech synthesis server 200 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, an imaging device, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can be made into the information processing device 70 that functions as the speech synthesis server 200 of the present disclosure.
[0162] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.
[0163] The present technology may also be configured as follows. (1) An information processing device comprising: a speaker search unit that selects speaker data from a plurality of speaker data by performing a search based on a topic of text data; and a speech synthesis unit that generates synthetic speech data for the text data using the speaker data selected by the speaker search unit. (2) The information processing device according to (1), wherein the speaker search unit acquires reference data selected based on the topic of the text data, and selects speaker data from the plurality of speaker data based on a similarity between speech features indicated in the reference data and speech features of the speaker data. (3) The information processing device according to (2), wherein the text data is transmitted to an external database and the reference data is received from the database. (4) The information processing device according to (2) or (3), wherein the reference data includes a topic vector that serves as an index for classifying the topics of reference content used to create the reference data. (5) The information processing device according to any of (2) to (4), wherein the reference data includes speech features obtained by extracting features from speech data of reference content used to create the reference data. (6) The information processing device according to any of (2) to (5), wherein the reference data includes information indicating reference content used to create the reference data. (7) The information processing device according to any of (1) to (6), further comprising a communication unit that performs processing to transmit information on speaker data selected by the speaker search unit and synthetic speech data generated by the speech synthesis unit to an information terminal that has transmitted the text data. (8) The information processing device according to (7), wherein the speaker search unit acquires reference data selected based on a topic of the text data, and selects speaker data from a plurality of speaker data based on a similarity between speech features indicated in the reference data and speech features of the speaker data, and the communication unit performs processing to transmit information on the reference content used to create the reference data, which is included in the reference data, to the information terminal that has transmitted the text data.(9) The information processing device according to any one of (1) to (8), wherein the speaker search unit selects speaker data for each of a plurality of reference data selected based on the topic of the text data based on a similarity between a voice feature indicated in the reference data and a voice feature of the speaker data. (10) The information processing device according to (9), further comprising a communication unit that performs processing for transmitting information of the speaker data selected by the speaker search unit and a synthetic voice generated by the voice synthesis unit to an information terminal that is a source of the text data, wherein the information of the speaker data selected based on the plurality of reference data is transmitted as information displayed as a list on the information terminal. (11) The information processing device according to (9) or (10), wherein the plurality of reference data are reference data further selected from a plurality of reference data selected in descending order of similarity with the topic of the text data, and wherein the topic of the original reference content includes first reference data that is determined to have the highest similarity to the topic of the text data, and one or more second reference data selected based on a similarity evaluation with the first reference data. (12) The information processing device according to (11), wherein the second reference data is reference data evaluated for similarity such that cosine similarity is orthogonal or opposite in direction based on speech features of the first reference data. (13) The information processing device according to (11) or (12), wherein the number of multiple reference data selected in descending order of similarity to the topic of the text data is set according to the variance of speech features. (14) An information processing method, wherein an information processing device executes: a speaker search process for selecting speaker data from multiple speaker data by performing a search based on the topic of the text data; and a speech synthesis process for generating synthetic speech for the text data using the speaker data selected in the speaker search process. (15) A program for causing an information processing device to execute: a speaker search process for selecting speaker data from multiple speaker data by performing a search based on the topic of the text data; and a speech synthesis process for generating synthetic speech for the text data using the speaker data selected in the speaker search process.(16) An information processing system comprising: a content analysis device that analyzes reference content to generate reference data including information on topics and information on voice features; a database that stores the reference data generated by the content analysis device and selects reference data based on the topic of text data; and a voice synthesis device, wherein the voice synthesis device comprises: a speaker search unit that acquires the reference data selected in the database based on the topic of the text data and selects speaker data from multiple speaker data based on the similarity between the voice features indicated in the reference data and the voice features of speaker data; and a voice synthesis unit that generates synthetic voice data of the text data using the speaker data selected by the speaker search unit.
[0164] 1 TTS speaker suggestion system 70 Information processing device 71 CPU 100 Information terminal 200 Speech synthesis server 210 Text-to-phoneme symbol conversion unit 220 Reference data acquisition unit 230 Speaker search unit 240 Retained speaker data unit 250 Speech synthesis unit 260 Network communication unit 300 Content analysis server 400 Database 500 Content provider RDT Topic vector RDX Speech feature RDU Content URL TD Text data AD Synthesized speech data RD TTS reference data TM Topic model TV Topic vector XV Speech feature CT1 Content (created by user) CT2 Reference content (for analysis)
Claims
1. An information processing device comprising: a speaker search unit that selects speaker data from among a plurality of speaker data by performing a search based on the topic of text data; and a voice synthesis unit that generates synthetic voice data for the text data using the speaker data selected by the speaker search unit.
2. The information processing device according to claim 1, wherein the speaker search unit acquires reference data selected based on the topic of the text data, and selects speaker data from among a plurality of speaker data based on the similarity between the voice characteristics indicated in the reference data and the voice characteristics of the speaker data.
3. The information processing apparatus according to claim 2, wherein the text data is transmitted to an external database, and the reference data is received from the database.
4. The information processing device according to claim 2, wherein the reference data includes a topic vector that is an index for classifying topics of the reference content used to create the reference data.
5. The information processing device according to claim 2, wherein the reference data includes audio features obtained by extracting features from audio data of the reference content used to create the reference data.
6. The information processing device according to claim 2, wherein the reference data includes information indicating reference content used to create the reference data.
7. An information processing device according to claim 1, further comprising a communication unit which performs processing for transmitting information on the speaker data selected by the speaker search unit and synthetic voice data generated by the voice synthesis unit to the information terminal which is the source of the text data.
8. The information processing device according to claim 7, wherein the speaker search unit acquires reference data selected based on the topic of the text data, and selects speaker data from among a plurality of speaker data based on the similarity between the voice characteristics indicated in the reference data and the voice characteristics of the speaker data, and the communication unit performs processing to transmit information contained in the reference data regarding the reference content used to create the reference data to the information terminal that sent the text data.
9. The information processing device according to claim 1, wherein the speaker search unit selects speaker data for each of a plurality of reference data selected based on the topic of the text data based on the similarity between the voice characteristics indicated in the reference data and the voice characteristics of the speaker data.
10. An information processing device as described in claim 9, further comprising a communication unit that performs processing to transmit information on speaker data selected by the speaker search unit and synthetic voice generated by the voice synthesis unit to an information terminal that is the source of the text data, and the information on speaker data selected based on a plurality of the reference data is transmitted as information displayed in a list on the information terminal.
11. The information processing device of claim 9, wherein the multiple reference data are reference data further selected from multiple reference data selected in order of similarity to the topic of the text data, and the topic of the original reference content includes a first reference data that is determined to have the highest similarity to the topic of the text data, and one or more second reference data selected based on a similarity evaluation with the first reference data.
12. The information processing device according to claim 11, wherein the second reference data is reference data whose similarity is evaluated as being orthogonal or opposite in cosine similarity based on the speech features of the first reference data.
13. The information processing device according to claim 11, wherein the number of the plurality of reference data selected in order of the degree of similarity to the topic of the text data is set according to the variance of the speech features.
14. An information processing method in which an information processing device executes a speaker search process that selects speaker data from a plurality of speaker data by conducting a search based on the topic of text data, and a voice synthesis process that generates synthetic voice for the text data using the speaker data selected in the speaker search process.
15. A program that causes an information processing device to execute a speaker search process that selects speaker data from multiple speaker data by searching based on the topic of text data, and a voice synthesis process that generates synthetic voice for the text data using the speaker data selected in the speaker search process.
16. An information processing system comprising: a content analysis device that analyzes reference content and generates reference data including information on topics and information on voice features; a database that stores the reference data generated by the content analysis device and selects reference data based on the topic of text data; and a voice synthesis device, wherein the voice synthesis device comprises: a speaker search unit that acquires reference data selected in the database based on the topic of the text data and selects speaker data from a plurality of speaker data based on the similarity between the voice features indicated in the reference data and the voice features of speaker data; and a voice synthesis unit that generates synthetic voice data of the text data using the speaker data selected by the speaker search unit.
Citation Information
Patent Citations
Pressurized water type nuclear reactor
JP1989296193A
Voice synthesizing system, voice synthesizing method, server, storage medium, program transmitting device, voice synthetic data storage medium and voice outputting equipment
JP2002023777A
Voice synthesizing system
JP2002358092A
Read-aloud system
JP2003122387A
Voice output method, voice output system and program
JP2020154050A
