Electronic device and method for performing speech synthesis
Patent Information
- Application Number
- TW114105838
- Authority / Receiving Office
- TW · TW
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-02-18
- Publication Date
- 2026-09-01
- Estimated Expiration
- 2045-02-17
AI Technical Summary
Existing speech synthesis systems face challenges in flexibility, naturalness, emotional expression, computational efficiency, and privacy due to high resource requirements, especially in edge devices and cross-language applications.
A speech synthesis system comprising a voice channel separation module, automatic speech recognition, speech emotion recognition, feature-adaptive text-to-speech, and audio mixing modules, which can operate on edge devices, adapt to target voice timbre and emotions, and select suitable models for real-time synthesis.
The system produces fluent, natural, and emotionally consistent speech on edge devices, reducing latency and computational costs while enhancing privacy and security.
Smart Images
Figure TWG2TA001073679_001 
Figure TWG2TA001073679_002 
Figure TWG2TA001073679_003
Abstract
Description
[Technical Field]
[0001] This invention relates to speech synchronization, and more particularly to a real-time and emotion-aware speech synthesis system and related methods. [Previous Technology]
[0002] For some existing speech synthesis systems (such as dubbing systems), appropriate speech segments may be selected from a speech database and spliced together to quickly generate synthesized speech. The disadvantages are poor flexibility in handling unexpected text or rare vocabulary, limited fluency and naturalness of the voice, and inconsistency in emotion recognition and emotional expression processing (especially for cross-language applications). For some existing speech synthesis systems, deep learning models can be trained based on large amounts of labeled speech and text data to automatically synthesize speech. The disadvantage is that a high number of model parameters is usually required to apply to different scenarios, significantly reducing computational efficiency. Furthermore, these existing speech synthesis systems can only fine-tune parameters in advance during the training phase based on the timbre characteristics of the target speech (such as breathing habits and articulation), which may cause problems in high-fidelity speech synthesis. Additionally, due to the high computational resource requirements and memory consumption of deep learning models, existing speech synthesis systems typically rely on large data centers or cloud services for computation, leading to latency, high computational costs, and privacy and security issues. Therefore, there is a great need for a novel speech synthesis system that can run on edge devices and produce fluent, natural and emotional speech output according to the needs of different application scenarios. [Summary of the Invention]
[0003] Therefore, one of the objectives of the present invention is to provide an instantaneous and emotion-aware speech synthesis system and related methods capable of operating on edge devices to solve the above-mentioned problems.
[0004] According to an embodiment of the present invention, an electronic device is provided, wherein the electronic device includes a storage device and a processor, and the storage device is used to store program code. The processor is used to load and execute the program code, wherein the program code instructs the processor to execute a speech synthesis system, and the speech synthesis system includes a human voice channel separation module, an automatic speech recognition module, a speech emotion recognition model, a feature-adaptive text-to-speech system, and an audio multi-channel mixing module. The human voice channel separation module is used to perform a separation operation on an audio input signal to generate multiple human voice channels and a single non-human voice channel. The automatic speech recognition module is used to select a target human voice channel based on a source language text from multiple human voice channels. The speech emotion recognition model is used to perform a prediction operation on each of the multiple human voice channels to generate a prediction result, wherein the prediction result indicates an emotion classification for each human voice channel. The feature-adaptive text-to-speech system performs speech synthesis based on source language text, target language text, target human voice channels, and emotion classification to produce a synthesized human voice. The audio multi-channel mixing module mixes a single non-human voice channel with the synthesized human voice to produce an audio output signal.
[0005] According to an embodiment of the present invention, a method for performing speech synthesis is provided. The method includes: performing a separation operation on an audio input signal to generate a plurality of human voice channels and a single non-human voice channel; selecting a target human voice channel from the plurality of human voice channels based on a source language text; performing a prediction operation on each of the plurality of human voice channels to generate a prediction result, wherein the prediction result indicates an emotion classification for each human voice channel; performing a speech synthesis operation based on the source language text, a target language text, the target human voice channel, and the emotion classification to generate a synthesized human voice; and performing a mixing operation on the single non-human voice channel and the synthesized human voice to generate an audio output signal.
[0006] One advantage of the present invention is that, through the speech synthesis system of the present invention, feature vectors indicating the timbre characteristics of the target human voice can be extracted from the target human voice, and speech synthesis operations can be performed based on these feature vectors, enabling the generated synthesized human voice to adapt to the timbre characteristics of the target human voice. Furthermore, the speech synthesis system of the present invention can automatically identify and adapt to the emotional needs of the current scene, selecting a suitable speech synthesis model from multiple candidate speech synthesis models. This improves the accuracy of emotion recognition and maintains the consistency and authenticity of emotional expression during speech synthesis. Additionally, the speech synthesis system of the present invention can run on resource-constrained edge devices, significantly reducing latency, computational costs, and enhancing privacy and security.
Implementation Method
[0007] Figure 1 is a schematic diagram of an electronic device 10 according to an embodiment of the present invention. For example, but not limited to, the electronic device 10 may be a multi-functional mobile phone, a tablet computer, a portable device, and a personal computer (e.g., a desktop computer or a laptop computer). The electronic device 10 may include a processor 12 and a storage device 14 (e.g., memory). The processor 12 may be a single-core processor or a multi-core processor. The storage device 14 may be used to store program code PROG, source language text (SLT), target language text (TLT), and multiple candidate speech synthesis models CSSM_1 to CSSM_N, where "N" may be a positive integer greater than one. The processor 12 has software execution capability. When the processor 12 loads and executes the program code PROG, the program code PROG instructs the processor 12 to execute the speech synthesis system proposed in this invention. The electronic device 10 can be regarded as a computer system using a computer program product, wherein the computer program product includes a computer-readable medium having program code PROG, that is, the speech synthesis system proposed in this invention can be implemented on the electronic device 10.
[0008] For example, when a video with the original audio in the first language is played on the player, and the speech synthesis system is pre-set with the first language subtitle text (i.e., source language text SLT) corresponding to the original audio and the translated second language subtitle text (i.e., target language text TLT), the speech synthesis system can generate the corresponding target audio based on the original audio, the first language subtitle text, and the second language subtitle text. The target audio can simulate the timbre characteristics (such as breathing habits and articulation) and related emotions of the original audio to provide higher fidelity speech synthesis and ensure the consistency and authenticity of the emotional expression between the original audio and the target audio.
[0009] Figure 2 is a schematic diagram of a speech synthesis system 20 according to an embodiment of the present invention. The speech synthesis system 20 can be a real-time dubbing system based on artificial intelligence (AI) and can include multiple modules / models implemented by the processor 12 executing program code PROG. The source language text SLT, the target language text TLT, and the audio input signal AU_IN can be input to the speech synthesis system 20, and the speech synthesis system 20 can be used to perform speech synthesis processing on the audio input signal AU_IN according to the source language text SLT and the target language text TLT to generate an audio output signal AU_OUT containing a synthesized vocal voice SVV, wherein the audio input signal AU_IN corresponds to the source language text SLT, and the synthesized vocal voice SVV corresponds to the target language text TLT. As shown in Figure 2, the speech synthesis system 20 may include a vocal voice channel separation module 200, an automatic speech recognition (ASR) module 202, a speech emotion recognition model 204, a feature-adaptive text-to-speech (TTS) system 206 (labeled as "feature-adaptive TTS system" in Figure 2 for simplicity), and an audio multi-channel mixing module 208.
[0010] The voice channel separation module 200 can separate the audio input signal AU_IN into multiple vocal voice channel (VVC) and a single non-voice channel (NVVC) (e.g., background noise). For example, the voice channel separation module 200 may include a demixing model based on the MDX-NET architecture and a speaker extraction model based on the X-SepFormer architecture. The demixing model can separate the audio input signal AU_IN into a single vocal channel and a single non-voice channel (NVVC). The speaker extraction model can separate the speech of multiple speakers in a single vocal channel into multiple vocal channel (VVC) respectively.
[0011] The automatic speech recognition module 202 can receive multiple human voice channel VVCs and source language text SLT, and select the target human voice channel TVVC based on the source language text SLT from the multiple human voice channel VVCs. The automatic speech recognition module 202 may include an AI-based model (e.g., a Whisper model), wherein the input of the Whisper model may be multiple human voice channel VVCs (e.g., multiple audio signals carrying different human voices), and the Whisper model can parse the text described by each human voice channel in the multiple human voice channel VVCs to generate a parsed text as the output of the Whisper model. The automatic speech recognition module 202 can perform a comparison operation between the parsed text and the source language text SLT to generate a comparison result COM_RT for selecting the target human voice channel TVVC, wherein the comparison result COM_RT can indicate the similarity between the parsed text and the source language text SLT.
[0012] The speech emotion recognition model 204 can be an AI-based model (e.g., a model based on the ECAPA-TDNN architecture). The input to this model can be multiple human voice channels (VVCs) (e.g., multiple audio signals carrying different human voices), and the model can perform a prediction operation on each of the multiple human voice channel VVCs to generate a corresponding emotion classification EC as the model's output. Specifically, the emotion classification EC can be further configured during the model's training phase according to design requirements. For example, the speaker's emotions can be pre-classified as joy, anger, sorrow, and happiness as emotion classification ECs. Alternatively, the emotion classification ECs can be further divided into male / female voices or high / low voices, depending on design considerations. That is, the types of emotion classification ECs can be predetermined during the model's training phase, and the model can be trained based on the determined emotion classification ECs to complete the emotion recognition operation.
[0013] The feature-adaptive text-to-speech system 206 can receive source language text SLT, target language text TLT, target human voice channel TVVC, and emotion classification EC, and obtain a synthesized human voice SVV of the target language with the timbre features of the audio input signal AU_IN based on the above inputs. Specifically, refer to Figure 3. Figure 3 is a schematic diagram of a feature-adaptive text-to-speech system 300 (labeled as "feature-adaptive TTS system" in Figure 3 for simplicity) according to an embodiment of the present invention, wherein the feature-adaptive text-to-speech system 206 shown in Figure 2 can be implemented by the feature-adaptive text-to-speech system 300. As shown in Figure 3, the feature-adaptive text-to-speech system 300 may include a text-to-phoneme (TP) module 302 (labeled as "TTP module" in Figure 3 for simplicity), a reference feature extraction module 304, and a speech synthesis model selection module 306.
[0014] The text-to-phoneme module 302 can perform tokenization operations on the source language text SLT and the target language text TLT respectively to generate the source phoneme ST and the target phoneme TT. In particular, the source language text SLT and the target language text TLT can be mapped to the source phoneme ST and the target phoneme TT respectively by a mapping method.
[0015] The reference feature extraction module 304 can be an AI-based model (e.g., a HuberT-based model) and can perform extraction operations on the target human voice channel TVVC (e.g., an audio signal carrying the target human voice) to generate a feature vector FV. The feature vector FV indicates the timbre characteristics (e.g., breathing habits and articulation) of the target human voice channel TVVC (or the target human voice), and can be input into a subsequent speech synthesis model to achieve adaptive adaptation of the timbre characteristics of the target human voice. For example, the reference feature extraction module 304 can treat the target human voice channel TVVC as a sequence and perform tokenization and related transformer computation operations on the sequence to obtain the feature vector FV.
[0016] The speech synthesis model selection module 306 can select a speech synthesis model that matches the emotion indicated by the emotion classification EC from the candidate speech synthesis models CSSM_1~CSSM_N in the storage device 14 according to the emotion classification EC, and use it as the speech synthesis model 308 to better simulate the emotion corresponding to the target human voice channel TVVC, so as to achieve the purpose of emotion perception and emotional expression. In this way, when a trade-off is required between model performance and model efficiency, the speech synthesis model selection module 306 can avoid the problem of a single model needing to adapt to different use scenarios (or different emotion classifications). It can reduce the size of the model and the resource requirements, and thus enable speech synthesis to be performed on edge devices with limited resources.
[0017] The speech synthesis model 308 can generate a synthesized human voice SVV based on the feature vector FV, the source phoneme ST, and the target phoneme TT. Specifically, each candidate speech synthesis model (e.g., speech synthesis model 308) among the candidate speech synthesis models CSSM_1 to CSSM_N can be composed of a first model based on a transformer-decoder architecture and a second model based on VITS. The first model can generate an embedding vector with the timbre features of the target human voice and corresponding to the target phoneme TT based on the source phoneme ST, the target phoneme TT, and the feature vector FV. The second model can generate a synthesized human voice SVV with the timbre features of the target human voice based on the embedding vector. In addition, a speed parameter can be set for the speech synthesis model 308 to control the speech rate of the synthesized human voice SVV, thereby further controlling the overall audio length of the synthesized human voice SVV.
[0018] Return to Figure 2. After generating a single non-human voice channel NVVC and a synthesized human voice SVV by the human voice channel separation module 200 and the feature-adaptive text-to-speech system 206 respectively, the audio multi-channel mixing module 208 can perform a mixing operation on the single non-human voice channel NVVC and the synthesized human voice SVV to generate an audio output signal AU_OUT.
[0019] It should be noted that when the processor 12 loads and executes the program code PROG, the program code PROG can also instruct the processor 12 to execute a real-time video delay module 210, which can delay a video input signal VIDEO_IN according to the system delay time of the speech synthesis system 20 to achieve the function of real-time dubbing. Specifically, assuming an audio-visual signal includes an audio input signal AU_IN and a video input signal VIDEO_IN, when the audio input signal AU_IN is input to the speech synthesis system 20, the video input signal VIDEO_IN will be input to the real-time video delay module 210, so that the video input signal VIDEO_IN is delayed by a specific time (e.g., the system delay time of the speech synthesis system 20) and then output as a video output signal VIDEO_OUT. In this way, the purpose of aligning the audio output signal AU_OUT with the video output signal VIDEO_OUT can be achieved.
[0020] Figure 4 is a flowchart of a method for performing speech synthesis according to an embodiment of the present invention. If the same result can be obtained, the steps do not necessarily have to be performed sequentially according to the process shown in Figure 4. For example, the method shown in Figure 4 can be implemented by the speech synthesis system 20 shown in Figure 2 (in particular, the modules / models therein).
[0021] In step S400, a separation operation is performed on the audio input signal AU_IN to generate multiple human voice channel VVCs and a single non-human voice channel NVVC.
[0022] In step S402, the target human voice channel TVVC is selected from multiple human voice channels VVC according to the source speech text SLT.
[0023] In step S404, a prediction operation is performed on each of the multiple human voice channels (VVC) to produce a prediction result, wherein the prediction result indicates the emotion classification (EC) of each human voice channel.
[0024] In step S406, a speech synthesis operation is performed based on the source language text SLT, the target language text TLT, the target human voice channel TVVC, and the emotion classification EC to generate a synthesized human voice SVV.
[0025] In step S408, a mixing operation is performed on the single non-human voice channel NVVC and the synthesized human voice SVV to generate an audio output signal AU_OUT.
[0026] Since those skilled in the art can easily understand the operation of each step shown in Figure 4 through the content of the speech synthesis system 20 shown in Figure 2 in the above-mentioned specification, similar content will not be repeated here in this embodiment for the sake of brevity.
[0027] In summary, the speech synthesis system of this invention can extract feature vectors indicating the timbre characteristics of the target human voice from the target human voice, and perform speech synthesis operations based on these feature vectors. This enables the synthesized human voice to adapt to the timbre characteristics of the target human voice. Furthermore, the speech synthesis system of this invention can automatically identify and adapt to the emotional needs of the current scene, selecting a suitable speech synthesis model from multiple candidate speech synthesis models. This improves the accuracy of emotion recognition and maintains consistency and authenticity of emotional expression during speech synthesis. Additionally, the speech synthesis system of this invention can run on resource-constrained edge devices, significantly reducing latency, computational costs, and enhancing privacy and security. The above description is merely a preferred embodiment of this invention. All equivalent variations and modifications made within the scope of the claims of this invention should be considered within the scope of this invention. [Simplified Explanation of the Diagram]
[0028] Figure 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 2 is a schematic diagram of a speech synthesis system according to an embodiment of the present invention. Figure 3 is a schematic diagram of a feature-adaptive text-to-speech system according to an embodiment of the present invention. Figure 4 is a flowchart of a method for performing speech synthesis according to an embodiment of the present invention.
Claims
1. An electronic device comprising: a storage device for storing code; and a processor for loading and executing the code, wherein the code instructs the processor to execute a speech synthesis system, and the speech synthesis system comprising: a voice channel separation module for performing a separation operation on an audio input signal to generate a plurality of human voice channels and a single non-human voice channel; an automatic speech recognition module for selecting a target human voice channel based on source language text from the plurality of human voice channels; and a speech emotion recognition model for performing a prediction operation on each of the plurality of human voice channels to generate a prediction result, wherein the prediction result indicates an emotion classification for each human voice channel. A feature-adaptive text-to-speech system for performing a speech synthesis operation based on the source language text, a target language text, the target human voice channel, and the emotion classification to generate a synthesized human voice; and an audio multi-channel mixing module for mixing the single non-human voice channel with the synthesized human voice to generate an audio output signal.
2. The electronic device as described in claim 1, wherein the voice channel separation module comprises: an anti-mixing model for separating the audio input signal into a single voice channel and the single non-voice channel; and a speaker separation model for separating the single voice channel into the plurality of voice channels.
3. The electronic device as described in claim 1, wherein the automatic speech recognition module is further configured to: parse a text in each of the plurality of human voice channels to generate a parsed text; and perform a comparison operation between the parsed text and the source language text to generate a comparison result for selecting the target human voice channel, wherein the comparison result indicates a similarity between the parsed text and the source language text.
4. The electronic device as described in claim 1, wherein the feature-adaptive text-to-speech system comprises: a text-to-phoneme module for performing a tokenization operation on the source language text and the target language text respectively to generate a source phoneme and a target phoneme.
5. The electronic device as described in claim 4, wherein the text-to-phoneme module maps the source language text and the target language text to the source phoneme and the target phoneme respectively by means of a mapping method.
6. The electronic device as described in claim 4, wherein the feature-adaptive text-to-speech system further comprises: a reference feature extraction module for performing an extraction operation on the target human voice channel to generate a feature vector, wherein the feature vector indicates a timbre feature of the target human voice channel.
7. The electronic device as described in claim 6, wherein the storage device further stores a plurality of candidate speech synthesis models, and the feature-adaptive text-to-speech system further includes: a speech synthesis model selection module for selecting from the plurality of candidate speech synthesis models based on the emotion classification to generate a selected speech synthesis model.
8. The electronic device as described in claim 7, wherein the selected speech synthesis model performs the speech synthesis operation based on the feature vector, the source phoneme, and the target factor to generate the synthesized human voice; and the synthesized human voice has the timbre feature.
9. The electronic device as described in claim 8, wherein the selected speech synthesis model is further used to: generate an embedding vector based on the source phonemes and the target factors; and generate the synthesized human voice based on the embedding vector.
10. A method for speech synthesis, comprising: performing a separation operation on an audio input signal to generate a plurality of human voice channels and a single non-human voice channel; selecting a target human voice channel from the plurality of human voice channels based on a source language text; performing a prediction operation on each of the plurality of human voice channels to generate a prediction result, wherein the prediction result indicates an emotion classification of each human voice channel; performing a speech synthesis operation based on the source language text, a target language text, the target human voice channel, and the emotion classification to generate a synthesized human voice; and performing a mixing operation on the single non-human voice channel and the synthesized human voice to generate an audio output signal.