Speech synthesis methods and apparatus, electronic devices, storage media and software products

Abstract

This disclosure relates to speech synthesis methods and apparatuses, electronic devices, storage media, and program products. The method includes: acquiring source language dubbing audio corresponding to source language subtitle text of a source video, and target language subtitle text translated from the source language subtitle text; extracting audio emotion features from the source language dubbing audio using an emotion extractor, the audio emotion features representing the emotion expressed by the source language dubbing audio; converting the target language subtitle text into a phoneme sequence, and encoding the phoneme sequence using a text encoder to obtain text encoding features; fusing the audio emotion features and text encoding features to obtain emotion text features; and generating target language audio based on the emotion text features using a decoder, the target language audio being used as dubbing audio for the source video in the target language. Therefore, high-quality target language audio with the emotion of the source dubbing audio can be automatically and efficiently generated, with low cost and high efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of speech synthesis, and more particularly to a speech synthesis method and apparatus, electronic device, storage medium and program product. Background Technology

[0002] In recent years, with the continuous improvement in the quality of domestically produced films and television dramas, many popular domestic productions have also appeared on foreign television screens and overseas broadcasting platforms. To better promote the export of films and television dramas, subtitles can be translated into the target market's language and dubbed. However, currently, the internationalization of films and television dramas mainly relies on professional voice actors dubbing episode by episode, but this method requires not only professional dubbing skills and conditions, but also consumes a lot of human and financial resources. Summary of the Invention

[0003] In view of this, this disclosure proposes a speech synthesis method and apparatus, electronic device, storage medium and program product that can automatically and efficiently generate high-quality target language audio with source voice-over audio emotion, with low cost and high efficiency.

[0004] According to one aspect of this disclosure, a speech synthesis method is provided, comprising: acquiring source language dubbing audio corresponding to source language subtitle text of a source video, and target language subtitle text translated from the source language subtitle text, wherein the source language and the target language are different; extracting audio emotion features from the source language dubbing audio using an emotion extractor, wherein the audio emotion features characterize the emotion expressed by the source language dubbing audio; converting the target language subtitle text into a phoneme sequence, and encoding the phoneme sequence using a text encoder to obtain text encoding features; fusing the audio emotion features and the text encoding features to obtain emotion text features; and generating target language audio based on the emotion text features using a decoder, wherein the target language audio is used as dubbing audio for the source video in the target language.

[0005] In one possible implementation, the audio emotion feature includes the emotion feature of each frame of audio in L frames of the source language dubbing audio, where L represents the number of frames in the source language dubbing audio; wherein, fusing the audio emotion feature with the text encoding feature to obtain the emotion text feature includes: calculating the mean of the emotion features of the L frames of audio in the audio emotion feature to obtain a first emotion feature; performing a high-dimensional mapping on the first emotion feature to obtain a second emotion feature, the second emotion feature having the same dimension as the text encoding feature; and adding the second emotion feature to the text encoding feature to obtain the emotion text feature.

[0006] In one possible implementation, the method further includes: extracting semantic features from the phoneme sequence using a semantic extractor, the semantic features representing the semantics of the target language subtitle text; generating target language audio using a decoder based on the emotional text features, comprising: fusing the emotional text features with the semantic features to obtain a comprehensive feature; and generating target language audio using a decoder based on the comprehensive feature.

[0007] In one possible implementation, after generating the target language audio, the method further includes: converting the target language audio into recognized text using speech recognition technology corresponding to the target language; determining the character error rate of the recognized text based on the difference between the recognized text and the target language subtitle text, wherein the character error rate characterizes the pronunciation error rate of the target language audio; and, if the character error rate is higher than a preset threshold, selecting target reference audio from a preset source language audio library that matches the emotion expressed by the source language dubbing audio, so as to regenerate the target language audio using the audio emotion features of the target reference audio; wherein the source language audio library includes reference audio from various source languages ​​with different emotions.

[0008] In one possible implementation, the source language audio library includes a first audio library, where reference audios are labeled with emotion categories. The step of selecting target reference audios from the preset source language audio library that match the emotion expressed by the source language dubbing audio includes: performing emotion recognition on the source language dubbing audio using an emotion recognition model to obtain an emotion classification result for the source language dubbing audio, where the emotion classification result characterizes the emotion category expressed by the source language dubbing audio; and, based on the emotion categories labeled on each reference audio in the first audio library, using reference audios whose emotion categories match the emotion classification results as target reference audios.

[0009] In one possible implementation, the source language audio library includes a second audio library, wherein reference audios in the second audio library are labeled with emotional audio features extracted using the emotion extractor. The step of selecting target reference audios from the preset source language audio library that match the emotions expressed by the source language dubbing audio includes: calculating the similarity between the audio emotional features of the source language dubbing audio and the emotional audio features labeled in each reference audio in the second audio library, and selecting the reference audio with the highest similarity as the target reference audio.

[0010] In one possible implementation, the training process of the emotion extractor, text encoder, and decoder includes: in a first training phase, an initial text encoder and an initial decoder are trained using a first dataset to obtain a trained first text encoder and a first decoder, wherein the first dataset includes: multiple first target language text samples and first target language audio samples corresponding to each first target language text sample; in a second training phase, the first text encoder, the first decoder, and the initial emotion extractor are trained using a second dataset to obtain a trained second text encoder, a second decoder, and a first emotion extractor; wherein the second dataset includes: multiple source language audio samples, second target language text samples corresponding to each source language audio sample, and corresponding second target language audio samples; in a third training phase, the second text encoder, the second decoder, and the first emotion extractor are fine-tuned using a third dataset to obtain a trained text encoder, decoder, and emotion extractor, wherein the third dataset includes: multiple third target language texts and third target language audio samples corresponding to each third target language text, wherein the third target language audio samples are obtained by collecting audio of the third target language text read aloud in standard pronunciation.

[0011] In one possible implementation, training an initial text encoder and an initial decoder using a first dataset to obtain a trained first text encoder and a first decoder includes: for any first target language text sample, encoding the phoneme sequence of the first target language text sample using the initial text encoder to obtain text encoding features of the first target language text sample; using the initial decoder based on the text encoding features of the first target language text sample to generate a predicted target language audio corresponding to the first target language text sample; and adjusting the parameters of the initial text encoder and the initial decoder based on the difference between the predicted target language audio corresponding to the first target language text sample and the first target language audio sample corresponding to the first target language text sample to obtain the trained first text encoder and the first decoder.

[0012] In one possible implementation, the step of fine-tuning the second text encoder, the second decoder, and the first emotion extractor using a third dataset to obtain a trained text encoder includes: for any third target language text sample, encoding the phoneme sequence of the third target language text sample using the second text encoder to obtain the text encoding features of the third target language text sample; extracting the audio emotion features of the third target language audio sample from the third target language audio sample corresponding to the third target language text sample using the first emotion extractor; fusing the text encoding features of the third target language text sample with the audio emotion features of the corresponding third target language audio sample to obtain the emotion text features corresponding to the third target language audio sample; generating a predicted target language audio corresponding to the third target language text sample using the second decoder based on the emotion text features corresponding to the third target language audio sample; and adjusting the parameters of the second text encoder, the second decoder, and the first emotion extractor based on the difference between the predicted target language audio corresponding to the third target language text sample and the third target language audio sample to obtain the trained text encoder, emotion extractor, and decoder.

[0013] In one possible implementation, the construction process of the second dataset includes: acquiring bilingual subtitle files, source language dubbing audio data, and target language dubbing audio data corresponding to the original long video data, wherein the bilingual subtitle files include the start and end times of each bilingual subtitle text display, and the bilingual subtitle text includes target language subtitle text and source language subtitle text; based on the start and end times of each bilingual subtitle text display in the bilingual subtitle files, segmenting the source language dubbing audio data and target language dubbing audio data to obtain multiple segmented first audio groups, each first audio group including segmented source language audio segments and target language audio segments; using the speech recognition technology corresponding to the target language, converting the target language audio segments in each first audio group into recognized text, and comparing the recognized text converted from each target language audio segment with the target language subtitle text corresponding to each target language audio segment. The differences between the audio segments are used to determine the character error rate of the recognized text converted from each target language audio segment; the first audio groups containing target language audio segments with character error rates higher than a preset threshold are filtered out to obtain multiple second audio groups; by performing voice separation on the source language audio segments and target language audio segments in each second audio group, multiple third audio groups are obtained, and the source language audio segments and target language audio segments in each third audio group contain human voice tracks; the audio quality of the source language audio segments and target language audio segments in each third audio group is detected, and the third audio groups with audio quality lower than a preset quality threshold are filtered out to obtain multiple fourth audio groups, wherein the audio quality includes naturalness and / or noise value; based on the source language audio segments and target language audio segments in the multiple fourth audio groups, and the target language subtitle text corresponding to the target language audio segments in each fourth audio group, the second dataset is obtained.

[0014] According to another aspect of this disclosure, a speech synthesis apparatus is provided, comprising: an acquisition module, configured to acquire source language dubbing audio corresponding to source language subtitle text of a source video, and target language subtitle text translated from the source language subtitle text, wherein the source language and the target language are different; a feature extraction module, configured to extract audio emotion features from the source language dubbing audio using an emotion extractor, wherein the audio emotion features characterize the emotion expressed by the source language dubbing audio; a text encoding module, configured to convert the target language subtitle text into a phoneme sequence and encode the phoneme sequence using a text encoder to obtain text encoding features; a feature fusion module, configured to fuse the audio emotion features and the text encoding features to obtain emotion text features; and an audio generation module, configured to generate target language audio using a decoder based on the emotion text features, wherein the target language audio is used as dubbing audio for the source video in the target language.

[0015] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0016] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0018] According to various aspects of this disclosure, by extracting the audio emotional features of the source language dubbing audio and fusing them with the text encoding features of the target language subtitle text, emotional text features containing the emotional information of the source language dubbing audio and the text information of the target language subtitle text are obtained. Based on these emotional text features, target language audio with the emotional expression of the source language dubbing audio can be generated. In other words, high-quality target language audio with the emotion of the source language dubbing audio can be automatically and efficiently generated without the need for professional dubbing equipment and conditions, or the expenditure of a large amount of human and financial resources. This makes the generation cost of dubbing audio from the source language to the target language lower and the efficiency higher.

[0019] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0021] Figure 1 A flowchart of a speech synthesis method according to an embodiment of the present disclosure is shown.

[0022] Figure 2 A flowchart is shown for another speech synthesis method according to an embodiment of the present disclosure.

[0023] Figure 3 A schematic diagram of a Chinese-Thai speech synthesis process according to an embodiment of the present disclosure is shown.

[0024] Figure 4 A flowchart is shown for another speech synthesis method according to an embodiment of the present disclosure.

[0025] Figure 5A flowchart of a three-stage training method according to an embodiment of the present disclosure is shown.

[0026] Figure 6 A block diagram of a speech synthesis apparatus according to an embodiment of the present disclosure is shown.

[0027] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0029] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0030] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0031] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0032] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0033] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0034] As mentioned above, the current overseas dubbing of films and television dramas mainly relies on professional voice actors dubbing episode by episode. However, this method requires not only professional dubbing skills and conditions but also consumes a lot of human and financial resources. With the development of artificial intelligence (AI) technology, it is possible to achieve seamless cross-language voice conversion for actors in film and television works, allowing audiences of different languages ​​to hear "authentic" dialogue, that is, to match the tone, intonation, and emotions of the characters in the original dubbing. This paper mainly studies the implementation and deployment of film and television drama-level speech synthesis projects, so that the synthesized speech can reach a level that meets the recognition of local people in terms of tone, intonation, and emotion. Text-to-Speech (TTS), as one of the most important technologies of artificial intelligence, has been successfully applied to various fields, including human-computer interaction, intelligent customer service, and virtual assistants. However, speech synthesis technology is rarely applied to AI dubbing of film and television dramas at present. The main reason is that current speech synthesis technology cannot achieve the subtle emotional synthesis of speech, let alone the "authentic" emotional expression of film and television dramas.

[0035] Therefore, in order to achieve professional-grade AI dubbing effects for film and television dramas, this disclosure proposes a speech synthesis method that can seamlessly convert domestic Chinese film and television dramas into overseas works, enabling seamless cross-language speech and emotion transfer for actors, allowing audiences of different languages ​​to hear the "authentic" dialogue. Combining the characteristics of film and television dramas, the speech synthesis method proposed in this disclosure is equivalent to a cross-language source-target language joint emotion transfer speech synthesis method. It extracts subtle emotional expressions from the source language dubbing audio (such as Chinese dubbing audio) using an emotion extractor, transferring these emotions to the target language speech synthesis to achieve synthesized audio with rich emotions.

[0036] The speech synthesis method of this disclosure can be deployed on various terminal devices through software or hardware modifications. The terminal devices involved in this disclosure can refer to devices with wireless and / or wired connection functions. Wireless connection means that they can connect to other devices via wireless methods such as Wi-Fi and Bluetooth. The terminal devices involved in this disclosure can also communicate with other devices via wired connection functions. The terminal devices involved in this disclosure can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking or swiping on the display screen using fingers or styluses. Non-touchscreen devices can connect to input devices such as mice, keyboards, and touch panels to control the terminal device. Screenless devices can be, for example, screenless Bluetooth speakers. For example, the terminal devices in this application can include, but are not limited to, user equipment (UE), mobile devices, user terminals, terminals, handheld devices, tablet computers, laptops, PDAs, and computing devices.

[0037] The speech synthesis method of this disclosure can also be deployed on a server, which can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine or container, with wireless communication capabilities. These wireless communication capabilities can be configured in the server's chip (system) or other components. This can refer to a device with wireless connectivity, meaning it can connect to other servers or terminal devices via wireless connections such as Wi-Fi or Bluetooth. The server involved in this disclosure can also have wired communication capabilities. For example, the server in this disclosure can be located in the cloud, communicating with terminal devices, receiving source language subtitle text and target language subtitle text translated from the source language subtitle text sent by the terminal devices, and using the speech synthesis method deployed on the server to generate target language audio based on the target language subtitle text and the source language subtitle audio, and returning it to the terminal devices for displaying the generated target language audio to the user on the terminal devices.

[0038] Figure 1 A flowchart illustrating a speech synthesis method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S11 to S15.

[0039] In step S11, the source language dubbing audio corresponding to the source language subtitle text of the source video, and the target language subtitle text translated from the source language subtitle text are obtained, where the source language and the target language are different.

[0040] The source video can be any video for which dubbing audio in the target language is to be generated. The source video can be, for example, a movie, TV series, a short drama, a variety show, etc. The source video can be a complete video or a segment of a video. This embodiment of the disclosure does not limit this.

[0041] In this embodiment, the source language subtitle text and the target language subtitle text are texts in different languages. For example, the source language subtitle text may be Chinese, and the target language subtitle text may be Thai, English, etc.; of course, the source language subtitle text can also be Thai or English, etc., and the target language subtitle text may be Chinese. It should be understood that those skilled in the art can translate the source language subtitle text into any target language subtitle text according to actual needs, and this embodiment does not limit this.

[0042] Considering that source videos typically correspond to multiple source language subtitle texts, and these source language subtitle texts correspond to multiple target language subtitle texts, to improve the efficiency of subsequent target language audio generation, corresponding target language audio can be generated separately for each target language subtitle text. Therefore, in one possible implementation, a subtitle file corresponding to the source video can be obtained. This subtitle file includes multiple source language subtitle texts corresponding to the source video and a subtitle timeline. This timeline represents the subtitle display time of each source language subtitle text, including the start and end times of the subtitle display. The subtitle timeline of the source language subtitle text is also the subtitle timeline that the target language subtitle text should use. Then, according to the subtitle display time of each source language subtitle text indicated by the timeline, audio segments corresponding to each source language subtitle text can be segmented from the complete dubbing audio file corresponding to the source video as the source language dubbing audio for each source language subtitle text. In this way, based on the source language dubbing audio corresponding to each source language subtitle text and the target language subtitle text translated from each source language subtitle text, the target language audio corresponding to each target language subtitle text can be generated separately. Here, "one" subtitle text can refer to the subtitle text between a start time and a corresponding end time in the subtitle timeline, that is, the subtitle text that is displayed on the screen at the same time.

[0043] In step S12, an emotion extractor is used to extract audio emotion features from the source language dubbing audio. The audio emotion features represent the emotions expressed by the source language dubbing audio.

[0044] As mentioned above, a source video can correspond to multiple source language subtitle texts, and each source language subtitle text can be segmented into its corresponding source language dubbing audio. That is, multiple source language subtitle texts can correspond to multiple source language dubbing audios. Therefore, an emotion extractor can be used to extract the audio emotion features in the source language dubbing audio corresponding to each source language subtitle text.

[0045] In practical applications, those skilled in the art can use artificial intelligence models disclosed in the art for extracting audio emotion features. For example, a Speech Emotion Recognition (SER) model can be used to extract audio emotion features. Specifically, the implicit emotion encoding vector output by the intermediate layer of the SER model can be used as the audio emotion feature extracted from the source language dubbing audio. Of course, a self-developed model can also be used, as long as it can achieve the function that the emotion extractor can achieve. This disclosure does not limit this aspect.

[0046] There are two encoding methods for the emotion extractor: Wav-level (audio wave dimension) encoding and Frame-level (audio frame dimension) encoding. The Wav-level encoding method outputs a one-dimensional feature vector, such as a (1, 1024) feature vector, which extracts the overall emotional features of the entire source language dubbing audio. The Frame-level encoding method outputs a two-dimensional feature vector, such as a (L, 1024) feature vector, where the first dimension "L" represents the frame dimension, which also represents the number of frames in the source language dubbing audio. This method extracts the emotional features of each frame in the L frames of the source language dubbing audio. In other words, the audio emotional features can include the overall emotional features of the source language dubbing audio, or the emotional features of each frame in the L frames of the source language dubbing audio.

[0047] It should be understood that when using the Wav-level encoding method to extract audio emotion features, the slight noise in the low-dimensional space may significantly affect the accuracy of the extracted one-dimensional audio emotion features. In contrast, the high-dimensional audio emotion features extracted using the frame-level encoding method can smooth out the noise in the high-dimensional space and improve the accuracy of emotion feature extraction.

[0048] In step S13, the target language subtitle text is converted into a phoneme sequence, and the phoneme sequence is encoded using a text encoder to obtain text encoding features.

[0049] As mentioned above, the source video can correspond to multiple source language subtitle texts, and the multiple source language subtitle texts correspond to multiple target language subtitle texts. Therefore, each target language subtitle text can be converted into a phoneme sequence, and a text encoder can be used to encode the phoneme sequence converted from each target language subtitle text to obtain the text encoding features of each target language subtitle text.

[0050] In this embodiment, the target language subtitle text can be converted into a phoneme sequence based on the phoneme dictionary corresponding to the target language. This disclosure does not limit the conversion method from text to phonemes.

[0051] Those skilled in the art can use text encoders disclosed in the field of speech synthesis, such as the text encoder in the VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) model, to encode the phoneme sequence converted from the target language subtitle text and obtain text encoding features. Of course, self-developed models can also be used, as long as they can achieve the functions that the text encoder can achieve. This disclosure does not limit this aspect.

[0052] In step S14, the audio emotion features are fused with the text encoding features to obtain the emotion text features.

[0053] As described above, the source video can correspond to multiple source language subtitle texts, and these multiple source language subtitle texts correspond to multiple target language subtitle texts. In step S12, an emotion extractor can be used to extract the audio emotion features from the source language dubbing audio corresponding to each source language subtitle text. In step S13, a text encoder can be used to encode the phoneme sequence of the target language subtitle text translated from each source language subtitle text, obtaining the text encoding features of each target language subtitle text. Therefore, in step S14, for each target language subtitle text, the text encoding features of each target language subtitle text can be fused with the audio emotion features from the source language dubbing audio of the corresponding source language subtitle text to obtain the emotion text features corresponding to each target language subtitle text. This is equivalent to embedding the emotion features of the source language dubbing audio of each source language subtitle text into the text encoding features of the corresponding target language subtitle text, thereby obtaining the emotion text features with emotion information corresponding to each target language subtitle text.

[0054] As mentioned above, audio emotion features can be one-dimensional feature vectors. Considering that text encoding features are usually high-dimensional, one-dimensional audio emotion features can be mapped to the feature dimension of text encoding features. Then, the high-dimensional emotion features mapped to the feature dimension of text encoding features are added to the text encoding features to obtain the emotion text features.

[0055] As mentioned above, audio emotion features can also be two-dimensional feature vectors. That is, audio emotion features can include the emotion features of each frame of audio in the L frames of the source language dubbing audio. Therefore, fusing audio emotion features with text encoding features yields emotional text features, which can include:

[0056] The first emotion feature is obtained by calculating the mean of the emotion features of L frames of audio in the audio emotion feature.

[0057] The first emotion feature is mapped in a high dimension to obtain the second emotion feature. The dimension of the second emotion feature is the same as the dimension of the text encoding feature.

[0058] The second emotion feature is added to the text encoding feature to obtain the emotion text feature.

[0059] It should be understood that when using Wav-level encoding to extract audio emotion features, the slight noise in the low-dimensional space may reduce the accuracy of the extracted one-dimensional audio emotion features. In contrast, the two-dimensional audio emotion features derived using frame-level encoding can smooth out the noise in the two-dimensional space, thus improving the accuracy of emotion feature extraction. Therefore, when audio emotion features are represented as two-dimensional feature vectors, the mean of the audio emotion features across the frame dimension can be fused with the text feature vector. This method can significantly improve the accuracy and stability of emotion feature embedding.

[0060] In practical applications, high-dimensional feature mapping and feature fusion can be achieved by adding an emotion mapping module and a fusion module to the aforementioned text encoder. Specifically, the emotion mapping module maps the low-dimensional emotion vector represented by the first emotion feature to a high-dimensional space to align with the vector dimension of the text encoding features extracted by the text encoder. For example, the emotion mapping module can use a fully connected neural network (FCL) for high-dimensional mapping, so that the mapped high-dimensional emotion vector can be better fused with the text encoding features while preserving the expressive power of the emotion features. The fusion module can be used to add the high-dimensional second emotion feature to the text encoding features. For example, broadcasting technology can be used to add information from different features: Fused emb =Phoneme emb +Emotion emb Fusedemb Representing emotional text features, Phoneme emb Emotion represents text encoding features. emb Representing the second emotion feature, the superposition of feature vectors allows emotional information to directly influence the speech synthesis process, thereby generating speech with a specific emotion. In this way, the emotional features of the source language dubbing audio extracted by the emotion extractor can be embedded into the text encoder of a speech synthesis model (such as VITS), constructing an emotion-embedded speech synthesis method with emotion transfer capabilities. This not only generates target language audio with specific emotions but also achieves cross-language emotion transfer, that is, transferring the emotion of the source language audio to the target language audio.

[0061] In step S15, the decoder generates target language audio based on emotional text features. The target language audio is used as dubbing audio for the source video in the target language.

[0062] In this context, generating target language audio using a decoder based on emotional text features is equivalent to synthesizing emotionally charged audio using text features containing emotional information. It should be understood that those skilled in the art can use decoders disclosed in the field of speech synthesis, such as the decoder in the VITS model, to decode emotional text features and generate target language audio. Alternatively, they can develop their own decoders, as long as they can achieve the functions that a decoder can perform. This disclosure does not impose any limitations on this aspect.

[0063] As mentioned above, the source video can correspond to multiple source language subtitle texts, and the multiple source language subtitle texts correspond to multiple target language subtitle texts. Steps S12 to S14 can be used to obtain the emotional text features corresponding to each target language subtitle text. Therefore, in step S15, the decoder can be used to generate the target language audio for each target language subtitle text based on the emotional text features corresponding to each target language subtitle text, and each target language audio carries the emotional expression from the corresponding source language dubbing audio.

[0064] In practical applications, after obtaining the target language audio for each target language subtitle text, the target language audio of each target language subtitle text can be spliced ​​and processed with audio-visual synchronization according to the subtitle timeline of the source language subtitle text (which is also the subtitle timeline required for the target language subtitle text) to obtain the complete dubbing audio of the source video in the target language, and the complete dubbing audio in the target language is synchronized with the video footage of the source video.

[0065] According to the speech synthesis method of this disclosure, the audio emotion features of the source language dubbing audio are extracted and fused with the text encoding features of the target language subtitle text to obtain emotion text features containing emotion information of the source language dubbing audio and text information of the target language subtitle text. Based on these emotion text features, target language audio with the emotion expression of the source language dubbing audio can be generated. In other words, high-quality target language audio with the emotion of the source language dubbing audio can be automatically and efficiently generated without the need for professional dubbing equipment and conditions, or the expenditure of a large amount of manpower and financial resources. This makes the generation cost of dubbing audio from the source language to the target language lower and the efficiency higher.

[0066] Figure 2 A flowchart of another speech synthesis method according to an embodiment of the present disclosure is shown, such as Figure 2 As shown, the method includes:

[0067] Step S21: Obtain the source language dubbing audio corresponding to the source language subtitle text of the source video, and the target language subtitle text translated from the source language subtitle text. The source language and the target language are different.

[0068] Step S22: Use an emotion extractor to extract audio emotion features from the source language dubbing audio. The audio emotion features represent the emotions expressed in the source language dubbing audio.

[0069] Step S23: Convert the target language subtitle text into a phoneme sequence, and use a text encoder to encode the phoneme sequence to obtain text encoding features;

[0070] Step S24: Fuse the audio emotion features with the text encoding features to obtain the emotion text features;

[0071] Step S25: Use a semantic extractor to extract semantic features from the phoneme sequence, wherein the semantic features characterize the semantics of the target language subtitle text;

[0072] Step S26: Fuse the emotional text features with semantic features to obtain comprehensive features;

[0073] Step S27: Generate target language audio using the decoder based on comprehensive features.

[0074] The implementation methods of steps S21 to S24 can refer to the implementation methods of steps S11 to S14 in the above embodiments of this disclosure, and will not be repeated here.

[0075] In step S25, those skilled in the art can employ artificial intelligence models capable of extracting text semantics. For example, BERT (Bidirectional Encoder Representations from Transformers) can be used as a semantic extractor to extract semantic features from the phoneme sequence. Alternatively, a self-developed semantic feature extraction model can be used; this disclosure does not limit the scope of the embodiments. As described above, the source video can correspond to multiple source language subtitle texts, and these multiple source language subtitle texts correspond to multiple target language subtitle texts. Therefore, a semantic extractor can be used to extract the semantic features of each target language subtitle text from its phoneme sequence.

[0076] Steps S26 and S27 can be implemented as a way of implementing step S15. In step S26, if the dimension of the semantic feature is the same as the dimension of the emotional text feature, the emotional text feature and the semantic feature can be directly added together to obtain a comprehensive feature; if the dimension of the semantic feature is different from the dimension of the emotional text feature, the semantic feature can be mapped to a semantic feature with the same dimension as the emotional text feature, and then the semantic feature mapped to the same dimension can be added together with the emotional text feature to obtain a comprehensive feature; this comprehensive feature is equivalent to integrating emotional information, semantic information and text information. In this way, the comprehensive feature can make the generated target language audio have a high level in terms of tone, pitch and emotion, that is, it can make the character's speech in the target language audio more in line with the language expression of the target language while possessing the emotional expression of the source language dubbing audio, making the character's speech in the target language audio more natural and realistic.

[0077] As mentioned above, the source video can correspond to multiple source language subtitle texts, and the multiple source language subtitle texts correspond to multiple target language subtitle texts. The emotional text features of each target language subtitle text can be obtained by using steps S22 to S24, and the semantic features of each target language subtitle text can be obtained by using step S25. Therefore, in step S26, the emotional text features and the corresponding semantic features of each target language subtitle text can be fused to obtain the comprehensive features of each target language subtitle text.

[0078] In step S27, the decoder can, for example, use the decoder in the VITS model to generate target language audio based on comprehensive features. Alternatively, a custom decoder can be developed, as long as it can achieve the functionality required by the decoder; this embodiment does not impose any limitations on this. As described above, the source video can correspond to multiple source language subtitle texts, and these multiple source language subtitle texts correspond to multiple target language subtitle texts. After obtaining the comprehensive features corresponding to each target language subtitle text using steps S22 to S26, in step S27, the decoder can be used to generate target language audio for each target language subtitle text based on these comprehensive features. This target language audio carries the emotional expression from the source language dubbing audio and matches the tone, intonation, and other linguistic expressions of the target language.

[0079] In practical applications, after obtaining the target language audio of each target language subtitle text in multiple target language subtitle texts through steps S21 to S27, the target language audio of each target language subtitle text can be spliced ​​and processed according to the subtitle timeline of the source language subtitle text to obtain the complete dubbing audio of the source video in the target language, and the complete dubbing audio in the target language is synchronized with the video screen of the source video.

[0080] Based on the above steps S21 to S27, Figure 3 A schematic diagram of a Chinese-Thai speech synthesis process is shown, such as... Figure 3 As shown, the process includes: using a Thai phoneme dictionary (which can be manually corrected to obtain accurate phoneme sequences in Thai), converting Thai text (i.e., subtitle text in Thai) into phoneme sequences, inputting the phoneme sequences into a text encoder to extract text encoding features, and simultaneously inputting Chinese audio into an emotion recognition model (i.e., an emotion extractor) to extract audio emotion features. An emotion mapping module and a fusion module are set in the text encoder to fuse the text encoding features with the audio emotion features, resulting in text & emotion features (i.e., emotion text features). Then, the fusion module fuses the semantic features extracted by the semantic extractor (not shown in the figure) with the text & emotion features to obtain text & emotion & semantic features (i.e., comprehensive features). Finally, the text & emotion & semantic features are input into the decoder to obtain Thai audio.

[0081] According to the speech synthesis method of this disclosure, by extracting the audio emotion features of the source language dubbing audio and fusing them with the text encoding features of the target language subtitle text, an emotional text feature containing the emotional information of the source language dubbing audio and the text information of the target language subtitle text is obtained. Then, by extracting the semantic features of the target language subtitle text and fusing the semantic features with the emotional text feature, a comprehensive feature containing emotional information, text information, and semantic information can be obtained. Based on this comprehensive feature, a target language audio with the emotional expression of the source language dubbing audio and the tone, intonation, and other language expression methods of the target language can be generated. In other words, the speech of the characters in the target language audio can be more in line with the language expression methods of the target language while possessing the speaking emotions of the source language dubbing audio, making the speech of the characters in the target language audio more natural and realistic. It does not require professional dubbing equipment and dubbing conditions, nor does it require a lot of manpower and financial resources, making the generation cost of dubbing audio from the source language to the target language lower, more efficient, and of higher quality.

[0082] Considering that the source language dubbing audio is needed as an emotional reference during audio production, but the quality of the source language dubbing audio (such as the original dubbing audio corresponding to Chinese videos) cannot be controlled in actual production, and the source language dubbing audio may even contain extreme noise, resulting in uncontrollable quality of the produced target language audio, in order to ensure the quality of the produced target language audio, in one possible implementation, after generating the target language audio based on the source language dubbing audio, that is, after steps S11 to S15, or steps S21 to S27, as follows... Figure 4 As shown, speech synthesis methods may also include:

[0083] Step S41: Using speech recognition technology corresponding to the target language, convert the target language audio into recognized text;

[0084] Step S42: Based on the difference between the identified text and the target language subtitle text, determine the character error rate of the identified text, wherein the character error rate characterizes the pronunciation error rate of the target language audio;

[0085] Step S43: If the character error rate is higher than a preset threshold, select target reference audio that matches the emotion expressed by the source language dubbing audio from a preset source language audio library, so as to regenerate the target language audio using the audio emotion features of the target reference audio; wherein, the source language audio library includes reference audio of the source language with multiple emotions.

[0086] In step S41, if the target language is Thai, Thai audio can be converted into recognized text using Automatic Speech Recognition (ASR) technology, i.e., audio-to-text speech recognition is performed on the target language audio to obtain the recognized text. Of course, if the target language is English, English audio can be converted into recognized text using Automatic Speech Recognition (ASR) technology. Furthermore, in step S42, the recognized text can be compared and analyzed with the original target language subtitle text to calculate the character error rate (CER, a commonly used metric for speech recognition accuracy) of the recognized text compared to the target language subtitle text. The character error rate is used to evaluate the pronunciation error rate (i.e., pronunciation accuracy) of the target language audio, i.e., to evaluate the difference between the converted recognized text and the original target language subtitle text.

[0087] It should be understood that if the character error rate of the recognized text is lower than the preset threshold, it can be considered that the pronunciation error rate of the target language is low, or that the pronunciation accuracy is high. In this case, there is no need to select the target reference audio from the source language audio library to regenerate the target language audio. If the character error rate of the recognized text is higher than the preset threshold, it can be considered that the pronunciation error rate of the target language audio is too high, or that the pronunciation accuracy is low. In this way, the target language audio with pronunciation problems can be identified efficiently and accurately. If the character error rate of the recognized text is higher than the preset threshold, it can be considered that the source language dubbing audio used to generate the target language audio with pronunciation problems is of poor quality.

[0088] Furthermore, after identifying the target language audio with pronunciation problems, a high-quality target reference audio with matching emotion can be searched from a preset source language audio library to replace the corresponding source language dubbing audio. The target language audio is then regenerated using the audio emotion features of the target reference audio. In practical applications, the audio emotion features of each reference audio in the source language audio can be extracted and stored in advance using an emotion extractor. This way, after searching for any target reference audio from the source language audio library, the audio emotion features of the target reference audio can be directly used to regenerate the target language audio, which helps to ensure that the emotional expression of the produced target language audio is consistent and the pronunciation is stable.

[0089] In one possible implementation, the source language audio library may include a first audio library, in which reference audios are labeled with emotion categories. This first audio library can be called a discrete audio library. It can be composed of high-quality source language reference audios containing discrete emotions, collected manually. For example, the first audio library may include reference audios for a total of nine emotions (such as calm, surprise, joy, anger, sadness, shouting, fear, excitement, and frustration), with each emotion corresponding to at least one reference audio. Each reference audio can be labeled with a corresponding emotion category. In this embodiment, the content expressed by each reference audio in the first audio library is not limited; only the emotion expressed by the reference audio is required. That is, as long as the emotion expressed by each reference audio matches its labeled emotion category, it is acceptable. Based on this first audio library, step S43 above, which involves selecting target reference audios from the preset source language audio library that match the emotions expressed by the source language dubbing audio, may include:

[0090] An emotion recognition model is used to perform emotion recognition on the source language dubbed audio, and the emotion classification result of the source language dubbed audio is obtained. The emotion classification result represents the emotion category expressed by the source language dubbed audio.

[0091] Based on the emotion categories labeled in each reference audio in the first audio library, the reference audio whose emotion category matches the emotion classification result is used as the target reference audio.

[0092] The emotion recognition model can be any model disclosed in the art for recognizing audio emotions, such as the SER model, and this embodiment of the present disclosure does not limit this. After the emotion recognition model identifies the emotion classification result of the source language dubbing audio, reference audio with the same emotion category as the emotion classification result can be searched from the first audio library as the target reference audio to regenerate the target language audio.

[0093] In one possible implementation, the source language audio library may include a second audio library. The reference audio in the second audio library is labeled with audio emotion features extracted using an emotion extractor. This second audio library can be called an emotion vector retrieval library. It can also be a collection of high-quality source language reference audio collected manually. Unlike a second audio library that only contains nine emotions, this second audio library can contain reference audio with more emotions. In this embodiment, the content expressed by each reference audio in the second audio library is not limited; only the emotion expressed by the reference audio is required. Users can collect reference audio with more emotions and extract the emotion features of each reference audio to label the emotion expressed by the reference audio. Thus, the emotion extractor can be used to pre-extract the audio emotion features of each reference audio in the second audio library to characterize the emotion category of each reference audio. Based on this, in step S43 above, selecting target reference audio that matches the emotion expressed by the source language dubbing audio from the preset source language audio library may include:

[0094] The similarity between the audio emotional features of the source language dubbing audio and the emotional audio features annotated in each reference audio in the second audio library is calculated, and the reference audio with the highest similarity is taken as the target reference audio. This method can be understood as obtaining the audio emotional features of the source language dubbing audio with pronunciation problems, and then obtaining the reference audio in the second audio library that is most similar to the audio emotional features of the source language dubbing audio by calculating the similarity between the audio emotional features of the source language dubbing audio and the emotional audio features annotated in each reference audio in the second audio library. This embodiment of the present disclosure does not limit the method of calculating the similarity between two features; for example, the distance between two features or cosine similarity can be calculated, and this embodiment of the present disclosure does not impose any limitations on this.

[0095] It should be understood that if adopted Figure 1 The illustrated speech synthesis method, in the process of regenerating target language audio using the audio emotion features of the target reference audio, may include: fusing the audio emotion features of the target reference audio with the text encoding features of the target language subtitle text to obtain emotion text features, and then using a decoder to generate target language audio based on these emotion text features. If using... Figure 2 The speech synthesis method shown can be used to regenerate target language audio using the audio emotion features of the target reference audio. This process can include: fusing the audio emotion features of the target reference audio with the text encoding features of the target language subtitle text to obtain emotion text features; fusing the emotion text features with the semantic features of the target language subtitle text to obtain comprehensive features; and then using a decoder to generate target language audio based on the comprehensive features.

[0096] As described above, the source video can correspond to multiple source language subtitle texts, and the multiple source language subtitle texts correspond to multiple target language subtitle texts. Then, by using the above steps S11 to S15, or steps S21 to S27, the target language audio of each target language subtitle text can be obtained. Therefore, for the target language audio of each target language subtitle text, the above steps S41 to S43 can be executed respectively. When the pronunciation error rate of the target language audio of any target language subtitle text is high (that is, the pronunciation has problems), the target language audio of the target language subtitle text can be regenerated based on the reference audio in the source language audio library to obtain high-quality target language audio for each target language subtitle text.

[0097] In practical applications, in utilizing Figure 1 Combination Figure 4 speech synthesis methods, or Figure 2 Combination Figure 4 The speech synthesis method obtains the target language audio of each target language subtitle text from multiple target language subtitle texts. Then, according to the subtitle timeline of the source language subtitle text, the target language audio of each target language subtitle text can be spliced ​​and processed with audio-visual synchronization to obtain the complete dubbing audio of the source video in the target language. The complete dubbing audio is synchronized with the video screen of the source video.

[0098] In this embodiment of the disclosure, by utilizing a first audio library and a second audio library, high-quality target reference audio can be retrieved when the quality of the source language dubbing audio is low, thereby producing new high-quality target language audio and improving the quality of dubbing audio generated from the source video in the target language.

[0099] In view of the emotion extractor, text encoder, and decoder used in the above-mentioned speech synthesis methods, embodiments of this disclosure also provide... Figure 5 A three-stage training method is shown, which can improve the stability of the entire emotion transfer speech synthesis model (i.e., emotion extractor, text encoder, and decoder), such as... Figure 5 As shown, the training method includes:

[0100] Step S51: In the first training phase, the initial text encoder and initial decoder are trained using the first dataset to obtain the trained first text encoder and first decoder. The first dataset includes: multiple first target language text samples and first target language audio samples corresponding to each first target language text sample.

[0101] Step S52: In the second training phase, the first text encoder, the first decoder, and the initial emotion extractor are trained using the second dataset to obtain the trained second text encoder, the second decoder, and the first emotion extractor; wherein, the second dataset includes: multiple source language audio samples, a second target language text sample corresponding to each source language audio sample, and a corresponding second target language audio sample.

[0102] Step S53: In the third training phase, the second text encoder, the second decoder, and the first emotion extractor are fine-tuned using the third dataset to obtain the trained text encoder, decoder, and emotion extractor. The third dataset includes: multiple third target language texts and third target language audio samples corresponding to each third target language text. The third target language audio samples are obtained by collecting audio of the third target language text read aloud in standard pronunciation.

[0103] In step S51, the first training stage is equivalent to training the pronunciation model of the entire emotion transfer speech synthesis model. This can be achieved by collecting a large amount of public corpus in the target language and then performing audio preprocessing and audio quality detection on the public corpus in the target language to obtain the first dataset. In other words, the public corpus in the target language can be processed into a high-quality first dataset. For example, when the target language is Thai, the first dataset may include a total of 35 speakers and 52 hours of Thai training samples (i.e., Thai text samples and corresponding Thai audio samples). The audio preprocessing process may include, for example, obtaining the audio sample corresponding to each text sample based on the start and end times of each text sample in the public corpus. Simultaneously, for each text sample and audio sample, ASR technology is used to convert the audio sample into corresponding recognized text. Then, the recognized text is compared and analyzed with the text sample, and the accuracy of the audio sample's pronunciation is evaluated by calculating the character error rate (CER) of the recognized text. Audio samples with CER results higher than a preset threshold are filtered out based on the CER results of each audio sample. This CER filtering process can remove audio samples with pronunciation problems. Next, background noise, ambient noise, and other sound effects in the audio track of the filtered audio sample are separated. For example, known human voice separation techniques can be used, such as the Demucs model, to accurately extract the human voice track from the mixed audio track of the audio sample, eliminating interference from background music or ambient noise, ensuring the purity and usability of the human voice in the generated audio sample. This human voice separation process ensures that the audio sample mainly contains the human voice track. The Demucs model employs the U-Net convolutional architecture, which incorporates a bidirectional long short-term memory (BiLSTM) network between the encoder and decoder, enabling effective separation of vocals and accompaniment.

[0104] The audio samples obtained through the above audio preprocessing process may still contain noise and low naturalness of speech, which may affect the training of the model. Therefore, an audio quality detection process can be designed to further screen audio samples from the perspective of audio quality. The audio quality detection process may include: detecting the naturalness and noise value of each audio sample, and removing audio samples whose naturalness and noise values ​​are lower than preset scores. For example, if the maximum score for naturalness and noise value is 5, the preset score can be 3. When any audio sample has a naturalness or noise value that is lower than the preset score, it can be considered that the audio sample is of unqualified quality. Moreover, these audio samples are difficult to improve in terms of audio quality through speech enhancement or other methods. Therefore, these audio samples with unqualified audio quality can be removed to ensure that only high-quality audio samples with scores greater than the preset score are included in the first dataset.

[0105] In the first training phase, the emotion extractor in the emotion transfer speech synthesis model is essentially hidden, and the text encoder and decoder are trained first. This means that emotion learning can be ignored in the first training phase, with the main focus on training the model to learn the pronunciation and prosody of the target language. Therefore, cross-emotion transfer is not considered initially; that is, the emotion extractor is not working, and no parameter adjustments are made. The input and output are both in the target language. The goal is to first train the pronunciation model (i.e., to train the first text encoder and first decoder that can convert target language text into target language audio). Thus, in step S51, the initial text encoder and initial decoder are trained using the first dataset to obtain the trained first text encoder and first decoder, which may include:

[0106] Step S511: For any first target language text sample, the phoneme sequence of the first target language text sample is encoded using the initial text encoder to obtain the text encoding features of the first target language text sample.

[0107] Step S512: Using the initial decoder based on the text encoding features of the first target language text sample, generate the predicted target language audio corresponding to the first target language text sample;

[0108] Step S513: Based on the difference between the predicted target language audio corresponding to the first target language text sample and the first target language audio sample corresponding to the first target language text sample, adjust the parameters of the initial text encoder and the initial decoder to obtain the trained first text encoder and the first decoder.

[0109] In step S511, the phoneme sequence of the first target language text sample can be obtained by phoneme conversion of the first target language text sample based on the target language phoneme dictionary. This embodiment of the present disclosure does not limit the type and structure of the initial text encoder and the initial decoder.

[0110] In step S513, based on the difference between the predicted target language audio corresponding to the first target language text sample and the first target language audio sample corresponding to the first target language text sample, the parameters of the initial text encoder and the initial decoder are adjusted. This can be understood as calculating the loss based on the difference between the predicted target language audio corresponding to the first target language text sample and the first target language audio sample corresponding to the first target language text sample. For example, the L1 loss, L2 loss, cross-entropy loss, etc., between the Mel spectrum of the predicted target language audio and the Mel spectrum of the first target language audio sample can be calculated. This embodiment of the present disclosure does not limit this. Then, the parameters of the initial text encoder and the initial decoder can be adjusted using backpropagation, gradient descent, etc., to obtain the trained first text encoder and the first decoder. It should be understood that in the first training stage, the initial text encoder and the initial decoder can be trained iteratively multiple times. Therefore, training batches can be divided based on the first dataset to utilize multiple batches of training samples to iteratively train the initial text encoder and the initial decoder. This embodiment of the present disclosure does not limit this.

[0111] In step S52, the second training phase is equivalent to training the emotional model of the entire emotion transfer speech synthesis model. The model input is target language text and source language audio, and the output is target language audio with the emotion of the source language audio. The training results of the first training phase are used as the training objects. For example, the second dataset can be obtained by collecting film and television resources from the source language to the target language (e.g., Chinese to Thai film and television resources), and then performing audio preprocessing and audio quality detection on the film and television resources. That is, the film and television resources from the source language to the target language can be processed into a high-quality second dataset. For example, when the target language is Thai, the second dataset can include 9 speakers, a total of 7 hours of Chinese corpus (i.e., Chinese subtitles and Chinese dubbing audio) and 7 hours of Thai corpus (i.e., Thai subtitles and Thai dubbing audio). Therefore, in one possible implementation, the construction process of the above-mentioned second dataset can include:

[0112] Step S61: Obtain the bilingual subtitle file, source language dubbing audio data, and target language dubbing audio data corresponding to the original long video data. The bilingual subtitle file includes the start and end times of each bilingual subtitle text display. The bilingual subtitle text includes the target language subtitle text and the source language subtitle text.

[0113] Step S61: Based on the start and end times of the bilingual subtitle text display in the bilingual subtitle file, the source language dubbing audio data and the target language dubbing audio data are segmented to obtain multiple segmented first audio groups. Each first audio group includes the segmented source language audio segment and the target language audio segment.

[0114] Step S62: Using the speech recognition technology corresponding to the target language, the target language audio segments in each first audio group are converted into recognized text, and the character error rate of the recognized text converted from each target language audio segment is determined based on the difference between the recognized text converted from each target language audio segment and the target language subtitle text corresponding to each target language audio segment.

[0115] Step S63: Filter out the first audio groups containing target language audio segments with character error rates higher than a preset threshold from multiple first audio groups to obtain multiple second audio groups;

[0116] Step S64: By performing voice separation on the source language audio segment and the target language audio segment in each second audio group, multiple third audio groups are obtained, and the source language audio segment and the target language audio segment in each third audio group contain a voice track.

[0117] Step S65: Detect the audio quality of the source language audio segment and the target language audio segment in each third audio group, and filter out multiple third audio groups whose audio quality is lower than a preset quality threshold to obtain multiple fourth audio groups, wherein the audio quality includes naturalness and / or noise value;

[0118] Step S66: Based on the source language audio segments and target language audio segments in the plurality of fourth audio groups, and the target language subtitle text corresponding to the target language audio segments in each fourth audio group, the second dataset is obtained.

[0119] The original long video data can be, for example, video resources such as movies, TV series, short dramas, and variety shows with bilingual subtitles and dubbing, and this disclosure does not limit this. It should be understood that the bilingual subtitle file corresponding to the original long video data may contain a subtitle timeline, which can indicate the start time (start_time) and end time (end_time) of each bilingual subtitle text display; or, if the original long video data does not have a bilingual subtitle file, subtitle translation can be performed based on the source language subtitles, and based on the subtitle translation file, the start_time and end_time of each subtitle in the source language subtitles and the target language subtitles can be aligned to ensure that the subtitles and audio from the source language to the target language are aligned. Since the target language dubbing audio may come from professional voice actors, who may not perfectly match the target language subtitle text, inconsistencies may arise between the target language dubbing audio and the target language subtitle text. Therefore, after segmenting the audio based on the start and end times of the bilingual subtitle text display to obtain the segmented source language audio segments and target language audio segments, speech recognition technology corresponding to the target language (such as ASR for Thai) can be used to convert the target language audio segments in each first audio group into recognized text. Based on the difference between the recognized text converted from each target language audio segment and the corresponding target language subtitle text, the character error rate of the recognized text converted from each target language audio segment is determined. This allows for the filtering out of first audio groups containing target language audio segments with character error rates exceeding a preset threshold, effectively filtering out target language audio and corresponding source language audio from multiple first audio groups whose pronunciation does not match the subtitles.

[0120] Among them, known human voice separation techniques in the field, such as the Demucus model, can be used to separate the source language audio segments and target language audio segments in each second audio group. This is equivalent to extracting the human voice signals of the source language audio segments and target language audio segments in each second audio group, so that the source language audio segments and target language audio segments in the third audio group mainly contain pure human voice tracks, thereby eliminating the influence of background music and environmental noise.

[0121] In practical applications, noise detection methods known in the art, such as spectrum analysis, power spectral density measurement, and signal-to-noise ratio calculation, can be used to detect the noise values ​​of the source language audio segments and the target language audio segments. Furthermore, naturalness detection methods known in the art can be used, such as Mel-frequency cepstral distortion, fundamental frequency (F0) continuity, and speech pause distribution, to detect audio naturalness. Alternatively, pre-trained audio quality detection models can be used to detect the naturalness of the source language audio segments and the target language audio segments in each third audio group. This disclosure does not limit the scope of these methods. Audio quality (noise value and naturalness) can be measured using a scoring system; a higher score indicates higher naturalness and lower noise value. Therefore, audio with quality below a preset quality threshold (i.e., a preset scoring threshold) can be considered substandard. This allows for the elimination of third audio groups with quality below the preset quality threshold, resulting in multiple fourth audio groups with higher quality. Then, a second dataset can be constructed based on the source language audio segments and target language audio segments in multiple fourth audio groups, as well as the target language subtitle text corresponding to the target language audio segments in each fourth audio group. That is, each source language audio sample in the second dataset can be a source language audio segment in each fourth audio group, the second target language text sample can be the target language subtitle text corresponding to the target language audio segments in each fourth audio group, and the second target language audio sample can be a target language audio segment in each fourth audio group.

[0122] It should be understood that high-quality audio samples can be obtained through the construction process of the second dataset described above. In practical applications, it is also possible to further locate the roles corresponding to the source language audio segments and target language audio segments in each fourth audio group, i.e., who speaks each audio segment. For example, a speaker recognition model disclosed in this field can be used to extract the voiceprint features (i.e., timbre features) of the source language audio segments in each fourth audio group. This speaker recognition model can be a speaker recognition model based on a densely connected time-delay neural network, which has accurate speaker recognition performance and faster inference speed. Specifically, the feature extraction component in the speaker recognition model can be used to extract voiceprint features separately for the source language audio segments in each fourth audio group, and the voiceprint features can be compressed into a vector space of the same dimension. Then, an unsupervised clustering algorithm is used to cluster the extracted voiceprint features. This clustering process can group the source language audio segments in each fourth audio group into different categories according to their respective voiceprint features. It can group audios with similar timbres (i.e., similar voiceprints) together. Each category can represent a role. During this process, the clustering parameters can be continuously adjusted to maximize the distinction between different categories. At the same time, noisy audio that cannot be classified is filtered out. Thus, the role type corresponding to the source language audio segment in each fourth audio group can be obtained, which is equivalent to obtaining the role type corresponding to the target language audio segment in each fourth audio group. In this way, some or all of the fourth audio groups corresponding to the role types can be selected in the second dataset to construct the second dataset. Thus, the audio samples in the second dataset can be multi-timbre (i.e., multiple people speaking) audio. This embodiment of the disclosure does not limit this.

[0123] Based on the second dataset mentioned above, step S52, which involves training the first text encoder, the first decoder, and the initial emotion extractor using the second dataset to obtain the trained second text encoder, the second decoder, and the first emotion extractor, may include:

[0124] Step S521: For any source language audio sample corresponding to a second target language text sample, use the first text encoder to encode the phoneme sequence of the second target language text sample to obtain the text encoding features of the second target language text sample.

[0125] Step S522: Use the initial emotion extractor to extract the audio emotion features of the source language audio samples from the source language audio samples.

[0126] Step S523: The text encoding features of the second target language text sample are fused with the audio emotion features of the source language audio sample to obtain the emotion text features corresponding to the second target language text sample.

[0127] Step S524: Using the first decoder, generate the predicted target language audio corresponding to the second target language text sample based on the emotional text features corresponding to the second target language text sample;

[0128] Step S525: Based on the difference between the predicted target language audio corresponding to the second target language text sample and the second target language audio sample corresponding to the second target language text sample, adjust the parameters of the first text encoder, the first decoder, and the initial emotion extractor to obtain the trained second text encoder, the first emotion extractor, and the second decoder.

[0129] The implementation methods of steps S521 to S524 can refer to the implementation methods of steps S12 to S15 in the above embodiments of this disclosure, and will not be repeated here.

[0130] In step S525, based on the difference between the predicted target language audio corresponding to the second target language text sample and the corresponding second target language audio sample, the parameters of the first text encoder, the first decoder, and the initial sentiment extractor are adjusted. This can be understood as calculating the loss based on the difference between the predicted target language audio corresponding to the second target language text sample and the corresponding second target language audio sample. For example, the L1 loss, L2 loss, and cross-entropy loss between the Mel spectrum of the predicted target language audio and the Mel spectrum of the second target language audio sample can be calculated. This embodiment of the present disclosure does not limit this. Then, the parameters of the first text encoder, the first decoder, and the initial sentiment extractor can be adjusted using backpropagation, gradient descent, etc., to obtain the trained second text encoder, the first sentiment extractor, and the second decoder. It should be understood that in the second training phase, the first text encoder, the first decoder, and the initial sentiment extractor can be trained iteratively multiple times. Therefore, training batches can be divided based on the second dataset to utilize multiple batches of training samples to iteratively train the first text encoder, the first decoder, and the initial sentiment extractor. This embodiment of the present disclosure does not limit this.

[0131] In step S53, a more stable emotion-transfer speech synthesis model can be trained using the third dataset. The model input includes target language text and target language audio, and the output is target language audio. The training results from the second training phase are used as the training objects. Since the model trained in the second training phase, while exhibiting strong emotional expression, is highly dependent on the quality of the input source language audio, fine-tuning can be performed on multiple high-quality monophonic third datasets to improve pronunciation stability. Specifically, the third target language audio samples in the third dataset can be obtained by recording the pronunciation of native speakers with standard pronunciation reading various preset third target language texts using professional equipment. This indicates that the pronunciation quality of the third target language audio samples in the third dataset is high, thus eliminating the need for preprocessing or audio quality detection. In this third dataset, the audio samples of the third target language can be monophonic, that is, each audio sample of the third target language in the third dataset can be from the same speaker. In practical applications, multiple monophonic third datasets can be constructed for the third stage of training. For example, a third dataset containing two monophonic sounds can be constructed, and the audio duration of each sound can be 1 hour. This embodiment of the present disclosure does not limit this.

[0132] Based on the aforementioned third dataset, in step S53, the second text encoder, the second decoder, and the first emotion extractor are fine-tuned using the third dataset to obtain a trained text encoder, which may include:

[0133] Step S531: For any third target language text sample, use the second text encoder to encode the phoneme sequence of the third target language text sample to obtain the text encoding features of the third target language text sample.

[0134] Step S532: Use the first emotion extractor to extract the audio emotion features of the third target language audio sample from the third target language audio sample corresponding to the third target language text sample;

[0135] Step S533: The text encoding features of the third target language text sample are fused with the audio emotion features of the corresponding third target language audio sample to obtain the emotional text features of the third target language audio sample.

[0136] Step S534: Using the second decoder, based on the emotional text features corresponding to the third target language audio sample, generate the predicted target language audio corresponding to the third target language text sample;

[0137] Step S535: Based on the difference between the predicted target language audio corresponding to the third target language text sample and the third target language audio sample, adjust the parameters of the second text encoder, the second decoder and the first emotion extractor to obtain the trained text encoder, emotion extractor and decoder.

[0138] It should be understood that steps S531 to S534 can be implemented with reference to steps S12 to S15 in the above embodiments of this disclosure, and will not be repeated here. The difference is that in step S523, the audio emotion features of the third target language audio sample are extracted, so that the emotion features in the third target language audio sample are used as an emotion reference to generate the target language audio.

[0139] In step S535, based on the difference between the predicted target language audio corresponding to the third target language text sample and the third target language audio sample, the parameters of the second text encoder, the second decoder, and the first emotion extractor are adjusted. This can be understood as calculating the loss based on the difference between the predicted target language audio corresponding to the third target language text sample and the corresponding third target language audio sample. For example, the L1 loss, L2 loss, and cross-entropy loss between the Mel spectrum of the predicted target language audio and the Mel spectrum of the third target language audio sample can be calculated. This embodiment of the present disclosure does not limit this. Then, the parameters of the second text encoder, the second decoder, and the first emotion extractor can be adjusted using backpropagation, gradient descent, etc., to obtain the trained text encoder, emotion extractor, and decoder. It should be understood that in the third training stage, the second text encoder, the second decoder, and the first emotion extractor can be trained iteratively multiple times. Therefore, training batches can be divided based on the third dataset to utilize multiple batches of training samples to iteratively train the second text encoder, the second decoder, and the first emotion extractor. This embodiment of the present disclosure does not limit this.

[0140] In practical applications, it has been found during model iterative training that there are significant differences between men and women in terms of pronunciation, emotion, and rhythm. This makes training male and female voices separately more effective than training models with mixed male and female voices. Therefore, we can construct a first dataset, a second dataset, and a third dataset consisting entirely of male voices, and a first dataset, a second dataset, and a third dataset consisting entirely of female voices, and train two sets of emotion transfer speech synthesis models respectively using the three-stage training method described above to generate target language audio in male and female voices. Alternatively, we can use a dataset with mixed male and female voices as the first dataset, and datasets with separate male and female voices as the second and third datasets to train two sets of emotion transfer speech synthesis models. This disclosure does not limit the scope of the embodiments.

[0141] Based on this, in the above Figure 1 or Figure 2 In the illustrated speech synthesis method, after acquiring any source language dubbing audio, it can first detect whether the voice in the source language dubbing audio is male or female. If it is male, the emotion transfer speech synthesis model corresponding to male voices is used to generate target language audio; if it is female, the emotion transfer speech synthesis model corresponding to female voices is used to generate target language audio. Of course, a first dataset, a second dataset, and a third dataset containing a mixture of male and female voices can also be used to train an emotion transfer speech synthesis model, and this emotion transfer speech synthesis model can be used to generate target language audio in both male and female voices. This embodiment of the present disclosure does not limit this approach.

[0142] According to the three-stage training method of this disclosure, the first training stage uses a large amount of target language audio corpus to train the base model of the speech synthesis model. The second training stage, based on the parameters of the base model, uses source-target language joint training to create an emotion-infused speech synthesis model. The third training stage, to ensure the stability of timbre and pronunciation, performs fine-tuning based on single-timbre target language corpus. Through this three-stage training method, an emotion-transfer speech synthesis model with stable pronunciation and nuanced emotions can be obtained.

[0143] As mentioned above, a semantic extractor can also be used in the speech synthesis method. This semantic extractor can be a pre-trained semantic extraction model. In the three-stage training process mentioned above, the semantic extractor can be used directly to extract the semantic features of the phoneme sequence of each target language text sample. The semantic features extracted by the semantic extractor are then fused with the emotional text features corresponding to the target language audio sample to obtain comprehensive features, which are then used in the model training process of each training stage. For example, in the second training stage, the semantic extractor can be used to extract the semantic features of the phoneme sequence of the second target language text sample. The semantic features of the phoneme sequence of the second target language text sample are then fused with the emotional text features corresponding to the second target language text sample to obtain comprehensive features corresponding to the second target language text sample. Then, the first decoder is used to generate the predicted target language audio corresponding to the second target language text sample based on the comprehensive features corresponding to the second target language text sample. Finally, the model parameters are adjusted according to step S525.

[0144] It should be understood that the semantic extractor participates in model training in the third training stage in the same way as in the second training stage, and will not be repeated here. The semantic extractor may not participate in the model training in the first training stage, or it may participate in the first training stage. When the semantic extractor participates in the first training stage, the semantic features extracted by the semantic extractor can be fused with the text encoding features, and the initial decoder can be used to decode the fused features to generate the predicted target language audio corresponding to the first target language text sample, and then the model parameters can be adjusted. This disclosure does not limit this aspect.

[0145] According to the three-stage training method of this disclosure, a data extraction and quality detection process is implemented, a batch of high-quality joint emotion corpora is accumulated, and a source-target joint training approach is adopted, which can provide better emotion transfer effects across languages. Furthermore, the three-stage training method, compared to the prior art which only performs one fine-tuning training on the pre-trained model, significantly improves the pronunciation effect of the target language audio output by the emotion transfer speech synthesis module. Additionally, a multi-class emotion recognition model is used to extract emotion features. Compared to existing speech synthesis schemes that rely on the CLIP model (Contrastive Language-Image Pre-training) for emotion feature extraction (primarily utilizing the CLIP model's multimodal capabilities between text and audio, but with limited emotion extraction capabilities), the multi-class emotion recognition model can extract emotion features more effectively.

[0146] Based on the speech synthesis method and three-stage training method proposed in this disclosure, a cross-language source-target language joint emotion transfer speech synthesis framework is realized. This framework mainly involves the following technical aspects: 1) Construction of joint source-target language corpus: Based on the rich audio resources of film and television dramas on video platforms, high-quality multi-voice joint TTS training corpus of source and target languages ​​can be constructed through audio preprocessing, quality detection models, speaker clustering, and other methods, providing strong support for training a refined emotion speech synthesis model. 2) Emotion feature extraction: Using a multi-classification emotion recognition model trained with film and television drama emotion corpus, the intermediate layer of the model is explicitly output as the emotion vector feature of the audio. 3) Emotion transfer speech synthesis: The emotion feature vector output by the emotion extractor is embedded into a classic speech synthesis model. By modifying modules such as the text encoder of VITS, a speech synthesis model with emotion transfer capabilities is constructed. 4) A joint training framework for source and target languages: To improve the stability of the trained emotion transfer speech synthesis model, a three-stage training method was adopted, which improved the pronunciation of the target language audio output by the emotion transfer speech synthesis model. 5) A stable production framework for emotion-based speech synthesis: During audio production, the quality of the target language audio is uncontrollable due to the inability to control the quality of the source language audio. To ensure the quality of the produced audio, the target language audio with poor pronunciation can be manually verified or automatically verified using ASR. Then, new target reference audio is matched based on a discrete audio library and an emotion vector retrieval library to replace the source language dubbing audio, thus ensuring that the produced audio is emotionally rich and has stable pronunciation.

[0147] Figure 6 A block diagram of a speech synthesis apparatus according to an embodiment of the present disclosure is shown, such as Figure 6 As shown, the device includes:

[0148] The acquisition module 601 is used to acquire the source language dubbing audio corresponding to the source language subtitle text of the source video, and the target language subtitle text translated from the source language subtitle text, wherein the source language and the target language are different;

[0149] The feature extraction module 602 is used to extract audio emotion features from the source language dubbing audio using an emotion extractor, wherein the audio emotion features characterize the emotion expressed by the source language dubbing audio.

[0150] The text encoding module 603 is used to convert the target language subtitle text into a phoneme sequence, and to encode the phoneme sequence using a text encoder to obtain text encoding features;

[0151] The feature fusion module 604 is used to fuse the audio emotion features with the text encoding features to obtain emotion text features;

[0152] The audio generation module 605 is used to generate target language audio based on the emotional text features using a decoder, and the target language audio is used as dubbing audio for the source video in the target language.

[0153] In one possible implementation, the audio emotion feature includes the emotion feature of each frame of audio in L frames of the source language dubbing audio, where L represents the number of frames in the source language dubbing audio; wherein, fusing the audio emotion feature with the text encoding feature to obtain the emotion text feature includes: calculating the mean of the emotion features of the L frames of audio in the audio emotion feature to obtain a first emotion feature; performing a high-dimensional mapping on the first emotion feature to obtain a second emotion feature, the second emotion feature having the same dimension as the text encoding feature; and adding the second emotion feature to the text encoding feature to obtain the emotion text feature.

[0154] In one possible implementation, the device further includes:

[0155] A semantic extraction module is used to extract semantic features from the phoneme sequence using a semantic extractor, wherein the semantic features characterize the semantics of the target language subtitle text;

[0156] The feature fusion module 604 is further configured to fuse the emotional text features with the semantic features to obtain a comprehensive feature;

[0157] The audio generation module 605 is also used to generate target language audio based on the comprehensive features using the decoder.

[0158] In one possible implementation, after generating the target language audio, the apparatus further includes:

[0159] The conversion module is used to convert the target language audio into recognized text using the speech recognition technology corresponding to the target language.

[0160] An error rate determination module is used to determine the character error rate of the identified text based on the difference between the identified text and the target language subtitle text, wherein the character error rate characterizes the pronunciation error rate of the target language audio;

[0161] The matching module is used to select target reference audio from a preset source language audio library that matches the emotion expressed by the source language dubbing audio when the character error rate is higher than a preset threshold, so as to regenerate the target language audio using the audio emotion features of the target reference audio.

[0162] The source language audio library includes reference audio in various source languages ​​representing different emotions.

[0163] In one possible implementation, the source language audio library includes a first audio library, where reference audios are labeled with emotion categories. The step of selecting target reference audios from the preset source language audio library that match the emotion expressed by the source language dubbing audio includes: performing emotion recognition on the source language dubbing audio using an emotion recognition model to obtain an emotion classification result for the source language dubbing audio, where the emotion classification result characterizes the emotion category expressed by the source language dubbing audio; and, based on the emotion categories labeled on each reference audio in the first audio library, using reference audios whose emotion categories match the emotion classification results as target reference audios.

[0164] In one possible implementation, the source language audio library includes a second audio library, wherein reference audios in the second audio library are labeled with emotional audio features extracted using the emotion extractor. The step of selecting target reference audios from the preset source language audio library that match the emotions expressed by the source language dubbing audio includes: calculating the similarity between the audio emotional features of the source language dubbing audio and the emotional audio features labeled in each reference audio in the second audio library, and selecting the reference audio with the highest similarity as the target reference audio.

[0165] In one possible implementation, the training process of the emotion extractor, text encoder, and decoder includes: in a first training phase, an initial text encoder and an initial decoder are trained using a first dataset to obtain a trained first text encoder and a first decoder, wherein the first dataset includes: multiple first target language text samples and first target language audio samples corresponding to each first target language text sample; in a second training phase, the first text encoder, the first decoder, and the initial emotion extractor are trained using a second dataset to obtain a trained second text encoder, a second decoder, and a first emotion extractor; wherein the second dataset includes: multiple source language audio samples, second target language text samples corresponding to each source language audio sample, and corresponding second target language audio samples; in a third training phase, the second text encoder, the second decoder, and the first emotion extractor are fine-tuned using a third dataset to obtain a trained text encoder, decoder, and emotion extractor, wherein the third dataset includes: multiple third target language texts and third target language audio samples corresponding to each third target language text, wherein the third target language audio samples are obtained by collecting audio of the third target language text read aloud in standard pronunciation.

[0166] In one possible implementation, training an initial text encoder and an initial decoder using a first dataset to obtain a trained first text encoder and a first decoder includes: for any first target language text sample, encoding the phoneme sequence of the first target language text sample using the initial text encoder to obtain text encoding features of the first target language text sample; using the initial decoder based on the text encoding features of the first target language text sample to generate a predicted target language audio corresponding to the first target language text sample; and adjusting the parameters of the initial text encoder and the initial decoder based on the difference between the predicted target language audio corresponding to the first target language text sample and the first target language audio sample corresponding to the first target language text sample to obtain the trained first text encoder and the first decoder.

[0167] In one possible implementation, the step of fine-tuning the second text encoder, the second decoder, and the first emotion extractor using a third dataset to obtain a trained text encoder includes: for any third target language text sample, encoding the phoneme sequence of the third target language text sample using the second text encoder to obtain the text encoding features of the third target language text sample; extracting the audio emotion features of the third target language audio sample from the third target language audio sample corresponding to the third target language text sample using the first emotion extractor; fusing the text encoding features of the third target language text sample with the audio emotion features of the corresponding third target language audio sample to obtain the emotion text features corresponding to the third target language audio sample; generating a predicted target language audio corresponding to the third target language text sample using the second decoder based on the emotion text features corresponding to the third target language audio sample; and adjusting the parameters of the second text encoder, the second decoder, and the first emotion extractor based on the difference between the predicted target language audio corresponding to the third target language text sample and the third target language audio sample to obtain the trained text encoder, emotion extractor, and decoder.

[0168] In one possible implementation, the construction process of the second dataset includes: acquiring bilingual subtitle files, source language dubbing audio data, and target language dubbing audio data corresponding to the original long video data, wherein the bilingual subtitle files include the start and end times of each bilingual subtitle text display, and the bilingual subtitle text includes target language subtitle text and source language subtitle text; based on the start and end times of each bilingual subtitle text display in the bilingual subtitle files, segmenting the source language dubbing audio data and target language dubbing audio data to obtain multiple segmented first audio groups, each first audio group including segmented source language audio segments and target language audio segments; using the speech recognition technology corresponding to the target language, converting the target language audio segments in each first audio group into recognized text, and comparing the recognized text converted from each target language audio segment with the target language subtitle text corresponding to each target language audio segment. The differences between the audio segments are used to determine the character error rate of the recognized text converted from each target language audio segment; the first audio groups containing target language audio segments with character error rates higher than a preset threshold are filtered out to obtain multiple second audio groups; by performing voice separation on the source language audio segments and target language audio segments in each second audio group, multiple third audio groups are obtained, and the source language audio segments and target language audio segments in each third audio group contain human voice tracks; the audio quality of the source language audio segments and target language audio segments in each third audio group is detected, and the third audio groups with audio quality lower than a preset quality threshold are filtered out to obtain multiple fourth audio groups, wherein the audio quality includes naturalness and / or noise value; based on the source language audio segments and target language audio segments in the multiple fourth audio groups, and the target language subtitle text corresponding to the target language audio segments in each fourth audio group, the second dataset is obtained.

[0169] According to the speech synthesis method of this disclosure, the audio emotion features of the source language dubbing audio are extracted and fused with the text encoding features of the target language subtitle text to obtain emotion text features containing emotion information of the source language dubbing audio and text information of the target language subtitle text. Based on these emotion text features, target language audio with the emotion expression of the source language dubbing audio can be generated. In other words, high-quality target language audio with the emotion of the source language dubbing audio can be automatically and efficiently generated without the need for professional dubbing equipment and conditions, or the expenditure of a large amount of manpower and financial resources. This makes the generation cost of dubbing audio from the source language to the target language lower and the efficiency higher.

[0170] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0171] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0172] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0173] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0174] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 7 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0175] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0176] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0177] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0178] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0179] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0180] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0181] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0182] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0184] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the source language dubbing audio corresponding to the source language subtitle text of the source video, and the target language subtitle text translated from the source language subtitle text, wherein the source language and the target language are different; An emotion extractor is used to extract audio emotion features from the source language dubbing audio, and the audio emotion features represent the emotions expressed by the source language dubbing audio. The target language subtitle text is converted into a phoneme sequence, and the phoneme sequence is encoded using a text encoder to obtain text encoding features; The audio emotion features are fused with the text encoding features to obtain the emotion text features; The decoder generates target language audio based on the emotional text features, and the target language audio is used as the dubbing audio for the source video in the target language. The method further includes, after generating the target language audio, the following: Using the speech recognition technology corresponding to the target language, the audio in the target language is converted into recognized text; Based on the difference between the identified text and the target language subtitle text, the character error rate of the identified text is determined, and the character error rate represents the pronunciation error rate of the target language audio; If the character error rate is higher than a preset threshold, target reference audio that matches the emotion expressed by the source language dubbing audio is selected from a preset source language audio library, so as to regenerate the target language audio using the audio emotion features of the target reference audio. The source language audio library includes reference audio in various source languages ​​representing different emotions.

2. The method according to claim 1, characterized in that, The audio emotional features include the emotional features of each frame of the L frames of the source language dubbing audio, where L represents the number of frames of the source language dubbing audio. The step of fusing the audio emotion features with the text encoding features to obtain emotion text features includes: The first emotional feature is obtained by calculating the mean of the emotional features of L frames of audio in the audio emotional features. The first emotion feature is mapped in a high dimension to obtain the second emotion feature, and the dimension of the second emotion feature is the same as the dimension of the text encoding feature. The second emotion feature is added to the text encoding feature to obtain the emotion text feature.

3. The method according to claim 1, characterized in that, The method further includes: A semantic extractor is used to extract semantic features from the phoneme sequence, and the semantic features characterize the semantics of the target language subtitle text; The step of generating target language audio using a decoder based on the emotional text features includes: The emotional text features are fused with the semantic features to obtain a comprehensive feature; The decoder generates audio in the target language based on the comprehensive features.

4. The method according to claim 1, characterized in that, The source language audio library includes a first audio library, in which reference audios are labeled with emotion categories. The step of selecting target reference audios from the preset source language audio library that match the emotions expressed in the source language dubbing audio includes: An emotion recognition model is used to perform emotion recognition on the source language dubbing audio, and an emotion classification result of the source language dubbing audio is obtained. The emotion classification result represents the emotion category expressed by the source language dubbing audio. Based on the emotion categories labeled in each reference audio in the first audio library, the reference audio whose emotion category matches the emotion classification result is used as the target reference audio.

5. The method according to claim 1, characterized in that, The source language audio library includes a second audio library, in which reference audio is labeled with emotional audio features extracted using the emotion extractor. The step of selecting target reference audio from the preset source language audio library that matches the emotion expressed in the source language dubbing audio includes: Calculate the similarity between the audio emotion features of the source language dubbing audio and the emotional audio features of each reference audio in the second audio library, and take the reference audio with the highest similarity as the target reference audio.

6. The method according to claim 1, characterized in that, The training process for the emotion extractor, text encoder, and decoder includes: In the first training phase, an initial text encoder and an initial decoder are trained using the first dataset to obtain the trained first text encoder and first decoder. The first dataset includes: multiple first target language text samples and first target language audio samples corresponding to each first target language text sample. In the second training phase, the first text encoder, the first decoder, and the initial emotion extractor are trained using the second dataset to obtain the trained second text encoder, the second decoder, and the first emotion extractor; wherein, the second dataset includes: multiple source language audio samples, a second target language text sample corresponding to each source language audio sample, and a corresponding second target language audio sample; In the third training phase, the second text encoder, the second decoder, and the first emotion extractor are fine-tuned using the third dataset to obtain the trained text encoder, decoder, and emotion extractor. The third dataset includes multiple third target language texts and third target language audio samples corresponding to each third target language text. The third target language audio samples are obtained by collecting audio of the third target language texts read aloud in standard pronunciation.

7. The method according to claim 6, characterized in that, The step of training an initial text encoder and an initial decoder using the first dataset to obtain a trained first text encoder and a first decoder includes: For any text sample in the first target language, the phoneme sequence of the text sample in the first target language is encoded using the initial text encoder to obtain the text encoding features of the text sample in the first target language. The initial decoder is used to generate the predicted target language audio corresponding to the first target language text sample based on the text encoding features of the first target language text sample; Based on the difference between the predicted target language audio corresponding to the first target language text sample and the first target language audio sample corresponding to the first target language text sample, the parameters of the initial text encoder and the initial decoder are adjusted to obtain the trained first text encoder and first decoder.

8. The method according to claim 6, characterized in that, The step of fine-tuning the second text encoder, the second decoder, and the first emotion extractor using a third dataset to obtain a trained text encoder includes: For any third target language text sample, the phoneme sequence of the third target language text sample is encoded using the second text encoder to obtain the text encoding features of the third target language text sample. The first emotion extractor is used to extract the audio emotion features of the third target language audio sample from the third target language audio sample corresponding to the third target language text sample; The text encoding features of the third target language text sample are fused with the audio emotion features of the corresponding third target language audio sample to obtain the emotion text features corresponding to the third target language audio sample. The second decoder is used to generate a predicted target language audio corresponding to the third target language audio sample based on the emotional text features corresponding to the third target language audio sample; Based on the difference between the predicted target language audio corresponding to the third target language text sample and the third target language audio sample, the parameters of the second text encoder, the second decoder, and the first emotion extractor are adjusted to obtain the trained text encoder, emotion extractor, and decoder.

9. The method according to claim 6, characterized in that, The construction process of the second dataset includes: Obtain the bilingual subtitle file, source language dubbing audio data, and target language dubbing audio data corresponding to the original long video data. The bilingual subtitle file includes the start and end times of the display of each bilingual subtitle text, and the bilingual subtitle text includes the target language subtitle text and the source language subtitle text. Based on the start and end times of each bilingual subtitle text in the bilingual subtitle file, the source language dubbing audio data and the target language dubbing audio data are segmented to obtain multiple segmented first audio groups. Each first audio group includes segmented source language audio segments and target language audio segments. Using the speech recognition technology corresponding to the target language, the target language audio segments in each first audio group are converted into recognized text, and the character error rate of the recognized text converted from each target language audio segment is determined based on the difference between the recognized text converted from each target language audio segment and the target language subtitle text corresponding to each target language audio segment. The first audio groups containing target language audio segments with character error rates higher than a preset threshold are filtered out to obtain multiple second audio groups; By separating the source language audio segment and the target language audio segment in each second audio group, multiple third audio groups are obtained. Each third audio group contains a human voice track in the source language audio segment and the target language audio segment. The audio quality of the source language audio segment and the target language audio segment in each third audio group is detected, and the third audio groups with audio quality lower than a preset quality threshold are filtered out to obtain multiple fourth audio groups, wherein the audio quality includes naturalness and / or noise value; The second dataset is obtained based on the source language audio segments and target language audio segments in the plurality of fourth audio groups, and the target language subtitle text corresponding to the target language audio segments in each fourth audio group.

10. A speech synthesis device, characterized in that, include: The acquisition module is used to acquire the source language dubbing audio corresponding to the source language subtitle text of the source video, and the target language subtitle text translated from the source language subtitle text, wherein the source language and the target language are different; The feature extraction module is used to extract audio emotion features from the source language dubbing audio using an emotion extractor, wherein the audio emotion features characterize the emotion expressed by the source language dubbing audio. The text encoding module is used to convert the target language subtitle text into a phoneme sequence, and to encode the phoneme sequence using a text encoder to obtain text encoding features; The feature fusion module is used to fuse the audio emotion features with the text encoding features to obtain emotion text features; An audio generation module is used to generate target language audio based on the emotional text features using a decoder, and the target language audio is used as dubbing audio for the source video in the target language; After generating the target language audio, the device further includes: The conversion module is used to convert the target language audio into recognized text using the speech recognition technology corresponding to the target language. An error rate determination module is used to determine the character error rate of the identified text based on the difference between the identified text and the target language subtitle text, wherein the character error rate characterizes the pronunciation error rate of the target language audio; The matching module is used to select target reference audio from a preset source language audio library that matches the emotion expressed by the source language dubbing audio when the character error rate is higher than a preset threshold, so as to regenerate the target language audio using the audio emotion features of the target reference audio. The source language audio library includes reference audio in various source languages ​​representing different emotions.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

12. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images