A method and device for emotion recognition in dubbing of a movie or a TV series

Abstract

The present disclosure relates to a method and device for emotion recognition in dubbing of a movie or TV series, which comprises: obtaining movie or TV series audio in a source language and text to be dubbed in a target language; processing the movie or TV series audio by using a trained emotion recognition model corresponding to the source language to extract target emotion features; wherein the emotion recognition model is trained based on an audio pre-training sub-model and a feature extraction sub-model using labeled training data; the feature extraction sub-model is used to extract audio features, and the audio pre-training sub-model is used to encode the extracted audio features to generate emotion features; and using a trained speech generation model corresponding to the target language to perform speech synthesis on the text to be dubbed based on the target emotion features to generate dubbing in the target language corresponding to the movie or TV series audio. According to the present disclosure, the original emotion is retained in the dubbing, realizing seamless conversion of speech across languages, so that the dubbing achieves the effect of "original taste".

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a method and device for emotion recognition in dubbing of a movie or TV series. BACKGROUND

[0002] In the process of overseasization of a movie or TV series, dubbing of the movie or TV series is needed to realize the spread and promotion in different language countries. In the related art, professional dubbers mainly complete dubbing through episodes, but this way needs to have professional dubbing ability and dubbing conditions, and also needs to consume a large amount of manpower and financial resources, and has high cost. SUMMARY

[0003] Therefore, the present disclosure provides a method and device for emotion recognition in dubbing of a movie or TV series, an electronic device, a storage medium and a computer program product.

[0004] According to an aspect of the present disclosure, a method for emotion recognition in dubbing of a movie or TV series is provided, and the method comprises:

[0005] obtaining movie or TV series audio in a source language and to-be-dubbed text in a target language; wherein the movie or TV series audio and the to-be-dubbed text have a corresponding relationship;

[0006] processing the movie or TV series audio by using a trained emotion recognition model corresponding to the source language to extract target emotion features; wherein the emotion recognition model is based on an audio pre-training sub-model and a feature extraction sub-model, and is trained by using labeled training data; the feature extraction sub-model is used to extract audio features, and the audio pre-training sub-model is used to perform feature coding on the extracted audio features to generate emotion features;

[0007] generating dubbing in the target language corresponding to the movie or TV series audio by using a trained speech generation model corresponding to the target language based on the target emotion features.

[0008] In a possible implementation manner, the generating dubbing in the target language corresponding to the movie or TV series audio by using the trained speech generation model corresponding to the target language based on the target emotion features comprises:

[0009] extracting phoneme features of the to-be-dubbed text by using the speech generation model;

[0010] fusing the phoneme features and the target emotion features to obtain fused features;

[0011] performing speech synthesis based on the fused features to generate the dubbing in the target language corresponding to the movie or TV series audio.

[0012] In a possible implementation, the labeled training data comprises a plurality of audio samples in the source language and an emotion label corresponding to each audio sample.

[0013] The method further comprises:

[0014] The audio pre-training sub-model and the feature extraction sub-model are trained in a self-supervised learning manner.

[0015] The plurality of audio samples and the emotion label corresponding to each audio sample are used to train a first to-be-trained model to obtain a trained emotion recognition model; the first to-be-trained model comprises the audio pre-training sub-model, the feature extraction sub-model, and an emotion classification layer, wherein the emotion classification layer is configured to classify the features extracted by the audio pre-training sub-model to obtain a corresponding predicted emotion.

[0016] In a possible implementation, the method comprises:

[0017] The emotion recognition model is embedded into a second to-be-trained model.

[0018] The parameters of the emotion recognition model are fixed, and the second to-be-trained model is trained to obtain the speech generation model.

[0019] In a possible implementation, the classifying the features extracted by the audio pre-training sub-model to obtain a corresponding predicted emotion comprises: calculating a preset emotion type probability distribution corresponding to the features extracted by the audio pre-training sub-model; and taking an emotion type corresponding to the highest probability as the predicted emotion.

[0020] and / or,

[0021] The classifying the features extracted by the audio pre-training sub-model to obtain a corresponding predicted emotion comprises: determining a numerical value of a three-dimensional evaluation index corresponding to the features extracted by the audio pre-training sub-model; and determining the predicted emotion based on the numerical value of the three-dimensional evaluation index; wherein the three-dimensional evaluation index comprises: pleasantness, arousal, and dominance.

[0022] In a possible implementation, the preset emotion types comprise:

[0023] surprise, anger, joy, scream, depression, fear, calm, sadness, excitement;

[0024] or,

[0025] surprise, love, anger, joy, seriousness, scream, embarrassment, depression, fear, whisper, calm, suspicion, sadness, excitement, chat, news broadcast.

[0026] In one possible implementation, the method further includes: analyzing the characteristics of the source language and determining the type of training data corresponding to the source language based on the analysis results; wherein the type of training data includes: continuous emotion corpus or discrete emotion corpus.

[0027] In one possible implementation, the audio pre-training sub-model is a Hubert model, and the feature extraction sub-model is a Wav2vec2 model.

[0028] According to another aspect of this disclosure, an emotion recognition device for film and television dubbing is provided, the device comprising:

[0029] The acquisition module is used to acquire audio of the film / television drama in the source language and the text to be dubbed in the target language; wherein the audio of the film / television drama and the text to be dubbed have a corresponding relationship;

[0030] An emotion recognition module is used to process the audio of the film and television drama using a trained emotion recognition model corresponding to the source language, and extract target emotion features. The emotion recognition model is based on an audio pre-training sub-model and a feature extraction sub-model, and is trained using labeled training data. The feature extraction sub-model is used to extract audio features, and the audio pre-training sub-model is used to encode the extracted audio features to generate emotion features.

[0031] The speech synthesis module is used to synthesize the text to be dubbed using a trained speech generation model corresponding to the target language, based on the target emotion features, to generate a dubbing in the target language corresponding to the audio of the film or television drama.

[0032] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0033] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0034] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0035] Through aspects of the present disclosure, for the cross-language dubbing scene of a film and television drama, the audio of the film and television drama in a source language and the text to be dubbed in a target language are obtained; wherein the audio of the film and television drama and the text to be dubbed have a corresponding relationship; the audio of the film and television drama is processed by using a trained emotion recognition model corresponding to the source language, and target emotion features are extracted; wherein the emotion recognition model is trained based on an audio pre-training sub-model and a feature extraction sub-model, and labeled training data is used for training; the feature extraction sub-model is used to extract audio features, and the audio pre-training sub-model is used to perform feature coding on the extracted audio features to generate emotion features; in this way, unlike the way of emotion recognition based on text or emotion recognition combined with text, image and other multi-modal data, in the present embodiment, the audio of the film and television drama in the source language, which is the single-modal data most accurately reflecting the emotion of the actor, is taken as the processing object, and the emotion recognition model of the source language trained based on the audio pre-training sub-model and the feature extraction sub-model is used to extract accurate target emotion features; at the same time, the emotion recognition model focuses on extracting emotion features, without performing other tasks such as recognizing the speaker, avoiding the influence of redundant processing on feature extraction, thereby ensuring the accuracy of the extracted target emotion features. Further, based on the target emotion features, the text to be dubbed is synthesized by using a trained speech generation model corresponding to the target language, to generate the dubbing in the target language corresponding to the audio of the film and television drama; in this way, the emotion in the audio of the film and television drama in the source language is migrated to the corresponding dubbing synthesis in the target language, and the generated dubbing retains the rich emotion of the actor in the film and television drama in the source language, thereby realizing seamless conversion of the cross-language speech of the film and television drama, making the cross-language dubbing of the film and television drama achieve the effect of "original flavor", and enabling audiences in different languages to hear the "original flavor" dialogue of the film and television drama.

[0036] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated in and form a part of the specification, illustrate examples of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0038] Figure 1 A flowchart of an emotion recognition method in film and television dubbing according to an embodiment of the present disclosure is shown;

[0039] Figure 2 A flowchart of an emotion recognition model training method according to an embodiment of the present disclosure is shown;

[0040] Figure 3 A schematic diagram of the distribution of each emotion in an emotion space according to an embodiment of the present disclosure is shown;

[0041] Figure 4 FIG. 1 shows a structural diagram of an emotion recognition model according to an embodiment of the disclosure;

[0042] Figure 5 FIG. 2 shows a schematic diagram of an emotion recognition method in dubbing according to an embodiment of the disclosure;

[0043] Figure 6 FIG. 3 shows a structural diagram of an emotion recognition apparatus in dubbing according to an embodiment of the disclosure;

[0044] Figure 7 FIG. 4 shows a block diagram of an electronic device 1900 according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0045] Various exemplary embodiments, features, and aspects of the disclosure will be described below in detail with reference to the accompanying drawings. The same reference numbers in the drawings represent the same elements or features having the same or similar functions. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0046] As used herein, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components, or functions but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or groups thereof.

[0047] When an element is referred to as being "connected", "coupled", "responsive", or "related" to another element, it can be directly connected, coupled, responsive, or related to the other element, or intervening elements can be present.

[0048] Although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of the present inventive concept.

[0049] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0050] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated herein can not be described in detail. It should be appreciated that the present disclosure can be practiced with the exact details as other implementations can omit, add, or modify.

[0051] In the process of overseasization of films and television dramas, dubbing can be performed through artificial intelligence (AI) technology to reduce the cost of dubbing. However, due to the complexity and diversity of human emotional expression, the emotion categories supported when dubbing through AI technology in related technologies are very limited, and the accuracy is insufficient. When processing audio of films and television dramas including rich emotions, the generated dubbing is poor in emotional expression, and cannot meet the delicate emotional expression requirements of films and television dramas.

[0052] To solve the above technical problems, an emotion recognition method in film and television drama dubbing is provided in the embodiments of the present disclosure (see the detailed description below). When dubbing films and television dramas across languages, considering the requirement for delicate expression of emotions in film and television drama dubbing and the particularity of films and television dramas, the audio of films and television dramas in the source language is selected as the processing object by analyzing data of different modalities. Based on the audio pre-training sub-model and the feature extraction sub-model, and using the labeled training data to train the emotion recognition model of the source language, the emotion features of the actors in the audio of films and television dramas in the source language are extracted using the emotion recognition model of the source language. Then, voice synthesis in the target language is performed based on the emotion features of the actors, and the generated dubbing retains the rich emotions of the actors in the source language films and television dramas, thereby realizing seamless conversion of voice across languages for films and television dramas, and achieving the effect of "original flavor" for cross-language dubbing of films and television dramas.

[0053] In the embodiments of the present disclosure, the audio of a film or television play is selected as the processing object. On the one hand, the audio of a film or television play is often the most direct and profound manifestation of the emotions of actors, and can more truly and accurately reflect the emotions of actors than other modalities of data such as images and texts. For example, the audio of a film or television play contains rich information of sound, such as the pitch, speed, pause, laughter, or crying, which cannot be directly conveyed by modalities of data such as texts and images. For example, for the same line of dialogue "I'm fine", if the actor speaks in a low and slow voice, it may express sadness, and if the actor speaks in a light and fast voice, it may really be fine. At this time, the amount of information of the audio is greater, and the emotions of the actor cannot be accurately captured through texts or images. On the other hand, other modalities of data are difficult to obtain or have poor data quality. For example, during the filming process of a film or television play, sound is recorded throughout the process or sound is supplemented later, and the voice of each line of dialogue will not be missing. However, the text of the subtitles of a film or television play is usually generated by automatic speech recognition (ASR) technology. However, due to the fact that the accuracy rate of the ASR technology cannot reach 100%, error accumulation occurs when the text is generated by the ASR technology, and in the case of intense emotions, the ASR cannot convert the audio into text, or even into completely incorrect text, resulting in missing or incorrect information in the generated text. For another example, in a film or television play, in many cases, the actor may not be shown or the facial expression may not be obvious (for example, turning his head, wearing sunglasses, or insufficient light), resulting in limitations of image data. In this way, the true emotions of the actor cannot be extracted from the facial image.

[0054] Exemplarily, the emotion recognition method in film or television play dubbing proposed in the embodiments of the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be a desktop terminal or a mobile terminal, for example, can be a notebook computer, a tablet computer, a desktop computer, a smart phone, a smart speaker, a smart watch, a smart television, a vehicle-mounted terminal, and various types of electronic devices. The server can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks, and basic cloud computing services such as big data and artificial intelligence platforms.

[0055] It should be noted that the emotion recognition in the above-mentioned film and television dubbing scene described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and does not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art can know that for the emergence of other similar or new scenes, for example, in the field of traffic safety, the degree of fatigue of the driver is monitored through voice emotion recognition, and in the field of medical health, the emotion state of the patient is recognized by the doctor through voice emotion recognition, and the like, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0056] Figure 1 A flowchart of a method for emotion recognition in film and television dubbing is shown according to an embodiment of the present disclosure. As shown in Figure 1 may include the following steps:

[0057] Step 101, obtaining film and television audio in a source language and text to be dubbed in a target language; the film and television audio and the text to be dubbed have a corresponding relationship.

[0058] Among them, the source language is the language used by the actor when speaking in the process of shooting or producing the film and television, for example, Chinese, English, Japanese, Thai, etc. The target language is the language used for dubbing the film and television. Among them, the target language and the source language are different languages.

[0059] Exemplarily, the film and television audio can include the voices of different characters in the film and television, and the text to be dubbed can include the dialogue texts of different characters in the film and television; wherein the film and television audio and the text to be dubbed have a corresponding relationship, which means that the film and television audio and the text to be dubbed respectively include the voices and dialogue texts at the same time in the film and television, that is, the dialogue text of the same actor is synchronized with the audio.

[0060] For example, in the scene where a certain Chinese TV series needs to be dubbed in Thai for playing in Thailand, the source language is Chinese, and the target language is Thai; the film and television audio in the source language includes the Chinese voices of different characters in the Chinese TV series, and the text to be dubbed in the target language includes the Thai dialogue of different characters in the Chinese TV series.

[0061] Step 102, using a trained emotion recognition model corresponding to the source language to process the film and television audio, and extracting target emotion features; wherein the emotion recognition model is based on an audio pre-training sub-model and a feature extraction sub-model, and is trained using labeled training data; the feature extraction sub-model is used to extract audio features, and the audio pre-training sub-model is used to encode the extracted audio features to generate emotion features.

[0062] Exemplarily, the audio pre-training sub-model and the feature extraction sub-model can be deep learning models. The audio pre-training sub-model and the feature extraction sub-model can be pre-trained, and then retrained by using the labeled training data in the source language to obtain a trained emotion recognition model (Speech Emotion Recognition, SER) corresponding to the source language.

[0063] The feature extraction sub-model in the trained emotion recognition model corresponding to the source language can extract effective features from the audio of the movie and television drama in the source language and perform vectorization representation, that is, extract audio features; while retaining as much useful physical acoustic information as possible. In this way, the audio features in the audio of the movie and television drama in the source language are extracted by the feature extraction sub-model, and the acoustic characteristics in the audio of the movie and television drama in the source language are retained. Exemplarily, the audio of the movie and television drama in the source language can be discretely sampled, and the discretely sampled data can be input into the feature extraction sub-model. The feature extraction sub-model processes the discretely sampled data to extract audio features.

[0064] The audio pre-training sub-model of the trained emotion recognition model corresponding to the source language is used to encode the audio features extracted from the audio of the movie and television drama in the source language, and convert the audio features into abstract features more suitable for representing emotional states, that is, emotional features. The emotional features have a mapping relationship with the preset emotional types in the emotion space, and can represent the preset emotional types (such as happy, sad, high-pitched, low-pitched, etc.) in the emotion space.

[0065] Exemplarily, the audio of the movie and television drama in the source language can be split into different audio segments, and each audio segment can be input into the trained emotion recognition model corresponding to the source language in turn to generate emotional features corresponding to each audio segment. Each audio segment can correspond to multiple lines of dialogue spoken by different actors, or can be a fixed time length segment, or can be multiple lines of dialogue spoken by the same actor in succession.

[0066] As an example, the audio pre-training sub-model is a Hubert (Hidden Units Bert) model, and the feature extraction sub-model can be a Wav2vec2 (Unsupervised Pre-training for Speech Recognition) model. Based on the Hubert model and the Wav2vec2 model, and by using the labeled training data of the source language for training, a source language corresponding emotion recognition model is obtained. In the emotion recognition model, the Wav2vec2 model can extract robust audio features; the Hubert model is a BERT-based self-supervised learning model, and after training using the training data of the source language, the Hubert model can perform feature coding on the audio features to extract emotion features; accordingly, the emotion recognition model can be referred to as a Hubert-based emotion recognition model (Hubert-SER). Considering that the manual feature extraction based on MFCCs and the like cannot fully represent the emotions in the audio, therefore, the use of the feature extraction sub-model to extract audio features and the use of the audio pre-training sub-model to further extract emotion features help the well-trained source language corresponding emotion recognition model to understand the complex emotions in the audio of the source language movie and television drama. Through a large number of research and verification, the Hubert model is selected as the audio pre-training sub-model, and the Wav2vec2 model is selected as the feature extraction sub-model, which obtains a higher emotion recognition accuracy, and the accuracy of emotion recognition compared to the manual feature extraction method is obviously improved. For example, when the source language is Chinese, the Chinese version of the HuBERT model is selected as the audio pre-training sub-model, and after training using the training data of the source language, the Chinese version of the HuBERT model can extract emotion features that can accurately reflect the emotions in the Chinese audio.

[0067] It should be noted that considering that different languages have different emotion expression characteristics, for different languages, the corresponding emotion recognition model can be pre-trained using the training data corresponding to the language; wherein the architectures of the emotion recognition models corresponding to different languages can be the same. Further, for any source language, the well-trained emotion recognition model corresponding to the source language is selected to process the audio of the movie and television drama of the source language and extract emotion features.

[0068] Step 103, using the well-trained voice generation model corresponding to the target language, performing voice synthesis on the text to be dubbed based on the target emotion features to generate dubbed audio of the movie and television drama in the target language.

[0069] The extracted target emotion feature can capture subtle changes in the audio features related to emotions, such as pitch, rhythm, and intensity. This vectorized emotion representation (i.e., the target emotion feature) can be used to guide the speech generation model to generate speech with specific emotions.

[0070] In one possible implementation, the step can include: extracting phoneme features of the to-be-dubbed text by the speech generation model; fusing the phoneme features with the emotion features to obtain fused features; and performing speech synthesis based on the fused features to generate the dubbing in the target language corresponding to the audio of the film or television series. In this way, by using the trained speech generation model corresponding to the target language, the phoneme features are fused with the extracted target emotion features, and the fused features have emotion expression capability, so that speech containing emotions, i.e., dubbing in the target language corresponding to the audio of the film or television series, can be generated based on the fused features.

[0071] For example, the trained speech generation model in the target language can extract phoneme features and prosody features of the to-be-dubbed text in the target language; then, the extracted phoneme features and prosody features are fused with the extracted target emotion features to obtain fused features; and further, speech synthesis is performed based on the fused features to generate dubbing in the target language corresponding to the audio of the film or television series in the source language.

[0072] It should be noted that for different languages, a corresponding speech generation model (Text-to-Speech, TTS) can be pre-trained using training data corresponding to the language. In this way, for a target language, a speech generation model corresponding to the target language can be selected, and based on the target emotion features extracted from the audio of the film or television series in the source language, speech synthesis is performed on the to-be-dubbed text in the target language to generate dubbing in the target language.

[0073] Exemplarily, a pre-trained emotion recognition model of a source language can be embedded in a speech generation model of a target language, which provides target emotion features for the speech generation model in the process of speech synthesis, so that the speech generation model generates speech with emotion. Correspondingly, the speech generation model embedded with the emotion recognition model can be referred to as an emotional speech generation model (Emotional Text-to-Speech, Emotional-TTS). As an example, the speech generation model can be a model constructed based on a conditional variational autoencoder (Variational Inference with adversarial learning for end-to-end Text-to-Speech, VITS) with adversarial learning for end-to-end speech synthesis; VITS is a speech synthesis architecture based on the combination of a variational autoencoder (Variational Autoencoder, VAE) and a generative adversarial network (Generative Adversarial Network, GAN); wherein the variational autoencoder is responsible for efficiently encoding the fusion features, which can map the fusion features to a low-dimensional latent space, and then restore them to high-quality speech signals through the decoder; in the training stage of the VITS model, the discriminator in the generative adversarial network can perform real-time quality evaluation and optimization on the generated speech signal; by distinguishing the difference between the speech generated by the decoder in the variational autoencoder and the real speech, feedback is provided to the decoder in the variational autoencoder, so as to continuously optimize the ability of the decoder to generate speech.

[0074] Exemplarily, the speech generation model can include an encoder, a projection layer, a stochastic duration predictor, a flow model, and a decoder; the trained emotion recognition model corresponding to the source language can be embedded in the encoder of the speech generation model corresponding to the target language. The encoder is configured to extract phoneme features of the text to be dubbed in the target language, and obtain the target emotion features extracted by the emotion recognition model corresponding to the source language, and fuse the phoneme features and the target emotion features to generate fused features. For example, the phoneme features corresponding to each phoneme and the target emotion features corresponding to each phoneme can be weighted and summed, spliced, or multiplied to generate the fused features corresponding to each phoneme. The projection layer is configured to map the fused features to a latent space. The stochastic duration predictor is configured to predict the pronunciation duration of each phoneme corresponding to the text to be dubbed based on the fused features, map the pronunciation duration of each phoneme to the latent space, and align the fused features with the pronunciation duration of each phoneme. The flow model is configured to generate latent hidden variables based on the alignment result. The decoder is configured to decode the latent hidden variables to generate speech corresponding to the text to be dubbed, i.e., dubbed voice in the target language corresponding to the audio of the movie or TV series. In this way, the speech generation model corresponding to the target language embedded with the emotion recognition model corresponding to the source language can generate high-quality speech with the same emotion as the audio of the movie or TV series in the source language. The training process of the speech generation model can be implemented by existing technologies, and is not limited herein.

[0075] In the embodiments of the present disclosure, for the cross-language dubbing scene of a film or television series, the audio of a film or television series in a source language and the text to be dubbed in a target language are obtained, wherein the audio of the film or television series and the text to be dubbed have a corresponding relationship; the audio of the film or television series is processed by using an emotion recognition model corresponding to the source language that has been trained, and target emotion features are extracted; wherein the emotion recognition model is obtained by training a labeled training data based on an audio pre-training sub-model and a feature extraction sub-model; the feature extraction sub-model is used to extract audio features, and the audio pre-training sub-model is used to encode the extracted audio features to generate emotion features; in this way, unlike the way of emotion recognition based on text or emotion recognition combined with text, image and other multi-modal data, in the embodiments of the present disclosure, the audio of the film or television series in the source language, which is the single-modal data most accurately reflecting the emotions of actors, is taken as the processing object, and the emotion recognition model corresponding to the source language is trained based on the audio pre-training sub-model and the feature extraction sub-model to extract accurate target emotion features; at the same time, the emotion recognition model focuses on extracting emotion features and does not need to perform other tasks such as recognizing speakers, avoiding the influence of redundant processing on feature extraction, thereby ensuring the accuracy of the extracted target emotion features. Furthermore, based on the target emotion features, the text to be dubbed is synthesized by voice based on a trained voice generation model corresponding to the target language, and the dubbing in the target language corresponding to the audio of the film or television series is generated; in this way, the emotions in the audio of the film or television series in the source language are transferred to the corresponding dubbing synthesis in the target language, and the generated dubbing retains the rich emotions of the actors in the film or television series in the source language, thereby realizing the seamless conversion of the voice of the film or television series in different languages, making the cross-language dubbing of the film or television series achieve the effect of "original flavor", and enabling audiences in different languages to hear the "original flavor" dialogue of the film or television series.

[0076] The training process of the emotion recognition model corresponding to the source language and the voice generation model corresponding to the target language will be described below.

[0077] As an example, the labeled training data corresponding to the source language includes a plurality of audio samples in the source language and emotion labels corresponding to each audio sample.

[0078] For example, rich film or television series resources on various platforms can be collected to obtain a plurality of audio samples in different languages, and for any audio sample, the emotion labels of the characters in the audio sample can be labeled, thereby completing the construction of labeled training data with rich emotions of film or television series. For example, the emotion labels of the characters in each audio sample can be labeled by manual labeling.

[0079] Exemplarily, data partitioning and data normalization can be performed on the audio samples to eliminate numerical instability in the calculation process caused by different sizes, prevent model overfitting, and remove noise interference (e.g., background sound such as traffic vehicle sound and multi-person conversation) in the audio samples through spectral subtraction, wavelet denoising algorithm, or Wiener filtering algorithm, so that the audio samples are clearer and easier to identify.

[0080] Figure 2 A flowchart of an emotion recognition model training method according to an embodiment of the present disclosure is shown.

[0081] As shown in Figure 2 The method can include the following steps:

[0082] Step 201: training the audio pre-training sub-model and the feature extraction sub-model in a self-supervised learning manner.

[0083] The self-supervised learning of the feature extraction sub-model can be implemented by existing methods, and is not limited in this regard. Through self-supervised learning, the feature extraction sub-model can learn useful audio features from unlabeled audio data. As an example, the feature extraction sub-model can be a Wav2Vec2 model.

[0084] The self-supervised learning of the audio pre-training sub-model can be implemented by existing methods, and is not limited in this regard. Through self-supervised learning, the audio pre-training sub-model can further capture deep features of the audio data. As an example, the audio pre-training sub-model is HuBERT, which can extract high-quality deep features through self-supervised learning. The deep features extracted by the HuBERT model are not related to the speaker and do not bring other redundant information to mislead the model. At the same time, HuBERT can capture the correlation between continuous features in the audio, and the emotion recognition task is not like the speaker recognition task. As a continuous signal, it is crucial for emotion to obtain the correlation between continuous features.

[0085] Step 202: training the first to-be-trained model using the plurality of audio samples and the emotion labels corresponding to each audio sample to obtain a trained emotion recognition model; the first to-be-trained model includes the audio pre-training sub-model, the feature extraction sub-model, and an emotion classification layer, wherein the emotion classification layer is used to classify the features extracted by the audio pre-training sub-model to obtain the corresponding predicted emotion.

[0086] Exemplarily, after obtaining the feature extraction sub-model and the audio pre-training sub-model, the last output layer of the feature extraction sub-model and the output layer of the audio pre-training sub-model can be removed, and the intermediate layer of the feature extraction sub-model can be spliced with the input layer of the audio pre-training sub-model, so that the audio features extracted by the feature extraction sub-model can be used as the input of the audio pre-training sub-model; at the same time, an emotion classification layer can be configured after the intermediate layer of the audio pre-training sub-model, so that the features extracted by the audio pre-training sub-model can be used as the input of the emotion classification layer, thereby obtaining the first to-be-trained model. Exemplarily, when the intermediate layer of the feature extraction sub-model is spliced with the input layer of the audio pre-training sub-model, a convolutional adaptation layer can be configured to adjust the feature dimension output by the intermediate layer of the feature extraction sub-model to adapt to the feature dimension input by the input layer of the audio pre-training sub-model.

[0087] Since the deep features extracted by the audio pre-training sub-model obtained by the above-mentioned self-supervised learning have universality, in this step, the audio pre-training sub-model in the first to-be-trained model is retrained, so that the audio pre-training sub-model can learn the mapping relationship between the emotion features and the emotion states. The training process adopts a supervised learning manner, and uses a plurality of audio samples corresponding to a source language and emotion labels corresponding to each audio sample pre-labeled as training data; through training, the parameters of the audio pre-training sub-model are constantly updated, exemplarily, the parameters of the audio pre-training sub-model can be constantly adjusted by using an optimization algorithm such as gradient descent, so as to minimize the difference between the predicted emotion output by the emotion classification layer in the first to-be-trained model and the real emotion label. Exemplarily, in the training process, the parameters of the feature extraction sub-model, the audio pre-training sub-model, and the emotion classification layer can also be adjusted to minimize the difference between the predicted emotion output by the emotion classification layer in the first to-be-trained model and the real emotion label. In this way, by retraining the audio pre-training sub-model, the vector representation output by the audio pre-training sub-model can be adjusted to a vector representation having a mapping relationship with the emotion, that is, the deep features extracted by the audio pre-training sub-model are adjusted to emotion features that can express emotion states, so that the audio pre-training sub-model has the ability to accurately extract emotion features.

[0088] Exemplarily, after the training is completed, the emotion classification layer in the first to-be-trained model can be removed, thereby obtaining a trained emotion recognition model corresponding to the source language, which can analyze the audio of a movie or a TV series in the source language to extract audio features, and further generate emotion features that can represent the emotion states of an actor.

[0089] In a possible implementation, the classifying the feature extracted by the audio pre-training sub-model to obtain a corresponding predicted emotion comprises: calculating a preset emotion type probability distribution corresponding to the feature extracted by the audio pre-training sub-model; and taking an emotion type corresponding to a highest probability as the predicted emotion.

[0090] It should be noted that the types of emotion types (i.e., the types of emotion labels) can be flexibly set according to the characteristics of different languages and the richness of training data of different voices, so as to meet the emotion expression needs of different languages. These preset emotion types constitute an emotion space. The richer the training data is, the more types of preset emotion types can be, and correspondingly, the richer the different emotion features extracted by the trained emotion recognition model can express emotions. In the training process, for any audio sample, the feature extraction sub-model in the first to-be-trained model extracts the audio feature, and then the audio pre-training sub-model in the first to-be-trained model encodes the audio feature. The emotion classification layer in the first to-be-trained model calculates the preset emotion type probability distribution corresponding to the encoded feature of the audio pre-training sub-model, takes the emotion type corresponding to the highest probability as the predicted emotion of the audio sample, and adjusts the parameters in the first to-be-trained model based on the preset emotion and the emotion label corresponding to the audio sample. In this way, through training, the first to-be-trained model has the ability to accurately identify the emotion type, and the audio pre-training sub-model in the first to-be-trained model has the ability to accurately extract the emotion feature.

[0091] Taking Chinese as the source language, as an example, the preset emotion types include: surprise, anger, joy, shouting, disappointment, fear, calm, sadness, and excitement. In this way, by analyzing the emotion expression of Chinese, the preset emotion types are set to the above-mentioned 9 types, which can effectively improve the efficiency of emotion recognition and quickly and accurately extract the target emotion feature while meeting the emotion expression needs of Chinese audio of TV series. As another example, the preset emotion types include: surprise, joy, anger, joy, seriousness, shouting, embarrassment, disappointment, fear, whisper, calm, suspicion, sadness, excitement, chatting, and peace. In this way, by combining the characteristics of emotion expression in Chinese TV series, the preset emotion types are set to the above-mentioned 16 types, so as to best match and cover the delicate emotion expression of Chinese audio of TV series. The trained emotion recognition model can extract more delicate emotion features in the audio of TV series to represent the rich emotion states of the actors in the TV series.

[0092] Figure 3 A schematic diagram showing the distribution of each emotion in the emotion space according to an embodiment of the present disclosure is shown as follows: Figure 3As shown, the emotion space is a circle, where each sector of the circle represents a type of emotion, and the size of each sector represents the proportion of the corresponding type of emotion in the source language audio of the movie or TV show relative to all types of emotions in the source language audio of the movie or TV show. The light blue color represents "amazed", the blue-purple color represents "cute", the dark gray color represents "angry", the medium green color represents "cheerful", the deep yellow color represents "serious", the light green color represents "shouting", the olive green color represents "embarrassed", the light brown color represents "depressed", the light gray color represents "terrified", the medium blue color represents "whispering", the dark blue color represents "calm", the medium gray color represents "doubt", the medium yellow color represents "sad", the dark brown color represents "excited", the deep green color represents "chat", and the light yellow color represents "newscast". Compared with existing emotion spaces, this emotion space is further refined and includes more types of emotions. For example, Figure 3 As shown, the distribution of different emotion types in the emotion space is different, for example, "calm" occupies a large proportion, while other emotions such as "amazed" and "cute" occupy a smaller proportion. This distribution reflects that in this movie or TV show, the expression of calm emotion by the actors is more common, while the expression of other emotions is relatively less common.

[0093] In one possible implementation, the classification of the features extracted by the audio pre-training sub-model to obtain the corresponding predicted emotion includes: determining the numerical value of the three-dimensional evaluation index corresponding to the features extracted by the audio pre-training sub-model; and determining the predicted emotion based on the numerical value of the three-dimensional evaluation index; wherein the three-dimensional evaluation index includes: valence, arousal, and dominance.

[0094] Among them, valence (Valence): represents the positive or negative nature of emotion; arousal (Arousal): represents the active degree of emotion; dominance (Dominance): represents the control degree of emotion. By dividing emotions into these three-dimensional evaluation indexes, the complexity of emotions can be better understood and expressed. For example, a highly active and high control degree emotion can be excitement, and a low active and high control degree emotion can be calm.

[0095] Further, the three evaluation indexes of valence, arousal and dominance can be quantified into different numerical values, for example, quantified into numerical values in 1 to 9, and then the corresponding emotion type is determined according to different numerical value combinations. Among them, the numerical value corresponding to valence from small to large represents the change of emotion from negative to positive; the numerical value corresponding to arousal from small to large represents the change of emotion intensity from low to high; the numerical value corresponding to dominance from small to large represents the change of emotion control degree from low to high.

[0096] Exemplarily, the corresponding relationship between different numerical value combinations of the three evaluation indexes and emotions can be preset, for example, Valence = 1, Arousal = 1, Dominance = 1, which represents that the emotion is negative, low in activity and low in control, and the corresponding emotion is frustration; for another example, Valence = 9, Arousal = 9, Dominance = 1, which represents that the emotion is positive, high in activity and low in control, and the corresponding emotion is excitement; for another example, Valence = 5, Arousal = 1, Dominance = 9, which represents that the emotion is neutral (i.e. positive and negative are equivalent), low in activity and high in control, and the corresponding emotion is calm; for another example, Valence = 3, Arousal = 9, Dominance = 3, which represents that the emotion is negative, high in activity and moderate in control, and the corresponding emotion is suspicion. In this way, the three evaluation indexes are quantified into numerical values in 1 to 9, and the corresponding relationship between different numerical value combinations and emotions is set, so as to represent different emotions by different numerical value combinations of the three evaluation indexes, thereby the emotions can be expressed and described more finely.

[0097] It should be noted that the numerical value range of the three evaluation indexes can be flexibly configured as needed, wherein when the numerical value range of the three evaluation indexes is large and the preset emotion type is relatively small, for similar numerical value combinations, they can be set to correspond to the same emotion, for example, Valence = 3, Arousal = 9, Dominance = 3, Valence = 3, Arousal = 8, Dominance = 3 and Valence = 3, Arousal = 9, Dominance = 2 can all correspond to the emotion of suspicion.

[0098] In the training process, for any audio sample, an audio feature is extracted by a feature extraction sub-model in the first to-be-trained model, and then the audio feature is encoded by an audio pre-training sub-model in the first to-be-trained model, a value of a three-dimensional evaluation index corresponding to the encoded feature of the audio pre-training sub-model is calculated by an emotion classification layer in the first to-be-trained model, a predicted emotion corresponding to the audio sample is determined based on a preset corresponding relationship between different value combinations and emotions, and parameters in the first to-be-trained model are adjusted based on a preset emotion and an emotion label corresponding to the audio sample. In this way, through training, the first to-be-trained model has the ability to accurately identify the type of emotion, and the audio pre-training sub-model in the first to-be-trained model has the ability to accurately extract emotion features.

[0099] In a possible implementation, the method further includes: analyzing the characteristics of the source language, and determining the type of training data corresponding to the source language based on the analysis result; wherein the type of training data includes continuous emotion corpus or discrete emotion corpus.

[0100] The continuous emotion corpus means that each emotion is regarded as a point in a multi-dimensional space, and each emotion can be represented by a value combination of three-dimensional evaluation indexes (pleasure, arousal, dominance). The discrete emotion corpus means that emotions are divided into limited and explicit categories, and each emotion can be labeled with an adjective label, for example, happy, sad.

[0101] Exemplarily, the type of training data suitable for the source language can be determined by analyzing the characteristics of the source language, such as the expression manner of the source language, the cultural background of the source language, and the like. For example, for a source language of western culture, such as Spanish, English, and the like, the expression of basic emotions (such as anger, happiness, and sadness) is relatively direct, and therefore discrete emotion corpus can be selected as the type of training data corresponding to the source language. For a source language of eastern culture, such as Chinese, Thai, Korean, and the like, the expression of emotions is usually more delicate and complex, and therefore continuous emotion corpus can be selected as the type of training data corresponding to the source language. In this way, for the characteristics of different source languages, suitable training data is selected for training, so that the trained emotion recognition model has the ability to more accurately extract emotion features.

[0102] In a possible implementation, in the process of training the first to-be-trained model, for any audio sample, the audio sample is preprocessed; the result of the preprocessing is input into a feature extraction sub-model to extract audio features of the audio sample; the audio features of the audio data are input into an audio pre-training sub-model for feature encoding, and the encoded features are input into an emotion classification layer to obtain a predicted emotion corresponding to the audio sample; based on the predicted emotion corresponding to the audio sample and an emotion label corresponding to the audio sample, a loss function is determined, and the parameters of the first to-be-trained model are iteratively updated using a stochastic gradient descent algorithm until the value of the loss function converges or a preset number of iterations is reached to stop training; and based on the first to-be-trained model corresponding to the minimum loss value of the loss function, the feature extraction sub-model and the audio pre-training sub-model are obtained to obtain a trained emotion recognition model.

[0103] Illustratively, using the trained emotion recognition model, for any audio of a film or television drama, an emotion feature can be obtained for subsequent speech synthesis. In addition, in some scenarios, the first to-be-trained model after training can also output an emotion type, and then a reference audio corresponding to a different emotion type can be determined from a plurality of preselected audios. When performing speech synthesis, a user can select a reference audio corresponding to a corresponding emotion type according to needs to achieve personalized dubbing requirements.

[0104] Further, the training method can further include the following step 203 to train a speech generation model.

[0105] Step 203, embedding the emotion recognition model into a second to-be-trained model; fixing the parameters of the emotion recognition model, and training the second to-be-trained model to obtain the speech generation model.

[0106] Illustratively, the second to-be-trained model can be a model with speech generation capability, for example, can be a VITS model, the trained emotion recognition model can be embedded into an encoder, and the parameters of the emotion recognition model are fixed, and the second to-be-trained model is trained using training data corresponding to a target language to obtain a trained speech generation model corresponding to the target language. Wherein, the existing training method can be used to train the second to-be-trained model, which is not limited.

[0107] As an example, the training data of the source language can be used in advance to train a speech generation model corresponding to the source language, and then the trained speech generation model can be used to train the second to-be-trained model. Figure 3The first to-be-trained model is trained in the emotion space shown, and a trained emotion recognition model is obtained, which can accurately capture subtle emotion features in the source language audio of the film and television drama, and then the emotion recognition model is embedded into a text encoder in the VITS model, and the VITS model is trained using training data in the target language, so that the VITS model has the ability to synthesize speech with emotion, and finally a trained speech generation model in the target language is obtained. In this way, the trained speech generation model has the cross-lingual emotion transfer capability, realizes emotion mapping between different languages, enhances the emotion richness and appeal of the generated film and television dubbing, so as to be used to produce subtle and consistent film and television dubbing sound tracks with the emotions in the source language audio of the film and television drama.

[0108] In the embodiments of the present disclosure, through the above steps 201-202, a trained emotion recognition model corresponding to the source language can be obtained, and then the above step 203 can be performed to obtain a trained speech generation model corresponding to the target language.

[0109] For example, taking Chinese as the source language, Figure 4 A structural schematic diagram of a first to-be-trained model according to an embodiment of the present disclosure is shown, as shown in FIG. 1. Figure 4As shown, the first model to be trained may include: Wav2Vec2 Feature Extractor, a Chinese version of Hubert-large, a fully connected network (FC), and a softmax function. Wav2Vec2 Feature Extractor serves as the feature extraction sub-model, the Chinese version of Hubert-large serves as the audio pre-training sub-model, and the fully connected network and softmax function serve as the emotion classification layer. During training, the Wav2Vec2 Feature Extractor extracts audio features from the input audio, which can be represented as (samples, sr*s), where samples is the number of audio samples in the input audio, sr*s represents the dimension of each audio sample, sr represents the sampling rate, and s represents the size of the feature extraction window. The Wav2Vec2 Feature Extractor extracts audio features from the audio sample, which can be represented as (samples, 75000), where each sample has 75000 feature dimensions. Hubert-large further encodes these audio features. The last hidden layer (last_hidden_state) in Hubert-large outputs the encoded features (i.e., emotion features), which can be represented as (samples, 234, 1024). Then, a fully connected network transforms the feature dimensions, resulting in (samples, 9), meaning each sample has 9 feature dimensions. Finally, the Softmax function is used to calculate the probability distribution of the 9 emotion types and outputs the emotion type with the highest probability. After training, the Wav2Vec2 Feature Extractor and the Chinese version of Hubert-large are used as the trained Chinese emotion recognition model.

[0110] For example, when performing feature dimension transformation in a fully connected network, the feature dimension transformation can convert the emotion feature (B,E,H) into (B,emotions), where B (Batch Size) represents the number of samples in an input audio, E (Embedding Size) represents the dimension of the encoded feature, and H (Hidden Size) represents the dimension of the hidden layer in the Hubert model.

[0111] Figure 5 This diagram illustrates an emotion recognition method in film and television dubbing according to an embodiment of the present disclosure, as shown below. Figure 5As shown, the system can be divided into two main parts: a trained emotion recognition model corresponding to the source language and a trained speech generation model corresponding to the target language. Among them, the emotion recognition model includes: Feature Extractor (feature extractor, i.e. feature extraction sub-model), Emotion Encoder (emotion encoder, i.e. audio pre-training sub-model), and the speech generation model includes: Text Encoder (text encoder), VITS Alignment (VITS alignment module), Decoder (decoder). For the source language movie audio and the target language text to be dubbed, the source language movie audio can be input into the Feature Extractor to extract the audio features; the Emotion Encoder encodes the audio features to generate the encoded emotion features (Emotion Embedding), and inputs the emotion features into the Text Encoder. At the same time, the target language text to be dubbed can be input into the Text processor (text processor), and based on the preset Phoneme Dictionary (phoneme dictionary), the text to be dubbed is converted into Phoneme (phoneme) and input into the TextEncoder. Further, the Text Encoder extracts the phoneme features of the phonemes, and the emotion features are fused, and then the fused features are input into the VITS Alignment for alignment, and the Decoder decodes based on the aligned features to generate the target language dubbing. In this way, the speech generation model generates the dubbing with the corresponding emotion through the emotion features extracted by the emotion recognition model in the process of speech synthesis, thereby completing the dubbing of the movie across languages.

[0112] As shown in Figure 5 The encoded features (i.e. emotion features) of the Emotion Encoder can have two embedding representation methods: Wav-level embedding (waveform level embedding) for representing the emotion features of the entire source language movie audio; and Frame-level embedding (frame level embedding) for representing the emotion features of each frame of the source language movie audio.

[0113] Based on the same inventive concept of the above method embodiment, the embodiments of the present disclosure also provide an emotion recognition device in movie dubbing. The device can be used to execute the technical solutions described in the above method embodiments.

[0114] Figure 6 The structure diagram of an emotion recognition device in movie dubbing according to an embodiment of the present disclosure is shown in Figure 6As shown, the apparatus comprises: an acquisition module 601 configured to acquire a source language audio of a film / TV series and a target language text to be dubbed; wherein the audio of the film / TV series and the text to be dubbed have a corresponding relationship; an emotion recognition module 602 configured to process the audio of the film / TV series by using a trained emotion recognition model corresponding to the source language, and extract target emotion features; wherein the emotion recognition model is obtained by training a labeled training data based on an audio pre-training sub-model and a feature extraction sub-model; the feature extraction sub-model is configured to extract audio features, and the audio pre-training sub-model is configured to perform feature coding on the extracted audio features to generate emotion features; and a speech synthesis module 603 configured to perform speech synthesis on the text to be dubbed based on the target emotion features by using a trained speech generation model corresponding to the target language, and generate dubbing of the target language corresponding to the audio of the film / TV series.

[0115] In the embodiments of the present disclosure, for the cross-language dubbing scene of a film / TV series, the audio of the film / TV series in a source language and the text to be dubbed in a target language are acquired; wherein the audio of the film / TV series and the text to be dubbed have a corresponding relationship; the audio of the film / TV series is processed by using a trained emotion recognition model corresponding to the source language, and target emotion features are extracted; wherein the emotion recognition model is obtained by training a labeled training data based on an audio pre-training sub-model and a feature extraction sub-model; the feature extraction sub-model is configured to extract audio features, and the audio pre-training sub-model is configured to perform feature coding on the extracted audio features to generate emotion features; in this way, unlike the way of emotion recognition based on text or emotion recognition combined with text, image and other multi-modal data, in the embodiments of the present disclosure, the audio of the film / TV series in the source language, which is the single-modal data most accurately reflecting the emotions of actors, is taken as the processing object, and the emotion recognition model of the source language is trained based on the audio pre-training sub-model and the feature extraction sub-model to extract accurate target emotion features; at the same time, the emotion recognition model focuses on extracting emotion features, without performing other tasks such as recognizing speakers, avoiding the influence of redundant processing on feature extraction, thereby ensuring the accuracy of the extracted target emotion features. Furthermore, the text to be dubbed is subjected to speech synthesis based on the target emotion features by using a trained speech generation model corresponding to the target language, and dubbing of the target language corresponding to the audio of the film / TV series is generated; in this way, the emotions in the audio of the film / TV series in the source language are migrated to the corresponding dubbing synthesis in the target language, the generated dubbing retains the rich emotions of actors in the film / TV series in the source language, thereby realizing seamless conversion of the speech of the film / TV series across languages, making the cross-language dubbing of the film / TV series achieve the effect of "original flavor", and enabling audiences in different languages to hear the "original flavor" dialogue of the film / TV series.

[0116] In a possible implementation, the speech synthesis module 603 is further configured to: extract phoneme features of the text to be dubbed by using the speech generation model; fuse the phoneme features with the target emotion features to obtain fused features; and perform speech synthesis based on the fused features to generate the dubbing in the target language corresponding to the audio of the film or television series.

[0117] In a possible implementation, the labeled training data includes a plurality of audio samples in the source language and emotion labels corresponding to the audio samples; and the emotion recognition module 602 is further configured to: train the audio pre-training sub-model and the feature extraction sub-model in a self-supervised learning manner; and train a first to-be-trained model by using the plurality of audio samples and the emotion labels corresponding to the audio samples to obtain a trained emotion recognition model, where the first to-be-trained model includes the audio pre-training sub-model, the feature extraction sub-model, and an emotion classification layer, and the emotion classification layer is configured to classify the features extracted by the audio pre-training sub-model to obtain a corresponding predicted emotion.

[0118] In a possible implementation, the speech synthesis module 603 is further configured to: embed the emotion recognition model into a second to-be-trained model; fix the parameters of the emotion recognition model, and train the second to-be-trained model to obtain the speech generation model.

[0119] In a possible implementation, the classifying the features extracted by the audio pre-training sub-model to obtain a corresponding predicted emotion includes: calculating a preset emotion type probability distribution corresponding to the features extracted by the audio pre-training sub-model; and taking an emotion type corresponding to the highest probability as the predicted emotion; and / or, the classifying the features extracted by the audio pre-training sub-model to obtain a corresponding predicted emotion includes: determining a value of a three-dimensional evaluation index corresponding to the features extracted by the audio pre-training sub-model; and determining the predicted emotion based on the value of the three-dimensional evaluation index, where the three-dimensional evaluation index includes: pleasantness, arousal, and dominance.

[0120] In a possible implementation, the preset emotion types include: surprise, anger, joy, shout, disappointment, fear, calm, sadness, excitement; or, surprise, love, anger, joy, seriousness, shout, embarrassment, disappointment, fear, whisper, calm, suspicion, sadness, excitement, chat, and news broadcast.

[0121] In a possible implementation, the emotion recognition module 602 is further configured to: analyze the characteristics of the source language, and determine a type of training data corresponding to the source language based on an analysis result; and the type of training data includes: continuous emotion corpus or discrete emotion corpus.

[0122] In a possible implementation, the audio pre-training sub-model is a Hubert model, and the feature extraction sub-model is a Wav2vec2 model.

[0123] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be described here.

[0124] The embodiments of the present disclosure also provide an electronic device, including a memory, a processor and a computer program stored in the memory, the processor executes the computer program to implement the steps of the above method.

[0125] The embodiments of the present disclosure also provide a non-volatile computer readable storage medium having a computer program stored thereon, the computer program is executed by a processor to implement the steps of the above method.

[0126] The embodiments of the present disclosure also provide a computer program product, including a computer program or a non-volatile computer readable storage medium carrying a computer program, the computer program is executed by a processor to implement the steps of the above method.

[0127] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932, for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0128] The electronic device 1900 can also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input and output interface 1958 (I / O interface). The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0129] In example embodiments, a non-transitory computer-readable storage medium, e.g., memory 1932 including computer program instructions, is also provided that can be executed by processing component(s) 1922 of electronic device 1900 to perform the above-described methods.

[0130] A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor, or any other suitable combination of technologies to

[0131] Computer programs (or computer readable program instructions) described herein can be downloaded from a computer readable storage medium to respective computing / processing devices, or be downloaded from a network, e.g., the Internet, a local area network, a wide area network, and / or a wireless network, to external computer or external storage devices. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0132] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0133] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0134] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0135] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0136] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0137] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for emotion recognition in dubbing of a movie or a TV series, characterized in that, The method comprises: acquiring a source language audio of a film and television drama and a target language text to be dubbed; wherein the film and television drama audio and the text to be dubbed have a corresponding relationship; processing the film and television drama audio by using a trained emotion recognition model corresponding to the source language to extract target emotion features; wherein the emotion recognition model is trained based on an audio pre-training sub-model and a feature extraction sub-model by using labeled training data corresponding to the source language; the feature extraction sub-model is used to extract audio features, and the audio pre-training sub-model is used to perform feature coding on the extracted audio features to generate emotion features; the emotion features are abstract features representing emotion states; based on the target emotion features, performing speech synthesis on the text to be dubbed by using a trained speech generation model corresponding to the target language to generate dubbing of the film and television drama audio in the target language; the method further comprises: analyzing characteristics of the source language and determining a type of training data corresponding to the source language based on an analysis result; wherein the type of training data includes continuous emotion corpus or discrete emotion corpus; the characteristics of the source language include expression modes of the source language and cultural backgrounds of the source language. The labeled training data comprises a plurality of audio samples of the source language and emotion labels corresponding to each audio sample. The method further comprises: training the audio pre-training sub-model and the feature extraction sub-model by using a self-supervised learning method; training a first to-be-trained model by using the plurality of audio samples and the emotion labels corresponding to each audio sample to obtain a trained emotion recognition model; the first to-be-trained model comprises the audio pre-training sub-model, the feature extraction sub-model and an emotion classification layer; wherein the emotion classification layer is used to classify features extracted by the audio pre-training sub-model to obtain a corresponding predicted emotion.

2. The method of claim 1, wherein, The method comprises: embedding the emotion recognition model into a second to-be-trained model; fixing parameters of the emotion recognition model and training the second to-be-trained model to obtain the speech generation model. The method comprises:

3. The method of claim 1, wherein, calculating a preset emotion type probability distribution corresponding to the features extracted by the audio pre-training sub-model; and taking an emotion type corresponding to the highest probability as the predicted emotion; and / or, ​ 4. The method of claim 2, wherein, ​ ​ The classified features extracted by the audio pre-training sub-model are classified to obtain a corresponding predicted emotion, including: determining the numerical value of the three-dimensional evaluation index corresponding to the features extracted by the audio pre-training sub-model; and determining the predicted emotion based on the numerical value of the three-dimensional evaluation index; wherein the three-dimensional evaluation index includes: pleasantness, arousal, and dominance.

5. The method of claim 4, wherein, The preset emotion types include: surprise, anger, joy, shouting, depression, fear, calm, sadness, excitement; or, surprise, joy, anger, joy, seriousness, shouting, embarrassment, depression, fear, whisper, calm, suspicion, sadness, excitement, chat, peace.

6. The method of claim 1, wherein, The audio pre-training sub-model is a Hubert model, and the feature extraction sub-model is a Wav2vec2 model.

7. An emotion recognition device in dubbing of a movie or a drama, characterized by, The device includes: An acquisition module is configured to acquire a source language audio of a film or television drama and a target language text to be dubbed; wherein the film or television drama audio and the text to be dubbed have a corresponding relationship; An emotion recognition module is configured to use a trained emotion recognition model corresponding to the source language to process the film or television drama audio and extract target emotion features; wherein the emotion recognition model is trained based on an audio pre-training sub-model and a feature extraction sub-model using labeled training data corresponding to the source language; the feature extraction sub-model is configured to extract audio features, and the audio pre-training sub-model is configured to perform feature encoding on the extracted audio features to generate emotion features; the emotion features are abstract features representing emotional states; A speech synthesis module is configured to use a trained speech generation model corresponding to the target language to perform speech synthesis on the text to be dubbed based on the target emotion features to generate dubbing of the film or television drama audio in the target language; The speech synthesis module is specifically configured to extract phoneme features of the text to be dubbed through the speech generation model; fuse the phoneme features with the target emotion features to obtain fused features; and perform speech synthesis based on the fused features to generate dubbing of the film or television drama audio in the target language; The emotion recognition module is further configured to analyze the characteristics of the source language and determine the type of training data corresponding to the source language based on the analysis result; wherein the type of training data includes continuous emotion corpus or discrete emotion corpus; and the characteristics of the source language include the expression manner of the source language and the cultural background of the source language.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-7. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program or a non-transitory computer-readable storage medium bearing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images