Dubbing method and device, electronic equipment and storage medium

Abstract

The invention relates to a dubbing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target dubbing table book of a to-be-dubbed video; wherein the target dubbing table book comprises at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing table book is different from the language corresponding to the video to be dubbed; performing voice synthesis on the target dubbing table book to generate target audio data; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing table book; performing timbre conversion on the target audio data; and performing audio and picture synchronization processing on the audio data after tone conversion and the video to be dubbed to generate a dubbed video. According to the method and the device, the cross-language automatic dubbing of the video is realized through processing such as speech synthesis, tone conversion and audio and picture synchronization.

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing technology, and in particular to a dubbing method, device, electronic device and storage medium. Background Art

[0002] In order to achieve the dissemination and promotion of film and television works in countries with different languages, they need to be dubbed. However, manual dubbing is difficult to meet the dubbing needs of a large number of film and television works. Summary of the invention

[0003] In view of this, the present disclosure proposes a dubbing method, device, electronic device, storage medium and computer program product.

[0004] According to one aspect of the present disclosure, a dubbing method is provided, the method comprising:

[0005] Obtain a target dubbing script for the video to be dubbed; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed;

[0006] Generate target audio data by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script;

[0007] Performing timbre conversion on the target audio data;

[0008] The audio data after the timbre conversion is processed synchronously with the video to be dubbed to generate a dubbed video.

[0009] In a possible implementation, the generating target audio data by performing speech synthesis on the target dubbing script includes:

[0010] Extracting the rhythmic features and phoneme features of the target dubbing script;

[0011] The rhythmic features and phoneme features of the target dubbing script are fused to generate text features;

[0012] Speech synthesis is performed based on the text features to generate the target audio data.

[0013] In a possible implementation, the method further includes:

[0014] Extracting emotional features of the target dubbing script;

[0015] The performing speech synthesis based on the text features to generate the target audio data includes:

[0016] Speech synthesis is performed based on the emotional features of the target dubbing script and the text features to generate the target audio data.

[0017] In a possible implementation, the performing timbre conversion on the target audio data includes:

[0018] Determining a target timbre that matches at least one character corresponding to the target audio data from among a plurality of preset reference timbres; the language corresponding to the plurality of reference timbres is the same as the language corresponding to the target audio data;

[0019] Based on the target timbre matched by the at least one character, the target audio data is converted into timbre.

[0020] In a possible implementation, the extracting the phoneme features of the target dubbing script includes:

[0021] Based on a preset phoneme dictionary, generate a phoneme sequence corresponding to the target dubbing script;

[0022] Extracting the phoneme feature corresponding to each phoneme in the phoneme sequence;

[0023] The fusion processing of the rhythmic features and phoneme features of the target dubbing script to generate text features includes:

[0024] Processing the prosodic features of the target dubbing script to determine the prosodic features corresponding to each phoneme in the phoneme sequence;

[0025] The prosodic feature corresponding to each phoneme in the phoneme sequence is fused with the phoneme feature corresponding to each phoneme to generate the text feature.

[0026] In a possible implementation, the step of extracting the emotional features of the target dubbing script includes:

[0027] The emotional characteristics of the target dubbing script are extracted through an emotion recognition model; wherein the emotion recognition model is trained using a transfer learning algorithm based on a preset emotion space, and the preset emotion space includes at least: joy, sadness, surprise, anger, disappointment, pride, jealousy, rage, bittersweet, and happiness.

[0028] In a possible implementation, the performing timbre conversion on the target audio data based on the target timbre matched by the at least one character includes:

[0029] Acquire a preset timbre feature corresponding to a target timbre matched by the at least one character;

[0030] Extracting semantic features corresponding to the at least one role in the target audio data;

[0031] By fusing the preset timbre feature with the semantic feature, audio data after timbre conversion is generated.

[0032] In a possible implementation, the step of performing audio and video synchronization processing on the audio data after the timbre conversion and the video to be dubbed to generate the dubbed video includes:

[0033] Decomposing the audio data after the timbre conversion into a plurality of audio segments, wherein each audio segment corresponds to a line of dialogue;

[0034] Determine a processing method corresponding to each audio segment; wherein the processing method includes: one or more of: keeping unchanged, accelerating, combining, and shifting, wherein combining means combining two adjacent audio segments into one audio segment, wherein shifting means moving the start time and / or end time of the audio segment, and wherein accelerating means shortening the duration of the audio segment:

[0035] The multiple audio segments are adjusted based on the processing method of each audio segment, and the adjusted audio segments are aligned with the pictures in the video to be dubbed in time sequence to generate the dubbed video.

[0036] In a possible implementation, the method further includes:

[0037] Acquire source audio data of the video to be dubbed; the language corresponding to the source audio data is the same as the language corresponding to the video to be dubbed;

[0038] Performing speech recognition on the source audio data to generate a source dubbing script;

[0039] The source dubbing script is translated to generate the target dubbing script.

[0040] According to another aspect of the present disclosure, a dubbing device is provided, the device comprising:

[0041] The acquisition module is used to acquire a target dubbing script for the video to be dubbed; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed;

[0042] A speech synthesis module, used to generate target audio data by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script;

[0043] A timbre conversion module, used for performing timbre conversion on the target audio data;

[0044] The audio and video synchronization module is used to perform audio and video synchronization processing on the audio data after timbre conversion and the video to be dubbed to generate a dubbed video.

[0045] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0046] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0047] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0048] Through the above aspects of the present disclosure, the target dubbing script of the video to be dubbed is obtained; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed; the target audio data is generated by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script; the target audio data is converted to timbre; the audio data after timbre conversion is synchronized with the video to be dubbed to generate a dubbed video. In this way, by performing speech synthesis, timbre conversion, audio and video synchronization and other processing, cross-language automatic dubbing of videos is realized, without the need for expensive manual dubbing, which greatly reduces the cost of video dubbing and improves the production capacity of dubbing production; it provides convenience for a large number of film and television works and other content to go overseas, and the cost of going overseas is greatly reduced.

[0049] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0051] Figure 1 A flowchart of a dubbing method according to an embodiment of the present disclosure is shown;

[0052] Figure 2A schematic diagram showing the structure of a speech synthesis model according to an embodiment of the present disclosure is shown;

[0053] Figure 3 A schematic diagram showing timbre conversion according to an embodiment of the present disclosure;

[0054] Figure 4 A schematic diagram showing adjustment of time information of an audio segment according to an embodiment of the present disclosure;

[0055] Figure 5 A schematic diagram showing a hard synchronization strategy for audio and video according to an embodiment of the present disclosure;

[0056] Figure 6 A schematic diagram showing a soft synchronization strategy for audio and video according to an embodiment of the present disclosure is shown;

[0057] Figure 7 A flow chart of a method for speech synthesis according to an embodiment of the present disclosure is shown;

[0058] Figure 8 A schematic diagram showing a method of extracting prosodic features and phoneme features according to an embodiment of the present disclosure is shown;

[0059] Fig. 9 A schematic diagram showing a method of extracting phoneme features and prosodic features according to an embodiment of the present disclosure is shown;

[0060] Fig.10 A schematic diagram showing a preset emotion space according to an embodiment of the present disclosure;

[0061] Fig.11 A schematic diagram showing a method of determining subtitle text corresponding to audio according to an embodiment of the present disclosure;

[0062] Fig.12 A schematic diagram showing speech-text alignment according to an embodiment of the present disclosure is shown;

[0063] Fig.13 A structural diagram of a dubbing device according to an embodiment of the present disclosure is shown;

[0064] Fig.14 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0065] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0066] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present disclosure include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the phrases "exemplary", "in one embodiment", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0067] In the present disclosure, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or plural.

[0068] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0069] Film and television works are a powerful mass communication medium in an international context, and localized dubbing for film and television dramas is particularly important. With the massive amount of domestic film and television works that need to be exported in recent years, a TV series generally has dozens of episodes, and the workload of manual foreign language dubbing is huge. At the same time, the emotional expression and depth of understanding required for film and television works far exceed those of ordinary radio dramas, which requires dubbing actors to have a higher level of skills and in-depth interpretation of film and television dramas. However, qualified dubbing actors are relatively scarce, especially for dubbing in small languages, and it is particularly difficult to find suitable dubbing talents.

[0070] In order to solve the above-mentioned problems, the embodiments of the present disclosure propose a dubbing method (see below for detailed description). By performing speech synthesis (Text-to-Speech, TTS), voice conversion (Voice Conversion, VC), audio and video synchronization and other processing, it realizes cross-language automatic dubbing of videos, effectively reducing the manual workload, eliminating the need for expensive manual dubbing, and greatly reducing the cost of video dubbing.

[0071] Exemplarily, the method can be executed by an electronic device such as a terminal device or a server, wherein the terminal device can be a desktop terminal or a mobile terminal, for example, it can be various types of electronic devices such as a laptop, a tablet computer, a desktop computer, a smart phone, a smart speaker, a smart watch, a smart TV, a car terminal, etc. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms.

[0072] Figure 1 A flowchart of a dubbing method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the following steps are included:

[0073] Step 101, obtaining a target dubbing script for a video to be dubbed; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed.

[0074] The video to be dubbed can be a movie, TV series, animation, music video, TV program, sports game, documentary, self-media video, advertisement, etc. The language corresponding to the video to be dubbed can also be called the source language, that is, the language used by the characters in the process of shooting or making the video to be dubbed, such as Chinese, English, Japanese, Thai, etc.

[0075] The target dubbing script is a text prepared by the dubbing actor, which is used to guide the dubbing actor to express the lines of the designated role at the corresponding moment in the video to be dubbed. The language corresponding to the target dubbing script can also be called the target language, that is, the language used by the dubbing actor to dub the video to be dubbed. Exemplarily, the target dubbing script includes the target language lines corresponding to each role, and the time corresponding to any line in the target dubbing script can include: the starting time corresponding to the line. Exemplarily, the target dubbing script can also include information such as the age and gender of each role.

[0076] For example, in a scenario where a domestic TV series needs to be dubbed and then broadcast in Thailand, the video to be dubbed is a domestic TV series, the language corresponding to the video to be dubbed is Chinese, and the language corresponding to the target dubbing script is Thai. The target dubbing script contains the Thai lines of different characters in the domestic TV series, and the corresponding time of each Thai line.

[0077] Exemplarily, the target dubbing script can be generated from the source audio data of the video to be dubbed. Among them, the language corresponding to the source audio data of the video to be dubbed is the same as the language corresponding to the video to be dubbed. The source audio data can be the audio of the actor's speech recorded on the spot during the shooting of the video to be dubbed, or the audio recorded by the dubbing actor in the recording studio in the later stage of the video; Exemplarily, the source audio data can include: the audio clips of the source language lines of each character and the starting time and duration of each audio clip; The source audio data has a sound and picture synchronization effect with the picture in the video to be dubbed, and the source language lines in the source audio data are displayed in the picture of the video according to the starting time and duration of the corresponding audio clip, so as to form subtitles. In a possible implementation method, the source audio data of the video to be dubbed can be obtained; the source audio data is subjected to speech recognition to generate a source dubbing script; the source dubbing script is translated to generate the target dubbing script. Among them, the language corresponding to the source dubbing script of the video to be dubbed is the same as the language corresponding to the video to be dubbed; speech recognition and translation can be implemented by existing technologies, which are not limited. As an example, for different languages, speech recognition models corresponding to different languages ​​can be pre-trained, and the speech recognition model can identify the identity of the speaker in the audio and the content of what he said; then for the language corresponding to the video to be dubbed, select the trained language recognition model corresponding to the language, and input the source audio data of the video to be dubbed into the language recognition model. The language recognition model distinguishes the different roles involved in the source audio data through voiceprint recognition, and at the same time, recognizes the lines spoken by different roles through semantics, and records the time corresponding to each line, and matches the identified different roles with the identified lines, so as to obtain the source dubbing script; illustratively, the source dubbing script can also include information such as the age and gender of each role; then, the source dubbing script can be translated through a machine translation model, so as to translate the lines of each role from the source language to the target language, and generate a target dubbing script. In another possible implementation, the source dubbing script of the video to be dubbed can be obtained, and the source dubbing script can be translated to generate the target dubbing script.

[0078] Step 102: Generate target audio data by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script.

[0079] Exemplarily, speech synthesis models corresponding to different languages ​​can be pre-trained, and the trained speech synthesis models are used to synthesize speech in the same language based on text. For the language corresponding to the target audio data, a trained speech synthesis model corresponding to the language is selected, the target dubbing script is input into the speech synthesis model, and the target audio data is output. The target audio data is the audio of each character in the video to be dubbed speaking in the target language, and can include audio clips of the target language lines of each character and the starting time and duration of each audio clip.

[0080] As an example, the speech synthesis model may be a model built based on Variational Inference with adversarial learning for end-to-end Text-to-Speech (VITS). Figure 2 A schematic diagram of the structure of a speech synthesis model according to an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the trained speech synthesis model may include: a text encoder, a mapping layer, a stochastic duration predictor, a flow model, and a decoder; wherein the text encoder is used to encode the target dubbing script to obtain the text intermediate features corresponding to the target dubbing script; the mapping layer is used to map the text intermediate features to the latent space; the stochastic duration predictor is used to preset the pronunciation duration of each phoneme in the target dubbing script and map it to the latent space; the flow model is used to generate latent latent variables, and the decoder is used to decode the latent latent variables to generate target audio data corresponding to the target dubbing script. In this way, a high-quality speech that sounds natural and matches the target dubbing script can be generated by a trained speech synthesis model; wherein the training process of the speech synthesis model can be achieved through existing technologies and is not limited to this.

[0081] Step 103: Perform timbre conversion on the target audio data.

[0082] Among them, timbre can be used to describe the voice characteristics of different people speaking, which is usually related to the vibration mode of the vocal cords, the shape and size of the oral cavity, and the gender and age of the speaker. In this step, by converting the timbre of the target audio data, the timbre distinction of each character in the target audio data is improved, thereby improving the user's experience of watching and understanding the plot.

[0083] In one possible implementation, the timbre conversion of the target audio data includes: determining a target timbre that matches at least one character corresponding to the target audio data among a plurality of preset reference timbres; the language corresponding to the plurality of reference timbres is the same as the language corresponding to the target audio data; and based on the target timbre that matches the at least one character, converting the timbre of the target audio data.

[0084] Considering that a TV series often covers hundreds of characters, such as an octogenarian, a playful lolita, a domineering president, a gentle lady, etc., it is very challenging to accurately interpret these character traits when dubbing; illustratively, a timbre library containing different languages ​​can be pre-built based on existing corpus, wherein the timbre library of each language includes multiple reference timbres, as well as timbre features corresponding to each reference timbre, and character information corresponding to each reference timbre (such as age (or age range), gender, etc.). Exemplarily, for the same language, the timbre of actors of different genders and ages can be used as reference timbre, and the timbre features can be extracted based on the actor's audio data as the timbre features corresponding to the reference timbre, thereby pre-building a timbre library containing different dimensional features such as language, gender, age, and timbre; further, timbre matching can be performed in the timbre library for different characters involved in the target audio data. Exemplarily, determining a target timbre that matches at least one character corresponding to the target audio data from a plurality of preset reference timbres may include: selecting a timbre library corresponding to the target language; extracting the timbre features of each character from the source audio data of the video to be dubbed, and obtaining the age and gender of each character from the target dubbing script; for any character, comparing the extracted timbre features of the character with the timbre features of each reference timbre in the timbre library for similarity, and combining the similarity of age and gender, comprehensively selecting the reference timbre with the highest similarity as the target timbre that matches the character, for example, for For any reference timbre, the similarity between the timbre characteristics of the character and the timbre characteristics of the reference timbre, the similarity between the age of the character and the age corresponding to the reference timbre, and the similarity between the gender of the character and the gender corresponding to the reference timbre are calculated respectively, and the sum of the three similarities is calculated, so that the reference timbre with the highest sum of the three similarities is used as the target timbre; it can be understood that the character corresponding to the target timbre data is the same as the character corresponding to the source audio data, and the target timbre is the target timbre that matches the character corresponding to the target audio data, thereby achieving a high similarity match between the character and the target timbre in the target audio data. Furthermore, based on the target timbre that matches the character, the timbre conversion can be performed on the audio clip corresponding to the character in the target audio data.

[0085] Considering that in the same film and television work, as the plot develops, the age of the same character may change, and the timbre of the same character at different ages may be different, for example, the timbre of a teenager may be different from that of an adult. Therefore, according to the age of each character in the target dubbing script, the target timbre matching different ages of each character can be further obtained, and the audio clips corresponding to the character in the target timbre data can be divided into audio clips corresponding to different ages of the character according to the age, so that the timbre of the audio clips corresponding to the corresponding age can be converted based on the target timbre matching different ages of the character. Exemplarily, when obtaining the target timbre matching any age of the character, the audio data corresponding to the age of the character can be obtained from the source audio data of the video to be dubbed, and the timbre features can be extracted from the audio data, and then the similarity between the timbre features and the timbre features of each reference timbre, the similarity between the age and the age corresponding to each reference timbre, and the similarity between the gender of the character and the gender corresponding to each reference timbre are calculated, and the sum of the three similarities is calculated, so that the reference timbre with the highest sum of the three similarities is used as the target timbre matching the age of the character.

[0086] Exemplarily, the timbre conversion of the target audio data based on the target timbre matched by the at least one character may include: obtaining a preset timbre feature corresponding to the target timbre matched by the at least one character; extracting a semantic feature corresponding to the at least one character in the target audio data; and generating audio data after timbre conversion by fusing the preset timbre feature with the semantic feature. The semantic feature is a feature in the target audio data that is irrelevant to the timbre of the character and is related to content information, emotional information, rhythmic information, etc. In this way, by extracting the semantic feature, the timbre conversion is performed while keeping the semantic feature unchanged, thereby avoiding the influence of the change of the timbre feature of the target audio data on the semantic information, emotional information, rhythmic information, etc. of the target audio data; thus, the timbre of the character can be intelligently converted without losing the emotional expression characteristics and maintaining the same semantics, so as to accurately shape the sound characteristics of the character in the target audio data.

[0087] Figure 3 A schematic diagram of timbre conversion according to an embodiment of the present disclosure is shown as follows: Figure 3As shown, for any character, the target audio data contains the semantics and timbre corresponding to the character, the semantics of the character are encoded, and the semantic features are extracted; based on the target timbre matched by the character determined above, the preset timbre features corresponding to the target timbre are obtained from the preset timbre library; then, the preset timbre features and semantic features are decoded, and after fusion processing, the audio data after timbre conversion is generated. Since the semantic features do not change during the timbre conversion process, it is ensured that the audio data after timbre conversion is the same as the semantics corresponding to the character in the target audio data before conversion; at the same time, the timbre features change during the timbre conversion process, which ensures that the timbre of the character in the audio data after timbre conversion is the same as the target timbre, thereby realizing the timbre conversion in the semantic space of the same language.

[0088] In one possible implementation, a timbre conversion model can be pre-trained. The trained timbre conversion model has the ability to convert timbre in the semantic space of the same language. The source audio data and target audio data of the video to be dubbed are input into the timbre conversion model, and the audio data after timbre conversion is output. The language corresponding to the audio data after timbre conversion is still the target language, and the semantics corresponding to each character in the target audio data before timbre conversion are the same, but the timbre of each character has changed. As an example, the timbre conversion model may include a timbre extraction network, a timbre removal network, and a vocoder. The timbre extraction network can be used to extract the timbre features of each character in the target audio data from the source audio data of the video to be dubbed. Based on the similarity between the timbre features of each character and the timbre features of each reference timbre in the timbre library, the similarity between the age of each character and the age corresponding to each reference timbre, and the similarity between the gender of each character and the gender corresponding to each reference timbre, a target timbre matching each character is selected from the timbre library. By removing the timbre network, the semantic features of each character in the target audio data can be obtained, and the features in the target audio data that are unrelated to the timbre of each character and are related to content information, emotional information, and rhythmic information can be accurately obtained. Through the vocoder, fusion processing is performed based on the semantic features and the preset timbre features corresponding to the target timbre to generate audio data after timbre conversion, thereby converting the timbre of each character in the target audio data into the target timbre while keeping the original semantics unchanged. As another example, the timbre conversion model may include a timbre extraction network, a timbre removal network, and a vocoder. The timbre extraction network can be used to extract the timbre characteristics of each character at different ages from the source audio data of the video to be dubbed, and then based on the similarity between the timbre characteristics of each character at different ages and the timbre characteristics of each reference timbre in the timbre library, the similarity between different ages and the ages corresponding to each reference timbre, and the similarity between the gender of each character and the gender corresponding to each reference timbre, the target timbre matching the different ages of each character is selected from the timbre library; by removing the timbre network, the semantic features corresponding to the different ages of each character in the target audio data can be obtained, and by the vocoder, the semantic features of the same character at the same age and the preset timbre features corresponding to the matching target timbre are fused to generate audio data after the timbre conversion of the character at different ages.

[0089] Exemplarily, the audio data after timbre conversion includes audio segments of the target language lines of each character and the starting time and duration of each audio segment; it can be understood that the starting time and duration of each audio segment in the audio data after timbre conversion are consistent with the starting time and duration of the corresponding audio segment in the target audio data.

[0090] Step 104: Perform audio and video synchronization processing on the audio data after timbre conversion and the video to be dubbed to generate a dubbed video.

[0091] Considering the differences in pronunciation systems between the target language and the source language, there may be differences in the time information between the audio data after the timbre conversion and the source audio data of the video to be dubbed, wherein the time information may include the duration, starting time, etc. of each audio segment in the audio data; for example, for the same line of the same character, the duration of the audio segment corresponding to the target language line in the audio data after the timbre conversion may be different from the duration of the audio segment corresponding to the source language line in the source audio data. Therefore, the audio data after the timbre conversion can be aligned with the time information of the source audio data of the video to be dubbed through audio-visual synchronization processing, so that the sound and the picture are coordinated, the synchronization of the audio and video content is guaranteed, and the immersive experience of the audience when watching the dubbed video is ensured. Exemplarily, the time information in the audio data after the timbre conversion can be adjusted so that the sound is coordinated with the picture in the video to be dubbed, thereby completing the audio-visual synchronization processing.

[0092] Exemplarily, the audio data after timbre conversion can be decomposed into multiple audio segments, each audio segment corresponds to a line in the target language, and for each audio segment, the corresponding audio segment of the source language line in the source audio data is determined, and the duration of the audio segment is compared to see whether it exceeds the duration of the audio segment of the source language line. If it exceeds, it is further determined whether the exceeded duration is greater than a preset threshold, and if it is greater than the preset threshold, the audio segment is accelerated, for example, 1.3 times the speed, 1.5 times the speed, etc., to reduce the difference between the duration of the audio segment of the target language line and the duration of the corresponding audio segment of the source language line, wherein the specific value of the preset threshold can be set according to demand.

[0093] In a possible implementation, the audio data after timbre conversion is processed synchronously with the video to be dubbed to generate the dubbed video, including: decomposing the audio data after timbre conversion into multiple audio segments; wherein each audio segment corresponds to a line of dialogue; determining a processing method corresponding to each audio segment; wherein the processing method includes: one or more of: keeping unchanged, accelerating, aligning axes, and shifting axes; adjusting the multiple audio segments based on the processing method of each audio segment, and aligning the adjusted audio segments with the pictures in the video to be dubbed in time sequence to generate the dubbed video.

[0094] Among them, keeping unchanged means not adjusting the time information of the audio clip, and speeding up, aligning, and shifting all mean adjusting the time information of the audio clip. Figure 4 A schematic diagram of adjusting the time information of an audio segment according to an embodiment of the present disclosure is shown, wherein the merging axis indicates merging two adjacent audio segments into one audio segment, the shifting axis indicates moving the start time and / or end time of the audio segment, and the speeding up indicates shortening the duration of the audio segment.

[0095] Exemplarily, when determining the processing method corresponding to each audio segment, multiple audio segments can be comprehensively considered, and the processing method corresponding to each audio segment in the multiple audio segments can be adaptively determined according to the duration of the multiple audio segments and the duration of the audio segments in the corresponding source audio data; wherein, in the process of adaptive determination, the acceleration ratio, axis shift offset, axis combination number and other parameters corresponding to different processing method combinations can be accurately calculated, and finally the optimal processing method is determined, wherein the axis shift offset can include the sum of the time differences of the target start time after the shift of each audio segment relative to the original start time before the shift, and / or the sum of the time differences of the target end time after the shift relative to the original start time before the shift. The sum of the time differences of the original end times; as an example, different weight values ​​can be assigned to the acceleration ratio, the axis shift offset, and the number of axes together. Based on the weight values ​​and the acceleration ratio, the axis shift offset, and the number of axes together corresponding to each processing method combination, the size of the modification value is calculated, and the processing method combination with the smallest modification value is used as the optimal processing method combination, that is, the optimal processing method corresponding to multiple audio clips, so as to achieve accurate synchronization of audio and video and ensure the perfect fusion of video audio and video; among them, the weight values ​​assigned to the acceleration ratio, the axis shift offset, and the number of axes together can be set according to needs, and the value obtained by weighted summation of the assigned weight values ​​to the acceleration ratio, the axis shift offset, and the number of axes together can be used as the modification value.

[0096] Exemplarily, for any audio segment, if the audio segment is accelerated, it is determined that the processing method of the audio segment is one of keeping unchanged, aligning the axis, and shifting the axis, so as to avoid the problem of unclear pronunciation caused by re-acceleration.

[0097] Based on the number of roles to which the multiple audio clips belong when adaptively determining the processing method corresponding to each audio clip in the multiple audio clips, it can be divided into an audio-visual hard synchronization strategy and an audio-visual soft synchronization strategy.

[0098] As an example, the hard synchronization strategy for audio and video means adjusting multiple audio clips of a character's multiple consecutive target language lines, that is, taking a character's multiple consecutive target language lines as a whole, and comprehensively determining the corresponding processing method for each of the multiple audio clips of the character's multiple consecutive target language lines in an adaptive manner, so that the total length of the multiple audio clips of the target language lines processed in this way is consistent with the total length of the multiple audio clips of the corresponding source language lines. For example, taking Chinese as the source language and Thai as the target language, Figure 5 A schematic diagram showing a hard synchronization strategy for audio and video according to an embodiment of the present disclosure is shown as follows: Figure 5As shown, the Chinese subtitle timeline, the Chinese subtitle text, the Thai subtitle text, and the Thai audio queue can be matched according to time, wherein the Chinese subtitle timeline is the audio clips of each Chinese line in the Chinese audio data arranged in time sequence, the Chinese subtitle text is the Chinese lines arranged in the display time sequence in the video screen, the Thai subtitle text is the Thai lines arranged in the display time sequence in the video screen, and the Thai audio queue is the Thai audio data after timbre conversion obtained by the above processing, wherein, for any audio clip of the Chinese line, the length, The duration of the Chinese lines in the subtitle text of the Chinese language and the duration of the Thai lines in the Thai subtitle text are the same, but the duration of the audio clips corresponding to the Thai lines in the Thai audio queue may be different; the five audio clips of role 1 (speaker 1) in the Thai audio queue (i.e., 1, 2, 3, 4, and 5 in the figure) are adaptively determined to process each of the five audio clips; among them, the duration of audio clip 1 is less than the duration of the line "Auspicious Day" in the corresponding Chinese subtitles, and accordingly, there will be a blank in the audio queue; audio clips 3 and 4 have been accelerated. Through adaptive processing, the processing methods corresponding to the five audio clips are determined as follows: the processing method corresponding to audio clip 1 is to remain unchanged; the processing method corresponding to audio clip 2 is to speed up, that is, character 1 speeds up the line "Miss" in Thai; the processing method corresponding to audio clip 3 is to shift the axis forward and backward in both directions, that is, character 1 slows down the line "It's time to get married" in Thai; the processing methods corresponding to audio clips 4 and 5 are both 4 and 5 combined, that is, character 1 connects the two lines "Miss" and "You should get married" into one line in Thai, thereby reducing the time gap between the two lines. The audio clips are processed according to the processing methods corresponding to each audio clip to generate the vocal track shown in the figure, which is the audio data after the audio and video synchronization processing; among them, the time length reduced by the combination of 4 and 5 is the same as the time length of the backward axis shift of audio clip 3, and the time length reduced by the acceleration of audio clip 2 is the same as the time length of the forward axis shift of audio clip 3. In the vocal track, the total duration of audio clips 1-5 in which character 1 speaks the lines in Thai is consistent with the total duration of the corresponding audio clips in the Chinese subtitle timeline, thus achieving timing alignment.

[0099] As another example, the audio-visual soft synchronization strategy indicates that multiple audio clips corresponding to adjacent target language lines of different characters are adjusted together, that is, multiple consecutive target language lines of multiple characters are taken as a whole, and the processing method corresponding to each of the multiple audio clips corresponding to the multiple target language lines of these multiple characters is comprehensively determined in an adaptive manner, so that the total duration of the multiple audio clips processed according to the processing method is consistent with the total duration of the multiple audio clips obtained by the corresponding source language lines. Figure 6A schematic diagram showing a soft audio-visual synchronization strategy according to an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, for the four audio clips of role 1 (speaker 1) (i.e., 1, 2, 3, and 4 in the figure) and one audio clip of role 2 (speaker 2) (i.e., 5 in the figure) in the Thai audio queue, the processing methods corresponding to the five audio clips are adaptively determined; wherein, the duration of audio clip 1 is less than the duration of the line "Auspicious Day" in the corresponding Chinese subtitles, and accordingly, there will be a blank period in the audio queue; audio clip 5 has been accelerated, and audio clip 4 and audio clip 5 belong to two characters, and there is a certain overlap in time, that is, there is a situation in the Thai audio queue where role 1 and role 2 speak at the same time; the processing methods corresponding to the five audio clips are determined through adaptive processing: the processing method corresponding to audio clip 1 is to keep unchanged; the processing method corresponding to audio clips 2 and 3 is the combination of 2 and 3, that is, character 1 connects the two lines "Miss" and "It's time to get married" into one line in Thai, thereby reducing the time gap between the two lines; the processing method corresponding to audio clip 4 is acceleration, that is, character 1 speeds up the line "Miss" in Thai; the processing method corresponding to audio clip 5 is forward axis shift, that is, character 2 speaks the line "You should get married" in Thai in advance; the audio clips are processed according to the processing methods corresponding to each audio clip to generate the vocal track shown in the figure, which is the audio data processed after the synchronization of sound and picture; among them, the forward length of audio clip 5 occupies a part of the blank time in addition to the length reduced by the acceleration of audio clip 4 and the length reduced by the combination of 2 and 3. In the voice track of this character, after character 1 says "Miss" in Thai, character 2 says "It's time for you to get on the sedan" in Thai, thus avoiding the overlap of speaking time of different characters. At the same time, the sum of the duration of character 1 speaking audio clips 1-4 and the duration of character 2 speaking audio clip 5 is consistent with the total duration of the corresponding audio clips on the Chinese subtitle timeline, achieving timing alignment.

[0100] It is understandable that the appropriate audio and video synchronization strategy can be flexibly selected according to needs. For example, if you want to maintain the relative independence of speech between different roles, you can choose the audio and video hard synchronization strategy. If you want to keep the original rhythm of the speech content as much as possible, you can use the audio and video soft synchronization strategy.

[0101] In the disclosed embodiment, a target dubbing script of a video to be dubbed is obtained; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed; target audio data is generated by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script; the target audio data is converted into timbre; the audio data after timbre conversion is synchronized with the video to be dubbed to generate a dubbed video. In this way, by performing speech synthesis, timbre conversion, audio and video synchronization and other processing, cross-language automated dubbing of videos is achieved, without the need for expensive manual dubbing, greatly reducing the cost of video dubbing and increasing the production capacity of dubbing production; it provides convenience for a large number of film and television works and other content to be exported overseas, and the cost of exporting overseas is greatly reduced.

[0102] The following is an exemplary description of a possible implementation method of generating target audio data by performing speech synthesis on the target dubbing script in the above step 102.

[0103] Figure 7 A flow chart of a method for speech synthesis according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, the following steps are included:

[0104] Step 701: extract the prosody embedding and phoneme embedding of the target dubbing script.

[0105] Among them, phoneme refers to the smallest speech unit divided according to the natural properties of speech, and the phoneme features are used to represent pronunciation information; prosody refers to the sound pattern in language, including stress, intonation, pauses, etc., and prosodic features are used to represent information such as the intonation, rhythm and intensity of speech.

[0106] Exemplarily, the above-mentioned speech synthesis model can be used to extract the prosodic features and phoneme features of the target dubbing script. For example, a trained prosodic feature extraction model and a phoneme feature extraction model can be configured in the speech synthesis model. As an example, the phoneme feature extraction model can be a grapheme-to-phoneme (G2P) model, and the prosodic feature extraction model can be a BERT Bidirectional Encoder Representations from Transformers model, wherein BERT is a pre-trained model of the Transformer architecture, which exhibits extremely high performance by pre-training on large-scale unlabeled text and then fine-tuning on specific tasks. For example, taking the speech synthesis model as a VITS-based model, it can be used in Figure 2 The BERT model and the G2P model are configured in the text encoder of VITS shown in the figure. Figure 8 A schematic diagram of extracting prosodic features and phoneme features according to an embodiment of the present disclosure is shown as follows: Figure 8 As shown in the figure, the BERT model is used to extract prosodic features, and the G2P model is used to extract phoneme features.

[0107] In a possible implementation, the extraction of the phoneme features of the target dubbing script includes: generating a phoneme sequence corresponding to the target dubbing script based on a preset phoneme dictionary; extracting the phoneme features corresponding to each phoneme in the phoneme sequence. The phoneme dictionary contains the correspondence between keywords and phonemes in the target dubbing script, so that the target dubbing script is divided into phonemes based on the preset phoneme dictionary to generate a phoneme sequence; it is understandable that for the same target dubbing script, the phoneme sequences divided by different phoneme dictionaries may be different. Exemplarily, phoneme dictionaries of different languages ​​are constructed in advance according to the characteristics of different languages, and the preset phoneme dictionary may include more fine-grained phonemes, so that the pronunciation of the characters in the generated target audio data is more natural; for example, for the same line, more phonemes can be divided according to the preset phoneme dictionary.

[0108] Fig. 9 A schematic diagram of extracting phoneme features and prosodic features according to an embodiment of the present disclosure is shown as follows: Fig. 9 As shown, the G2P model is configured with a word segmenter, through which each line in the target dubbing script is divided into a phoneme sequence based on a preset Thai phoneme dictionary. The phoneme sequences corresponding to each line are concatenated to form the phoneme sequence corresponding to the target dubbing script, and the G2P model is pre-trained to enable it to have the ability to extract the phoneme features of each phoneme; Fig. 9 As shown, for the Thai lines " ", the relevant technology is divided into the word " The phoneme of " is "sawatdi". In the embodiment of the present disclosure, the G2P model is configured with two word segmenters, which further segment the line based on different phoneme dictionaries to obtain more fine-grained phonemes; wherein, through word segmenter 1, based on the corresponding phoneme dictionary, " "The participle is " "," "," "Three key words, among which, " "The corresponding phoneme is "sawa", The corresponding phoneme is " ”, “ "The corresponding phoneme is "di". Through word segmenter 2, based on another phoneme dictionary, " "The participle is " "and" "Two key words; among them, " "The corresponding phoneme is "sawat", ” The corresponding phoneme is “di”.

[0109] In a possible implementation, the method further includes: extracting emotional features of the target dubbing script.

[0110] Among them, emotional characteristics are used to characterize emotion types (such as happy, sad, high-pitched, low-pitched, etc.), as well as attitude levels (such as affirmation, negation, praise, criticism, irony, etc.).

[0111] Exemplarily, a pre-trained emotion recognition model can be configured in the speech synthesis model to extract the emotional features of the target dubbing script. Taking the speech synthesis model as a VITS-based model as an example, the emotion recognition model can be configured in the text encoder in the VITS model. As an example, the emotion recognition model can be a Hubert-based language emotion recognition model (HuBert with speech emotion recognition, Hubert-SER), where Hubert is a self-supervised speech representation learning model based on BERT.

[0112] As an example, the extracting of emotional features of the target dubbing script includes: extracting emotional features of the target dubbing script through an emotion recognition model; wherein the emotion recognition model is trained using a transfer learning algorithm based on a preset emotional space, and the preset emotional space includes at least: joy, sadness, surprise, anger, disappointment, pride, jealousy, rage, bittersweet, and happiness.

[0113] Fig.10A schematic diagram of a preset emotion space according to an embodiment of the present disclosure is shown as follows: Fig.10 As shown, the left side is the existing emotional space, and the right side is the preset emotional space used in the embodiment of the present disclosure; wherein, in the preset emotional space used in the embodiment of the present disclosure, yellow represents "Joy", blue represents "Sadness", light blue represents "Surprise", red represents "Anger", dark blue represents "Disappointment", orange represents "Pride", purple represents "Envy", dark purple represents "Outrage", olive green represents "Bittersweetness", and green represents "Delight"; relative to the existing emotional space, it is further refined so as to better extract the delicate emotional features of the target dubbing script.

[0114] Among them, the transfer learning algorithm is a deep learning technique that allows a model to be trained on one task and then fine-tuned on another related task. This method can improve the performance of the model on new tasks while reducing training time and data requirements. For example, a sample dubbing script in the source language can be used in advance. Fig.10 In the preset emotional space shown in the figure, the emotion recognition model is trained so that the emotion recognition model can accurately capture the delicate emotions in the source language, and then the dubbing script samples in the target language are collected. Fig.10 The emotion recognition model is further trained in the preset emotion space shown, thereby finally obtaining a trained emotion recognition model. In this way, the trained emotion recognition model has the ability to recognize and transfer emotions across languages, realizes emotion mapping between different languages, and thus enhances the emotional richness and appeal of the synthesized target audio data.

[0115] Step 702: Fusing the rhythmic features and phoneme features of the target dubbing script to generate text features.

[0116] For example, as mentioned above Figure 8 As shown in Figure 1, the prosodic features output by the BERT model can be fused with the phoneme features output by the G2P model to generate text features.

[0117] In a possible implementation, the rhythmic features and phoneme features of the target dubbing script are fused to generate text features, including: processing the rhythmic features of the target dubbing script to determine the rhythmic features corresponding to each phoneme in the phoneme sequence; fusing the rhythmic features corresponding to each phoneme in the phoneme sequence with the phoneme features corresponding to each phoneme to generate the text features.

[0118] Since the phonemes are further subdivided by using a preset phoneme dictionary, the trained phoneme feature extraction model configured in the speech synthesis model has the ability to extract the phoneme features of each subdivided phoneme, and the trained rhythmic feature extraction model has the ability to extract the rhythmic features of the target dubbing script. The rhythmic features of the target dubbing script include the phoneme features of all phonemes corresponding to the target dubbing script; therefore, the rhythmic features of the target dubbing script can be processed to determine the rhythmic features corresponding to each phoneme in the phoneme sequence, so as to align the feature dimensions of the rhythmic features and the phoneme features, so as to facilitate the fusion of the rhythmic features corresponding to each phoneme in the phoneme sequence with the phoneme features corresponding to each phoneme.

[0119] For example, Fig. 9 As shown, the dimensional alignment of prosodic features and phoneme features can be achieved through three alignment methods: Alignment method 1: Average and broadcast. This method averages the prosodic features of each dimension and propagates the average value to each phoneme in the entire phoneme sequence. The prosodic features corresponding to each phoneme obtained are the same. Alignment method 2: Greedy repeat. This method evenly distributes the prosodic features to each phoneme according to the dimension and the order of the phonemes in the phoneme sequence. For the prosodic features of the remaining dimensions, the order of the phonemes is still followed, and the prosodic features of the remaining dimensions are preferentially allocated to the phonemes with the highest order. Fig. 9 As shown in the figure, the dimension of the prosodic feature is 7, and the number of phonemes is 3. The prosodic feature dimension corresponding to each phoneme is 2, and the prosodic feature of the remaining dimension is assigned to the first phoneme. Alignment method 3: Accurate alignment. This method assigns the prosodic feature to each phoneme according to the length of each phoneme. As shown in the figure, the length of the phoneme "sawat" is 5, the length of "di" is 2, and the dimension of the prosodic feature is 7. The prosodic features of the first 5 dimensions can be assigned to the phoneme "sawat", and the prosodic features of the last 2 dimensions can be assigned to the phoneme "di".

[0120] Step 703: Perform speech synthesis based on the text features to generate the target audio data.

[0121] For example, the above Figure 2 The speech synthesis model shown, based on the text features generated by the text encoder, performs speech synthesis based on the text features to generate target audio data.

[0122] Exemplarily, speech synthesis is performed based on text features that are a fusion of prosodic features and phoneme features, and the generated target audio data contains prosodic information and phoneme information, which effectively improves the pronunciation of each character in the target audio data.

[0123] In one possible implementation, the generating the target audio data by performing speech synthesis based on the text features includes: generating the target audio data by performing speech synthesis based on the emotional features of the target dubbing script and the text features. In this way, the generated target audio data contains rhythmic information, phoneme information and emotional information, realizing cross-language joint emotional transfer speech synthesis (Emotinal Text-to-Speech, Emotional-TTS), which greatly improves the realism and rhythmic expression of the target language of each character. For example, taking the speech synthesis model as a VITS-based model, it can be Figure 2 The text encoder of VITS shown in the figure is configured with a Hubert-SER model, a BERT model and a G2P model. The BERT model is used to extract rhythmic features, the G2P model is used to extract phoneme features, and the Hubert-SER model is used to extract emotional features. The text encoder fuses the rhythmic features with the phoneme features to generate text features, and fuses the text features with the emotional features to generate text intermediate features, and then passes the text intermediate features to the mapping layer. The mapping layer maps the text intermediate features to the latent space, and the flow model generates latent hidden variables in the latent space. The decoder decodes the latent hidden variables to generate target audio data.

[0124] In the disclosed embodiments, the rhythmic features and phoneme features of the target dubbing script are extracted; the rhythmic features and phoneme features of the target dubbing script are fused to generate text features; speech synthesis is performed based on the text features to generate the target audio data; thus, pronunciation is improved by extracting the rhythmic features, thereby achieving film and television-level dubbing synthesis; as an example, the emotional features of the target dubbing script are extracted, and speech synthesis is performed based on the emotional features of the target dubbing script and the text features to generate the target audio data, thereby achieving speech synthesis with localized rhythm and cross-language emotion transfer, and the generated target audio data is full of rich emotions.

[0125] Furthermore, the target audio data obtained is further processed with timbre conversion and audio-visual synchronization, thereby realizing the dubbing process of speech synthesis, emotion transfer, timbre matching, and audio-visual synchronization, and automatically producing dubbing audio with multiple timbres and emotions.

[0126] For example, the dubbed video can be further processed for sound channel optimization as required. For example, the audio data in the dubbed video can be finely analyzed and processed using audio separation and restoration technology to restore or enhance specific emotional sounds, including but not limited to crying and laughing, to achieve higher authenticity.

[0127] In addition, the disclosed embodiments make full use of the extensive dubbing resources within the platform and provide a method for generating training data, which enables rapid acquisition of high-quality training data through audio preprocessing, speaker separation, and text annotation. This not only enhances the quality and applicability of the training data, but also expands the diversity of the corpus, covering multiple language fields such as Thai, Vietnamese, Spanish, and English, so that the models involved in the above dubbing method can be pre-trained using these training data.

[0128] For example, when generating training data, audio and text alignment (audio-text alignment) can be performed. Fig.11 A schematic diagram showing a method of determining subtitle text corresponding to audio according to an embodiment of the present disclosure is shown as follows: Fig.11 As shown, for any audio segment (Audio), where the audio can be a dialogue audio in a film or TV work, a sound track in a video, etc.; automatic speech recognition is performed on the audio to generate a speech recognition text (ASR Text), and then a similarity calculation is performed between the speech recognition text and the subtitle text (ASR Text), where the subtitle text can be pre-produced based on all the film and TV works where the audio is located; thereby finding the subtitle text corresponding to the audio segment.

[0129] For example, when generating training data, speech-text alignment can be performed. Fig.12 A schematic diagram of performing speech-to-text alignment according to an embodiment of the present disclosure is shown as follows: Fig.12 As shown, for any audio segment, speech-text alignment processing is performed based on the audio and the subtitle text corresponding to the audio. For example, alignment processing can be performed through Montreal Forced Aligner (MFA) to generate speech-text alignment data.

[0130] The “fusion” between the features involved in the embodiments of the present disclosure can be achieved by selecting a specific fusion method in the relevant technology as needed, such as splicing, summation, multiplication, etc., and the present disclosure does not impose any restrictions on this.

[0131] Based on the same inventive concept of the above method embodiment, an embodiment of the present disclosure further provides a dubbing device, which can be used to execute the technical solution described in the above method embodiment.

[0132] Fig.13 A structural diagram of a dubbing device according to an embodiment of the present disclosure is shown as follows: Fig.13As shown, the device includes: an acquisition module 1301, used to acquire a target dubbing script of a video to be dubbed; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed; a speech synthesis module 1302, used to generate target audio data by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script; a timbre conversion module 1303, used to perform timbre conversion on the target audio data; and an audio-visual synchronization module 1304, used to perform audio-visual synchronization processing on the audio data after timbre conversion and the video to be dubbed to generate a dubbed video.

[0133] In the disclosed embodiment, a target dubbing script of a video to be dubbed is obtained; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed; target audio data is generated by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script; the target audio data is converted into timbre; the audio data after timbre conversion is synchronized with the video to be dubbed to generate a dubbed video. In this way, by performing speech synthesis, timbre conversion, audio and video synchronization and other processing, cross-language automated dubbing of videos is achieved, without the need for expensive manual dubbing, greatly reducing the cost of video dubbing and increasing the production capacity of dubbing production; it provides convenience for a large number of film and television works and other content to be exported overseas, and the cost of exporting overseas is greatly reduced.

[0134] In a possible implementation, the speech synthesis module 1302 is also used to: extract the rhythmic features and phonemic features of the target dubbing script; fuse the rhythmic features and phonemic features of the target dubbing script to generate text features; perform speech synthesis based on the text features to generate the target audio data.

[0135] In a possible implementation, the speech synthesis module 1302 is further used to: extract the emotional features of the target dubbing script; perform speech synthesis based on the emotional features of the target dubbing script and the text features to generate the target audio data.

[0136] In a possible implementation, the timbre conversion module 1303 is further used to: determine, from a plurality of preset reference timbres, a target timbre that matches at least one character corresponding to the target audio data; the language corresponding to the plurality of reference timbres is the same as the language corresponding to the target audio data; and perform timbre conversion on the target audio data based on the target timbre that matches the at least one character.

[0137] In a possible implementation, the speech synthesis module 1302 is also used to: generate a phoneme sequence corresponding to the target dubbing script based on a preset phoneme dictionary; extract the phoneme features corresponding to each phoneme in the phoneme sequence; process the rhythmic features of the target dubbing script to determine the rhythmic features corresponding to each phoneme in the phoneme sequence; and fuse the rhythmic features corresponding to each phoneme in the phoneme sequence with the phoneme features corresponding to each phoneme to generate the text features.

[0138] In one possible implementation, the speech synthesis module 1302 is also used to: extract emotional characteristics of the target dubbing script through an emotion recognition model; wherein the emotion recognition model is trained using a transfer learning algorithm based on a preset emotional space, and the preset emotional space includes at least: joy, sadness, surprise, anger, disappointment, pride, jealousy, rage, bittersweet, and happiness.

[0139] In a possible implementation, the timbre conversion module 1303 is also used to: obtain a preset timbre feature corresponding to the target timbre matching the at least one role; extract a semantic feature corresponding to the at least one role in the target audio data; and generate audio data after timbre conversion by fusing the preset timbre feature with the semantic feature.

[0140] In a possible implementation, the audio-visual synchronization module 1304 is further used to: decompose the audio data after timbre conversion into multiple audio segments; wherein each audio segment corresponds to a line of dialogue; determine the processing method corresponding to each audio segment; wherein the processing method includes: one or more of: keeping unchanged, accelerating, merging, and shifting, wherein the merging means merging two adjacent audio segments into one audio segment, the shifting means moving the start time and / or end time of the audio segment, and the acceleration means shortening the duration of the audio segment: adjusting the multiple audio segments based on the processing method of each audio segment, and aligning the adjusted audio segments with the pictures in the video to be dubbed in time sequence to generate the dubbed video.

[0141] In a possible implementation, the acquisition module 1301 is also used to: acquire source audio data of the video to be dubbed; the language corresponding to the source audio data is the same as the language corresponding to the video to be dubbed; perform speech recognition on the source audio data to generate a source dubbing script; translate the source dubbing script to generate the target dubbing script.

[0142] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0143] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0144] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0145] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0146] Fig.14 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.14 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0147] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0148] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0149] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0150] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0151] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0152] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0153] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0154] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0155] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0156] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0157] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A dubbing method, characterized in that: The method comprises: Obtain a target dubbing script for the video to be dubbed; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed; Generate target audio data by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script; Performing timbre conversion on the target audio data; The audio data after the timbre conversion is processed synchronously with the video to be dubbed to generate a dubbed video.

2. The method according to claim 1, characterized in that The generating target audio data by performing speech synthesis on the target dubbing script includes: Extracting the rhythmic features and phoneme features of the target dubbing script; The rhythmic features and phoneme features of the target dubbing script are fused to generate text features; Speech synthesis is performed based on the text features to generate the target audio data.

3. The method according to claim 2, characterized in that The method further comprises: Extracting emotional features of the target dubbing script; The performing speech synthesis based on the text features to generate the target audio data includes: Speech synthesis is performed based on the emotional features of the target dubbing script and the text features to generate the target audio data.

4. The method according to claim 1, characterized in that: The step of converting the target audio data into timbre includes: Determining a target timbre that matches at least one character corresponding to the target audio data from among a plurality of preset reference timbres; the language corresponding to the plurality of reference timbres is the same as the language corresponding to the target audio data; Based on the target timbre matched by the at least one character, the target audio data is converted into timbre.

5. The method according to claim 2, characterized in that: The step of extracting the phoneme features of the target dubbing script includes: Based on a preset phoneme dictionary, generate a phoneme sequence corresponding to the target dubbing script; Extracting the phoneme feature corresponding to each phoneme in the phoneme sequence; The fusion processing of the rhythmic features and phoneme features of the target dubbing script to generate text features includes: Processing the prosodic features of the target dubbing script to determine the prosodic features corresponding to each phoneme in the phoneme sequence; The prosodic feature corresponding to each phoneme in the phoneme sequence is fused with the phoneme feature corresponding to each phoneme to generate the text feature.

6. The method according to claim 3, characterized in that: The step of extracting the emotional features of the target dubbing script includes: The emotional characteristics of the target dubbing script are extracted through an emotion recognition model; wherein the emotion recognition model is trained using a transfer learning algorithm based on a preset emotion space, and the preset emotion space includes at least: joy, sadness, surprise, anger, disappointment, pride, jealousy, rage, bittersweet, and happiness.

7. The method according to claim 4, characterized in that The step of converting the target audio data into a timbre based on the target timbre matched by the at least one character comprises: Acquire a preset timbre feature corresponding to a target timbre matched by the at least one character; Extracting semantic features corresponding to the at least one role in the target audio data; By fusing the preset timbre feature with the semantic feature, audio data after timbre conversion is generated.

8. The method according to claim 1, characterized in that The step of performing audio and video synchronization processing on the audio data after the timbre conversion and the video to be dubbed to generate the dubbed video includes: Decomposing the audio data after the timbre conversion into a plurality of audio segments, wherein each audio segment corresponds to a line of dialogue; Determine a processing method corresponding to each audio segment; wherein the processing method includes: one or more of: keeping unchanged, accelerating, combining, and shifting, wherein combining means combining two adjacent audio segments into one audio segment, wherein shifting means moving the start time and / or end time of the audio segment, and wherein accelerating means shortening the duration of the audio segment: The multiple audio segments are adjusted based on the processing method of each audio segment, and the adjusted audio segments are aligned with the pictures in the video to be dubbed in time sequence to generate the dubbed video.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire source audio data of the video to be dubbed; the language corresponding to the source audio data is the same as the language corresponding to the video to be dubbed; Performing speech recognition on the source audio data to generate a source dubbing script; The source dubbing script is translated to generate the target dubbing script.

10. A dubbing device, characterized in that: The device comprises: The acquisition module is used to acquire a target dubbing script for the video to be dubbed; wherein the target dubbing script includes at least one line and the time and role corresponding to each line; the language corresponding to the target dubbing script is different from the language corresponding to the video to be dubbed; A speech synthesis module, used to generate target audio data by performing speech synthesis on the target dubbing script; wherein the language corresponding to the target audio data is the same as the language corresponding to the target dubbing script; A timbre conversion module, used for performing timbre conversion on the target audio data; The audio and video synchronization module is used to perform audio and video synchronization processing on the audio data after timbre conversion and the video to be dubbed to generate a dubbed video.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images