Dubbing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a dubbing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. The specific implementation scheme is as follows: obtaining a target line text and an original line audio; performing acoustic feature extraction on the original line audio to obtain at least one acoustic feature; based on the at least one acoustic feature, generating natural language description information about the acoustic feature of the original line audio; and based on the natural language description information, performing audio synthesis processing on the target line text to obtain a dubbing audio which has acoustic characteristics of the original line audio and belongs to the target language. Visibly, according to the scheme of the application, the dubbing efficiency for the target line text can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a dubbing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Video platforms have a need to re-dub certain videos. For example, if the audio in an overseas video is in English, it needs to be re-dubbed into Mandarin.

[0003] In related technologies, voice actors are required to manually dub the videos to be dubbed. Specifically, the voice actors dub according to the text of the dialogue in the video to be dubbed (i.e., the text of the dialogue in the video to be dubbed). This method of dubbing is inefficient. Summary of the Invention

[0004] The purpose of this application is to provide a dubbing method, apparatus, electronic device, and storage medium to improve the efficiency of dubbing target dialogue text. The specific technical solution is as follows:

[0005] In a first aspect of this application, a dubbing method is provided, the method comprising:

[0006] Obtain the target dialogue text and the original dialogue audio; wherein the target dialogue text and the original dialogue audio represent the same dialogue semantics, and the language of the target dialogue text is the target language to be dubbed;

[0007] The original audio of the dialogue is subjected to acoustic feature extraction to obtain at least one acoustic feature;

[0008] Based on the at least one acoustic feature, generate natural language description information about the acoustic features of the original dialogue audio;

[0009] Based on the natural language description information, the target dialogue text is processed by audio synthesis to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0010] In a second aspect of this application, a dubbing device is also provided, the device comprising:

[0011] The acquisition module is used to acquire the target dialogue text and the original dialogue audio; wherein the target dialogue text and the original dialogue audio represent the same dialogue semantics, and the language of the target dialogue text is the target language to be dubbed;

[0012] The extraction module is used to extract acoustic features from the original dialogue audio to obtain at least one acoustic feature;

[0013] A generation module is used to generate natural language description information about the acoustic features of the original dialogue audio based on the at least one acoustic feature;

[0014] The synthesis module is used to perform audio synthesis processing on the target dialogue text based on the natural language description information to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0015] In a third aspect of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0016] Memory, used to store computer programs;

[0017] The processor, when executing a program stored in memory, implements any of the dubbing methods described above.

[0018] In a fourth aspect of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements any of the aforementioned dubbing methods. In yet another aspect of this application, a computer program product comprising instructions is also provided, which, when run on a computer, causes the computer to perform any of the aforementioned dubbing methods.

[0019] The solution of this application, for target dialogue text in a target language, extracts acoustic features from the original dialogue audio corresponding to the target dialogue text, obtaining at least one acoustic feature. The original dialogue audio has the same semantic content as the target dialogue text but a different language. Based on the obtained acoustic features, natural language description information about the acoustic features of the original dialogue audio is generated. Based on the natural language description information, audio synthesis processing is performed on the target dialogue text to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language. Therefore, the solution of this application can improve the dubbing efficiency for target dialogue text. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0021] Figure 1 This is a flowchart illustrating a dubbing method provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a process for generating target audio based on transcribed text and descriptive text, provided in an embodiment of this application.

[0023] Figure 3 This is a schematic diagram of the structure of a dubbing device provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0026] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0027] This application provides a dubbing method, apparatus, electronic device, and storage medium that can improve the efficiency of dubbing target dialogue text.

[0028] The following is a description of a dubbing method provided in an embodiment of this application. This dubbing method can be applied to electronic devices, such as servers or terminal devices. This application does not limit the specific form of the electronic device.

[0029] The dubbing method provided in this application embodiment may include:

[0030] Obtain the target dialogue text and the original dialogue audio; wherein the target dialogue text and the original dialogue audio represent the same dialogue semantics, and the language of the target dialogue text is the target language to be dubbed;

[0031] The original audio of the dialogue is subjected to acoustic feature extraction to obtain at least one acoustic feature;

[0032] Based on the at least one acoustic feature, generate natural language description information about the acoustic features of the original dialogue audio;

[0033] Based on the natural language description information, the target dialogue text is processed by audio synthesis to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0034] The solution of this application, for target dialogue text in a target language, extracts acoustic features from the original dialogue audio corresponding to the target dialogue text, obtaining at least one acoustic feature. The original dialogue audio has the same semantic content as the target dialogue text but a different language. Based on the obtained acoustic features, natural language description information about the acoustic features of the original dialogue audio is generated. Based on the natural language description information, audio synthesis processing is performed on the target dialogue text to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language. Therefore, the solution of this application can improve the dubbing efficiency for target dialogue text.

[0035] like Figure 1 As shown in the embodiment of this application, a dubbing method includes:

[0036] S101, Obtain the target dialogue text and the original dialogue audio; wherein, the target dialogue text and the original dialogue audio represent the same dialogue semantics, and the language of the target dialogue text is the target language to be dubbed.

[0037] This application is applicable to video dubbing scenarios, specifically, scenarios where existing dialogue in a video is dubbed into another language. For example, in video A, the existing dialogue is dubbed in English, and dubbing the dialogue in video A into Chinese could be a current need. This application does not limit the language of the existing dialogue in the video to be dubbed or the target language to be dubbed.

[0038] The original audio dialogue refers to the existing audio dialogue in the video to be dubbed. The target text dialogue refers to the text dialogue to be dubbed, which can be translated from the text dialogue corresponding to the original audio dialogue. Therefore, although the target text dialogue and the original audio dialogue are in different languages, they represent the same semantic meaning of the dialogue.

[0039] The target dialogue text can be a single line, all the lines of the same character in the entire video, or all the lines in the entire video. The specific granularity of the division can be set according to actual needs.

[0040] S102, perform acoustic feature extraction on the original dialogue audio to obtain at least one acoustic feature.

[0041] The acoustic features can include speech rate, emotion, gender, pitch, reverberation, and noise, etc. There are many specific types of acoustic features, and this application does not exhaustively list or limit them.

[0042] For the same line, the meaning of the line is the same in different languages, so the acoustic features corresponding to the dubbing of the line in different languages ​​can be the same.

[0043] Furthermore, different acoustic features can be extracted in different ways.

[0044] For example, regarding speech rate, we can identify each effective audio segment in the original audio script; an effective audio segment is an audio segment containing dialogue content; analyze the number of words in each effective audio segment, and then use the ratio of the total number of words in each effective audio segment to the total duration of each effective audio segment to determine the speech rate.

[0045] Gender can be determined using an audio gender classification model. This model is trained using sample audio files and ground truth values. The sample audio files contain both male and female voices, while the ground truth values ​​represent the gender of the sample audio files. Therefore, by inputting the original dialogue audio into the audio gender classification model, the gender classification result for that original dialogue audio can be obtained.

[0046] Of course, large language models can also be used to identify multiple acoustic features of the original dialogue audio at once.

[0047] In this application, there are many types of acoustic features, and there can be multiple ways to extract each type of acoustic feature, which will not be listed here. Moreover, the method of acoustic feature extraction is not the main point of this application. Any method that can achieve acoustic feature extraction can be applied to the solution of this application, and this application does not limit it.

[0048] Acoustic features can be represented in the form of text, numbers, tables, and images, and can also be specific parameters.

[0049] Because different acoustic features require different extraction methods, the resulting acoustic feature outputs can have different formats. For example, the acoustic feature output could be: speech rate: fast, gender: male, etc.

[0050] S103, Based on the at least one acoustic feature, generate natural language description information about the acoustic features of the original dialogue audio.

[0051] The natural language description information describes acoustic features using human language. Utilizing natural language description information allows for a standardized format and accurate description of the acoustic features of the original dialogue audio across various dimensions. Therefore, the solution in this application does not impose specific requirements on the format of each acoustic feature. Furthermore, natural language description information allows for the association of at least one acoustic feature, thereby improving the correlation between various acoustic features.

[0052] The acoustic features of different languages, such as phonemes, prosodic rules, and emotional expression habits, are fundamentally different.

[0053] The language of the natural language description information can be the same as the target dialogue text. For example, using Chinese natural language description to generate speech corresponding to the Chinese target dialogue text results in better matching.

[0054] Natural language descriptions can abstract specific acoustic features, describing them from a semantic level of style and intent (e.g., "a passionate male voice, fast-paced, with a sad tone"). This abstract description is language-independent and not a specific parameter (e.g., the sound is x Hz), thus allowing for a more accurate expression of acoustic features.

[0055] Furthermore, compared to directly transplanting the acoustic features of the original dialogue audio into the generated target language audio (ensuring that the parameters of each acoustic feature in the original dialogue audio are consistent with the parameters of each acoustic feature in the generated target language audio), the solution in this application can generate speech that is more in line with the language's habits, more natural, and more expressive based on the semantic understanding of natural language description information (such as "sadness") and combined with the rhythm and pronunciation rules of the target language itself. For example, the speaking speed, pauses, and pitch variation patterns of expressing sadness in Chinese are different from those in English. The description information can make the generated audio interpret "sadness" in the Chinese way, thereby reducing the awkwardness of directly transplanting the parameters of acoustic features. Therefore, the natural language description information of acoustic features in this application is fundamentally different from the parameters of the specific acoustic features.

[0056] Furthermore, the natural language description information is easily understood by humans, making it easy for users to detect. During the generation process, users can also adjust the natural language description information as needed, offering a high degree of freedom.

[0057] For example, the target dialogue text could be: "The woods where Timothy wandered alone were desolate and remote, inhabited by fierce lynxes that often howled and fought at night"; the natural language description could be: "A woman reads the work, her voice is highly captivating and mesmerizing. Her pronunciation is very clear and pleasant to the ear, and the recording quality is impeccable." Here, "woman," "highly captivating," "clear and pleasant to the ear," and "the recording quality is impeccable" can be considered the aforementioned acoustic features.

[0058] S104, Based on the natural language description information, perform audio synthesis processing on the target dialogue text to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0059] The executing entity can perform audio synthesis processing on the target dialogue text according to the acoustic features guaranteed by the natural language description information, thereby obtaining a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language, thus completing the dubbing of the target dialogue text in the target language.

[0060] The solution of this application, for target dialogue text in a target language, extracts acoustic features from the original audio of the dialogue text to obtain at least one acoustic feature. Based on the obtained acoustic feature, it generates natural language description information about the acoustic features of the original audio. Based on the natural language description information, it performs audio synthesis processing on the target dialogue text to obtain a dubbing audio that has the acoustic features of the original audio and belongs to the target language. Therefore, the solution of this application can improve the dubbing efficiency for target dialogue text.

[0061] Optionally, the step of performing audio synthesis processing on the target dialogue text based on the natural language description information to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language includes steps A1-A3;

[0062] Step A1: Vectorize the target dialogue text and the natural language description information respectively to obtain a first vector corresponding to the target dialogue text and a second vector corresponding to the natural language description information;

[0063] Since the input to the pre-trained target speech synthesis model is a vector, the target dialogue text can be vectorized to obtain the first vector, and the natural language description information can be vectorized to obtain the second vector.

[0064] Specifically, the first vector may include a first marker in addition to the vector containing the target dialogue text, to indicate that it is the first vector. Similarly, the second vector may include a second marker in addition to the vector containing natural language description information, to indicate that it is the second vector.

[0065] Step A2: Input the first vector and the second vector into the pre-trained target speech synthesis model so that the target speech synthesis model generates a target audio vector based on the first vector and the second vector; wherein the target audio vector is used to represent the audio of the target dialogue text.

[0066] The pre-trained target speech synthesis model can process the first and second input vectors to obtain the target audio vector.

[0067] The target audio vector can be understood as the vector form of audio.

[0068] Step A3: Decode the target audio vector into audio to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0069] Decoding the target audio vector yields a dubbing audio that possesses the acoustic features of the original dialogue audio and belongs to the target language.

[0070] Using the target speech synthesis model provided in this application, a target audio vector that has the acoustic features of the original dialogue audio and belongs to the target language can be generated based on the first vector and the second vector. By decoding the target audio vector, the dubbing audio can be obtained.

[0071] Optionally, in step A2, the target speech synthesis model generates the target audio vector based on the first and second vectors in the following ways:

[0072] The target speech synthesis model performs cross-attention processing and self-attention processing based on the first vector and the second vector to obtain intermediate processing results, and generates target audio vectors based on the intermediate processing results.

[0073] Specifically, the intermediate processing result is an audio token sequence, and the target audio vector is composed of the joint vectors corresponding to each audio token in the audio token sequence. Generating the target audio vector based on the intermediate processing result can be achieved by determining the joint vector of each audio token in the audio token sequence from the RVQ (Residual Vector Quantization) codebook, and then constructing the target audio vector using these joint vectors.

[0074] The RVQ codebook is a pre-defined set of fixed vectors. Each vector corresponds to a specific acoustic feature of the audio (such as frequency, timbre, loudness, etc.), and each vector has a unique integer index (i.e., a single-level token). In a multi-level RVQ codebook, there is a one-to-one correspondence between joint tokens and joint vectors. Each level of RVQ has a fixed codebook, and each level of codebook has a corresponding set of fixed vectors. Each joint token is a combination of indices from multiple levels of codebooks. For example, for a 3-level codebook, index 1 from the first level, index 2 from the second level, and index 3 from the third level can be selected to form a joint token.

[0075] The following describes the generation process of each audio token in the audio token sequence. The target speech synthesis model performs self-attention processing on the generated audio tokens (in actual processing, since the audio token is equivalent to the index of the joint vector, the target speech synthesis model performs self-attention processing on the joint vector corresponding to the generated audio token, and when generating the first audio token, since there is no previously generated audio token, there is no need to use the previously generated audio token), which can obtain a context state vector. The context state vector can represent the correlation between the generated audio tokens. Specifically, the correlation can include temporal dependency information of the audio (such as pitch changes, speech rate coherence, etc.), which can ensure the fluency of the generated audio.

[0076] Cross-attention processing between the first vector and the context state vector can make the generated audio tokens correspond one-to-one with the text semantics (e.g., "climbing a mountain" corresponds to a specific audio token). Cross-attention processing between the second vector and the context state vector can make the target speech synthesis model refer to the style information (such as accent, emotion, sound quality, etc.) in the natural language description information when generating audio tokens, so that the generated audio token sequence conforms to the characteristics of the natural language description information. The first and second vectors can be cross-attention processed with the context state vector to obtain the first cross vector and the second cross vector, respectively. (This process can involve using the context state vector as the query and the first vector as the key / value pair, or vice versa.) The first and second cross vectors obtained after cross-attention processing can be weighted and fused. Based on the weighted fused vector, the similarity between the current level residual (initially the fused vector) and the corresponding single-layer vector in the codebook is calculated sequentially according to the RVQ hierarchy (the residual of each layer = the input residual of the previous layer - the single-layer vector selected in the previous layer). The token corresponding to the single-layer vector with the highest similarity is selected layer by layer, and the selected single-layer tokens are combined into a joint token, which is the audio token generated in this iteration. The sequence of audio tokens generated is the audio token sequence.

[0077] For each joint token (audio token), the single-level vectors corresponding to each index of the joint token are calculated according to the RVQ residual superposition rule (i.e., summing the single-level vectors corresponding to each index). The result is the joint vector of the joint token. Using the joint vectors of the audio tokens in the audio token sequence, the target audio vector can be constructed. For example, the joint vectors can be arranged in temporal order to obtain the target audio vector.

[0078] Cross-attention processing enables the establishment of a dynamic relationship between the target dialogue text and the natural language description information. It allows for the understanding of both the target dialogue text and the natural language description information, ensuring that each acoustic feature (such as timbre, intonation, and sound effects) is presented in the correct position and manner, ultimately synthesizing the target audio. In essence, cross-attention processing instructs the target speech synthesis model to generate an audio vector containing the acoustic features represented by the natural language description information, based on the target dialogue text.

[0079] Optionally, generating natural language description information about the acoustic features of the original dialogue audio based on the at least one acoustic feature includes:

[0080] The at least one acoustic feature and the cue word are input into a predetermined large language model, so that the large language model analyzes and processes the at least one acoustic feature according to the cue word to generate natural language description information about the acoustic features of the original dialogue audio, wherein the cue word is at least used to indicate the description of the acoustic feature by natural language.

[0081] Large language models are deep learning models trained on massive amounts of data and possessing a huge number of parameters (usually billions or even trillions). They are capable of understanding and generating human language and performing complex tasks such as reasoning and logical analysis. The solution proposed in this application can use any large language model.

[0082] The prompt words can be used to instruct the large language model to generate natural language description information using at least one acoustic feature in a specified format. For example, the format could be: This audio clip, the speaker's gender is <>, the speech rate is <>, and the pitch is <>; the acoustic feature to be filled in is in <>.

[0083] The large language model can understand the meaning of the at least one acoustic feature, and thus can generate natural language description information about the acoustic features of the original dialogue audio according to a specified format. Therefore, this application does not limit the format of each acoustic feature among the at least one acoustic feature. It should be emphasized that the large language model can understand the meaning of the at least one acoustic feature and describe the acoustic feature from the perspective of semantic style and intent, rather than simply filling in blanks according to a specified format.

[0084] Therefore, this embodiment can utilize a predetermined large language model to quickly generate natural language description information, thereby further improving the efficiency of dubbing.

[0085] Optionally, the training method for the target speech synthesis model includes:

[0086] The speech synthesis model to be trained is pre-trained using the first sample data;

[0087] Using the second sample data, the pre-trained speech synthesis model is trained to obtain the target speech synthesis model;

[0088] The first sample data includes: first sample text, first natural language description information, and first sample audio; the first sample text and the first sample audio represent the same semantic content, the first sample audio is audio for the language to which the first sample text belongs, and the first natural language description information is used to represent the acoustic features of the first sample audio.

[0089] The second sample data includes: second sample text, second natural language description information, and second sample audio; the second sample text is a dialogue text, and the second sample audio is a dubbing audio obtained by a specified user for the language to which the second sample text belongs. The dialogue semantics represented by the second sample text and the second sample audio are the same, and the second natural language description information is used to represent the acoustic features of the second sample audio.

[0090] The training process of the target speech synthesis model described in this application is divided into two training processes. In this embodiment, the first stage involves pre-training the speech synthesis model to be trained using first sample data; the second stage involves training the pre-trained speech synthesis model using second sample data to obtain the target speech synthesis model. In the first stage, the first sample data used can be any available sample data, such as various open-source speech data, and it is not limited whether the speech content is dialogue, as long as it is text content; thus, the first stage can use a large amount of first sample data for pre-training to obtain a pre-trained speech synthesis model.

[0091] In the second stage, the second sample audio used in the sample data is obtained by dubbing by a designated user. It's understandable that video platforms have requirements for the timbre of the dubbing. Therefore, training the model using the second sample audio of a designated user allows it to learn the timbre of that user, enabling it to generate audio vectors based on that learned timbre.

[0092] The designated user can be one or more users, allowing the target speech synthesis model to learn the timbre of one or more designated users. In the specific application of the target speech synthesis model, the timbre of one of multiple designated users can be pre-assigned to the target lines of a character. This ensures that when the target speech synthesis model generates the audio vector for that character, it utilizes the assigned designated user, thus maintaining the consistency of the character's timbre. It is understood that the target lines for different characters can be pre-differentiated. Before inputting the first and second vectors into the pre-trained target speech synthesis model, the timbre of the audio vector generated by the target speech synthesis model can be controlled to be the timbre of the assigned designated user.

[0093] Timbre refers to the unique quality or "color" of a sound, allowing us to distinguish sounds of the same pitch and volume from different sound sources (such as different instruments or human voices). For example, the sounds of a child and an elderly person producing the same note sound completely different. In other words, assuming other acoustic characteristics are the same, differences in the timbre of a dubbing performance can significantly alter the audience's experience.

[0094] Furthermore, in the second stage, the accuracy of the second natural language description information in the second sample data used is more accurate than that of the first natural language description information in the first stage. For example, the second natural language description information can be generated using at least one manually labeled acoustic feature, so the audio vector generated by the target speech synthesis model trained using the second sample data is also more accurate.

[0095] The second stage of the training process allows for fine-tuning of the parameters of the pre-trained speech synthesis model to obtain a more accurate target speech synthesis model. Specifically, fine-tuning can be applied to only a subset of the parameters of the pre-trained speech synthesis model. For example, parameters affecting timbre can be adjusted. In scenarios involving adjusting only certain parameters, the specific parameters to be adjusted can be set according to the actual situation, and this application does not impose any limitations on this.

[0096] Furthermore, in the second sample data, the second sample audio is obtained by a designated user dubbing it. The accuracy of the second natural language description information is more accurate than that of the first natural language description information in the first stage. In addition, the generation cost of the second sample data is higher, so the number of second sample data is usually less than that of the first sample data. However, this application can train a more accurate target speech synthesis model at a lower cost by reasonably utilizing the first and second sample data through a two-stage training process.

[0097] The pre-trained speech synthesis model can be called a base model. In a specific implementation, the base model can be trained using a large amount of first sample data (for example, the total duration of the first sample audio in each first sample data can be greater than 30,000 hours) to make the generated speech more natural.

[0098] Optionally, the process of pre-training the speech synthesis model to be trained using the first sample data includes steps B1-B5:

[0099] Step B1: Vectorize the first sample text, the first natural language description information, and the first sample audio respectively to obtain the vector of the first sample text, the vector of the first natural language description information, and the audio vector of the first sample audio.

[0100] Step B2: Input the vector of the first sample text and the vector of the first natural language description information into the speech synthesis model to be trained for processing, and obtain the predicted probability value of each candidate audio token at each token element position generated by the speech synthesis model to be trained; the token element position is: the element position in the audio token sequence that serves as the intermediate processing result of the speech synthesis model to be trained, and the intermediate processing result is the processing result used to generate the predicted audio vector as the output result of the model;

[0101] Step B3: For each token element position, determine the predicted probability value of the candidate audio token that is the same as the corresponding real audio token from the predicted probability values ​​of each candidate audio token at that token element position, and use it as the predicted probability value of that token element position; the real audio token corresponding to that token element position is: the audio token located at that token element position in the audio token sequence obtained by converting the audio vector of the first sample audio.

[0102] Step B4: Determine the loss value based on the predicted probability value of each token element position;

[0103] Step B5: Adjust the parameters of the speech synthesis model to be trained based on the obtained loss value until the speech synthesis model to be trained converges.

[0104] The process of training the pre-trained speech synthesis model using the second sample data includes steps C1-C5.

[0105] Step C1: Vectorize the second sample text, the second natural language description information, and the second sample audio respectively to obtain the vector of the second sample text, the vector of the second natural language description information, and the audio vector of the second sample audio.

[0106] Step C2: Input the vector of the second sample text and the vector of the second natural language description information into the pre-trained speech synthesis model for processing to obtain the predicted probability value of each candidate audio token at each token element position generated by the pre-trained speech synthesis model; the token element position is: the element position in the audio token sequence that is the intermediate processing result of the pre-trained speech synthesis model, and the intermediate processing result is the processing result used to generate the predicted audio vector as the model output result;

[0107] Step C3: For each token element position, determine the predicted probability value of the candidate audio token that is the same as the corresponding real audio token from the predicted probability values ​​of each candidate audio token at that token element position, and use it as the predicted probability value of that token element position; the real audio token corresponding to that token element position is: the audio token located at that token element position in the audio token sequence obtained by converting the audio vector of the second sample audio.

[0108] Step C4: Determine the loss value based on the predicted probability value of each token element position;

[0109] Step C5: Adjust the parameters of the pre-trained speech synthesis model based on the obtained loss value until the pre-trained speech synthesis model converges.

[0110] When calculating the loss value in steps B4 and C4, the loss function used can be the cross-entropy loss function, specifically, it can be:

[0111] Loss = -ΣΣ log P(c_t^k | c <t, text, description);

[0112] Among them: the subscripts of the two summation symbols in the loss function can be 1-T and 1-K respectively, where T is the length of the audio sequence (number of time steps), K is the number of codebook layers, and c_t^k is the true audio token of the codebook at the t-th step and the k-th layer. c<t represents all the generated audio tokens, text refers to the first vector corresponding to the target line text, and description refers to the second vector corresponding to the natural language description information.

[0113] P(c_t^k | c_<t, text, description) is the predicted probability value corresponding to the true audio token of the codebook at the t-th step and the k-th layer. P(c_t^k | c_<t, text, description) is a conditional probability, where (c_<t,text, description) is the condition, indicating that when generating the audio token of the codebook at the t-th step and the k-th layer, it needs to depend on all the generated audio tokens, the first vector corresponding to the target line text, and the second vector corresponding to the natural language description information. The principle of depending on each condition here corresponds to step A2 above.

[0114] When calculating the loss value using the loss function, the greater the predicted probability of the true audio token corresponding to each token element position, the smaller the calculated loss value.

[0115] The speech synthesis model can be based on the decoder only transformer model. The decoder only transformer model is a model variant that only retains the decoder part in the original Transformer (a deep learning model) architecture and is an autoregressive model. The above steps B1 - B5 are the training process of the first stage using the first sample data for training, and steps C1 - C5 are the training process of the second stage using the second sample data.

[0116] It is understandable that the number of first sample data can be multiple, and each sample data includes a first sample text, a first natural language description, and a first sample audio, which have a corresponding relationship. In the specific training process, the first sample data can be divided into multiple batches, each batch including one or more first sample data. For each batch of first sample data, steps B1-B3 are followed to obtain the predicted probability value of each element position in the audio token sequence of each first sample data included in that batch. Then, steps B4-B5 are executed. Specifically, based on the predicted probability values ​​of each token element position, a loss value is determined, and the parameters of the speech synthesis model to be trained are adjusted based on the obtained loss value. When the batch includes multiple first samples, the loss value used to adjust the parameters of the pre-trained speech synthesis model can be the average or weighted sum of the loss values ​​of each first sample data. Then, steps B1-B5 can be executed again using another batch of first sample data until the pre-trained speech synthesis model converges. The conditions for the pre-trained speech synthesis model to converge may include one or more of the following: the loss value is not greater than a target threshold and the preset maximum number of training rounds is reached.

[0117] Similarly, the number of second sample data can be multiple, and each sample data includes a corresponding relationship between the second sample text, the second natural language description information, and the second sample audio. In the specific training process, each second sample data can be divided into multiple batches, each batch including one or more second sample data. For each batch of second sample data, steps C1-C3 are followed to obtain the predicted probability value of each element position in the audio token sequence of each second sample data included in that batch. Then, steps C4-C5 are executed. Specifically, based on the predicted probability value of each token element position, a loss value is determined and calculated. The parameters of the pre-trained speech synthesis model are adjusted based on the obtained loss value. When the batch includes multiple second samples, the loss value used to adjust the parameters of the pre-trained speech synthesis model can be the average or weighted sum of the loss values ​​of each second sample data. Then, steps C1-C5 can be executed again using another batch of second sample data until the pre-trained speech synthesis model converges. The conditions for the pre-trained speech synthesis model to converge can include one or more of the following: the loss value is not greater than a target threshold and the preset maximum number of training rounds is reached.

[0118] The differences between the first and second phases of the training process, including the differences between the first and second sample data and the differences in the adjusted parameters, have been described in the above embodiments and will not be repeated here.

[0119] By training the model in this embodiment, an audio vector can be obtained that can accurately generate audio of the target dialogue text that conforms to the natural language description information.

[0120] Optionally, the audio vectors of the first sample audio and the second sample audio are obtained by encoding using a target vocoder.

[0121] Step A3 decodes the target audio vector into audio, obtaining dubbing audio that possesses the acoustic features of the original dialogue audio and belongs to the target language, including:

[0122] The target audio vector is input to the target vocoder so that the target vocoder decodes the target audio vector to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0123] The target vocoder can encode audio to obtain an audio vector, and decode the audio vector to obtain audio.

[0124] The target speech synthesis model takes vectors as input and outputs vectors. Therefore, for the first sample audio in the first sample data and the second sample audio in the second sample data, the target vocoder can be used to encode the first sample audio to obtain the audio vector of the first sample audio, and encode the second sample audio to obtain the audio vector of the first sample audio. Furthermore, the target vocoder can decode the target audio vector to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0125] It is understandable that there may be differences between the encoding and decoding results of different vocoders. Therefore, in this application, the same vocoder is used for encoding the first and second sample audio samples, as well as for decoding the target audio vector. This enhances the consistency of the encoding and decoding processes, resulting in more accurate dubbing audio. This application does not specifically limit the vocoder; the same vocoder can be used throughout the processing.

[0126] The following is a specific embodiment illustrating the process by which S104, based on the natural language description information, performs audio synthesis processing on the target dialogue text to obtain a dubbing audio with the acoustic features of the original dialogue audio and belonging to the target language. Figure 2 This is a schematic diagram illustrating a process for generating target audio based on transcribed text and descriptive text, as provided in an embodiment of this application. Figure 2 As shown, it may include the following steps:

[0127] S201, the transcribed text is vectorized to obtain the first vector.

[0128] S202, the description text is vectorized to obtain the second vector.

[0129] S201 and S202 correspond to step A1 above. The transcribed text is the target dialogue text, and the descriptive text is the natural language description information. Vectorization of the descriptive text can be achieved using a pre-trained T5 model with fixed parameters. Figure 2 The pre-trained parameters are fixed. The T5 model is a text processing model. The first vector can be... Figure 2 The vector corresponding to the preceding text tokens.

[0130] S203, based on the first vector and the second vector, perform cross-attention processing and self-attention processing to obtain the processing result, and generate the target audio vector based on the processing result.

[0131] S203 corresponds to step A2 above. Further details will not be provided here. Figure 2 The codebook in the above is the codebook of RVQ (Residual Vector Quantization). Sequence steps are the quantization levels of RVQ. Each level can correspond to one codebook. The token after residual vector quantization is the processing result. Figure 2 t1-t n For Token.

[0132] S204, decode the target audio vector to obtain the target audio.

[0133] S204 corresponds to step A3 above. The target audio vector is input to the target vocoder so that the target vocoder decodes the target audio vector to obtain the target audio. The target audio is the dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0134] Furthermore, before step S202, step 205 may be included: extracting acoustic features from the original dialogue audio to obtain meta-information (meta-information refers to the aforementioned acoustic features, such as speech rate, emotion, gender, pitch, reverberation, etc.). The descriptive text is generated using the meta-information. S202 corresponds to S102 above.

[0135] The proposed solution constructs sound features through natural language description, which can effectively preserve the sound features of the dialogue in the original audio, such as emotions and other sound details, and the output dubbing is more in line with the scene of the original video.

[0136] Based on the above-described embodiments of the dubbing method, this application also provides a dubbing device. Figure 3 This is a schematic diagram of the structure of a dubbing device provided in an embodiment of this application, as shown below. Figure 3 As shown, the dubbing device may include:

[0137] The acquisition module 301 is used to acquire the target dialogue text and the original dialogue audio; wherein the target dialogue text and the original dialogue audio represent the same dialogue semantics, and the language of the target dialogue text is the target language to be dubbed;

[0138] Extraction module 302 is used to extract acoustic features from the original dialogue audio to obtain at least one acoustic feature;

[0139] The generation module 303 is used to generate natural language description information about the acoustic features of the original dialogue audio based on the at least one acoustic feature;

[0140] The synthesis module 304 is used to perform audio synthesis processing on the target dialogue text based on the natural language description information to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0141] The solution of this application, for target dialogue text in a target language, extracts acoustic features from the original dialogue audio corresponding to the target dialogue text, obtaining at least one acoustic feature. The original dialogue audio has the same semantic content as the target dialogue text but a different language. Based on the obtained acoustic features, natural language description information about the acoustic features of the original dialogue audio is generated. Based on the natural language description information, audio synthesis processing is performed on the target dialogue text to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language. Therefore, the solution of this application can improve the dubbing efficiency for target dialogue text.

[0142] Optionally, the synthesis module includes:

[0143] The vectorization processing unit is used to perform vectorization processing on the target dialogue text and the natural language description information respectively to obtain a first vector corresponding to the target dialogue text and a second vector corresponding to the natural language description information.

[0144] An input unit is configured to input the first vector and the second vector into a pre-trained target speech synthesis model, so that the target speech synthesis model generates a target audio vector based on the first vector and the second vector; wherein the target audio vector is used to represent the audio of the target dialogue text;

[0145] The decoding unit is used to decode the target audio vector into audio, thereby obtaining dubbing audio that has the timbre of the original dialogue audio and belongs to the target language.

[0146] Optionally, the target speech synthesis model generates the target audio vector based on the first vector and the second vector in the following ways:

[0147] The target speech synthesis model performs cross-attention processing and self-attention processing based on the first vector and the second vector to obtain intermediate processing results, and generates target audio vectors based on the intermediate processing results.

[0148] Optionally, the generation module includes:

[0149] A generation unit is configured to input the at least one acoustic feature and a cue word into a predetermined large language model, so that the large language model analyzes and processes the at least one acoustic feature according to the cue word to generate natural language description information about the acoustic features of the original dialogue audio, wherein the cue word is at least used to indicate the description of the acoustic feature by natural language.

[0150] Optionally, the training method for the target speech synthesis model includes:

[0151] The speech synthesis model to be trained is pre-trained using the first sample data;

[0152] Using the second sample data, the pre-trained speech synthesis model is trained to obtain the target speech synthesis model;

[0153] The first sample data includes: first sample text, first natural language description information, and first sample audio; the first sample text and the first sample audio represent the same semantic content, the first sample audio is audio for the language to which the first sample text belongs, and the first natural language description information is used to represent the acoustic features of the first sample audio.

[0154] The second sample data includes: second sample text, second natural language description information, and second sample audio; the second sample text is a dialogue text, and the second sample audio is a dubbing audio obtained by a specified user for the language to which the second sample text belongs. The dialogue semantics represented by the second sample text and the second sample audio are the same, and the second natural language description information is used to represent the acoustic features of the second sample audio.

[0155] Optionally, the process of pre-training the speech synthesis model to be trained using the first sample data includes:

[0156] The first sample text, the first natural language description information, and the first sample audio are vectorized respectively to obtain the vector of the first sample text, the vector of the first natural language description information, and the audio vector of the first sample audio.

[0157] The vector of the first sample text and the vector of the first natural language description information are input into the speech synthesis model to be trained for processing to obtain the predicted probability value of each candidate audio token at each token element position generated by the speech synthesis model to be trained; the token element position is: the element position in the audio token sequence that is the intermediate processing result of the speech synthesis model to be trained, and the intermediate processing result is the processing result used to generate the predicted audio vector that is the output result of the model.

[0158] For each token element position, from the predicted probability values ​​of each candidate audio token at that token element position, determine the predicted probability value of the candidate audio token that is the same as the corresponding real audio token, and use it as the predicted probability value of that token element position; the real audio token corresponding to that token element position is: the audio token located at that token element position in the audio token sequence obtained by converting the audio vector of the first sample audio.

[0159] The loss value is determined based on the predicted probability value of each token element's position;

[0160] The parameters of the speech synthesis model to be trained are adjusted based on the obtained loss value until the speech synthesis model to be trained converges.

[0161] Optionally, the process of training the pre-trained speech synthesis model using the second sample data includes:

[0162] The second sample text, the second natural language description information, and the second sample audio are vectorized respectively to obtain the vector of the second sample text, the vector of the second natural language description information, and the audio vector of the second sample audio.

[0163] The vector of the second sample text and the vector of the second natural language description information are input into the pre-trained speech synthesis model for processing to obtain the predicted probability value of each candidate audio token at each token element position generated by the pre-trained speech synthesis model; the token element position is the element position in the audio token sequence that is the intermediate processing result of the pre-trained speech synthesis model, and the intermediate processing result is the processing result used to generate the predicted audio vector as the model output result.

[0164] For each token element position, from the predicted probability values ​​of each candidate audio token at that token element position, determine the predicted probability value of the candidate audio token that is the same as the corresponding real audio token, and use it as the predicted probability value of that token element position; the real audio token corresponding to that token element position is: the audio token located at that token element position in the audio token sequence obtained by converting the audio vector of the second sample audio.

[0165] The loss value is determined based on the predicted probability value of each token element's position;

[0166] The parameters of the pre-trained speech synthesis model are adjusted based on the obtained loss value until the pre-trained speech synthesis model converges.

[0167] Optionally, the audio vectors of the first sample audio and the second sample audio are obtained by encoding using a target vocoder.

[0168] The decoding unit includes:

[0169] The decoding subunit is used to input the target audio vector into the target vocoder so that the target vocoder can decode the target audio vector to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

[0170] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0171] Memory 403 is used to store computer programs;

[0172] The processor 401, when executing the program stored in the memory 403, implements any of the dubbing methods described above.

[0173] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0174] The communication interface is used for communication between the aforementioned terminal and other devices.

[0175] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0176] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0177] In another embodiment provided in this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the dubbing methods described in the above embodiments.

[0178] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the dubbing methods described in the above embodiments.

[0179] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0180] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0181] The various embodiments in this specification are described in a related manner. For the same or similar parts between the various embodiments, please refer to each other. Each embodiment focuses on describing the differences from other embodiments.

[0182] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A dubbing method, characterized in that, The method includes: Obtain the target dialogue text and the original dialogue audio; wherein the target dialogue text and the original dialogue audio represent the same dialogue semantics, and the language of the target dialogue text is the target language to be dubbed; The original audio of the dialogue is subjected to acoustic feature extraction to obtain at least one acoustic feature; Based on the at least one acoustic feature, generate natural language description information about the acoustic features of the original dialogue audio; Based on the natural language description information, the target dialogue text is processed by audio synthesis to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

2. The method according to claim 1, characterized in that, The step of performing audio synthesis processing on the target dialogue text based on the natural language description information to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language includes: The target dialogue text and the natural language description information are vectorized respectively to obtain a first vector corresponding to the target dialogue text and a second vector corresponding to the natural language description information. The first vector and the second vector are input into a pre-trained target speech synthesis model so that the target speech synthesis model generates a target audio vector based on the first vector and the second vector; wherein the target audio vector is used to represent the audio of the target dialogue text; The target audio vector is decoded into audio to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

3. The method according to claim 2, characterized in that, The target speech synthesis model generates target audio vectors based on the first vector and the second vector in the following ways: The target speech synthesis model performs cross-attention processing and self-attention processing based on the first vector and the second vector to obtain intermediate processing results, and generates target audio vectors based on the intermediate processing results.

4. The method according to any one of claims 1-3, characterized in that, The process of generating natural language description information about the acoustic features of the original dialogue audio based on the at least one acoustic feature includes: The at least one acoustic feature and the cue word are input into a predetermined large language model, so that the large language model analyzes and processes the at least one acoustic feature according to the cue word to generate natural language description information about the acoustic features of the original dialogue audio, wherein the cue word is at least used to indicate the description of the acoustic feature by natural language.

5. The method according to claim 2, characterized in that, The training methods for the target speech synthesis model include: The speech synthesis model to be trained is pre-trained using the first sample data; Using the second sample data, the pre-trained speech synthesis model is trained to obtain the target speech synthesis model; The first sample data includes: first sample text, first natural language description information, and first sample audio; the first sample text and the first sample audio represent the same semantic content, the first sample audio is audio for the language to which the first sample text belongs, and the first natural language description information is used to represent the acoustic features of the first sample audio. The second sample data includes: second sample text, second natural language description information, and second sample audio; the second sample text is a dialogue text, and the second sample audio is a dubbing audio obtained by a specified user for the language to which the second sample text belongs. The dialogue semantics represented by the second sample text and the second sample audio are the same, and the second natural language description information is used to represent the acoustic features of the second sample audio.

6. The method according to claim 5, characterized in that, The process of pre-training the speech synthesis model to be trained using the first sample data includes: The first sample text, the first natural language description information, and the first sample audio are vectorized respectively to obtain the vector of the first sample text, the vector of the first natural language description information, and the audio vector of the first sample audio. The vector of the first sample text and the vector of the first natural language description information are input into the speech synthesis model to be trained for processing to obtain the predicted probability value of each candidate audio token at each token element position generated by the speech synthesis model to be trained; the token element position is: the element position in the audio token sequence that is the intermediate processing result of the speech synthesis model to be trained, and the intermediate processing result is the processing result used to generate the predicted audio vector as the output result of the model. For each token element position, from the predicted probability values ​​of each candidate audio token at that token element position, determine the predicted probability value of the candidate audio token that is the same as the corresponding real audio token, and use it as the predicted probability value of that token element position; the real audio token corresponding to that token element position is: the audio token located at that token element position in the audio token sequence obtained by converting the audio vector of the first sample audio. The loss value is determined based on the predicted probability value of each token element's position; The parameters of the speech synthesis model to be trained are adjusted based on the obtained loss value until the speech synthesis model to be trained converges.

7. The method according to claim 5, characterized in that, The process of training the pre-trained speech synthesis model using the second sample data includes: The second sample text, the second natural language description information, and the second sample audio are vectorized respectively to obtain the vector of the second sample text, the vector of the second natural language description information, and the audio vector of the second sample audio. The vector of the second sample text and the vector of the second natural language description information are input into the pre-trained speech synthesis model for processing to obtain the predicted probability value of each candidate audio token at each token element position generated by the pre-trained speech synthesis model; the token element position is the element position in the audio token sequence that is the intermediate processing result of the pre-trained speech synthesis model, and the intermediate processing result is the processing result used to generate the predicted audio vector as the model output result. For each token element position, from the predicted probability values ​​of each candidate audio token at that token element position, determine the predicted probability value of the candidate audio token that is the same as the corresponding real audio token, and use it as the predicted probability value of that token element position; the real audio token corresponding to that token element position is: the audio token located at that token element position in the audio token sequence obtained by converting the audio vector of the second sample audio. The loss value is determined based on the predicted probability value of each token element's position; The parameters of the pre-trained speech synthesis model are adjusted based on the obtained loss value until the pre-trained speech synthesis model converges.

8. The method according to claim 6 or 7, characterized in that, The audio vectors of the first sample audio and the second sample audio are obtained by encoding using a target vocoder. The step of decoding the target audio vector into audio to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language includes: The target audio vector is input to the target vocoder so that the target vocoder decodes the target audio vector to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

9. A dubbing device, characterized in that, The device includes: The acquisition module is used to acquire the target dialogue text and the original dialogue audio; wherein the target dialogue text and the original dialogue audio represent the same dialogue semantics, and the language of the target dialogue text is the target language to be dubbed; The extraction module is used to extract acoustic features from the original dialogue audio to obtain at least one acoustic feature; A generation module is used to generate natural language description information about the acoustic features of the original dialogue audio based on the at least one acoustic feature; The synthesis module is used to perform audio synthesis processing on the target dialogue text based on the natural language description information to obtain a dubbing audio that has the acoustic features of the original dialogue audio and belongs to the target language.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images