Voice determination method, electronic equipment and storage medium

Abstract

The invention discloses a voice determination method and device, electronic equipment and a storage medium. The method comprises the following steps: based on to-be-dubbed voice of a target object in a target video, determining timbre characteristics of the target object and a text sequence of the to-be-dubbed voice in a target language; the language of the to-be-dubbed voice is different from the target language; speech feature synthesis processing is carried out based on the tone features and the text sequence, and a candidate speech feature sequence under the target language is obtained; performing speech synthesis processing based on the mouth type vector sequence of the target object and the candidate speech feature sequence to obtain target speech of the to-be-dubbed speech in a target language; the mouth shape vector sequence is used for indicating the mouth shape change when the target object speaks the to-be-dubbed voice. According to the method provided by the invention, the target of automatically generating the target voice of the target language required by dubbing based on the target video is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic information, and more particularly, to a voice determination method, an electronic device, and a storage medium. BACKGROUND

[0002] At present, in the field of multi-language video dubbing, one or more dubbing actors speaking the target language are needed to record the dubbing voice of the target language for the target video, and then add the dubbing voice to the target video to realize dubbing of the target video. However, using this dubbing method, the dubbing efficiency is low and the dubbing cost is high. SUMMARY

[0003] Therefore, the embodiments of the present application provide a voice determination method, device, electronic device, and storage medium.

[0004] In a first aspect, the embodiments of the present application provide a voice determination method, which includes: determining a timbre feature of a target object and a text sequence of a to-be-dubbed voice of the target object in a target language based on the to-be-dubbed voice of the target object in a target video; the language of the to-be-dubbed voice is different from the target language; performing voice feature synthesis processing based on the timbre feature and the text sequence to obtain a candidate voice feature sequence in the target language; and performing voice synthesis processing based on a mouth shape vector sequence of the target object and the candidate voice feature sequence to obtain a target voice of the to-be-dubbed voice in the target language; the mouth shape vector sequence is used to indicate a mouth shape change when the target object speaks the to-be-dubbed voice.

[0005] In a second aspect, the embodiments of the present application provide a voice determination device, which includes: a determination module configured to determine a timbre feature of a target object and a text sequence of a to-be-dubbed voice of the target object in a target language based on the to-be-dubbed voice of the target object in a target video; the language of the to-be-dubbed voice is different from the target language; a feature synthesis module configured to perform voice feature synthesis processing based on the timbre feature and the text sequence to obtain a candidate voice feature sequence in the target language; and a voice synthesis module configured to perform voice synthesis processing based on a mouth shape vector sequence of the target object and the candidate voice feature sequence to obtain a target voice of the to-be-dubbed voice in the target language; the mouth shape vector sequence is used to indicate a mouth shape change when the target object speaks the to-be-dubbed voice.

[0006] In a third aspect, the embodiments of the present application provide an electronic device, which includes a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

[0007] In a fourth aspect, the embodiments of the present application provide a computer-readable storage medium, which stores program codes, wherein the program codes are executed by a processor to perform the above method.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the electronic device to perform the method described above.

[0009] The voice determination method and device, the electronic device, and the storage medium provided in the embodiments of the present application first determine a timbre feature indicating a timbre of a target object and a text sequence of a to-be-dubbed voice in a target language based on the to-be-dubbed voice of the target object in a target video, perform voice feature synthesis based on the timbre feature and the text sequence to obtain a candidate voice feature sequence in the target language, and then perform voice synthesis processing based on a mouth shape vector sequence of the target object and the candidate voice feature sequence to obtain a target voice of the to-be-dubbed voice in the target language. In this way, the target voice in the target language is automatically generated based on the to-be-dubbed voice, and a dubbing actor is not needed to dub, thereby saving the dubbing process of the dubbing actor, improving dubbing efficiency, saving the labor cost of the dubbing actor, and performing voice synthesis processing by referring to the mouth shape vector sequence of the target object and the candidate voice feature sequence, so that the target voice in the target language is consistent with the to-be-dubbed voice of the target object in the target video, the accuracy of the target voice is better, and the dubbing effect is improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, the drawings can also be obtained without creative labor. Figure 1 A flowchart of a voice determination method according to an embodiment of the present application is shown; Figure 2 A flowchart of steps before step S120 in the embodiment of Figure 1 A flowchart of steps before step S130 in the embodiment of Figure 3 A flowchart of steps before step S130 in the embodiment of Figure 1 A flowchart of steps before step S130 in the embodiment of Figure 4 A block diagram of a voice determination device according to an embodiment of the present application is shown; Figure 5 A structural block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0011] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. According to the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0012] In the following description, the terms "first\second" are only to distinguish similar objects, and do not represent the specific order of the objects. It can be understood that "first\second" can be exchanged in specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0014] Please refer to Figure 1 , Figure 1 A flowchart of a voice determination method according to an embodiment of the present application is shown, the method can be used in an electronic device, and the method comprises: S110, based on the to-be-dubbed voice of the target object in the target video, determining the timbre feature of the target object and the text sequence of the to-be-dubbed voice in the target language.

[0015] Wherein, the language of the to-be-dubbed voice (in order to distinguish, the language in the to-be-dubbed voice is the source language) is different from the target language. In the present application, the language can refer to the language of a certain independent language, or different dialects under the same language, for example, the language can be Chinese, English, French and Spanish, and for example, the language can be northeast dialect, Sichuan dialect and Cantonese, etc.

[0016] The target video can refer to a video to be dubbed, and the target video can include at least one to-be-dubbed voice. The to-be-dubbed voice is the to-be-dubbed voice in the present application, and the object speaking the to-be-dubbed voice in the target video is the target object. The target object can be a real human or a virtual character, for example, the target object can be a virtual dog or a virtual gunman, etc.

[0017] In the present application, the timbre feature of the target object can be extracted based on the to-be-dubbed voice; the to-be-dubbed voice is processed by voice-to-text to obtain the initial text sequence corresponding to the to-be-dubbed voice; and the initial text sequence is translated to obtain the text sequence in the target language.

[0018] Firstly, the timbre features of the to-be-voiced speech can be extracted based on the to-be-voiced speech by a pre-trained timbre feature extraction model (for example, S-Vector, d-Vector or X-Vector, etc.), and the extracted features are taken as the timbre features of the target object. The timbre features of the target object can indicate the timbre and the speech characteristics (for example, including the speech speed and the intonation, etc.) of the target object, so that the generated target speech is consistent with the speaking manner of the target object, and the obtained target speech is more accurate.

[0019] The to-be-voiced speech can be converted into text by a speech recognition technology (Automatic Speech Recognition, ASR), and a sequence of original texts of the to-be-voiced speech (a sequence of texts arranged in the speaking order of the target object) is obtained. Then, the original texts are translated into the target language, and a text sequence in the target language is obtained. At this time, the text sequence is the text sequence of the original texts of the to-be-voiced speech translated into the target language and arranged in the speaking order of the target language. The machine translation technology or the Large Language Model (LLM) can be used to realize the translation.

[0020] In S120, speech feature synthesis processing is performed based on the timbre features and the text sequence, and a candidate speech feature sequence in the target language is obtained.

[0021] After obtaining the timbre features of the target object and the text sequence, speech feature synthesis can be performed based on the timbre features of the target object and the text sequence, and a candidate speech feature sequence is obtained. The candidate speech feature sequence is a feature representation of the content in the to-be-voiced speech spoken by the target object in the target language.

[0022] In this application, the text sequence can be encoded to obtain an encoded text feature sequence. Then, the encoded text feature sequence and the timbre features are fused to obtain a candidate speech feature sequence in the target language.

[0023] For example, the text sequence can be encoded by a text encoder (Text Encoder) to obtain an encoded text feature sequence. At this time, the feature corresponding to one text (or phoneme) in the encoded text feature sequence is taken as a frame of speech feature. Of course, in this application, in order to support different phoneme and character systems of different languages, the text encoder can be a unified phoneme embedding or use a language ID as an additional input vector.

[0024] After that, the timbre features are linearly transformed through a linear layer to obtain linearly transformed timbre features, and the linear transformation processing is fused with the encoded text feature sequence to obtain a candidate speech feature sequence in the target language, so that the speaking manner in the speech decoded from the obtained candidate speech feature sequence is consistent with the target object. The fusion here can include channel splicing, addition operation or additional conditional attention mechanism, which is not limited in the present application.

[0025] In S130, speech synthesis processing is performed based on the mouth shape vector sequence of the target object and the candidate speech feature sequence to obtain target speech of the to-be-dubbed speech in the target language.

[0026] The mouth shape vector sequence is used to indicate the mouth shape change when the target object speaks the to-be-dubbed speech, and the mouth shape vector sequence can be determined based on the to-be-dubbed speech. The mouth shape can be predicted based on the to-be-dubbed speech by a pre-trained mouth shape prediction model to obtain the mouth shape vector sequence corresponding to the to-be-dubbed speech.

[0027] Alternatively, the video segments in the target video that include the to-be-dubbed audio can be frame-processed by a video frame window to obtain frame-processed video segments (each video segment is arranged in chronological order), and then the mouth shape (Lip Shape) representation of the video segment is extracted by face detection and mouth key point positioning technology (such as a convolutional network or key point model) to serve as the mouth shape vector of each video segment (the mouth shape vector corresponding to a video segment as a frame mouth shape vector). The mouth shape vectors of each video segment are arranged in chronological order according to the order of each video segment in the target video, and the obtained sequence is the mouth shape vector sequence. The duration of the video frame window can be set based on requirements, for example, the video frame window is 20ms or 40ms, etc. For example, when the video is frame-processed into T video segments, the obtained mouth shape vector sequence is {L1, L2, …, LT} The mouth shape vector sequence and the candidate speech feature sequence can be fused to obtain a fusion feature sequence, and then a vocoder is used to perform speech synthesis based on the fusion feature sequence to obtain target speech in the target language. The fusion here can include channel splicing, addition operation or additional conditional attention mechanism, and the vocoder can be a pre-trained HiFi-GAN, WaveGlow, etc. At this time, the obtained target speech includes the timbre and speech characteristics of the target object.

[0028] In some embodiments, the candidate speech feature sequence can be aligned based on the mouth shape vector sequence to obtain a target speech feature sequence in the target language, and speech generation is performed based on the target speech feature sequence to obtain target speech in the target language.

[0029] Since the mouth shape vector sequence and the candidate speech feature sequence can not be accurately aligned, when dubbing the target video according to the generated target speech, there is a problem of audio-visual asynchrony. Therefore, the mouth shape vector sequence needs to be aligned with the candidate speech feature sequence, so that when dubbing the target video according to the generated target speech, the audio-visual synchronization degree of the dubbed video is high.

[0030] In some embodiments, the candidate speech feature sequence can be interpolated or compressed based on the mouth shape vector sequence, so that the number of mouth shape vectors in the mouth shape vector sequence is consistent with the number of speech features in the candidate speech feature sequence (one frame of mouth feature corresponds to one frame of speech feature), the mouth shape vector sequence and the candidate speech feature sequence are aligned, and the target speech feature sequence in the target language is obtained.

[0031] In yet some embodiments, the candidate speech feature sequence can be adjusted based on the mouth shape vector sequence through an attention mechanism, so that the candidate speech feature sequence is aligned with the mouth shape vector sequence in terms of time length and / or factor boundary dimension, and the target speech feature sequence in the target language is obtained.

[0032] That is, through the attention mechanism (self-attention mechanism, cross-attention mechanism, multi-head attention mechanism, etc.), the candidate speech feature sequence and the mouth shape vector sequence are operated based on the mouth shape vector sequence and the candidate speech feature sequence, so as to align the candidate speech feature sequence with the mouth shape vector sequence in terms of time length and / or factor boundary dimension, and the result of the operation is obtained as the target speech feature sequence in the target language. Wherein, the query vector Q can be determined based on the mouth shape vector sequence, the key vector K and the value vector V can be determined based on the candidate speech feature sequence, and the operation of the attention mechanism is performed based on the determined QKV, and the target speech feature sequence in the target language is obtained.

[0033] After obtaining the target speech feature sequence in the target language, the target speech is directly synthesized through a vocoder based on the aligned target speech feature sequence.

[0034] After obtaining the target speech, the target speech can be inserted into the target video as the dubbing soundtrack of the target video, so as to realize dubbing of the target object in the target video, and the dubbed speech is consistent with the speaking tone or personality of the target object.

[0035] In this embodiment, based on the to-be-dubbed speech of the target object in the target video, the timbre feature indicating the timbre of the target object and the text sequence of the to-be-dubbed speech in the target language are determined, and the speech feature synthesis is performed based on the timbre feature and the text sequence to obtain the candidate speech feature sequence in the target language. Then, the speech synthesis processing is performed based on the mouth shape vector sequence of the target object and the candidate speech feature sequence to obtain the target speech of the to-be-dubbed speech in the target language. In this way, the target speech in the target language is automatically generated based on the to-be-dubbed speech, and the dubbing actor is not needed to dub, thereby saving the dubbing process of the dubbing actor, improving the dubbing efficiency, saving the labor cost of the dubbing actor, and performing the speech synthesis processing by referring to the mouth shape vector sequence of the target object and the candidate speech feature sequence, so that the target speech in the target language is consistent with the to-be-dubbed speech of the target object in the target video, the accuracy of the target speech is better, and the dubbing effect is improved.

[0036] Secondly, the candidate speech feature sequence is generated based on the timbre feature indicating the timbre of the target object, so that the target speech generated based on the candidate speech feature sequence is consistent with the speaking timbre or personality of the target object, the target speech is more accurate, and the dubbing effect is better.

[0037] In some embodiments, S120 is implemented by a speech synthesis model (also called a TTS (Text-to-Speech) model), that is, the speech synthesis model is used to obtain the candidate speech feature sequence (indicating the features of the speech) in the target language based on the text (text sequence) in the target language, as shown in Figure 2 Before S120, the method further includes: S210, obtaining a first sample speech of a first sample object in a first sample language and a labeled text sequence corresponding to the first sample speech.

[0038] The first sample language includes the target language. That is, in order to enable the speech synthesis model to implement the text-to-speech feature conversion of the target language, the samples for training the speech synthesis model need to include the target language, so that the speech synthesis model has high text-to-speech feature conversion capability for the target language and has good conversion effect.

[0039] The first sample object refers to an object that speaks the first sample speech according to the first sample language, and the labeled text sequence refers to a text sequence obtained by performing speech-to-text processing on the first sample speech. The speech-to-text processing can be performed on the first sample speech by using a speech recognition technology to obtain the labeled text sequence corresponding to the first sample speech (a sequence in which texts are arranged in the speaking order of the first sample object).

[0040] S220, determine the first sample timbre feature of the first sample object and the first labeled speech feature sequence of the first sample object in the first sample language based on the first sample speech.

[0041] The timbre feature extraction model can be pre-trained, and the timbre feature can be extracted based on the first sample speech to obtain the extracted feature as the first sample timbre feature of the first sample object. The first sample timbre feature of the first sample object can indicate the timbre and the speech characteristics (such as including the speech speed and the tone, etc.) of the first sample object.

[0042] The feature extraction process includes endpoint detection, speech framing window framing processing, and acoustic feature extraction (such as Mel spectrum or linear spectrum), so as to take the extracted acoustic feature as the correct speech feature sequence of the first sample speech in the first sample language-the first labeled speech feature sequence. Wherein, the speech framing window is 20ms or 40ms, etc.

[0043] That is, the starting point and the ending point of the speech are determined in the first sample speech by endpoint detection, and then the speech segment between the starting point and the ending point is divided into multiple ordered speech segments by framing window framing processing, and then the acoustic feature of each speech segment is extracted to obtain the acoustic feature of each speech segment. The acoustic features of the multiple speech segments are arranged in the order of the speech segments in the first sample speech, and the obtained sequence is taken as the first labeled speech feature sequence. At this time, the first labeled speech feature sequence can accurately indicate the first sample speech.

[0044] S230, perform speech feature synthesis processing based on the first sample timbre feature and the labeled text sequence through the speech feature synthesis model to obtain the first sample speech feature sequence in the first sample language.

[0045] In this embodiment, the speech feature synthesis model can adopt the Transformer structure, and the speech feature synthesis model can include a text encoder (which can adopt the Transformer structure) and a speaker adaptation network layer (which can adopt the Transformer structure). Thus, the text encoder in the speech feature synthesis model encodes the labeled text sequence to obtain a labeled encoded text feature sequence; and then the speaker adaptation network layer in the speech feature synthesis model fuses the labeled encoded text feature sequence and the first sample timbre feature to obtain the first sample speech feature sequence in the first sample language.

[0046] The speaker adaptation network layer may include a linear mapping layer, which performs a linear transformation on the first sample timbre features to obtain the first sample linearly transformed timbre features, and fuses the first sample linearly transformed timbre features with the annotated coded text feature sequence to obtain the fused result as the first sample speech feature sequence in the first sample language. The fusion here may include channel-based concatenation, addition, or the use of an additional conditional attention mechanism, which is not limited in this application.

[0047] S240: Training a speech synthesis model based on the difference between the first sample speech feature sequence and the first annotated speech feature sequence.

[0048] Based on the difference between the first sample speech feature sequence and the first labeled speech feature sequence, a loss value can be determined through L1 loss function, L2 loss function, cross entropy loss function, etc. as the first loss value, and then the speech synthesis model is trained through the first loss value so that the speech synthesis model learns the ability to convert text into speech.

[0049] Of course, in order to improve the capabilities of the speech synthesis model, the first sample languages ​​involved may include multiple different languages, so that the speech synthesis model can learn the text-to-speech conversion capabilities for different languages. Moreover, the first sample speech may also include the speech of the same first sample subject in different first sample languages, so that the speech synthesis model can learn the text-to-speech conversion capabilities of the same person in different languages. In addition, there may also be multiple first sample subjects, so that the speech synthesis model can learn the speaking habits of different subjects and more accurately achieve text-to-speech conversion for different subjects.

[0050] After multiple rounds of iterative training, the speech synthesis model learned to output speech in the target language with the timbre characteristics of any speaker, given text in any language and the timbre characteristics of any speaker. The trained speech synthesis model can, given the timbre characteristics of user s1 and a text sequence in a language l1, generate a feature representation—a speech feature sequence—of the speech that would indicate user s1 speaking the text sequence in that language l1.

[0051] In this embodiment, by training the speech synthesis model, the speech synthesis model can have a better ability to convert text into speech, so that when the speech synthesis model performs speech feature synthesis based on the above-mentioned timbre features and text sequence, the output candidate speech feature sequence is more accurate and fits the timbre of the target object, so that the final target speech is more accurate, can fit the timbre of the target object, and the dubbing effect is better.

[0052] In some embodiments, S130 is determined by a dubbing model, that is, the dubbing model is used for speech synthesis processing based on the mouth shape vector sequence of the target object and the candidate speech feature sequence, and the to-be-dubbed speech is the target speech of the target language, such as Figure 3 As shown, the training process of the dubbing model includes: S310, based on the second sample speech of the second sample object in the sample video, determining the second sample timbre feature of the second sample object and the sample text sequence of the second sample speech in the second sample language.

[0053] The second sample language includes the target language.

[0054] The sample video can refer to a video including speech for training the dubbing model, the speech in the sample video as the second sample speech, the object speaking the second sample speech in the sample video as the second sample object, and the second sample object can be a real human or a virtual character, etc. For example, the second sample object can be a virtual dog or a virtual gunman, etc.

[0055] In the present application, the timbre feature can be extracted based on the second sample speech to obtain the second sample timbre feature corresponding to the second sample object; the speech-to-text processing is performed based on the second sample speech to obtain the initial sample text sequence corresponding to the second sample speech; and the initial text sequence is translated to obtain the sample text sequence in the second sample language. The language of the second sample speech and the second sample language can be different to realize the translation of the text.

[0056] Firstly, the timbre feature can be extracted based on the second sample speech by using the pre-trained timbre feature extraction model, and the extracted feature is used as the second sample timbre feature corresponding to the second sample object. The second sample timbre feature of the second sample object can indicate the timbre and the speech characteristics (such as including the speech speed and the tone, etc.) of the second sample object.

[0057] The speech-to-text processing can be performed on the second sample speech by using the speech recognition technology to obtain the sequence of the original text of the second sample speech (the sequence arranged according to the speaking order of the second sample object), and then the sequence of the original text is used as the second sample language to obtain the sample text sequence in the second sample language. At this time, the sample text sequence is the text translated from the original text of the second sample speech into the second sample language, arranged according to the speaking order of the second sample language.

[0058] S320, based on the second sample timbre feature and the sample text sequence, performing speech feature synthesis processing to obtain the second sample speech feature sequence in the second sample language.

[0059] After obtaining the second sample timbre feature of the second sample object and the sample text sequence, speech feature synthesis can be performed based on the second sample timbre feature and the sample text sequence to obtain a second sample speech feature sequence in a second sample language, wherein the second sample speech feature sequence is a feature representation of the content in the second sample speech spoken by the second sample object in the second sample language.

[0060] In this application, the sample text sequence can be encoded to obtain a second sample encoded text feature sequence; then, the second sample encoded text feature sequence and the second sample timbre feature are fused to obtain a second sample speech feature sequence in a second sample language.

[0061] For example, the sample text sequence can be encoded by a text encoder in the speech synthesis model to obtain a second sample encoded text feature sequence, and at this time, the feature corresponding to one text (or phoneme) in the second sample encoded text feature sequence is taken as a frame of speech feature.

[0062] Then, the linear layer included in the speaker adaptation network layer in the speech synthesis model performs linear transformation processing on the second sample timbre feature to obtain a second sample linearly transformed timbre feature, and fuses the second sample linearly transformed timbre feature with the second sample encoded text feature sequence to obtain a second sample speech feature sequence in a second sample language, so that the speaking manner in the speech decoded from the obtained second sample speech feature sequence is consistent with the second sample object. The fusion here can include channel splicing, addition operation or additional conditional attention mechanism, which is not limited in this application.

[0063] After obtaining the second sample speech feature sequence, speech synthesis can be performed based on the second sample speech feature sequence by a vocoder to obtain a synthesized speech as an initial paired sample speech paired with the second sample speech. The initial paired sample speech is consistent with the content of the second sample speech, but the language is different.

[0064] However, the initial paired sample speech and the sample video may not have achieved sound and picture synchronization, so it is still necessary to align the initial paired sample speech and the sample video based on the sample mouth shape vector sequence to align the initial paired sample speech and the sample mouth shape vector sequence in time length and / or in factor boundary dimension to obtain an aligned speech as a paired sample speech; wherein the sample mouth shape vector sequence is used to indicate the mouth shape change of the second sample object when speaking the second sample speech, and the sample mouth shape vector sequence can be determined based on the second sample speech. The determination process of the sample mouth shape vector sequence is as described above in the process of determining the mouth shape vector sequence based on the target speech, which will not be repeated here.

[0065] In the present application, the second sample voice can be extracted from the sample video (s) according to actual needs, thereby obtaining a plurality of second sample objects S = {s1, s2, …, sN}, and each second sample object si extracts the timbre feature s-i from the second sample voice.

[0066] The initial sample text sequence T = {T1, T2, …, TN} is obtained by extracting the text sentence from each second sample voice. Of course, in order to improve the applicability of the trained dubbing model, the initial sample text sequence extracted from the sample video can cover a variety of common scenes (such as greetings, commonly used short sentences, short speech paragraphs, etc.), and ensure that it has consistent meaning in multiple languages.

[0067] Each sample text sequence Tj can be translated into a plurality of second sample languages (for example, K), thereby obtaining a large number of sample text sequences , thereby obtaining the sample text sequence set of the corresponding multi-language (K second sample languages) At this time, the obtained multi-language sample text sequence set includes KxN sample text sequences.

[0068] For each second sample object si, each sample text sequence Tj (assuming it is a Chinese text), and each second sample language , taking as the input, the trained voice synthesis model is called to synthesize the corresponding paired sample voice of the speaker si in language .

[0069] It is worth mentioning that when is consistent with the language of the second sample voice (consistent before and after translation), a paired sample voice can also be synthesized, which can be used for subsequent data enhancement or verification of the self-loop consistency of the voice synthesis model; for example, in the case where the paired sample voice synthesized by the voice synthesis model is consistent with the language of the original second sample voice, the self-loop consistency effect of the voice synthesis model is verified based on the difference between the synthesized paired sample voice and the second sample voice, whether the synthesis effect of the voice synthesis model is qualified or meets the standard.

[0070] Finally, a complete parallel corpus The corpus includes pairs of speech of the same second sample object si in different second sample languages (including the original second sample speech and the paired sample speech), and the content is semantically consistent at the text level in multiple languages, meeting the "parallel corpus" requirement of the dubbing model training process, achieving the goal of one second sample object having one second sample speech corresponding to multiple paired sample speech in different second sample languages, and obtaining a large amount of corpus, which is used for the training process of the dubbing model.

[0071] In S330, a speech synthesis process is performed on the sample mouth shape vector sequence and the second sample speech feature sequence based on the dubbing model to obtain a sample target speech in the second sample language.

[0072] The sample mouth shape vector sequence and the second sample speech feature sequence can be fused to obtain a sample fusion feature sequence, and then a vocoder is used to perform speech synthesis based on the sample fusion feature sequence to obtain a sample target speech in the second sample language. The fusion can include channel splicing, addition operation, or additional conditional attention mechanism, etc. At this time, the sample target speech obtained includes the timbre and tone characteristics of the second sample object.

[0073] In some embodiments, the second sample speech feature sequence can be aligned based on the sample mouth shape vector sequence to obtain a sample target speech feature sequence in the second sample language; and speech generation is performed based on the sample target speech feature sequence to obtain a sample target speech in the second sample language.

[0074] In some embodiments, the second sample speech feature sequence can be interpolated or compressed based on the sample mouth shape vector sequence, so that the number of mouth shape vectors in the sample mouth shape vector sequence is consistent with the number of speech features in the second sample speech feature sequence (one frame of mouth feature corresponds to one frame of speech feature), so as to align the sample mouth shape vector sequence and the second sample speech feature sequence, and obtain a sample target speech feature sequence in the second sample language.

[0075] In yet some embodiments, the second sample speech feature sequence can be adjusted based on the sample mouth shape vector sequence through an attention mechanism to align the second sample speech feature sequence and the sample mouth shape vector sequence in terms of time length and / or factor boundary dimension, and obtain a sample target speech feature sequence in the second sample language.

[0076] That is, through the attention mechanism (self-attention mechanism, cross-attention mechanism, multi-head attention mechanism, etc.), the second sample speech feature sequence is aligned with the sample mouth shape vector sequence in time length and / or in factor boundary dimension based on the sample mouth shape vector sequence and the second sample speech feature sequence, and the result after operation is obtained as the sample target speech feature sequence in the second sample language. Wherein, the query vector Q can be determined based on the sample mouth shape vector sequence, the key vector K and the value vector V are determined based on the second sample speech feature sequence, and the attention mechanism operation is performed based on the determined QKV, to obtain the sample target speech feature sequence in the second sample language.

[0077] By introducing the attention mechanism into the dubbing model, the dubbing model will adaptively learn the correspondence between vision and acoustics during training, without manually setting the time length prediction or frame insertion rules, thereby realizing natural and smooth speech and mouth shape synchronization effect.

[0078] Of course, after obtaining the sample target speech feature sequence, the vocoder performs speech synthesis based on the sample target speech feature sequence to obtain the sample target speech in the second sample language.

[0079] S340, based on the difference between the predicted sample speech feature sequence extracted from the sample target speech and the second labeled speech feature sequence, the difference between the time length of the second sample speech and the time length of the sample target speech, and the similarity between the predicted sample timbre feature extracted from the sample target speech and the second sample timbre feature, the dubbing model is trained.

[0080] Wherein, the second labeled speech feature sequence is determined based on the paired sample speech, and the paired sample speech is determined based on the second sample speech feature sequence. The process of extracting the second labeled speech feature sequence A_t^tgt from the paired sample speech is referred to the process of extracting the first labeled speech feature sequence from the first sample speech, which is not described here.

[0081] The sample target speech can be sequentially processed by endpoint detection, speech framing window framing, and extraction of acoustic features (such as Mel spectrum or linear spectrum) to extract acoustic features, and the extracted acoustic features are the predicted sample speech feature sequence Ã_t^tgt, which is the feature representation of the speech generated by the dubbing model.

[0082] And through the pre-trained timbre feature extraction model, the timbre feature is extracted based on the sample target speech, and the extracted feature is obtained as the predicted sample timbre feature.

[0083] Thus, the loss value can be determined based on the difference between the predicted sample voice feature sequence extracted from the sample target voice and the second sample voice feature sequence, the difference between the length of the second sample voice and the length of the sample target voice, and the similarity between the predicted sample timbre feature extracted from the sample target voice and the second sample timbre feature, and the voice dubbing model is trained through the loss value.

[0084] Alternatively, the loss value can be determined based on the difference between the predicted sample voice feature sequence extracted from the sample target voice and the second labeled voice feature sequence through an L1 loss function, an L2 loss function, and a cross-entropy loss function, as a second loss value; the difference between the length of the second sample voice and the length of the sample target voice through an L1 loss function, an L2 loss function, and a cross-entropy loss function, as a third loss value; and the similarity (such as cosine similarity, Euclidean distance, and Manhattan distance) between the predicted sample timbre feature extracted from the sample target voice and the second sample timbre feature s orig, as a fourth loss value. Then, the second loss value, the third loss value, and the fourth loss value are summed or weighted summed to obtain a total loss value, and then the voice dubbing model is trained through the total loss value.

[0085] In some embodiments, S330 can further include determining a predicted sample mouth shape vector sequence corresponding to the sample target voice based on the sample target voice; and training the voice dubbing model based on the difference between the predicted sample voice feature sequence extracted from the sample target voice and the second labeled voice feature sequence, the difference between the length of the second sample voice and the length of the sample target voice, the similarity between the predicted sample timbre feature extracted from the sample target voice and the second sample timbre feature, and the difference between the sample mouth shape vector sequence and the predicted sample mouth shape vector sequence.

[0086] The predicted sample mouth shape vector sequence corresponding to the sample target voice can be obtained by predicting the mouth shape based on the sample target voice through a pre-trained mouth shape prediction model, and then determining the loss value based on the difference between the sample mouth shape vector sequence and the predicted sample mouth shape vector sequence through an L1 loss function, an L2 loss function, and a cross-entropy loss function, as a fifth loss value. Then, the second loss value, the third loss value, the fourth loss value, and the fifth loss value are summed or weighted summed to obtain a total loss value, and then the voice dubbing model is trained through the total loss value.

[0087] ​As described above, for each second sample language, each second sample voice can correspond to a paired sample voice in the second sample language, and therefore, for each second sample voice, the difference between the predicted sample voice feature sequence extracted from the sample target voice and the second labeled voice feature sequence, the difference between the time length of the second sample voice and the time length of the sample target voice, the similarity between the predicted sample timbre feature extracted from the sample target voice and the second sample timbre feature, and the difference between the sample lip vector sequence and the predicted sample lip vector sequence can be determined based on the second sample voice, and a loss can be determined. The losses of the second sample voices in the second sample languages are summed to obtain a total loss, and the dubbing model is trained by the total loss. In this way, the dubbing model is trained by the second sample voices in the second sample languages, the dubbing model is comprehensively trained, and the training effect of the dubbing model is improved.

[0088] In this embodiment, the dubbing model is trained to have better dubbing capability, so that when the dubbing model performs voice synthesis processing based on the lip vector sequence of the target object and the candidate voice feature sequence, the output target voice is more accurate.

[0089] Moreover, the difference between the time length of the second sample voice and the time length of the sample target voice and the similarity between the predicted sample timbre feature extracted from the sample target voice and the second sample timbre feature are introduced in the training process, so that when the dubbing model is trained, the dubbing model can learn the time length alignment capability and the timbre restoration capability, so that the time length of the target voice generated by the dubbing model is closer to the time length of the voice to be dubbed, and the voice timbre in the target voice generated by the dubbing model is close to the voice to be dubbed, and the dubbing effect is better.

[0090] Furthermore, the difference between the sample lip vector sequence and the predicted sample lip vector sequence is also introduced to the dubbing model, so that when the dubbing model is trained, the dubbing model can learn the alignment capability of voice and lip, so that the target voice generated by the dubbing model can be more synchronized with the video picture of the target video, and the audio-visual synchronization is further improved.

[0091] Moreover, the multilingual voice synthesis model and the dubbing model only need to be fine-tuned, and the multilingual voice synthesis model and the dubbing model can be extended to more languages and more speaker scenarios. For a new speaker, only the single voice sample of the speaker is needed to generate a cross-language voice by the multilingual voice synthesis model, and the cross-language voice is used for fine-tuning of the dubbing model, which greatly improves the dubbing efficiency.

[0092] Please refer to Figure 4 , Figure 4 A block diagram of a voice determination device according to an embodiment of the present application is shown, and the device 700 includes: The determining module 710 is configured to determine a timbre feature of the target object and a text sequence of the to-be-synthesized speech in the target language based on the to-be-synthesized speech of the target object in the target video, wherein the language of the to-be-synthesized speech is different from the target language. The feature synthesizing module 720 is configured to perform speech feature synthesis processing based on the timbre feature and the text sequence to obtain a candidate speech feature sequence in the target language. The speech synthesizing module 730 is configured to perform speech synthesis processing based on the mouth shape vector sequence of the target object and the candidate speech feature sequence to obtain target speech of the to-be-synthesized speech in the target language, wherein the mouth shape vector sequence is used to indicate mouth shape changes of the target object when speaking the to-be-synthesized speech.

[0093] Optionally, the feature synthesizing module 720 is further configured to perform encoding processing on the text sequence to obtain an encoded text feature sequence, and fuse the encoded text feature sequence and the timbre feature to obtain the candidate speech feature sequence in the target language.

[0094] Optionally, the speech feature synthesis processing is implemented through a speech synthesis model; the apparatus further includes a training module configured to obtain a first sample speech of a first sample object in a first sample language and a labeled text sequence corresponding to the first sample speech; the first sample language includes the target language; determine a first sample timbre feature of the first sample object and a first labeled speech feature sequence of the first sample object in the first sample language based on the first sample speech; perform speech feature synthesis processing through the speech synthesis model based on the first sample timbre feature and the labeled text sequence to obtain a first sample speech feature sequence in the first sample language; and train the speech synthesis model based on a difference between the first sample speech feature sequence and the first labeled speech feature sequence.

[0095] Optionally, the speech synthesizing module 730 is further configured to perform alignment processing on the candidate speech feature sequence based on the mouth shape vector sequence to obtain a target speech feature sequence in the target language; and perform speech generation based on the target speech feature sequence to obtain the target speech in the target language.

[0096] Optionally, the speech synthesizing module 730 is further configured to adjust the candidate speech feature sequence based on the mouth shape vector sequence through an attention mechanism to align the candidate speech feature sequence and the mouth shape vector sequence in terms of time length and / or in a factor boundary dimension, so as to obtain the target speech feature sequence in the target language.

[0097] Optionally, the speech synthesis processing is determined through the dubbing model; the training module is further configured to determine, based on the second sample voice of the second sample object in the sample video, a second sample timbre feature of the second sample object and a sample text sequence of the second sample voice in a second sample language; the second sample language comprises the target language; perform speech feature synthesis processing based on the second sample timbre feature and the sample text sequence to obtain a second sample voice feature sequence in the second sample language; perform speech synthesis processing on the sample target voice in the second sample language through the dubbing model based on the sample mouth shape vector sequence of the second sample object and the second sample voice feature sequence; the sample mouth shape vector sequence is used to indicate the mouth shape change when the second sample object speaks the second sample voice; train the dubbing model based on the difference between the predicted sample voice feature sequence extracted from the sample target voice and the second labeled voice feature sequence, the difference between the duration of the second sample voice and the duration of the sample target voice, and the similarity between the predicted sample timbre feature extracted in the sample target voice and the second sample timbre feature; wherein the second labeled voice feature sequence is determined based on a paired sample voice, and the paired sample voice is determined based on the second sample voice feature sequence.

[0098] Optionally, the training module is further configured to determine, based on the sample target voice, a predicted sample mouth shape vector sequence corresponding to the sample target voice; train the dubbing model based on the difference between the predicted sample voice feature sequence extracted from the sample target voice and the second labeled voice feature sequence, the difference between the duration of the second sample voice and the duration of the sample target voice, the similarity between the predicted sample timbre feature extracted in the sample target voice and the second sample timbre feature, and the difference between the sample mouth shape vector sequence and the predicted sample mouth shape vector sequence.

[0099] Optionally, the determination module 710 is further configured to extract the timbre feature based on the voice to be dubbed to obtain the timbre feature of the target object; perform speech-to-text processing on the voice to be dubbed to obtain an initial text sequence corresponding to the voice to be dubbed; and perform translation processing on the initial text sequence to obtain a text sequence in the target language.

[0100] It should be noted that the device embodiments in the present application correspond to the foregoing method embodiments, and the specific principles in the device embodiments can be referred to the foregoing method embodiments, which will not be described here.

[0101] Figure 5 A structural block diagram of an electronic device is shown, which is used to execute the voice determination method according to an embodiment of the present application. As shown in FIG. 7, the electronic device comprises a processor 710 and a memory 720. The memory 720 stores a computer program. The processor 710 is configured to execute the computer program stored in the memory 720 to perform the voice determination method according to an embodiment of the present application. Figure 5As shown, the electronic device 1200 includes a central processing unit (CPU) 1201 which can perform various appropriate actions and processes in accordance with a program stored in a read-only memory (ROM) 1202 or a program loaded from the storage section 1208 into a random access memory (RAM) 1203, such as executing the methods in the above-described embodiments. In the RAM 1203, various programs and data required for system operation are also stored. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0102] Connected to the I / O interface 1205 are an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as necessary. A removable recording medium 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1210 as necessary, so that a computer program read therefrom is installed into the storage section 1208 as necessary.

[0103] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from the removable recording medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are performed.

[0104] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, transmit, propagate or transport a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0105] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0106] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0107] As another aspect, the present application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The above computer readable storage medium carries computer readable instructions, which, when executed by a processor, implement the method in any of the above embodiments.

[0108] According to an aspect of the embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the electronic device to perform the method in any of the above embodiments.

[0109] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into modules or units embodied by multiple modules or units.

[0110] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or network, and includes a number of instructions to enable a computing device (which can be a personal computer, server, touch terminal, or network device, etc.) to perform the methods according to the embodiments of the present application.

[0111] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims. Various modifications and changes can be made thereto by those skilled in the art which fall within the scope of the application as known from the before-described teachings. The scope of the application is not to be limited by any of the foregoing description but is only to be limited by the following claims.

[0112] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing and illustrating, not limiting, the technical solutions of the present application; even though the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions out of the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A voice determination method characterized by, The method comprises: determining a timbre feature of the target object and a text sequence of the to-be-dubbed speech in a target language based on the to-be-dubbed speech of the target object in the target video; the language of the to-be-dubbed speech is different from the target language; performing speech feature synthesis processing based on the timbre feature and the text sequence to obtain a candidate speech feature sequence in the target language; performing speech synthesis processing based on the mouth shape vector sequence of the target object and the candidate speech feature sequence to obtain target speech of the to-be-dubbed speech in the target language; the mouth shape vector sequence is used to indicate the mouth shape change when the target object speaks the to-be-dubbed speech.

2. The method of claim 1, wherein, The speech feature synthesis processing based on the timbre feature and the text sequence to obtain a candidate speech feature sequence in the target language comprises: performing encoding processing on the text sequence to obtain an encoded text feature sequence; fusing the encoded text feature sequence and the timbre feature to obtain a candidate speech feature sequence in the target language.

3. The method of claim 1, wherein, The speech feature synthesis processing is implemented through a speech synthesis model; Before the speech feature synthesis processing based on the timbre feature and the text sequence to obtain a candidate speech feature sequence in the target language, the method further comprises: obtaining a first sample speech of a first sample object in a first sample language and a labeled text sequence corresponding to the first sample speech; the first sample language comprises the target language; determining a first sample timbre feature of the first sample object and a first labeled speech feature sequence of the first sample object in the first sample language based on the first sample speech; performing speech feature synthesis processing on the first sample timbre feature and the labeled text sequence based on the speech feature synthesis model to obtain a first sample speech feature sequence in the first sample language; training the speech synthesis model based on the difference between the first sample speech feature sequence and the first labeled speech feature sequence.

4. The method of claim 1, wherein, The speech synthesis processing based on the mouth shape vector sequence of the target object and the candidate speech feature sequence to obtain target speech of the to-be-dubbed speech in the target language comprises: performing alignment processing on the candidate speech feature sequence based on the mouth shape vector sequence to obtain a target speech feature sequence in the target language; generating speech based on the target speech feature sequence to obtain target speech in the target language.

5. The method of claim 4, wherein, The alignment processing on the candidate speech feature sequence based on the mouth shape vector sequence to obtain a target speech feature sequence in the target language comprises: adjusting the candidate speech feature sequence based on the mouth shape vector sequence through an attention mechanism to align the candidate speech feature sequence with the mouth shape vector sequence in terms of time length and / or in a factor boundary dimension to obtain a target speech feature sequence in the target language.

6. The method of claim 1, wherein, The speech synthesis processing is determined through a dubbing model; Before the speech synthesis processing is performed based on the mouth shape vector sequence of the target object and the candidate speech feature sequence, the method further includes: determining a second sample voice feature of a second sample object and a sample text sequence of the second sample voice in a second sample language based on a second sample voice of the second sample object in the sample video; the second sample language includes the target language; performing speech feature synthesis processing based on the second sample voice feature and the sample text sequence to obtain a second sample speech feature sequence in the second sample language; performing speech synthesis processing on the second sample object based on a sample mouth shape vector sequence of the second sample object and the second sample speech feature sequence by using the dubbing model to obtain a sample target voice in the second sample language; the sample mouth shape vector sequence is used to indicate the mouth shape change of the second sample object when speaking the second sample voice; training the dubbing model based on the difference between the predicted sample speech feature sequence extracted from the sample target voice and a second labeled speech feature sequence, the difference between the duration of the second sample voice and the duration of the sample target voice, and the similarity between the predicted sample voice feature extracted from the sample target voice and the second sample voice feature; wherein the second labeled speech feature sequence is determined based on a paired sample voice, and the paired sample voice is determined based on the second sample speech feature sequence.

7. The method of claim 6, wherein, The training of the dubbing model based on the difference between the predicted sample speech feature sequence extracted from the sample target voice and a second labeled speech feature sequence, the difference between the duration of the second sample voice and the duration of the sample target voice, and the similarity between the predicted sample voice feature extracted from the sample target voice and the second sample voice feature includes: determining a predicted sample mouth shape vector sequence corresponding to the sample target voice based on the sample target voice; training the dubbing model based on the difference between the predicted sample speech feature sequence extracted from the sample target voice and the second labeled speech feature sequence, the difference between the duration of the second sample voice and the duration of the sample target voice, the similarity between the predicted sample voice feature extracted from the sample target voice and the second sample voice feature, and the difference between the sample mouth shape vector sequence and the predicted sample mouth shape vector sequence.

8. The method of claim 1, wherein, The determining of the voice feature of the target object and the text sequence of the voice to be dubbed in the target language based on the voice to be dubbed of the target object in the target video includes: extracting the voice feature based on the voice to be dubbed to obtain the voice feature of the target object; performing speech-to-text processing based on the voice to be dubbed to obtain an initial text sequence corresponding to the voice to be dubbed; performing translation processing on the initial text sequence to obtain a text sequence in the target language.

9. An electronic device, comprising: includes: one or more processors; memory; one or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, the one or more programs configured to perform the method of any of claims 1-8.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program codes, and the program codes can be invoked by the processor to execute the method of any of claims 1-8.

Images