Dubbing additional recording method and device, storage medium and electronic equipment

By obtaining the target emotions and matching audio of the supplementary lines of the film and television drama, using pre-trained speech synthesis large models to generate high-quality supplementary audio, the problem of low efficiency of manual supplementary dubbing in film and television drama production is solved, and high-quality and naturalness are achieved High emotional consistency audio generation.

CN118942490APending Publication Date: 2024-11-12CHENGDU IQIYI INTELLIGENT INNOVATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411184772.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

During the production of film and television dramas, manual re-recording and dubbing work efficiency is low. The traditional TTS algorithm cannot meet the high requirements of voice nature, emotion and timbre consistency, resulting in long and low re-recording time, affecting project management and online time.

Method used

By obtaining the target emotions of the supplementary lines, matching the audio of the corresponding emotions in the dubbing database of the target voice actor, and using pre-trained speech synthesis large models, input lines and audio into the model to generate high-quality, emotionally consistent supplementary audio.

Benefits of technology

The work efficiency of supplementary dubbing is improved, ensuring that the generated audio is consistent with the original sound in terms of emotional and sound characteristics, and achieving natural and smooth high-quality supplementary recording effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942490A_ABST
    Figure CN118942490A_ABST
Patent Text Reader

Abstract

The invention relates to a dubbing additional recording method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring additional recorded lines and target dubbers of the additional recorded lines; a target emotion of the additional recorded lines is determined, a target audio is obtained from a dubbing database of the target dubbler according to the target emotion, and the emotion of the target audio is the same as the target emotion; obtaining a target speech synthesis large model of the target dubbing person; and inputting the additional recorded lines and the target audio into the target speech synthesis large model to obtain additional recorded audio of the additional recorded lines. The technical problem that the working efficiency of manual additional recording and dubbing in the movie and television play making process is too low is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent analysis technology, and in particular to a dubbing recording method, device, storage medium and electronic device. Background Art

[0002] The dubbing of current film and television dramas is divided into manual dubbing and AI (Artificial Intelligence) dubbing, and both dubbing modes rely on the input of manual dubbing actors. However, in the post-dubbing production process, there are many cases of missing dubbing, line polishing, etc. that require manual dubbing actors to re-dub. Statistics show that the average time period from re-recording feedback to the completion of re-recording by manual dubbing actors according to feedback is 3 days. For key projects with urgent launch time, this waiting time is unacceptable. For inventory projects, long-term and large-scale accumulation of re-recording is not conducive to project management and reduces launch efficiency. Traditional TTS (text-to-speech) algorithm models can only receive text input and output natural speech, but the dubbing scene has very high requirements for speech naturalness, speech emotion, and dubbing timbre consistency, and traditional TTS algorithm models cannot meet this requirement. Summary of the invention

[0003] The present application provides a dubbing re-recording method, device, storage medium and electronic device to solve the technical problem of low efficiency of manual dubbing re-recording in the process of film and television drama production.

[0004] In a first aspect, the present application provides a method for supplementary dubbing, comprising: obtaining supplementary lines and a target dubbing actor for the supplementary lines; determining a target emotion for the supplementary lines, and obtaining a target audio from a dubbing database of the target dubbing actor based on the target emotion, wherein the emotion of the target audio is the same as the target emotion; obtaining a target speech synthesis model for the target dubbing actor; inputting the supplementary lines and the target audio into the target speech synthesis model to obtain the supplementary audio of the supplementary lines.

[0005] In a second aspect, the present application provides a dubbing supplementary recording device, comprising: a first acquisition module, used to obtain supplementary recording lines and a target dubbing actor for the supplementary recording lines; a determination module, used to determine the target emotion of the supplementary recording lines, and according to the target emotion, obtain the target audio in the dubbing database of the target dubbing actor, wherein the emotion of the target audio is the same as the target emotion; a second acquisition module, used to obtain a target speech synthesis large model of the target dubbing actor; a supplementary recording module, used to input the supplementary recording lines and the target audio into the target speech synthesis large model to obtain the supplementary recording audio of the supplementary recording lines.

[0006] As an optional example, the above-mentioned emotion includes an emotion category and an emotion degree, and the above-mentioned determination module includes: a first determination unit, used to determine the target emotion category and target emotion degree of the above-mentioned supplementary lines; a second determination unit, used to determine the emotion category and emotion degree of each audio in the above-mentioned dubbing database; a third determination unit, used to determine a first audio group from the above-mentioned dubbing database, wherein the emotion category of each audio in the above-mentioned first audio group is the same as the above-mentioned target emotion category; a fourth determination unit, used to determine the above-mentioned target audio from the above-mentioned first audio group, wherein the difference between the emotion degree of the above-mentioned target audio and the above-mentioned target emotion degree is minimal.

[0007] As an optional example, the fourth determination unit includes: a calculation subunit, used to calculate the difference between the emotional level of each audio in the first audio group and the target emotional level, wherein the difference value is the absolute value of the difference between each emotional level and the target emotional level; and a determination subunit, used to determine the audio with the smallest difference value in the first audio data group as the target audio.

[0008] As an optional example, the target speech synthesis large model includes a first target generation model and a second target generation model, and the device also includes: a third acquisition module, used to obtain a first training data set and a first verification data set of the target voice actor before obtaining the target speech synthesis large model of the target voice actor, wherein the first training data set includes multiple data pairs, one data pair consists of a pair of lines and audio with the same emotion, and the first verification data set includes a comprehensive semantic representation of each data pair; a first conversion module, used to input the first training data set into the first generation model to obtain a first generation result data set of the first training data set; a first calculation module, used to calculate the difference between the first generation result data set and the first verification data set using a loss function to obtain a first loss value; a first adjustment module, used to adjust the model parameters of the first generation model according to the first loss value, and recalculate the first loss value until the recalculated first loss value is less than the target threshold, so as to obtain the first target generation model.

[0009] As an optional example, the above-mentioned device also includes: a second conversion module, which is used to adjust the model parameters of the above-mentioned first generation model according to the above-mentioned loss value until the above-mentioned first loss value is less than the first threshold value, and the above-mentioned first target generation model is obtained, and then the above-mentioned first training data set is input into the above-mentioned first target generation model to obtain a second training data set; a fourth acquisition module, which is used to obtain a second verification data set of the above-mentioned target voice actor, wherein the above-mentioned second verification data set includes the audio corresponding to each line in the above-mentioned first training data set; a third conversion module, which is used to input the above-mentioned second training data set into the second generation model to obtain a second generation result data set of the above-mentioned second training data set; a second calculation module, which is used to calculate the difference between the above-mentioned second generation result data set and the above-mentioned second verification data set using the above-mentioned loss function to obtain a second loss value; a second adjustment module, which is used to adjust the model parameters of the above-mentioned second generation model according to the above-mentioned second loss value, and recalculate the above-mentioned second loss value until the recalculated second loss value is less than the above-mentioned target threshold value, and obtain the above-mentioned second target generation model.

[0010] As an optional example, the supplementary recording module includes: a first conversion unit, used to input the supplementary recording lines and the target audio into the first target generation model of the target speech synthesis model to convert the supplementary recording lines and the target audio into a comprehensive semantic representation; a second conversion unit, used to input the comprehensive semantic representation into the second target generation model of the target speech synthesis model to convert the comprehensive semantic representation into the supplementary recording audio.

[0011] As an optional example, the above-mentioned first conversion unit includes: a first encoding subunit, used to encode the above-mentioned supplementary lines into a set of text semantic representations; a second encoding subunit, used to encode the above-mentioned target audio into a set of audio semantic representations; and a third encoding subunit, used to combine the above-mentioned text semantic representation and the above-mentioned audio semantic representation to generate the above-mentioned comprehensive semantic representation.

[0012] In a third aspect, the present application provides a storage medium storing a computer program, wherein the computer program executes the above-mentioned dubbing supplementary recording method when executed by a processor.

[0013] In a fourth aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned dubbing supplementary recording method through the computer program.

[0014] In an embodiment of the present application, a method is adopted for obtaining supplementary lines and a target voice actor for the supplementary lines; determining a target emotion for the supplementary lines, and obtaining a target audio from a dubbing database of the target voice actor according to the target emotion, wherein the emotion of the target audio is the same as the target emotion; obtaining a target speech synthesis model for the target voice actor; and inputting the supplementary lines and the target audio into the target speech synthesis model to obtain the supplementary audio of the supplementary lines. In the above method, by determining the voice actor and emotion corresponding to the supplementary lines and matching the audio clips of the corresponding emotions, a pre-trained speech synthesis model for the voice actor is obtained, and the supplementary lines and the audio clips are input into the speech synthesis model to generate supplementary audio, ensuring that the generated supplementary audio is consistent with the original sound in terms of emotion and sound features, thereby achieving the purpose of ensuring a natural and smooth high-quality supplementary recording effect while improving the work efficiency of supplementary dubbing, thereby solving the technical problem of too low work efficiency of manual supplementary dubbing in the production process of film and television dramas. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0018] Figure 1 is a flow chart of an optional dubbing supplementary recording method according to an embodiment of the present application;

[0019] Figure 2 is a diagram of the training process of a first target generation model of an optional dubbing supplementary recording method according to an embodiment of the present application;

[0020] Figure 3 is a diagram of the training process of a second target generation model of an optional dubbing supplementary recording method according to an embodiment of the present application;

[0021] Figure 4 is a process diagram of dubbing re-recording of an optional dubbing re-recording method according to an embodiment of the present application;

[0022] Figure 5 is a schematic structural diagram of an optional dubbing supplementary recording device according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0026] According to a first aspect of an embodiment of the present application, a dubbing supplementary recording method is provided, optionally, as follows: Figure 1 As shown, the above method includes:

[0027] S102, obtaining the supplementary lines and the target voice actor for the supplementary lines;

[0028] S104, determining a target emotion for the supplementary recording of the lines, and acquiring a target audio from a dubbing database of a target dubbing artist according to the target emotion, wherein the emotion of the target audio is the same as the target emotion;

[0029] S106, obtaining a target speech synthesis model of a target voice actor;

[0030] S108, input the supplementary recorded lines and the target audio into the target speech synthesis large model to obtain the supplementary recorded audio of the supplementary recorded lines.

[0031] Optionally, in this embodiment, the target speech synthesis model adopts a TTS model trained with massive open source speech data, and is quickly fine-tuned by a small amount of high-quality corpus to obtain a target speech synthesis model for different voice actors. The target speech synthesis model includes an acoustic model text2semantic and a vqgan vocoder based on an autoregressive language model, that is, a first target generation model and a second target generation model, wherein the vqgan vocoder can complete the encoding of speech to semantic representation and the decoding of semantic representation to speech, and the text2semantic model can receive audio and input text, and complete the mapping of text to semantic representation. The model is based on an autoregressive language model, and the timbre and emotion can be specified by input audio, so that the output supplementary audio is consistent with the timbre of the voice actor, the emotion meets the requirements, and the naturalness also exceeds the traditional TTS algorithm.

[0032] Optionally, in the present embodiment, the supplementary lines that need to be supplemented are obtained from the script, and the corresponding target dubbing actor and target emotion are analyzed and determined, including the target emotion category and the target emotion degree, for example, the target emotion category is happy, and the target emotion degree is 50. According to the emotion category of the supplementary lines, the target audio matching this emotion category is obtained in the dubbing database of the target dubbing actor. The pre-trained target speech synthesis large model is obtained, and the target speech synthesis large model is specially trained for the target dubbing actor, and can understand and generate the voice features of the dubbing actor. The supplementary lines and the target audio are input into the text2semant ic model of the target speech synthesis large model, and converted to generate a comprehensive semantic representation. Finally, the comprehensive semantic representation is input into the vqgan vocoder of the target speech synthesis large model, and the final supplementary audio is generated according to the comprehensive semantic representation.

[0033] Optionally, in the present embodiment, by matching audio clips of the same emotion, emotional consistency can be better maintained during the supplementary recording, making the supplementary recording audio more natural and real. Utilizing a speech synthesis large model specially trained for the voice actor, a voice that matches the voice actor's voice characteristics can be more accurately generated, thereby improving the quality and naturalness of the supplementary recording audio. Through comprehensive semantic representation, the content and emotion of the lines, as well as the audio characteristics of the voice actor, can be better understood, laying the foundation for the next step of speech generation. Utilizing comprehensive semantic representation to generate supplementary recording audio, it is ensured that the generated audio matches the original lines in semantics and emotion, while maintaining the voice characteristics of the voice actor, thereby achieving high-quality supplementary recording effect.

[0034] As an optional example, the emotion includes an emotion category and an emotion degree, and obtaining the target audio in the dubbing database of the target dubbing artist according to the target emotion includes:

[0035] Determine the target emotion category and target emotion level for supplementary recording of lines;

[0036] Determine the emotion category and emotion degree of each audio in the dubbing database;

[0037] Determine a first audio group from the dubbing database, wherein the emotion category of each audio in the first audio group is the same as the target emotion category;

[0038] A target audio is determined from the first audio group, wherein the difference between the emotion level of the target audio and the target emotion level is minimal.

[0039] Optionally, in this embodiment, first, the lines that need to be supplemented are analyzed to determine their emotional categories (such as anger, happiness, sadness, etc.) and emotional degrees (the intensity of emotions, which can be expressed in numerical values), the audio database of the target dubbing actor is analyzed, and the emotional categories and emotional degrees are marked for each audio clip, and all audio clips with the same emotional categories as the target emotional categories of the supplemented lines are screened out in the database to form a first audio group, narrow the screening range, and ensure that all candidate audios are consistent with the target emotions in emotional categories. Further screening is performed in the first audio group to select the audio clip with the smallest difference in emotional degree from the target emotional degree of the supplemented lines, and determine it as the target audio, ensuring that the audio finally selected is closest to the supplemented lines in emotional intensity, so that the subtle differences of the target emotions can be better reflected during supplementary recording. By marking the emotional categories and emotional degrees in detail, the emotional characteristics of the supplemented lines can be accurately matched, ensuring that the supplemented audio is emotionally consistent with the lines. By matching the emotional degrees, the audio clip closest to the emotional degree of the lines is selected, so that the supplemented audio can be made more natural and real, and the overall audio quality is improved.

[0040] As an optional example, determining the target audio from the first audio group includes:

[0041] Calculating a difference between an emotion level of each audio in the first audio group and a target emotion level, wherein the difference is an absolute value of a difference between each emotion level and the target emotion level;

[0042] The audio with the smallest difference value in the first audio data group is determined as the target audio.

[0043] Optionally, in this embodiment, for each audio clip in the first audio group, the difference value between its emotional degree and the target emotional degree is calculated. Specifically, this difference value is the absolute value of the difference between the emotional degree of each audio clip and the target emotional degree, that is, the difference value = |emotional degree - target emotional degree|. By calculating the emotional degree difference value of each audio, the degree of closeness of each audio clip to the target emotion can be quantified, providing data support for subsequent screening. After calculating the emotional degree difference value of each audio clip, the audio clip with the smallest difference value is selected as the target audio, which means that the emotional degree of the audio clip is closest to the target emotional degree. By selecting the audio clip with the smallest difference value, it is ensured that the final selected audio is most closely matched with the target emotion in terms of emotional intensity, so that the expected emotion can be better conveyed during the supplementary recording.

[0044] As an optional example, the target speech synthesis large model includes a first target generation model and a second target generation model. Before obtaining the target speech synthesis large model of the target voice actor, the method further includes:

[0045] Obtaining a first training data set and a first verification data set of a target voice actor, wherein the first training data set includes a plurality of data pairs, each data pair is composed of a pair of lines and audio with the same emotion, and the first verification data set includes a comprehensive semantic representation of each data pair;

[0046] Inputting the first training data set into the first generation model to obtain a first generation result data set of the first training data set;

[0047] Using the loss function to calculate the difference between the first generated result data set and the first verification data set, to obtain a first loss value;

[0048] The model parameters of the first generation model are adjusted according to the first loss value, and the first loss value is recalculated until the recalculated first loss value is less than the target threshold, thereby obtaining the first target generation model.

[0049] Optionally, in this embodiment, the target speech synthesis model includes a first target generation model and a second target generation model. Each voice actor has a corresponding speech synthesis model. Before using the target speech synthesis model for dubbing, it is necessary to train the target speech synthesis model based on a large number of lines and dubbing audio of the voice actor to form the final target speech synthesis model. Therefore, Figure 2As shown in the figure, the training process of the first target generative model includes: collecting and preparing the speech data of the target voice actor, dividing it into a first training data set and a first verification data set, the first training data set includes multiple data pairs, each data pair consists of lines and audio with the same emotion, and the first verification data set contains the comprehensive semantic representation of these data pairs. The first training data set is input into the initial first generative model, such as the text2semantic model, and the first generative model converts these data pairs into their corresponding comprehensive semantic representations to generate a first generated result data set. The difference between the first generated result data set and the first verification data set is calculated using the loss function, that is, the difference between the comprehensive semantic representation generated by the calculation model and the expected representation is calculated to obtain a first loss value. According to the calculated first loss value, the parameters of the model are adjusted to reduce the loss value. Repeat this process, recalculate the loss value until the loss value is less than the preset target threshold, and obtain the final first target generative model. By repeatedly adjusting and optimizing the model parameters, it is ensured that the model can accurately convert the lines and audio pairs into comprehensive semantic representations, thereby improving the performance and accuracy of the model.

[0050] As an optional example, after adjusting the model parameters of the first generation model according to the loss value until the first loss value is less than the first threshold and the first target generation model is obtained, the method further includes:

[0051] Inputting the first training data set into the first target generation model to obtain a second training data set;

[0052] Obtaining a second verification data set of the target voice actor, wherein the second verification data set includes the audio corresponding to each line in the first training data set;

[0053] Inputting the second training data set into the second generation model to obtain a second generation result data set of the second training data set;

[0054] Using the loss function to calculate the difference between the second generated result data set and the second verification data set, to obtain a second loss value;

[0055] The model parameters of the second generation model are adjusted according to the second loss value, and the second loss value is recalculated until the recalculated second loss value is less than the target threshold, thereby obtaining the second target generation model.

[0056] Optionally, in this embodiment, if Figure 3As shown in the figure, the training process of the second target generation model includes: inputting the prepared first training data set (including lines and audio pairs with the same emotions) into the previously trained first target generation model to generate the corresponding comprehensive semantic representation and form the second training data set. Collect and prepare the audio data of the target voice actor to ensure that each line has a corresponding audio to form a second verification data set. Input the generated second training data set into the initial second generation model, for example, the vqgan vocoder, and the model converts these comprehensive semantic representations into corresponding audio to generate a second generation result data set. The loss function is used to calculate the difference between the second generation result data set (audio generated by the model) and the second verification data set (standard audio) to obtain the second loss value. According to the calculated second loss value, adjust the parameters of the second generation model to reduce the loss value. Repeat this process and recalculate the loss value until the loss value is less than the preset target threshold to obtain the final second target generation model. By repeatedly adjusting and optimizing the model parameters, it is ensured that the model can accurately convert the comprehensive semantic representation into the corresponding audio, thereby improving the performance and accuracy of the model.

[0057] As an optional example, the supplementary recorded lines and the target audio are input into the target speech synthesis model to obtain the supplementary recorded audio of the supplementary recorded lines, including:

[0058] Inputting the supplementary recorded lines and the target audio into the first target generation model of the target speech synthesis large model to convert the supplementary recorded lines and the target audio into a comprehensive semantic representation;

[0059] The comprehensive semantic representation is input into the second target generation model of the target speech synthesis large model to convert the comprehensive semantic representation into supplementary audio.

[0060] Optionally, in this embodiment, if Figure 4 As shown in the process diagram of dubbing re-recording, the specific process of using the target speech synthesis model to re-record the re-recorded lines includes: first, the lines to be re-recorded are converted into corresponding text semantic representations through an encoder, and the text information is converted into a semantic representation form that the model can understand. Then, the target audio is converted into the corresponding audio semantic representation through an encoder, and the audio information is converted into a semantic representation form that the model can process. Next, the text semantic representation and the audio semantic representation are combined to generate a comprehensive semantic representation. This comprehensive semantic representation contains the content information of the lines and the emotional and timbre characteristics of the target audio. Finally, the generated comprehensive semantic representation is input into the second target generation model of the target speech synthesis model, and the model converts this comprehensive semantic representation into re-recorded audio. Through the second target generation model, it is ensured that the generated re-recorded audio is highly consistent with the target audio in terms of voice features, emotional expression and timbre, so as to achieve high-quality audio re-recording.

[0061] As an optional example, inputting the supplementary recorded lines and the target audio into the first target generation model of the target speech synthesis large model to convert the supplementary recorded lines and the target audio into a comprehensive semantic representation includes:

[0062] Encode the supplementary lines into a set of textual semantic representations;

[0063] Encoding the target audio into a set of audio semantic representations;

[0064] Combine text semantic representation and audio semantic representation to generate a comprehensive semantic representation.

[0065] Optionally, in this embodiment, the first target generation model of the target speech synthesis model can convert the lines that need to be supplemented into corresponding text semantic representations, and convert the text information into a semantic representation form that the model can understand. Then, the target audio is converted into a corresponding audio semantic representation, and the audio information is converted into a semantic representation form that the model can process. Finally, the text semantic representation and the audio semantic representation are combined to generate a comprehensive semantic representation, which includes the content information of the lines and the emotional and timbre characteristics of the target audio.

[0066] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0067] According to another aspect of the embodiment of the present application, a dubbing recording device is also provided. Figure 5 As shown, including:

[0068] A first acquisition module 502 is used to acquire supplementary lines and a target voice actor for supplementary lines;

[0069] The determination module 504 is used to determine the target emotion of the supplementary recording of the lines, and obtain the target audio from the dubbing database of the target dubbing artist according to the target emotion, wherein the emotion of the target audio is the same as the target emotion;

[0070] The second acquisition module 506 is used to acquire a target speech synthesis model of a target voice actor;

[0071] The supplementary recording module 508 is used to input the supplementary recorded lines and the target audio into the target speech synthesis large model to obtain the supplementary recorded audio of the supplementary recorded lines.

[0072] Optionally, in this embodiment, the target speech synthesis model adopts a TTS model trained with massive open source speech data, and is quickly fine-tuned by a small amount of high-quality corpus to obtain a target speech synthesis model for different voice actors. The target speech synthesis model includes an acoustic model text2semantic and a vqgan vocoder based on an autoregressive language model, that is, a first target generation model and a second target generation model, wherein the vqgan vocoder can complete the encoding of speech to semantic representation and the decoding of semantic representation to speech, and the text2semantic model can receive audio and input text, and complete the mapping of text to semantic representation. The model is based on an autoregressive language model, and the timbre and emotion can be specified by input audio, so that the output supplementary audio is consistent with the timbre of the voice actor, the emotion meets the requirements, and the naturalness also exceeds the traditional TTS algorithm.

[0073] Optionally, in the present embodiment, the supplementary lines that need to be supplemented are obtained from the script, and the corresponding target dubbing actor and target emotion are analyzed and determined, including the target emotion category and the target emotion degree, for example, the target emotion category is happy, and the target emotion degree is 50. According to the emotion category of the supplementary lines, the target audio matching this emotion category is obtained in the dubbing database of the target dubbing actor. The pre-trained target speech synthesis large model is obtained, and the target speech synthesis large model is specially trained for the target dubbing actor, and can understand and generate the voice features of the dubbing actor. The supplementary lines and the target audio are input into the text2semant ic model of the target speech synthesis large model, and converted to generate a comprehensive semantic representation. Finally, the comprehensive semantic representation is input into the vqgan vocoder of the target speech synthesis large model, and the final supplementary audio is generated according to the comprehensive semantic representation.

[0074] Optionally, in the present embodiment, by matching audio clips of the same emotion, emotional consistency can be better maintained during the supplementary recording, making the supplementary recording audio more natural and real. Utilizing a speech synthesis large model specially trained for the voice actor, a voice that matches the voice actor's voice characteristics can be more accurately generated, thereby improving the quality and naturalness of the supplementary recording audio. Through comprehensive semantic representation, the content and emotion of the lines, as well as the audio characteristics of the voice actor, can be better understood, laying the foundation for the next step of speech generation. Utilizing comprehensive semantic representation to generate supplementary recording audio, it is ensured that the generated audio matches the original lines in semantics and emotion, while maintaining the voice characteristics of the voice actor, thereby achieving high-quality supplementary recording effect.

[0075] As an optional example, the emotion includes an emotion category and an emotion degree, and the determination module includes:

[0076] A first determination unit is used to determine a target emotion category and a target emotion degree of the supplementary recording lines;

[0077] A second determination unit, used to determine the emotion category and emotion degree of each audio in the dubbing database;

[0078] A third determining unit is used to determine a first audio group from the dubbing database, wherein the emotion category of each audio in the first audio group is the same as the target emotion category;

[0079] The fourth determining unit is configured to determine a target audio from the first audio group, wherein the difference between the emotion level of the target audio and the target emotion level is minimal.

[0080] Optionally, in this embodiment, first, the lines that need to be supplemented are analyzed to determine their emotional categories (such as anger, happiness, sadness, etc.) and emotional degrees (the intensity of emotions, which can be expressed in numerical values), the audio database of the target dubbing actor is analyzed, and the emotional categories and emotional degrees are marked for each audio clip, and all audio clips with the same emotional categories as the target emotional categories of the supplemented lines are screened out in the database to form a first audio group, narrow the screening range, and ensure that all candidate audios are consistent with the target emotions in emotional categories. Further screening is performed in the first audio group to select the audio clip with the smallest difference in emotional degree from the target emotional degree of the supplemented lines, and determine it as the target audio, ensuring that the audio finally selected is closest to the supplemented lines in emotional intensity, so that the subtle differences of the target emotions can be better reflected during supplementary recording. By marking the emotional categories and emotional degrees in detail, the emotional characteristics of the supplemented lines can be accurately matched, ensuring that the supplemented audio is emotionally consistent with the lines. By matching the emotional degrees, the audio clip closest to the emotional degree of the lines is selected, so that the supplemented audio can be made more natural and real, and the overall audio quality is improved.

[0081] As an optional example, the fourth determining unit includes:

[0082] A calculation subunit, configured to calculate a difference between an emotion level of each audio in the first audio group and a target emotion level, wherein the difference is an absolute value of a difference between each emotion level and the target emotion level;

[0083] The determination subunit is used to determine the audio with the smallest difference value in the first audio data group as the target audio.

[0084] Optionally, in this embodiment, for each audio clip in the first audio group, the difference value between its emotional degree and the target emotional degree is calculated. Specifically, this difference value is the absolute value of the difference between the emotional degree of each audio clip and the target emotional degree, that is, the difference value = |emotional degree - target emotional degree|. By calculating the emotional degree difference value of each audio, the degree of closeness of each audio clip to the target emotion can be quantified, providing data support for subsequent screening. After calculating the emotional degree difference value of each audio clip, the audio clip with the smallest difference value is selected as the target audio, which means that the emotional degree of the audio clip is closest to the target emotional degree. By selecting the audio clip with the smallest difference value, it is ensured that the final selected audio is most closely matched with the target emotion in terms of emotional intensity, so that the expected emotion can be better conveyed during the supplementary recording.

[0085] As an optional example, the target speech synthesis model includes a first target generation model and a second target generation model, and the device further includes:

[0086] A third acquisition module is used to acquire a first training data set and a first verification data set of the target voice actor before acquiring the target speech synthesis large model of the target voice actor, wherein the first training data set includes a plurality of data pairs, each data pair is composed of a pair of lines and audio with the same emotion, and the first verification data set includes a comprehensive semantic representation of each data pair;

[0087] A first conversion module, used for inputting the first training data set into the first generation model to obtain a first generation result data set of the first training data set;

[0088] A first calculation module, used to calculate the difference between the first generated result data set and the first verification data set using a loss function to obtain a first loss value;

[0089] The first adjustment module is used to adjust the model parameters of the first generation model according to the first loss value, and recalculate the first loss value until the recalculated first loss value is less than the target threshold, so as to obtain the first target generation model.

[0090] Optionally, in this embodiment, the target speech synthesis model includes a first target generation model and a second target generation model. Each voice actor has a corresponding language model. Before using the target speech synthesis model for dubbing, it is necessary to train based on a large number of lines and dubbing audio of the voice actor to form the final target speech synthesis model. Therefore, Figure 2As shown in the figure, the training process of the first target generative model includes: collecting and preparing the speech data of the target voice actor, dividing it into a first training data set and a first verification data set, the first training data set includes multiple data pairs, each data pair consists of lines and audio with the same emotion, and the first verification data set contains the comprehensive semantic representation of these data pairs. The first training data set is input into the initial first generative model, such as the text2semantic model, and the first generative model converts these data pairs into their corresponding comprehensive semantic representations to generate a first generated result data set. The difference between the first generated result data set and the first verification data set is calculated using the loss function, that is, the difference between the comprehensive semantic representation generated by the calculation model and the expected representation is calculated to obtain a first loss value. According to the calculated first loss value, the parameters of the model are adjusted to reduce the loss value. Repeat this process, recalculate the loss value until the loss value is less than the preset target threshold, and obtain the final first target generative model. By repeatedly adjusting and optimizing the model parameters, it is ensured that the model can accurately convert the lines and audio pairs into comprehensive semantic representations, thereby improving the performance and accuracy of the model.

[0091] As an optional example, the above device further includes:

[0092] A second conversion module is used for adjusting the model parameters of the first generation model according to the loss value until the first loss value is less than the first threshold value, and after obtaining the first target generation model, inputting the first training data set into the first target generation model to obtain a second training data set;

[0093] A fourth acquisition module is used to acquire a second verification data set of the target voice actor, wherein the second verification data set includes the audio corresponding to each line in the first training data set;

[0094] A third conversion module is used to input the second training data set into the second generation model to obtain a second generation result data set of the second training data set;

[0095] A second calculation module, used to calculate the difference between the second generated result data set and the second verification data set using a loss function to obtain a second loss value;

[0096] The second adjustment module is used to adjust the model parameters of the second generation model according to the second loss value, and recalculate the second loss value until the recalculated second loss value is less than the target threshold, so as to obtain the second target generation model.

[0097] Optionally, in this embodiment, if Figure 3As shown in the figure, the training process of the second target generation model includes: inputting the prepared first training data set (including lines and audio pairs with the same emotions) into the previously trained first target generation model to generate the corresponding comprehensive semantic representation and form the second training data set. Collect and prepare the audio data of the target voice actor to ensure that each line has a corresponding audio to form a second verification data set. Input the generated second training data set into the initial second generation model, for example, the vqgan vocoder, and the model converts these comprehensive semantic representations into corresponding audio to generate a second generation result data set. The loss function is used to calculate the difference between the second generation result data set (audio generated by the model) and the second verification data set (standard audio) to obtain the second loss value. According to the calculated second loss value, adjust the parameters of the second generation model to reduce the loss value. Repeat this process and recalculate the loss value until the loss value is less than the preset target threshold to obtain the final second target generation model. By repeatedly adjusting and optimizing the model parameters, it is ensured that the model can accurately convert the comprehensive semantic representation into the corresponding audio, thereby improving the performance and accuracy of the model.

[0098] As an optional example, the supplementary recording module includes:

[0099] A first conversion unit, used for inputting the supplementary recorded lines and the target audio into a first target generation model of the target speech synthesis large model, so as to convert the supplementary recorded lines and the target audio into a comprehensive semantic representation;

[0100] The second conversion unit is used to input the comprehensive semantic representation into the second target generation model of the target speech synthesis large model to convert the comprehensive semantic representation into supplementary audio.

[0101] Optionally, in this embodiment, if Figure 4 As shown in the process diagram of dubbing re-recording, the specific process of using the target speech synthesis model to re-record the re-recorded lines includes: first, the lines to be re-recorded are converted into corresponding text semantic representations through an encoder, and the text information is converted into a semantic representation form that the model can understand. Then, the target audio is converted into the corresponding audio semantic representation through an encoder, and the audio information is converted into a semantic representation form that the model can process. Next, the text semantic representation and the audio semantic representation are combined to generate a comprehensive semantic representation. This comprehensive semantic representation contains the content information of the lines and the emotional and timbre characteristics of the target audio. Finally, the generated comprehensive semantic representation is input into the second target generation model of the target speech synthesis model, and the model converts this comprehensive semantic representation into re-recorded audio. Through the second target generation model, it is ensured that the generated re-recorded audio is highly consistent with the target audio in terms of voice features, emotional expression and timbre, so as to achieve high-quality audio re-recording.

[0102] As an optional example, the first conversion unit includes:

[0103] A first encoding subunit is used to encode the supplementary lines into a set of text semantic representations;

[0104] A second encoding subunit, configured to encode the target audio into a set of audio semantic representations;

[0105] The third encoding subunit is used to combine the text semantic representation and the audio semantic representation to generate a comprehensive semantic representation.

[0106] Optionally, in this embodiment, the first target generation model of the target speech synthesis model can convert the lines that need to be supplemented into corresponding text semantic representations, and convert the text information into a semantic representation form that the model can understand. Then, the target audio is converted into a corresponding audio semantic representation, and the audio information is converted into a semantic representation form that the model can process. Finally, the text semantic representation and the audio semantic representation are combined to generate a comprehensive semantic representation, which includes the content information of the lines and the emotional and timbre characteristics of the target audio.

[0107] For other examples of this embodiment, please refer to the above examples and will not be repeated here.

[0108] Figure 6 is a schematic diagram of an optional electronic device according to an embodiment of the present application, such as Figure 6 As shown, it includes a processor 602, a communication interface 604, a memory 606 and a communication bus 608, wherein the processor 602, the communication interface 604 and the memory 606 communicate with each other through the communication bus 608, wherein,

[0109] Memory 606, used to store computer programs;

[0110] The processor 602 is used to implement the following steps when executing the computer program stored in the memory 606:

[0111] Obtain the over-recording lines and the target voice actor for the over-recording lines;

[0112] Determine a target emotion for the supplementary recording of the lines, and obtain a target audio from a dubbing database of a target dubbing artist according to the target emotion, wherein the emotion of the target audio is the same as the target emotion;

[0113] Obtain a target speech synthesis model of the target voice actor;

[0114] The supplementary recorded lines and the target audio are input into the target speech synthesis large model to obtain the supplementary recorded audio of the supplementary recorded lines.

[0115] Optionally, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The communication interface is used for communication between the above electronic device and other devices.

[0116] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0117] As an example, the memory 606 may include, but is not limited to, the first acquisition module 502, the determination module 504, the second acquisition module 506, and the supplementary recording module 508 in the dubbing supplementary recording device. In addition, it may also include, but is not limited to, other module units in the dubbing supplementary recording device, which will not be repeated in this example.

[0118] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0119] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0120] It can be understood by those skilled in the art that Figure 6The structure shown is for illustration only, and the device for implementing the above-mentioned dubbing supplementary recording method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 6 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 6 Different configurations shown.

[0121] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0122] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program executes the steps in the above-mentioned dubbing supplementary recording method when executed by a processor.

[0123] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0124] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0125] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0126] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A dubbing recording method, characterized in that: include: Obtaining supplementary lines and target voice actors for the supplementary lines; Determining a target emotion of the supplementary recorded lines, and acquiring a target audio in a dubbing database of the target dubbing artist according to the target emotion, wherein the emotion of the target audio is the same as the target emotion; Obtaining a target speech synthesis large model of the target voice actor; The supplementary recorded lines and the target audio are input into the target speech synthesis large model to obtain the supplementary recorded audio of the supplementary recorded lines.

2. The method according to claim 1, characterized in that The emotion includes an emotion category and an emotion degree, and acquiring a target audio in a dubbing database of the target dubbing artist according to the target emotion includes: Determining a target emotion category and a target emotion degree of the supplementary recorded lines; Determine the emotion category and emotion degree of each audio in the dubbing database; Determine a first audio group from the dubbing database, wherein the emotion category of each audio in the first audio group is the same as the target emotion category; The target audio is determined from the first audio group, wherein the difference between the emotion level of the target audio and the target emotion level is minimal.

3. The method according to claim 2, characterized in that The determining the target audio from the first audio group comprises: Calculating a difference between an emotion level of each audio in the first audio group and the target emotion level, wherein the difference is an absolute value of a difference between each emotion level and the target emotion level; The audio with the smallest difference value in the first audio data group is determined as the target audio.

4. The method according to claim 1, characterized in that The target speech synthesis large model includes a first target generation model and a second target generation model. Before obtaining the target speech synthesis large model of the target voice actor, the method further includes: Acquire a first training data set and a first verification data set of the target voice actor, wherein the first training data set includes a plurality of data pairs, one of the data pairs is composed of a pair of lines and audio with the same emotion, and the first verification data set includes a comprehensive semantic representation of each of the data pairs; Inputting the first training data set into a first generation model to obtain a first generation result data set of the first training data set; Using a loss function to calculate the difference between the first generated result data set and the first verification data set, to obtain a first loss value; The model parameters of the first generation model are adjusted according to the first loss value, and the first loss value is recalculated until the recalculated first loss value is less than a target threshold, so as to obtain the first target generation model.

5. The method according to claim 4, characterized in that After adjusting the model parameters of the first generation model according to the loss value until the first loss value is less than a first threshold value to obtain the first target generation model, the method further includes: Inputting the first training data set into the first target generation model to obtain a second training data set; Acquire a second verification data set of the target voice actor, wherein the second verification data set includes audio corresponding to each line in the first training data set; Inputting the second training data set into a second generation model to obtain a second generation result data set of the second training data set; Calculate the difference between the second generated result data set and the second verification data set using the loss function to obtain a second loss value; Adjust the model parameters of the second generation model according to the second loss value, and recalculate the second loss value until the recalculated second loss value is less than the target threshold, so as to obtain the second target generation model.

6. The method according to claim 1, characterized in that The step of inputting the supplementary recorded lines and the target audio into the target speech synthesis large model to obtain the supplementary recorded audio of the supplementary recorded lines comprises: Inputting the supplementary recorded lines and the target audio into a first target generation model of the target speech synthesis large model to convert the supplementary recorded lines and the target audio into a comprehensive semantic representation; The comprehensive semantic representation is input into the second target generation model of the target speech synthesis large model to convert the comprehensive semantic representation into the supplementary audio.

7. The method according to claim 6, characterized in that The step of inputting the supplementary recorded lines and the target audio into the first target generation model of the target speech synthesis large model to convert the supplementary recorded lines and the target audio into a comprehensive semantic representation includes: Encoding the supplementary lines into a set of text semantic representations; Encoding the target audio into a set of audio semantic representations; The text semantic representation and the audio semantic representation are combined to generate the comprehensive semantic representation.

8. A dubbing recording device, characterized in that: include: A first acquisition module is used to acquire supplementary lines and a target voice actor for the supplementary lines; A determination module, used to determine the target emotion of the supplementary recorded lines, and obtain a target audio in the dubbing database of the target dubbing artist according to the target emotion, wherein the emotion of the target audio is the same as the target emotion; A second acquisition module is used to acquire a target speech synthesis model of the target voice actor; The supplementary recording module is used to input the supplementary recorded lines and the target audio into the target speech synthesis large model to obtain the supplementary recorded audio of the supplementary recorded lines.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.