Video content role dubbing method and device, equipment and storage medium

Abstract

The invention relates to a role dubbing method and device for video content, equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed video file; based on the to-be-processed video file, obtaining a to-be-analyzed text and role voice feature information; based on a target language selected by a user, judging whether the to-be-analyzed text needs to be translated or not; if the to-be-analyzed text needs to be translated, translating the to-be-analyzed text to obtain a target language text; based on the target language text and the character speech feature information, obtaining a synthetic speech; performing synchronization processing on the audio information of the synthetic voice and the video information of the video file to be processed to obtain adjusted audio information; and synthesizing the adjusted audio information with the video information to obtain a target video file. According to the method, the target language can be selected according to needs, translation and synthesis are automatically performed, dependence on professionals is reduced, and the requirement of a user for multi-language dubbing can be met.

Description

Technical Field

[0001] The present application relates to the field of video content processing and artificial intelligence technology, and in particular to a method, device, equipment and storage medium for dubbing a role in video content. Background Art

[0002] Video dubbing means replacing the original dialogue or narration with a new voice while retaining the original video image. The new voice can be a translation into a different language, or the same language but with a different character or style. The main purpose is to enable the video content to transcend language and cultural barriers so that more viewers can understand and appreciate it.

[0003] Traditional video dubbing technology mainly involves translating the dialogue in the original video into the target language, then hiring professional dubbing actors to record according to the translated script, and then editing the recorded audio to achieve the final dubbing effect.

[0004] Although manual dubbing can ensure the naturalness of speech and emotional expression, the professional capabilities of manual dubbing are limited, that is, the language coverage is limited and cannot meet users' needs for multilingual dubbing. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for dubbing a character in a video content, aiming to solve the problem that manual dubbing cannot meet the user's demand for multi-language dubbing.

[0006] In a first aspect, an embodiment of the present application provides a method for dubbing a character in a video content, the method comprising:

[0007] Get the video file to be processed;

[0008] Based on the video file to be processed, the text to be analyzed and the voice feature information of the character are obtained;

[0009] Based on the target language selected by the user, determine whether the text to be analyzed needs to be translated;

[0010] If the text to be analyzed needs to be translated, the text to be analyzed is translated to obtain the target language text;

[0011] Based on the target language text and the character's voice feature information, synthesized speech is obtained;

[0012] Synchronously processing the audio information of the synthesized speech and the video information of the video file to be processed to obtain adjusted audio information;

[0013] The adjusted audio information is synthesized with the video information to obtain a target video file.

[0014] In some implementations, obtaining character voice feature information based on the video file to be processed includes:

[0015] Perform language feature extraction on the video file to be processed to obtain speech features;

[0016] Inputting speech features into a pre-trained deep learning model to obtain a speech waveform segment output by the pre-trained deep learning model;

[0017] The speech waveform segment is input into the character speech model to obtain the character speech feature information.

[0018] In some implementations, the audio information of the synthesized speech includes durations corresponding to a plurality of speech waveform segments. After obtaining the video file to be processed, the method includes:

[0019] Recognize the role actions of the video file to be processed to obtain the role action features;

[0020] Extracting the duration of the character's action features as video information of the video file to be processed;

[0021] The audio information of the synthesized speech is synchronously processed with the video information of the video file to be processed to obtain the adjusted audio information, including:

[0022] The durations corresponding to the multiple speech waveform segments are synchronously processed with the durations of the character's action features to obtain an adjusted speech waveform duration.

[0023] In some implementations, the durations corresponding to the plurality of speech waveform segments are synchronously processed with the durations of the character action features to obtain the adjusted speech waveform durations, including:

[0024] Obtain the duration of each character's action feature and the duration of each corresponding waveform segment respectively;

[0025] Based on the duration of the character's action feature, the duration of the corresponding speech waveform segment is dynamically adjusted to obtain an adjusted speech waveform duration.

[0026] In some implementations, dynamically adjusting the duration of the corresponding speech waveform segment based on the duration of the character's action feature includes:

[0027] If the duration of the speech waveform segment is greater than the duration of the corresponding character action, shorten the duration of the speech waveform segment to the duration of the corresponding character action;

[0028] If the duration of the speech waveform segment is shorter than the duration of the corresponding character action, the duration of the speech waveform segment is extended to the duration of the corresponding character action.

[0029] In some implementations, obtaining a text to be analyzed based on a video file to be processed includes:

[0030] Separating the audio from the video file to be processed to obtain an audio file;

[0031] Performing speech recognition on the audio file to obtain the speech content of the audio file;

[0032] The speech content is transcribed to obtain the text to be analyzed.

[0033] In some implementations, based on the target language selected by the user, determining whether to translate the text to be analyzed, the method further includes:

[0034] If the text to be analyzed does not need to be translated, a synthesized speech is obtained based on the text to be analyzed and the character's speech feature information.

[0035] In a second aspect, an embodiment of the present application further provides a device for dubbing a character in video content, which includes a unit for executing the above method.

[0036] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the above method is implemented when the processor executes the computer program.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0038] The embodiment of the present application provides a method, device, equipment and storage medium for dubbing a role in video content. The method includes: obtaining a video file to be processed; obtaining a text to be analyzed and character voice feature information based on the video file to be processed; judging whether the text to be analyzed needs to be translated based on the target language selected by the user; if the text to be analyzed needs to be translated, translating the text to be analyzed to obtain the target language text; obtaining a synthesized voice based on the target language text and the character voice feature information; synchronizing the audio information of the synthesized voice with the video information of the video file to be processed to obtain the adjusted audio information; synthesizing the adjusted audio information with the video information to obtain the target video file.

[0039] Since the embodiment of the present application automatically determines whether the text to be analyzed needs to be translated based on the target language selected by the user, and if the text to be analyzed needs to be translated, the text to be analyzed will be translated. Therefore, the character dubbing system of the video content of the present application can support translation and synthesis of multiple languages. Users can select the target language as needed, and automatically perform translation and synthesis without additional manual intervention, thereby reducing dependence on professionals and meeting users' needs for multi-language dubbing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0043] Figure 1 A flowchart of a first embodiment of a method for dubbing a role in video content provided by the present application;

[0044] Figure 2 A schematic diagram of another embodiment of dubbing a character in a video content provided by the present application;

[0045] Figure 3 A schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0047] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0048] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0049] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0050] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0051] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0052] Video dubbing means replacing the original dialogue or narration with a new voice while retaining the original video image. The new voice can be a translation into a different language, or the same language but with a different character or style. The main purpose is to enable the video content to transcend language and cultural barriers so that more viewers can understand and appreciate it.

[0053] Traditional video dubbing technology mainly involves translating the dialogue in the original video into the target language, then hiring professional dubbing actors to record according to the translated script, and then editing the recorded audio to achieve the final dubbing effect.

[0054] Although manual dubbing can ensure the naturalness of speech and emotional expression, the professional capabilities of manual dubbing are limited, that is, the language coverage is limited and cannot meet users' needs for multilingual dubbing.

[0055] In order to solve the problem that manual dubbing in the prior art cannot meet the user's demand for multi-language dubbing, the present application provides a method for character dubbing of video content, which can select the target language as needed, automatically translate and synthesize, reduce the dependence on professionals, and can meet the user's demand for multi-language dubbing.

[0056] See also Figure 1 , Figure 1 This is a flow chart of a first embodiment of a method for dubbing a role in video content provided by the present application. The method comprises the following steps:

[0057] Step 110: Obtain the video file to be processed.

[0058] Step 120: Based on the video file to be processed, obtain the text to be analyzed and the character's voice feature information.

[0059] Step 130: Based on the target language selected by the user, determine whether the text to be analyzed needs to be translated.

[0060] Step 140: If the text to be analyzed needs to be translated, the text to be analyzed is translated to obtain a target language text.

[0061] Step 150: Obtaining synthesized speech based on the target language text and the character's speech feature information.

[0062] Step 160: Synchronize the audio information of the synthesized speech with the video information of the video file to be processed to obtain adjusted audio information.

[0063] Step 170: synthesize the adjusted audio information and video information to obtain a target video file.

[0064] Since this embodiment automatically determines whether the text to be analyzed needs to be translated based on the target language selected by the user, and if the text to be analyzed needs to be translated, the text to be analyzed will be translated. Therefore, the character dubbing system of the video content of this application can support translation and synthesis of multiple languages. Users can select the target language as needed, and the system automatically translates and synthesizes without additional manual intervention, thereby reducing dependence on professionals and meeting users' needs for multi-language dubbing.

[0065] Referring to the second embodiment of a method for dubbing a role in video content provided by the present application, the second embodiment includes the following steps:

[0066] Step 210: Obtain the video file to be processed.

[0067] Step 220: Based on the video file to be processed, obtain the text to be analyzed.

[0068] The character voice feature information can be obtained by following the steps below:

[0069] Step 221: extract language features from the video file to be processed to obtain speech features.

[0070] Step 222: Input the speech features into the pre-trained deep learning model to obtain the speech waveform segment output by the pre-trained deep learning model.

[0071] Among them, the pre-trained deep learning model can be a Tacotron, WaveNet or other model, which can be selected according to actual conditions and is not limited in this application.

[0072] Step 223: Input the speech waveform segment into the character speech model to obtain the character speech feature information.

[0073] For steps 221 to 223, it is assumed that there is an animated character, which has the following actions and corresponding dialogues in the video:

[0074] Action 1: Look up->Duration: 1 second->Dialogue: "Hello!".

[0075] Action 2: waving -> duration 2 seconds -> dialogue: "How about you?".

[0076] Action 3: Nod -> Duration 1 second -> Dialogue: "Nice to meet you."

[0077] Then, a voice analysis is performed on the video to obtain the voice information, i.e., the dialogue "Hello! How are you? Nice to meet you.".

[0078] “Hello! How are you? Nice to meet you.” is subjected to text analysis and prosody prediction, and then a deep learning model (such as Tacotron) is used to generate speech waveform segments. The obtained speech waveform segments are: “Hello!” (0.5 seconds in length), “How are you?” (1.5 seconds in length), and “Nice to meet you.” (1 second in length).

[0079] Step 230: Based on the target language selected by the user, determine whether the text to be analyzed needs to be translated.

[0080] Step 240: If the text to be analyzed needs to be translated, the text to be analyzed is translated to obtain a target language text.

[0081] Among them, a translation engine, such as Google Translate or Microsoft Translator, can be used to translate the text to be analyzed into a target language text.

[0082] In addition, to improve the quality of translation, in some embodiments, a domain-specific terminology library can be built into the character dubbing system of the video content to automatically optimize the translation results. For content with proper nouns or different cultural backgrounds, manual adjustments can be made.

[0083] Step 250: Obtain synthesized speech based on the target language text and the character's speech feature information.

[0084] Step 260: Synchronize the audio information of the synthesized speech with the video information of the video file to be processed to obtain adjusted audio information.

[0085] Step 270: synthesize the adjusted audio information and video information to obtain a target video file.

[0086] In some embodiments, the audio information of the synthesized speech includes the durations corresponding to multiple speech waveform segments, and synchronizing the audio information of the synthesized speech with the video information of the video file to be processed may be synchronizing the durations corresponding to the speech waveform segments with the durations of the character's action features.

[0087] Specifically, after obtaining the video file to be processed, the following steps may be taken:

[0088] Step 310: Recognize the character's actions in the video file to be processed to obtain the character's action features.

[0089] Among them, the recognition and analysis of character actions on the video file to be processed mainly includes action recognition and action timing recording.

[0090] Specifically, in some embodiments, computer vision technology (such as OpenPose, posture estimation) can be used to identify the specific actions of the character, and record the start time, end time and rhythm of each action.

[0091] Step 320: Then extract the duration of the character action feature as the video information of the video file to be processed.

[0092] Step 330: Synchronize the durations corresponding to the multiple speech waveform segments with the durations of the character's action features to obtain an adjusted speech waveform duration.

[0093] Wherein, step 330 may include the following steps:

[0094] Step 331: respectively obtain the duration of each character action feature and the duration of each corresponding waveform segment.

[0095] Step 332: Based on the duration of the character's action feature, dynamically adjust the duration of the corresponding speech waveform segment to obtain an adjusted speech waveform duration.

[0096] The timing in the generated voice file (such as the duration of the character's action features) is aligned with the action timing (such as the duration of the character's action features) so that the key sentences match the character's lip shape and action moments.

[0097] There are two main ways to dynamically adjust the duration of the corresponding speech waveform segment:

[0098] 1) If the duration of the speech waveform segment is longer than the duration of the corresponding character action, shorten the duration of the speech waveform segment to the duration of the corresponding character action;

[0099] 2) If the duration of the speech waveform segment is shorter than the duration of the corresponding character action, the duration of the speech waveform segment is extended to the duration of the corresponding character action.

[0100] For details on the dynamic adjustment of the duration of the speech waveform segment, please refer to the following:

[0101] Assume that the duration of the character action feature is:

[0102] Character action feature 1: raising head, duration 1 second, corresponding to the dialogue "Hello!".

[0103] Character action feature 2: waving, duration 2 seconds, corresponding to the dialogue "How are you?".

[0104] Character action feature 3: Nod, lasting 1 second, corresponding to the dialogue "Nice to meet you."

[0105] The speech waveform segments obtained after speech synthesis are: "Hello!" (0.5 seconds in length), "How are you?" (1.5 seconds in length), and "Nice to meet you." (1 second in length).

[0106] The dialogue "Hello!" (0.5 seconds) in the speech waveform segment needs to be aligned with the head-raising action (1 second). Since the dialogue "Hello!" (0.5 seconds) in the speech waveform segment is shorter than the head-raising action (1 second), the speech duration in the speech waveform segment can be stretched to 1 second, that is, extended to the duration of the corresponding head-raising action.

[0107] Similarly, the dialogue "Howareyou?" (1.5 seconds) in the speech waveform segment needs to be aligned with the waving action (2 seconds). Since the dialogue "Howareyou?" (1.5 seconds) in the speech waveform segment is shorter than the waving action (2 seconds), the speech duration in the speech waveform segment can be slightly stretched to 2 seconds, that is, extended to the duration of the corresponding waving action.

[0108] The dialogue “Nice to meet you.” (1 second) in the speech waveform segment matches the duration of the nodding action (1 second), so there is no need to adjust the duration of the speech waveform segment.

[0109] In this way, the duration of the newly generated synthesized speech can be dynamically adjusted as needed. For example, the timing in the newly generated synthesized speech can be time scaled or stretched by modifying the playback speed of the speech to match the timing of the character's action features without changing the pitch of the speech, thereby making the dubbing more vivid and natural.

[0110] In some embodiments, the acquisition of the text to be analyzed may be performed by the following steps:

[0111] Step 410: Separate the audio from the video file to be processed to obtain an audio file.

[0112] Step 420: Perform speech recognition on the audio file to obtain the speech content of the audio file.

[0113] Step 430: transcribe the speech content to obtain the text to be analyzed.

[0114] Since the embodiment of the present application automatically determines whether the text to be analyzed needs to be translated based on the target language selected by the user, and if the text to be analyzed needs to be translated, the text to be analyzed will be translated. Therefore, the character dubbing system of the video content of the present application can support translation and synthesis of multiple languages. Users can select the target language as needed, and automatically perform translation and synthesis without additional manual intervention, thereby reducing dependence on professionals and meeting users' needs for multi-language dubbing.

[0115] Referring to the third embodiment of a method for dubbing a role in video content provided by the present application, the third embodiment may include the following steps:

[0116] Step 510: Obtain the video file to be processed.

[0117] Step 520: Based on the video file to be processed, obtain the text to be analyzed and the character's voice feature information.

[0118] Step 530: Based on the target language selected by the user, determine whether the text to be analyzed needs to be translated.

[0119] Step 540: If the text to be analyzed does not need to be translated, a synthesized speech is obtained based on the text to be analyzed and the character's speech feature information.

[0120] Step 550: Synchronize the audio information of the synthesized speech with the video information of the video file to be processed to obtain adjusted audio information.

[0121] Step 560: synthesize the adjusted audio information and video information to obtain a target video file.

[0122] Since the embodiment of the present application automatically determines whether the text to be analyzed needs to be translated based on the target language selected by the user, and if the text to be analyzed needs to be translated, the text to be analyzed will be translated. Therefore, the character dubbing system of the video content of the present application can support translation and synthesis of multiple languages. Users can select the target language as needed, and automatically perform translation and synthesis without additional manual intervention, thereby reducing dependence on professionals and meeting users' needs for multi-language dubbing.

[0123] Based on the above embodiments, the present application also provides a character dubbing system for video content, which is mainly used in the production and release of various video content such as movies, TV series, animations, advertisements, online education, and travel promotional films.

[0124] See also Figure 2 The character dubbing system 10 for video content provided in the embodiment of the present application includes a speech recognition and transcription module 100, a character speech feature extraction module 200, a video content analysis module 300, a multilingual translation module 400, a speech synthesis module 500 and an audio and video synchronization module 600.

[0125] The speech recognition and transcription module 100 is mainly used for the following operations:

[0126] 1) Audio separation.

[0127] That is, the audio is separated from the video file to be processed to obtain an audio file.

[0128] 2) Voice recognition.

[0129] That is, voice recognition is performed on the audio file to obtain the voice content of the audio file.

[0130] 3) Convert to text.

[0131] That is, the voice content is transcribed to obtain the text to be analyzed.

[0132] The character voice feature extraction module 200 is mainly used for the following operations:

[0133] 1) Speech feature extraction.

[0134] 2) Obtain speech waveform fragments based on the deep learning model.

[0135] The extracted speech features are input into a pre-trained deep learning model to obtain a speech waveform segment output by the pre-trained deep learning model.

[0136] 3) Character voice model training.

[0137] That is, the speech waveform segment is input into the character speech model to obtain the character speech feature information.

[0138] The video content analysis module 300 is mainly used for the following operations:

[0139] 1) Video frame extraction.

[0140] 2) Character action recognition.

[0141] That is, the role action is recognized in the video file to be processed, the role action features are obtained and the video information is output to the audio and video synchronization module 600 .

[0142] After obtaining the text to be analyzed, it can be determined whether translation is required based on the language selected by the user. If so, the text to be analyzed is input into the multilingual translation module 400 for translation. If not, the converted text information, i.e. the text to be analyzed, is directly input into the speech synthesis module 500 for speech synthesis.

[0143] The multilingual translation module 400 mainly translates the input transcribed text, ie, the text to be analyzed, into multiple languages ​​to generate a translated text, ie, a target language text, and then inputs the translated text information into the speech synthesis module 500 for speech synthesis.

[0144] The speech synthesis module 500 is mainly used for the following operations:

[0145] 1) Receive input text.

[0146] The input text may be converted text information or translated text information.

[0147] 2) Character voice matching.

[0148] That is, the character voice feature information is matched with the input text.

[0149] 3) Synthesized speech output.

[0150] The newly generated audio information can be extracted from the output synthesized speech and input into the audio and video synchronization module 600, so that the audio and video synchronization module 600 synchronizes the video information and the newly generated audio to ensure that the time points of the audio and video are consistent, and then outputs the generated video file.

[0151] Among them, for the synchronization operation of the video information and the newly generated audio, please refer to the relevant statements of the above embodiments, and this application will not go into details here.

[0152] In this way, the character dubbing system of the video content of the present application can support translation and synthesis of multiple languages. Users can select the target language as needed, and the system automatically translates and synthesizes without additional human intervention, reducing dependence on professionals and meeting users' needs for multi-language dubbing.

[0153] Corresponding to the above video content character dubbing method, the present application also provides a video content character dubbing device. The video content character dubbing device includes a unit for executing the above video content character dubbing method, and the video content character dubbing device can be configured in a desktop computer, tablet computer, laptop, etc.

[0154] like Figure 3 As shown, the embodiment of the present application provides a computer device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0155] Memory 113, used for storing computer programs;

[0156] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the method for dubbing a character of video content provided by any of the aforementioned method embodiments, including:

[0157] Get the video file to be processed;

[0158] Based on the video file to be processed, the text to be analyzed and the voice feature information of the character are obtained;

[0159] Based on the target language selected by the user, determine whether the text to be analyzed needs to be translated;

[0160] If the text to be analyzed needs to be translated, the text to be analyzed is translated to obtain the target language text;

[0161] Based on the target language text and the character's voice feature information, synthesized speech is obtained;

[0162] Synchronously processing the audio information of the synthesized speech and the video information of the video file to be processed to obtain adjusted audio information;

[0163] The adjusted audio information is synthesized with the video information to obtain a target video file.

[0164] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0165] Therefore, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for dubbing a character of video content provided in any of the aforementioned method embodiments are implemented, including:

[0166] Get the video file to be processed;

[0167] Based on the video file to be processed, the text to be analyzed and the voice feature information of the character are obtained;

[0168] Based on the target language selected by the user, determine whether the text to be analyzed needs to be translated;

[0169] If the text to be analyzed needs to be translated, the text to be analyzed is translated to obtain the target language text;

[0170] Based on the target language text and the character's voice feature information, synthesized speech is obtained;

[0171] Synchronously processing the audio information of the synthesized speech and the video information of the video file to be processed to obtain adjusted audio information;

[0172] The adjusted audio information is synthesized with the video information to obtain a target video file.

[0173] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which can store program codes. Computer-readable storage media can be non-volatile or volatile.

[0174] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0176] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0178] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0179] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for dubbing a role in a video content, characterized in that: The method comprises: Get the video file to be processed; Based on the video file to be processed, obtaining the text to be analyzed and the character voice feature information; Based on the target language selected by the user, determining whether the text to be analyzed needs to be translated; If the text to be analyzed needs to be translated, the text to be analyzed is translated to obtain a text in a target language; Based on the target language text and the character voice feature information, obtaining synthesized speech; Synchronously processing the audio information of the synthesized speech and the video information of the video file to be processed to obtain adjusted audio information; The adjusted audio information is synthesized with the video information to obtain a target video file.

2. The method according to claim 1, characterized in that Based on the video file to be processed, character voice feature information is obtained, including: Extracting language features from the video file to be processed to obtain speech features; Inputting the speech feature into a pre-trained deep learning model to obtain a speech waveform segment output by the pre-trained deep learning model; The speech waveform segment is input into the character speech model to obtain the character speech feature information.

3. The method according to claim 2, characterized in that The audio information of the synthesized speech includes the durations corresponding to a plurality of speech waveform segments. After obtaining the video file to be processed, the method includes: Recognizing the action of the character in the video file to be processed to obtain the action features of the character; Extracting the duration of the character action feature as the video information of the video file to be processed; The step of synchronously processing the audio information of the synthesized speech and the video information of the video file to be processed to obtain the adjusted audio information includes: The durations corresponding to the multiple speech waveform segments are synchronously processed with the duration of the character action feature to obtain an adjusted speech waveform duration.

4. The method according to claim 3, characterized in that The step of synchronizing the durations corresponding to the plurality of speech waveform segments with the duration of the character action feature to obtain an adjusted speech waveform duration comprises: Obtain the duration of each character's action feature and the duration of each corresponding waveform segment respectively; Based on the duration of the character action feature, the duration of the corresponding speech waveform segment is dynamically adjusted to obtain an adjusted speech waveform duration.

5. The method according to claim 4, characterized in that The dynamically adjusting the duration of the corresponding speech waveform segment based on the duration of the character action feature includes: If the duration of the speech waveform segment is greater than the duration of the corresponding character action, shortening the duration of the speech waveform segment to the duration of the corresponding character action; If the duration of the speech waveform segment is shorter than the duration of the corresponding character action, the duration of the speech waveform segment is extended to the duration of the corresponding character action.

6. The method according to claim 1, characterized in that The step of obtaining a text to be analyzed based on the video file to be processed includes: Separating the audio from the video file to be processed to obtain an audio file; Performing speech recognition on the audio file to obtain the speech content of the audio file; The speech content is transcribed to obtain a text to be analyzed.

7. The method according to claim 1, characterized in that The method further comprises: determining whether to translate the text to be analyzed based on the target language selected by the user. If the text to be analyzed does not need to be translated, a synthesized speech is obtained based on the text to be analyzed and the character's speech feature information.

8. A device for dubbing characters in video content, characterized in that: The method comprises a unit for executing the method according to any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images