A dubbing method, device, electronic device and storage medium
By correcting and synthesising the dubbing audio and video image information of film and television works, the problem of insufficient dubbing quality in the existing technology is solved, the coordination of character lip styling and voice is achieved, and the dubbing quality and audience experience are improved.
Patent Information
- Application Number
- CN202210778985.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-06-30
AI Technical Summary
The existing technology is difficult to improve the dubbing quality of overseas film and television works, resulting in the inconsistency of the character's lip and voice when watching the dubbing video.
By obtaining the first dubbing audio corresponding to the video to be dubbed, the image information in the video is lip-shaped and the corrected image information is synthesized with the audio to ensure that the character's lip-shaped and dubbed are consistent.
It improves the dubbing quality, reduces the audience's bad viewing experience caused by incoordination of character lip shape and voice, and improves the difficulty of video comprehension and audience satisfaction.
Smart Images

Figure CN115174825B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of dubbing, and in particular to a dubbing method, device, electronic device and storage medium. Background Art
[0002] In order to increase the popularity of overseas film and television works in China, it is necessary to localize the language of the film and television content, which mainly includes two processing methods: "dubbing" and "original sound plus subtitles". Among them, the "dubbing" method can fully reduce the difficulty of audiences' understanding of film and television works and is deeply loved by the audience.
[0003] At present, the promotion of overseas film and television works is limited. Therefore, how to improve the dubbing quality of overseas film and television works is an urgent problem to be solved. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a dubbing method, device, electronic device and storage medium.
[0005] In a first aspect, the present disclosure provides a dubbing method, comprising:
[0006] Obtaining a first dubbing audio corresponding to the video to be dubbed;
[0007] Based on the first dubbing audio, performing lip shape correction on image information corresponding to the first dubbing audio in the video to be dubbed;
[0008] The image information after lip shape correction is synthesized with the first dubbing audio to obtain a dubbed video.
[0009] In a second aspect, the present disclosure further provides a dubbing device, comprising:
[0010] An acquisition module, used for acquiring a first dubbing audio corresponding to the video to be dubbed;
[0011] A correction module, configured to correct the lip shape of the image information corresponding to the first dubbing audio in the video to be dubbed based on the first dubbing audio;
[0012] The synthesis module is used to synthesize the image information after lip shape correction with the first dubbing audio to obtain a dubbed video.
[0013] In a third aspect, the present disclosure further provides an electronic device, the electronic device comprising:
[0014] one or more processors;
[0015] A storage device for storing one or more programs;
[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the dubbing method as described above.
[0017] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the dubbing method as described above when executed by a processor.
[0018] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0019] The technical solution provided by the embodiment of the present disclosure obtains the first dubbing audio corresponding to the video to be dubbed; based on the first dubbing audio, performs lip shape correction on the image information corresponding to the first dubbing audio in the video to be dubbed; synthesizes the image information after lip shape correction with the first dubbing audio to obtain the dubbed video. By correcting the lip shape of the character in the image information, the lip shape of the character in the corrected image is consistent with the dubbing, thereby achieving the purpose of improving the dubbing quality and reducing the probability of the audience watching the dubbed video due to the incoordination between the lip shape of the character and the voice of the character, which leads to the bad situation of being out of the play during the viewing process. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0022] Figure 1 A flowchart of a dubbing method provided by an embodiment of the present disclosure;
[0023] Figure 2 A flowchart of another dubbing method provided by an embodiment of the present disclosure;
[0024] Figure 3 A schematic diagram of a translation proofreading interface provided by an embodiment of the present disclosure;
[0025] Figure 4 A schematic diagram of another translation proofreading interface provided by an embodiment of the present disclosure;
[0026] Figure 5 A flowchart of another dubbing method provided by an embodiment of the present disclosure;
[0027] Figure 6A flowchart of another dubbing method provided by an embodiment of the present disclosure;
[0028] Figure 7 is a structural schematic diagram of a dubbing device in an embodiment of the present disclosure;
[0029] Figure 8 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0030] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0032] Figure 1 This is a flowchart of a dubbing method provided in an embodiment of the present disclosure. This embodiment is applicable to the case where dubbing is performed on a video in a client. The method can be executed by a dubbing device, which can be implemented in software and / or hardware. The device can be configured in an electronic device, such as a terminal, specifically including but not limited to a smart phone, a PDA, a tablet computer, a wearable device with a display screen, a desktop computer, a laptop computer, an all-in-one machine, a smart home device, etc. Alternatively, this embodiment is applicable to the case where dubbing is performed on a video in a server. The method can be executed by a dubbing device, which can be implemented in software and / or hardware. The device can be configured in an electronic device, such as a server.
[0033] like Figure 1 As shown, the method may specifically include:
[0034] S110: Obtain a first dubbing audio corresponding to the video to be dubbed.
[0035] The video to be dubbed refers to a video that needs dubbing. Optionally, the video to be dubbed includes a web series, a documentary, and other fast-produced works, or a large-scale film and television work, such as a movie, a TV series, and the like.
[0036] The first dubbing audio is the voice information that is desired to be presented after dubbing. The first dubbing audio can be obtained by recording the dubbing actor, or can be generated by software. In practice, the first dubbing audio is the voice information obtained by dubbing the dialogue, confession or narration of the character in the video.
[0037] The language of the first dubbing audio and the language of the video to be dubbed can be the same or different. Different languages refer to different types of languages, which can be divided according to countries or regions. For example, Chinese and English are different languages, Korean and Japanese are different languages, Sichuan dialect and Shanghai dialect are different languages, British English and American English are different languages.
[0038] There are many methods for implementing this step, which are not limited in this application. Exemplarily, the method for implementing this step includes: obtaining an original text corresponding to the video to be dubbed; the language of the original text is the same as the language of the video to be dubbed; based on the original text, a final translation text is obtained; the absolute value of the difference between the number of phonemes corresponding to the final translation text and the target number of phonemes is less than or equal to a set threshold; the target number of phonemes is determined based on the duration of the audio information corresponding to the original text in the video to be dubbed; and obtaining a first dubbing audio corresponding to the final translation text.
[0039] S120: Based on the first dubbing audio, perform lip shape correction on image information corresponding to the first dubbing audio in the video to be dubbed.
[0040] There are many methods for implementing this step, and this application does not limit this. Exemplarily, the method for implementing this step includes: determining the phonemes included in the first dubbing audio; determining the lip shape corresponding to each phoneme; using the lip shape corresponding to each phoneme to correct the lip shape of the image information corresponding to the first dubbing audio in the dubbing video. Among them, phoneme refers to the smallest speech unit divided according to the natural properties of speech. From the perspective of acoustic properties, phoneme is the smallest speech unit divided from the perspective of sound quality. From the perspective of physiological properties, one pronunciation action forms a phoneme. For example, [ma] contains two pronunciation actions [m] and [a], which are two phonemes.
[0041] In one embodiment, a lip shape database is pre-built, and the lip shape database stores a plurality of data sets of correspondence between phonemes and lip shapes. By parsing the first dubbing audio, all phonemes corresponding to the first dubbing audio can be obtained. Further, in combination with the video to be dubbed, the timestamp corresponding to each phoneme can be obtained.
[0042] Those skilled in the art can understand that in order to achieve the dubbing effect, the first dubbing audio and the image information of the video to be dubbed need to be synthesized later. After the synthesis, each phoneme in the first dubbing audio will correspond to a specific time information, which is the timestamp corresponding to the phoneme. By querying the lip shape database, the lip shape corresponding to each phoneme in the first dubbing audio can be obtained, and based on the timestamp corresponding to each phoneme, the image information corresponding to each phoneme can be obtained. The image information corresponding to each phoneme can be corrected using the lip shape corresponding to each phoneme to obtain the corrected image information.
[0043] For example, the timestamp of the first phoneme [m] in the first dubbing audio is 1:20:35. The lip shape of the phoneme [m] is lip shape M, and the image information with the timestamp of 1:20:35 is separated from the video to be dubbed, and the image information with the timestamp of 1:20:35 is corrected using lip shape M. This is repeated until the image information in the video to be dubbed is corrected based on each phoneme in the first dubbing audio, and the image information is finally obtained after the lip shape correction.
[0044] Optionally, the lip shape database can be calculated based on the audio in the video and the lip shape in the image information to obtain the lip shape database.
[0045] Optionally, in practice, a public lip shape database can be constructed, that is, no matter which video to be dubbed is dubbed, the public lip shape database is queried to obtain the lip shape corresponding to each phoneme in the first dubbing audio. Alternatively, a lip shape database corresponding to each video to be dubbed can be established. When performing lip shape correction, the lip shape database corresponding to the video to be dubbed is queried. Alternatively, a lip shape database corresponding to each actor of each role can be established. When performing lip shape correction, the lip shape database corresponding to the actor of the role to which the first dubbing audio belongs is queried.
[0046] S130: synthesize the image information after the lip shape correction with the first dubbing audio to obtain a dubbed video.
[0047] The above technical solution obtains the first dubbing audio corresponding to the video to be dubbed; based on the first dubbing audio, performs lip shape correction on the image information corresponding to the first dubbing audio in the video to be dubbed; synthesizes the image information after lip shape correction with the first dubbing audio to obtain the dubbed video. By correcting the lip shape of the character in the image information, the lip shape of the character in the corrected image is consistent with the dubbing, thereby achieving the purpose of improving the dubbing quality and reducing the probability of the audience watching the dubbed video due to the incoordination between the lip shape and the voice of the character, which leads to the bad situation of being out of the play during the viewing process.
[0048] Figure 2 A method for implementing the present disclosure is provided in an embodiment Figure 1 Flow chart of step S110 method in FIG. Figure 2 , the method comprising:
[0049] S111, obtaining an original text corresponding to the video to be dubbed; if the language of the original text is the same as the language of the video to be dubbed, executing S112.
[0050] The original text is the text information used as the basis for translation. In practice, if the video to be dubbed includes subtitle information, the original text is the subtitle information directly separated from the video to be dubbed. If the video to be dubbed does not include subtitle information, the original text is the text information obtained by extracting audio from the video to be dubbed and performing speech recognition based on the extracted audio.
[0051] S112. Based on the original text, an intermediate translation text is obtained, and S113 is executed.
[0052] The intermediate translation text is the translation result obtained by translating the original text.
[0053] S113, determining whether the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to a set threshold; if so, executing S114; if not, executing S115.
[0054] In this step, the concept of phonemes is consistent with the concept of phonemes mentioned in the previous section on lip shape correction.
[0055] The number of phonemes corresponding to the intermediate translation text refers to the number of phonemes corresponding to the audio after the intermediate translation text is converted into audio. In other words, the number of phonemes corresponding to the intermediate translation text refers to the total number of phonemes that need to be spoken when speaking the intermediate translation text.
[0056] The target number of phonemes is determined based on the duration of the audio information corresponding to the original text in the audio and video. Specifically, the target number of phonemes refers to the number of phonemes that can be accommodated within the duration of the audio information corresponding to the original text in the video to be dubbed. Among them, "can accommodate" should be understood as the number of phonemes that can be spoken within the duration of dubbing at a preset speaking speed. This application does not impose any restrictions on the preset speaking speed, but it is necessary to ensure that at the preset speaking speed, the user can understand the content expressed by the dubbing.
[0057] S114: Take the intermediate translation text as the final translation text and execute S116.
[0058] If the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to the set threshold, the intermediate translation text is considered to meet the standard, and therefore, the intermediate translation text is used as the final translation text.
[0059] S115. In response to the instruction to modify the intermediate translation text, modify the intermediate translation text and execute S113.
[0060] If the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is greater than the set threshold, the intermediate translation text is considered to be substandard and needs to be further modified. After the modification, S113 is executed, that is, it is determined again whether the modified intermediate translation text meets the standard.
[0061] S116: Obtain a first dubbing audio corresponding to the final translation text.
[0062] The above technical solution can ensure that the number of phonemes corresponding to the final translation text is relatively appropriate, and the number of phonemes corresponding to the first dubbing audio obtained based on the final translation text is relatively appropriate. After the first dubbing audio is synthesized with the corrected image information, the obtained video, while achieving the effect of "sound and picture synchronization", has appropriate speaking speeds for each character, and no character speaks too fast or too slow, thereby reducing the difficulty of understanding the video and improving the quality of dubbing.
[0063] On the basis of the above technical solution, optionally, the method further includes: displaying a translation proofreading interface, the translation proofreading interface includes the original text, the intermediate translation text and the proofreading target information; the proofreading target information is used to indicate whether the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to the set threshold. Among them, the translation proofreading interface refers to a page that assists the proofreader in proofreading the intermediate translation text. In addition, the translation proofreading interface allows the proofreader to edit and modify the intermediate translation text. Since the translation proofreading interface can directly and synchronously display the original text, the intermediate translation text and the proofreading target information, during the proofreading process, the proofreader clarifies the problems existing in the current intermediate translation according to the proofreading target information, and modifies the intermediate translation text information based on the original text information to achieve the purpose of proofreading, which can reduce the difficulty of proofreading the intermediate translation text.
[0064] Optionally, in response to the modification instruction of the intermediate translation text, after the intermediate translation text is modified, the method further includes: updating the proofreading target information, so as to assist the proofreader to understand again whether there are still problems with the modified intermediate translation text.
[0065] On the basis of the above technical solutions, the original text includes one or more sub-original texts; the intermediate translation text includes one or more sub-intermediate translation texts; the proofreading target information includes one or more sub-proofreading target information; a sub-original text, a sub-intermediate translation text and a sub-proofreading target information constitute a proofreading information group; in any proofreading information group, the sub-original text, the sub-intermediate translation text and the sub-proofreading target information have a corresponding relationship. The essence of such a setting is to reduce the difficulty of proofreading the intermediate translation text by breaking the whole into parts. Optionally, in practice, a sentence is a sub-original text.
[0066] In practice, there are many ways to display the original text, the intermediate translation text, and the proofreading target information using the translation proofreading interface, and this application does not limit this. Two display methods are given below as examples.
[0067] Method 1
[0068] The translation proofreading interface includes a first area and a second area. Each sub-original text is displayed vertically in the first area, and each sub-intermediate translation text is displayed vertically in the second area; each sub-proofreading target information is displayed in the second area, and the distance between the display position of each sub-proofreading target information and the display position of the corresponding sub-intermediate translation text is less than a set distance threshold; in any proofreading information group, the sub-original text and the sub-intermediate translation text are in a horizontal comparison relationship.
[0069] Among them, the distance between the display position of each sub-proofreading target information and the display position of the corresponding sub-intermediate translation text is less than the set distance threshold, which means that the distance between each sub-proofreading target information and the corresponding sub-intermediate translation text is relatively close, so that the proofreader can intuitively understand the correspondence between each sub-proofreading target information and each sub-intermediate translation text.
[0070] Figure 3 A schematic diagram of a translation proofreading interface provided by an embodiment of the present disclosure. Figure 3 , the translation proofreading interface includes a first area A and a second area B. The original text 10 includes a sub-original text 11, a sub-original text 12, and a sub-original text 13. The sub-original text 11, the sub-original text 12, and the sub-original text 13 are displayed in the first area A in sequence along the vertical direction. The intermediate translation text 20 includes a sub-intermediate translation text 21, a sub-intermediate translation text 22, and a sub-intermediate translation text 23. The sub-intermediate translation text 21, the sub-intermediate translation text 22, and the sub-intermediate translation text 23 are displayed in the second area B in sequence along the vertical direction. The proofreading target information 30 includes a sub-proofreading target information 31, a sub-proofreading target information 32, and a sub-proofreading target information 33. The sub-proofreading target information 31, the sub-proofreading target information 32, and the sub-proofreading target information 33 are displayed in the second area B in sequence along the vertical direction.
[0071] The sub-original text 11, the sub-intermediate translation text 21 and the sub-proofreading target information 31 constitute a proofreading information group. The sub-intermediate translation text 21 is the translation result of the sub-original text 11, and the sub-proofreading target information 31 is used to indicate whether the absolute value of the difference between the number of phonemes corresponding to the sub-intermediate translation text 21 and the target number of phonemes of the proofreading information group is less than or equal to a set threshold. The target number of phonemes of the proofreading information group is determined based on the duration of the audio information corresponding to the sub-original text 11 in the video to be dubbed. Therefore, the sub-original text 11, the sub-intermediate translation text 21 and the sub-proofreading target information 31 have a corresponding relationship.
[0072] The relationship between the sub-original text 12, the sub-intermediate translation text 22 and the sub-proofreading target information 32, and the relationship between the sub-original text 13, the sub-intermediate translation text 23 and the sub-proofreading target information 33 are similar to the relationship between the sub-original text 11, the sub-intermediate translation text 21 and the sub-proofreading target information 31, and will not be repeated here.
[0073] Continue to see Figure 3 In any proofreading information group, the sub-original texts and the sub-intermediate translation texts are in a horizontal comparison relationship, and the sub-proofreading target information is located at the upper right corner of the corresponding sub-intermediate translation text, so that the proofreader can identify the problems existing in each sub-intermediate translation text and proofread each sub-intermediate translation text with reference to each sub-original text, which helps to improve the proofreading efficiency and accuracy.
[0074] Method 2
[0075] The translation proofreading interface includes a third area; each proofreading information group is displayed in the third area in sequence along the vertical direction; in any proofreading information group, the sub-original text and the sub-intermediate translation text are in a vertical comparison relationship; the display position of each sub-proofreading target information and the display position of the corresponding sub-intermediate translation text are less than a set distance threshold.
[0076] Similarly, if the distance between the display position of each sub-proofreading target information and the display position of the corresponding sub-intermediate translation text is less than the set distance threshold, it means that the distance between each sub-proofreading target information and the corresponding sub-intermediate translation text is relatively close, so that the proofreader can intuitively understand the correspondence between each sub-proofreading target information and each sub-intermediate translation text.
[0077] Figure 4 This is a schematic diagram of another translation proofreading interface provided by an embodiment of the present disclosure. Figure 4, the translation proofreading interface includes a third area C; each proofreading information group is arranged vertically in the third area C. The sub-original texts and sub-intermediate translation texts in any proofreading information group are arranged vertically. For example, the sub-original text 11 and the sub-intermediate translation text 21 in the first proofreading information group are arranged vertically. The sub-original text 12 and the sub-intermediate translation text 22 in the second proofreading information group are arranged vertically. The sub-original text 13 and the sub-intermediate translation text 23 in the third proofreading information group are arranged vertically.
[0078] Each sub-proofreading target information is located at the upper right corner of the corresponding sub-intermediate translation text. Exemplarily, the sub-proofreading target information 31 is located at the upper right corner of the corresponding sub-intermediate translation text 21, so that the proofreader can identify the problems existing in each sub-intermediate translation text and proofread each sub-intermediate translation text with reference to each sub-original text, which helps to improve the proofreading efficiency and accuracy.
[0079] It should be noted that, in the above embodiment, the display position of each sub-proofreading target information may also be located on the left, right, above or below the display position of the corresponding sub-intermediate translation text.
[0080] In one embodiment, in any proofreading information group, if the absolute value of the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes corresponding to the proofreading information group is less than or equal to a set threshold, the sub-proofreading target information is in a target achievement state; if the absolute value of the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes corresponding to the proofreading information group is greater than the set threshold, the sub-proofreading target information is in a target non-achievement state; the target number of phonemes corresponding to the proofreading information group is determined based on the sub-original text in the proofreading information group.
[0081] For example, the threshold is set to 0. Figure 3 or Figure 4 In the first proofreading information group, the difference between the number of phonemes corresponding to the sub-intermediate translation text 21 and the target number of phonemes of the proofreading information group is equal to 0, and the sub-proofreading target information 31 of the proofreading information group is in the target achievement state. In the second proofreading information group, the difference between the number of phonemes corresponding to the sub-intermediate translation text 22 and the target number of phonemes corresponding to the proofreading information group is equal to -5, and its absolute value (5) is greater than 0, and the sub-proofreading target information 32 of the proofreading information group is in the target unachieved state. In the third proofreading information group, the difference between the number of phonemes corresponding to the sub-intermediate translation text 23 and the target number of phonemes corresponding to the proofreading information group is equal to +1, and its absolute value (1) is greater than 0, and the sub-proofreading target information 33 of the proofreading information group is in the target unachieved state.
[0082] In one embodiment, the specific implementation method of "the target number of phonemes corresponding to the proofreading information group is determined based on the sub-original text in the proofreading information group" may include: based on the video to be dubbed, obtaining the start time and end time of any sub-original text, and then obtaining the duration of any sub-original text. Since the duration is proportional to the number of phonemes it can accommodate, a functional relationship between the duration and the number of phonemes it can accommodate can be pre-constructed, and the functional relationship uses the duration as the independent variable and the number of phonemes that can be accommodated as the dependent variable. Based on this functional relationship and the duration of any sub-original text, the number of phonemes that can be accommodated within the duration of any sub-original text can be obtained, that is, the target number of phonemes in the proofreading information group to which any sub-original text belongs.
[0083] In one embodiment, the sub-proofreading target information in the target achievement state includes a target achievement prompt; the sub-proofreading target information in the target unachieved state includes the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes. Figure 3 or Figure 4 , the target achievement prompt is "√". This setting can help proofreaders quickly understand which sub-intermediate translation texts meet the standards and do not need to be revised; which sub-intermediate translation texts do not meet the standards and need to be revised, thereby improving proofreading efficiency.
[0084] Optionally, in practice, the sub-proofreading target information in the target unachieved state may also be set to include the difference between the target number of phonemes and the number of phonemes corresponding to the current sub-intermediate translation text.
[0085] In another embodiment, the background color of the sub-proofreading target information in the target-achieved state is different from the background color of the sub-proofreading target information in the target-unachieved state; and / or the border color of the proofreading target information in the target-achieved state is different from the border color of the proofreading target information in the target-unachieved state. This setting can help the proofreader quickly understand which sub-intermediate translation texts have met the standards and do not need to be further revised; which sub-intermediate translation texts have not met the standards and need to be further revised, thereby improving the efficiency of proofreading.
[0086] On the basis of the above technical solutions, optionally, continue to refer to Figure 3 or Figure 4 The translation proofreading interface also includes an audio and video playback window D, which is used to play the video to be dubbed. This arrangement allows the proofreader to proofread the intermediate translation text while watching the video and / or listening to the audio, which can meet the proofreader's diverse proofreading needs.
[0087] Figure 5 A flowchart of another dubbing method provided in an embodiment of the present disclosure. Figure 5 for Figure 1 See Figure 5 , the method comprising:
[0088] S210: Obtain a first dubbing audio corresponding to the video to be dubbed.
[0089] The first dubbing audio can be obtained by recording the dubbing actor, or can be generated by software.
[0090] If the first dubbing audio is obtained by recording a dubbing actor, it includes the emotion characteristic information and speech flow characteristic information assigned by the dubbing actor, but lacks the timbre characteristic information of the character to which the first dubbing audio belongs.
[0091] Speech flow is a language expression process created by the semantic materials composed of characters, words and sentences. In speech flow, due to the influence of adjacent phonemes of adjacent syllables, the initial consonants, finals or tones in some syllables will undergo phonetic changes. This phenomenon is called speech flow sound change. Speech flow sound change includes assimilation, alienation, weakening and dropout. Assimilation means that two adjacent different sounds in speech flow, one of which is affected by the other, becomes the same or similar to the other in one or more features. Alienation means that one of two originally identical or similar sounds becomes different from its original pronunciation for some reason. Weakening means that in speech flow, the pronunciation of some sounds may become weaker and lighter than before. Dropout means that some sounds in speech flow are no longer pronounced or disappear due to contraction and other reasons when they are read together. Speech flow feature information includes at least one of assimilation, alienation, weakening and dropout.
[0092] If the first dubbing audio is generated by software, the first dubbing audio lacks timbre feature information, emotion feature information, and speech flow feature information of the character to which the first dubbing audio belongs.
[0093] S220, fusing the first dubbing audio with the target voice feature information to obtain a second dubbing audio; the target voice feature information is the voice feature information of the character to which the first dubbing audio belongs in the video to be dubbed.
[0094] Voice feature information refers to the feature information presented by a character when speaking. Since different characters have different voice features when speaking, the audience can easily distinguish which character is speaking based on the voice feature information. The target voice feature information is the voice feature information of the character to which the first dubbing audio belongs in the video to be dubbed.
[0095] For example, if the video to be dubbed includes a dialogue between character A and character B, character A is dubbed, and the words spoken by character A are "ZZZZZZZZ", the dubbing artist emotionally says "ZZZZZZZZ" based on the plot to be dubbed, and the voice recording device is used to collect the voice of "ZZZZZZZZ" spoken by the dubbing artist to obtain the first dubbing audio. The target voice feature information refers to the voice feature information of character A.
[0096] The second dubbing audio is a result of fusing the first dubbing audio with the target voice feature information.
[0097] There are many ways to implement this step, and this application does not limit this. For example, based on a cross-language TTS (Text To Speech) model, the first dubbing audio is fused with the target speech feature information to obtain the second dubbing audio; the cross-language TTS model is learned based on the original text and original audio corresponding to the dubbing video.
[0098] Specifically, firstly, machine learning is performed based on the original text and original audio corresponding to the video to be dubbed to obtain a cross-language TTS model. The obtained cross-language TTS model has learned the target voice feature information. After the first dubbing audio is input into the cross-language TTS model, the cross-language TTS model can fuse the first dubbing audio with the learned target voice feature information to obtain the second dubbing audio.
[0099] Optionally, if the first dubbing audio is obtained by recording a dubbing actor, the target voice feature information includes the timbre feature information of the character to which the first dubbing audio belongs. Alternatively, if the first dubbing audio is generated by software, the target voice feature information includes one or more of the timbre feature information, emotion feature information, and speech flow feature information of the character to which the first dubbing audio belongs.
[0100] S230: Perform lip shape correction on the image information corresponding to the first dubbing audio in the video to be dubbed.
[0101] S240: synthesize the image information after the lip shape correction with the second dubbing audio to obtain a dubbed video.
[0102] Since the second dubbing audio is the first dubbing audio superimposed with the target voice feature information, if the lip shape of the video image after dubbing is consistent with the first dubbing audio, it means that the lip shape of the video image after dubbing is consistent with the second dubbing audio. The above technical scheme obtains the second dubbing audio by fusing the first dubbing audio with the target voice feature information; the target voice feature information is the voice feature information of the character to which the first dubbing audio belongs in the video to be dubbed; the image information corresponding to the first dubbing audio in the video to be dubbed is corrected in lip shape; the image information after lip shape correction is synthesized with the second dubbing audio to obtain the video after dubbing. It can make the dubbing audio have the voice feature information of the original character, and the lip shape of the video image is consistent with the dubbing, thereby compensating to a certain extent the influence of the recording equipment, recording venue, dubbing time and the dubbing artist's own level on the dubbing effect, so as to improve the dubbing quality and reduce the number of times that the video audience is distracted by the poor dubbing quality.
[0103] On the basis of the above technical solutions, optionally, the method further includes: determining a target duration corresponding to the first dubbing audio, and adjusting the duration of the first dubbing audio so that the absolute value of the difference between the duration of the first dubbing audio and the target duration is less than or equal to a preset time length. The target duration refers to the time length that the first dubbing audio is allowed to play in the video to be dubbed. In the scenario where the video to be dubbed includes the original dubbing, the target duration refers to the time spent by the character to which the first dubbing audio belongs in the video to be dubbed to speak the original text.
[0104] Exemplarily, if the video to be dubbed includes a plot in which character A and character B have a conversation, character A is dubbed, and character A says "XXXXXX" in the first language, and "XXXXXX" is the original text. "XXXXXX" is translated to obtain the final translated text "YYYYYYYY" in the second language, and the dubbing artist says "YYYYYYYY" based on the final translated text. The words spoken by the dubbing artist are the first dubbing audio. The duration of the first dubbing audio is the length of time required to play the first dubbing audio. The target duration corresponding to the first dubbing audio refers to the length of time required for character A to say "XXXXXX" in the video to be dubbed.
[0105] The duration of the first dubbing audio is adjusted, specifically including but not limited to speeding up or slowing down the playing speed of the first dubbing audio. By adjusting the duration of the first dubbing audio so that the absolute value of the difference between the duration of the first dubbing audio and the target duration is less than or equal to the preset time length, the effect of "sound and picture synchronization" can be further achieved.
[0106] Based on the above technical solutions, the method may further include: determining background audio based on the video to be dubbed; synthesizing the background audio with the first dubbing audio to obtain a third dubbing audio; S130 is replaced by: synthesizing the image information after the lip shape correction with the third dubbing audio to obtain the video after dubbing. The essence of this setting is to make the video after dubbing still include the background audio of the video before dubbing, thereby ensuring that there is no information loss in the video after dubbing, and ensuring that the video after dubbing has a higher artistic effect.
[0107] Optionally, before synthesizing the background sound audio with the first dubbing audio to obtain the third dubbing audio, the background sound audio is optimized, such as eliminating noise information in the background sound audio, etc. The optimized background sound audio is synthesized with the first dubbing audio to obtain the third dubbing audio.
[0108] Figure 6 A flowchart of another dubbing method provided in an embodiment of the present disclosure. Figure 6 for Figure 1 See Figure 6 ,The dubbing method can be divided into three stages during its execution, namely background sound ,processing, TTS model training and dubbing lip shape simulation.
[0109] In the background sound processing stage, the video to be dubbed is obtained; the original audio in the video to be dubbed is separated into tracks to obtain background sound audio and human voice audio, where the human voice audio is the original audio. The background sound audio is optimized to obtain optimized background sound audio.
[0110] In the TTS model training phase, it is determined whether the video to be dubbed includes subtitles. If the video to be dubbed includes subtitles, the subtitles are the original text. If the video to be dubbed does not include subtitles, speech recognition is performed on the original audio to obtain the original text. Machine learning is performed on the original audio and original text to obtain a cross-language TTS model with the speech features of the character to which the original audio belongs.
[0111] In the dubbing lip simulation stage, the original text is translated to obtain an intermediate translation text. It is determined whether the absolute value of the difference between the number of phonemes corresponding to a single sentence of the current intermediate translation text and the target number of phonemes corresponding to the sentence is less than or equal to the set threshold. If the absolute value of the difference between the number of phonemes corresponding to a single sentence of the intermediate translation text and the target number of phonemes corresponding to the sentence is less than or equal to the set threshold, the intermediate translation text is determined as the final translation text. The dubbing audio (i.e., the first dubbing audio) formed by the dubbing artist reading the final translation text is obtained. If the absolute value of the difference between the number of phonemes corresponding to a single sentence of the intermediate translation text and the target number of phonemes is greater than the set threshold, the proofreading target information is output to prompt that the absolute value of the difference between the number of phonemes corresponding to the current intermediate translation text and the target number of phonemes corresponding to the sentence is greater than the set threshold, so that the translation proofreader can modify the intermediate translation text based on the proofreading target information. After the intermediate translation text is modified, it is determined again whether the absolute value of the difference between the number of phonemes corresponding to the current intermediate translation text and the target number of phonemes is less than or equal to the set threshold. This process is repeated until the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to the set threshold.
[0112] The dubbing audio (ie, the first dubbing audio) is input into the cross-language TTS model to obtain the dubbing audio (ie, the second dubbing audio) synthesized with the target voice feature information. The target voice feature information includes the timbre feature information of the character to which the dubbing audio belongs.
[0113] Determine whether the absolute value of the difference between the duration of the dubbing audio (i.e., the second dubbing audio) and the audio track corresponding to the picture corresponding to the dubbing audio in the video to be dubbed (i.e., the target duration) is less than or equal to the preset time length. If so, keep the duration of the dubbing audio unchanged; if not, adjust the duration of the dubbing audio (i.e., the second dubbing audio) until the absolute value of the difference between the duration of the dubbing audio (i.e., the second dubbing audio) and the target duration is less than or equal to the preset time length. After determining that the absolute value of the difference between the duration of the dubbing audio (i.e., the second dubbing audio) and the audio track corresponding to the picture corresponding to the dubbing audio in the video to be dubbed (i.e., the target duration) is less than or equal to the preset time length, perform reverberation processing on the dubbing audio and the optimized background audio to obtain the third dubbing audio.
[0114] Usually, the lip shape database is queried to determine the lip shape corresponding to each phoneme in the dubbing audio (i.e., the first dubbing audio); the lip shape corresponding to each phoneme is used to correct the lip shape of the image information corresponding to the dubbing audio (i.e., the first dubbing audio) in the dubbing video. The image information after the lip shape correction is synthesized with the third dubbing audio to obtain the dubbed video.
[0115] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0116] Figure 7 Schematic diagram of the structure of a dubbing device in an embodiment of the present disclosure. The dubbing device provided in the embodiment of the present disclosure can be configured in a client or in a server. Figure 7 , the dubbing device specifically comprises:
[0117] An acquisition module 310 is used for acquiring a first dubbing audio corresponding to the video to be dubbed;
[0118] A correction module 320, configured to correct the lip shape of the image information corresponding to the first dubbing audio in the video to be dubbed based on the first dubbing audio;
[0119] The synthesis module 330 is used to synthesize the image information after the lip shape correction with the first dubbing audio to obtain a dubbed video.
[0120] Furthermore, the correction module is used to:
[0121] Determining the phonemes corresponding to the first dubbing audio;
[0122] Determining the mouth shape corresponding to each of the phonemes;
[0123] The lip shapes corresponding to the phonemes are used to correct the lip shapes of the image information corresponding to the first dubbing audio in the video to be dubbed.
[0124] Furthermore, the language of the first dubbing audio is different from the language of the video to be dubbed.
[0125] Furthermore, a module is obtained for:
[0126] Acquire an original text corresponding to the video to be dubbed; the language of the original text is the same as the language of the video to be dubbed;
[0127] Based on the original text, a final translation text is obtained; the absolute value of the difference between the number of phonemes corresponding to the final translation text and the target number of phonemes is less than or equal to a set threshold; the target number of phonemes is determined based on the duration of the audio information corresponding to the original text in the video to be dubbed;
[0128] A first dubbing audio corresponding to the final translated text is obtained.
[0129] Furthermore, a module is obtained for:
[0130] Based on the original text, an intermediate translation text is obtained;
[0131] Determining whether the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to a set threshold;
[0132] If yes, use the intermediate translation text as the final translation text;
[0133] If not, in response to an instruction to modify the intermediate translation text, modify the intermediate translation text.
[0134] Furthermore, the device also includes a display module, which is used to:
[0135] A translation proofreading interface is displayed, wherein the translation proofreading interface includes the original text, the intermediate translation text, and proofreading target information; the proofreading target information is used to indicate whether the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to a set threshold.
[0136] Furthermore, the display module is used for: responding to a modification instruction for the intermediate translation text, modifying the intermediate translation text, and then updating the proofreading target information.
[0137] Furthermore, the original text includes one or more sub-original texts; the intermediate translation text includes one or more sub-intermediate translation texts; the proofreading target information includes one or more sub-proofreading target information;
[0138] One sub-original text, one sub-intermediate translation text and one sub-proofreading target information constitute a proofreading information group; in any one of the proofreading information groups, the sub-original text, the sub-intermediate translation text and the sub-proofreading target information have a corresponding relationship.
[0139] Furthermore, in any one of the proofreading information groups, if the absolute value of the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes is less than or equal to a set threshold, the sub-proofreading target information is in a target achievement state; if the absolute value of the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes is greater than the set threshold, the sub-proofreading target information is in a target non-achievement state; the target number of phonemes is determined based on the sub-original text in the proofreading information group.
[0140] Further, the sub-proofreading target information in the target achievement state includes a target achievement prompt;
[0141] The sub-proofreading target information in the target-unreached state includes the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes; or, the sub-proofreading target information in the target-unreached state includes the difference between the target number of phonemes and the number of phonemes corresponding to the current intermediate translation text.
[0142] Further, the background color of the sub-proofreading target information in the target-achieved state is different from the background color of the sub-proofreading target information in the target-unachieved state; and / or,
[0143] The border color of the sub-proofreading target information in the target-achieved state is different from the border color of the sub-proofreading target information in the target-unachieved state.
[0144] Furthermore, the synthesis module is also used for: after obtaining the first dubbing audio corresponding to the video to be dubbed, fusing the first dubbing audio with the target voice feature information to obtain the second dubbing audio; the target voice feature information is the voice feature information of the character to which the first dubbing audio belongs in the video to be dubbed;
[0145] The image information after lip shape correction is synthesized with the second dubbing audio to obtain a dubbed video.
[0146] Furthermore, the synthesis module is also used to:
[0147] Based on a cross-language TTS model, the first dubbing audio is fused with target speech feature information to obtain a second dubbing audio; the cross-language TTS model is learned based on the original text and original audio corresponding to the video to be dubbed.
[0148] Further, the first dubbing audio is obtained by recording a dubbing artist, and the target voice feature information includes timbre feature information of the character to which the first dubbing audio belongs; or
[0149] The first dubbing audio is generated by software, and the target voice feature information includes timbre feature information and emotion feature information of the character to which the first dubbing audio belongs.
[0150] Furthermore, the device also includes an adjustment module, which is used to:
[0151] determining a target duration corresponding to the first dubbing audio,
[0152] The duration of the first dubbing audio is adjusted so that the absolute value of the difference between the duration of the first dubbing audio and the target duration is less than or equal to a preset time length.
[0153] Furthermore, the synthesis module is further used to obtain the first dubbing audio corresponding to the video to be dubbed, and then determine the background sound audio based on the video to be dubbed;
[0154] Synthesize the background sound audio with the first dubbing audio to obtain a third dubbing audio;
[0155] The step of synthesizing the image information after the lip shape correction with the first dubbing audio to obtain a dubbed video includes:
[0156] The image information after lip shape correction is synthesized with the third dubbing audio to obtain a dubbed video.
[0157] The dubbing device provided in the embodiment of the present disclosure can execute the steps executed by the client or server in the dubbing method provided in the embodiment of the method of the present disclosure, and has the execution steps and beneficial effects, which will not be repeated here.
[0158] Figure 8 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 8, which shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing the embodiment of the present disclosure. The electronic device 1000 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0159] like Figure 8 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes to implement the dubbing method of the embodiment described in the present disclosure according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1008 to the random access memory (RAM) 1003. In the RAM 1003, various programs and information required for the operation of the electronic device 1000 are also stored. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0160] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange information. Although Figure 8 The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0161] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the dubbing method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0162] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include an information signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated information signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0163] In some embodiments, the client and the server may communicate using any known or future developed network protocol such as HTTP (HyperText TransferProtocol), and may be interconnected with any form or medium of digital information communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any known or future developed network.
[0164] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0165] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:
[0166] Obtaining a first dubbing audio corresponding to the video to be dubbed;
[0167] Based on the first dubbing audio, performing lip shape correction on image information corresponding to the first dubbing audio in the video to be dubbed;
[0168] The image information after lip shape correction is synthesized with the first dubbing audio to obtain a dubbed video.
[0169] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.
[0170] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0171] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0172] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.
[0173] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0174] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0175] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:
[0176] one or more processors;
[0177] A memory for storing one or more programs;
[0178] When the one or more programs are executed by the one or more processors, the one or more processors implement any dubbing method provided in the present disclosure.
[0179] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the program implements any dubbing method provided by the present disclosure.
[0180] The embodiments of the present disclosure also provide a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the dubbing method described above is implemented.
[0181] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0182] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A dubbing method, characterized in that: include: Obtaining a first dubbing audio corresponding to the video to be dubbed; Based on the first dubbing audio, performing lip shape correction on image information corresponding to the first dubbing audio in the video to be dubbed; synthesizing the image information after the lip shape correction with the first dubbing audio to obtain a dubbed video; The step of obtaining the first dubbing audio corresponding to the video to be dubbed includes: Acquire an original text corresponding to the video to be dubbed; the language of the original text is the same as the language of the video to be dubbed; Based on the original text, a final translation text is obtained; the absolute value of the difference between the number of phonemes corresponding to the final translation text and the target number of phonemes is less than or equal to a set threshold; the target number of phonemes is determined based on the duration of the audio information corresponding to the original text in the video to be dubbed; Acquire a first dubbing audio corresponding to the final translation text; The final translation text is obtained based on the original text, including: Based on the original text, an intermediate translation text is obtained; Determining whether the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to a set threshold; If yes, use the intermediate translation text as the final translation text; If not, in response to the modification instruction for the intermediate translation text, modify the intermediate translation text; The method also includes: displaying a translation proofreading interface, the translation proofreading interface including the original text, the intermediate translation text and proofreading target information; the proofreading target information is used to indicate whether the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to a set threshold.
2. The dubbing method according to claim 1, characterized in that: The step of performing lip shape correction on image information corresponding to the first dubbing audio in the video to be dubbed based on the first dubbing audio includes: Determining the phonemes corresponding to the first dubbing audio; Determining the mouth shape corresponding to each of the phonemes; The lip shapes corresponding to the phonemes are used to correct the lip shapes of the image information corresponding to the first dubbing audio in the video to be dubbed.
3. The dubbing method according to claim 1, characterized in that: The language of the first dubbing audio is different from the language of the video to be dubbed.
4. The dubbing method according to claim 1, characterized in that: After the intermediate translation text is modified in response to the modification instruction for the intermediate translation text, the method further includes: The proofreading target information is updated.
5. The dubbing method according to claim 1, characterized in that: The original text includes one or more sub-original texts; the intermediate translation text includes one or more sub-intermediate translation texts; the proofreading target information includes one or more sub-proofreading target information; One sub-original text, one sub-intermediate translation text and one sub-proofreading target information constitute a proofreading information group; in any one of the proofreading information groups, the sub-original text, the sub-intermediate translation text and the sub-proofreading target information have a corresponding relationship.
6. The dubbing method according to claim 5, characterized in that: In any of the proofreading information groups, if the absolute value of the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes is less than or equal to a set threshold, the sub-proofreading target information is in a target-achieved state; if the absolute value of the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes is greater than the set threshold, the sub-proofreading target information is in a target-unachieved state; The target number of phonemes is determined based on the sub-original text in the proofreading information group.
7. The method according to claim 6, characterized in that The sub-proofreading target information in the target achievement state includes a target achievement prompt; The sub-proofreading target information in the target-unachieved state includes the difference between the number of phonemes corresponding to the current sub-intermediate translation text and the target number of phonemes; or, the sub-proofreading target information in the target-unachieved state includes the difference between the target number of phonemes and the number of phonemes corresponding to the current sub-intermediate translation text.
8. The method according to claim 6, characterized in that The background color of the sub-proofreading target information in the target-achieved state is different from the background color of the sub-proofreading target information in the target-unachieved state; and / or, The border color of the sub-proofreading target information in the target-achieved state is different from the border color of the sub-proofreading target information in the target-unachieved state.
9. The dubbing method according to claim 1, characterized in that: After obtaining the first dubbing audio corresponding to the video to be dubbed, the method further includes: The first dubbing audio is merged with the target voice feature information to obtain a second dubbing audio; the target voice feature information is the voice feature information of the character to which the first dubbing audio belongs in the video to be dubbed; The step of synthesizing the image information after the lip shape correction with the first dubbing audio to obtain a dubbed video includes: The image information after lip shape correction is synthesized with the second dubbing audio to obtain a dubbed video.
10. The dubbing method according to claim 9, characterized in that: The step of fusing the first dubbing audio with the target voice feature information to obtain the second dubbing audio includes: Based on a cross-language TTS model, the first dubbing audio is fused with target speech feature information to obtain a second dubbing audio; the cross-language TTS model is learned based on the original text and original audio corresponding to the video to be dubbed.
11. The dubbing method according to claim 9, characterized in that: The first dubbing audio is obtained by recording a dubbing actor, and the target voice feature information includes timbre feature information of the character to which the first dubbing audio belongs; or, The first dubbing audio is generated by software, and the target voice feature information includes at least one of timbre feature information, emotion feature information, and speech flow feature information of the character to which the first dubbing audio belongs.
12. The dubbing method according to claim 1, characterized in that: Also includes: determining a target duration corresponding to the first dubbing audio, The duration of the first dubbing audio is adjusted so that the absolute value of the difference between the duration of the first dubbing audio and the target duration is less than or equal to a preset time length.
13. The dubbing method according to claim 1, characterized in that: After obtaining the first dubbing audio corresponding to the video to be dubbed, the method further includes: Determine the background audio based on the video to be dubbed; Synthesize the background sound audio with the first dubbing audio to obtain a third dubbing audio; The step of synthesizing the image information after the lip shape correction with the first dubbing audio to obtain a dubbed video includes: The image information after lip shape correction is synthesized with the third dubbing audio to obtain a dubbed video.
14. A dubbing device, characterized in that: include: An acquisition module, used for acquiring a first dubbing audio corresponding to the video to be dubbed; A correction module, configured to correct the lip shape of the image information corresponding to the first dubbing audio in the video to be dubbed based on the first dubbing audio; A synthesis module, used for synthesizing the image information after the lip shape correction with the first dubbing audio to obtain a dubbed video; The acquisition module is specifically used for: Acquire an original text corresponding to the video to be dubbed; the language of the original text is the same as the language of the video to be dubbed; Based on the original text, a final translation text is obtained; The absolute value of the difference between the number of phonemes corresponding to the final translation text and the target number of phonemes is less than or equal to a set threshold; The target number of phonemes is determined based on the duration of the audio information corresponding to the original text in the video to be dubbed; Acquire a first dubbing audio corresponding to the final translation text; The acquisition module is specifically used for: Based on the original text, an intermediate translation text is obtained; Determining whether the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to a set threshold; If yes, use the intermediate translation text as the final translation text; If not, in response to the modification instruction for the intermediate translation text, modify the intermediate translation text; The device also includes a display module, which is used to: display a translation proofreading interface, wherein the translation proofreading interface includes the original text, the intermediate translation text, and proofreading target information; the proofreading target information is used to indicate whether the absolute value of the difference between the number of phonemes corresponding to the intermediate translation text and the target number of phonemes is less than or equal to a set threshold.
15. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 13.
16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.
