Video processing method, video processing device, electronic equipment and medium

Abstract

The invention discloses a video processing method, a video processing device, electronic equipment and a medium, and belongs to the technical field of computers. The video processing method comprises the following steps: receiving a first input for a first interface, wherein the first interface comprises a picture of a first video; in response to the first input, acquiring dubbing timbre features and dubbing rhythm features; acquiring a dubbing audio based on the dubbing timbre feature and the dubbing rhythm feature; and generating a second video based on the dubbing audio and the first video.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a video processing method, a video processing device, an electronic device and a medium. Background Art

[0002] As the functions of electronic devices, such as mobile phones, continue to improve, using electronic devices to dub videos is becoming more and more common.

[0003] At present, when using an electronic device to dub a video, it is necessary to first divide the video according to the dialogue or conversation in the video to obtain multiple sub-videos, and obtain the dubbing text corresponding to each sub-video. Among them, each sub-video corresponds to a line of dialogue or conversation, and the dubbing text corresponding to each sub-video is the text of a line of dialogue or conversation corresponding to each sub-video. Then, the electronic device can use the audio recorded by the user according to the dubbing text corresponding to each sub-video and the corresponding sub-video to generate a new video.

[0004] Thus, when using an electronic device to dub a video, the user is required to continuously record the audio of the entire video, resulting in low efficiency of video dubbing and inflexible methods of video dubbing. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a video processing method, a video processing device, an electronic device and a medium, which can improve the efficiency and flexibility of video dubbing.

[0006] In a first aspect, an embodiment of the present application provides a video processing method, the method comprising:

[0007] Receiving a first input to a first interface, where the first interface includes a screen of a first video;

[0008] In response to the first input, acquiring a dubbing timbre feature and a dubbing rhythm feature;

[0009] Based on the dubbing timbre characteristics and dubbing rhythm characteristics, obtaining dubbing audio;

[0010] A second video is generated based on the dubbing audio and the first video.

[0011] In a second aspect, an embodiment of the present application provides a video processing device, the device comprising:

[0012] A receiving module, configured to receive a first input to a first interface, wherein the first interface includes a screen of a first video;

[0013] A processing module, for obtaining a dubbing timbre feature and a dubbing rhythm feature in response to a first input;

[0014] Based on the dubbing timbre characteristics and dubbing rhythm characteristics, obtaining dubbing audio;

[0015] A second video is generated based on the dubbing audio and the first video.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the video processing method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the video processing method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the video processing method as described in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the video processing method as described in the first aspect.

[0020] In an embodiment of the present application, a first input to a first interface is received, and the first interface includes a screen of a first video; in response to the first input, dubbing timbre characteristics and dubbing rhythm characteristics are obtained; based on the dubbing timbre characteristics and dubbing rhythm characteristics, dubbing audio is obtained; based on the dubbing audio and the first video, a second video is generated. In this way, the electronic device can obtain the corresponding dubbing timbre characteristics and dubbing rhythm characteristics according to the user's input, and then generate dubbing audio according to the dubbing timbre characteristics and dubbing rhythm characteristics, thereby improving the flexibility of the video dubbing method. In addition, there is no need for the user to continuously record the audio of the entire video, thereby improving the efficiency of video dubbing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0022] Figure 2A is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0023] Figure 2B is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0024] Figure 2C is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0025] Figure 2D is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0026] Figure 2E is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0027] Figure 2F is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0028] Figure 2G is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0029] Figure 2H is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0030] Fig.2I is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0031] Figure 2J is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0032] Figure 2K is a schematic diagram of a video processing interface provided in an embodiment of the present application;

[0033] Figure 3 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0034] Figure 4 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0035] Figure 5 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0036] Figure 6 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0037] Figure 7 It is a flowchart of a dubbing audio generation method provided in an embodiment of the present application;

[0038] Figure 8 It is a structural schematic diagram of a timbre acquisition module provided in an embodiment of the present application;

[0039] Fig. 9 is a structural diagram of a rhythm acquisition module provided in an embodiment of the present application;

[0040] Fig.10 is a schematic diagram of a phoneme sequence provided in an embodiment of the present application;

[0041] Fig.11 This is a flow chart of a video processing method provided in an embodiment of the present application.

[0042] Fig.12 is a structural schematic diagram of a video processing device provided in an embodiment of the present application;

[0043] Fig.13 is a structural schematic diagram of a video processing device provided in an embodiment of the present application;

[0044] Fig.14 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0045] Fig.15 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0047] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0048] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0049] The video processing method, video processing device, electronic device and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0050] The video processing method, video processing device, electronic device and medium provided in the embodiments of the present application can be applied to video dubbing scenarios, such as using a mobile phone to dub a video.

[0051] The execution subject of the video processing method provided in the embodiment of the present application may be a video processing device. Exemplarily, the video processing device may be an electronic device, or a functional component or functional entity in the electronic device. The video processing method provided in the embodiment of the present application will be described exemplarily below by taking the execution subject as an electronic device as an example.

[0052] Figure 1 is a flow chart of a video processing method provided in an embodiment of the present application, such as Figure 1 As shown, the video processing method provided in the embodiment of the present application may include the following steps 101 to 104.

[0053] Step 101: An electronic device receives a first input to a first interface.

[0054] In some embodiments of the present application, the first interface includes a screen of a first video, and the first video is a video to be dubbed.

[0055] In some embodiments of the present application, the above-mentioned first input may include any of the following: user's click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0056] In some embodiments of the present application, the above-mentioned gesture input may include but is not limited to at least one of the following: click gesture, sliding gesture, drag gesture, pressure recognition gesture, long press gesture, area change gesture, double press gesture, double click gesture, specific gesture input or other possible gesture inputs. The specific gesture input form can be determined according to actual needs and is not limited to this in some embodiments.

[0057] In some embodiments of the present application, the above-mentioned click input can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long-press input or a short-press input, which is not limited in some embodiments.

[0058] In some embodiments of the present application, the above-mentioned sliding input can be a sliding input in any direction, such as sliding up, sliding down, sliding to the left or sliding to the right, etc., which is not limited in some embodiments.

[0059] Step 102: The electronic device obtains dubbing timbre characteristics and dubbing rhythm characteristics in response to the first input.

[0060] In some embodiments of the present application, the first input includes a third sub-input and a fourth sub-input, and the step 102 may be implemented by the following steps 102a and 102b:

[0061] Step 102a: The electronic device displays at least one dubbing mode on the first interface in response to the third sub-input.

[0062] In some embodiments of the present application, each dubbing mode corresponds to a method for acquiring dubbing timbre characteristics and dubbing rhythm characteristics.

[0063] In some embodiments of the present application, the third sub-input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0064] For example, when the electronic device is a mobile phone, Figure 2A As shown, the first interface 20 can be displayed on the screen 20a of the mobile phone, and the first interface 20 includes a picture 21 of the first video and a dubbing window prompt control 22. When the mobile phone receives a click input from the user on the dubbing window prompt control 22, the mobile phone can respond to the click input of the user on the dubbing window prompt control 22, in combination with Figure 2A ,like Figure 2B As shown, a dubbing window 23 is displayed on the first interface 20, and the dubbing window 23 includes a dubbing mode selection control 231. When the mobile phone receives a click input of the user on the dubbing mode selection control 231, the mobile phone can respond to the click input of the user on the dubbing mode selection control 231, in combination with Figure 2B ,like Figure 2C As shown, a dubbing mode selection window 24 is displayed on the first interface 21, and the displayed dubbing mode selection window 24 includes dubbing mode 1, dubbing mode 2 and dubbing mode 3. The user's click input on the dubbing mode selection control 231 is the above-mentioned third sub-input.

[0065] It should be noted that the shapes of the dubbing window prompt control, dubbing window, and dubbing mode selection window can be any possible shapes such as circle, rectangle, triangle, diamond, ring or polygon. Figure 2A , Figure 2B , Figure 2C The shapes of the dubbing window prompt control, the dubbing window, and the dubbing mode selection window are only given as examples, and can be determined according to actual usage requirements, and are not limited in the embodiments of the present invention.

[0066] Step 102b: the electronic device obtains dubbing timbre features and dubbing rhythm features in response to the fourth sub-input of the target dubbing mode according to the target dubbing timbre features and dubbing rhythm features corresponding to the target dubbing mode.

[0067] In some embodiments of the present application, the target dubbing mode is a dubbing mode among the at least one dubbing mode mentioned above.

[0068] In some embodiments of the present application, the fourth sub-input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0069] Combination Figure 2C As shown, dubbing mode 1 can correspond to obtaining user timbre features and user rhythm features, dubbing mode 2 can correspond to obtaining video timbre features and user rhythm features, and dubbing mode 3 can correspond to obtaining user timbre features and video rhythm features. When the mobile phone receives the fourth sub-input of dubbing mode 1, the mobile phone can extract the dubbing timbre features and dubbing rhythm features from the user audio in response to the fourth sub-input of dubbing mode 1. When the mobile phone receives the fourth sub-input of dubbing mode 2, the mobile phone can extract the dubbing timbre features from the first audio and the dubbing rhythm features from the user audio in response to the fourth sub-input of dubbing mode 2. When the mobile phone receives the fourth sub-input of dubbing mode 3, the mobile phone can extract the dubbing timbre features from the user audio in response to the fourth sub-input of dubbing mode 3 and the dubbing rhythm features from the first audio.

[0070] Step 103: The electronic device obtains the dubbing audio based on the dubbing timbre characteristics and the dubbing rhythm characteristics.

[0071] In some embodiments of the present application, the dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a video rhythm feature; Figure 1 ,like Figure 3 As shown, the above step 103 can be implemented by the following step 103a1 and the following step 103a2:

[0072] Step 103a1: extract user timbre features from the user audio, and extract video rhythm features from the first audio.

[0073] In some embodiments of the present application, the first audio is the audio in the first video, the user timbre features extracted from the user audio are used to characterize the timbre of the user's voice, and the video rhythm features extracted from the first audio are used to characterize the rhythm of the first video sound.

[0074] In some embodiments of the present application, the electronic device may perform audio feature extraction on the user audio to obtain the audio features of the user audio, and then perform timbre feature extraction on the audio features to obtain the timbre features of the user.

[0075] In some embodiments of the present application, the electronic device may perform prosodic feature extraction and vector quantization on the first audio to obtain the above-mentioned video prosodic feature.

[0076] Step 103a2: The electronic device generates dubbing audio based on the user's timbre characteristics and the video's rhythmic characteristics.

[0077] It should be noted that the implementation process of the above step 103a2 can refer to the following step A1, and to avoid repetition, this embodiment will not be repeated here.

[0078] In this way, the electronic device extracts the user's timbre features from the user audio and the video's rhythm features from the first audio; generates the dubbing audio based on the user's timbre features and the video's rhythm features, so that the electronic device can generate the dubbing audio with the user's timbre and the video's rhythm, thereby improving the flexibility of the video dubbing method. In addition, the user does not need to continuously record the audio of the complete video, thereby improving the efficiency of the video dubbing.

[0079] In some embodiments of the present application, the dubbing timbre feature is a video timbre feature, and the dubbing rhythm feature is a user rhythm feature; Figure 1 ,like Figure 4 As shown, the above step 103 can be implemented by the following step 103b1 and the following step 103b2:

[0080] Step 103b1: extract user timbre features and user rhythm features from user audio.

[0081] In some embodiments of the present application, prosodic features are extracted from user audio to characterize the prosody of the user's voice.

[0082] It should be noted that the process of the electronic device extracting user timbre features and user rhythm features from user audio can refer to the process of extracting user timbre features from user audio and extracting video rhythm features from the first audio. To avoid repetition, this embodiment will not be repeated here.

[0083] Step 103b2: The electronic device generates dubbing audio based on the user's timbre characteristics and the user's prosody characteristics.

[0084] It should be noted that the implementation process of the above step 103b2 can refer to the following step A1, and to avoid repetition, this embodiment will not be repeated here.

[0085] In this way, the electronic device extracts the user's timbre features and the user's rhythm features from the user's audio; generates the dubbing audio based on the user's timbre features and the user's rhythm features, so that the electronic device can generate the dubbing audio with the user's timbre and the user's rhythm, thereby improving the flexibility of the video dubbing method. In addition, the user does not need to continuously record the audio of the complete video, thereby improving the efficiency of the video dubbing.

[0086] In some embodiments of the present application, the dubbing timbre feature is a video timbre feature, and the dubbing rhythm feature is a user rhythm feature; Figure 1 ,like Figure 5 As shown, the above step 103 can be implemented by the following step 103c1 and the following step 103c2:

[0087] Step 103c1: The electronic device extracts video timbre features from the first audio, and extracts user prosody features from the user audio.

[0088] In some embodiments of the present application, timbre features are extracted from the first audio to characterize the timbre of the video sound.

[0089] It should be noted that the process of the electronic device extracting video timbre features from the first audio and extracting user rhythm features from the user audio can refer to the process of extracting user timbre features from the user audio and extracting video rhythm features from the first audio. To avoid repetition, this embodiment will not be repeated here.

[0090] Step 103c2: The electronic device generates dubbing audio based on the video timbre characteristics and the user prosody characteristics.

[0091] It should be noted that the implementation process of the above step 103c2 can refer to the following step A1, and to avoid repetition, this embodiment will not be repeated here.

[0092] In this way, the electronic device extracts the video timbre feature from the first audio and the user rhythm feature from the user audio; generates the dubbing audio based on the video timbre feature and the user rhythm feature, so that the electronic device can generate the dubbing audio with the same timbre as the video and the same rhythm as the user rhythm, thereby improving the flexibility of the video dubbing method. In addition, the user does not need to continuously record the audio of the complete video, thereby improving the efficiency of the video dubbing.

[0093] In some embodiments of the present application, the first interface further includes a text acquisition control; Figure 1 ,like Figure 6 As shown, before the above step 103, the video processing method provided in the embodiment of the present application may further include the following steps 105a and 105b, and the above step 103 may be implemented by the following step A1:

[0094] Step 105a: The electronic device receives a third input to the dubbing text acquisition control.

[0095] In some embodiments of the present application, the dubbing text acquisition control is used to instruct acquisition of the dubbing text of the first video.

[0096] In some embodiments of the present application, the shape of the above-mentioned dubbing text acquisition control can be any possible shape such as a circle, rectangle, triangle, diamond, ring or polygon, which can be determined according to actual usage requirements and is not limited by the embodiments of the present invention.

[0097] In some embodiments of the present application, the third input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0098] Step 105b: The electronic device displays the dubbing text in response to the third input.

[0099] In some embodiments of the present application, the electronic device may display a text prompt box in response to the fifth input, where the text prompt box includes the dubbing text.

[0100] In some embodiments of the present application, the dubbing text is the text corresponding to the first video or the text input by the user.

[0101] In some embodiments of the present application, the dubbing text is the text corresponding to the first video; the step 105b may be implemented by the following step B1:

[0102] Step B1: In response to a third input, the electronic device displays a dubbing text when a first audio is detected in a first video.

[0103] In some embodiments of the present application, the dubbing text is the text corresponding to the first audio in the first video.

[0104] In some embodiments of the present application, the electronic device may detect the first audio in the first video in response to the third input. When the first audio is detected, the text corresponding to the first audio is obtained, and the text corresponding to the first audio is displayed in the text prompt box as the dubbing text.

[0105] For example Figure 2B The dubbing window 23 may further include a dubbing text acquisition control 232. When the mobile phone receives a click input from the user to the dubbing text acquisition control 232, the mobile phone may respond to the click input from the user to the dubbing text acquisition control 232 and, in the case where the first audio is detected in the first video, combine the first audio with the first audio. Figure 2B ,like Figure 2D As shown, a text prompt box 25 is displayed on the first interface 20. The text prompt box 25 displays a dubbing text 251 and a fourth prompt information 252. The fourth prompt information 252 is used to prompt that the dubbing text 251 is a text obtained from the first video, that is, the dubbing text 251 is the text corresponding to the first audio. The user's click input on the dubbing text acquisition control 232 is the above-mentioned third input.

[0106] In this way, the electronic device displays the dubbing text in response to the third input when the first audio is detected in the first video, so that the electronic device can generate dubbing audio with corresponding timbre and rhythm according to the text corresponding to the first audio detected in the first video, thereby improving the flexibility of video dubbing.

[0107] In some embodiments of the present application, the dubbing text is a text input by a user, and the third input includes a first sub-input and a second sub-input; the step 105b can be implemented by the following steps C1 and C2:

[0108] Step C1: The electronic device displays a text input box on the first interface in response to a first sub-input.

[0109] In some embodiments of the present application, the above-mentioned first sub-input may include any of the following: user's click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0110] In some embodiments of the present application, the shape of the above-mentioned text input box can be any possible shape such as a circle, rectangle, triangle, diamond, ring or polygon, which can be determined according to actual usage requirements and is not limited by the embodiments of the present invention.

[0111] Step C2: The electronic device displays the dubbing text in the text input box in response to the second sub-input to the text input box.

[0112] In some embodiments of the present application, the second sub-input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0113] In some embodiments of the present application, the dubbing text is a text corresponding to the second sub-input.

[0114] Combination Figure 2BAs shown, when the mobile phone receives a click input from the user on the dubbing text acquisition control 232, the mobile phone can respond to the click input from the user on the dubbing text acquisition control 232, and in the case where the first audio is not detected in the first video, as shown in FIG. Figure 2E As shown, a text prompt box 25 is displayed on the first interface 20, and the text prompt box 25 displays and the fifth prompt information 253, and the fifth prompt information 253 is used to prompt the user to input the dubbing text. When the mobile phone receives the second sub-input of the user to the text prompt box 25 according to the fifth prompt information 253, the text corresponding to the fourth sub-input is displayed as the dubbing text 251 in the text prompt box 25, and the user's click input on the dubbing text acquisition control 232 is the above-mentioned first sub-input. For example, if the text input by the user is "Hello, nice to meet you", "Hello, nice to meet you" is displayed as the dubbing text 251 in the text prompt box 25.

[0115] In this way, the electronic device displays a text input box on the first interface in response to the first sub-input, and displays the dubbing text in the text input box in response to the second sub-input to the text input box, so that the electronic device can generate dubbing audio with corresponding timbre and rhythm according to the text input by the user, thereby improving the flexibility of video dubbing.

[0116] Step A1: The electronic device generates dubbing audio based on timbre features, rhythm features and dubbing text.

[0117] In this way, the electronic device receives a third input to the dubbing text acquisition control; displays the dubbing text in response to the third input; and generates dubbing audio based on the timbre characteristics, rhythmic characteristics and the dubbing text, so that the electronic device can generate dubbing audio with corresponding timbre and rhythm based on the dubbing text obtained by the user operating the dubbing text acquisition control, thereby improving the flexibility of video dubbing.

[0118] In some embodiments of the present application, the above step A1 can be implemented by the following steps D1 to D4:

[0119] Step D1, the electronic device obtains a first phoneme sequence corresponding to the dubbing text;

[0120] In some embodiments of the present application, the above step D1 can be implemented by the following steps a1 to a3:

[0121] Step a1: The electronic device obtains at least one first phoneme and at least one first phoneme duration of the dubbing text.

[0122] In some embodiments of the present application, each first phoneme duration is the duration of a first phoneme.

[0123] In some embodiments of the present application, the electronic device may obtain the pronunciation of each character in the dubbed text, and then perform phoneme segmentation on the pronunciation of each character in the dubbed text to obtain at least one first phoneme of the above-mentioned dubbed text.

[0124] For example, the dubbed text may be "你好" (Hello in Chinese), and the corresponding pronunciations are "ni" and "hao" respectively. Then, performing phoneme segmentation on the pronunciation of each character in "你好", the at least one first phoneme obtained may be "sil n i3 h ao3 sil", where the "sil" at the beginning and end represents the silent phoneme, "i3" represents the third tone of "你", and "ao3" represents the third tone of "好".

[0125] In some embodiments of the present application, the electronic device may input the above at least one phoneme into a duration prediction model to obtain the at least one first phoneme duration. The above duration prediction model may be a model trained using a training set, and the training set includes a plurality of phoneme samples and a plurality of labels, and each label represents the duration corresponding to a phoneme sample.

[0126] Step a2: The electronic device determines the first quantity of each first phoneme in the at least one first phoneme based on the at least one first phoneme duration.

[0127] In some embodiments of the present application, the electronic device may calculate the ratio of the at least one first phoneme duration and the second duration to obtain the first quantity of each first phoneme in the at least one first phoneme, and the second duration is the duration of a single-frame audio.

[0128] For example, the first phoneme time corresponding to the "n" phoneme is 0.11 s, and the first duration is 10 ms. Then, the first quantity corresponding to the "n" phoneme is 0.11 s / 10 ms, that is, the first quantity corresponding to the "j" phoneme is 10.

[0129] Step a3: The electronic device sorts the at least one first phoneme based on the first quantity to obtain a first phoneme sequence.

[0130] In some embodiments of the present application, the first quantity corresponding to each first phoneme is the quantity of each first phoneme in the first phoneme sequence.

[0131] For example, the first quantities corresponding to “siln i3 h ao3 sil” are [22, 6, 16, 11, 22, 25] respectively, and the first phoneme sequence is “sil sil...sil nn...ni3 i3...i3 hh...h ao3 ao3...ao3 silsil...sil”. Among them, the number of the first “sil” in the first phoneme sequence is 22, the number of “n” is 6, the number of “i3” is 16, the number of “h” is 11, the number of “ao3” is 22, and the number of “sil” at the end of the first phoneme sequence is 25.

[0132] In this way, the electronic device uses at least one first phoneme and at least one first phoneme duration of the dubbing text; based on the at least one first phoneme duration, determines the first quantity of each first phoneme in at least one first phoneme; based on the first quantity, sorts at least one first phoneme to obtain a first phoneme sequence, and can accurately and quickly obtain a first timbre sequence.

[0133] Step D2: The electronic device obtains a second phoneme sequence corresponding to the target audio.

[0134] In some embodiments of the present application, the target audio is the audio in the first video or the user audio.

[0135] In some embodiments of the present application, the above step D2 can be implemented by the following steps b1 to b4:

[0136] Step b1: The electronic device obtains a target text corresponding to the target audio.

[0137] In some embodiments of the present application, the electronic device may first perform noise reduction processing on the target audio to obtain the noise-reduced target audio, and then perform recognition processing on the noise-reduced target audio to obtain the target text corresponding to the noise-reduced target audio.

[0138] Step b2: The electronic device obtains at least one second phoneme and at least one second phoneme duration of the target text.

[0139] In some embodiments of the present application, the duration of each second phoneme is the duration of one second phoneme.

[0140] It should be noted that the specific process of the electronic device acquiring at least one second phoneme of the target text can refer to the specific process of the electronic device acquiring at least one first phoneme of the dubbing text. To avoid repetition, this embodiment will not be repeated here.

[0141] In some embodiments of the present application, the electronic device may obtain a third duration of the target audio, and then align the target audio with the at least one second phoneme according to the third duration to obtain a second phoneme duration of each second phoneme.

[0142] Step b3, the electronic device determines the second quantity of each first phoneme in the at least one second phoneme based on the duration of the at least one second phoneme;

[0143] Step b4: The electronic device sorts at least one second phoneme based on the second quantity to obtain a second phoneme sequence.

[0144] It should be noted that the specific implementation process of the above step b3 and the above step b4 can refer to the above step a2 and the above step a3. To avoid repetition, this embodiment will not be repeated here.

[0145] In this way, the electronic device obtains the target text corresponding to the target audio; obtains at least one second phoneme and at least one second phoneme duration of the target text; determines the second quantity of each first phoneme in at least one second phoneme based on the duration of at least one second phoneme; and sorts at least one second phoneme based on the second quantity to obtain a second phoneme sequence, thereby being able to accurately and quickly obtain the first timbre sequence.

[0146] Step D3: The electronic device generates audio features of the dubbing audio based on the first phoneme sequence, the second phoneme sequence, the dubbing timbre features, and the dubbing rhythm features.

[0147] In some embodiments of the present application, the above step D3 can be implemented by the following steps c1 to c5:

[0148] Step c1: The electronic device obtains a first splicing feature based on the second phoneme sequence and the dubbing rhythm feature.

[0149] In some embodiments of the present application, the electronic device may respectively use a first embedding layer and a second embedding layer to convert the second phoneme sequence and the dubbing rhythm feature into a first feature vector and a second feature vector, and then splice the first feature vector and the second feature vector to obtain the first splicing feature.

[0150] Step c2: The electronic device inputs the first concatenation feature into a prosody prediction model to obtain a predicted prosody feature of the ith phoneme in the first phoneme sequence.

[0151] In some embodiments of the present application, the predicted prosodic feature may be the prosodic feature of the i-th phoneme predicted by the prosodic prediction model.

[0152] In some embodiments of the present application, the first phoneme sequence includes N phonemes, N is a positive integer, and i is an integer from 1 to N.

[0153] Step c3: The electronic device obtains a second concatenation feature based on the predicted prosodic feature and the i-th phoneme.

[0154] In some embodiments of the present application, the electronic device may respectively use a first embedding layer and a second embedding layer to convert the predicted prosodic feature and the i-th phoneme into a third feature vector and a fourth feature vector, and then concatenate the third feature vector and the fourth feature vector to obtain the second concatenated feature.

[0155] Step c4: the electronic device concatenates the second concatenation feature with the first concatenation feature to obtain the first concatenation feature again, until i=N, and obtains at least one predicted prosodic feature, each predicted prosodic feature corresponding to a phoneme in the first phoneme sequence;

[0156] Step c5: The electronic device obtains a third splicing feature based on at least one predicted prosody feature and the dubbing timbre feature, and decodes the third splicing feature to generate an audio feature of the dubbing text.

[0157] In some embodiments of the present application, the electronic device may embed each predicted prosodic feature and dubbing timbre feature in a first embedding layer and a second embedding layer respectively, convert each predicted prosodic feature and dubbing timbre feature into a fifth feature vector and a sixth feature vector, and then splice the fifth feature vector and the sixth feature vector to obtain a third spliced ​​feature corresponding to each predicted prosodic feature.

[0158] In some embodiments of the present application, the electronic device may decode the third splicing feature through a decoder to obtain the audio feature.

[0159] In this way, the electronic device obtains a first splicing feature based on the second phoneme sequence and the dubbing rhythmic feature; inputs the first splicing feature into the rhythmic prediction model to obtain the predicted rhythmic feature of the i-th phoneme in the first phoneme sequence; obtains a second splicing feature based on the predicted rhythmic feature and the i-th phoneme, and the electronic device splices the second splicing feature with the first splicing feature to obtain the first splicing feature again until i=N, and obtains at least one predicted rhythmic feature, each predicted rhythmic feature corresponding to a phoneme in the first phoneme sequence; obtains a third splicing feature based on at least one predicted rhythmic feature and the dubbing timbre feature, and decodes the third splicing feature to generate the audio feature of the dubbing text, so as to accurately and quickly obtain the audio feature of the dubbing text.

[0160] Step D4: The electronic device encodes the audio features to obtain dubbing audio.

[0161] In some embodiments of the present application, the electronic device may use a vocoder to encode the above-mentioned audio features to obtain the above-mentioned dubbing audio.

[0162] In this way, the electronic device obtains a first phoneme sequence corresponding to the dubbing text; obtains a second phoneme sequence corresponding to the target audio; generates audio features of the dubbing audio based on the first phoneme sequence, the second phoneme sequence, the dubbing timbre features and the dubbing rhythm features; encodes the audio features to obtain the dubbing audio, and can obtain the dubbing audio accurately and quickly, thereby improving the efficiency of video dubbing.

[0163] Step 104: The electronic device generates a second video based on the dubbing audio and the first video.

[0164] In some embodiments of the present application, the electronic device may synchronize the dubbing audio with the first video to obtain the second video.

[0165] In some embodiments of the present application, the above step 104 may be implemented by the following step 104a:

[0166] Step 104a: The electronic device generates a second video based on the dubbing audio and the first video in response to a fourth input to the video generation control.

[0167] In some embodiments of the present application, the fourth input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0168] In some embodiments of the present application, the video generation control is used to instruct the generation of the second video.

[0169] In some embodiments of the present application, the shape of the above-mentioned video generation control can be any possible shape such as a circle, a rectangle, a triangle, a diamond, a ring or a polygon, which can be determined according to actual usage requirements and is not limited by the embodiments of the present invention.

[0170] For example Figure 2B As shown, the dubbing window 23 may further include a video generation control 233. When the mobile phone receives a click input from the user to the video generation control 233, the mobile phone may generate a second video based on the dubbing audio and the first video in response to the click input from the user to the video generation control 233, and the click input from the user to the video generation control 233 is the fourth input mentioned above.

[0171] In the video processing method provided in the embodiment of the present application, by receiving a first input to a first interface, the first interface includes a screen of a first video; in response to the first input, obtaining dubbing timbre characteristics and dubbing rhythm characteristics; obtaining dubbing audio based on the dubbing timbre characteristics and dubbing rhythm characteristics; generating a second video based on the dubbing audio and the first video. In this way, the electronic device can obtain the corresponding dubbing timbre characteristics and dubbing rhythm characteristics according to the user's input, and then generate dubbing audio according to the dubbing timbre characteristics and dubbing rhythm characteristics, thereby improving the flexibility of the video dubbing method. In addition, there is no need for the user to continuously record the audio of the entire video, thereby improving the efficiency of video dubbing.

[0172] In some embodiments of the present application, the first interface further includes an audio acquisition control; before the above step 101, the video processing method provided in the embodiment of the present application may further include the following step 106a and the following step 106b:

[0173] Step 106a: The electronic device receives a second input to the audio acquisition control.

[0174] In some embodiments of the present application, the second input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0175] In some embodiments of the present application, the audio acquisition control may be a video audio acquisition control or a user audio acquisition control.

[0176] In some embodiments of the present application, the above-mentioned video and audio acquisition control is used to indicate the acquisition of video and audio, and the above-mentioned user audio acquisition control is used to indicate the acquisition of user audio.

[0177] In some embodiments of the present application, the shape of the audio acquisition control may be any possible shape such as a circle, rectangle, triangle, diamond, ring or polygon, which may be determined according to actual usage requirements and is not limited in the embodiments of the present invention.

[0178] Step 106b: The electronic device displays a first prompt message in response to the second input.

[0179] In some embodiments of the present application, the second prompt information is used to prompt that the target audio is acquired, and the target audio is the audio in the first video or the user audio.

[0180] In some embodiments of the present application, when the audio acquisition control is a video audio acquisition control, the electronic device can extract audio from the first video in response to the second input and display a first prompt message indicating that the video audio has been acquired.

[0181] For example Figure 2B As shown, the dubbing window 23 may also include a video and audio acquisition control 234. When the mobile phone receives a click input from the user to the video and audio acquisition control 234, the mobile phone may respond to the click input from the user to the video and audio acquisition control 234, collect video and audio, and combine the collected video and audio with the video and audio. Figure 2B ,like Figure 2F As shown, the first prompt information 261 of extracting audio from the first video is displayed on the first interface 20, and the user's click input on the video and audio acquisition control 234 is the above-mentioned second input.

[0182] It should be noted that if the mobile phone does not capture video and audio, Figure 2B ,like Figure 2G As shown, the sixth prompt information 27 is displayed on the first interface 20, and the sixth prompt information 27 is used to prompt that the video and audio are not collected.

[0183] In some embodiments of the present application, when the above-mentioned audio acquisition control is a user audio acquisition control, the electronic device can collect user audio in response to the above-mentioned second input, and display a first prompt information that the user audio is acquired.

[0184] For example Figure 2F As shown, when the sixth prompt information 27 is displayed on the first interface 20, the user can click and input the user audio acquisition control 235 in the dubbing window 23. The mobile phone can respond to the user's click input on the user audio acquisition control 235, combined with Figure 2F ,like Figure 2H As shown, a guide text prompt box 28 is displayed, and the guide text prompt box 28 includes a guide text 281 and an audio recording control 282. After the user clicks and inputs the audio recording control 282, the mobile phone can collect the audio recorded by the user according to the guide text 281, and after collecting the audio recorded by the user according to the guide text 281, the mobile phone can combine the audio recorded by the user according to the guide text 281 to record the audio. Figure 2G ,like Fig.2I As shown, the first prompt information 262 that the user audio is extracted is displayed on the first interface 20.

[0185] In this way, the electronic device receives a second input to the audio acquisition control and displays the first prompt information in response to the second input, so that the electronic device can display the prompt information of the target audio obtained by the electronic device according to the user's operation of the audio acquisition control, thereby improving the flexibility of video dubbing.

[0186] In some embodiments of the present application, after "obtaining at least one first phoneme duration of the dubbing text" in the above step a1, the video processing method provided in the embodiment of the present application may further include the following steps 108a to 108e:

[0187] Step 108a: The electronic device calculates the sum of the duration of at least one first phoneme to obtain a first duration of the dubbing audio.

[0188] For example, the first phoneme durations corresponding to at least one "siln i3 h ao3 sil" are 0.22s, 0.06s, 0.16s, 0.11s, 0.22s and 0.25s respectively, then the first duration of the dubbing audio is 0.22s+0.06s+0.16s, 0.11s+0.22s+0.25s, that is, the first duration of the dubbing audio is 1.04s.

[0189] Step 108b: The electronic device calculates the ratio of the first duration to the video duration of the first video.

[0190] In some embodiments of the present application, the above ratio is the interpolation coefficient of the above audio feature, and the electronic device can calculate the above interpolation coefficient using the following formula:

[0191] α=L o / L t ; (1)

[0192] Among them, α represents the above interpolation coefficient, L o represents the first duration mentioned above, L t Indicates the duration of the above video.

[0193] Step 108c: If the ratio is within the first value range, the electronic device performs interpolation processing on the audio feature based on the ratio to obtain a processed audio feature.

[0194] In some embodiments of the present application, the electronic device may use the following formula to perform interpolation processing on the audio feature:

[0195] feat n =acoustic_feat [n*α] ; (2)

[0196] Among them, feat represents the processed audio features, acoustic_feat represents the above audio features, n is an integer from 0 to M, M is the ratio of the above video duration to the first duration, and [n*α] represents rounding n*α to the nearest integer.

[0197] It should be noted that after obtaining the processed audio features, the electronic device can use a voice coder to decode the processed audio features to obtain the dubbing audio again, and then synchronize the obtained dubbing audio with the above-mentioned first video to obtain the second video again.

[0198] In this way, the electronic device obtains the first duration of the dubbing audio by calculating the sum of at least one first phoneme duration; calculates the ratio of the first duration to the video duration of the first video, and when the ratio is within a first numerical range, interpolates the audio features based on the ratio to obtain processed audio features, and can flexibly adjust the audio features according to the ratio of the first duration to the video duration of the first video, and then decode the adjusted audio features, so as to adjust the dubbing audio, thereby improving the flexibility of video dubbing.

[0199] Step 108d: If the ratio is outside the first numerical range and is smaller than the minimum value of the first numerical range, the electronic device displays a second prompt message.

[0200] In some embodiments of the present application, the second prompt information is used to prompt the expansion of the dubbing text.

[0201] In some embodiments of the present application, if the speech speed of the dubbing audio is too fast or too slow, the sound and the picture will be out of sync during dubbing. If the ratio is outside the first numerical range and the ratio is less than the minimum value of the first numerical range, it means that the speech speed of the dubbing audio is too fast, and the electronic device can display a third prompt message to prompt the user to expand the dubbing text to generate a dubbing audio close to the first duration.

[0202] For example, the first numerical range may be 0.85 to 1.1. When the mobile phone determines that the ratio is 0.6, which is outside the first numerical range of 0.85 to 1.1, and the ratio is less than the minimum value of 0.85 in the first numerical range, Figure 2J As shown, a second prompt message 291 for prompting to expand the dubbing text may be displayed on the first interface 20 .

[0203] Step 108e: If the ratio is outside the first numerical range and is greater than the maximum value of the first numerical range, the electronic device displays a third prompt message.

[0204] In some embodiments of the present application, the third prompt information is used to prompt shortening of the dubbing text.

[0205] In some embodiments of the present application, if the ratio is outside the first numerical range and is less than the minimum value of the first numerical range, it means that the speaking speed of the dubbing audio is too fast. The electronic device can display a fourth prompt message to prompt the user to shorten the dubbing text to generate a dubbing audio close to the first duration.

[0206] For example, the first numerical range may be 0.85 to 1.1. When the mobile phone determines that the ratio is 1.5, which is outside the first numerical range 0.85 to 1.1, and the ratio is greater than the maximum value 1.1 of the first numerical range, Figure 2KAs shown, a third prompt message 292 for prompting to shorten the dubbing text may be displayed on the first interface 20 .

[0207] In this way, the electronic device displays the second prompt information when the ratio is outside the first numerical range and is less than the minimum value of the first numerical range, and displays the third prompt information when the ratio is outside the first numerical range and is greater than the maximum value of the first numerical range. This enables the electronic device to adjust the dubbing audio according to the dubbing text re-entered by the user by viewing the second prompt information and the third prompt information, thereby improving the flexibility of the video dubbing method.

[0208] Next, combine Figure 7 , which explains how electronic devices generate dubbing audio. Figure 7 is a flow chart of the dubbing audio generation method provided in the embodiment of the present application. Figure 7 As shown, the method includes the following steps 701 to 707.

[0209] Step 701: Obtain a target text corresponding to the target audio and at least one second phoneme and at least one second phoneme duration corresponding to the target text.

[0210] For example, the electronic device can first reduce the noise of the target audio. For audio generation tasks, too much noise in the source audio will affect the final effect. Therefore, the electronic device can use the open source project DeepFilterNet3 to reduce the noise of the target audio and obtain a clean target audio.

[0211] Exemplarily, there are many speech recognition models available for electronic devices. The present invention uses whisper to recognize the target audio after noise reduction to obtain the target text corresponding to the target audio.

[0212] Exemplarily, after obtaining the target text, the electronic device can process the text: convert the numbers, symbols, etc. of the target text into corresponding Chinese / English; convert the polyphonetic characters into correct pronunciations, and finally convert them into at least one corresponding second phoneme. For example, when the target text is "Are you feeling OK today?", the at least one second phoneme corresponding to the target text is "siljin1 t ian1 x in1 q ing2 OW0 K EY1 m a5sil", where the first and last "sil" represent silent phonemes.

[0213] Exemplarily, the electronic device can use the MFA forced alignment tool to perform phoneme alignment on the target audio and at least one second phoneme according to the duration of the target audio, obtain the second phoneme duration corresponding to each second phoneme, and then align the target audio with the at least one second phoneme according to the duration of the target audio.

[0214] Step 702: extract timbre features of the target audio.

[0215] Exemplarily, the electronic device can perform Mel feature extraction on the target audio to obtain Mel features. The number of Mel features is consistent with the total number of frames of the target audio. For example, if the target audio duration is 1.01s, taking 10ms / frame as an example, the total number of frames of the target audio is 104, and the number of Mel features is also 104. Exemplarily, the electronic device can input the Mel features into the timbre extraction module, and extract a timbre feature vector timbre_feat with a dimension of (8, 246) from the Mel features; the timbre extraction module is mainly a convolution structure, and the structure of the timbre extraction module can be as follows: Figure 8 As shown, the timbre extraction module 80 includes a first linear and normalization module linear&layernorm 801, N cascaded convolution modules ConvBlock 802 and a second linear&layernorm 803. Where N=246.

[0216] Step 703: extract prosodic features from the target audio.

[0217] For example, the electronic device may use a rhythm extraction module to extract rhythm features from the target audio. The main structure of the rhythm extraction module is similar to that of the timbre extraction module. Fig. 9 As shown, the rhythm extraction module 90 includes a first convolution model ConvModule 91, a maximum pooling module Max Pool 92, a second convolution model ConvModule 93 and a vector quantization module Vector Quantizer 94. Among them, the structures of the first convolution model ConvModule 91 and the second convolution model Conv Module 93 are the same, and are of the same structure type as the timbre extraction module. The first convolution model ConvModule91 may include a third linear and normalization module linear&layernorm911, M cascaded convolution modules ConvBlock 912 and a fourth linear&layernorm913, where M=4. The input of the rhythm extraction module is consistent with the timbre extraction module, both of which are Mel features. The number of rhythm features of the output target audio is the same as the number of frames of the target audio. For example, if the target audio duration is 1.04s, taking 10ms / frame as an example, the total number of frames of the target audio is 104, and the number of rhythm features of the target audio is also 104.

[0218] Step 704: Obtain at least one first phoneme sequence corresponding to the dubbing text.

[0219] It should be noted that the process of the electronic device acquiring at least one first phoneme corresponding to the dubbing text can refer to the process of acquiring at least one second phoneme corresponding to the target text, and will not be repeated here to avoid repetition.

[0220] Exemplarily, after the electronic device obtains at least one first phoneme corresponding to the dubbing text, it can input the at least one first phoneme into the duration prediction model to predict the first phoneme duration corresponding to each first phoneme, and then determine the first quantity corresponding to each first timbre according to the at least one first timbre duration. The structure of the duration prediction is similar to the timbre extraction module. For example, the dubbing text can be "sound replicated", and the at least one first phoneme corresponding to the dubbing text is "sil sh eng1 in1fu4 k e4 sil". The first phoneme duration of each first phoneme predicted by the duration model is [0.24s, 0.12, s 0.17s, 0.21s, 0.08s, 0.11s, 0.06s, 0.17s, 0.27s]. Assuming that the duration of a single frame of audio is 10ms, the first quantity corresponding to each first phoneme is [24, 12, 17, 21, 8, 11, 6, 17, 27]. The first phoneme sequence obtained after expanding and sorting at least one first phoneme according to the first quantity can be as follows. Fig.10 As shown, the number below each phoneme can express the first quantity of the phoneme.

[0221] Step 705: The electronic device determines a second phoneme sequence corresponding to the target text based on the at least one second phoneme duration, and performs prosody prediction on the dubbing text based on the second phoneme sequence.

[0222] Exemplarily, the electronic device can use a prosody prediction model to predict the prosody of the dubbing text. The prosody prediction model is a transformer decoder-only structure. The structure used in the present invention is Llama. The model can predict the result of the nth step based on the results of the previous n-1 steps. This model structure can make full use of the above information to guide the generation of the current result.

[0223] Exemplarily, the electronic device may determine the second quantity corresponding to each second phoneme according to the duration of the at least one second phoneme, and then obtain a second phoneme sequence by expanding and sorting the at least one second phoneme according to the second quantity.

[0224] Exemplarily, the electronic device can extract the second phoneme sequence and the prosodic feature through the embedding module respectively to obtain the first feature and the second feature, and then splice the first feature and the second feature to obtain the first spliced ​​feature. The dimension of the second phoneme sequence can be 1*104. After the embedding feature extraction, the first feature with a dimension of 104*127 can be extracted. The dimension of the prosodic feature is 1*104. After the embedding feature extraction, the second feature with a dimension of 104*246 can be extracted. The second output is 104*246. After splicing the first feature and the second feature, the first spliced ​​feature prompt_feat with a dimension of 104*384 can be obtained.

[0225] The prosody prediction model can predict the result of the next step based on the input at the current time step. For example, in the application of a text large model, when the input is "The weather is great today. Let's go out", the model can predict the next word "stroll". Then, "stroll" is also input into the model, that is, "The weather is great today. Let's go out stroll", and the model can predict "stroll". Continuing to input "stroll" into the model, and finally inputting "bar" into the model.至此, the model has predicted the complete sentence "The weather is great today. Let's go out for a stroll". Similarly, the first concatenated feature is input into the prosody prediction model to predict the first prosody feature, which is used as the prosody feature target_prosody_1 of the first phoneme in the first phoneme sequence. The target_prosody_1 and the first phoneme feature are respectively subjected to feature extraction through embedding and then concatenated to obtain the second concatenated feature target_feat. The prompt_feat and the target_feat are concatenated to obtain the first concatenated feature prompt_feat again. Here, the concatenation is performed in the dimension of the number of features. For example, the dimension of prompt_feat is [104, 384], and the dimension of target_feat is [1, 384]. After concatenation, the dimension of prompt_feat is [105, 384]. Continuing to send the concatenated prompt_feat into the prosody prediction module to predict the prosody feature target_prosody_2 of the second phoneme in the first phoneme sequence. The prosody feature and the second phoneme of 104.2 are respectively subjected to feature extraction through embedding and then concatenated to obtain a new target feat, which is then concatenated with the prompt_feat and sent into the prosody prediction module for prosody prediction. This process is repeated until the prosody feature target_prosody_n of the last phoneme in the first phoneme sequence is predicted, and finally at least one predicted prosody feature t_prosody_feats = [trager_prosody_1, target_prosody_2,..., target_prosody_n] is obtained.

[0226] Step 706: Obtain a third concatenated feature based on at least one predicted prosody feature and a timbre feature, and use a decoder to process the third concatenated feature to obtain the acoustic feature of the target audio.

[0227] Exemplarily, the electronic device can concatenate each predicted prosody feature in at least one predicted prosody feature with the timbre feature to obtain at least one third concatenated feature.至此, the dubbed text can be transformed into a feature that simultaneously has the timbre and prosody of the target audio. Then, the decoder processes the third concatenated feature to decode the acoustic feature (i.e., the above audio feature) acoustic_feats.

[0228] For example, the dimension of each predicted prosody feature may be 1*384, the dimension of the timbre feature may be 8,246, and the dimension of each third splicing feature in at least one third splicing feature obtained by combining each predicted prosody feature and timbre feature may be 8*640.

[0229] Step 707: The vocoder encodes the acoustic features to obtain dubbing audio.

[0230] For example, the electronic device can encode the acoustic features using an open source vocos. The Vocos vocos does not directly model the audio sample in the time domain, but generates spectral coefficients and quickly reconstructs the audio with the help of inverse Fourier transform, thereby achieving the unity of high efficiency and high quality. The acoustic_feats obtained in step 706 are synthesized into the final dubbing audio through the vocos.

[0231] Combine the following Fig.11 , when the electronic device is a mobile phone, how to generate a dubbing video on the mobile phone interface is explained. Fig.11 As shown, the video processing method provided by the present application may include the following steps 1101 to 1106:

[0232] Step 1101: The mobile phone displays the first interface.

[0233] Exemplarily, the first interface may be an interface of a video dubbing application (Application, APP), or the first interface may be an interface of a video to be dubbed (ie, the first video), and the first interface may include the screen to be dubbed.

[0234] For example, after the mobile phone opens the video dubbing APP and displays the first interface, Figure 2B As shown, a dubbing window 23 may be displayed on the first interface 20, and the dubbing window may include a video import control 236. When the user clicks and inputs the video import control 236, the electronic device may display at least one video to be imported on the first interface 20 in response to the user's click input on the video import control 236. When the user clicks and inputs a first video among the at least one video, the mobile phone may display a video screen of the first video on the first interface 20 in response to the user's click input on the first video.

[0235] Step 1102: The mobile phone extracts the human voice from the first video (ie, extracts the first audio).

[0236] For example, Figure 2BAs shown, when the user clicks and inputs the video and audio acquisition control 234, the mobile phone can extract the human voice (i.e., the first audio) from the first video in response to the user's click and input on the video and audio acquisition control 234, save the extracted human voice audio, and record the start and end time points of the human voice. Figure 2F As shown, a first prompt message 261 is displayed on the first interface 20, prompting the user that the extraction of the video voice is completed; otherwise Figure 2G As shown, a sixth prompt message 27 is displayed on the first interface 20, prompting that no human voice is extracted.

[0237] Step 1103: The mobile phone obtains user audio.

[0238] For example, Figure 2B As shown, when the user clicks and inputs the user audio acquisition control 235 (i.e., the microphone-shaped control in FIG. 2 ), the mobile phone can respond to the user's click input on the user audio acquisition control 235 in combination with the user's click input on the user audio acquisition control 235. Figure 2F ,like Figure 2H As shown, a guide text prompt box 28 is displayed, and the guide text prompt box 28 includes a guide text 281 and an audio recording control 282. After the user clicks and inputs the audio recording control 262, the mobile phone can collect the audio recorded by the user according to the guide text 281 and save it. After saving the audio recorded by the user according to the guide text 281, the mobile phone can collect the audio recorded by the user according to the guide text 281 and save it. Figure 2H ,like Fig.2I As shown, the first prompt information 262 that the user audio is extracted is displayed on the first interface 20.

[0239] Step 1104: The mobile phone selects a dubbing mode in response to the user's first input.

[0240] Exemplarily, there may be three dubbing modes provided by the mobile phone, wherein the video human voice is obtained by step 202 , and the user audio is obtained by step 203 .

[0241] Mode 1: User rhythm + user voice timbre

[0242] If the user likes his or her own voice and hopes to use his or her own voice and pronunciation style to generate dubbing audio, then step 702 extracts the timbre of the user's audio, step 703 extracts the user's rhythm, and uses the rhythm as the input of the rhythm prediction model in step 705 to predict the rhythm of the dubbing text, and finally generates dubbing audio consistent with the user's timbre and rhythm.

[0243] Mode 2: User voice + video vocal rhythm

[0244] If the user likes the pronunciation style of the human voice in the video and only wants to modify the timbre, then: step 702 extracts the timbre of the user's audio, step 703 extracts the rhythm of the video human voice audio and uses the rhythm as the input of the rhythm prediction module 705 to predict the rhythm of the dubbing text, and finally generates dubbing audio consistent with the user's timbre and rhythm.

[0245] Mode 3: User rhythm + video voice chromaticity

[0246] If the user likes the timbre of the human voice in the video and hopes to use the timbre of the human voice in the video and his own pronunciation style to generate audio, then step 702 extracts the timbre of the human voice audio in the video, step 703 extracts the rhythm of the user audio, and uses the rhythm as the input of the rhythm prediction model in step 705 to predict the rhythm of the dubbing text, and finally generates dubbing audio consistent with the user's timbre and rhythm.

[0247] For example, Figure 2B As shown, when the user can click and input the dubbing mode selection control 231, the mobile phone can respond to the user's click and input on the dubbing mode selection control 231, display the dubbing mode selection window 23, and display the dubbing mode selection window 24 including dubbing mode 1, dubbing mode 2 and dubbing mode 3. The user's click and input on the dubbing mode selection control 231 is the above-mentioned first input. When the human voice is not extracted in step 702, the color of the dubbing mode 2 and mode 3 option buttons becomes lighter and cannot be clicked; when the human voice is detected, the three mode option buttons are displayed normally and can all be selected.

[0248] Step 1105: The mobile phone obtains the dubbing text.

[0249] For example, Figure 2B As shown, when the mobile phone receives the user's click input on the dubbing text acquisition control 233, it can respond to the user's click input on the dubbing text acquisition control 233, such as Figure 2D As shown, a text prompt box 25 is displayed on the first interface 20. If a human voice is extracted in step 702, the text prompt box is filled with the text of the extracted human voice by default, and this text is obtained by the ASR tool. The user can also change it to what he wants. If no human voice is detected in step 702, Figure 2E As shown, the text prompt box 25 displays a text prompting the user to input.

[0250] Step 1106: The mobile phone generates a dubbing video (ie, the second video mentioned above).

[0251] For example, Figure 2B As shown, when the mobile phone receives a click input from the user to the video generation control 233, the mobile phone can generate a dubbing video based on the dubbing audio and the first video in response to the click input from the user to the video generation control 233.

[0252] Exemplarily, when the duration of the generated dubbing audio is inconsistent with the duration of the first video or the duration of the human voice audio of the video, it is necessary to expand or shorten the duration of the generated dubbing audio. The mobile phone can use the nearest neighbor interpolation method to adjust the duration of the dubbing audio. Exemplarily, the mobile phone can change the number of audio features through the interpolation coefficient, thereby adjusting the duration of the dubbing audio. For details, please refer to the above steps 108a to 108c, which will not be repeated here to avoid repetition.

[0253] The generated audio needs to be inserted into the video. The speaking speed is adjusted through interpolation. A speaking speed that is too fast or too slow will cause the sound to be out of sync with the picture. In step 704, the duration of the final synthesized dubbing audio can be obtained. When the calculated interpolation coefficient α is not within the set range, the user can be prompted to re-given the text to be synthesized to generate audio that is close to the length of the video or the length of the video voice. According to research, the interpolation coefficient α is best when it is between 0.85 and 1.1. When α<0.85, the user is prompted to expand the dubbing text; when α>1.1, the user is prompted to shorten the dubbing text. For details, please refer to Figure 2J and Figure 2K , I will not go into details here.

[0254] Exemplarily, the mobile phone may use the audio / video processing tool ffmpeg to merge the audio and video.

[0255] Exemplarily, when there is no human voice in the first video, the ffmpeg tool is directly used to combine the audio / video; when there is human voice in the first source video, the synthesized audio is combined with the ffmpeg tool according to the start and end time points of the human voice in the video.

[0256] For example, to increase the fun, if the first video is content containing lip movements, such as interviews, speeches, talk shows, etc., the mobile phone can use the MyHeyGen open source tool to align the synthesized audio with the lip movements of the video content to generate a lively and interesting re-dubbed video.

[0257] It should be noted that the specific implementation process of the video processing method in the above steps can refer to the relevant description of the above embodiment. To avoid repetition, this embodiment will not be repeated here.

[0258] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not impose any restrictions on this.

[0259] The video processing method provided in the embodiment of the present application can be executed by a video processing device. In the embodiment of the present application, the video processing device provided in the embodiment of the present application is described by taking the video processing method executed by the video processing device as an example.

[0260] Fig.12 1 is a schematic diagram of the structure of a video processing device 1200 provided in an embodiment of the present application. The video processing device 1200 includes: a receiving module 1201 and a processing module 1202.

[0261] The receiving module 1201 is used to receive a first input to a first interface, where the first interface includes a screen of a first video;

[0262] The processing module 1202 is used for acquiring the dubbing timbre feature and the dubbing rhythm feature in response to the first input;

[0263] Based on the dubbing timbre feature and the dubbing rhythm feature, obtaining dubbing audio;

[0264] A second video is generated based on the dubbing audio and the first video.

[0265] In some embodiments of the present application, the dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a video rhythm feature;

[0266] The processing module 1202 is specifically used for:

[0267] Extracting the user timbre feature from the user audio, and extracting the video rhythm feature from the first audio, where the first audio is the audio in the first video;

[0268] The dubbing audio is generated based on the user's timbre characteristics and the video prosody characteristics.

[0269] In some embodiments of the present application, the dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a user rhythm feature;

[0270] The processing module 1202 is specifically used for:

[0271] Extracting the user timbre feature and the user prosody feature from the user audio;

[0272] The dubbing audio is generated based on the user timbre feature and the user prosody feature.

[0273] In some embodiments of the present application, the dubbing timbre feature is a video timbre feature, and the dubbing rhythm feature is a user rhythm feature;

[0274] The processing module 1202 is specifically used for:

[0275] Extracting the video timbre feature from a first audio, and extracting the user prosody feature from a user audio, wherein the first audio is the audio in the first video;

[0276] The dubbing audio is generated based on the video timbre characteristics and the user prosody characteristics.

[0277] In some embodiments of the present application, the first interface further includes an audio acquisition control;

[0278] The receiving module 1201 is further configured to receive a second input to the audio acquisition control;

[0279] Combination Fig.12 ,like Fig.13 As shown, the device 1200 also includes: a display module 1203, which is used to display first prompt information in response to the second input, and the first prompt information is used to prompt that the target audio is obtained, and the target audio is the audio in the first video or the user audio.

[0280] In some embodiments of the present application, the first interface further includes a dubbing text acquisition control;

[0281] The receiving module 1201 is further configured to receive a third input to the dubbing text acquisition control before generating the dubbing audio based on the timbre feature and the rhythm feature;

[0282] A display module 1203 is used to display a dubbing text in response to the third input, where the dubbing text is a text corresponding to the first video or a text input by a user;

[0283] The processing module 1202 is specifically used for:

[0284] The dubbing audio is generated based on the timbre feature, the rhythm feature and the dubbing text.

[0285] In some embodiments of the present application, the dubbing text is the text corresponding to the first video;

[0286] The display module 1203 is specifically used for:

[0287] In response to the third input, when the first audio is detected in the first video, the dubbing text is displayed, where the dubbing text is the text corresponding to the first audio in the first video.

[0288] In some embodiments of the present application, the dubbing text is a text input by a user, and the third input includes a first sub-input and a second sub-input;

[0289] The display module 1203 is specifically used for:

[0290] In response to the first sub-input, displaying a text input box on the first interface;

[0291] In response to a second sub-input to the text input box, the dubbing text is displayed in the text input box, the dubbing text being text corresponding to the second sub-input.

[0292] In some embodiments of the present application, the processing module 1202 is specifically used to:

[0293] Acquire a first phoneme sequence corresponding to the dubbing text;

[0294] Acquire a second phoneme sequence corresponding to a target audio, where the target audio is audio in the first video or user audio;

[0295] generating an audio feature of the dubbing audio based on the first phoneme sequence, the second phoneme sequence, the dubbing timbre feature, and the dubbing rhythm feature;

[0296] The audio features are encoded to obtain the dubbing audio.

[0297] In some embodiments of the present application, the processing module 1202 is specifically used to:

[0298] Acquire at least one first phoneme and at least one first phoneme duration of the dubbing text, where each first phoneme duration is the duration of a first phoneme;

[0299] determining a first quantity of each first phoneme in the at least one first phoneme based on the at least one first phoneme duration;

[0300] Based on the first number, the at least one first phoneme is sorted to obtain the first phoneme sequence.

[0301] In some embodiments of the present application, the processing module 1202 is further configured to, after acquiring at least one first phoneme duration of the dubbing text, calculate the sum of the at least one first phoneme duration to obtain the first duration of the dubbing audio;

[0302] Calculating a ratio of the first duration to a video duration of the first video;

[0303] If the ratio is within a first value range, performing interpolation processing on the audio feature based on the ratio to obtain the processed audio feature;

[0304] The display module 1203 is further configured to display a second prompt message if the ratio is outside the first numerical range and is smaller than a minimum value of the first numerical range, wherein the second prompt message is configured to prompt the user to expand the dubbing text.

[0305] If the ratio is outside the first numerical range and is greater than the maximum value of the first numerical range, a third prompt message is displayed, and the third prompt message is used to prompt to shorten the dubbing text.

[0306] In some embodiments of the present application, the processing module 1202 is specifically used to:

[0307] Based on the second phoneme sequence and the dubbing rhythm feature, obtaining a first splicing feature;

[0308] Inputting the first concatenation feature into a prosody prediction model to obtain a predicted prosody feature of an i-th phoneme in the first phoneme sequence, wherein the first phoneme sequence includes N phonemes, where N is a positive integer and i is an integer from 1 to N;

[0309] Based on the predicted prosodic feature and the i-th phoneme, obtaining a second concatenation feature;

[0310] splicing the second splicing feature and the first splicing feature to obtain the first splicing feature again, until i=N, to obtain at least one predicted prosodic feature, each predicted prosodic feature corresponding to a phoneme in the first phoneme sequence;

[0311] Based on the at least one predicted prosody feature and the dubbing timbre feature, a third splicing feature is obtained, and the third splicing feature is decoded to generate an audio feature of the dubbing text.

[0312] In the video processing device provided in the embodiment of the present application, a first input to a first interface is received, and the first interface includes a screen of a first video; in response to the first input, dubbing timbre characteristics and dubbing rhythm characteristics are obtained; based on the dubbing timbre characteristics and dubbing rhythm characteristics, dubbing audio is obtained; based on the dubbing audio and the first video, a second video is generated. In this way, the electronic device can obtain the corresponding dubbing timbre characteristics and dubbing rhythm characteristics according to the user's input, and then generate dubbing audio according to the dubbing timbre characteristics and dubbing rhythm characteristics, thereby improving the flexibility of the video dubbing method. In addition, there is no need for the user to continuously record the audio of the entire video, thereby improving the efficiency of video dubbing.

[0313] The video processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device, an augmented reality / virtual reality device, a robot, a wearable device, a super mobile personal computer, a netbook or a personal digital assistant, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0314] The video processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0315] The video processing device provided in the embodiment of the present application can implement each process implemented by each embodiment of the above-mentioned video processing method. To avoid repetition, it will not be described again here.

[0316] Alternatively, if Fig.14 As shown, an embodiment of the present application further provides an electronic device 1400, including a processor 1401 and a memory 1402, wherein the memory 1402 stores programs or instructions that can be executed on the processor 1401, and when the program or instructions are executed by the processor 1401, the various steps of the above-mentioned video processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0317] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0318] Fig.15 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0319] The electronic device 1500 includes but is not limited to: a radio frequency unit 1501, a network module 1502, an audio output unit 1503, an input unit 1504, a sensor 1505, a display unit 1506, a user input unit 1507, an interface unit 1508, a memory 1509, and a processor 1510 and other components.

[0320] Those skilled in the art will appreciate that the electronic device 1500 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.15 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0321] The user input unit 1507 is used to receive a first input to a first interface, where the first interface includes a screen of a first video;

[0322] Processor 1510, for obtaining dubbing timbre characteristics and dubbing rhythm characteristics in response to the first input;

[0323] Based on the dubbing timbre feature and the dubbing rhythm feature, obtaining dubbing audio;

[0324] A second video is generated based on the dubbing audio and the first video.

[0325] In some embodiments of the present application, the dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a video rhythm feature;

[0326] The processor 1510 is specifically configured to:

[0327] Extracting the user timbre feature from the user audio, and extracting the video rhythm feature from the first audio, where the first audio is the audio in the first video;

[0328] The dubbing audio is generated based on the user's timbre characteristics and the video prosody characteristics.

[0329] In some embodiments of the present application, the dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a user rhythm feature;

[0330] The processor 1510 is specifically configured to:

[0331] Extracting the user timbre feature and the user prosody feature from the user audio;

[0332] The dubbing audio is generated based on the user timbre feature and the user prosody feature.

[0333] In some embodiments of the present application, the dubbing timbre feature is a video timbre feature, and the dubbing rhythm feature is a user rhythm feature;

[0334] The processor 1510 is specifically configured to:

[0335] Extracting the video timbre feature from a first audio, and extracting the user prosody feature from a user audio, wherein the first audio is the audio in the first video;

[0336] The dubbing audio is generated based on the video timbre characteristics and the user prosody characteristics.

[0337] In some embodiments of the present application, the first interface further includes an audio acquisition control;

[0338] The user input unit 1507 is further used to receive a second input to the audio acquisition control;

[0339] The display unit 1506 is used to display first prompt information in response to the second input, where the first prompt information is used to prompt that the target audio is acquired, and the target audio is the audio in the first video or the user audio.

[0340] In some embodiments of the present application, the first interface further includes a dubbing text acquisition control;

[0341] The user input unit 1507 is further used to receive a third input to the dubbing text acquisition control before generating the dubbing audio based on the timbre feature and the rhythm feature;

[0342] A display unit 1506 is used to display a dubbing text in response to the third input, where the dubbing text is a text corresponding to the first video or a text input by a user;

[0343] The processor 1510 is specifically configured to:

[0344] The dubbing audio is generated based on the timbre feature, the rhythm feature and the dubbing text.

[0345] In some embodiments of the present application, the dubbing text is the text corresponding to the first video;

[0346] The display unit 1506 is specifically used for:

[0347] In response to the third input, when the first audio is detected in the first video, the dubbing text is displayed, where the dubbing text is the text corresponding to the first audio in the first video.

[0348] In some embodiments of the present application, the dubbing text is a text input by a user, and the third input includes a first sub-input and a second sub-input;

[0349] The display unit 1506 is specifically used for:

[0350] In response to the first sub-input, displaying a text input box on the first interface;

[0351] In response to a second sub-input to the text input box, the dubbing text is displayed in the text input box, the dubbing text being text corresponding to the second sub-input.

[0352] In some embodiments of the present application, the processor 1510 is specifically configured to:

[0353] Acquire a first phoneme sequence corresponding to the dubbing text;

[0354] Acquire a second phoneme sequence corresponding to a target audio, where the target audio is audio in the first video or user audio;

[0355] generating an audio feature of the dubbing audio based on the first phoneme sequence, the second phoneme sequence, the dubbing timbre feature, and the dubbing rhythm feature;

[0356] The audio features are encoded to obtain the dubbing audio.

[0357] In some embodiments of the present application, the processor 1510 is specifically configured to:

[0358] Acquire at least one first phoneme and at least one first phoneme duration of the dubbing text, where each first phoneme duration is the duration of a first phoneme;

[0359] determining a first quantity of each first phoneme in the at least one first phoneme based on the at least one first phoneme duration;

[0360] Based on the first number, the at least one first phoneme is sorted to obtain the first phoneme sequence.

[0361] In some embodiments of the present application, the processor 1510 is further configured to, after acquiring at least one first phoneme duration of the dubbing text, calculate the sum of the at least one first phoneme duration to obtain a first duration of the dubbing audio;

[0362] Calculating a ratio of the first duration to a video duration of the first video;

[0363] If the ratio is within a first value range, performing interpolation processing on the audio feature based on the ratio to obtain the processed audio feature;

[0364] The display unit 1506 is further configured to display second prompt information if the ratio is outside the first numerical range and is less than a minimum value of the first numerical range, wherein the second prompt information is used to prompt the user to expand the dubbing text;

[0365] If the ratio is outside the first numerical range and is greater than the maximum value of the first numerical range, a third prompt message is displayed, and the third prompt message is used to prompt to shorten the dubbing text.

[0366] In some embodiments of the present application, the processor 1510 is specifically configured to:

[0367] Based on the second phoneme sequence and the dubbing rhythm feature, obtaining a first splicing feature;

[0368] Inputting the first concatenation feature into a prosody prediction model to obtain a predicted prosody feature of an i-th phoneme in the first phoneme sequence, wherein the first phoneme sequence includes N phonemes, where N is a positive integer and i is an integer from 1 to N;

[0369] Based on the predicted prosodic feature and the i-th phoneme, obtaining a second concatenation feature;

[0370] splicing the second splicing feature and the first splicing feature to obtain the first splicing feature again, until i=N, to obtain at least one predicted prosodic feature, each predicted prosodic feature corresponding to a phoneme in the first phoneme sequence;

[0371] Based on the at least one predicted prosody feature and the dubbing timbre feature, a third splicing feature is obtained, and the third splicing feature is decoded to generate an audio feature of the dubbing text.

[0372] In the electronic device provided in the embodiment of the present application, a first input to a first interface is received, and the first interface includes a screen of a first video; in response to the first input, dubbing timbre characteristics and dubbing rhythm characteristics are obtained; based on the dubbing timbre characteristics and dubbing rhythm characteristics, dubbing audio is obtained; based on the dubbing audio and the first video, a second video is generated. In this way, the electronic device can obtain the corresponding dubbing timbre characteristics and dubbing rhythm characteristics according to the user's input, and then generate dubbing audio according to the dubbing timbre characteristics and dubbing rhythm characteristics, thereby improving the flexibility of the video dubbing method. In addition, there is no need for the user to continuously record the audio of the entire video, thereby improving the efficiency of video dubbing.

[0373] It should be understood that in the embodiment of the present application, the input unit 1504 may include a graphics processor (Graphics Processing Unit, GPU) 15041 and a microphone 15042, and the graphics processor 15041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1506 may include a display panel 15061, and the display panel 15061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1507 includes a touch panel 15071 and at least one of other input devices 15072. The touch panel 15071 is also called a touch screen. The touch panel 15071 may include two parts: a touch detection device and a touch controller. Other input devices 15072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0374] The memory 1509 can be used to store software programs and various data. The memory 1509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1509 may include a volatile memory or a non-volatile memory, or the memory 1509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0375] The processor 1510 may include one or more processing units; optionally, the processor 1510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1510.

[0376] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0377] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0378] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0379] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0380] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0381] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0382] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0383] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Receiving a first input to a first interface, wherein the first interface includes a screen of a first video; In response to the first input, acquiring a dubbing timbre feature and a dubbing rhythm feature; Based on the dubbing timbre feature and the dubbing rhythm feature, obtaining dubbing audio; A second video is generated based on the dubbing audio and the first video.

2. The method according to claim 1, characterized in that The dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a video rhythm feature; The step of obtaining the dubbing audio based on the dubbing timbre feature and the dubbing rhythm feature comprises: Extracting the user timbre feature from the user audio, and extracting the video rhythm feature from the first audio, where the first audio is the audio in the first video; The dubbing audio is generated based on the user's timbre characteristics and the video prosody characteristics.

3. The method according to claim 1, characterized in that: The dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a user rhythm feature; The step of obtaining the dubbing audio based on the dubbing timbre feature and the dubbing rhythm feature comprises: Extracting the user timbre feature and the user prosody feature from the user audio; The dubbing audio is generated based on the user timbre feature and the user prosody feature.

4. The method according to claim 1, characterized in that: The dubbing timbre feature is a video timbre feature, and the dubbing rhythm feature is a user rhythm feature; The step of obtaining the dubbing audio based on the dubbing timbre feature and the dubbing rhythm feature comprises: Extracting the video timbre feature from a first audio, and extracting the user prosody feature from a user audio, wherein the first audio is the audio in the first video; The dubbing audio is generated based on the video timbre characteristics and the user prosody characteristics.

5. The method according to any one of claims 2 to 4, characterized in that The first interface also includes an audio acquisition control; The method further comprises: receiving a second input to the audio acquisition control; In response to the second input, a first prompt message is displayed, where the first prompt message is used to prompt that a target audio is acquired, where the target audio is the audio in the first video or the user audio.

6. The method according to claim 1, characterized in that The first interface also includes a dubbing text acquisition control; Before generating the dubbing audio based on the dubbing timbre feature and the dubbing rhythm feature, the method further includes: Receiving a third input to the dubbing text acquisition control; In response to the third input, displaying a dubbing text, where the dubbing text is a text corresponding to the first video or a text input by a user; The generating of the dubbing audio based on the dubbing timbre feature and the dubbing rhythm feature comprises: The dubbing audio is generated based on the dubbing timbre feature, the dubbing rhythm feature and the dubbing text.

7. The method according to claim 6, characterized in that The dubbing text is the text corresponding to the first video; The step of displaying the dubbing text in response to the third input comprises: In response to the third input, when the first audio is detected in the first video, the dubbing text is displayed, where the dubbing text is the text corresponding to the first audio in the first video.

8. The method according to claim 6, characterized in that The dubbing text is a text input by a user, and the third input includes a first sub-input and a second sub-input; The step of displaying the dubbing text in response to the third input comprises: In response to the first sub-input, displaying a text input box on the first interface; In response to a second sub-input to the text input box, the dubbing text is displayed in the text input box, the dubbing text being text corresponding to the second sub-input.

9. The method according to claim 6, characterized in that The step of generating the dubbing audio based on the dubbing timbre feature, the dubbing rhythm feature and the dubbing text comprises: Acquire a first phoneme sequence corresponding to the dubbing text; Acquire a second phoneme sequence corresponding to a target audio, where the target audio is audio in the first video or user audio; generating an audio feature of the dubbing audio based on the first phoneme sequence, the second phoneme sequence, the dubbing timbre feature, and the dubbing rhythm feature; The audio features are encoded to obtain the dubbing audio.

10. The method according to claim 9, characterized in that The step of obtaining a first phoneme sequence corresponding to the dubbing text includes: Acquire at least one first phoneme and at least one first phoneme duration of the dubbing text, where each first phoneme duration is the duration of a first phoneme; determining a first quantity of each first phoneme in the at least one first phoneme based on the at least one first phoneme duration; Based on the first number, the at least one first phoneme is sorted to obtain the first phoneme sequence.

11. The method according to claim 10, characterized in that After obtaining at least one first phoneme duration of the dubbing text, the method further includes: Calculating the sum of the duration of the at least one first phoneme to obtain a first duration of the dubbing audio; Calculating a ratio of the first duration to a video duration of the first video; If the ratio is within a first value range, performing interpolation processing on the audio feature based on the ratio to obtain the processed audio feature; If the ratio is outside the first numerical range and the ratio is less than the minimum value of the first numerical range, a second prompt message is displayed, wherein the second prompt message is used to prompt the expansion of the dubbing text; If the ratio is outside the first numerical range and is greater than the maximum value of the first numerical range, a third prompt message is displayed, and the third prompt message is used to prompt to shorten the dubbing text.

12. The method according to claim 9, characterized in that The step of generating the audio feature of the dubbing audio based on the first phoneme sequence, the second phoneme sequence, the dubbing timbre feature and the dubbing rhythm feature comprises: Based on the second phoneme sequence and the dubbing rhythm feature, obtaining a first splicing feature; Inputting the first concatenation feature into a prosody prediction model to obtain a predicted prosody feature of an i-th phoneme in the first phoneme sequence, wherein the first phoneme sequence includes N phonemes, where N is a positive integer and i is an integer from 1 to N; Based on the predicted prosodic feature and the i-th phoneme, obtaining a second concatenation feature; splicing the second splicing feature and the first splicing feature to obtain the first splicing feature again, until i=N, to obtain at least one predicted prosodic feature, each predicted prosodic feature corresponding to a phoneme in the first phoneme sequence; Based on the at least one predicted prosody feature and the dubbing timbre feature, a third splicing feature is obtained, and the third splicing feature is decoded to generate an audio feature of the dubbing text.

13. A video processing device, characterized in that: The device comprises: A receiving module, configured to receive a first input to a first interface, wherein the first interface includes a screen of a first video; A processing module, for obtaining dubbing timbre characteristics and dubbing rhythm characteristics in response to the first input; Based on the dubbing timbre feature and the dubbing rhythm feature, obtaining dubbing audio; A second video is generated based on the dubbing audio and the first video.

14. The device according to claim 13, characterized in that The dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a video rhythm feature; The processing module is specifically used for: Extracting the user timbre feature from the user audio, and extracting the video rhythm feature from the first audio, where the first audio is the audio in the first video; The dubbing audio is generated based on the user's timbre characteristics and the video prosody characteristics.

15. The device according to claim 13, characterized in that The dubbing timbre feature is a user timbre feature, and the dubbing rhythm feature is a user rhythm feature; The processing module is specifically used for: Extracting the user timbre feature and the user prosody feature from the user audio; The dubbing audio is generated based on the user timbre feature and the user prosody feature.

16. The device according to claim 13, characterized in that The dubbing timbre feature is a video timbre feature, and the dubbing rhythm feature is a user rhythm feature; The processing module is specifically used for: Extracting the video timbre feature from a first audio, and extracting the user prosody feature from a user audio, wherein the first audio is the audio in the first video; The dubbing audio is generated based on the video timbre characteristics and the user prosody characteristics.

17. The device according to claim 13, characterized in that The first interface also includes a dubbing text acquisition control; The receiving module is further used to receive a third input to the dubbing text acquisition control before generating the dubbing audio based on the timbre feature and the rhythm feature; The device further includes: a display module, configured to display a dubbing text in response to the third input, wherein the dubbing text is a text corresponding to the first video or a text input by a user; The processing module is specifically used for: The dubbing audio is generated based on the timbre feature, the rhythm feature and the dubbing text.

18. The device according to claim 17, characterized in that The processing module is specifically used for: Acquire a first phoneme sequence corresponding to the dubbing text; Acquire a second phoneme sequence corresponding to a target audio, where the target audio is audio in the first video or user audio; generating an audio feature of the dubbing audio based on the first phoneme sequence, the second phoneme sequence, the dubbing timbre feature, and the dubbing rhythm feature; The audio features are encoded to obtain the dubbing audio.

19. The device according to claim 18, characterized in that The processing module is specifically used for: Based on the second phoneme sequence and the dubbing rhythm feature, obtaining a first splicing feature; Inputting the first concatenation feature into a prosody prediction model to obtain a predicted prosody feature of an i-th phoneme in the first phoneme sequence, wherein the first phoneme sequence includes N phonemes, where N is a positive integer and i is an integer from 1 to N; Based on the predicted prosodic feature and the i-th phoneme, obtaining a second concatenation feature; splicing the second splicing feature and the first splicing feature to obtain the first splicing feature again, until i=N, to obtain at least one predicted prosodic feature, each predicted prosodic feature corresponding to a phoneme in the first phoneme sequence; Based on the at least one predicted prosody feature and the dubbing timbre feature, a third splicing feature is obtained, and the third splicing feature is decoded to generate an audio feature of the dubbing text.

20. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the video processing method according to any one of claims 1 to 12 are implemented.

21. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the video processing method according to any one of claims 1 to 12 are implemented.

Images