Video content modification method and apparatus, electronic device, and storage medium

Abstract

The application provides a video content modification method and device, electronic equipment and storage medium, and relates to the technical field of multimedia. The method comprises the following steps: extracting the initial audio of a target person in an original video, extracting target voice and audio text according to the initial audio; modifying the audio text to obtain target text, and synthesizing target audio according to the target text and the target voice; recording a face lip video according to the target audio to obtain a driving video; performing lip calibration according to the driving video and the target audio to obtain a standard driving video; and inputting the standard driving video and the original video into an implicit key point driving model to obtain a synthesized video. The application provides natural lip, expression and head posture by recording a face driving video, so that the generated video is more realistic and smooth in terms of lip, and ensures that the final video effect is more in line with expectations. The driving video and the audio are inferred and trained in a hidden space, so that the speed and quality of video generation are improved.

Description

Technical Field

[0001] This invention relates to the field of multimedia technology, specifically to a method, apparatus, electronic device, and storage medium for modifying video content. Background Technology

[0002] In the fields of media production and content creation, situations frequently arise where adjustments or modifications are needed to audio segments in recorded videos. For example, in interview scenarios, journalists or content creators may discover problems with certain statements made by the interviewee after the interview. If these issues are not corrected, they may affect the accuracy and effectiveness of the information conveyed. However, due to time and resource constraints, it is often impossible to recreate the original interview scene for re-recording. In the secondary creation of film and television works, creators frequently need to extract segments from existing works and modify or re-edit the dialogue to create new, innovative, and engaging videos.

[0003] Existing methods typically involve first synthesizing audio from edited text, ensuring consistency with the original speaker's voice, then using this audio to drive lip-syncing in the video, and finally trimming away unnecessary video clips. However, this method can only modify lip movements; it cannot alter eyes, eyebrows, facial expressions, or head posture. Furthermore, it produces poor lip-syncing results for Chinese text and side-profile videos, easily losing some of the original person's mouth features, and the resulting video has low resolution. Therefore, a method is urgently needed to address the problem of poor video quality when synthesizing content after modification. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides a video content modification method, apparatus, electronic device and storage medium, which effectively solves the problem of poor video quality when modifying existing video content.

[0005] In a first aspect, the present invention provides a method for modifying video content, the method comprising:

[0006] Extract the initial audio of the target person from the original video, and extract the target voice and audio text based on the initial audio;

[0007] Modify the audio text to obtain the target text, and synthesize the target audio based on the target text and the target human voice;

[0008] Record facial lip-sync video based on the target audio to obtain driving video;

[0009] Lip-sync calibration is performed based on the driving video and the target audio to obtain a standard driving video;

[0010] Input the standard driving video and the original video into an implicit key point driving model to obtain a synthesized video.

[0011] Further, the initial audio of the target person in the original video is extracted, and target voice and audio text are extracted according to the initial audio, including:

[0012] The initial audio of the target person in the original video is extracted by using a clipping tool;

[0013] The initial audio is input into a voice separation technology model to obtain a voice file;

[0014] The voice file is input into a noise reduction model to obtain target voice;

[0015] The initial audio is identified by using a speech recognition model to obtain the audio text.

[0016] Further, the target text is obtained by modifying the audio text, and target audio is synthesized according to the target text and the target voice, including:

[0017] The audio text is edited and modified to obtain a target text;

[0018] The target text and the target voice are input into a sound cloning model for synthesis to obtain target audio.

[0019] Further, the standard driving video is obtained according to the driving video and the target audio, including:

[0020] The driving video and the target audio are input into an audio-video synchronization model for matching calculation to obtain a synchronization score;

[0021] The driving video is calibrated frame by frame according to the synchronization score to obtain a best audio timestamp corresponding to each frame of video;

[0022] The driving video is adjusted according to the best audio timestamp and the lip shape of the driving video to obtain the standard driving video.

[0023] Further, the driving video and the target audio are input into an audio-video synchronization model for matching calculation to obtain a synchronization score, including:

[0024] The driving video and the target audio are respectively converted into a video feature vector and an audio feature vector;

[0025] The video feature vector and the audio feature vector are respectively subjected to spatio-temporal feature extraction to obtain video spatio-temporal feature information and audio spatio-temporal feature information;

[0026] correlation between the video spatio-temporal feature information and the audio spatio-temporal feature information is calculated to generate a similarity graph;

[0027] maximum pooling processing is performed on the similarity graph to obtain a time offset;

[0028] The synchronization score is generated according to the time offset.

[0029] Further, the standard driving video and the original video are input into an implicit key point driving model to obtain a synthesized video, including:

[0030] Each frame of driving picture of the standard driving video and each frame of original picture of the original video are obtained;

[0031] The driving picture and the original picture are encoded from a pixel space to a hidden space to obtain a hidden space driving feature map and a hidden space original feature map;

[0032] A hidden space synthesized feature map is generated according to the hidden space driving feature map and the hidden space original feature map;

[0033] The hidden space synthesized feature map is decoded from the hidden space to the pixel space to obtain each frame of synthesized picture;

[0034] The synthesized video is generated according to the synthesized picture and the target audio.

[0035] Further, the hidden space synthesized feature map is generated according to the hidden space driving feature map and the hidden space original feature map, including:

[0036] An appearance feature of the target person in the hidden space original feature map is extracted to obtain a target appearance feature;

[0037] Action features of the target person in the hidden space original feature map and the hidden space driving feature map are respectively extracted to obtain an original target action feature and a driving target action feature;

[0038] The target appearance feature, the original target action feature and the driving target action feature are feature-encapsulated to obtain an encapsulated feature map;

[0039] The encapsulated feature map is decoded to obtain the hidden space synthesized feature map.

[0040] In a second aspect, the present application provides a video content modification device, the device comprising:

[0041] An audio extraction module is configured to extract an initial audio of a target person in an original video, extract a target voice and audio text according to the initial audio;

[0042] An audio synthesis module is used to modify the audio text to obtain the target text, and to synthesize the target audio based on the target text and the target human voice;

[0043] The video recording module is used to record facial lip-sync video based on the target audio to obtain driving video;

[0044] The video calibration module is used to perform lip-sync calibration based on the driving video and the target audio to obtain a standard driving video;

[0045] The video synthesis module is used to input the standard driving video and the original video into the implicit keypoint driving model to obtain a synthesized video.

[0046] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video content modification method as described in the first aspect of the present invention.

[0047] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video content modification method as described in the first aspect of the present invention.

[0048] This invention provides a video content modification method, apparatus, electronic device, and storage medium. It uses cloning technology to generate speech similar to the target speaker's voice, while simultaneously recording facial-driven video to provide natural lip movements, facial expressions, and head postures, capturing natural lip movements of the actor. Post-production adjustment capabilities allow for fine-tuning of details, making the generated video more realistic and fluid in terms of lip movements, ensuring the final video effect better meets expectations. Driving video and audio through inference and training in the latent space, utilizing the encoding and decoding capabilities of a variational encoder, significantly improves computational efficiency, increasing the speed and quality of video generation. The use of audio-video synchronization technology ensures precise synchronization between the video and audio, allowing adjustments even for minor time differences during recording, ensuring the quality of the final video. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a first schematic diagram of the video content modification method provided in this embodiment of the invention;

[0051] Figure 2 is a second schematic diagram of a video content modification method provided by an embodiment of the present application;

[0052] Figure 3 is a third schematic diagram of a video content modification method provided by an embodiment of the present application;

[0053] Figure 4 is a fourth schematic diagram of a video content modification method provided by an embodiment of the present application;

[0054] Figure 5 is a fifth schematic diagram of a video content modification method provided by an embodiment of the present application;

[0055] Figure 6 is a sixth schematic diagram of a video content modification method provided by an embodiment of the present application;

[0056] Figure 7 is a schematic diagram of an architecture of an implicit key point driven model in an embodiment of the present application;

[0057] Figure 8 is a seventh schematic diagram of a video content modification method provided by an embodiment of the present application;

[0058] Figure 9 is a schematic diagram of a structure of a video content modification device provided by an embodiment of the present application;

[0059] Figure 10 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present application.

[0060] Main element symbol explanation:

[0061] 900, video content modification device; 910, audio extraction module; 920, audio synthesis module; 930, video recording module; 940, video calibration module; 950, video synthesis module; 1000, electronic device; 1010, processor; 1020, communication interface; 1030, memory; 1040, communication bus. DETAILED DESCRIPTION

[0062] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be further clearly and completely described below with reference to the accompanying drawings of the embodiments of the present application. It should be noted that the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0063] In addition, the terms "first", "second", "third", etc. are used herein only to describe different instances, and are not used to indicate or imply relative importance or a number of indicated technical features. Thus, features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0065] In the field of media production and content creation, it is often necessary to adjust or modify the voice segment of a recorded video. The existing method usually synthesizes the edited text content into audio, maintains the same tone as the original speaker, and uses the audio to drive the video synthesis mouth shape to complete the mouth shape synchronization, and finally clips the redundant video segment. However, this method can only modify the mouth shape, and the eyes, eyebrows, facial expressions and head posture cannot be modified, and the mouth shape synthesis effect for Chinese and side face videos is not good, which easily loses some features of the original character's mouth, and the generated video has low resolution and poor quality.

[0066] Embodiment 1

[0067] The embodiment of the present application provides a video content modification method, which effectively solves the problem of poor video effect in the existing video content modification. Figure 1 is the first schematic diagram of the video content modification method provided by the embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0068] S100, extracting the initial audio of the target person in the original video, and extracting the target voice and audio text according to the initial audio.

[0069] Figure 2 is the second schematic diagram of the video content modification method provided by the embodiment of the present application, as shown in Figure 2 The audio extraction specifically comprises the following steps:

[0070] S110, using a clip tool to extract the initial audio of the target person in the original video.

[0071] In the embodiment of the present application, the video segment needing to be modified in the original video can be directly cut out by using a cutting tool as the original video material. Then, the audio part of the same target person speaking in the original video material needing to be modified is extracted as the initial audio. The audio length of the initial audio can be 10-15 seconds, which can be set according to actual conditions.

[0072] In S120, the initial audio is input into a human voice separation technology model to obtain a human voice file.

[0073] Optionally, the human voice separation technology model can adopt a UVR5 model, which can separate human voice and background music or noise from mixed audio. The initial audio is input into the UVR5 model to separate human voice and background noise by the model to obtain the human voice file after separation.

[0074] In S130, the human voice file is input into a noise reduction model to obtain target human voice.

[0075] Optionally, the noise reduction model can adopt an acoustic-noise-suppression model of ModelScope, which can remove unnecessary noise from audio and improve the clarity of the audio. The human voice file is input into the noise reduction model to further remove residual background noise by the model to obtain the final target human voice.

[0076] In S140, a speech recognition model is used to recognize the initial audio to obtain an audio text.

[0077] In the embodiment of the present application, a FunASR or Whisper speech recognition model can be used to recognize the initial audio after noise suppression processing to obtain the corresponding audio text.

[0078] In S200, a target text is obtained by modifying the audio text, and target audio is synthesized according to the target text and the target human voice.

[0079] Figure 3 is a third schematic diagram of a video content modification method provided by the embodiment of the present application, as shown in the figure, the audio synthesis specifically includes the following steps: Figure 3

[0080] In S210, the audio text is edited and modified to obtain a target text.

[0081] The audio text is modified according to actual conditions to edit a new script to obtain the target text.

[0082] In S220, the target text and the target human voice are input into a voice cloning model for synthesis to obtain target audio.

[0083] ​In the embodiment of the present application, a 1-shot TTS voice cloning model available such as F5-TTS or GPT-Sovits can be selected, the target text and the target voice are input into the voice cloning model, the model generates new voice output according to the timbre and characteristics learned from the target voice and according to the target text, and the target audio is obtained. The 1-shot TTS type of cloning model is more suitable for the scene where the video material is short and compact, so it is only necessary to extract 10-15 seconds of target voice from the original video to realize cloning.

[0084] S300, record a face lip video according to the target audio to obtain a driving video.

[0085] In the embodiment of the present application, the recording of the driving video can let ordinary staff act as a lip actor, and the lip actor can practice the lip according to the target audio for several times. Specifically, the camera is placed in front of the lip actor, and the actor performs the lip while the audio is played, and the driving video is recorded. At least the face of the lip actor is completely recorded to obtain the driving video. If the recorded driving video cannot be completely aligned with the target audio, the lip of the actor can be slightly delayed from the target audio for subsequent audio and picture alignment.

[0086] S400, perform lip calibration according to the driving video and the target audio to obtain a standard driving video.

[0087] The recorded driving video may have some unaligned conditions where the lip lags behind the sound at some time stamps, so lip calibration is needed. Figure 4 is a fourth schematic diagram of a video content modification method provided by the embodiment of the present application, as shown in Figure 4 The lip calibration specifically includes the following steps:

[0088] S410, input the driving video and the target audio into an audio and video synchronization model for matching calculation to obtain a synchronization score.

[0089] In the embodiment of the present application, the SyncNet model can be used for matching calculation to obtain the synchronization score. The SyncNet model is a neural network model for judging whether the audio and the video are synchronized, and can measure the synchronization between the lip movement of a person and the corresponding voice. Figure 5 is a fifth schematic diagram of a video content modification method provided by the embodiment of the present application, as shown in Figure 5 The obtaining of the synchronization score specifically includes the following steps:

[0090] S411, convert the driving video and the target audio into a video feature vector and an audio feature vector respectively.

[0091] Specifically, by sampling, filtering and feature extraction of the driving video and target audio, video feature vectors and audio feature vectors representing the content of the video and audio can be extracted.

[0092] S412, respectively, the video feature vector and the audio feature vector are subjected to spatio-temporal feature extraction to obtain video spatio-temporal feature information and audio spatio-temporal feature information.

[0093] In the embodiment of the present application, the SyncNet model extracts spatio-temporal features of the video feature vector and the audio feature vector through two convolutional neural network-based encoders, thereby capturing key information in the driving video and the target audio, such as spectral features of the audio and inter-frame differences of the video, to form the video spatio-temporal feature information and the audio spatio-temporal feature information.

[0094] S413, the correlation between the video spatio-temporal feature information and the audio spatio-temporal feature information is calculated to generate a similarity map.

[0095] According to the video spatio-temporal feature information and the audio spatio-temporal feature information, the similarity between the video feature vector and the audio feature vector can be calculated to generate a similarity map. This similarity can be obtained by calculating the dot product, cosine similarity or other measurement methods between the feature vectors. The similarity map is a two-dimensional matrix reflecting the matching degree of the video and audio data at different time offsets.

[0096] S414, the similarity map is subjected to max-pooling processing to obtain a time offset.

[0097] Specifically, the max-pooling processing selects the maximum value in a local area of the similarity map, which usually corresponds to the time point with the highest matching degree between the audio and video features. By sliding a window over the entire similarity map and applying the max-pooling operation, a series of candidate time offsets can be determined. The offset with the highest similarity (i.e. the maximum value after max-pooling) is selected as the final time offset, which represents the relative delay or advance between the audio and video, and can be used to adjust the playback time of the audio and video data to achieve synchronization.

[0098] S415, a synchronization score is generated according to the time offset.

[0099] The synchronization score is usually a numerical value representing the audio-video synchronization confidence, and the higher the numerical value, the more synchronized the audio and video are. The synchronization score can include information such as time offset, minimum distance, and confidence score, wherein the time offset of the audio relative to the video reflects the synchronization error between the audio and video; the lower the minimum distance value of the features of the audio and video at different time offsets, the higher the matching degree of the audio and video, that is, the more synchronized; and the confidence score of the audio-video synchronization reflects the degree of trust of the SyncNet model for the audio-video synchronization result, and the higher the numerical value, the more certain the SyncNet model is that the audio and video are synchronized.

[0100] S420, according to the synchronization score, frame-by-frame calibration is performed on the driving video to obtain the best audio timestamp corresponding to each frame of video.

[0101] In the embodiments of the present application, the output synchronization score of the SyncNet model can be used to perform frame-by-frame calibration on the video frames, so as to find the best audio timestamp corresponding to each frame of video. By comparing the mouth shape in the video frame with the timestamp of the audio signal, it can be identified which video frames lag behind the audio and which video frames are redundant.

[0102] S430, adjusting the driving video according to the best audio timestamp and the mouth shape of the driving video to obtain a standard driving video.

[0103] Specifically, for the lagging video frames, they need to be advanced to the position aligned with the best audio timestamp, which usually involves cutting and rearranging the video. For the redundant video frames, they need to be deleted to ensure the compactness and continuity of the video content. The synchronization problem between the obtained standard driving video mouth shape and the sound is effectively solved, thereby improving the viewing experience of the video.

[0104] S500, inputting the standard driving video and the original video into an implicit key point driving model to obtain a synthesized video.

[0105] In the embodiments of the present application, the LivePortrait framework and the Face_vid2vid framework are used as the basic framework to build the main model for generating new facial expression videos, that is, the implicit key point driving model. The LivePortrait framework can convert a static photo into a realistic dynamic video, and the Face_vid2vid framework can migrate the head speaking action of a person in a video to a person in a source image to generate a motion video of the person in the source image by using a single target person source image and an arbitrary head speaking action video.

[0106] Specifically, the implicit key point of the implicit key point driven model is a k group vector representation in the hidden layer, not a three-dimensional space key point in the traditional face mark. By using a standard driving video, the head and face changes of the lip actor in the standard driving video are migrated to the target person in the original video to realize the reproduction of the lip, expression and head posture. Figure 6 is a sixth schematic diagram of a video content modification method provided by an embodiment of the present application, Figure 7 is a schematic diagram of an implicit key point driven model in an embodiment of the present application, as shown in Figure 6 and Figure 7 As shown in the figure, the video synthesis includes the following steps:

[0107] S510, obtaining each driving picture of a standard driving video and each original picture of an original video.

[0108] In the embodiment of the present application, the pictures are extracted frame by frame from the standard driving video and the original video, and the resolution of the obtained driving picture and the original picture is 512x512, so that the resolution of the synthesized image of the two can be effectively improved.

[0109] S520, encoding the driving picture and the original picture from the pixel space to the hidden space to obtain a hidden space driving feature map and a hidden space original feature map.

[0110] In the embodiment of the present application, the input end of the implicit key point driven model is provided with a variational encoder model image encoder with frozen parameters, and the variational encoder image encoder model maps high-dimensional data (such as images) to a low-dimensional latent space (hidden space). Since the pixel space usually has high dimensionality, training directly in the pixel space requires processing a large amount of data, and after reducing the image dimension to the hidden space, training can be performed in a lower-dimensional space, thereby reducing the amount of calculation. Training in the hidden space can converge to the optimal solution more quickly, because the data points in the hidden space are usually more compact and dense, so the training process can find the internal structure of the data more quickly. Sampling and decoding in the hidden space can generate new image samples, and since the hidden space is continuous, new samples between known samples can be generated by interpolation and other methods. Freezing parameters means that the parameters of these input models will not be updated during the training process, which can maintain the feature extraction capability while reducing the amount of calculation in the training process.

[0111] S530, generating a hidden space synthesis feature map according to the hidden space driving feature map and the hidden space original feature map.

[0112] Figure 8 is a seventh schematic diagram of a video content modification method provided by an embodiment of the present application, as shown in Figure 8 The generation of the hidden space synthesis feature map specifically includes the following steps:

[0113] S531, extract the appearance feature of the target person in the hidden space original feature map to obtain a target appearance feature.

[0114] In the embodiment of the application, the appearance feature extractor is used to extract the appearance feature of the target person in the hidden space original feature map, and the appearance feature extractor can automatically extract the representative appearance feature of the target person from the image data. The appearance feature extractor can be a pre-trained deep learning model, or can use image processing algorithms such as SIFT algorithm and SURF algorithm.

[0115] S532, respectively extract the action feature of the target person in the hidden space original feature map and the hidden space driving feature map to obtain original target action features and driving target action features.

[0116] In the embodiment of the application, the action feature extractor is used to extract the action feature of the target person, and the action feature extractor extracts the action feature of the feature map by integrating multiple modules, including a standard key point extraction module, a head pose estimation module, and an expression deformation estimation module.

[0117] The standard key point extraction module is the basic part of the action feature extractor, which is responsible for extracting key feature points from the input feature map. These feature points are usually a set of points in the image that have stability and distinction, such as corner points, edge points or interest points. By defining detection criteria, this module can accurately identify and extract these key points, providing a basis for subsequent feature description and recognition.

[0118] The head pose estimation module is an important component of the action feature extractor, which is used to estimate the attitude angle of the head, including the pitch angle, the yaw angle and the roll angle. Optionally, the module is implemented by the following steps: first, the key points of the face, such as the eye corners and the mouth corners, are detected from the input feature map. Then, the detected key points are matched with a pre-defined 3D face model to obtain the 3D spatial position of the key points. Next, the attitude matrix of the head relative to the camera is obtained by solving the transformation relationship between the 3D points and the corresponding key points. Finally, the pitch angle, the yaw angle and the roll angle are solved according to the rotation matrix, thereby completing the estimation of the head pose.

[0119] The expression deformation estimation module is another key part of the action feature extractor, which is used to estimate the deformation of the facial expression. Optionally, the module is implemented by the following steps: first, the feature points of the facial organs, the texture regions and the pre-defined feature points are located and extracted from the input feature map. Then, the facial expression features are extracted using feature extraction methods such as global method, local method and deep learning model. Next, the deformation of the facial expression is estimated using machine learning or deep learning algorithms according to the extracted expression features. Finally, the parameters or feature vectors of the expression deformation are output for subsequent recognition, analysis or synthesis tasks.

[0120] The action feature extractor can comprehensively and accurately extract the original target action features and the driving target action features by integrating multiple modules such as a standard key point extraction module, a head posture estimation module and an expression deformation estimation module.

[0121] S533, encapsulating the target appearance features, the original target action features and the driving target action features to obtain an encapsulated feature map.

[0122] In the embodiment of the present application, the deformation field estimator is used for feature encapsulation, and the deformation field estimator is used to predict the deformation field of the input target appearance features, the original target action features and the driving target action features, and the deformation field represents a vector field representing the displacement of each pixel or point of an image or object during the deformation process. According to the predicted deformation field, the target appearance features, the original target action features and the driving target action features are encapsulated to obtain an encapsulated feature map.

[0123] S534, decoding the encapsulated feature map to obtain a latent space synthesis feature map.

[0124] Optionally, the deformed features in the encapsulated feature map are decoded by the decoder to generate a latent space synthesis feature map, and the latent space synthesis feature map is a picture frame modified from the driven picture.

[0125] S540, decoding the latent space synthesis feature map from the latent space to the pixel space to obtain each frame of the synthesized picture.

[0126] In the embodiment of the present application, the output end of the implicit key point driving model is provided with a variational encoder model image decoder with frozen parameters, and the variational encoder model image decoder decodes the latent space synthesis feature map back to the pixel space to obtain each frame of the synthesized picture, and the resolution of the synthesized picture is also 512x512.

[0127] S550, generating a synthesized video according to the synthesized picture and the target audio.

[0128] Optionally, a video editing tool is used to set parameters such as frame rate, resolution and encoding format of the video according to needs, and the synthesized picture and the target audio are imported to generate a synthesized video.

[0129] The method for modifying the content of speech of a character in a video provided by the embodiment of the present application can be used to perform low-cost and rapid secondary editing without the need for re-shooting, keep the timbre unchanged during the modification process, and synthesize a mouth shape that matches the new content. The cloned voice technology is used to generate speech similar to the voice of the target speaker, and at the same time, the natural mouth shape, expression and head posture are provided by recording the face driving video, so that the natural mouth shape change of the actor can be captured. The post-adjustment capability allows fine tuning of details, which makes the generated video more realistic and smooth in terms of mouth shape, and ensures that the final video effect is more in line with expectations.

[0130] Embodiment 2

[0131] Based on the same technical concept as the method embodiment of embodiment 1, the embodiment of the present application provides a video content modification device, Figure 9 is a structural schematic diagram of the video content modification device provided by the embodiment of the present application, as Figure 9 shown, the video content modification device 900 comprises:

[0132] The audio extraction module 910 is configured to extract the initial audio of the target character in the original video, and extract the target voice and audio text according to the initial audio.

[0133] The audio synthesis module 920 is configured to modify the audio text to obtain the target text, and synthesize the target audio according to the target text and the target voice.

[0134] The video recording module 930 is configured to record the face mouth shape video according to the target audio, and obtain the driving video.

[0135] The video calibration module 940 is configured to calibrate the mouth shape according to the driving video and the target audio, and obtain the standard driving video.

[0136] The video synthesis module 950 is configured to input the standard driving video and the initial video into the implicit key point driving model, and obtain the synthesis video.

[0137] The video content modification device provided by the embodiment of the present application uses cloned voice technology to generate speech similar to the voice of the target speaker, and at the same time, natural mouth shape, expression and head posture are provided by recording the face driving video. The driving video and the audio are inferred and trained in the latent space, and the coding and decoding capabilities of the variational encoder are used to significantly improve the computing efficiency and improve the speed and quality of video generation. The audio-video synchronization technology ensures the accurate synchronization between the picture and the audio, and even if there is a slight time difference during the recording process, it can be adjusted to ensure the quality of the final video.

[0138] It can be understood that the implementation manners of the video content modification method described in Embodiment 1 above are also applicable to this embodiment and can achieve the same technical effects, and thus will not be described herein again.

[0139] Embodiment 3

[0140] Based on the same concept, the embodiment of the present application also provides an electronic device, Figure 10 is a structural schematic diagram of an electronic device provided by the embodiment of the present application, as Figure 10 shown, the electronic device 1000 can include a processor 1010, a communications interface 1020, a memory 1030 and a communications bus 1040, wherein the processor 1010, the communications interface 1020 and the memory 1030 complete mutual communication through the communications bus 1040. The processor 1010 can invoke the logical instructions in the memory 1030 to execute the steps of the video content modification method as described in the above embodiments. For example, it includes:

[0141] S100, extracting the initial audio of the target person in the original video, extracting the target voice and audio text according to the initial audio;

[0142] S200, modifying the audio text to obtain the target text, and synthesizing the target audio according to the target text and the target voice;

[0143] S300, recording the facial lip movement video according to the target audio to obtain the driving video;

[0144] S400, performing lip movement calibration according to the driving video and the target audio to obtain the standard driving video;

[0145] S500, inputting the standard driving video and the original video into the implicit key point driving model to obtain the synthesized video.

[0146] The processor 1010 can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. chips, or a combination of the above various types of chips.

[0147] Moreover, the logic instructions in the memory 1030 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0148] The memory 1030 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0149] Embodiment 4

[0150] Based on the same idea, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program includes at least one code, which can be executed by a host device to control the host device to implement the steps of the video content modification method as described in the above embodiments. For example, it includes:

[0151] S100, extracting an initial audio of a target person in an original video, extracting a target voice and an audio text according to the initial audio;

[0152] S200, modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and the target voice;

[0153] S300, recording a face lip video according to the target audio to obtain a driving video;

[0154] S400, performing lip calibration according to the driving video and the target audio to obtain a standard driving video;

[0155] S500, input the standard driving video and the original video into the implicit key point driving model to obtain a synthesized video.

[0156] Based on the same technical concept, the embodiment of the application further provides a computer program for implementing the above-mentioned method embodiment when the computer program is executed by a host device.

[0157] The computer program can be stored entirely or partially on a computer readable storage medium packaged together with the processor, or partially or entirely on a memory not packaged together with the processor.

[0158] Based on the same technical concept, the embodiment of the application further provides a processor for implementing the above-mentioned method embodiment. The processor can be a chip.

[0159] In summary, the video content modification method, device, electronic equipment and storage medium provided by the application use cloning sound technology to generate speech similar to the voice of the target speaker, and at the same time provide natural lip shapes, expressions and head poses through recording face-driven videos, which can capture the natural changes in the actor's lip shapes. The post-adjustment capability allows fine-tuning of details, which makes the generated video more realistic and smooth in terms of lip shapes, ensuring that the final video effect is more in line with expectations. The driving video and audio are inferred and trained within the latent space, using the encoding and decoding capabilities of the variational encoder, which significantly improves the computational efficiency and improves the speed and quality of video generation. The audio-video synchronization technology ensures accurate synchronization between the picture and the audio, and even if there is a slight time difference during recording, it can be adjusted to ensure the quality of the final video.

[0160] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described herein can be combined with other embodiments.

[0161] The above-described embodiments only express several implementation manners of the application, and the description is more specific and detailed, but it should not be understood as a limitation on the patent scope of the application. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which are all within the protection scope of the application. Therefore, the protection scope of the patent of the application should be subject to the appended claims.

[0162] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for modifying video content, characterized in that, The method includes: Extract the initial audio of the target person from the original video, and extract the target voice and audio text based on the initial audio; Modify the audio text to obtain the target text, and synthesize the target audio based on the target text and the target human voice; Record facial lip-sync video based on the target audio to obtain driving video; Lip-sync calibration is performed based on the driving video and the target audio to obtain a standard driving video; The standard driving video and the original video are input into the implicit keypoint driving model to obtain a synthesized video, including: Acquire each frame of the standard driven video and each frame of the original video; The driving image and the original image are encoded from the pixel space to the latent space to obtain the latent space driving feature map and the latent space original feature map; A latent space composite feature map is generated based on the latent space driving feature map and the latent space original feature map; The latent space synthesized feature map is decoded from the latent space to the pixel space to obtain each frame of synthesized image; The synthesized video is generated based on the synthesized image and the target audio; The step of generating a latent space composite feature map based on the latent space driving feature map and the latent space original feature map includes: Extract the appearance features of the target person from the original feature map of the latent space to obtain the target appearance features; The action features of the target person in the original latent space feature map and the driving latent space feature map are extracted respectively to obtain the original target action features and the driving target action features; The target appearance features, the original target action features, and the driving target action features are encapsulated to obtain an encapsulated feature map; The encapsulated feature map is decoded to obtain the latent space synthesized feature map; The step of encapsulating the target appearance features, the original target action features, and the driving target action features to obtain an encapsulated feature map includes: The deformation field is predicted by the deformation field estimator for the target appearance features, the original target motion features and the driving target motion features, wherein the deformation field is a vector field of displacement of each pixel or point of the image or object during the deformation process; The target appearance features, the original target motion features, and the driven target motion features are encapsulated based on the deformation field to obtain an encapsulated feature map.

2. The video content modification method according to claim 1, characterized in that, The step of extracting the initial audio of the target person from the original video, and extracting the target voice and audio text based on the initial audio, includes: The initial audio of the target person in the original video is extracted using editing tools; The initial audio is input into the human voice separation technology model to obtain the human voice file; The voice file is input into the noise reduction model to obtain the target voice. The initial audio is identified using a speech recognition model to obtain the audio text.

3. The video content modification method according to claim 1, characterized in that, The step of modifying the audio text to obtain the target text, and synthesizing the target audio based on the target text and the target human voice, includes: The audio text is edited and modified to obtain the target text; The target text and the target human voice input sound clone model are synthesized to obtain the target audio.

4. The video content modification method according to claim 1, characterized in that, The step of performing lip-sync calibration based on the driving video and the target audio to obtain a standard driving video includes: The driving video and the target audio input audio-video synchronization model are matched and calculated to obtain a synchronization score; The driving video is calibrated frame by frame based on the synchronization score to obtain the best audio timestamp for each frame of video. The standard driving video is obtained by adjusting the driving video according to the optimal audio timestamp and the lip movements of the driving video.

5. The video content modification method according to claim 4, characterized in that, The step of matching and calculating the synchronization score between the driving video and the target audio input audio-video synchronization model includes: The driving video and the target audio are respectively converted into video feature vectors and audio feature vectors; Spatiotemporal feature extraction is performed on the video feature vector and the audio feature vector respectively to obtain video spatiotemporal feature information and audio spatiotemporal feature information; Calculate the correlation between the video spatiotemporal feature information and the audio spatiotemporal feature information, and generate a similarity map; Max pooling is performed on the similarity map to obtain the time offset; The synchronization score is generated based on the time offset.

6. A video content modification device, characterized in that, The device includes: The audio extraction module is used to extract the initial audio of the target person in the original video, and extract the target voice and audio text based on the initial audio; An audio synthesis module is used to modify the audio text to obtain the target text, and to synthesize the target audio based on the target text and the target human voice; The video recording module is used to record facial lip-sync video based on the target audio to obtain driving video; The video calibration module is used to perform lip-sync calibration based on the driving video and the target audio to obtain a standard driving video; The video synthesis module is used to input the standard driving video and the original video into the implicit keypoint driving model to obtain a synthesized video, including: Acquire each frame of the standard driven video and each frame of the original video; The driving image and the original image are encoded from the pixel space to the latent space to obtain the latent space driving feature map and the latent space original feature map; A latent space composite feature map is generated based on the latent space driving feature map and the latent space original feature map; The latent space synthesized feature map is decoded from the latent space to the pixel space to obtain each frame of synthesized image; The synthesized video is generated based on the synthesized image and the target audio; The step of generating a latent space composite feature map based on the latent space driving feature map and the latent space original feature map includes: Extract the appearance features of the target person from the original feature map of the latent space to obtain the target appearance features; The action features of the target person in the original latent space feature map and the driving latent space feature map are extracted respectively to obtain the original target action features and the driving target action features; The target appearance features, the original target action features, and the driving target action features are encapsulated to obtain an encapsulated feature map; The encapsulated feature map is decoded to obtain the latent space synthesized feature map; The step of encapsulating the target appearance features, the original target action features, and the driving target action features to obtain an encapsulated feature map includes: The deformation field is predicted by the deformation field estimator for the target appearance features, the original target motion features and the driving target motion features, wherein the deformation field is a vector field of displacement of each pixel or point of the image or object during the deformation process; The target appearance features, the original target motion features, and the driven target motion features are encapsulated based on the deformation field to obtain an encapsulated feature map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the video content modification method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video content modification method as described in any one of claims 1 to 5.

Images