Video content modification method and device, electronic equipment and storage medium

Abstract

The invention provides a video content modification method and device, electronic equipment and a storage medium, and relates to the technical field of multimedia, and the method comprises the steps: extracting the initial audio of a target person in an original video, and extracting the sound of the target person and an audio text according to the initial audio; modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and the target human voice; recording a face mouth shape video according to the target audio to obtain a driving video; performing mouth shape calibration according to the driving video and the target audio to obtain a standard driving video; and inputting the standard driving video and the original video into the implicit key point driving model to obtain a synthetic video. According to the method, the natural mouth shape, expression and head posture are provided by recording the face driving video, so that the generated video is more vivid and smoother in the mouth shape aspect, the final video effect is ensured to better conform to the expectation, the driving video and the audio are subjected to reasoning and training in the hidden space, and the video generation speed and quality are improved.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technology, and in particular to a video content modification method, device, electronic equipment and storage medium. Background Art

[0002] In the field of media production and content creation, it is often necessary to adjust or modify the vocal clips in the recorded video. For example, in an interview scene, after completing the interview, the reporter or content creator may find that there are problems with some of the interviewee's words. If these problems are not corrected, they may affect the accuracy and communication effect of the information. However, due to time and resource constraints, it is often impossible to restore the original interview scene for re-recording. In the secondary creation of film and television works, creators often need to intercept clips from the original film and television works and create new videos that are novel and attractive by modifying or re-editing the dialogue content.

[0003] Existing methods usually synthesize audio from the edited text content, and keep the timbre consistent with the original speaker, then use the audio to drive the video synthesis lip shape to complete lip synchronization, and finally cut out the redundant video clips through editing. However, this method can only modify the lip shape, and the eyes, eyebrows, facial expressions and head posture cannot be modified. At the same time, the lip shape synthesis effect of Chinese and side face videos is not good, and some features of the original person's mouth are easily lost. The generated video quality resolution is not high. Therefore, there is an urgent need for a method to solve the problem of poor video synthesis effect when the existing video content is modified. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a video content modification method, device, electronic device and storage medium, which effectively solve the problem of poor synthesized video effect when modifying the existing video content.

[0005] In a first aspect, the present invention provides a method for modifying video content, the method comprising:

[0006] Extracting the initial audio of the target person in the original video, and extracting the target person's voice and audio text according to the initial audio;

[0007] Modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and the target human voice;

[0008] Record the facial lip-sync video according to the target audio to obtain a driving video;

[0009] Perform lip calibration according to the driving video and the target audio to obtain a standard driving video;

[0010] The standard driven video and the original video are input into an implicit key point driven model to obtain a synthesized video.

[0011] Furthermore, the extracting of the initial audio of the target person in the original video, and extracting the target person's voice and audio text according to the initial audio, includes:

[0012] Using an editing tool to extract the initial audio of the target person in the original video;

[0013] Inputting the initial audio into a vocal separation technology model to obtain a vocal file;

[0014] Inputting the human voice file into a noise reduction model to obtain a target human voice;

[0015] The initial audio is recognized by using a speech recognition model to obtain the audio text.

[0016] Furthermore, the step of modifying the audio text to obtain a target text and synthesizing a target audio according to the target text and the target human voice includes:

[0017] Editing and modifying the audio text to obtain a target text;

[0018] The target text and the target human voice are input into a sound cloning model for synthesis to obtain a target audio.

[0019] Further, performing lip calibration according to the driving video and the target audio to obtain a standard driving video includes:

[0020] The driving video and the target audio are input into an audio-video synchronization model for matching calculation to obtain a synchronization score;

[0021] Calibrate the driving video frame by frame according to the synchronization score to obtain the best audio timestamp corresponding to each frame of the video;

[0022] The driving video is adjusted according to the optimal audio timestamp and the lip shape of the driving video to obtain the standard driving video.

[0023] Furthermore, the step of inputting the driving video and the target audio into an audio-video synchronization model for matching calculation to obtain a synchronization score includes:

[0024] Convert the driving video and the target audio into a video feature vector and an audio feature vector respectively;

[0025] Extracting spatiotemporal features from the video feature vector and the audio feature vector respectively to obtain video spatiotemporal feature information and audio spatiotemporal feature information;

[0026] Calculating the correlation between the video spatiotemporal feature information and the audio spatiotemporal feature information to generate a similarity graph;

[0027] Performing maximum pooling processing on the similarity graph to obtain a time offset;

[0028] The synchronization score is generated according to the time offset.

[0029] Furthermore, the step of inputting the standard driving video and the original video into an implicit key point driving model to obtain a synthesized video includes:

[0030] Acquire each frame of the driving picture of the standard driving video and each frame of the original picture of the original video;

[0031] Encoding the driving picture and the original picture from the pixel space to the latent space to obtain a latent space driving feature map and a latent space original feature map;

[0032] Generate a latent space synthetic feature map according to the latent space driving feature map and the latent space original feature map;

[0033] Decoding the latent space synthetic feature map from the latent space to the pixel space to obtain each frame of synthetic picture;

[0034] The synthetic video is generated according to the synthetic picture and the target audio.

[0035] Furthermore, generating a latent space synthetic feature map according to the latent space driving feature map and the latent space original feature map includes:

[0036] Extract the appearance features of the target person in the original feature map of the latent space to obtain the target appearance features;

[0037] Extract the action features of the target person in the latent space original feature map and the latent space driven feature map respectively to obtain the original target action features and the driven target action features;

[0038] Performing feature encapsulation on the target appearance feature, the original target motion feature, and the driven target motion feature to obtain a packaged feature graph;

[0039] The encapsulated feature map is decoded to obtain the latent space synthetic feature map.

[0040] In a second aspect, the present invention provides a video content modification device, the device comprising:

[0041] An audio extraction module is used to extract the initial audio of the target person in the original video, and extract the target person's voice and audio text according to the initial audio;

[0042] An audio synthesis module, used for modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and the target human voice;

[0043] A video recording module, used for recording a facial lip-sync video according to the target audio to obtain a driving video;

[0044] A video calibration module, used for performing lip calibration according to the driving video and the target audio to obtain a standard driving video;

[0045] The video synthesis module is used to input the standard driving video and the original video into the implicit key point driving model to obtain a synthesized video.

[0046] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video content modification method as described in the first aspect of the present invention.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video content modification method as described in the first aspect of the present invention.

[0048] The video content modification method, device, electronic device and storage medium provided by the present invention use cloning sound technology to generate speech similar to the target speaker's voice, and at the same time provide natural lip shape, expression and head posture by recording facial driving video, which can capture the natural lip shape changes of the actor. The post-adjustment capability allows fine-tuning of details, which makes the generated video more realistic and smooth in terms of lip shape, ensuring that the final video effect is more in line with expectations. The driving video and audio are inferred and trained in the latent space, and the encoding and decoding capabilities of the variational encoder are utilized to significantly improve the computing efficiency, improve the speed and quality of video generation. The use of audio and video synchronization technology ensures the precise synchronization between the picture and the audio, and can be adjusted even if there is a slight time difference during the recording process to ensure the quality of the final video. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a first schematic diagram of the video content modification method provided by an embodiment of the present invention;

[0051] Figure 2 is a second schematic diagram of the video content modification method process provided by an embodiment of the present invention;

[0052] Figure 3 is a third schematic diagram of the video content modification method provided by an embodiment of the present invention;

[0053] Figure 4 is a fourth schematic diagram of the video content modification method process provided by an embodiment of the present invention;

[0054] Figure 5 is a fifth schematic diagram of the video content modification method provided by an embodiment of the present invention;

[0055] Figure 6 is a sixth schematic diagram of the video content modification method provided by an embodiment of the present invention;

[0056] Figure 7 Schematic diagram of the architecture of the implicit key point driving model in an embodiment of the present invention;

[0057] Figure 8 is a seventh schematic diagram of the video content modification method process provided by an embodiment of the present invention;

[0058] Fig. 9 is a schematic diagram of the structure of a video content modification device provided by an embodiment of the present invention;

[0059] Fig.10 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention.

[0060] Description of main component symbols:

[0061] 900, video content modification device; 910, audio extraction module; 920, audio synthesis module; 930, video recording module; 940, video calibration module; 950, video synthesis module; 1000, electronic device; 1010, processor; 1020, communication interface; 1030, memory; 1040, communication bus. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0063] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0065] In the field of media production and content creation, it is often necessary to adjust or modify the vocal clips in the recorded video. Existing methods usually synthesize the edited text content into audio, and keep it consistent with the original speaker's timbre, then use the audio to drive the video synthesis lip shape to complete lip synchronization, and finally cut out the redundant video clips through editing. However, this method can only modify the lip shape, and the eyes, eyebrows, facial expressions and head posture cannot be modified. At the same time, the lip shape synthesis effect of Chinese and side face videos is not good, and it is easy to lose some features of the original person's mouth, and the generated video quality resolution is not high.

[0066] Example 1

[0067] The embodiment of the present invention provides a video content modification method, which effectively solves the problem of poor synthesized video effect when modifying the existing video content. Figure 1 is a first schematic diagram of the video content modification method provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0068] S100, extracting the initial audio of the target person in the original video, and extracting the target person's voice and audio text according to the initial audio.

[0069] Figure 2 is a second schematic diagram of the video content modification method provided by an embodiment of the present invention. Figure 2 As shown, audio extraction specifically includes the following steps:

[0070] S110, extracting the initial audio of the target person in the original video using an editing tool.

[0071] In the embodiment of the present invention, the video clips to be modified in the original video can be directly edited by using editing tools as the original video material. Then, the audio part of the target person speaking in the original video material to be modified is extracted as the initial audio, and the audio length of the initial audio can be 10 to 15 seconds, which can be set according to the actual situation.

[0072] S120: Input the initial audio into a vocal separation technology model to obtain a vocal file.

[0073] Optionally, the human voice separation technology model may adopt the UVR5 model, which is capable of separating human voice and background music or noise from mixed audio. The initial audio is input into the UVR5 model, and the model is allowed to separate the human voice and background noise to obtain the separated human voice file.

[0074] S130: Input the human voice file into the noise reduction model to obtain the target human voice.

[0075] Optionally, the noise reduction model can use the acoustic-noise-suppression model of ModelScope, which can remove unwanted noise from the audio and improve the clarity of the audio. Input the human voice file into the noise reduction model, and let the model further remove the residual background noise to obtain the final target human voice.

[0076] S140: Using a speech recognition model to recognize the initial audio to obtain an audio text.

[0077] In the embodiment of the present invention, a speech recognition model such as FunASR or Whisper may be used to recognize the initial audio after noise suppression processing to obtain the corresponding audio text.

[0078] S200, modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and a target voice.

[0079] Figure 3 is a third schematic diagram of the video content modification method provided by an embodiment of the present invention. Figure 3 As shown, audio synthesis specifically includes the following steps:

[0080] S210: Edit and modify the audio text to obtain a target text.

[0081] Modify the audio text according to the actual situation, edit a new copy, and obtain the target text.

[0082] S220: Input the target text and the target human voice into the sound cloning model for synthesis to obtain the target audio.

[0083] In the embodiment of the present invention, an available 1-shot TTS voice cloning model such as F5-TTS or GPT-Sovits can be selected, and the target text and the target voice can be input into the voice cloning model, so that the model can generate a new voice output based on the timbre and features learned from the target voice and the target text to obtain the target audio. The 1-shot TTS type cloning model is more suitable for scenes with short and compact video materials, so cloning can be achieved by extracting only 10 to 15 seconds of the target voice from the original video.

[0084] S300, recording a facial lip-sync video according to the target audio to obtain a driving video.

[0085] In an embodiment of the present invention, the recording of the driving video can allow ordinary staff to act as lip sync actors, and let the lip sync actors listen to the target audio several times to practice lip syncing. Specifically, a camera is placed in front of the lip sync actor, and then the actor is asked to lip sync and record while playing the audio, and at least the face of the lip sync actor is recorded completely to obtain the driving video. If the recorded driving video cannot be completely aligned with the target audio, the actor's lip sync can be slightly lagged behind the target audio to facilitate subsequent audio and video alignment.

[0086] S400: Perform lip calibration according to the driving video and the target audio to obtain a standard driving video.

[0087] The recorded driving video may have some timestamps where the lip movement lags behind the sound, which is visible to the naked eye, so lip alignment is required. Figure 4 4 is a schematic diagram of the video content modification method provided by an embodiment of the present invention. Figure 4 As shown, lip calibration specifically includes the following steps:

[0088] S410: Input the driving video and the target audio into an audio and video synchronization model for matching calculation to obtain a synchronization score.

[0089] In an embodiment of the present invention, a SyncNet model may be used to perform matching calculations to obtain a synchronization score. The SyncNet model is a neural network model used to determine whether audio and video are synchronized, and may measure the synchronization between human lip movements and corresponding speech. Figure 5 5 is a fifth schematic diagram of the video content modification method provided by an embodiment of the present invention. Figure 5 As shown, obtaining the synchronization score specifically includes the following steps:

[0090] S411, converting the driving video and the target audio into a video feature vector and an audio feature vector respectively.

[0091] Specifically, by performing steps such as sampling, filtering and feature extraction on the driving video and the target audio, a video feature vector and an audio feature vector that can represent the video and audio contents can be extracted.

[0092] S412: Extract spatiotemporal features from the video feature vector and the audio feature vector respectively to obtain video spatiotemporal feature information and audio spatiotemporal feature information.

[0093] In an embodiment of the present invention, the SyncNet model extracts spatiotemporal features of the video feature vector and the audio feature vector respectively through two encoders based on convolutional neural networks, thereby capturing key information in the driving video and the target audio, such as the spectral characteristics of the audio and the frame differences of the video, to form video spatiotemporal feature information and audio spatiotemporal feature information.

[0094] S413: Calculate the correlation between the video spatiotemporal feature information and the audio spatiotemporal feature information to generate a similarity graph.

[0095] Based on the video spatiotemporal feature information and the audio spatiotemporal feature information, the similarity between the video feature vector and the audio feature vector can be calculated to generate a similarity graph. This similarity can be obtained by calculating the dot product, cosine similarity or other measurement methods between the feature vectors. The similarity graph is a two-dimensional matrix that reflects the matching degree of the video and audio data at different time offsets.

[0096] S414: Perform maximum pooling processing on the similarity graph to obtain a time offset.

[0097] Specifically, the maximum pooling process selects the maximum value in a local area of ​​the similarity graph, which usually corresponds to the time point that best matches the audio and video features. By sliding a window across the entire similarity graph and applying the maximum pooling operation, a series of candidate time offsets can be determined. The offset with the highest similarity (i.e., the maximum value after maximum pooling) is selected as the final time offset, which represents the relative delay or advance between audio and video, and can be used to adjust the playback time of audio and video data to achieve synchronization.

[0098] S415: Generate a synchronization score according to the time offset.

[0099] The synchronization score is usually a value that represents the confidence of audio and video synchronization. The higher the value, the more synchronized the audio and video are. The synchronization score can include information such as time offset, minimum distance, and confidence score. The time offset of audio relative to video reflects the synchronization error between audio and video; the lower the minimum distance value of audio and video features at different time offsets, the higher the matching degree of audio and video, that is, the more synchronized; the confidence score of audio and video synchronization reflects the degree of trust of the SyncNet model in the audio and video synchronization results. The higher the value, the more certain the SyncNet model is that the audio and video are synchronized.

[0100] S420: Calibrate the driving video frame by frame according to the synchronization score to obtain the best audio timestamp corresponding to each frame of the video.

[0101] In the embodiment of the present invention, the output synchronization score of the SyncNet model can be used to calibrate the video frames frame by frame, so as to find the best audio timestamp corresponding to each frame of video. By comparing the lip shape in the video frame with the timestamp of the audio signal, it is possible to identify which video frames lag behind the audio and which video frames are redundant.

[0102] S430: Adjust the driving video according to the best audio timestamp and the lip shape of the driving video to obtain a standard driving video.

[0103] Specifically, the lagging video frames need to be advanced to a position aligned with the best audio timestamp, which usually involves cutting and rearranging the video. The redundant video frames need to be deleted to ensure the compactness and coherence of the video content. The synchronization problem between the standard-driven video lip shape and the sound is effectively solved, thereby improving the video viewing experience.

[0104] S500, input the standard driving video and the original video into the implicit key point driving model to obtain a synthesized video.

[0105] In the embodiment of the present invention, the LivePortrait framework and the Face_vid2vid framework are used as the basic frameworks to construct the main model of video-driven generation of new facial expression videos, that is, the implicit key point driven model. The LivePortrait framework can convert static photos into realistic dynamic videos, and the Face_vid2vid framework uses a single target person source image and an arbitrary head speaking action video to migrate the head speaking action of the person in the video to the person in the source image, thereby generating a motion video of the person in the source image.

[0106] Specifically, the implicit key points of the implicit key point driven model are represented by k groups of vectors in the hidden layer, rather than the three-dimensional spatial key points in traditional face marking. By using a standard driving video, the head and facial changes of the lip sync actor in the video are transferred to the target person in the original video, thereby achieving the reproduction of lip shape, expression and head posture. Figure 6 is a sixth schematic diagram of the video content modification method provided by an embodiment of the present invention, Figure 7 is a schematic diagram of the architecture of the implicit key point driving model in an embodiment of the present invention, such as Figure 6 and Figure 7 As shown, video synthesis includes the following steps:

[0107] S510: Acquire each frame of a driving picture of a standard driving video and each frame of an original picture of an original video.

[0108] In the embodiment of the present invention, pictures are extracted frame by frame from the standard driving video and the original video, and the resolution of the obtained driving pictures and original pictures is 512x512, thereby effectively improving the resolution of the image synthesized by the two.

[0109] S520, encoding the driving image and the original image from the pixel space to the latent space to obtain a latent space driving feature map and a latent space original feature map.

[0110] In an embodiment of the present invention, a variational encoder model image encoder with frozen parameters is set at the input end of the implicit key point driving model. The variational encoder image encoder model maps high-dimensional data (such as images) to a low-dimensional potential space (hidden space). Since the pixel space is usually high-dimensional, training directly in the pixel space requires processing a large amount of data, and after reducing the image to the hidden space, it can be trained in a lower-dimensional space, thereby reducing the amount of calculation. Training in the hidden space can converge to the optimal solution faster, because the data points in the hidden space are usually more compact and dense, so the training process can find the intrinsic structure of the data faster. Sampling and decoding in the hidden space can generate new image samples. Since the hidden space is continuous, new samples between known samples can be generated by interpolation and other methods. Freezing parameters means that the parameters of these input models will not be updated during the training process, which can maintain its feature extraction capabilities while reducing the amount of calculation during the training process.

[0111] S530, generating a latent space synthetic feature map according to the latent space driving feature map and the latent space original feature map.

[0112] Figure 8 7 is a seventh schematic diagram of the video content modification method provided by an embodiment of the present invention. Figure 8 As shown in Figure 2, the generation of latent space synthetic feature maps specifically includes the following steps:

[0113] S531, extracting the appearance features of the target person in the original feature map of the latent space to obtain the target appearance features.

[0114] In an embodiment of the present invention, an appearance feature extractor is used to extract the appearance features of the target person in the original feature map of the latent space. The appearance feature extractor can automatically extract representative appearance features of the target person from the image data. The appearance feature extractor can be a pre-trained deep learning model, or it can use image processing algorithms such as SIFT algorithm and SURF algorithm.

[0115] S532, respectively extracting the action features of the target person in the latent space original feature map and the latent space driving feature map to obtain the original target action features and the driving target action features.

[0116] In an embodiment of the present invention, an action feature extractor is used to extract the action features of the target person. The action feature extractor extracts the action features of the feature graph by integrating multiple modules, including a standard key point extraction module, a head posture estimation module and an expression deformation estimation module.

[0117] The standard key point extraction module is the basic part of the action feature extractor, which is responsible for extracting key feature points from the input feature map. These feature points are usually stable and distinctive point sets in the image, such as corner points, edge points or interest points. By defining the detection criteria, the module can accurately identify and extract these key points, providing a basis for subsequent feature description and recognition.

[0118] The head posture estimation module is an important part of the motion feature extractor. It is used to estimate the posture angles of the head, including pitch, yaw and roll. Optionally, the module is implemented by the following steps: First, the key points of the face, such as the corners of the eyes and the corners of the mouth, are detected from the input feature map. Then the detected key points are matched with the predefined 3D face model to obtain the 3D spatial position of the key points. Then, by solving the conversion relationship between the 3D points and the corresponding key points, the posture matrix of the head relative to the camera is obtained. Finally, the pitch, yaw and roll angles are solved according to the rotation matrix to complete the estimation of the head posture.

[0119] The expression deformation estimation module is another key part of the action feature extractor, which is used to estimate the deformation of facial expressions. Optionally, this module is implemented by the following steps: first, the facial organ features, texture areas and predefined feature points are located and extracted from the input feature map. Then, feature extraction methods such as the overall method, local method and deep learning model are used to extract facial expression features. Then, based on the extracted expression features, machine learning or deep learning algorithms are used to estimate the deformation of facial expressions. Finally, the expression deformation parameters or feature vectors are output for subsequent recognition, analysis or synthesis tasks.

[0120] The action feature extractor integrates multiple modules such as the standard key point extraction module, the head posture estimation module and the expression deformation estimation module, and can comprehensively and accurately extract the original target action features and the driving target action features.

[0121] S533, encapsulating the target appearance features, the original target motion features, and the driven target motion features to obtain an encapsulated feature graph.

[0122] In the embodiment of the present invention, a deformation field estimator is used for feature encapsulation, and the deformation field estimator is used to predict the deformation field of the input target appearance features, original target motion features, and driven target motion features, where the deformation field represents a vector field representing the displacement of each pixel or point of an image or object during the deformation process. According to the predicted deformation field, the target appearance features, original target motion features, and driven target motion features are encapsulated to obtain an encapsulated feature map.

[0123] S534. Decode the encapsulated feature map to obtain a latent space synthetic feature map.

[0124] Optionally, the deformed features in the encapsulated feature map are decoded by a decoder to generate a latent space synthetic feature map, where the latent space synthetic feature map is a picture frame modified by the driven picture.

[0125] S540, decoding the latent space synthesis feature map from the latent space to the pixel space to obtain each frame of the synthesized picture.

[0126] In an embodiment of the present invention, a variational encoder model image decoder with frozen parameters is set at the output end of the implicit key point driving model. The variational encoder model image decoder decodes the latent space synthetic feature map back to the pixel space, thereby obtaining each frame of the synthetic picture, and the resolution of the synthetic picture is also 512×512.

[0127] S550: Generate a synthesized video according to the synthesized image and the target audio.

[0128] Optionally, a video editing tool is used to set parameters such as the frame rate, resolution, and encoding format of the video as required, and the synthesized image and target audio are imported to generate a synthesized video.

[0129] The video content modification method provided by the embodiment of the present invention is used to modify the content of the speech of the characters in the video, perform low-cost and fast secondary editing without reshooting, keep the timbre unchanged during the modification process, and synthesize the lip shape that matches the new content. The voice cloning technology is used to generate speech similar to the voice of the target speaker, and the natural lip shape, expression and head posture are provided by recording facial-driven video, which can capture the natural lip shape changes of the actor. The post-adjustment capability allows fine-tuning of details, which makes the generated video more realistic and smooth in terms of lip shape, ensuring that the final video effect is more in line with expectations.

[0130] Example 2

[0131] Based on the same technical concept as the method embodiment of the above embodiment 1, the embodiment of the present invention provides a video content modification device, Fig. 9 is a schematic diagram of the structure of a video content modification device provided by an embodiment of the present invention, such as Fig. 9 As shown, the video content modification device 900 includes:

[0132] The audio extraction module 910 is used to extract the initial audio of the target person in the original video, and extract the target person's voice and audio text based on the initial audio.

[0133] The audio synthesis module 920 is used to modify the audio text to obtain the target text, and synthesize the target audio according to the target text and the target voice.

[0134] The video recording module 930 is used to record the facial lip-sync video according to the target audio to obtain the driving video.

[0135] The video calibration module 940 is used to perform lip calibration according to the driving video and the target audio to obtain a standard driving video.

[0136] The video synthesis module 950 is used to input the standard driving video and the initial video into an implicit key point driving model to obtain a synthesized video.

[0137] The video content modification device provided by the embodiment of the present invention uses the sound cloning technology to generate speech similar to the target speaker's voice, and at the same time provides natural lip shape, expression and head posture by recording facial driving video. The driving video and audio are inferred and trained in the latent space, and the encoding and decoding capabilities of the variational encoder are utilized to significantly improve the computational efficiency and the speed and quality of video generation. The use of audio and video synchronization technology ensures the precise synchronization between the picture and the audio, and can be adjusted even if there is a slight time difference during the recording process to ensure the quality of the final video.

[0138] It can be understood that the implementation method of the video content modification method described in the above embodiment 1 is also applicable to this embodiment and can achieve the same technical effect, so it will not be repeated here.

[0139] Example 3

[0140] Based on the same concept, an embodiment of the present invention further provides an electronic device, Fig.10 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig.10 As shown, the electronic device 1000 may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the steps of the video content modification method described in the above embodiments. For example, it includes:

[0141] S100, extracting the initial audio of the target person in the original video, and extracting the target person's voice and audio text according to the initial audio;

[0142] S200, modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and a target voice;

[0143] S300, recording the facial lip shape video according to the target audio to obtain a driving video;

[0144] S400, performing lip calibration according to the driving video and the target audio to obtain a standard driving video;

[0145] S500, input the standard driving video and the original video into the implicit key point driving model to obtain a synthesized video.

[0146] The processor 1010 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0147] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0148] The memory 1030 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] Example 4

[0150] Based on the same concept, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program, and the computer program includes at least one code segment, which can be executed by a main control device to control the main control device to implement the steps of the video content modification method described in the above embodiments. For example, it includes:

[0151] S100, extracting the initial audio of the target person in the original video, and extracting the target person's voice and audio text according to the initial audio;

[0152] S200, modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and a target voice;

[0153] S300, recording the facial lip shape video according to the target audio to obtain a driving video;

[0154] S400, performing lip calibration according to the driving video and the target audio to obtain a standard driving video;

[0155] S500, input the standard driving video and the original video into the implicit key point driving model to obtain a synthesized video.

[0156] Based on the same technical concept, an embodiment of the present invention further provides a computer program, which is used to implement the above method embodiment when executed by a main control device.

[0157] The computer program may be stored in whole or in part on a computer-readable storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor.

[0158] Based on the same technical concept, an embodiment of the present invention further provides a processor, which is used to implement the above method embodiment. The above processor can be a chip.

[0159] In summary, the video content modification method, device, electronic device and storage medium provided by the present invention use cloning sound technology to generate speech similar to the target speaker's voice, and at the same time provide natural lip shape, expression and head posture by recording facial driving video, which can capture the natural lip shape changes of the actor. The post-adjustment capability allows fine-tuning of details, which makes the generated video more realistic and smooth in terms of lip shape, ensuring that the final video effect is more in line with expectations. The driving video and audio are reasoned and trained in the latent space, and the encoding and decoding capabilities of the variational encoder are utilized to significantly improve the computational efficiency, improve the speed and quality of video generation. The use of audio and video synchronization technology ensures the precise synchronization between the picture and the audio, and can be adjusted even if there is a slight time difference during the recording process to ensure the quality of the final video.

[0160] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0161] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video content modification method, characterized in that: The method comprises: Extracting the initial audio of the target person in the original video, and extracting the target person's voice and audio text according to the initial audio; Modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and the target human voice; Record the facial lip-sync video according to the target audio to obtain a driving video; Perform lip calibration according to the driving video and the target audio to obtain a standard driving video; The standard driven video and the original video are input into an implicit key point driven model to obtain a synthesized video.

2. The video content modification method according to claim 1, characterized in that: The step of extracting the initial audio of the target person in the original video and extracting the target person's voice and audio text according to the initial audio includes: Using an editing tool to extract the initial audio of the target person in the original video; Inputting the initial audio into a vocal separation technology model to obtain a vocal file; Inputting the human voice file into a noise reduction model to obtain a target human voice; The initial audio is recognized by using a speech recognition model to obtain the audio text.

3. The video content modification method according to claim 1, characterized in that: The step of modifying the audio text to obtain a target text and synthesizing a target audio according to the target text and the target human voice includes: Editing and modifying the audio text to obtain a target text; The target text and the target human voice are input into a sound cloning model for synthesis to obtain a target audio.

4. The video content modification method according to claim 1, characterized in that: The step of performing lip calibration according to the driving video and the target audio to obtain a standard driving video includes: The driving video and the target audio are input into an audio-video synchronization model for matching calculation to obtain a synchronization score; Calibrate the driving video frame by frame according to the synchronization score to obtain the best audio timestamp corresponding to each frame of the video; The driving video is adjusted according to the optimal audio timestamp and the lip shape of the driving video to obtain the standard driving video.

5. The video content modification method according to claim 1, characterized in that: The step of inputting the driving video and the target audio into an audio-video synchronization model for matching calculation to obtain a synchronization score includes: Convert the driving video and the target audio into a video feature vector and an audio feature vector respectively; Extracting spatiotemporal features from the video feature vector and the audio feature vector respectively to obtain video spatiotemporal feature information and audio spatiotemporal feature information; Calculating the correlation between the video spatiotemporal feature information and the audio spatiotemporal feature information to generate a similarity graph; Performing maximum pooling processing on the similarity graph to obtain a time offset; The synchronization score is generated according to the time offset.

6. The video content modification method according to claim 1, characterized in that: The step of inputting the standard driving video and the original video into an implicit key point driving model to obtain a synthesized video comprises: Acquire each frame of the driving picture of the standard driving video and each frame of the original picture of the original video; Encoding the driving picture and the original picture from the pixel space to the latent space to obtain a latent space driving feature map and a latent space original feature map; Generate a latent space synthetic feature map according to the latent space driving feature map and the latent space original feature map; Decoding the latent space synthetic feature map from the latent space to the pixel space to obtain each frame of synthetic picture; The synthetic video is generated according to the synthetic picture and the target audio.

7. The video content modification method according to claim 6, characterized in that: The generating of a latent space synthetic feature map according to the latent space driving feature map and the latent space original feature map comprises: Extract the appearance features of the target person in the original feature map of the latent space to obtain the target appearance features; Extract the action features of the target person in the latent space original feature map and the latent space driven feature map respectively to obtain the original target action features and the driven target action features; Performing feature encapsulation on the target appearance feature, the original target motion feature, and the driven target motion feature to obtain a packaged feature graph; The encapsulated feature map is decoded to obtain the latent space synthetic feature map.

8. A video content modification device, characterized in that: The device comprises: An audio extraction module is used to extract the initial audio of the target person in the original video, and extract the target person's voice and audio text according to the initial audio; An audio synthesis module, used for modifying the audio text to obtain a target text, and synthesizing a target audio according to the target text and the target human voice; A video recording module, used for recording a facial lip-sync video according to the target audio to obtain a driving video; A video calibration module, used for performing lip calibration according to the driving video and the target audio to obtain a standard driving video; The video synthesis module is used to input the standard driving video and the original video into the implicit key point driving model to obtain a synthesized video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the video content modification method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video content modification method according to any one of claims 1 to 7 is implemented.

Images