Video content replacement method and system

Abstract

The invention relates to the technical field of data processing, in particular to a video content replacement method and system. The method comprises the following steps: acquiring a to-be-replaced video, and performing audio-video separation on the to-be-replaced video to obtain an audio stream and a video frame sequence; based on the audio stream, utilizing a preset voice generation model to generate oral broadcast voice of the target language; performing target detection, segmentation and tracking on the video frame sequence by using a preset target detection model to obtain a time sequence mask sequence of the target to be replaced; based on the time sequence mask sequence, performing background reconstruction on the video frame sequence by using a preset video restoration model to obtain a background video frame sequence after the to-be-replaced target is removed; and based on the oral broadcast voice and the background video frame sequence, generating a replaced target video by using a preset audio-driven video generation model. The method provided by the invention can improve the efficiency and quality of video content replacement.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a video content replacement method and system[1]. Background Technology

[0002] With the development of digital media technology, video has become a core medium for information transmission, cultural dissemination, and commercial promotion. In the post-production stage of video production, in order to meet the needs of audiences from different regions and cultural backgrounds, it is usually necessary to replace and reconstruct the core content of the video, such as characters, on-screen subtitles, and spoken audio.

[0003] In existing technologies, voice replacement typically relies on voice actors recording in the target language. This method is costly, inefficient, and prone to inconsistencies between the voice and the character's lip movements or tone. Character replacement, on the other hand, usually relies on green screen shooting and 2D / 3D animation redrawing, which also suffer from high costs and low efficiency. Furthermore, it is difficult to ensure that the new character blends naturally with the original video scene's lighting and movement, and it cannot achieve rapid switching of ethnicity or style. In addition, existing video restoration technologies, such as those based on optical flow or simple context completion, are mainly used to handle small-area defects. After removing large targets (such as characters), it is difficult to guarantee the temporal continuity and motion consistency of the restored background video, which can easily lead to screen flickering.

[0004] Therefore, improving the efficiency and quality of video content replacement is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] To address the technical problems of low efficiency and poor quality in video content replacement, the present invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides a video content replacement method, comprising: acquiring a video to be replaced; performing audio-video separation on the video to be replaced to obtain an audio stream and a video frame sequence; generating spoken speech in a target language based on the audio stream using a preset speech generation model; performing target detection, segmentation, and tracking on the video frame sequence using a preset target detection model to obtain a temporal mask sequence of the target to be replaced; reconstructing the background of the video frame sequence based on the temporal mask sequence using a preset video inpainting model to obtain a background video frame sequence after removing the target to be replaced; and generating a replaced target video based on the spoken speech and the background video frame sequence using a preset audio-driven video generation model.

[0007] Furthermore, generating spoken speech in the target language using a preset speech generation model includes: performing speech recognition on the audio stream to obtain source text; translating the source text into the target language in response to receiving a translation instruction to obtain target text; and inputting the target text and the audio stream into the speech generation model to obtain the spoken speech.

[0008] Furthermore, the speech generation model is the CosyVoice3 model, and the loss function used during the post-training of the CosyVoice3 model is:

[0009] In the formula, This is the loss function, which is also the optimization objective function of reinforcement learning. A sequence of speech tokens generated by the current speech generation model. This indicates the expected calculation. For the preset reward function, Let KL divergence function be used. For the current speech generation model, For reference speech generation models, This is a hyperparameter.

[0010] Furthermore, the target to be replaced includes subtitles and characters; after generating the replaced target video, the method further includes: generating a subtitle image sequence in the target language based on the target text; and adding the subtitle image sequence to the corresponding position in the target video.

[0011] Furthermore, the target detection model includes a Grounding DINO model and a SAM2 model; the target detection, segmentation, and tracking of video frames in the video frame sequence are performed using the preset target detection model, including: obtaining a text cue describing the target to be replaced; inputting the text cue and the video frame sequence into the Grounding DINO model for target detection to obtain a bounding box containing the target to be replaced; and inputting the bounding box as a cue, together with the video frame sequence, into the SAM2 model for segmentation and tracking.

[0012] Further, a background video frame sequence after removing the target to be replaced is generated using a preset video restoration model, resulting in the background video frame sequence after removing the target to be replaced. This includes: preprocessing the video frame sequence and the temporal mask sequence to obtain a mask latent feature sequence and a latent mask sequence; constructing optical flow guidance conditional features based on the mask latent feature sequence and the latent mask sequence; fusing the mask latent feature sequences of subsequent frames (excluding the first frame) with the corresponding latent mask sequences with added random noise to obtain a subsequent frame noise feature sequence; concatenating the optical flow guidance conditional features and the subsequent frame noise feature sequence and inputting them into a preset diffusion model to obtain the restored target latent feature sequence; and decoding the target latent feature sequence to obtain the background video frame sequence.

[0013] Furthermore, the diffusion model is a 3D-UNet diffusion model, and the processing procedure of the 3D-UNet diffusion model satisfies the following equation:

[0014] In the formula, For the target latent feature sequence, For the 3D-UNet diffusion model, The number of video frames. As the potential mask for frame 1, For the first The potential mask of the frame, For the 2nd frame to the 3rd frame The potential mask sequence of frames, For the mask potential features of the first frame, For the first Frame mask latent features, For the 2nd frame to the 3rd frame The mask latent feature sequence of the frame, For optical flow warp operators, For the first The optical flow field from frame 1 to frame 1 For Hadamard product, It is random noise. This is for splicing operations.

[0015] Further, the video frame sequence and the temporal mask sequence are preprocessed to obtain a mask latent feature sequence and a latent mask sequence, including: mapping the video frame sequence to a latent space to obtain a mask latent feature sequence; and downsampling the temporal mask sequence to the same dimension as the mask latent feature sequence to obtain a latent mask sequence.

[0016] Furthermore, the audio-driven video generation model is the Wan2.2-S2V model.

[0017] In a second aspect, the present invention provides a video content replacement system, including a processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the video content replacement method described in the first aspect.

[0018] The beneficial effects of this invention are as follows: by generating spoken audio in the target language through a preset speech generation model, the efficiency of video content replacement is improved, while ensuring that the timbre of the replaced spoken audio is the same as that of the original audio; by processing video frames using an object detection model and a video repair model, seamless restoration of the background video is achieved, improving the quality of video content replacement; by generating the target video using an audio-driven video generation model, the efficiency and quality of video content replacement are further improved, while ensuring that the replaced audio matches the lip movements of the replaced character, ultimately constructing an end-to-end automated video content replacement pipeline. Attached Figure Description

[0019] Figure 1 This is a flowchart of the video content replacement method in an embodiment of the present invention; Figure 2 This is a structural block diagram of the frequency content replacement system in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart of the video content replacement method in an embodiment of the present invention.

[0023] In a first aspect, the present invention provides a video content replacement method, such as... Figure 1 As shown, the method of the present invention includes the following steps.

[0024] S1. Obtain the video to be replaced, and perform audio-video separation on the video to be replaced to obtain the audio stream and video frame sequence.

[0025] Specifically, the process involves acquiring the video to be replaced uploaded by the user through the platform / terminal, then inputting the video into FFmpeg (a multimedia processing library) for audio-video separation to obtain an audio stream and a video frame sequence. The video frame sequence consists of multiple video frames. FFmpeg is an audio-video processing toolset that supports the splitting and merging of audio and video.

[0026] S2. Based on the audio stream, generate spoken speech in the target language using a preset speech generation model.

[0027] In this embodiment, the speech generation model adopts the CosyVoice3 model based on reinforcement learning training, which supports speech generation in multiple languages.

[0028] Specifically, ASR (Automatic Speech Recognition) technology is used to perform speech recognition on the audio stream to obtain the source text. If a translation command is received, the source text is translated into the target language to obtain the target text. The target text and audio stream are then input into a pre-trained speech generation model to obtain the spoken speech in the target language. It is understood that when the user selects a target language, the corresponding translation command is triggered. If the user does not select a target language, or the selected target language is the same as the original language, the source text and audio stream are directly input into the pre-trained speech generation model. In this embodiment, the input audio stream is used as a timbre reference to ensure that the timbre of the generated spoken speech is the same as the timbre of the original character.

[0029] The speech generation model can quickly generate spoken audio in any language and ensure that the timbre of the generated spoken audio is consistent with the original timbre, thus improving the quality and efficiency of video content replacement.

[0030] In one embodiment, the method of the present invention further includes: post-training the CosyVoice3 model using a preset loss function. Specifically, the loss function is:

[0031] In the formula, This is the loss function, which is also the optimization objective function of reinforcement learning. A sequence of speech tokens generated by the current speech generation model. This indicates the expected calculation. For the preset reward function, Let KL divergence function be used. For the current speech generation model, For reference speech generation models, This is a hyperparameter.

[0032] By introducing a reward function based on reinforcement learning, the generated speech becomes more realistic and natural. By introducing KL divergence, the generated speech is guaranteed to maintain quality without losing the accuracy of the text content and the coherence of the language.

[0033] S3. Using a preset target detection model, the video frame sequence is subjected to target detection, segmentation and tracking to obtain a temporal mask sequence of the target to be replaced.

[0034] In this embodiment, the target detection model includes a cascaded Grounding DINO model and a SAM2 model.

[0035] Specifically, a text prompt describing the target to be replaced is obtained, which can be input by the user; wherein the target to be replaced includes characters and subtitles; the text prompt and the video frame sequence are input into a pre-trained GroundingDINO model for target detection to obtain bounding boxes containing the target to be replaced, which include bounding boxes for characters and subtitles. In one embodiment, the process of generating bounding boxes can be represented as:

[0036] In the formula, For the first The bounding box of the frame. Representing the Grounding DINO model, For text prompts, For video frames .

[0037] Furthermore, the bounding boxes are used as prompts and input along with the video frame sequence into a pre-trained SAM2 model for segmentation and tracking, resulting in a temporal mask sequence of the target to be replaced.

[0038] Specifically, the bounding box is input as a cue into the SAM2 model to obtain the mask / mask of the current frame. In one embodiment, the mask of the current frame can be represented as:

[0039] In the formula, For the first Frame mask, Represents the SAM2 model. For the first The bounding box of the frame. For video frames .

[0040] By using bounding boxes as input cues into the SAM2 model, the model can quickly and accurately segment and locate characters and subtitles at the pixel level, thereby improving the accuracy and efficiency of subsequent video content replacement.

[0041] Furthermore, the mask of the current frame is stored in a temporal memory as prior knowledge for the segmentation of the next frame. Using the temporal propagation mechanism / tracking module of the SAM2 model, the mask of the first frame is propagated to subsequent video frames, resulting in a continuous temporal mask sequence. This temporal mask sequence includes a character temporal mask sequence and a subtitle temporal mask sequence. In one embodiment, the above propagation process can be as follows:

[0042] In the formula, For the first Frame mask, For the first Frame mask, Represents the SAM2 model. For the first The bounding box of the frame. For video frames .

[0043] Compared to existing technologies, this invention, through the SAM2 model, can continuously and stably track the same target in subsequent frames of the video and generate continuous, smooth, and consistent-shaped masks. This enables pixel-level and temporally continuous precise positioning and tracking of the subtitles waiting to be replaced in the video to be replaced. As a result, random jitter at the mask edges is avoided when the target to be replaced is subsequently erased, thereby ensuring the smoothness and coherence of the repaired background area and improving the quality of video content replacement.

[0044] It should be noted that the subsequent frames referred to are all video frames in the video frame sequence other than the first frame / first frame.

[0045] S4. Based on the temporal mask sequence, the background of the video frame sequence is reconstructed using a preset video restoration model to obtain the background video frame sequence after removing the target to be replaced.

[0046] In this embodiment, the video restoration model is the FFF-VDI (First Frame Filling VideoDiffusion Inpainting) model.

[0047] In one embodiment, obtaining the background video frame sequence after removing the target to be replaced includes the following steps.

[0048] S41. Preprocess the video frame sequence and the temporal mask sequence respectively to obtain the mask latent feature sequence and the latent mask sequence.

[0049] Specifically, a VAE (Variational Autoencoder) is used to map the video frame sequence to the latent space to obtain the mask latent feature sequence; the temporal mask sequence is downsampled to the same dimension as the mask latent feature sequence to obtain the latent mask sequence.

[0050] S42. Construct optical flow guidance condition features based on mask latent feature sequences and latent mask sequences.

[0051] Specifically, the optical flow field of subsequent frames relative to the first frame is calculated. This optical flow field represents the displacement vector of a pixel moving from the subsequent frame to the first frame. Existing optical flow estimation algorithms can be used for calculation, and there are no restrictions here.

[0052] Furthermore, the complementary mask of the first frame is multiplied with the latent mask feature of the first frame to obtain the background feature of the first frame; the background feature of subsequent frames is obtained in the same way.

[0053] Furthermore, the optical flow warp operator is invoked to spatially warp the background features of subsequent frames, guided by the optical flow field, to obtain aligned background features of subsequent frames. The aligned background features of subsequent frames are accumulated to obtain aggregated background features. The aggregated background features are multiplied with the latent mask of the first frame to obtain filling features only for the mask region. The filling features are added with the background features of the first frame to obtain optical flow guided conditional features. The optical flow guided conditional features are a complete feature map in the latent space that uses the physical information of subsequent frames to fill the holes in the first frame.

[0054] In one embodiment, the process of constructing this optical flow guiding condition feature can be represented as follows:

[0055] In the formula, The number of video frames. As the potential mask for frame 1, For the first The potential mask of the frame, For the mask potential features of the first frame, For the first Frame mask latent features, For the pixel point from the first The optical flow field from frame 1 to frame 1 For optical flow warp operators, This indicates the complementary mask of the first frame.

[0056] By using optical flow mapping to complete the background features located outside the mask in subsequent frames into the first frame, the problem of blurring in the first frame is solved, ensuring the spatiotemporal consistency of background features.

[0057] S43. The mask latent feature sequences of subsequent frames (excluding the first frame) are fused with the corresponding latent mask sequences with added random noise to obtain the noise feature sequences of subsequent frames.

[0058] Specifically, random noise is added to the potential mask of each subsequent frame; the sequence of potential masks with added random noise is then fused with the corresponding mask potential feature sequence to obtain the noise feature sequence of subsequent frames. In one embodiment, the process of obtaining the noise feature sequence of subsequent frames can be represented as:

[0059] In the formula, For the 2nd frame to the 3rd frame The mask latent feature sequence of a frame, i.e. the mask latent feature sequence of subsequent frames; For the 2nd frame to the 3rd frame The potential mask sequence of a frame, i.e., the potential mask sequence of subsequent frames; It is random noise; For Hadamard product, i.e., fusion; This represents the potential mask filling random noise in subsequent frames.

[0060] S44. After concatenating the optical flow guidance condition features and subsequent frame noise feature sequences, input them into the preset diffusion model to obtain the repaired target potential feature sequence.

[0061] In one embodiment, the process of obtaining the repaired target latent feature sequence can be represented as:

[0062] In the formula, For the target latent feature sequence, For the 3D-UNet diffusion model, Representation model, The number of video frames. As the potential mask for frame 1, For the first The potential mask of the frame, For the 2nd frame to the 3rd frame The potential mask sequence of frames, where m represents the mask and t represents the frame. For the mask potential features of the first frame, For the first Frame mask latent features, For the 2nd frame to the 3rd frame The mask latent feature sequence of the frame, For optical flow warp operators, For from the first The optical flow field from frame 1 to frame 1 For Hadamard product, It is random noise. In this embodiment, the splicing operation is performed along the temporal dimension.

[0063] This expression represents that the 3D-UNet diffusion model uses the integrity of the first frame to guide the background generation of subsequent frames by jointly processing optical flow guidance features with subsequent noise features.

[0064] S45. Decode the target latent feature sequence to obtain the background video frame sequence.

[0065] Specifically, VAE is used to decode the target latent feature sequence to obtain the background video frame sequence.

[0066] By using the FFF-VDI model as the core engine, seamless background reconstruction of large-area, arbitrary-shaped, and long-term missing regions is achieved. This engine only performs noise domain warp-fill on the mask region of the first frame once, and subsequent frames are generated at once by the 3D-UNet diffusion model, completely abandoning frame-by-frame propagation, eliminating error accumulation and spatiotemporal flicker, and ensuring the temporal continuity and motion consistency of the repaired background video, thereby improving the quality of video restoration and, in turn, improving the quality of video content replacement.

[0067] S5. Based on the spoken audio and background video frame sequence, generate the replacement target video using a preset audio-driven video generation model.

[0068] In this embodiment, the audio-driven video generation model is the Wan2.2-S2V model.

[0069] Specifically, keyframes in the background video frame sequence are used as pixel-level constraints or latent background variables to ensure that the Wan2.2-S2V model defines the area outside the human figure region, while other pixels remain unchanged. The corresponding spoken audio is used as the action-driven condition, and combined with the mask information of the area to be replaced and the text prompts or reference images of the target human figure, this information is input into the Wan2.2-S2V model. The Wan2.2-S2V model outputs a digital human video synchronized with the audio within the masked area, while maintaining the pixel count of the background video region. The mask information of the area to be replaced is either the human figure's bounding box or a temporal mask sequence.

[0070] It should be noted that in this embodiment, S2 and S3 are two independent steps that can be executed simultaneously.

[0071] By utilizing the Wan2.2-S2V model, the newly generated spoken speech can be efficiently and naturally mapped to the facial movements and lip movements of the target character, solving the problem that the replaced character is prone to stiff expressions, lack of facial details, or obvious motion artifacts in the existing technology.

[0072] In one embodiment, the method of the present invention further includes: generating a subtitle image sequence in the target language based on the target text; adding the subtitle image sequence to the corresponding position in the target video so that the subtitle image sequence is synchronized with the spoken speech on the time axis; and outputting the target video with the added subtitle image sequence as the final target video, wherein the target video is a digital human video that matches the spoken speech, and the subtitles also match the spoken speech and the digital human.

[0073] In another embodiment, the process may also include: identifying the subtitle region in the video frame (the subtitle region can be located by subtitle bounding boxes); then using OCR (Optical Character Recognition) technology to perform text recognition on the subtitle region to obtain subtitle text; if a translation instruction is received, translating the subtitle text into the target language to obtain target subtitle text; generating a subtitle image sequence in the target language based on the target subtitle text; adding the subtitle image sequence to the corresponding position in the target video; and outputting the target video with the added subtitle image sequence as the final target video.

[0074] Since generating subtitle image sequences based on text and adding the subtitle image sequences to the corresponding positions in the target video are existing technologies, they will not be described in detail here.

[0075] Existing video content replacement technologies typically involve multiple independent and complex steps that are manually linked together, including video disassembly, ASR (Automatic Speech Responsive Analysis), TTS (Text-to-Speech), human image synthesis, subtitle localization, and background restoration. These methods are characterized by cumbersome processes, excessive manual intervention, low efficiency, and a high susceptibility to errors. In contrast, the method of this invention seamlessly integrates target location, cross-language audio transfer, and seamless video restoration to create an end-to-end automated processing pipeline. Users only need to input the video to be replaced, and the system can automatically complete all intermediate steps and output a target video that meets the requirements. This achieves one-click, fully automated video content replacement based on target requirements, improving both the efficiency and quality of video content replacement.

[0076] Figure 2 This is a schematic diagram illustrating the structure of the video content replacement system according to this embodiment.

[0077] In a second aspect, the present invention also provides a video content replacement system. For example... Figure 2As shown, the system includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the video content replacement method described in the first aspect of the present invention.

[0078] The system also includes other components well known to those skilled in the art, such as communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0079] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0080] In the description of this specification, "multiple" means at least two, such as two, three or more, unless otherwise explicitly specified. Furthermore, the steps described above are for clarity only; in implementation, they can be combined into one step or some steps can be broken down into multiple steps, as long as they include the same logical relationships.

[0081] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for replacing video content, characterized in that, include: Obtain the video to be replaced, and perform audio-video separation on the video to be replaced to obtain an audio stream and a video frame sequence; Based on the audio stream, spoken speech in the target language is generated using a preset speech generation model; The video frame sequence is subjected to target detection, segmentation and tracking using a preset target detection model to obtain a temporal mask sequence of the target to be replaced; Based on the temporal mask sequence, the background of the video frame sequence is reconstructed using a preset video restoration model to obtain the background video frame sequence after removing the target to be replaced. Based on the spoken audio and the background video frame sequence, a preset audio-driven video generation model is used to generate the replacement target video.

2. The video content replacement method according to claim 1, characterized in that, Generate spoken audio in the target language using a pre-defined speech generation model, including: Perform speech recognition on the audio stream to obtain the source text; In response to receiving a translation instruction, the source text is translated into the target language to obtain the target text; The target text and the audio stream are input into the speech generation model to obtain the spoken speech.

3. The video content replacement method according to claim 2, characterized in that, The speech generation model is the CosyVoice3 model, and the loss function used during the training of the CosyVoice3 model is: In the formula, This is the loss function, which is also the optimization objective function of reinforcement learning. A sequence of speech tokens generated by the current speech generation model. This indicates the expected calculation. For the preset reward function, Let KL divergence function be used. For the current speech generation model, For reference speech generation models, This is a hyperparameter.

4. The video content replacement method according to claim 2, characterized in that, The targets to be replaced include subtitles and characters; after generating the replaced target video, it also includes: Generate a sequence of subtitle images in the target language based on the target text; Add the subtitle image sequence to the corresponding position in the target video.

5. The video content replacement method according to claim 1, characterized in that, The target detection model includes the Grounding DINO model and the SAM2 model; using the preset target detection model to perform target detection, segmentation, and tracking on video frames in the video frame sequence includes: Get a text prompt describing the target to be replaced; The text prompt and the video frame sequence are input into the Grounding DINO model for target detection to obtain a bounding box containing the target to be replaced; The bounding box is used as a cue and, together with the video frame sequence, is input into the SAM2 model for segmentation and tracking.

6. The video content replacement method according to claim 1, characterized in that, A pre-defined video restoration model is used to generate a sequence of background video frames after removing the target to be replaced, resulting in the following sequence: The video frame sequence and the temporal mask sequence are preprocessed respectively to obtain the mask latent feature sequence and the latent mask sequence; Optical flow guidance condition features are constructed based on the mask potential feature sequence and the potential mask sequence; The mask latent feature sequences of subsequent frames (excluding the first frame) are fused with the corresponding latent mask sequences with added random noise to obtain the noise feature sequences of subsequent frames. The optical flow guidance condition features and the subsequent frame noise feature sequences are concatenated and then input into a preset diffusion model to obtain the repaired target potential feature sequence. The target latent feature sequence is decoded to obtain the background video frame sequence.

7. The video content replacement method according to claim 6, characterized in that, The diffusion model is a 3D-UNet diffusion model, and the processing procedure of the 3D-UNet diffusion model satisfies the following formula: In the formula, For the target latent feature sequence, For the 3D-UNet diffusion model, The number of video frames. As the potential mask for frame 1, For the first The potential mask of the frame, For the 2nd frame to the 3rd frame The potential mask sequence of frames, For the mask potential features of the first frame, For the first Frame mask latent features, For the 2nd frame to the 3rd frame The mask latent feature sequence of the frame, For optical flow warp operators, For the first The optical flow field from frame 1 to frame 1 For Hadamard product, It is random noise. This is for splicing operations.

8. The video content replacement method according to claim 6, characterized in that, The video frame sequence and the temporal mask sequence are preprocessed respectively to obtain a mask latent feature sequence and a latent mask sequence, including: The video frame sequence is mapped to the latent space to obtain the mask latent feature sequence; The temporal mask sequence is downsampled to the same dimension as the potential feature sequence of the mask to obtain the potential mask sequence.

9. The video content replacement method according to claim 1, characterized in that, The audio-driven video generation model is the Wan2.2-S2V model.

10. A video content replacement system, characterized in that, It includes a processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the video content replacement method according to any one of claims 1-9.

Images