A video content completion method, electronic device, computer readable storage medium and program product

Abstract

The application relates to the field of computer vision, and provides a video content completion method, an electronic device, a computer readable storage medium and a program product. The method comprises the following steps: acquiring an original video and a corresponding original depth map sequence, performing mask processing on original image frames and depth maps respectively to obtain mask image frames and mask depth maps; performing latent space mapping on the mask image frames and the mask depth maps to obtain image frame latent variables and depth map latent variables, and performing noise processing on the image frame latent variables to obtain noisy latent variables; obtaining predicted noise based on the depth map latent variables and the noisy latent variables; performing iterative denoising on the noisy latent variables according to known region latent variables and the predicted noise to obtain denoised latent variables, and processing the denoised latent variables into denoised image frames; performing pixel fusion on the denoised image frames and the original image frames based on a known region mask to obtain completed image frames, so that a completed video is generated. The application can improve the quality of video completion content.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to a video content completion method, an electronic device, a computer-readable storage medium, and a program product. Background Technology

[0002] Video content completion is a typical task in computer vision, aiming to infer and generate the complete outline and content of occluded or missing areas in a video. However, in complex dynamic video scenes, due to mutual occlusion between objects and camera movement, a single frame image often only captures local information of objects. Completing spatiotemporally consistent content that conforms to physical geometry in a video remains a significant challenge. Traditional video completion methods calculate the movement direction and distance of each point in the image using image computation rules, complete the motion trajectory in the missing area, then stitch the intact pixels from the preceding and following frames to the area to be completed according to the motion trajectory, and finally optimize the stitched edges using a smoothing algorithm.

[0003] However, traditional processing methods lack geometric awareness, so the completed pixels can only be directly copied and stitched from the reference frame. Based on the simplified principle of planar translation, they perform 2D motion estimation, which cannot adapt to scene depth, object geometry, and non-rigid deformation. The completed pixels only perform planar translation, failing to conform to real-world spatial geometry. Furthermore, existing processing methods process a few adjacent frames independently, lacking global temporal constraints, resulting in poor temporal consistency. Errors accumulate frame by frame, causing jitter, flickering, and motion trajectory drift in the completed content, affecting video completion quality. Therefore, there is an urgent need for a method to improve video completion quality, enabling the completed content to conform to real-world spatial geometry and produce smooth footage. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a video content completion method, electronic device, computer-readable storage medium and program product, which improves the problems of lack of geometric perception, poor temporal consistency and geometric data leakage in existing video completion methods by injecting depth information representing the geometric distance relationship of the image as a geometric constraint into the video diffusion model, thereby improving the quality of video completion content.

[0005] To achieve the above technical objectives, the technical solution adopted in this application is as follows:

[0006] This application provides a video content completion method, an electronic device, a computer-readable storage medium, and a program product. The method includes:

[0007] Obtain the original video and the corresponding original depth map sequence, wherein the original video contains several consecutive original image frames and the original depth map sequence includes several depth maps corresponding to the original image frames.

[0008] Masking is performed on all the original image frames in the original video and all the depth maps in the original depth map sequence to obtain masked image frames and masked depth maps, respectively. Each masked image frame contains a known region mask.

[0009] The masked image frame and the masked depth map are latent space mapped by a variational autoencoder to obtain latent variables of the image frame and latent variables of the depth map. The latent variables of the image frame are then subjected to noise processing to obtain noisy latent variables.

[0010] Based on the depth map latent variables and the noisy latent variables, the noise of the noisy latent variables is predicted to obtain the predicted noise;

[0011] Based on the known region latent variables in the image frame latent variables and the predicted noise, the noisy latent variables are iteratively denoised to obtain denoised latent variables, and the denoised latent variables are processed into denoised image frames through a variational autoencoder.

[0012] Based on the known region mask, the denoised image frame and the original image frame are pixel-fused to obtain the completed image frame, thereby generating the completed video.

[0013] According to the first aspect, the step of iteratively denoising the noisy latent variables based on the known region latent variables in the image frame latent variables and the predicted noise to obtain denoised latent variables, and processing the denoised latent variables into a denoised image frame through a variational autoencoder, includes:

[0014] Based on the predicted noise and the noisy latent variable, iterative denoising is performed through a preset scheduler. If the number of current iterations of denoising is less than a first preset value, the known region latent variable and the noisy latent variable are fused and calculated to obtain the predicted latent variable, and the predicted latent variable obtained by the current iteration of denoising is used as the noisy latent variable for the next iteration of denoising.

[0015] If the number of current iterations of denoising is greater than or equal to the first preset value, the noisy latent variable is used as the predicted latent variable, and the predicted latent variable obtained from the current iteration of denoising is used as the noisy latent variable for the next iteration of denoising.

[0016] If the number of current iterations for denoising reaches a second preset value, the predicted latent variable obtained from the last iteration for denoising is used as the output result to obtain the denoised latent variable, where the second preset value is greater than the first preset value.

[0017] The features in the denoising latent variables are restored to the pixel space through the decoder of the variational autoencoder to obtain the denoised image frame.

[0018] According to the first aspect, if the number of iterations for denoising is less than a first preset value, the known region latent variables and the noisy latent variables are fused and calculated to obtain predicted latent variables, including:

[0019] The known region mask is compressed to obtain a soft edge transition mask, which has the same resolution as the noisy latent variable.

[0020] Based on the current number of iterations for denoising, the latent variables in the known region are denoised to obtain the denoised known latent variables;

[0021] Based on the soft edge transition mask, the known noisy latent variable and the noisy latent variable corresponding to the current iteration denoising number are weighted and fused to obtain the predicted latent variable.

[0022] According to the first aspect, the method is used for a deep learning model, the deep learning model including a first network branch and a second network branch, the first network branch including a downsampling block, an intermediate block, an upsampling block and an output layer, the second network branch including a depth downsampling block and a depth intermediate block, the step of predicting the noise of the noisy latent variables based on the depth map latent variables and the noisy latent variables to obtain predicted noise includes:

[0023] The noisy latent variable and the image frame latent variable are input into the first network branch, and feature extraction is performed through the downsampling block in the first network branch to obtain the first shallow feature.

[0024] The first shallow feature is input into the intermediate block of the first network branch, and the intermediate block processes the first shallow feature into a first deep feature.

[0025] The latent variables of the depth map are input into the second network branch, and features are extracted through the depth downsampling blocks of the second network branch to obtain the second shallow layer features;

[0026] The second shallow feature is input into the deep intermediate block of the second network branch, and the deep intermediate block processes the second shallow feature into a second deep feature;

[0027] The second shallow feature and the second deep feature are input into the zero convolutional layer. The zero convolutional layer processes the second shallow feature and the second deep feature into a first fused feature component and a second fused feature component, respectively. The weights and biases of the zero convolutional layer are set to 0 during initialization.

[0028] The first fused feature component is added to and fused with the first shallow feature to obtain the skip connection feature;

[0029] The second fused feature component is added to and fused with the first deep feature to obtain the bottleneck layer feature.

[0030] The skip connection features and the bottleneck layer features are input into the upsampling block, and the predicted noise is obtained by outputting the upsampling block.

[0031] According to the first aspect, the step of fusing the denoised image frame and the original image frame pixel by pixel based on the known region mask to obtain a completed image frame, in order to generate a completed video, includes:

[0032] Based on the known region mask, the pixels in the original image frame and the pixels in the denoised image frame are fused to obtain fused pixels:

[0033]

[0034] in, Indicates merged pixels; Represents the pixels of the original image frame; This represents the mask of the known region; This represents the pixels of the denoised image frame;

[0035] Based on the fused pixels, a complete image frame is obtained, and the complete video is obtained by arranging the complete image frames in the order of the original video.

[0036] According to the first aspect, before obtaining the original video and the corresponding original depth map sequence, the process also includes:

[0037] Obtain historical original video and corresponding historical original depth map sequence, as well as the complete video corresponding to the historical original video. The complete video includes several complete video frames, and each complete video frame has the real pixel content of the area to be filled in in the historical original video. The historical original video contains several consecutive historical original image frames, and the historical original depth map sequence includes several historical depth maps corresponding to the historical original image frames.

[0038] Masking is performed on all the original historical image frames and all the historical depth maps respectively to obtain historical masked image frames and historical masked depth maps. Each historical masked image frame contains a mask of a historically known region.

[0039] The historical mask image frame, the historical mask depth map, and the complete video are latent space mapped by a variational autoencoder to obtain the latent variables of the historical image frame, the latent variables of the historical depth map, and the latent variables of the complete video. The latent variables of the historical image frame are then subjected to noise processing to obtain the noisy historical latent variables.

[0040] Training noise is added to the complete video latent variables to obtain noisy complete video latent variables;

[0041] The historical depth map latent variables, the historical noisy latent variables, the noisy complete video latent variables, and the training noise are integrated into training data;

[0042] A deep learning model is trained and initialized based on the training data, so that after the deep learning model is executed by inputting the deep map latent variables and the noisy latent variables, the deep learning model outputs predicted noise.

[0043] Secondly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory coupled to each other, the memory storing a computer program, and when the computer program is executed by the processor, causing the electronic device to perform the method as described in the first aspect.

[0044] Thirdly, embodiments of this application also provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when run on a computer, causes the computer to perform the method described in the first aspect.

[0045] Fourthly, embodiments of this application also provide a program product, characterized in that it includes a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0046] The invention employing the above technical solution has the following advantages:

[0047] In the technical solution provided in this application, the original image frame and depth map are masked separately, so that the known regions and the regions to be completed in the masked image frame and masked depth map are distinguished by the mask, cutting off the cheating inference of the geometric information of the region to be completed during model inference and preventing data leakage. After mapping the masked image frame and masked depth map to the latent space through a variational autoencoder, the latent variables of the image frame and the latent variables of the depth map are obtained, and the latent variables of the image frame are noise-added to obtain noisy latent variables. The latent variables of the depth map are incorporated as geometric constraints into the process of generating prediction noise in the deep learning model, so that the prediction noise has spatiotemporal awareness and geometric constraints. Taking the noisy latent variables as the processing object, continuous iterative denoising is performed based on the latent variables of the known regions in the latent variables of the image frame, so that each frame of the video is gradually restored in the same latent variable evolution process, thereby reducing the content abrupt changes between adjacent frames and improving the temporal consistency of the completed image frame. The denoised image frame is fused with the original image frame to obtain the completed image frame, thereby generating the completed video and improving the quality of video content completion. Attached Figure Description

[0048] This application can be further illustrated by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate some embodiments of this application and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained from these drawings without any inventive effort.

[0049] Figure 1 This is a flowchart illustrating a video content completion method according to this application.

[0050] Figure 2 This is a schematic diagram of the architecture of the first network branch and the second network branch in one possible implementation of the method described in this application.

[0051] Figure 3 This is a schematic diagram illustrating the implementation effect of a video content completion method according to this application. Detailed Implementation

[0052] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts are referred to by the same reference numerals in the drawings or description. Implementations not shown or described in the drawings are forms known to those skilled in the art. In the description of this application, terms such as "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0053] Existing video content completion methods rely on hand-designed features or basic deep learning models. These methods often lack the ability to perceive the geometric structure of the scene, easily leading to perspective distortion in the generated completed regions. In recent years, video diffusion models, represented by Stable Video Diffusion (SVD), have demonstrated powerful generative capabilities. However, in practical applications, using two-dimensional image features makes it difficult to understand the physical structure of the scene, resulting in perspective distortion and unreasonable deformation of the generated occluded regions. Although video diffusion models exhibit strong spatiotemporal generation capabilities, they still require image constraints, such as using geometric priors from depth maps. However, how to inject depth information into the network and achieve temporally continuous completion without disrupting the original distribution of the pre-trained model remains a technical bottleneck. Furthermore, even if the depth map representing depth information is fed into the network, the geometric information of the region to be completed can easily be used by the model network during the inference stage, leading to data leakage. To enable video diffusion models to better compute image features in videos, variational autoencoders (VAEs) are typically used to map video sequences to the latent space. However, directly concatenating the latent variables obtained from mapping the video sequence can lead to severe 8×8 grid mosaic artifacts at the completion boundaries, thus affecting the consistency of the video sequence. Improving the geometric consistency, temporal coherence, and naturalness of boundary blending in video content completion is a key issue that needs to be addressed in video content completion methods.

[0054] Please refer to Figure 1 This application provides a video content completion method that can be applied to electronic devices, and the steps of the method can be executed or implemented by the electronic device. The electronic device can be, but is not limited to, personal computers, smartphones, and other electronic devices. The method described in this application can be executed by an electronic device equipped with corresponding software functional modules. The computer system of the electronic device can be equipped with an Ubuntu 20.04 or higher operating system, a high-performance multi-core processor, at least 32GB of system memory, and a dedicated graphics card supporting CUDA acceleration (such as an NVIDIA RTX 3090 or A6000 with at least 24GB of VRAM) to meet the computational power requirements for video diffusion model inference and training. The underlying framework adopts the PyTorch deep learning framework, combined with the HuggingFace Diffusers and Accelerate libraries, to achieve high-precision inference and multi-GPU isolated scheduling of the model.

[0055] A video content completion method may include the following steps:

[0056] S10 acquires the original video and the corresponding original depth map sequence. The original video contains several consecutive original image frames, and the original depth map sequence includes several depth maps corresponding to the original image frames.

[0057] S20 performs masking processing on all the original image frames in the original video and all the depth maps in the original depth map sequence to obtain masked image frames and masked depth maps respectively, and each masked image frame contains a known region mask;

[0058] S30 performs latent space mapping on the mask image frame and the mask depth map through a variational autoencoder to obtain latent variables of the image frame and latent variables of the depth map, and adds noise to the latent variables of the image frame to obtain noisy latent variables.

[0059] S40 predicts the noise of the noisy latent variables based on the depth map latent variables and the noisy latent variables, and obtains the predicted noise;

[0060] S50 performs iterative denoising on the noisy latent variables based on the known region latent variables in the image frame latent variables and the predicted noise to obtain denoised latent variables, and processes the denoised latent variables into a denoised image frame through a variational autoencoder.

[0061] S60 performs pixel fusion of the denoised image frame and the original image frame based on the known region mask to obtain a completed image frame, thereby generating a completed video.

[0062] In S10, the original video is an RGB video containing occlusion relationships. At least one original image frame in the original video exhibits an occlusion relationship between different objects or different image regions. Specifically, a foreground object closer to the video capture device can occlude a target object or background region located behind it, preventing part of the image content of the target object or background region from being directly displayed in the corresponding original image frame. For example, in the same original image frame, a pedestrian in front of the camera can occlude a vehicle behind the pedestrian; the area where the pedestrian is located is the foreground occlusion region, and the part of the vehicle occluded by the pedestrian is the region to be completed. The occlusion relationship can exist in multiple consecutive original image frames in the original video, or it can exist only in some original image frames. Since the occluding object, the occluded object, or the video capture device may move, the position, shape, and area of ​​the region to be completed in different original image frames may also change over time. The region to be completed refers to the area that needs to generate image content through subsequent processing due to foreground object occlusion, missing image content, or pre-specified video editing requirements. The known region and the region to be completed are complementary. The known region refers to the area in the original video and the corresponding original depth map sequence that is not occluded by objects and has complete and real pixel information and geometric information.

[0063] A raw depth map sequence refers to a sequence of multiple depth maps corresponding to each frame of the original video image, arranged in the same temporal order. The depth values ​​in the depth map characterize the distance or relative proximity of a scene point corresponding to a pixel location relative to the video capture device. While RGB raw image frames primarily represent the color, texture, and appearance information of objects, depth maps primarily represent the spatial hierarchy, distance variations, and spatial geometric relationships between different objects in the scene. The depth values ​​in the depth map can be absolute depth values ​​expressed in physical distance units such as meters or millimeters, or relative depth values ​​used only to represent the relative proximity of different pixel locations. For example, when using a monocular depth estimation network to process ordinary RGB raw image frames, the resulting depth map may not represent precise actual distances, but rather uses different depth values ​​to represent the relative proximity of objects corresponding to each pixel location.

[0064] In one optional implementation, when the video capture device only captures ordinary RGB video, the electronic device can process the original image frames using a pre-trained depth estimation network to obtain the depth map. Specifically, the original image frames can be sequentially input into the depth estimation network according to the frame order of the original video. The depth estimation network extracts features such as object contours, texture changes, perspective relationships, and scene structure from the original image frames, and outputs a depth map corresponding to each original image frame. The depth estimation network can be a pre-trained monocular depth estimation network, such as Depth Anything V2, or other neural networks capable of outputting pixel-level depth information from RGB images. This processing can be completed offline before video content completion is performed.

[0065] In one possible implementation, when the resolution of the original depth map differs from that of the corresponding original image frame, scaling, cropping, padding, or coordinate mapping can be performed on the depth map to ensure that the processed depth map has the same or corresponding spatial dimensions as the original image frame. Before using the original depth map sequence for subsequent processing, the depth values ​​in each depth map can be normalized to ensure that the depth values ​​of different original image frames are within a uniform numerical range. For example, the depth values ​​can be normalized to the range of [0,1] or [-1,1] based on the maximum and minimum depth values ​​in the original depth map sequence.

[0066] In S20, the masking process generates region marker data corresponding to each original image frame based on the position of the region to be completed in each original image frame. It then retains the original data in the known regions based on the region marker data, while masking or replacing the original data in the region to be completed, ensuring that the processed data clearly distinguishes between the known regions and the region to be completed. The known regions refer to areas in the original image frames that can be used as the basis for video content completion, while the regions to be completed refer to areas that need to be inferred and generated using a deep learning model. The regions to be completed can be target object regions occluded by foreground objects, or regions that need to be regenerated due to video editing, object removal, missing frames, or other reasons. It should be noted that although RGB pixels may still exist in the region to be completed, these RGB pixels usually represent foreground occluded objects, not the actual content of the object to be completed hidden under the occlusion relationship. Therefore, the original RGB pixels in the region to be completed cannot be directly used as a valid basis for the content to be completed and need to be masked before being input into the deep learning model. The region annotation data corresponding to the original image frame can be obtained, and the region to be completed in each original image frame can be determined based on the region annotation data.

[0067] The region annotation data can be obtained manually or through pre-trained image segmentation models, object detection models, object tracking models, or video segmentation models. For example, a user can annotate foreground objects to be removed in one or more original image frames of the original video, and determine the corresponding regions to be completed based on the positions of the foreground objects in each original image frame. When the object corresponding to the region to be completed moves in the original video sequence, the position of the object in subsequent original image frames can be determined by object tracking, and the corresponding regions to be completed can be determined based on the object positions in each original image frame. Therefore, the regions to be completed in different original image frames can have different positions, outlines, and areas. For example, if a pedestrian in the original video occupies part of a vehicle, and it is necessary to remove the pedestrian and complete the vehicle content occupied by the pedestrian, the image area occupied by the pedestrian in each original image frame can be determined as the region to be completed. As the pedestrian moves in the video, the region to be completed can also change frame by frame.

[0068] The mask value corresponding to the known region can be set to 1, and the mask value corresponding to the region to be completed can be set to 0. Thus, the region mask can be divided into: a known region mask M, used to identify the region where the original data needs to be preserved; and regions to be completed masks 1-M, used to identify the regions where the original data needs to be masked and the content generated by the deep learning model needs to be generated. Each original image frame in the original video corresponds to a region mask, and the region masks form a mask sequence according to the original arrangement order of the original image frames.

[0069] Based on the region mask corresponding to the t-th original image frame, pixel-by-pixel processing is performed on the original image frame. In one implementation, the masked image frame can be determined in the following way:

[0070]

[0071] in, This represents the t-th masked image frame; This represents the t-th original image frame in the original video; This represents the mask of the known region corresponding to the t-th original image frame; represents the corresponding mask for the area to be filled; C represents the preset video fill value. This indicates element-wise multiplication.

[0072] For pixel positions belonging to a known region, since the corresponding mask value is 1, the masked image frame retains the RGB pixel values ​​of the original image frame at that position. For pixel positions belonging to a region to be filled, since the corresponding mask value is 0, the masked image frame replaces the original RGB pixels at that position with a preset video fill value. The preset video fill value can be a pre-set fixed value or an invalid marker value adapted to the input data format of the deep learning model. For example, when the RGB video data is normalized to the [-1,1] interval, the preset video fill value can be set to -1.0; in other embodiments, 0, a preset mean, or other values ​​that indicate that the pixel position does not provide valid original image content can also be used. The same type of masking processing is performed on each original image frame in the original video, and the processing results are arranged according to the original temporal order of each original image frame to obtain a masked video sequence.

[0073] Similarly, by masking the original depth map sequence, we can obtain:

[0074]

[0075] in, This represents the depth map of the t-th mask. This represents the t-th depth map; This represents the mask for the corresponding known region; This indicates the preset depth fill value.

[0076] For pixel locations belonging to known regions, the masked depth map retains the depth values ​​from the original depth map; for pixel locations belonging to regions to be filled, the original depth values ​​are replaced with preset depth fill values. The preset depth fill value indicates that the corresponding location does not provide a valid depth condition. In one implementation, when the depth value is normalized to the [-1, 1] interval, the preset depth fill value can be set to -1.0. Alternatively, other pre-set fixed values ​​or invalid marker values ​​can be used depending on the specific requirements of the deep learning model for the input data. Masking is performed on each depth map in the original depth map sequence, and the original depth maps are arranged in chronological order according to the original image frames they correspond to, resulting in a masked depth map sequence.

[0077] When masking the original video and the original depth map sequence separately, the regions to be completed in both can be mutually masked and aligned. At the same time point in the video, for a pixel position in the masked image frame that is set as the region to be completed, the corresponding position in the masked depth map is also set as the region to be completed. Thus, RGB and depth information use consistent validity marking for the same scene location. Through masking, the masked video retains the original RGB information in the known region and masks occluded or invalid pixels in the region to be completed. The masked depth map sequence retains the depth geometry information in the known region and masks the depth information in the region to be completed. Therefore, the masked video and the masked depth map sequence can jointly provide effective and spatially aligned conditional information to the deep learning model, enabling the deep learning model to generate the content to be completed based on the color, texture, temporal variations, and spatial geometric relationships of the known region without knowing the true RGB content and true depth content of the region to be completed, and preventing the model from directly using the depth information of the region to be completed during inference.

[0078] In S30, a VAE (Variational Autoencoder) typically includes a corresponding encoder and decoder. The VAE encoder performs feature extraction and resolution compression on the input raw image frame or depth map, converting image data with a large number of pixels into latent variables with a smaller data volume. The VAE decoder performs the reverse processing, restoring the latent variables to the original image frame that can be displayed. The latent space refers to the feature space formed after the VAE variational autoencoder compresses the pixel data. The latent variables in this space no longer directly represent the color or depth of a single pixel, but rather represent the contours, textures, positional relationships, local structures, and other effective information in the input data in the form of multi-channel features. The variational autoencoder can be a pre-trained and available variational autoencoder in the prior art, outputting the input masked image frame as image frame latent variables with a preset number of channels. The image frame latent variables can be represented as a multi-dimensional tensor including the time dimension, the latent variable channel dimension, the latent space height dimension, and the latent space width dimension. Similarly, for the masked depth map, the VAE encoder converts the depth value distribution in the pixel space into depth features in the latent space. The latent variables in the depth map can characterize the relative distances, depth variations, object edges, and spatial hierarchy relationships at different locations within a known region. Since the original video comprises multiple high-resolution original image frames, performing multi-round diffusion denoising directly on all original image frames in pixel space would require processing a large amount of pixel data, resulting in significant computational and memory consumption. The VAE encoder reduces spatial resolution and extracts effective features, enabling subsequent deep learning models to perform computations in a smaller latent space.

[0079] Adding noise to the latent variables of an image frame requires a preset initial noise, which is a set of numerical data used to perturb the video latent variables. The preset initial noise can be randomly sampled from a standard Gaussian distribution; that is, each noise value in the preset initial noise can follow a Gaussian distribution with a mean of 0 and a variance of 1. The preset initial noise has the same data size as the latent variables of the image frame. Specifically, it can be expressed as:

[0080]

[0081] in: Represents noisy latent variables; Represents latent variables in an image frame; This represents the preset initial noise with the same dimension as the latent variables of the image frame. This represents the signal preservation coefficient corresponding to the initial noise level; This represents the noise addition coefficient corresponding to the initial noise level. The noise addition process is performed element-wise on the entire image frame latent variable tensor, meaning that the data in the image frame latent variable corresponding to the known region and the region to be completed are all processed to the same initial noise level.

[0082] Since the region to be filled in the masked image frame has been set to a preset video padding value, the corresponding latent variables of the image frame do not contain the true and complete image content of the object to be filled. If this latent variable is directly retained without introducing noise, the latent space data in the region to be filled will be strongly limited by the fixed padding value and its encoding result. The deep learning model may tend to retain this fixed state and find it difficult to generate new colors, textures, contours, and object structures in the region to be filled. By adding preset initial noise, the latent variables corresponding to the region to be filled can be transformed from a fixed missing state to an uncertain initial state for generation. Then, based on the video information of the known region, the inter-frame variation information, and the geometric information provided by the mask depth map, the deep learning model gradually generates latent space content that conforms to the current scene from this noisy state. The noise addition process is to establish an initial state for the deep learning model that can gradually reconstruct and generate the content to be filled.

[0083] In S40, the method is used for a deep learning model, which includes a first network branch and a second network branch. The first network branch includes a downsampling block, an intermediate block, an upsampling block, and an output layer. The second network branch includes a depth downsampling block and a depth intermediate block. The step of predicting the noise of the noisy latent variables based on the depth map latent variables and the noisy latent variables to obtain predicted noise includes:

[0084] S41 inputs the noisy latent variable and the image frame latent variable into the first network branch, and performs feature extraction through the downsampling block in the first network branch to obtain the first shallow feature;

[0085] S42 inputs the first shallow feature into the intermediate block of the first network branch, and the intermediate block processes the first shallow feature into a first deep feature;

[0086] S43 inputs the latent variables of the depth map into the second network branch, and performs feature extraction through the depth downsampling block of the second network branch to obtain the second shallow layer features;

[0087] S44 inputs the second shallow feature into the deep intermediate block of the second network branch, and the deep intermediate block processes the second shallow feature into a second deep feature;

[0088] S45 inputs the second shallow feature and the second deep feature into the zero convolutional layer. Through the zero convolutional layer, the second shallow feature and the second deep feature are processed into the first fused feature component and the second fused feature component, respectively. The weights and biases of the zero convolutional layer are both set to 0 during initialization.

[0089] S46 performs additive fusion of the first fused feature component and the first shallow feature to obtain the skip connection feature;

[0090] S47 performs additive fusion of the second fused feature component and the first deep feature to obtain the bottleneck layer feature;

[0091] S48 inputs the skip connection features and the bottleneck layer features into the upsampling block, and outputs the predicted noise through the upsampling block.

[0092] In S41, see Figure 2 In one optional implementation, the first network branch is the main branch in the deep learning model used to process video content information and predict noise, and can adopt a U-Net structure, i.e. Figure 2 The main branch is U-Net. In an alternative implementation, the second network branch can be an injection branch with injection functionality, i.e. Figure 2 The deep injection branch in the network is used. The main branch U-Net includes multiple downsampling blocks, intermediate blocks, multiple upsampling blocks, and an output layer connected in sequence. The noisy latent variables and the image frame latent variables are first concatenated along the channel dimension to form a joint input. For example, when each has 4 channels, the concatenation forms an 8-channel joint input, which is then input into the first downsampling block of the first network branch. The downsampling block is a network processing module used to extract features from the input features and progressively reduce the spatial resolution. Each downsampling block may include a convolutional layer, a residual processing unit, and a downsampling layer. The convolutional layer and the residual processing unit are used to extract features such as edges, textures, object contours, mask boundaries, and inter-frame variations from the noisy latent variables and the image frame latent variables. The downsampling layer reduces the height and width of the feature map through convolution with a preset stride or other size compression methods, so that subsequent downsampling blocks can extract video content features within a larger receptive range. Specifically, the joint input passes through downsampling block 1, downsampling block 2, and downsampling block 3 in sequence. Each downsampling block takes the output of the previous processing stage as input and outputs stage features with different spatial resolutions and different feature levels. The features output by the first downsampling block focus more on local edges, color, and texture information. Subsequent downsampling blocks further extract object outlines, scene layout, and the relationship between changes between video frames while gradually compressing the spatial size. The first shallow feature can be the stage feature output by downsampling block 3.

[0093] In S42, the intermediate block further processes the first shallow features at a lower spatial resolution. By expanding the feature receptive range and integrating the correlation information from different latent space locations and different original image frames, it further integrates the features such as local edges, textures, mask boundaries, and object contours obtained in the downsampling stage into first deep features that can characterize the overall structure of the object, scene layout, motion changes, and contextual relationships of the region to be filled. In specific implementation, the intermediate block may include a convolution processing unit, a residual processing unit, and a spatiotemporal feature processing unit. It retains the effective information in the input features through convolution calculation and residual mapping, and establishes the correlation between different regions and different original image frames through spatial feature interaction or temporal feature interaction. The first deep features output by the intermediate block are used to fuse with the deep geometric features output by the second network branch, and also serve as the basis for restoring spatial details and outputting prediction noise in the upsampling stage of the first network branch.

[0094] In S43, similar to S41, the second network branch is used to independently process the latent variables of the depth map and extract depth geometric features that can characterize the distance relationships of the scene, object edges, and spatial hierarchy. In one possible implementation, the second network branch is a depth injection branch, which includes multiple depth downsampling blocks and depth intermediate blocks connected in sequence, as shown in [reference needed]. Figure 2 The diagram shows depth downsampling blocks 1, 2, and 3. These depth downsampling blocks are feature extraction and size compression modules designed for latent variables in the depth map. Through convolutional feature extraction, residual processing, and downsampling operations, they progressively reduce the spatial resolution of the input features and increase the number of feature channels or expand the feature receptive range. Each depth downsampling block outputs features at different levels, maintaining a spatially compatible size with the first shallow features of the corresponding level in the first network branch. This allows for subsequent processing by zero-convolutional layers before being injected into the main branch U-Net, providing multi-scale depth geometric constraints for the main branch U-Net. Depth downsampling block 3 outputs second shallow features.

[0095] In S44, similar to S42, the depth intermediate block is the bottleneck processing module in the second network branch, located after multiple depth downsampling blocks. It is used to further integrate and abstract the second shallow features output by the last-level depth downsampling block. Through convolution processing, residual mapping, and spatial or temporal feature interaction, the depth intermediate block integrates depth information from various latent spatial locations and different original image frames within a large receptive range. It further processes local depth features into second deep features that can characterize the overall depth distribution of the scene, object spatial contours, and hierarchical relationships.

[0096] In S45, in an optional implementation, the zero-convolutional layer is a feature mapping layer disposed between the second network branch and the first network branch, used to convert the depth geometric features extracted by the second network branch into feature components that can be fused with the video features of the corresponding layer of the first network branch. The zero-convolutional layer can be implemented using convolution operations, for example, using a convolution kernel size of... The convolutional layers, while maintaining the input feature space resolution, map the number of channels and the feature values ​​of each channel of the input features. The convolutional weights and biases of the zero-convolutional layer are set to 0 during model initialization and subsequently updated according to the loss function during deep learning model training. In the trained deep learning model, the weights and biases of the zero-convolutional layer are frozen to fixed values. In one possible implementation, see... Figure 2 The depth injection branch comprises three sequentially connected depth downsampling blocks: 1, 2, 3, and an intermediate depth block. Each depth downsampling block is configured with a corresponding zero convolutional layer, and the intermediate depth block is also configured with a corresponding zero convolutional layer. The first-level depth features output from depth downsampling block 1 are input to the first zero convolutional layer, the second-level depth features output from depth downsampling block 2 are input to the second zero convolutional layer, the third-level depth features output from depth downsampling block 3 are input to the third zero convolutional layer, and the second-level depth features output from the intermediate depth block are input to the intermediate zero convolutional layer. Through this layer-by-layer corresponding processing method, fused feature components for injection into the main branch U-Net can be generated at different spatial resolutions and different feature levels. After passing through their corresponding zero convolutional layers, each depth downsampling block is connected to its corresponding downsampling layer in the first network branch, while the intermediate depth block, after passing through its corresponding zero convolutional layer, is connected to the intermediate block of the first network branch.

[0097] Specifically, the second shallow features output from each depth downsampling block are input into their corresponding trained zero-convolutional layers. These zero-convolutional layers, based on the trained convolutional weights and biases, perform convolutional mapping and channel adjustment on the second shallow features, resulting in multiple first fusion feature components that match the size of the first shallow features at the corresponding layer of the first network branch. Simultaneously, the second deep features output from the depth intermediate blocks are input into their corresponding trained zero-convolutional layers. After convolutional mapping, second fusion feature components matching the size of the first deep features are obtained. At this point, the weights and biases of the zero-convolutional layers have been updated from their initial 0 to the trained values, enabling the conversion of effective depth geometric information into feature components that can be injected into the first network branch. The second network branch provides auxiliary geometric conditions, while the first network branch still undertakes the main tasks of video content generation and noise prediction. By controlling the output of depth features at each layer through trainable zero-convolutional layers, the deep learning model can selectively inject effective depth information based on the training results of the zero-convolutional layers, reducing the adverse effects of irrelevant or inaccurate depth features on video content generation.

[0098] In S46-S47, additive fusion refers to adding two feature tensors with the same spatial size, temporal dimension, and number of channels element-wise at the same feature positions. In one possible implementation, both the first shallow feature and the first fused feature component can include multiple levels. Therefore, additive fusion is performed according to a one-to-one correspondence of network levels. For example, the first fused feature component corresponding to depth downsampling block 1 is added to the first shallow feature output by main branch downsampling block 1; the first fused feature component corresponding to depth downsampling block 2 is added to the first shallow feature output by main branch downsampling block 2; and the first fused feature component corresponding to depth downsampling block 3 is added to the first shallow feature output by main branch downsampling block 3, forming skip connection features at different levels. The skip connection feature refers to the feature passed to the corresponding upsampling block through the skip connection of the first network branch. Skip connections can bypass intermediate blocks, directly providing the higher spatial resolution features retained in the downsampling stage to the upsampling stage. After deep feature addition and fusion, the skip connection features not only include color, texture, edge, object outline, and inter-frame variation information extracted by the main branch, but also local depth variations, depth edges, and foreground / background occlusion relationships provided by the second network branch. Subsequent upsampling blocks can utilize these skip connection features to recover the boundaries and local details of the region to be incomplete.

[0099] The second fused feature component is added to the first deep feature to obtain the bottleneck layer feature. The bottleneck layer feature is a high-level fused feature located between the downsampling and upsampling stages of the first network branch. The first deep feature mainly includes the overall object structure, scene layout, video motion, and contextual relationships extracted by the main branch from noisy latent variables and image frame latent variables. The second fused feature component mainly includes the overall depth distribution of the scene, object spatial contours, and front-back hierarchical relationships. Adding the two element-wise allows for the incorporation of overall depth geometric constraints into the video content representation of the main branch, enabling the subsequent upsampling process to gradually recover latent variables based on both video content information and depth spatial information.

[0100] In S48, the upsampling block is a processing module located after the intermediate block in the first network branch, used to progressively restore the feature spatial resolution and generate prediction noise. At each upsampling level, the bottleneck layer features and the skip connection features corresponding to the current spatial resolution are input into the upsampling block. After multiple sequentially connected upsampling blocks, the output features restored to the preset latent spatial resolution are obtained. These features are then convolved through the output layer of the first network branch to convert the output features into prediction noise.

[0101] Using the second network branch as a depth geometry conditional branch, multi-scale feature extraction is performed on the latent variables of the depth map. The depth features output from each depth downsampling block and intermediate depth block are mapped through zero-convolutional layers and then injected into the shallow skip connections and bottleneck layers corresponding to the first network branch. This allows the main branch to determine the noise components to be removed not only based on video content and inter-frame changes, but also to impose geometric constraints on these noise components based on scene depth changes and occlusion boundaries. The final output predicted noise simultaneously includes spatiotemporal content-aware information and geometric conditional information. This helps reduce problems such as inter-frame flicker and content drift errors that occur in traditional frame-by-frame completion, thereby improving the temporal consistency and spatial structure rationality of the video completion results.

[0102] In S50, the step of iteratively denoising the noisy latent variables based on the known region latent variables in the image frame latent variables and the predicted noise to obtain denoised latent variables, and then processing the denoised latent variables into a denoised image frame through a variational autoencoder, includes:

[0103] S51 Based on the predicted noise and the noisy latent variable, iterative denoising is performed through a preset scheduler. If the number of current iterations of denoising is less than a first preset value, the known region latent variable and the noisy latent variable are fused and calculated to obtain the predicted latent variable, and the predicted latent variable obtained by the current iteration of denoising is used as the noisy latent variable for the next iteration of denoising.

[0104] S52 If the number of current iterations of denoising is greater than or equal to the first preset value, the noisy latent variable is used as the predicted latent variable, and the predicted latent variable obtained from the current iteration of denoising is used as the noisy latent variable for the next iteration of denoising.

[0105] S53 If the number of current iterations of denoising reaches the second preset value, the predicted latent variable obtained from the last iteration of denoising is used as the output result to obtain the denoised latent variable, where the second preset value is greater than the first preset value.

[0106] S54 uses the decoder of the variational autoencoder to restore the features in the denoising latent variables to the pixel space, thereby obtaining the denoised image frame.

[0107] In S51, the preset scheduler is a computational module in the deep learning model used to control the noise level of latent variables and the iterative update process of latent variables. Based on a pre-set noise scheduling sequence, the preset scheduler uses the predicted noise output from the first network branch to perform a numerical update on the latent variable currently at a certain noise level, converting it into the latent variable corresponding to the next noise level. In one possible implementation, the preset scheduler can be an Euler Discrete Scheduler. When using an Euler Discrete Scheduler, the noise-free latent variable corresponding to the current latent variable can be estimated first based on the current latent variable and the predicted noise.

[0108]

[0109] in, This represents the noise-free latent variable estimated based on the current predicted noise. Indicates the noise level corresponding to the current time step; Indicates the current latent variable; This represents the predicted noise. Subsequently, based on the difference between the current latent variable and the noise-free latent variable, the change in the current latent variable along the denoising direction is determined:

[0110]

[0111] in, This indicates the denoising direction corresponding to the current time step. When the deep learning model uses predicted noise as the network output, the denoising direction corresponds to the predicted noise. A preset scheduler further determines the current noise level. With the next noise level Noise step size between:

[0112]

[0113] Since the next noise level is lower than the current noise level, the noise step size is typically negative. Based on this noise step size, an Euler update is performed on the current latent variable to obtain the predicted latent variable. :

[0114]

[0115] The preset scheduler determines the direction in which the latent variable should change based on the predicted noise, and then determines the magnitude of the change based on the difference between adjacent noise levels, moving the current latent variable one step towards a lower noise state. The preset scheduler can also employ other scheduling methods capable of single latent variable updates based on the current latent variable, predicted noise, and time step; this application is not limited to a specific scheduler type. The preset scheduler is primarily used to convert the predicted noise output by the neural network into actual latent variable update results.

[0116] In S51, the first preset value is a threshold for the number of iterations used to divide the early and late stages of iterative denoising. It represents the point at which forced fusion of known regions ceases. This first preset value can be determined based on the total number of iterations and a pre-set fusion stage ratio. For example, 0.85 of the total number of iterations can be rounded down to obtain the first preset value. When the total number of iterations is N, the first preset value can be expressed as... The value 0.85 is merely an example; in actual implementation, it can be preset based on the motion amplitude of the video content, the size of the region to be completed, and the strength of the known region constraints. When the current iteration of denoising is less than the first preset value, a single iteration of denoising is performed to obtain the predicted latent variable. The predicted latent variable is the latent variable after a single denoising process and the addition of known region constraints.

[0117] In S51, if the number of iterations for denoising is less than a first preset value, the known region latent variables and the noisy latent variables are fused and calculated to obtain predicted latent variables, including:

[0118] S511 performs resolution compression on the known region mask to obtain a soft edge transition mask, the soft edge transition mask having the same resolution as the noisy latent variable;

[0119] S512 adds noise to the known region latent variables based on the current number of iterations for denoising, to obtain the denoised known latent variables;

[0120] S513, based on the soft edge transition mask, performs weighted fusion of the known noisy latent variable and the noisy latent variable corresponding to the current iteration denoising number to obtain the predicted latent variable.

[0121] In S511, based on the spatial dimensions of the noisy latent variable, the known region mask in the pixel space is downsampled using bilinear interpolation, compressing its height and width to the same resolution as the noisy latent variable, thus obtaining a soft edge transition mask. During this compression process, the mask values ​​inside the original known region mask and the region to be filled remain essentially 1 and 0, while multiple pixels near their boundary are aggregated into continuous values ​​between 0 and 1, thereby forming a soft edge that gradually changes from the known region to the region to be filled.

[0122] In step S512, based on the current denoising iteration count, the time step and noise intensity corresponding to that iteration count are determined from the noise scheduling sequence of the preset scheduler. Then, according to the known region mask, and based on the noise intensity corresponding to the time step, preset initial noise is added to the known region latent variable to obtain a noisy known latent variable. Specifically, in the early stages of iterative denoising, the noise intensity corresponding to the current iteration count is relatively high, keeping the noisy known latent variable in a high-noise state. As the current denoising iteration count increases, the corresponding noise intensity gradually decreases, causing the noisy known latent variable to synchronously change towards a low-noise state. Therefore, the noisy known latent variable and the predicted latent variable obtained in the current iteration are at a matching noise level, avoiding the direct fusion of known region latent variables with inconsistent noise levels with the noisy latent variable.

[0123] In S513, the deep learning model performs position-wise weighted fusion of the known noisy latent variable and the noisy latent variable at the current denoising iteration, based on a soft-edge transition mask with the same resolution as the noisy latent variable. In the soft-edge transition mask, the values ​​corresponding to the known region are close to 1, the values ​​corresponding to the region to be completed are close to 0, and the values ​​at the boundary between the known region and the region to be completed vary continuously between 0 and 1. This can be calculated in the following way:

[0124]

[0125] in, Indicates the prediction of latent variables. Indicates a soft-edge transition mask. Indicates a known latent variable for adding noise. The noisy latent variable representing the current iteration denoising count. This indicates multiplication by position.

[0126] In the known region, the known latent variables with added noise are mainly retained, while in the region to be completed, the noisy latent variables of the current iteration number are mainly used. At the boundary between the two, a gradual transition is performed according to the soft edge transition mask. In this way, the content constraints of the known region are maintained continuously during the iterative denoising process, while reducing the boundary abrupt changes caused by the direct switching of latent variables between the known region and the region to be completed.

[0127] In S52, when the current iteration denoising count is greater than or equal to the first preset value, it indicates that the iterative denoising process has entered the later stage. In the previous multiple iterations, the deep learning model has repeatedly weighted and fused the known noisy latent variables and the noisy latent variables using soft edge transition masks, essentially establishing the content constraints of the known region, the overall structure of the region to be completed, and the boundary relationship between them. At this point, the deep learning model no longer performs fusion calculations but directly uses the noisy latent variable obtained by the preset scheduler in the current iteration denoising count as the predicted latent variable, and allows this predicted latent variable to enter the next iteration step. In the later stages of iterative denoising, the deep learning model needs to further coordinate the texture, boundaries, and content continuity between different original image frames between the known region and the region to be completed. If the noisy known latent variables are still forcibly fused into the noisy latent variables, it may repeatedly overwrite the latent variable content already generated by the deep learning model, limiting the autonomous refinement of the region to be completed and its boundaries. Using noisy latent variables directly as prediction latent variables removes the forced fusion constraint of the known region on the latent variables. This allows subsequent iterations to further optimize the content to be completed, boundary transitions, and inter-frame continuity while preserving the content and geometric structure already formed in the previous iterations. It should be noted that this step stops the fusion calculation of the latent variables of the known region and the noisy latent variables, but does not stop the iterative denoising. The prediction latent variables serve as input for the next round of denoising processing.

[0128] In S53, the second preset value is the number of denoising iterations to terminate preset by the deep learning model, which can be determined according to the total number of denoising iterations configured by the preset scheduler. The second preset value is greater than the first preset value so that after the deep learning model stops fusing the latent variables of the known region with the noisy latent variables, it can still continue to perform several rounds of single denoising processing to further refine the content to be completed, boundary transitions, and the continuity between the original image frames. When the current denoising iteration reaches the second preset value, it means that all preset denoising iterations have been completed, and the predicted latent variable obtained in the last iteration is at the lowest noise level set by the preset scheduler. The deep learning model does not enter the next iteration, but instead uses the predicted latent variable as the final output to obtain the denoised latent variable.

[0129] In S54, the VAE decoder in the deep learning model is used to perform latent space decoding processing corresponding to the VAE encoder. Specifically, the denoising latent variables are input into the VAE decoder, and through stepwise feature transformation and resolution recovery, the video content, spatial structure, and inter-frame variation information contained in the denoising latent variables are mapped to the pixel space to obtain denoised image frames.

[0130] In S60, the step of fusing the denoised image frame and the original image frame pixel by pixel based on the known region mask to obtain a completed image frame and generate a completed video includes:

[0131] S61, based on the known region mask, performs pixel fusion between the pixels in the original image frame and the pixels in the denoised image frame to obtain fused pixels:

[0132]

[0133] in, Indicates merged pixels; Represents the pixels of the original image frame; This represents the mask of the known region; This represents the pixels of the denoised image frame;

[0134] S62 obtains the completed image frames based on the fused pixels, and obtains the completed video according to the order of the completed image frames in the original video.

[0135] In steps S61-S62, in the known region, the fused pixels are directly taken from the real pixels in the original image frame; in the region to be completed, the fused pixels are taken from the pixels of the denoised image frame. The above fusion is performed on each pixel position, preserving the original known content while embedding the generated content to be completed into the corresponding image frame. Based on the known region mask, the denoised image frame and the original image frame are pixel-fused.

[0136] Although the denoised image frame contains the generated region to be completed, its known regions have also undergone latent space mapping, iterative denoising, and VAE decoding, which may result in color deviation, texture blurring, brightness variations, or loss of local detail. For known regions in the original image frame that are not occluded and do not require completion, there is no need to replace real pixels with newly generated pixels. Therefore, selectively retaining pixels from the original image frame using the known region mask can avoid degrading image quality due to repeated generation of known regions, ensuring that the completion process only actually applies to the region to be completed. Furthermore, using the known region pixels from the original image frame can maintain the original color, texture, and motion variation relationships of the known regions between adjacent image frames. By recovering the known region pixels from the original image frame frame by frame, the known regions can maintain temporal continuity consistent with the original image frame, reducing unnecessary changes in non-completed regions.

[0137] In S40, the deep learning model can be trained in the following way: acquiring historical original videos and corresponding historical original depth map sequences, as well as complete videos corresponding to the historical original videos, wherein the complete videos include several complete video frames, each complete video frame having the real pixel content of the region to be filled in the historical original videos, the historical original videos containing several consecutive historical original image frames, and the historical original depth map sequence including several historical depth maps corresponding to the historical original image frames; performing masking processing on all the historical original image frames and all the historical depth maps respectively to obtain historical masked image frames and historical masked depth maps, each of the historical masked image frames containing a mask of a historically known region; A variational autoencoder is used to perform latent space mapping on the historical masked image frames, the historical masked depth maps, and the complete video to obtain latent variables for historical image frames, historical depth maps, and the complete video. Noise is added to the historical image frame latent variables to obtain noisy historical latent variables. Training noise is added to the complete video latent variables to obtain noisy complete video latent variables. The historical depth map latent variables, the noisy historical latent variables, the prediction noise, and the noisy complete video latent variables are integrated into training data. An initialized deep learning model is trained based on the training data, so that when the depth map latent variables and the noisy latent variables are input into the deep learning model, the deep learning model outputs prediction noise.

[0138] Specifically, before training the deep learning model, historical original videos, their corresponding historical original depth map sequences, and corresponding complete videos are acquired. The historical original videos simulate the input videos for actual video completion; the historical original depth map sequences provide spatial depth information for each historical original image frame; and the complete videos contain the actual pixel content of the region to be completed in the historical original videos, providing a reference for the training process regarding the "true state of the region to be completed." Masking is performed on all historical original image frames and all historical depth maps at the same location, resulting in historical masked image frames and historical masked depth maps. This ensures that the data state during training remains consistent with the actual application stage, meaning the deep learning model infers the region to be completed based on historically known regions and other available information. A variational autoencoder maps the historical masked image frames, historical masked depth maps, and complete videos to the latent space, resulting in latent variables for historical image frames, historical depth maps, and complete videos. The latent variables for complete videos represent the true representation of the complete video with the actual content of the region to be completed in the latent space. Noise is added to the historical image frame latent variables to obtain noisy historical latent variables. The historical noisy latent variables are used to simulate the noisy state faced by the deep learning model when actually performing denoising, enabling the deep learning model to learn how to gradually recover video content from noisy latent variables. The historical depth map latent variables then participate in subsequent noise prediction as spatial structure conditions, so that when the deep learning model determines which noise should be removed, it is simultaneously constrained by the depth relationships, spatial contours, and hierarchical relationships represented by the historical depth map latent variables.

[0139] On the other hand, pre-generated training noise is added to the latent variables of the complete video to obtain noisy complete video latent variables. Since the complete video latent variables themselves correspond to the real content of the complete video, artificially adding training noise allows us to clearly identify what training noise was added during the transformation from complete video latent variables to noisy complete video latent variables. This training noise can therefore serve as a true supervisory signal when the deep learning model outputs predicted noise. The historical depth map latent variables, the historical noisy latent variables, the noisy complete video latent variables, and the training noise are integrated into training data. The training noise is used as the supervisory signal, and the historical depth map latent variables, the historical noisy latent variables, and the noisy complete video latent variables are used as input features to construct a training set. After initializing the deep learning model, a loss function is used for training. After training, during the testing phase, only the depth map latent variables and the noisy latent variables need to be input for the deep learning model to output predicted noise. In this embodiment, the training dataset is divided into a training set and a test set in an 8:2 ratio, with 80% of the data used for model training and the remaining 20% ​​used for testing. The loss function of the deep learning model is the mean squared error between the historical predicted noise and the training noise.

[0140]

[0141] Where: L represents the loss value of the deep learning model; This represents the j-th value in the historical prediction noise; This represents the j-th value in the training noise; This represents the total number of values ​​included in the predicted noise, encompassing the frame dimension, channel dimension, height dimension, and width dimension. By continuously reducing the difference between the predicted noise and the training noise, the deep learning model gradually learns to identify noise that should be removed from historical noisy latent variables, under the constraints of historical depth map latent variables.

[0142] The weights and biases of each zero-convolutional layer are initialized to 0. At the start of training, the depth features output by the second network branch do not change the original output of the first network branch. As training progresses, the weights and biases of each zero-convolutional layer are gradually updated, and the depth geometric information in the historical depth map latent variables is gradually injected into the first network branch. During training, the deep learning model learns the denoising direction corresponding to each noise level through prediction noise loss. After training is complete, during inference, a pre-defined scheduler performs multiple rounds of single-pass denoising on the noisy latent variables based on the continuously output prediction noise of the deep learning model, gradually obtaining the denoised latent variables.

[0143] Please see Figure 3 As an example for effect verification, a video clip containing complex occlusion, background texture changes, and irregular target boundaries is selected as a demonstration of the video content completion effect. The method in this application makes the boundary transition between the generated region and the known region more natural, and the content changes between video frames more continuous.

[0144] To further verify the completion effect of the method in this application, in one possible implementation, quantitative experiments were conducted on the SAIL-VOS validation set and the TAO-Amodal validation set. The SAIL-VOS validation set contains 5432 video sequences with a total of 135800 frames, and the TAO-Amodal validation set contains 1386 video sequences with a total of 34650 frames. During evaluation, the method in this application does not explicitly output the complete target mask. Instead, it extracts the implicit complete target region from the generated RGB video frames and compares it with the manually annotated complete target region. The control method uses a Modal baseline that directly uses known regions as the predicted complete target.

[0145] Experimental results show that on the SAIL-VOS dataset, the overall mIoU of the Modal baseline is 67.89%, while the overall mIoU of the proposed method is improved to 71.55%. Furthermore, it achieves an mIoU_occ of 45.19% in unknown regions, a precision of 44.97% in unknown regions, and a recall of 45.01% in unknown regions. These results demonstrate that the video frames generated by the proposed method contain effective occlusion region shape information, rather than simply replicating known input regions.

[0146] On the TAO-Amodal real-world video dataset, the Modal baseline's mIoU, AP25, AP50, and AP75 are 75.68%, 93.71%, 82.16%, and 63.12%, respectively. The corresponding metrics of the proposed method are improved to 76.36%, 94.05%, 83.22%, and 64.44%. Notably, the method achieves a 1.32 percentage point improvement on the more stringent AP75 metric, indicating that it exhibits better generalization ability and more accurate complete target region recovery in real-world video occlusion scenarios.

[0147] Furthermore, video frames with different degrees of occlusion are grouped and statistically analyzed according to their visibility ratio. SAIL-VOS uses the ratio of the visible mask area to the complete mask area as the visibility ratio, while TAO-Amodal uses the IoU (Intersection over Union) between the visible and complete bounding boxes as an approximation. In SAIL-VOS scenarios with heavy, moderate, and light occlusion, the proposed method improves the mIoU by 16.07, 15.82, and 7.85 percentage points compared to the Modal baseline, respectively; in TAO-Amodal scenarios with heavy, moderate, and light occlusion, the proposed method improves the mIoU by 1.65, 2.03, and 1.59 percentage points, respectively.

[0148] The experimental results above show that the method of this application can generate video frames containing complete target structures using visible RGB information, known region masks and depth geometry priors without relying on explicit complete target mask prediction heads. Its advantages are mainly reflected in scenes with obvious occlusion, and it can effectively improve the problem that traditional known region prediction cannot recover invisible parts.

[0149] This application provides an electronic device that may include a processing module and a memory. The memory stores a computer program, which, when executed by the processor, enables the electronic device to perform the corresponding steps in the video content completion method described above.

[0150] In this embodiment, the processor can be an integrated circuit chip with signal processing capabilities. For example, the processor can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0151] The memory can be, but is not limited to, random access memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, etc.

[0152] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the electronic device described above can be referred to the corresponding steps in the aforementioned method, and will not be elaborated further here.

[0153] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the video content completion method as described in the above embodiments.

[0154] Computer-readable storage media may be magnetic disks, optical disks, read-only memory, random access memory, flash memory, USB flash drives, hard disks, or solid-state drives, etc., and may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implement the methods shown in the above embodiments.

[0155] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video content completion method described above. The computer program product may exist in a computer-readable storage medium in forms including, but not limited to, source files, executable files, and installation package files.

[0156] Based on the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, electronic device, or network device, etc.) to execute the methods described in the various implementation scenarios of this application.

[0157] In the embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0158] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video content completion method, characterized in that, The method includes: Obtain the original video and the corresponding original depth map sequence, wherein the original video contains several consecutive original image frames and the original depth map sequence includes several depth maps corresponding to the original image frames. Masking is performed on all the original image frames in the original video and all the depth maps in the original depth map sequence to obtain masked image frames and masked depth maps, respectively. Each masked image frame contains a known region mask. The masked image frame and the masked depth map are latent space mapped by a variational autoencoder to obtain latent variables of the image frame and latent variables of the depth map. The latent variables of the image frame are then subjected to noise processing to obtain noisy latent variables. Based on the depth map latent variables and the noisy latent variables, the noise of the noisy latent variables is predicted to obtain the predicted noise; Based on the known region latent variables in the image frame latent variables and the predicted noise, the noisy latent variables are iteratively denoised to obtain denoised latent variables, and the denoised latent variables are processed into denoised image frames through a variational autoencoder. Based on the known region mask, the denoised image frame and the original image frame are pixel-fused to obtain the completed image frame, thereby generating the completed video.

2. The method according to claim 1, characterized in that, The step of iteratively denoising the noisy latent variables based on the known region latent variables in the image frame latent variables and the predicted noise to obtain denoised latent variables, and then processing the denoised latent variables into a denoised image frame using a variational autoencoder, includes: Based on the predicted noise and the noisy latent variable, iterative denoising is performed through a preset scheduler. If the number of current iterations of denoising is less than a first preset value, the known region latent variable and the noisy latent variable are fused and calculated to obtain the predicted latent variable, and the predicted latent variable obtained by the current iteration of denoising is used as the noisy latent variable for the next iteration of denoising. If the number of current iterations of denoising is greater than or equal to the first preset value, the noisy latent variable is used as the predicted latent variable, and the predicted latent variable obtained from the current iteration of denoising is used as the noisy latent variable for the next iteration of denoising. If the number of current iterations for denoising reaches a second preset value, the predicted latent variable obtained from the last iteration for denoising is used as the output result to obtain the denoised latent variable, where the second preset value is greater than the first preset value. The features in the denoising latent variables are restored to the pixel space through the decoder of the variational autoencoder to obtain the denoised image frame.

3. The method according to claim 2, characterized in that, If the number of iterations for denoising is less than a first preset value, the known region latent variables and the noisy latent variables are fused and calculated to obtain predicted latent variables, including: The known region mask is compressed to obtain a soft edge transition mask, which has the same resolution as the noisy latent variable. Based on the current number of iterations for denoising, the latent variables in the known region are denoised to obtain the denoised known latent variables; Based on the soft edge transition mask, the known noisy latent variable and the noisy latent variable corresponding to the current iteration denoising number are weighted and fused to obtain the predicted latent variable.

4. The method according to claim 1, characterized in that, The method is used for a deep learning model, which includes a first network branch and a second network branch. The first network branch includes a downsampling block, an intermediate block, an upsampling block, and an output layer. The second network branch includes a depth downsampling block and a depth intermediate block. The step of predicting the noise of the noisy latent variables based on the depth map latent variables and the noisy latent variables to obtain predicted noise includes: The noisy latent variable and the image frame latent variable are input into the first network branch, and feature extraction is performed through the downsampling block in the first network branch to obtain the first shallow feature. The first shallow feature is input into the intermediate block of the first network branch, and the intermediate block processes the first shallow feature into a first deep feature. The latent variables of the depth map are input into the second network branch, and features are extracted through the depth downsampling blocks of the second network branch to obtain the second shallow layer features; The second shallow feature is input into the deep intermediate block of the second network branch, and the deep intermediate block processes the second shallow feature into a second deep feature; The second shallow feature and the second deep feature are input into the zero convolutional layer. The zero convolutional layer processes the second shallow feature and the second deep feature into a first fused feature component and a second fused feature component, respectively. The weights and biases of the zero convolutional layer are set to 0 during initialization. The first fused feature component is added to and fused with the first shallow feature to obtain the skip connection feature; The second fused feature component is added to and fused with the first deep feature to obtain the bottleneck layer feature. The skip connection features and the bottleneck layer features are input into the upsampling block, and the predicted noise is obtained by outputting the upsampling block.

5. The method according to claim 2, characterized in that, The step of fusing the denoised image frame and the original image frame pixel by pixel based on the known region mask to obtain a completed image frame, thereby generating a completed video, includes: Based on the known region mask, the pixels in the original image frame and the pixels in the denoised image frame are fused to obtain fused pixels: in, Indicates merged pixels; Represents the pixels of the original image frame; This represents the mask of the known region; This represents the pixels of the denoised image frame; Based on the fused pixels, a complete image frame is obtained, and the complete video is obtained by arranging the complete image frames in the order of the original video.

6. The method according to claim 4, characterized in that, Before acquiring the original video and the corresponding original depth map sequence, the following steps are also included: Obtain historical original video and corresponding historical original depth map sequence, as well as the complete video corresponding to the historical original video. The complete video includes several complete video frames, and each complete video frame has the real pixel content of the area to be filled in in the historical original video. The historical original video contains several consecutive historical original image frames, and the historical original depth map sequence includes several historical depth maps corresponding to the historical original image frames. Masking is performed on all the original historical image frames and all the historical depth maps respectively to obtain historical masked image frames and historical masked depth maps. Each historical masked image frame contains a mask of a historically known region. The historical mask image frame, the historical mask depth map, and the complete video are latent space mapped by a variational autoencoder to obtain the latent variables of the historical image frame, the latent variables of the historical depth map, and the latent variables of the complete video. The latent variables of the historical image frame are then subjected to noise processing to obtain the noisy historical latent variables. Training noise is added to the complete video latent variables to obtain noisy complete video latent variables; The historical depth map latent variables, the historical noisy latent variables, the noisy complete video latent variables, and the training noise are integrated into training data; A deep learning model is trained and initialized based on the training data, so that after the deep learning model is executed by inputting the deep map latent variables and the noisy latent variables, the deep learning model outputs predicted noise.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory coupled together, the memory storing a computer program that, when executed by the processor, causes the electronic device to perform the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 6.

9. A program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images