Complementation method and system for consistent objects in monocular video

Abstract

The invention provides a consistent object completion method and system in a monocular video, and the method comprises the steps: obtaining a to-be-processed video in the monocular video, carrying out the segmentation and marking through a pre-training segmentation model, and obtaining all video frame data containing a complete target object, and a mask image; according to the screened complete target object, searching an initial coverage position of the first shelter picture in the video initial frame; dynamically adjusting the size and the center of the shelter; compressing the video to be processed, the visible part mask sequence and the target object video added with noise into the same hidden space; constructing an adjusted stable video diffusion model, predicting a noise item needing to be removed, and performing supervision by using a truth value; a video object completion model is constructed, and time-space consistent video object completion is realized; constructing a staged completion and picture fusion long video processing model, and processing a long video; through an iterative test training strategy, the processing efficiency and accuracy are improved; and inference is carried out to obtain an unshielded complete video of the final target object.

Description

Technical Field

[0001] The present invention relates to the technical field of video image intelligent data processing, and more specifically, to a method and system for completing consistent objects in a monocular video. Background Art

[0002] Completing occluded objects in monocular videos is a challenging task in the field of computer vision. It aims to consistently restore the complete geometry, appearance, and motion of partially visible objects from monocular videos, so as to improve the understanding of videos and the 3D world and assist downstream tasks such as 3D reconstruction and pose estimation. The main difficulty in completing occluded objects in monocular videos comes from the uncertainty of the information provided by the monocular video and how to consistently complete the occluded parts while keeping the content of the unoccluded parts unchanged. Previous studies have proposed a variety of methods to address this challenge, including video restoration technology, or dividing the video into each frame and then using a single image completion model, but these methods have the following three main disadvantages: 1) These works ignore the maintenance of consistency between frames. This spatiotemporal consistency is extremely important for video understanding and good consistency is a prerequisite for downstream reconstruction tasks; 2) Existing video-based restoration work requires explicit specification of the area to be restored, which assumes a general understanding of the occluded area. The shape of the occluded object, rather than directly inferring the shape of the occluded area from the observable part of the input video; 3) It is difficult for existing video object completion models to utilize the temporal information in the video, while the temporal context can provide other viewpoint information and action information about the target object; many recent works have used video generation models to synthesize multi-perspective consistent content or temporally continuous action sequences, but there is still a lack of reasonable models to use such generation models to consistently complete objects in the video; how to filter out object candidate areas occluded by other objects to obtain video frame data and mask images, how to search for coverage positions in the video start frame and adjust the occluded video data set; how to predict the noise items that need to be removed and use the true value for supervision and achieve consistent video object completion in time and space, how to efficiently process long videos to improve processing efficiency and accuracy and obtain complete videos of target objects, etc., have yet to be solved; therefore, it is necessary to propose a consistent object completion method and system in monocular video to at least partially solve the problems existing in the prior art. Summary of the invention

[0003] A series of simplified concepts are introduced in the content of the invention, which will be further described in detail in the specific implementation method section; the content of the invention of the present invention does not mean to attempt to limit the key features and essential technical features of the technical solution claimed for protection, nor does it mean to attempt to determine the scope of protection of the technical solution claimed for protection.

[0004] To at least partially solve the above problems, the present invention provides a method for completing consistent objects in a monocular video, comprising: S10, obtaining a video to be processed in a monocular video, segmenting and annotating it using a pre-trained segmentation model, filtering out object candidate areas blocked by other objects, and obtaining all video frame data and mask images containing the complete target object; S20, searching for the starting coverage position of the first occluder image in the starting frame of the video according to the screened complete target object; dynamically adjusting the size and center of the occluder according to the movement of the object bounding box in the video to obtain a large-scale occluded-unoccluded video pair data set; S30, compressing the video to be processed, the visible part mask sequence and the target object video with noise into the same latent space; constructing an adjusted stable video diffusion model, predicting the noise items to be removed and using the true value for supervision; constructing a video object completion model to achieve time-space consistent video object completion; S40 builds a phased completion and image fusion long video processing model to process long videos; improves processing efficiency and accuracy through iterative test and training strategies; and obtains a complete video of the final target object without occlusion through reasoning.

[0005] Preferably, S10 includes: S101, obtaining a video to be processed in a monocular video, using object mask annotation or video shooting content text, and segmenting the target object and the object mask using a pre-trained segmentation model; S102, filtering out object candidate areas blocked by other objects through heuristic rules and annotations, thereby obtaining all video frame data containing the complete target object in the video to be processed, and the mask image of the target object in all video frame data.

[0006] Preferably, S20 includes: S201, randomly selecting a first occluder picture from the candidate occluder pictures according to the screened complete target object; searching for a starting coverage position of the first occluder picture in the starting frame of the video; S202, dynamically adjust the size and center of the occluder according to the movement of the bounding box of the object in the video, so that it can continuously and completely cover the original complete target object and uniformly cover the first occluder; select the next target object to uniformly cover the first occluder; obtain a large-scale occluded-unoccluded video pair data set, and obtain a large-scale data pipeline construction architecture; construct an occluded and unoccluded video pair data set around a certain object from image data sets in multiple monocular videos; The initial coverage position includes: being able to maintain the original complete target object, still having at least 30% visible after coverage, and no more than 70% invisible.

[0007] Preferably, S30 includes: S301, compressing the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space using a pre-trained variational autoencoder to obtain latent space compressed mixed data; S302, constructing an adjusted stable video diffusion model; after latent space compressed mixed data is connected, it is input into the adjusted stable video diffusion model, and the noise items to be removed are predicted and supervised by using the true value; S303, constructing a video object completion model by using a large-scale data pipeline construction architecture and a visible part mask obtained from an input video, and performing adjustment and completion on a video pair dataset to achieve temporally and spatially consistent video object completion; Constructing the adjusted stable video diffusion model includes: adjusting the pre-trained stable video diffusion model, changing the input structure of its U-Net network, so that it can additionally accept the input of the visible part mask image and zero-initialize the parameters of this part, and initialize the remaining parameters with the pre-trained parameters to obtain the adjusted stable video diffusion model; improving the accuracy of downstream tasks of object reconstruction and pose estimation in scenes with occlusion; A video object completion model is constructed by utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion, including: utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, adding a mask image that prompts a target object to a video dataset to be processed based on an adjusted stable video diffusion model, constructing a video object completion model, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion.

[0008] Preferably, S40 includes: S401, constructing a phased completion and image fusion long video processing model, and processing the long video according to the sliding window method; S402, improves processing efficiency and accuracy through iterative test and training strategies; randomly samples a noise and connects it with the input video and the visible part mask, performs step-by-step denoising sampling, and obtains the complete video of the final target object without occlusion through reasoning.

[0009] The present invention provides a consistent object completion system in a monocular video, comprising: The mask segmentation, annotation and filtering subsystem obtains the video to be processed in the monocular video, segments and annotates it using the pre-trained segmentation model, filters out the candidate areas of objects blocked by other objects, and obtains all the video frame data and mask images containing the complete target object; The data pipeline construction subsystem searches for the starting coverage position of the first occluder image in the starting frame of the video based on the screened complete target object; dynamically adjusts the size and center of the occluder according to the movement of the object bounding box in the video to obtain a large-scale occluded-unoccluded video pair dataset; The latent space adjustment and completion subsystem compresses the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space; constructs an adjusted stable video diffusion model to predict the noise items that need to be removed and uses the true value for supervision; constructs a video object completion model to achieve consistent video object completion in time and space; The completion and fusion iterative strategy subsystem builds a phased completion and image fusion long video processing model to process long videos; improves processing efficiency and accuracy through iterative testing and training strategies; and obtains a complete video of the final target object without occlusion through reasoning.

[0010] Preferably, the mask segmentation, annotation and filtering subsystem includes: The target object mask segmentation module obtains the video to be processed in the monocular video, uses the object mask annotation or the video shooting content text, and segments the target object and the object mask using the pre-trained segmentation model; The video data annotation and filtering module filters out the object candidate areas blocked by other objects through heuristic rules and annotations, thereby obtaining all the video frame data containing the complete target object in the video to be processed, and the mask image of the target object in all the video frame data.

[0011] Preferably, the data pipeline construction subsystem includes: The covering position search module randomly selects a first covering object picture from the candidate covering object pictures according to the screened complete target object; searches for the starting covering position of the first covering object picture in the starting frame of the video; The target object consistent coverage module dynamically adjusts the size and center of the occluder according to the movement of the object bounding box in the video, so that it can continuously and completely cover the original complete target object and consistently cover the first occluder; selects the next target object to consistently cover the first occluder; obtains a large-scale occluded-unoccluded video pair data set and obtains a large-scale data pipeline construction architecture; constructs an occluded and unoccluded video pair data set around a certain object from image data sets in multiple monocular videos; The initial coverage position includes: being able to maintain the original complete target object, still having at least 30% visible after coverage, and no more than 70% invisible.

[0012] Preferably, the latent space adjustment and completion subsystem includes: The latent space compression mixing module compresses the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space using a pre-trained variational autoencoder to obtain latent space compressed mixed data. The stable video diffusion adjustment module builds an adjusted stable video diffusion model. After the latent space compressed mixed data is connected, it is input into the adjusted stable video diffusion model to predict the noise items that need to be removed and supervised by the true value. The consistent video object completion module uses a large-scale data pipeline construction architecture and visible part masks obtained from the input video to build a video object completion model, adjust and complete the video pair dataset, and achieve temporally and spatially consistent video object completion; Constructing the adjusted stable video diffusion model includes: adjusting the pre-trained stable video diffusion model, changing the input structure of its U-Net network, so that it can additionally accept the input of the visible part mask image and zero-initialize the parameters of this part, and initialize the remaining parameters with the pre-trained parameters to obtain the adjusted stable video diffusion model; improving the accuracy of downstream tasks of object reconstruction and pose estimation in scenes with occlusion; A video object completion model is constructed by utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion, including: utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, adding a mask image that prompts a target object to a video dataset to be processed based on an adjusted stable video diffusion model, constructing a video object completion model, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion.

[0013] Preferably, the complementary fusion iteration strategy subsystem includes: The completion and fusion long video processing module builds a phased completion and image fusion long video processing model, and processes long videos using a sliding window method; The iterative training strategy inference module improves processing efficiency and accuracy through iterative testing of training strategies. After randomly sampling a noise and connecting it with the input video and the visible part mask, it performs step-by-step denoising sampling and obtains the complete video of the final target object without occlusion through inference.

[0014] The beneficial effects of the above technical solution include: The present invention provides a method and system for completing consistent objects in a monocular video. The method comprises the following steps: obtaining a video to be processed in a monocular video, segmenting and annotating the video using a pre-trained segmentation model, filtering out candidate regions of objects blocked by other objects, and obtaining all video frame data and mask images containing the complete target object; searching for the starting coverage position of the first blocking object image in the starting frame of the video according to the screened complete target object; dynamically adjusting the size and center of the blocking object according to the movement of the object bounding box in the video to obtain a scaled blocked-unblocked video pair data set; compressing the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space; constructing an adjusted stable video diffusion model, predicting the noise items to be removed, and supervising them using true values; constructing a video object completion model to achieve time-space consistent video object completion; constructing a staged completion and image fusion long video processing model to process the long video; and training the video through iterative testing. Strategy to improve processing efficiency and accuracy; obtain a complete video of the final target object without occlusion through reasoning; improve the generalization ability in a large number of scenarios such as robot grasping, unmanned driving, scene reconstruction, and its object reconstruction and pose estimation capabilities for assisted downstream tasks; propose a new framework for consistently completing objects from monocular videos, and use the adjusted stable video diffusion model and the visible part mask obtained from the input video to consistently synthesize the geometry, appearance and motion of the complete object, which improves the generalization ability in a large number of scenarios such as robot grasping, unmanned driving, scene reconstruction, and its ability to assist downstream tasks, including reconstruction and pose estimation; mainly includes four steps: identifying potential complete objects in the video, judging whether they are complete, selecting occluders, and consistently covering occluders in the video; improves the task processing accuracy in the downstream tasks of object reconstruction and pose estimation in scenes with occlusions; has important technical significance and significant effects.

[0015] The present invention provides a method and system for completing consistent objects in a monocular video. Other advantages, objectives and features of the present invention will be partially reflected in the following description, and will also be understood by technicians in this field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 A diagram showing an embodiment of a method and system for completing consistent objects in a monocular video according to the present invention.

[0017] Figure 2 This is a diagram of an application embodiment of a method and system for completing consistent objects in a monocular video of the present invention.

[0018] Figure 3FIG. 2 is another embodiment of a system for completing consistent objects in a monocular video according to the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments so that those skilled in the art can implement the invention with reference to the instructions. As shown in the drawings, the present invention provides a method for completing consistent objects in a monocular video, comprising: S10, obtaining a video to be processed in a monocular video, segmenting and annotating it using a pre-trained segmentation model, filtering out object candidate areas blocked by other objects, and obtaining all video frame data and mask images containing the complete target object; S20, searching for the starting coverage position of the first occluder image in the starting frame of the video according to the screened complete target object; dynamically adjusting the size and center of the occluder according to the movement of the object bounding box in the video to obtain a large-scale occluded-unoccluded video pair data set; S30, compressing the video to be processed, the visible part mask sequence and the target object video with noise into the same latent space; constructing an adjusted stable video diffusion model, predicting the noise items to be removed and using the true value for supervision; constructing a video object completion model to achieve time-space consistent video object completion; S40 builds a phased completion and image fusion long video processing model to process long videos; improves processing efficiency and accuracy through iterative test and training strategies; and obtains a complete video of the final target object without occlusion through reasoning.

[0020] The principle and effect of the above technical solution are as follows: the present invention provides a method for completing consistent objects in a monocular video, comprising: obtaining a video to be processed in a monocular video, segmenting and annotating the video using a pre-trained segmentation model, filtering out candidate object areas occluded by other objects, and obtaining all video frame data and mask images containing the complete target object; searching for the starting coverage position of the first occluder image in the starting frame of the video according to the screened complete target object; dynamically adjusting the size and center of the occluder according to the movement of the object bounding box in the video to obtain a scaled occluded-unoccluded video pair data set; compressing the video to be processed, the visible part mask sequence, and the noisy target object video into the same latent space; constructing an adjusted stable video diffusion model, predicting the noise items that need to be removed and using the true value for supervision; and constructing a video object completion model. , achieve time-space consistent video object completion; build a staged completion and image fusion long video processing model to process long videos; improve processing efficiency and accuracy through iterative test and training strategies; obtain a complete video of the final target object without occlusion through reasoning; improve the generalization ability in a large number of scenarios such as robot grasping, unmanned driving, scene reconstruction, and its ability to assist downstream tasks in object reconstruction and pose estimation; propose a new framework for consistently completing objects from monocular videos, using an adjusted stable video diffusion model and a visible part mask obtained from the input video to consistently synthesize the geometry, appearance and motion of the complete object, improving the generalization ability in a large number of scenarios such as robot grasping, unmanned driving, scene reconstruction, and its ability to assist downstream tasks, including reconstruction and pose estimation; overall design and training such as Figure 1 As shown, the data pipeline construction architecture is as follows Figure 2 As shown; it mainly includes four steps: identifying potential complete objects in the video, judging whether they are complete, selecting occluders, and consistently covering the occluders in the video; like Figure 1 As shown in the figure, the video to be processed in the monocular video is obtained, the video and prompt are input, and the pre-trained segmentation model is used for segmentation and annotation; the variational autoencoder and the connection merging model are used to compress the mixed data connection in the latent space; the multimodal pre-trained model and the video denoiser convolutional neural network U-Net are used for mixed processing, and the mixed processing result is output to the inference model, and the complete object and the complete mask are reconstructed after inference; when adjusting the stable video diffusion model, the sequence of the input video and the visible part mask is compressed and spliced ​​into the noisy latent variable The above are input to the U-Net of the diffusion model together, and a conditional diffusion model is obtained after retraining: in, For a video containing a complete object, is the mask of the visible part of the object of interest, is the input video, f represents the conditional diffusion model operation formula; it improves the task processing accuracy in the downstream tasks of object reconstruction and pose estimation in occluded scenes; it has important technical significance and significant effects.

[0021] In one embodiment, S10 includes: S101, obtaining a video to be processed in a monocular video, using object mask annotation or video shooting content text, and segmenting the target object and the object mask using a pre-trained segmentation model; S102, filtering out object candidate areas blocked by other objects through heuristic rules and annotations, thereby obtaining all video frame data containing the complete target object in the video to be processed, and the mask image of the target object in all video frame data.

[0022] The principle and effect of the above technical solution are: obtaining the video to be processed in the monocular video, using the object mask annotation or the video shooting content text to segment the target object and the object mask using the pre-trained segmentation model; filtering out the object candidate area blocked by other objects through heuristic rules and annotations, so as to obtain all the video frame data containing the complete target object in the video to be processed, and the mask image of the target object in all the video frame data; the heuristic rules include: depth estimation, edge detection or mask quality screening, the depth estimation implementation example is that the unobstructed object usually has a smaller depth value than the surrounding objects that are closer to the camera; the edge detection implementation example is that the object with intersection at the edge of the picture is usually not completely captured by the camera; the mask quality screening implementation example is that the mask with a large number of incomplete holes inside is usually a low-quality annotation.

[0023] In one embodiment, S20 includes: S201, randomly selecting a first occluder picture from the candidate occluder pictures according to the screened complete target object; searching for a starting coverage position of the first occluder picture in the starting frame of the video; S202, dynamically adjust the size and center of the occluder according to the movement of the bounding box of the object in the video, so that it can continuously and completely cover the original complete target object and uniformly cover the first occluder; select the next target object to uniformly cover the first occluder; obtain a large-scale occluded-unoccluded video pair data set, and obtain a large-scale data pipeline construction architecture; construct an occluded and unoccluded video pair data set around a certain object from image data sets in multiple monocular videos; The initial coverage position includes: being able to maintain the original complete target object, still having at least 30% visible after coverage, and no more than 70% invisible.

[0024] The principle and effect of the above technical solution are: according to the screened complete target object, randomly select the first occluder picture from the candidate occluder pictures; search for the starting coverage position of the first occluder picture in the starting frame of the video; according to the movement of the object bounding box in the video, dynamically adjust the size and center of the occluder so that it can continuously and completely cover the original complete target object and consistently cover the first occluder; select the next target object to consistently cover the first occluder; obtain a large-scale occluded-unoccluded video pair data set, and obtain a large-scale data pipeline construction architecture; construct an occluded and unoccluded video pair data set around a certain object from image data sets in multiple monocular videos; the starting coverage position includes: being able to maintain the original complete target object, still having at least 30% visible after covering, and not more than 70% invisible; the pre-trained segmentation model includes the SAM2 arbitrary segmentation model; the variational autoencoder includes the VAE model; the connection merging model includes the Concat model; the multimodal pre-training model includes the contrastive learning language image pre-training CLIP model.

[0025] In one embodiment, S30 includes: S301, compressing the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space using a pre-trained variational autoencoder to obtain latent space compressed mixed data; S302, constructing an adjusted stable video diffusion model; after latent space compressed mixed data is connected, it is input into the adjusted stable video diffusion model, and the noise items to be removed are predicted and supervised by using the true value; S303, constructing a video object completion model by using a large-scale data pipeline construction architecture and a visible part mask obtained from an input video, and performing adjustment and completion on a video pair dataset to achieve temporally and spatially consistent video object completion; Constructing the adjusted stable video diffusion model includes: adjusting the pre-trained stable video diffusion model, changing the input structure of its U-Net network, so that it can additionally accept the input of the visible part mask image and zero-initialize the parameters of this part, and initialize the remaining parameters with the pre-trained parameters to obtain the adjusted stable video diffusion model; improving the accuracy of downstream tasks of object reconstruction and pose estimation in scenes with occlusion; A video object completion model is constructed by utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion, including: utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, adding a mask image that prompts a target object to a video dataset to be processed based on an adjusted stable video diffusion model, constructing a video object completion model, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion.

[0026] The principle and effect of the above technical solution are as follows: the video to be processed, the visible part mask sequence and the target object video with noise are compressed into the same latent space by using a pre-trained variational autoencoder to obtain the latent space compressed mixed data; an adjusted stable video diffusion model is constructed; after the latent space compressed mixed data is connected, it is input into the adjusted stable video diffusion model to predict the noise items to be removed and supervised by the true value; a video object completion model is constructed by using a large-scale data pipeline construction architecture and the visible part mask obtained from the input video, and the video object completion is adjusted and completed on the video pair data set to achieve time-space consistent video object completion; the variational autoencoder includes a VAE model, the output includes the latent variable value and the latent variable mean and variance distribution parameters, and new data is generated by sampling the distribution parameters; Constructing an adjusted stable video diffusion model includes: adjusting the pre-trained stable video diffusion model, changing the input structure of its U-Net network so that it can additionally accept the input of a visible part mask image and zero-initialize this part of the parameters, and initializing the remaining parameters with the pre-trained parameters to obtain the adjusted stable video diffusion model; improving the accuracy of downstream tasks of object reconstruction and pose estimation in scenes with occlusion; using a large-scale data pipeline to construct an architecture and a visible part mask obtained from the input video, constructing a video object completion model, and adjusting and completing on a video pair dataset to achieve spatiotemporally consistent video object completion. The method includes: using a large-scale data pipeline to construct an architecture and a visible part mask obtained from the input video, adding a mask image that prompts the target object to the video dataset to be processed based on the adjusted stable video diffusion model, constructing a video object completion model, and adjusting and completing on the video pair dataset to achieve spatiotemporally consistent video object completion.

[0027] In one embodiment, S40 includes: S401, constructing a phased completion and image fusion long video processing model, and processing the long video according to the sliding window method; S402, improves processing efficiency and accuracy through iterative test and training strategies; randomly samples a noise and connects it with the input video and the visible part mask, performs step-by-step denoising sampling, and obtains the complete video of the final target object without occlusion through reasoning.

[0028] The principle and effect of the above technical solution are: constructing a staged completion and image fusion long video processing model, and processing long videos according to the sliding window method; improving processing efficiency and accuracy through iterative test and training strategies; randomly sampling a noise and connecting it with the input video and the visible part mask, performing step-by-step denoising sampling, and obtaining a complete video of the final target object without occlusion through reasoning; constructing a staged completion and image fusion long video processing model, and processing long videos according to the sliding window method includes: obtaining a total of N frames of the video to be processed, setting the mask image of the object to be completed, first selecting the starting k frames for completion, and reinserting the completed object into the original video; the staged completion and image fusion long video processing model includes: in, is the completed video, M is the object mask obtained based on the completed video, For the original video; To process the model output; ⊙ represents the same or same operation. If the two inputs are the same, the same or same result is true, and if the two inputs are different, the result is false. In the subsequent sliding window model, the last 5 frames of the video from the previous completion will be selected as the common part of the sliding window for further completion. A complete object video with only the object to be completed is output. The geometry, appearance and movement of the complete object are consistently synthesized. Applications include robot grasping, unmanned driving or scene reconstruction, and object reconstruction and pose estimation of downstream auxiliary tasks.

[0029] The present invention provides a consistent object completion system in a monocular video, comprising: The mask segmentation, annotation and filtering subsystem obtains the video to be processed in the monocular video, segments and annotates it using the pre-trained segmentation model, filters out the candidate areas of objects blocked by other objects, and obtains all the video frame data and mask images containing the complete target object; The data pipeline construction subsystem searches for the starting coverage position of the first occluder image in the starting frame of the video based on the screened complete target object; dynamically adjusts the size and center of the occluder according to the movement of the object bounding box in the video to obtain a large-scale occluded-unoccluded video pair dataset; The latent space adjustment and completion subsystem compresses the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space; constructs an adjusted stable video diffusion model to predict the noise items that need to be removed and uses the true value for supervision; constructs a video object completion model to achieve consistent video object completion in time and space; The completion and fusion iterative strategy subsystem builds a phased completion and image fusion long video processing model to process long videos; improves processing efficiency and accuracy through iterative testing and training strategies; and obtains a complete video of the final target object without occlusion through reasoning.

[0030] The principle and effect of the above technical solution are as follows: the present invention provides a consistent object completion system in a monocular video, including: a mask segmentation, annotation and filtering subsystem, which obtains a video to be processed in a monocular video, segments and annotates the video using a pre-trained segmentation model, filters out candidate object areas blocked by other objects, and obtains all video frame data and mask images containing the complete target object; a data pipeline construction subsystem, which searches for the starting coverage position of the first occluder image in the starting frame of the video according to the screened complete target object; dynamically adjusts the size and center of the occluder according to the movement of the object bounding box in the video, and obtains a scaled occluded-unoccluded video pair data set; a latent space adjustment and completion subsystem, which compresses the video to be processed, the visible part mask sequence and the target object video with noise into the same latent space; constructs an adjusted stable video diffusion model, predicts the noise items that need to be removed and uses the true value for supervision; Construct a video object completion model to achieve consistent video object completion in time and space; complete the fusion iteration strategy subsystem, build a phased completion and image fusion long video processing model to process long videos; improve processing efficiency and accuracy through iterative test and training strategies; obtain a complete video of the final target object without occlusion through reasoning; improve the generalization ability in a large number of scenarios such as robot grasping, unmanned driving, scene reconstruction, and its ability to assist downstream tasks in object reconstruction and pose estimation; propose a new framework for consistently completing objects from monocular videos, using an adjusted stable video diffusion model and a visible part mask obtained from the input video to consistently synthesize the geometry, appearance and motion of the complete object, improving the generalization ability in a large number of scenarios such as robot grasping, unmanned driving, scene reconstruction, and its ability to assist downstream tasks, including reconstruction and pose estimation; overall design and training such as Figure 1 As shown, the data pipeline construction architecture is as follows Figure 2 As shown; it mainly includes four steps: identifying potential complete objects in the video, judging whether they are complete, selecting occluders, and consistently covering the occluders in the video; as shown Figure 1 As shown in Figure 2, when adjusting the stable video diffusion model, the sequence of the input video and the visible part mask is compressed and spliced ​​into the noisy latent variable The above are input to the U-Net of the diffusion model together, and a conditional diffusion model is obtained after retraining: in, For a video containing a complete object, is the mask of the visible part of the object of interest, is the input video, f represents the conditional diffusion model operation formula; it improves the task processing accuracy in the downstream tasks of object reconstruction and pose estimation in occluded scenes; it has important technical significance and significant effects.

[0031] In one embodiment, the mask segmentation, annotation and filtering subsystem includes: The target object mask segmentation module obtains the video to be processed in the monocular video, uses the object mask annotation or the video shooting content text, and segments the target object and the object mask using the pre-trained segmentation model; The video data annotation and filtering module filters out the object candidate areas blocked by other objects through heuristic rules and annotations, thereby obtaining all the video frame data containing the complete target object in the video to be processed, and the mask image of the target object in all the video frame data.

[0032] The principle and effect of the above technical solution are: mask segmentation, annotation and filtering, through the target object mask segmentation module, the video to be processed in the monocular video is obtained, and the target object and the object mask are segmented out by the pre-trained segmentation model using the object mask annotation or the video shooting content text; the video data annotation and filtering module, through heuristic rules and annotations, filters out the object candidate area blocked by other objects, so as to obtain all the video frame data containing the complete target object in the video to be processed, and the mask image of the target object in all the video frame data; the heuristic rules include: depth estimation, edge detection or mask quality screening, the depth estimation implementation example is that the unobstructed object usually has a smaller depth value than the surrounding objects that are closer to the camera; the edge detection implementation example is that the object with intersection at the edge of the picture is usually not completely captured by the camera; the mask quality screening implementation example is that the mask with a large number of incomplete holes inside is usually a low-quality annotation.

[0033] In one embodiment, the data pipeline construction subsystem includes: The covering position search module randomly selects a first covering object picture from the candidate covering object pictures according to the screened complete target object; searches for the starting covering position of the first covering object picture in the starting frame of the video; The target object consistent coverage module dynamically adjusts the size and center of the occluder according to the movement of the object bounding box in the video, so that it can continuously and completely cover the original complete target object and consistently cover the first occluder; selects the next target object to consistently cover the first occluder; obtains a large-scale occluded-unoccluded video pair data set and obtains a large-scale data pipeline construction architecture; constructs an occluded and unoccluded video pair data set around a certain object from image data sets in multiple monocular videos; The initial coverage position includes: being able to maintain the original complete target object, still having at least 30% visible after coverage, and no more than 70% invisible.

[0034] The principle and effect of the above technical solution are as follows: data pipeline construction, through the coverage position search module, according to the screened complete target object, randomly select the first occluder picture from the candidate occluder pictures; search the starting coverage position of the first occluder picture in the starting frame of the video; the target object consistent coverage module dynamically adjusts the size and center of the occluder according to the movement of the object bounding box in the video, so that it can continuously and completely cover the original complete target object and consistently cover the first occluder; select the next target object to consistently cover the first occluder; obtain a large-scale occluded-unoccluded video pair data set, and obtain a large-scale data pipeline construction architecture; construct an occluded and unoccluded video pair data set around a certain object from image data sets in multiple monocular videos; the starting coverage position includes: being able to maintain the original complete target object, still having at least 30% visible after covering, and not more than 70% invisible; the pre-trained segmentation model includes the SAM2 arbitrary segmentation model; the variational autoencoder includes the VAE model; the connection merging model includes the Concat model; the multimodal pre-training model includes the contrastive learning language image pre-training CLIP model.

[0035] In one embodiment, the latent space adjustment and completion subsystem includes: The latent space compression mixing module compresses the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space using a pre-trained variational autoencoder to obtain latent space compressed mixed data. The stable video diffusion adjustment module builds an adjusted stable video diffusion model. After the latent space compressed mixed data is connected, it is input into the adjusted stable video diffusion model to predict the noise items that need to be removed and supervised by the true value. The consistent video object completion module uses a large-scale data pipeline construction architecture and visible part masks obtained from the input video to build a video object completion model, adjust and complete the video pair dataset, and achieve temporally and spatially consistent video object completion; Constructing the adjusted stable video diffusion model includes: adjusting the pre-trained stable video diffusion model, changing the input structure of its U-Net network, so that it can additionally accept the input of the visible part mask image and zero-initialize the parameters of this part, and initialize the remaining parameters with the pre-trained parameters to obtain the adjusted stable video diffusion model; improving the accuracy of downstream tasks of object reconstruction and pose estimation in scenes with occlusion; A video object completion model is constructed by utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion, including: utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, adding a mask image that prompts a target object to a video dataset to be processed based on an adjusted stable video diffusion model, constructing a video object completion model, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion.

[0036] The principle and effect of the above technical solution are as follows: latent space adjustment and completion, through the latent space compression and mixing module, the video to be processed, the visible part mask sequence and the target object video with noise are compressed into the same latent space by using the pre-trained variational autoencoder to obtain the latent space compressed mixed data; the stable video diffusion adjustment module builds the adjusted stable video diffusion model; after the latent space compressed mixed data is connected, it is input into the adjusted stable video diffusion model to predict the noise items to be removed and supervised by the true value; the consistent video object completion module uses the large-scale data pipeline construction architecture and the visible part mask obtained from the input video to build a video object completion model, adjust and complete the video pair data set, and realize the video object completion that is consistent in time and space; the variational autoencoder includes a VAE model, the output includes the latent variable value and the latent variable mean and variance distribution parameters, and new data is generated by sampling the distribution parameters; Constructing an adjusted stable video diffusion model includes: adjusting the pre-trained stable video diffusion model, changing the input structure of its U-Net network so that it can additionally accept the input of a visible part mask image and zero-initialize this part of the parameters, and initializing the remaining parameters with the pre-trained parameters to obtain the adjusted stable video diffusion model; improving the accuracy of downstream tasks of object reconstruction and pose estimation in scenes with occlusion; using a large-scale data pipeline to construct an architecture and a visible part mask obtained from the input video, constructing a video object completion model, and adjusting and completing on a video pair dataset to achieve spatiotemporally consistent video object completion. The method includes: using a large-scale data pipeline to construct an architecture and a visible part mask obtained from the input video, adding a mask image that prompts the target object to the video dataset to be processed based on the adjusted stable video diffusion model, constructing a video object completion model, and adjusting and completing on the video pair dataset to achieve spatiotemporally consistent video object completion.

[0037] In one embodiment, the completion fusion iteration strategy subsystem includes: The completion and fusion long video processing module builds a phased completion and image fusion long video processing model, and processes long videos using a sliding window method; The iterative training strategy inference module improves processing efficiency and accuracy through iterative testing of training strategies. After randomly sampling a noise and connecting it with the input video and the visible part mask, it performs step-by-step denoising sampling and obtains the complete video of the final target object without occlusion through inference.

[0038] The principle and effect of the above technical solution are as follows: the completion and fusion iteration strategy subsystem includes: a completion and fusion long video processing module, which constructs a phased completion and image fusion long video processing model, and processes long videos according to the sliding window method; an iterative training strategy reasoning module, which improves processing efficiency and accuracy through iterative testing and training strategies; after randomly sampling a noise and connecting it with the input video and the visible part mask, it performs step-by-step denoising sampling, and obtains the complete video of the final target object without occlusion through reasoning; constructs a phased completion and image fusion long video processing model, and processes long videos according to the sliding window method, including: obtaining a total of N frames of the video to be processed, setting the mask image of the object to be completed, first selecting the starting k frames for completion, and reinserting the completed object into the original video; the phased completion and image fusion long video processing model includes: in, is the completed video, M is the object mask obtained based on the completed video, For the original video; To process the model output; ⊙ represents the same or same operation. If the two inputs are the same, the same or same result is true, and if the two inputs are different, the result is false. In the subsequent sliding window model, the last 5 frames of the video from the previous completion will be selected as the common part of the sliding window for further completion. A complete object video with only the object to be completed is output. The geometry, appearance and movement of the complete object are consistently synthesized. Applications include robot grasping, unmanned driving or scene reconstruction, and object reconstruction and pose estimation of downstream auxiliary tasks.

[0039] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for completing consistent objects in a monocular video, characterized in that: include: S10, obtaining a video to be processed in a monocular video, segmenting and annotating it using a pre-trained segmentation model, filtering out object candidate areas blocked by other objects, and obtaining all video frame data and mask images containing the complete target object; S20, searching for the starting coverage position of the first occluder image in the starting frame of the video according to the screened complete target object; dynamically adjusting the size and center of the occluder according to the movement of the object bounding box in the video to obtain a large-scale occluded-unoccluded video pair data set; S30, compressing the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space; Build an adjusted stable video diffusion model to predict the noise items that need to be removed and use the true value for supervision; build a video object completion model to achieve spatiotemporally consistent video object completion; S40, builds a phased completion and image fusion long video processing model to process long videos; Improve processing efficiency and accuracy through iterative test training strategies; Through reasoning, the final complete video of the target object without occlusion is obtained.

2. A method for completing consistent objects in a monocular video according to claim 1, characterized in that: S10 includes: S101, obtaining a video to be processed in a monocular video, using object mask annotation or video shooting content text, and segmenting the target object and the object mask using a pre-trained segmentation model; S102, filtering out object candidate areas blocked by other objects through heuristic rules and annotations, thereby obtaining all video frame data containing the complete target object in the video to be processed, and the mask image of the target object in all video frame data.

3. The method for completing consistent objects in a monocular video according to claim 1, characterized in that: The S20 includes: S201, randomly selecting a first occluder picture from the candidate occluder pictures according to the screened complete target object; searching for a starting coverage position of the first occluder picture in the starting frame of the video; S202, dynamically adjust the size and center of the occluder according to the movement of the bounding box of the object in the video, so that it can continuously and completely cover the original complete target object and uniformly cover the first occluder; select the next target object to uniformly cover the first occluder; obtain a large-scale occluded-unoccluded video pair data set, and obtain a large-scale data pipeline construction architecture; construct an occluded and unoccluded video pair data set around a certain object from image data sets in multiple monocular videos; The initial coverage position includes: being able to maintain the original complete target object, still having at least 30% visible after coverage, and no more than 70% invisible.

4. The method for completing consistent objects in a monocular video according to claim 1, characterized in that: S30 includes: S301, compressing the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space using a pre-trained variational autoencoder to obtain latent space compressed mixed data; S302, constructing an adjusted stable video diffusion model; after latent space compressed mixed data is connected, it is input into the adjusted stable video diffusion model, and the noise items to be removed are predicted and supervised by using the true value; S303, constructing a video object completion model by using a large-scale data pipeline construction architecture and a visible part mask obtained from an input video, and performing adjustment and completion on a video pair dataset to achieve temporally and spatially consistent video object completion; Constructing the adjusted stable video diffusion model includes: adjusting the pre-trained stable video diffusion model, changing the input structure of its U-Net network, so that it can additionally accept the input of the visible part mask image and zero-initialize the parameters of this part, and initialize the remaining parameters with the pre-trained parameters to obtain the adjusted stable video diffusion model; improving the accuracy of downstream tasks of object reconstruction and pose estimation in scenes with occlusion; A video object completion model is constructed by utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion, including: utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, adding a mask image that prompts a target object to a video dataset to be processed based on an adjusted stable video diffusion model, constructing a video object completion model, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion.

5. The method for completing consistent objects in a monocular video according to claim 1, characterized in that: S40 includes: S401, constructing a phased completion and image fusion long video processing model, and processing the long video according to the sliding window method; S402, improves processing efficiency and accuracy through iterative test and training strategies; randomly samples a noise and connects it with the input video and the visible part mask, performs step-by-step denoising sampling, and obtains the complete video of the final target object without occlusion through reasoning.

6. A consistent object completion system in a monocular video, characterized in that: include: The mask segmentation, annotation and filtering subsystem obtains the video to be processed in the monocular video, segments and annotates it using the pre-trained segmentation model, filters out the candidate areas of objects blocked by other objects, and obtains all the video frame data and mask images containing the complete target object; The data pipeline construction subsystem searches for the starting coverage position of the first occluder image in the starting frame of the video based on the screened complete target object; dynamically adjusts the size and center of the occluder according to the movement of the object bounding box in the video to obtain a large-scale occluded-unoccluded video pair dataset; The latent space adjustment and completion subsystem compresses the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space; Build an adjusted stable video diffusion model to predict the noise items that need to be removed and use the true value for supervision; build a video object completion model to achieve spatiotemporally consistent video object completion; The completion and fusion iteration strategy subsystem builds a phased completion and image fusion long video processing model to process long videos; Improve processing efficiency and accuracy through iterative testing and training strategies; obtain a complete video of the final target object without occlusion through reasoning.

7. A consistent object completion system in a monocular video according to claim 6, characterized in that: The mask segmentation, annotation and filtering subsystem includes: The target object mask segmentation module obtains the video to be processed in the monocular video, uses the object mask annotation or the video shooting content text, and segments the target object and the object mask using the pre-trained segmentation model; The video data annotation and filtering module filters out the object candidate areas blocked by other objects through heuristic rules and annotations, thereby obtaining all the video frame data containing the complete target object in the video to be processed, and the mask image of the target object in all the video frame data.

8. The consistent object completion system in monocular video according to claim 6, characterized in that: The data pipeline construction subsystem includes: The covering position search module randomly selects a first covering object picture from the candidate covering object pictures according to the screened complete target object; searches for the starting covering position of the first covering object picture in the starting frame of the video; The target object consistent coverage module dynamically adjusts the size and center of the occluder according to the movement of the object bounding box in the video, so that it can continuously and completely cover the original complete target object and consistently cover the first occluder; selects the next target object to consistently cover the first occluder; obtains a large-scale occluded-unoccluded video pair data set and obtains a large-scale data pipeline construction architecture; constructs an occluded and unoccluded video pair data set around a certain object from image data sets in multiple monocular videos; The initial coverage position includes: being able to maintain the original complete target object, still having at least 30% visible after coverage, and no more than 70% invisible.

9. The consistent object completion system in monocular video according to claim 6, characterized in that: Hidden space adjustment and completion subsystem, including: The latent space compression mixing module compresses the video to be processed, the visible part mask sequence, and the target object video with noise into the same latent space using a pre-trained variational autoencoder to obtain latent space compressed mixed data. The stable video diffusion adjustment module builds an adjusted stable video diffusion model. After the latent space compressed mixed data is connected, it is input into the adjusted stable video diffusion model to predict the noise items that need to be removed and supervised by the true value. The consistent video object completion module uses a large-scale data pipeline construction architecture and visible part masks obtained from the input video to build a video object completion model, adjust and complete the video pair dataset, and achieve temporally and spatially consistent video object completion; Constructing the adjusted stable video diffusion model includes: adjusting the pre-trained stable video diffusion model, changing the input structure of its U-Net network, so that it can additionally accept the input of the visible part mask image and zero-initialize the parameters of this part, and initialize the remaining parameters with the pre-trained parameters to obtain the adjusted stable video diffusion model; improving the accuracy of downstream tasks of object reconstruction and pose estimation in scenes with occlusion; A video object completion model is constructed by utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion, including: utilizing a large-scale data pipeline to construct an architecture and a visible part mask obtained from an input video, adding a mask image that prompts a target object to a video dataset to be processed based on an adjusted stable video diffusion model, constructing a video object completion model, and adjustment and completion are performed on a video pair dataset to achieve spatiotemporally consistent video object completion.

10. The consistent object completion system in monocular video according to claim 6, characterized in that: Complete the fusion iteration strategy subsystem, including: The completion and fusion long video processing module builds a phased completion and image fusion long video processing model, and processes long videos using a sliding window method; The iterative training strategy inference module improves processing efficiency and accuracy through iterative testing of training strategies. After randomly sampling a noise and connecting it with the input video and the visible part mask, it performs step-by-step denoising sampling and obtains the complete video of the final target object without occlusion through inference.

Images