Scene object occlusion completion method and device based on spatiotemporal joint prior knowledge

Abstract

The present invention discloses a scene object occlusion completion method and device based on spatiotemporal joint prior knowledge: step 1, dividing a video into a training set and a verification set; step 2, constructing a scene object occlusion completion model, the scene object occlusion completion model including a space-time encoding module, an occlusion completion module and an optical flow decoding module; step 3, inputting the training set into the scene object occlusion completion model for training to obtain a trained scene object occlusion completion model; step 4, inputting the image to be completed or the video to be completed into the trained scene object occlusion completion model to obtain the corresponding completed graphics or videos. The present invention has stronger generalization ability, and in addition, it can not only handle object occlusion completion in static scenes, but also handle object occlusion problems in dynamic scenes, thereby improving the quality and accuracy of occlusion completion.

Description

Technical Field

[0001] The present invention belongs to the field of image and video processing and restoration, and relates to a scene object occlusion completion method and device based on spatiotemporal joint prior knowledge. The method and device use temporal motion information and features of adjacent areas in the scene to predict and complete occluded objects in images and videos. Background Art

[0002] Scene object occlusion completion aims to address the common problem of object occlusion in input images or video scenes, and aims to perceive, reason and restore the complete shape of objects when they are partially occluded. By completing occluded objects in images or videos, the intelligent agent can interact with the environment to obtain information, understand problems, make decisions and take actions, which plays a vital role in the generation of intelligent behavior and adaptability of the new generation of artificial intelligence systems.

[0003] Although deep learning methods have made significant progress in solving the problem of object occlusion completion, some challenges and limitations still exist. For example, inaccurate inference or incomplete completion may occur for complex occlusion situations and scenes with multiple overlapping objects. Most current methods mainly use shape statistics to model type priors, which makes it difficult to extend to open-world applications where the distribution of object categories is long-tailed. In addition, these models often use simpler datasets when testing, and still face challenges in self-supervised object discovery in complex scenes in the real world.

[0004] Patent applications with publication numbers CN117541510A and CN118069350A both involve methods for completing missing parts of scenes. The former obtains target scene image data, segments object contours, makes difference judgments on pixel points, replaces unqualified pixels, and generates contours of occluded areas of the image. However, this method models at the image level, and has poor effects when processing dynamic scenes or non-standard objects; it relies on type priors, and the model cannot process objects of categories that do not appear in the data set, and its generalization ability is limited. The latter models at the video level, and proposes edge detection of the optical flow map to obtain the edges of the optical flow map; an optical flow image completion network is used to complete the edges of the optical flow map and the missing optical flow to obtain the final completed image. However, this scheme only completes through optical flow maps and edge information, and cannot autonomously learn the spatiotemporal priors and object motion information of objects from video sequences. This limits the model's adaptability to different object categories and scenes, and its generalization performance is weak.

[0005] Therefore, how to design an occlusion completion method that performs well in dealing with rare categories of objects, complex scenes, and multi-object occlusion, and has high quality, high precision, and good generalization ability, is an important issue in the current field of image and video processing. Summary of the invention

[0006] The purpose of the present invention is to provide a method and device for object occlusion completion based on spatiotemporal joint prior knowledge, which solves the problems of low generalization and low accuracy of existing methods when dealing with rare categories of objects, complex scenes and multi-object occlusion.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] On the one hand, the present invention provides a scene object occlusion completion method based on spatiotemporal joint prior knowledge, comprising the following steps:

[0009] Step 1: Input the video and divide it into training set and verification set;

[0010] Step 2, constructing a scene object occlusion completion model, the scene object occlusion completion model includes a space-time encoding module, an occlusion completion module and an optical flow decoding module, the output end of the space-time encoding module is respectively connected to the input end of the occlusion completion module and the optical flow decoding module, and the output of the occlusion completion module and the optical flow decoding module are jointly used as the output of the scene object occlusion completion model; the space-time encoding module includes a data preprocessing, a convolutional layer sequence, a linear layer and an LSTM layer connected in sequence; the occlusion completion module includes a downsampling network, flattening and linear transformation, feature splicing, a linear layer and an upsampling network connected in sequence; the optical flow decoding module includes a downsampling network, flattening and linear transformation, feature splicing, feature processing and an upsampling network connected in sequence;

[0011] Step 3, inputting the training set into the scene object occlusion completion model for training to obtain a trained scene object occlusion completion model;

[0012] Step 4: Input the image to be completed or the video to be completed into the trained scene object occlusion completion model to obtain the corresponding completed image or video.

[0013] On the other hand, the present invention provides a scene object occlusion completion device based on spatiotemporal joint prior knowledge, which specifically includes the following modules:

[0014] The video input module is used to input videos and divide them into training sets and verification sets;

[0015] A model construction module is used to construct a scene object occlusion completion model, which includes a space-time encoding module, an occlusion completion module and an optical flow decoding module, wherein the output end of the space-time encoding module is respectively connected to the input end of the occlusion completion module and the optical flow decoding module, and the output of the occlusion completion module and the optical flow decoding module are jointly used as the output of the scene object occlusion completion model; the space-time encoding module includes a data preprocessing, a convolutional layer sequence, a linear layer and an LSTM layer connected in sequence; the occlusion completion module includes a downsampling network, a flattening and linear transformation, a feature splicing, a linear layer and an upsampling network connected in sequence; the optical flow decoding module includes a downsampling network, a flattening and linear transformation, a feature splicing, a feature processing and an upsampling network connected in sequence;

[0016] A training module, used for inputting a training set into a scene object occlusion completion model for training, to obtain a trained scene object occlusion completion model;

[0017] The completion module is used to input the image to be completed or the video to be completed into the trained scene object occlusion completion model to obtain the corresponding completed image or video.

[0018] Compared with the prior art, the method of the present invention has the following beneficial effects:

[0019] (1) By designing a space-time encoding module in the model, rich object features and motion information can be learned from the video, thereby improving the quality and accuracy of occlusion completion; space-time encoding can better understand and utilize the spatiotemporal information in the video, thereby improving the performance of video analysis; this method does not require explicit visual correspondence between pixels, but uses temporal information to model motion and complete dense object motion prediction; the advantage of this design is that it can autonomously learn the spatiotemporal priors of objects from video sequences, rather than relying solely on fixed object category priors.

[0020] (2) By designing an occlusion completion module in the model, we use the spatiotemporal continuity and the relationship between adjacent frame images to generate training supervision signals to learn the occluded parts of objects for processing video sequences and generating target segmentation masks. This module can capture features of different scales through the process of downsampling and upsampling, which helps to consider objects or regions of different sizes when generating masks; and enables the model to handle complex scene changes such as occlusion, lighting changes, and different backgrounds.

[0021] (3) By designing an optical flow decoding module in the model, the encoded optical flow is converted into a feature vector, and the feature vector and latent variables are used to reconstruct the complete object motion information through upsampling. This module is designed mainly to generate optical flow, that is, to describe the movement of pixels in an image between consecutive frames. The advantage of this design is that it can accurately capture video features; it enables the model to combine global information and local motion information to generate more accurate optical flow; through multi-layer convolution and deconvolution operations, it can handle complex scene changes, such as fast motion, occlusion, and lighting changes.

[0022] (4) The architecture of the model includes a space-time encoding module, an occlusion completion module, and an optical flow decoding module. The two decoders, the occlusion completion module and the optical flow decoding module, simultaneously receive the output of the space-time encoding module as input. The input data is decoded through the corresponding network layer to reconstruct the complete mask and flow. The combination of the three modules can not only handle object occlusion completion in static scenes, but also deal with object occlusion problems in dynamic scenes, with good generalization ability and applicability. Through spatiotemporal learning, the model can learn rich object features and motion information from video data, and can roughly learn the deformation of the visible parts in the past by using spatiotemporal continuity, convert the visible parts in other frames into self-supervisory signals, and generate training supervision signals by constructing dense object motion between frames to study the relationship between the visible parts in adjacent frames, thereby improving the quality and accuracy of the model's occlusion completion.

[0023] (5) During the model training process, the present invention introduces occlusion consistency loss and temporal consistency loss to ensure that the generated occlusion mask is consistent with the actual visible mask and maintains the consistency of predictions between different time frames. This can not only handle object occlusion completion in static scenes, but also deal with object occlusion problems in dynamic scenes.

[0024] In summary, compared with existing algorithms, the present invention can autonomously learn the spatiotemporal priors of objects from video sequences, rather than relying solely on fixed object category priors. This allows the model to adapt to objects of different categories, and can handle object occlusion completion not only in static scenes, but also in dynamic scenes. It is more versatile and can be applied to a variety of different scenes and data sets. Compared with some methods that may rely on modal information or specific scenes, the present invention has stronger generalization capabilities. This method can learn rich object features and motion information from video data, thereby improving the quality and accuracy of occlusion completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural diagram of the space-time coding module;

[0026] Figure 2It is the structural diagram of the occlusion completion module;

[0027] Figure 3 It is the structural diagram of the optical flow decoding module;

[0028] Figure 4 It is a schematic diagram of the shape completion result of the occluded part of the image;

[0029] Figure 5 This is a schematic diagram of the shape completion result of the occluded part of the video. DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Example 1

[0032] The object occlusion completion method based on spatiotemporal prior provided in this embodiment includes the following steps:

[0033] Step 1: Input the video and divide it into training set and validation set.

[0034] Step 2, construct a scene object occlusion completion model, the scene object occlusion completion model includes a space-time encoding module, an occlusion completion module and an optical flow decoding module, wherein the output end of the space-time encoding module is respectively connected to the input end of the occlusion completion module and the optical flow decoding module, and the output of the occlusion completion module and the optical flow decoding module are jointly used as the output of the scene object occlusion completion model.

[0035] (1) The space-time encoding module includes a sequentially connected data preprocessing, a sequence of convolutional layers (including 6 sequentially connected convolutional layers), a linear layer, and an LSTM layer, such as Figure 1 As shown. Among them:

[0036] Data preprocessing is to extract feature tensors from the input video. The specific process is as follows: obtain feature tensors from each frame of the input video, including image blocks, segmented visible parts and optical flow information.

[0037] The sequence of convolutional layers is used to adjust the shape of the feature tensor and encode the adjusted feature tensor to obtain the encoded feature tensor.

[0038] The linear layer reshapes the encoded feature tensor into spatiotemporal features of the original sequence length, and the LSTM layer encodes the spatiotemporal features to obtain the encoded data and encoded optical flow.

[0039] Through the above design, the space-time encoding module learns rich object features and motion information from the video, thereby improving the quality and accuracy of occlusion completion; space-time encoding can better understand and utilize the spatiotemporal information in the video, thereby improving the performance of video analysis; this method does not require explicit visual correspondence between pixels, but uses temporal information to model motion and complete dense object motion prediction; the advantage of this design is that it can autonomously learn the spatiotemporal priors of objects from video sequences, rather than relying solely on fixed object category priors.

[0040] (2) The occlusion completion module includes a downsampling network, flattening and linear transformation, feature concatenation, linear layer and upsampling network connected in sequence; the downsampling network is composed of multiple convolutional layers, group normalization layers and activation functions connected in sequence. The downsampling network is used to extract features from the encoded data output by the space-time encoding module; the upsampling network is used to reconstruct the complete mask of each object predicted in each frame of the image through a deconvolution operation using the feature vector output by the linear layer, so that the mask matches the input.

[0041] Based on the above design, the occlusion completion module uses the relationship between spatiotemporal continuity and adjacent frame images to generate training supervision signals to learn the occluded parts of objects (objects) for processing video sequences and generating target segmentation masks. This module can capture features of different scales through the process of downsampling and upsampling, which helps to consider objects or regions of different sizes when generating masks; and enables the model to handle complex scene changes such as occlusion, lighting changes, and different backgrounds.

[0042] (3) Optical flow decoding module, including a downsampling network, flattening and linear transformation, feature concatenation, feature processing and upsampling network connected in sequence; the downsampling network is composed of multiple convolutional layers, group normalization layers and activation functions connected in sequence. The downsampling network is used to extract features from the encoded optical flow output of the spatial-temporal encoding module; the upsampling network is used to reconstruct the dense motion of the mask of each object in each frame of the image through deconvolution operations on the feature vector, so that the object motion information matches the input.

[0043] Based on the above design, the optical flow decoding module converts the encoded optical flow into a feature vector, and uses the feature vector and latent variables to reconstruct the complete object motion information through upsampling. The design of this module is mainly used to generate optical flow, that is, to describe the movement of pixels in the image between consecutive frames. The advantage of this design is that it can accurately capture video features; it enables the model to combine global information and local motion information to generate more accurate optical flow; through multi-layer convolution and deconvolution operations, it can handle complex scene changes, such as fast motion, occlusion, and lighting changes.

[0044] The two decoders, the occlusion completion module and the optical flow decoding module, simultaneously receive the output of the space-time encoding module as input. The input data is decoded through the corresponding network layer to reconstruct the complete mask and flow. The combination of these three modules can not only handle object occlusion completion in static scenes, but also deal with object occlusion problems in dynamic scenes, with good generalization ability and applicability. Through spatiotemporal learning, the model can learn rich object features and motion information from video data, and can roughly learn the deformation of the visible parts in the past by using spatiotemporal continuity, convert the visible parts in other frames into self-supervisory signals, and generate training supervision signals by constructing dense object motion between frames to study the relationship between visible parts in adjacent frames, ultimately improving the quality and accuracy of the model's occlusion completion.

[0045] Step 3: Input the training set obtained in step 1 into the scene object occlusion completion model constructed in step 2 for training to obtain a trained scene object occlusion completion model.

[0046] Step 3 specifically includes the following sub-steps:

[0047] Step 3.1: Input the training set obtained in step 1 into the scene object occlusion completion model constructed in step 2 to obtain the first t Frame image prediction object k The complete mask of , No. t Objects in frame images k Intensive movement of the mask .in, t Represents the frame number in the video of the training set, T Represents the number of frames in the video of the training set, 0≤ t ≤ T ; k Indicates t The ordinal number of the object in the frame image, K Indicates t The number of objects in the frame image, K is a variable, the number of objects in different images is different, 0≤ k ≤ K .

[0048] Step 3.2, use the Warp Function to process the result of step 3.1 to obtain t +1 frame of the image object k The estimated value of the mask ,and satisfy:

[0049]

[0050] Where:

[0051] —No. t +1 frame of the image object k The estimated value of the mask;

[0052] (x, y) — coordinates of the current position;

[0053] , —No. t Frame image object k The horizontal and vertical movement of the predicted value of the mask is obtained by step 3.1. t Objects in frame images k Intensive movement of the mask Decomposed from two directions;

[0054] —No. t Frame image prediction object k The complete mask of is obtained from step 3.1.

[0055] Step 3.3, based on the result of step 3.2, calculate the value of the loss function.

[0056] In this embodiment, the loss function is as follows:

[0057] ;

[0058] Where:

[0059] — loss function;

[0060] —Occlusion consistency loss;

[0061] —loss of temporal consistency;

[0062] , —The weight coefficient of occlusion consistency loss and the weight coefficient of temporal consistency loss, + =1; In this embodiment, , Take 0.2 and 0.8 respectively;

[0063] ;

[0064] ;

[0065] Where:

[0066] — distance function;

[0067] —No. t +1 frame of the image object k The estimated value of the mask is obtained in step 3.2;

[0068] —No. t +The visible part of the segmented 1-frame image is obtained by data preprocessing of the space-time encoding module of the scene object occlusion completion model;

[0069] —Distance metric function;

[0070] —No. t Frame image prediction object k The complete mask of is obtained from step 3.1;

[0071] —No. t Frame image object k The estimated value of the mask is obtained in step 3.2.

[0072] In step 3.3, the occlusion consistency loss Aims to ensure that the generated occlusion mask is consistent with its corresponding visible mask. This loss quantifies the difference between the generated occlusion mask and the true visible mask to encourage the model to learn to generate more accurate occlusion masks. It is a loss function used to ensure that the model generates consistent predictions in time series data. By introducing temporal consistency loss, the model's predictions for each frame can be made consistent not only with the visible information of the current frame, but also with the complete mask predictions of the objects in the previous and next frames. The loss function used in this step ensures that the model can accurately predict the mask of the visible part, while maintaining consistency for the occluded part between different time frames.

[0073] In step 3.4, the training set is trained multiple times in an iterative manner. Steps 3.1-3.3 are executed in each round, and back-propagation is performed to update the model parameters. When the loss function value is minimum and stable, the trained scene object occlusion completion model is obtained in the current round.

[0074] Step 4, input the image to be completed or the video to be completed into the trained scene object occlusion completion model obtained in step 3, and obtain the corresponding completed image or video. It should be noted that when the video to be completed enters the trained model for processing, each frame of the image is input into the model to obtain an output image, and then the output images are spliced ​​in sequence to obtain the completed video.

[0075] In order to demonstrate the feasibility and effectiveness of the method of the present invention, the constructed model is inferred on the FISHBOWL dataset below, using video sequences to predict the complete shape of the object (the object is the object in the image) and the mask of the occluded part.

[0076] Compared with the Convex and Self-Deocclusion methods, the model constructed in the present invention performs well in mIoU (mean intersection over union) of both the complete shape and the occluded part, demonstrating its excellent generalization ability on unseen images or videos.

[0077] The scene object occlusion completion model constructed by the present invention is trained and tested on the public FISHBOWL dataset, which contains videos recorded in the public WebGL aquarium demonstration. The specific effect of completing the occluded objects in the video frame is as follows: Figure 4 As shown in the figure, the dataset contains 10,000 training videos and 1,000 validation and test videos. Each video contains 128 frames with a resolution of 480×320 pixels.

[0078] As can be seen from Table 1, when comparing the mIoU of different unsupervised algorithms on the FISHBOWL dataset, the Convex algorithm performs better in terms of complete shape mIoU, but worse in terms of occluded part mIoU, and is more suitable for overall object recognition; the Self-Deocclusion algorithm performs well in terms of complete shape mIoU and occluded part mIoU, but there is still room for improvement; the method proposed in the present invention achieves the best performance in terms of complete shape mIoU and occluded part mIoU, and the model has a good application effect on unseen pictures or videos, such as Figure 5 As shown, it is shown that the model constructed in the present invention has good generalization ability.

[0079] Table 1 IoU comparison of different algorithms on FISHBOWL dataset

[0080]

[0081] Example 2

[0082] This embodiment provides a scene object occlusion completion device based on spatiotemporal joint prior knowledge, which specifically includes the following modules:

[0083] The video input module is used to input videos and divide them into training sets and verification sets;

[0084] A model construction module is used to construct a scene object occlusion completion model, which includes a space-time encoding module, an occlusion completion module and an optical flow decoding module, wherein the output end of the space-time encoding module is respectively connected to the input end of the occlusion completion module and the optical flow decoding module, and the output of the occlusion completion module and the optical flow decoding module are jointly used as the output of the scene object occlusion completion model; the space-time encoding module includes a data preprocessing, a convolutional layer sequence, a linear layer and an LSTM layer connected in sequence; the occlusion completion module includes a downsampling network, a flattening and linear transformation, a feature splicing, a linear layer and an upsampling network connected in sequence; the optical flow decoding module includes a downsampling network, a flattening and linear transformation, a feature splicing, a feature processing and an upsampling network connected in sequence;

[0085] A training module, used for inputting a training set into a scene object occlusion completion model for training, to obtain a trained scene object occlusion completion model;

[0086] The completion module is used to input the image to be completed or the video to be completed into the trained scene object occlusion completion model to obtain the corresponding completed image or video.

Claims

1. A scene object occlusion completion method based on spatiotemporal joint prior knowledge, characterized in that: The steps include: Step 1: Input the video and divide it into training set and verification set; Step 2, constructing a scene object occlusion completion model, the scene object occlusion completion model includes a space-time encoding module, an occlusion completion module and an optical flow decoding module, the output end of the space-time encoding module is respectively connected to the input end of the occlusion completion module and the optical flow decoding module, and the output of the occlusion completion module and the optical flow decoding module are jointly used as the output of the scene object occlusion completion model; the space-time encoding module includes a data preprocessing, a convolutional layer sequence, a linear layer and an LSTM layer connected in sequence; the occlusion completion module includes a downsampling network, flattening and linear transformation, feature splicing, a linear layer and an upsampling network connected in sequence; the optical flow decoding module includes a downsampling network, flattening and linear transformation, feature splicing, feature processing and an upsampling network connected in sequence; Step 3, inputting the training set into the scene object occlusion completion model for training to obtain a trained scene object occlusion completion model; Step 4: Input the image to be completed or the video to be completed into the trained scene object occlusion completion model to obtain the corresponding completed image or video.

2. The scene object occlusion completion method based on spatiotemporal joint prior knowledge as claimed in claim 1, characterized in that: The convolutional layer sequence in the space-time encoding module consists of 6 convolutional layers connected in sequence.

3. The scene object occlusion completion method based on spatiotemporal joint prior knowledge as claimed in claim 1, characterized in that: The downsampling network in the occlusion completion module is composed of multiple convolutional layers, group normalization layers and activation functions connected in sequence.

4. The scene object occlusion completion method based on spatiotemporal joint prior knowledge according to claim 1, characterized in that: The downsampling network in the optical flow decoding module is composed of multiple convolutional layers, group normalization layers and activation functions connected in sequence.

5. The scene object occlusion completion method based on spatiotemporal joint prior knowledge as claimed in claim 1, characterized in that: Step 3 specifically includes the following sub-steps: Step 3.1: Input the training set obtained in step 1 into the scene object occlusion completion model constructed in step 2 to obtain the first t Frame image prediction object k The complete mask of , No. t Objects in frame images k Intensive movement of the mask ;in, t Represents the frame number in the video of the training set, T Represents the number of frames in the video of the training set, 0≤ t ≤ T ; k Indicates t The ordinal number of the object in the frame image, K Indicates t The number of objects in the frame image, 0≤ k ≤ K ; Step 3.2, use the Warp Function to process the result of step 3.1 to obtain t +1 frame of the image object k The estimated value of the mask ,and satisfy: Where: —No. t +1 frame of the image object k The estimated value of the mask; (x, y) — coordinates of the current position; , —No. t Frame image object k Horizontal movement and vertical movement of the predicted value of the mask; —No. t Frame image prediction object k The complete mask of Step 3.3, based on the result of step 3.2, calculate the value of the loss function, the loss function is as follows: ； Where: — loss function; —Occlusion consistency loss; —loss of temporal consistency; , —The weight coefficient of occlusion consistency loss and the weight coefficient of temporal consistency loss, + =1; ； ； Where: — distance function; —No. t +1 frame of the image object k The estimated value of the mask; —No. t +1 visible part of the image segmented; —Distance metric function; —No. t Frame image prediction object k The complete mask of —No. t Frame image object k The estimated value of the mask; In step 3.4, the training set is trained multiple times in an iterative manner. Steps 3.1-3.3 are executed in each round, and back-propagation is performed to update the model parameters. When the loss function value is minimum and stable, the trained scene object occlusion completion model is obtained in the current round.

6. The scene object occlusion completion method based on spatiotemporal joint prior knowledge according to claim 5, characterized in that: In step 3.3, , Take 0.2 and 0.8 respectively.

7. A scene object occlusion completion device based on spatiotemporal joint prior knowledge, characterized in that: The modules include: The video input module is used to input videos and divide them into training sets and verification sets; A model construction module is used to construct a scene object occlusion completion model, which includes a space-time encoding module, an occlusion completion module and an optical flow decoding module, wherein the output end of the space-time encoding module is respectively connected to the input end of the occlusion completion module and the optical flow decoding module, and the output of the occlusion completion module and the optical flow decoding module are jointly used as the output of the scene object occlusion completion model; the space-time encoding module includes a data preprocessing, a convolutional layer sequence, a linear layer and an LSTM layer connected in sequence; the occlusion completion module includes a downsampling network, a flattening and linear transformation, a feature splicing, a linear layer and an upsampling network connected in sequence; the optical flow decoding module includes a downsampling network, a flattening and linear transformation, a feature splicing, a feature processing and an upsampling network connected in sequence; A training module, used for inputting a training set into a scene object occlusion completion model for training, to obtain a trained scene object occlusion completion model; The completion module is used to input the image to be completed or the video to be completed into the trained scene object occlusion completion model to obtain the corresponding completed image or video.

Images