Monocular video scene dynamic three-dimensional reconstruction method based on optical flow

Abstract

The invention discloses a monocular video scene dynamic three-dimensional reconstruction method based on optical flow, and relates to the technical field of scene reconstruction. Calculating an optical flow of each pixel in each frame of image in the monocular video, and determining a dynamic region mask of the image; according to the dynamic region mask of each frame of image, determining a plurality of dynamic object instances with consistent time and space; performing four-dimensional Gaussian sputtering conversion on each frame of image to obtain four-dimensional Gaussian distribution representation; for any one dynamic object instance in any one frame of image, acquiring other images containing the dynamic object instance in different image frames, and according to the coordinates of the Gaussian point clouds of the image and other images, generating a visual angle point cloud, which is not observed in the image, of the object instance; and according to color information of each frame of Gaussian point cloud in the four-dimensional Gaussian distribution representation, performing color rendering on each frame of Gaussian point cloud generating the visual angle point cloud to obtain a reconstructed scene. The method can accurately realize scene reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of scene reconstruction, and in particular to a method for dynamic three-dimensional reconstruction of a monocular video scene based on optical flow. Background Art

[0002] With the rapid development of computer vision and deep learning technology, dynamic reconstruction has gradually become a research hotspot.

[0003] Motion reconstruction technology based on monocular video has broad application value across multiple industries, particularly in culture and tourism, film and television production, game development, and intelligent surveillance. Traditional reconstruction methods often rely on multi-view images or complex hardware, which limits their practical application. Reconstructing dynamic objects is particularly challenging due to lighting variations and occlusions, making accurate scene reconstruction a pressing issue. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for dynamic 3D reconstruction of monocular video scenes based on optical flow to address the above technical issues. This method can accurately achieve scene reconstruction.

[0005] The present invention adopts the following technical solutions: The present invention provides a method for dynamic 3D reconstruction of a monocular video scene based on optical flow, comprising: Calculate the optical flow of each pixel in each frame of a monocular video; For any frame image, the dynamic area mask of the image is determined based on the distribution difference between the optical flow of each pixel in the image and the average optical flow of all corresponding pixels; Determine multiple spatiotemporally consistent dynamic object instances based on the dynamic region mask of each frame; each spatiotemporally consistent dynamic object instance represents the same object in multiple frames of the image in a time sequence; Each frame of the image is transformed by three-dimensional Gaussian sputtering to obtain a three-dimensional Gaussian distribution representation. Based on the same dynamic object instance in different image frames, the three-dimensional Gaussian distribution representation of the image is transformed by four-dimensional Gaussian sputtering to obtain a four-dimensional Gaussian distribution representation. The three-dimensional Gaussian distribution representation includes the coordinates, scale, rotation and color information of the Gaussian point cloud. The four-dimensional Gaussian distribution representation has an additional time dimension than the three-dimensional Gaussian distribution representation. For any dynamic object instance in any frame of image, obtain other images containing the dynamic object instance in different image frames, and generate the viewpoint point cloud of the dynamic object instance that is not observed in each image frame based on the coordinates of the Gaussian point clouds of the image and other images; According to the color information of each frame of Gaussian point cloud in the four-dimensional Gaussian distribution representation, each frame of Gaussian point cloud generated from the perspective point cloud is rendered in color to obtain the reconstructed scene.

[0006] Optionally, calculate the optical flow of each pixel in each frame of the monocular video, including: Calculate the optical flow of each RGB image frame relative to the next frame in the monocular video through the RAFT2 neural network; The monocular depth estimation network MiDaS predicts the depth map of each frame of the image and calculates the optical flow of each depth map; the optical flow includes the optical flow of each pixel.

[0007] Optionally, determining a dynamic area mask of the image based on a distribution difference between the optical flow of each pixel in the image and the average optical flow of all corresponding pixels includes: Determine the mask area of ​​the RGB image based on the difference between the optical flow of each pixel in the RGB image and the average optical flow of all corresponding pixels; Determine the mask area of ​​the depth map based on the difference between the optical flow of each pixel in the depth map and the average optical flow of all corresponding pixels; The intersection between the mask area of ​​the RGB image and the mask area of ​​the depth map is determined as the dynamic area mask of the image.

[0008] Optionally, determining a plurality of spatiotemporally consistent dynamic object instances based on the dynamic region mask in each frame of the image includes: The video instance segmentation model is used to perform instance segmentation on the dynamic area of ​​each frame image to obtain the initial instance masks of multiple dynamic targets; The instance tracking method is used to perform target association between different frames on the initial instance mask of each dynamic target, and the instance masks of all dynamic targets that are consistent in time and space are obtained; Calculate the ratio of the intersection of each instance mask and the dynamic region mask to all instance masks; If the ratio exceeds a preset threshold, the instance mask is determined to be a dynamic object instance.

[0009] Optionally, according to the same dynamic object instance in different image frames, the three-dimensional Gaussian distribution representation of the image is converted into a four-dimensional Gaussian distribution representation by four-dimensional Gaussian sputtering, including: According to the same dynamic object instance in different image frames, the timestamp of the corresponding image is obtained; The three-dimensional Gaussian distribution of the image is mapped to the corresponding timestamp to obtain a four-dimensional Gaussian distribution representation.

[0010] Optionally, generating a point cloud of a viewpoint of the object instance not observed in the image according to the coordinates of the Gaussian point clouds of the image and other images includes: The coordinates of the Gaussian point clouds corresponding to object instances in the image and other images are input into the improved multi-view image generation model to obtain the viewpoint point clouds of the object instances that are not observed in the image; the improved multi-view image generation model is to expand the generation conditions of the Zero123 model to the two nearest viewpoints of the original viewpoint.

[0011] The present invention provides a device for dynamic three-dimensional reconstruction of a monocular video scene based on optical flow, comprising: The calculation module is used to calculate the optical flow of each pixel in each frame of the monocular video; A first determination module is configured to determine, for any frame image, a dynamic area mask of the image based on a distribution difference between an optical flow of each pixel in the image and an average optical flow of all corresponding pixels; The second determination module is configured to determine a plurality of spatiotemporally consistent dynamic object instances based on the dynamic region mask of each frame of image; each spatiotemporally consistent dynamic object instance represents the same object in multiple frames of image in a time sequence; A module is provided for performing a three-dimensional Gaussian sputtering conversion on each frame of the image to obtain a three-dimensional Gaussian distribution representation, and performing a four-dimensional Gaussian sputtering conversion on the three-dimensional Gaussian distribution representation of the image based on the same dynamic object instance in different image frames to obtain a four-dimensional Gaussian distribution representation; the three-dimensional Gaussian distribution representation includes the coordinates, scale, rotation and color information of the Gaussian point cloud; the four-dimensional Gaussian distribution representation has an additional time dimension than the three-dimensional Gaussian distribution representation; A generation module is configured to obtain, for any dynamic object instance in any frame of image, other images containing the dynamic object instance in different image frames, and generate a viewpoint point cloud of the dynamic object instance that is not observed in each image frame based on the coordinates of the Gaussian point clouds of the image and other images; The rendering module is used to perform color rendering on each frame of the generated viewpoint point cloud according to the color information of each frame of the Gaussian point cloud in the four-dimensional Gaussian distribution representation to obtain the reconstructed scene.

[0012] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for dynamic three-dimensional reconstruction of a monocular video scene based on optical flow.

[0013] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for dynamic three-dimensional reconstruction of a monocular video scene based on optical flow is implemented.

[0014] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects: In the present invention, the spatiotemporal consistency constraint of optical flow is realized by associating dynamic object instances through optical flow detection, thereby introducing the time dimension into the three-dimensional Gaussian distribution representation of the image. The addition of the time dimension enables the model to capture the motion trajectory of the object. In this way, the unobserved perspectives of the dynamic object instance are supplemented and colored by multiple images associated with the same dynamic object instance, which can accurately achieve end-to-end reconstruction from monocular video to three-dimensional scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0016] Figure 1 A schematic flow chart of a method for dynamic 3D reconstruction of a monocular video scene based on optical flow provided by the present invention; Figure 2 A schematic diagram of a perspective completion process provided by the present invention; Figure 3 A schematic diagram of a computer device for implementing a method for dynamic 3D reconstruction of a monocular video scene based on optical flow provided by the present invention. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] The technical solutions provided by various embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0019] Figure 1 The following is a flow chart of a method for dynamic 3D reconstruction of a monocular video scene based on optical flow in the present invention, which specifically includes the following steps: S101, calculating the optical flow of each pixel in each frame of the monocular video.

[0020] Optionally, calculating the optical flow of each pixel in each frame of the monocular video includes: calculating the optical flow of each RGB image frame in the monocular video relative to the next frame through the RAFT2 neural network; predicting the depth map of each frame through the monocular depth estimation network MiDaS, and calculating the optical flow of each depth map; the optical flow includes the optical flow of each pixel.

[0021] Since the frame difference method is easily affected by noise, the existing RAFT2 neural network is used to estimate the RGB image of each frame. The global optical flow relative to the next frame, i.e. pixel motion. RAFT2 is a deep learning model based on Convolutional Neural Network (CNN) and attention mechanism. It can provide accurate and fine-grained optical flow information by estimating pixel-level motion of consecutive image frames. At the same time, the monocular depth estimation network MiDaS is used to predict the depth map of each frame. MiDaS uses a deep convolutional neural network trained with a large amount of labeled data to accurately estimate the depth information of monocular images.

[0022] Optical flow is the motion vector of each pixel in the image, which indicates the moving direction and displacement of each pixel in the image between two consecutive frames. Optical flow is represented by two components: horizontal component ( ) and the vertical component ( ).

[0023] S102 : For any frame image, determine a dynamic area mask of the image according to a distribution difference between the optical flow of each pixel in the image and the average optical flow of all corresponding pixels.

[0024] In one embodiment, a dynamic area mask of an image is determined based on a distribution difference between an optical flow of each pixel in the image and an average optical flow of all corresponding pixels, including: determining a mask area of ​​the RGB image based on a difference between an optical flow of each pixel in the RGB image and an average optical flow of all corresponding pixels; determining a mask area of ​​the depth map based on a difference between an optical flow of each pixel in the depth map and an average optical flow of all corresponding pixels; and determining an intersection between the mask area of ​​the RGB image and the mask area of ​​the depth map as the dynamic area mask of the image.

[0025] For any frame image, based on the optical flow of the corresponding RGB image and depth map, the average optical flow size of the entire image is calculated. (or median optical flow) and the optical flow direction of all pixels , the calculation formula is as follows: A threshold is set to determine the difference between the motion of each pixel and the global pixel motion, thereby separating the dynamic object mask. Two masks are obtained: one is based on the RGB image optical flow, which marks pixels that have a large difference from the overall image optical flow; the other is based on the depth map optical flow mask, which marks the motion inconsistency in depth. Finally, the intersection of the two different optical flow masks is taken as the final dynamic area mask. , which is the dynamic object area to be separated in the end:

[0026] ( x,y )=I[| | ||- |> ] ∩ I[| | ||- |> ] S103 , determining a plurality of spatiotemporally consistent dynamic object instances according to the dynamic region mask in each frame of image; each spatiotemporally consistent dynamic object instance represents the same object in multiple frames of image in a time sequence.

[0027] In one embodiment, multiple spatiotemporally consistent dynamic object instances are determined based on the dynamic region mask in each frame of the image, including: using a video instance segmentation model to perform instance segmentation on each dynamic mask region to obtain initial instance masks of multiple dynamic targets; using an instance tracking method to perform target association between different frames on the initial instance mask of each dynamic target to obtain spatiotemporally consistent instance masks of all dynamic targets; calculating the ratio of the intersection of each instance mask and the dynamic region mask to all instance masks; if the ratio exceeds a preset threshold, determining that the instance mask is a dynamic object instance.

[0028] Specifically, in the above embodiment, the dynamic mask area in the input video or image sequence has been separated. Then, we use the existing video instance segmentation model (Segment Anything Model 2, SAM2) to perform instance segmentation on the dynamic mask area, segmenting the dynamic objects in each frame into separate instances to obtain the initial instance mask , and use the instance tracking method of SAM2 to mask the initial instance Perform object association between different frames to obtain the segmentation results (instance masks) of all instances that are consistent in time and space. Considering the loss of physical integrity in dynamic regions due to noise, it is necessary to extract a temporally consistent panoramic instance segmentation mask across the entire input video frame and then calculate each instance mask. With dynamic region mask The ratio of the intersection to all instance masks If the ratio exceeds the set threshold (for example, 0.5), the instance mask is considered to be a dynamic object instance and the instance is retained, so that each dynamic object instance is finally obtained. All are complete and time-consistent.

[0029] S104, performing a three-dimensional Gaussian sputtering conversion on each frame of the image to obtain a three-dimensional Gaussian distribution representation, and performing a four-dimensional Gaussian sputtering conversion on the three-dimensional Gaussian distribution representation of the image based on the same dynamic object instance in different image frames to obtain a four-dimensional Gaussian distribution representation; the three-dimensional Gaussian distribution representation includes the coordinates, scale, rotation and color information of the Gaussian point cloud; the four-dimensional Gaussian distribution representation has an additional time dimension than the three-dimensional Gaussian distribution representation.

[0030] The monocular video is filtered out based on the feature overlap rate of the previous and next frames and the video frame rate; the image pose and 3D sparse point cloud are extracted using the SFM method. Specifically, the key frame image feature extraction and matching are completed through the optical flow and depth information obtained in S1, and then the epipolar geometry method is used for preliminary 3D reconstruction: when the camera intrinsic parameter matrix K is known, the essential matrix E describes the matching points on the two images. and The corresponding relationship: .

[0031] The essential matrix can be obtained by solving the equation using the eight-point method or nonlinear method. Then the rotation matrix is ​​obtained by SVD decomposition and translation vectors And restore the initial 3D points by triangulation We will 3D point Projecting into the image to get 2D points , optimize the camera pose and the position of the 3D points by minimizing the reprojection error on all images through bundle adjustment (BA):

[0032] After extracting the camera pose and sparse 3D points of the scene using the SFM method, the sparse point cloud is then used to initialize and train a Gaussian (GS) model to obtain a GS-based 3D representation of the scene. Specifically, each 3D point obtained by SFM is initialized to a Gaussian distribution in the form of: ;in, is the spatial position of the Gaussian point, initialized by the spatial position of the sparse point; is the covariance matrix of the Gaussian points, which indicates the extent of the Gaussian points in space and is initialized to 1 for subsequent adjustment; Is an RGB vector, representing the color or texture information of the Gaussian point, represented by the spherical harmonic function of Gauss, and initialized here to the color corresponding to the point cloud; is the opacity, initialized to 0.5.

[0033] Next, using the projection formula, the three-dimensional Gaussian points are projected into the camera coordinate system using the Jacobian matrix, and then mapped onto the two-dimensional image plane to render the image. During rendering, all Gaussian points are sorted by depth from far to near, and alpha blending is used to calculate the final pixel color:

[0034] in, Indicates the cumulative transparency of all previous Gaussian points (to achieve occlusion effect); Indicates the i The color of the Gaussian points; is the opacity of the Gaussian point.

[0035] The position, color, opacity and other parameters of the Gaussian point can be directly optimized using the gradient descent method. Gradient of , applying the chain rule, will be The gradient is converted into the scaling factor s and the rotation quaternion Gradient of the parameters.

[0036] The position, color, opacity and other parameters of the Gaussian point can be directly optimized using the gradient descent method. Gradient of , applying the chain rule, will be The gradient is converted into the scaling factor s and the rotation quaternion The gradient of the parameters. The loss function is a combination of L1 and D-SSIM terms. The loss function is multiplied by the static scene mask, and the final optimization is the 3D Gaussian representation of the static scene without dynamic objects:

[0037] in, represents the key frame image of the original input, and Represents a 3D Gaussian rendered image computed by alpha blending from the same camera perspective. A 3D Gaussian model is trained and optimized under full supervision from real images.

[0038] In one embodiment, based on the same dynamic object instance in different image frames, the three-dimensional Gaussian distribution representation of the image is converted into a four-dimensional Gaussian sputtering to obtain a four-dimensional Gaussian distribution representation, including: based on the same dynamic object instance in different image frames, obtaining the timestamp of the corresponding image; mapping the three-dimensional Gaussian distribution of the image to the corresponding timestamp to obtain a four-dimensional Gaussian distribution representation.

[0039] To represent dynamic scenes, each frame is represented using a three-dimensional Gaussian sputtering (3DGS), which significantly consumes video memory and main memory. However, directly using a four-dimensional Gaussian sputtering (4DGS) cannot model long-term motion processes. Therefore, the motion duration S of the same object is divided into several small timestamps s based on the degree of change in optical flow (gradient value). The object changes at the same intensity within each timestamp s.

[0040] Specifically, a four-dimensional Gaussian sputtering (4DGS) is first initialized for each pixel of each dynamic instance; when an instance mask When the average direction or magnitude of the internal optical flow changes by more than twice the original value, the time is recorded. t And reinitialize a 4DGS. Each 4DGS generated in this way can be s The fixed-amplitude motion is represented internally, thus realizing the long-term dynamic reconstruction of a certain motion instance. t to render the corresponding 4DGS primitive.

[0041] 4D-GS is a method for real-time rendering of dynamic scenes. Its core idea is to introduce the time dimension based on 3D Gaussian splatter (3DGS) and model the movement and deformation of the Gaussian function through the deformation field. The disadvantage is that it can only model movement in a very short time (within a few seconds), and there are many static elements in the scene, and the dynamic amplitude of the objects is small. The initialization of 4DGS is as follows: Based on this, the deformation field is introduced , the standard Gaussian Mapping to timestamp Deformed Gaussian :

[0042] Among them, ΔX and ΔΣ are predicted by the deformation field network.

[0043] S105, for any dynamic object instance in any frame image, obtain other images containing the dynamic object instance in different image frames, and generate a viewpoint point cloud of the dynamic object instance that is not observed in each image frame based on the coordinates of the Gaussian point clouds of the image and other images.

[0044] When processing monocular video, since each frame only provides observations from one perspective, the point cloud data may be sparse and incomplete. To address this issue, the 4D Gaussian field (4DGS) needs to be completed and optimized.

[0045] In one embodiment, based on the coordinates of the Gaussian point clouds of the image and other images, a point cloud of the viewpoint of the object instance that is not observed in the image is generated, including: inputting the coordinates of the Gaussian point clouds corresponding to the object instances in the image and other images into an improved multi-view image generation model to obtain the point cloud of the viewpoint of the object instance that is not observed in the image; the improved multi-view image generation model is to expand the generation conditions of the Zero123 model to the two nearest viewpoints of the original viewpoint.

[0046] First, the pre-trained point cloud completion model PCN is used to densify the point cloud of each frame to generate a more complete three-dimensional shape. At the same time, the improved multi-view image generation model Zero123 model is used to infer the appearance of the unseen area of ​​the object at a certain moment based on the same object observed in the previous and next frames of the video, and generate an image of the object from a perspective not observed in the current frame. This generation process is done by inputting the features of the photos of other frames associated with the instance, thereby inferring the part of the object that is not seen in the current frame. t No. i Additional perspectives for frame generation The following formula expresses it:

[0047] in, The same image and text observed in the previous frame, current frame and next frame respectively i The characteristics of this process are as follows Figure 2 shown.

[0048] The original Zero123 model generates images from other viewpoints based on a single viewpoint camera position and image on a multi-view dataset, lacking constraints. When fine-tuning the improved Zero123 model, the present invention expands the generation condition to the two nearest viewpoints of the original viewpoint to maintain consistency with the above-mentioned usage process. The specific training process is as follows:

[0049] [|| ϵ - ] In the above formula is the main neural network of Zero123, It uses VAE neural network to encode adjacent views into latent variables, which is used to provide multiple view conditions for the improved Zero123 model. As a condition, the image of another viewpoint of a certain frame of the instance is predicted, which includes the invisible part of the instance. Integrating the above multiple viewpoints into 4DGS training can help optimize the 4D Gaussian Sputtering (4DGS) model.

[0050] S106 , performing color rendering on each frame of the generated viewpoint point cloud according to the color information of each frame of the Gaussian point cloud in the four-dimensional Gaussian distribution representation, to obtain a reconstructed scene.

[0051] The timestamp given in step S104 (coarse-grained) and time steps (fine-grained), the rendering process starts with the timestamp Load the corresponding 4DGS model, and then calculate the time step Calculate the deformation of the dynamic Gaussian and finally render the pixel color: in, Indicates the timestamp Next, the deformation field Calculated time step The corresponding 3DGS set: = ( , t ); Is the timestamp The local deformation field is trained using a multilayer perceptron (MLP) neural network, which only acts on Within the scope .

[0052] The training process can be expressed as: The first term in the above formula is the reconstruction loss, ensuring that each time step t Rendering result Close to the true value ; The last two terms are regularization losses, which are also a norm.

[0053] When applying the optical flow-based monocular video scene dynamic 3D reconstruction method provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0054] The above is a method for dynamic 3D reconstruction of a monocular video scene based on optical flow provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding device for dynamic 3D reconstruction of a monocular video scene based on optical flow, which includes: The calculation module is used to calculate the optical flow of each pixel in each frame of the monocular video; A first determination module is configured to determine, for any frame image, a dynamic area mask of the image based on a distribution difference between an optical flow of each pixel in the image and an average optical flow of all corresponding pixels; The second determination module is configured to determine a plurality of spatiotemporally consistent dynamic object instances based on the dynamic region mask of each frame of image; each spatiotemporally consistent dynamic object instance represents the same object in multiple frames of image in a time sequence; A module is provided for performing a three-dimensional Gaussian sputtering conversion on each frame of the image to obtain a three-dimensional Gaussian distribution representation, and performing a four-dimensional Gaussian sputtering conversion on the three-dimensional Gaussian distribution representation of the image based on the same dynamic object instance in different image frames to obtain a four-dimensional Gaussian distribution representation; the three-dimensional Gaussian distribution representation includes the coordinates, scale, rotation and color information of the Gaussian point cloud; the four-dimensional Gaussian distribution representation has an additional time dimension than the three-dimensional Gaussian distribution representation; A generation module is configured to obtain, for any dynamic object instance in any frame of image, other images containing the dynamic object instance in different image frames, and generate a viewpoint point cloud of the dynamic object instance that is not observed in each image frame based on the coordinates of the Gaussian point clouds of the image and other images; The rendering module is used to perform color rendering on each frame of the generated viewpoint point cloud according to the color information of each frame of the Gaussian point cloud in the four-dimensional Gaussian distribution representation to obtain the reconstructed scene.

[0055] Regarding the specific limitations of the monocular video scene dynamic three-dimensional reconstruction device based on optical flow, please refer to the limitations of the monocular video scene dynamic three-dimensional reconstruction method based on optical flow above, which will not be repeated here. Each module in the above-mentioned monocular video scene dynamic three-dimensional reconstruction device based on optical flow can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0056] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The proposed method is a dynamic 3D reconstruction method for monocular video scenes based on optical flow.

[0057] The present invention also provides Figure 3 The structural diagram of the computer equipment shown in FIG. Figure 3 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1The proposed method is a dynamic 3D reconstruction method for monocular video scenes based on optical flow.

[0058] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0059] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for dynamic 3D reconstruction of monocular video scenes based on optical flow, characterized in that: include: Calculate the optical flow of each pixel in each frame of a monocular video; For any frame image, the dynamic area mask of the image is determined based on the distribution difference between the optical flow of each pixel in the image and the average optical flow of all corresponding pixels; Determine multiple spatiotemporally consistent dynamic object instances based on the dynamic region mask of each frame; each spatiotemporally consistent dynamic object instance represents the same object in multiple frames of the image in a time sequence; Each frame of the image is transformed by three-dimensional Gaussian sputtering to obtain a three-dimensional Gaussian distribution representation. Based on the same dynamic object instance in different image frames, the three-dimensional Gaussian distribution representation of the image is transformed by four-dimensional Gaussian sputtering to obtain a four-dimensional Gaussian distribution representation. The three-dimensional Gaussian distribution representation includes the coordinates, scale, rotation and color information of the Gaussian point cloud. The four-dimensional Gaussian distribution representation has an additional time dimension than the three-dimensional Gaussian distribution representation. For any dynamic object instance in any frame of image, obtain other images containing the dynamic object instance in different image frames, and generate the viewpoint point cloud of the dynamic object instance that is not observed in each image frame based on the coordinates of the Gaussian point clouds of the image and other images; According to the color information of each frame of Gaussian point cloud in the four-dimensional Gaussian distribution representation, each frame of Gaussian point cloud generated from the perspective point cloud is rendered in color to obtain the reconstructed scene.

2. The method according to claim 1, characterized in that Calculate the optical flow of each pixel in each frame of a monocular video, including: Calculate the optical flow of each RGB image frame relative to the next frame in the monocular video through the RAFT2 neural network; The monocular depth estimation network MiDaS predicts the depth map of each frame of the image and calculates the optical flow of each depth map; the optical flow includes the optical flow of each pixel.

3. The method according to claim 2, characterized in that The dynamic area mask of the image is determined based on the distribution difference between the optical flow of each pixel in the image and the average optical flow of all corresponding pixels, including: Determine the mask area of ​​the RGB image based on the difference between the optical flow of each pixel in the RGB image and the average optical flow of all corresponding pixels; Determine the mask area of ​​the depth map based on the difference between the optical flow of each pixel in the depth map and the average optical flow of all corresponding pixels; The intersection between the mask area of ​​the RGB image and the mask area of ​​the depth map is determined as the dynamic area mask of the image.

4. The method according to claim 1, wherein Based on the dynamic region mask of each frame, multiple spatiotemporally consistent dynamic object instances are determined, including: The video instance segmentation model is used to perform instance segmentation on each frame of the image to obtain the initial instance masks of multiple dynamic targets; The instance tracking method is used to perform target association between different frames on the initial instance mask of each dynamic target, and the instance masks of all dynamic targets that are consistent in time and space are obtained; Calculate the ratio of the intersection of each instance mask and the dynamic region mask to all instance masks; If the ratio exceeds a preset threshold, the instance mask is determined to be a dynamic object instance.

5. The method according to claim 1, wherein According to the same dynamic object instance in different image frames, the three-dimensional Gaussian distribution representation of the image is converted into a four-dimensional Gaussian sputtering to obtain a four-dimensional Gaussian distribution representation, including: According to the same dynamic object instance in different image frames, the timestamp of the corresponding image is obtained; The three-dimensional Gaussian distribution of the image is mapped to the corresponding timestamp to obtain a four-dimensional Gaussian distribution representation.

6. The method according to claim 1, characterized in that Based on the coordinates of the Gaussian point clouds of the image and other images, a point cloud of the object instance at a viewpoint not observed in the image is generated, including: The coordinates of the Gaussian point clouds corresponding to object instances in the image and other images are input into the improved multi-view image generation model to obtain the viewpoint point clouds of the object instances that are not observed in the image; the improved multi-view image generation model is to expand the generation conditions of the Zero123 model to the two nearest viewpoints of the original viewpoint.

Images