Method and apparatus for 3d scene reconstruction based on neural network and multi-view consistency

Abstract

This invention discloses a method and apparatus for 3D scene reconstruction based on neural networks and multi-view consistency. The method includes: extracting image frames from video data and downsampling them; using a pre-trained monocular absolute depth estimation model to predict the downsampled images; using a scale offset recovery module to recover the inconsistent scale and offset between downsampled absolute depth maps; projecting the obtained scale offset consistent depth maps between downsampled image frames using optimizable camera parameters; selecting local and global keyframes; supervising optical flow information, geometric consistency, and regularization terms; optimizing depth and camera parameters; inputting the relevant parameters into an image-depth fusion module for processing; and outputting the 3D scene reconstruction result. This invention reduces the reliance on sensors in existing methods, improves the density and robustness of 3D scene reconstruction methods, and solves the problem of difficulty in achieving both robustness and density in existing 3D scene reconstruction methods.

Description

Technical Field

[0001] This invention relates to a 3D scene reconstruction method in the field of computer vision technology, specifically a method based on neural networks and multi-view... Figure 1 A consistent method and apparatus for three-dimensional scene reconstruction. Background Technology

[0002] With the development of artificial intelligence, dense 3D reconstruction has been widely applied in fields such as autonomous driving, virtual reality, augmented reality, and medical 3D modeling. Existing 3D scene reconstruction technologies can be divided into optimization methods based on multi-view geometric constraints and learning methods based on neural networks. Among them, optimization methods based on multi-view geometric constraints calculate the depth map of the image frame by matching the feature information between image frames from different perspectives and projecting it into 3D space for fusion, thereby realizing 3D scene reconstruction. For example, as described in the literature Im S, Jeon HG, Lin S, et al. DPSNet: End-to-end Deep Plane SweepStereo[C]. International Conference on Learning Representations.2018., similarity between pixel-level features of the image can be calculated to obtain matching information by simulating traditional planar scanning, thereby predicting the image depth information. Then, the camera pose is obtained through sensors, the camera intrinsic parameters are calculated through camera calibration, and the image is projected into 3D space for fusion to obtain 3D scene reconstruction.

[0003] Neural network-based learning methods train large neural networks to directly learn to fit and predict 3D geometric information from image frames. For example, as described in the literature Sun J, Xie Y, Chen L, et al. NeuralRecon: Real-time coherent 3D reconstruction from monocular video[C]. Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2021:15598-15607, convolutional neural networks are used to extract image features, project the features onto 3D space, and use a convolutional GRU fusion module to fuse the features into a 3D global hidden state. Finally, through training, the predicted output 3D scene reconstruction representation is obtained.

[0004] The above are representative methods for existing 3D scene reconstruction, with three main drawbacks. First, optimization methods based on multi-view geometric constraints perform poorly in low-texture areas of camera images, areas with drastic lighting changes, and occluded areas, and feature matching is prone to errors, resulting in overly sparse reconstructed 3D scenes. Second, neural network-based learning methods heavily rely on massive amounts of data to drive the learning of robust geometric information; however, the quantity and diversity of existing 3D scene reconstruction datasets are limited, making these methods lack robustness in real-world scenarios. Third, both methods largely depend on sensors to obtain camera pose, limiting their application scenarios. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a method based on neural networks and multi-view... Figure 1 A consistent 3D scene reconstruction method and apparatus, which utilizes neural networks to learn robust depth information and combines multi-view... Figure 1 By optimizing the consistency geometric constraints, the reliance of existing methods on sensors is reduced, while the density and robustness of 3D scene reconstruction methods are improved.

[0006] The first aspect of this invention provides a method based on neural networks and multiple views. Figure 1 A consistent 3D scene reconstruction method includes the following steps:

[0007] Step 1: Construct a pre-trained monocular absolute depth estimation model;

[0008] Step 2: Extract image frames from the video data to be processed and downsample them to obtain downsampled image frames. Input the downsampled image frames into the pre-trained monocular absolute depth estimation model in Step 1 to obtain monocular predicted absolute depth maps. Use the optimizable depth parameters in the model's predicted absolute depth scale offset recovery module to recover the inconsistent scale and offset between monocular predicted absolute depth maps to obtain scale offset consistent depth maps. Project the scale offset consistent depth maps between downsampled image frames using optimizable camera parameters.

[0009] Step 3: Use the LoFTR algorithm to extract sparse features from each downsampled image frame and match them with other downsampled image frames to select local keyframes. Select global keyframes based on the relative rotation angle between two downsampled images. Construct a supervision signal L based on the projection results. Train the depth parameters and camera parameters based on the supervision signal L to optimize the depth parameters and camera parameters.

[0010] Step 4: Input the monocular predicted absolute depth map and the optimized depth parameters into the model predicted absolute depth scale offset recovery module to obtain a scale offset consistent depth map. Input the obtained scale offset consistent depth map, downsampled image frames, and optimized camera parameters into the image-depth fusion module to output the 3D scene reconstruction result.

[0011] This invention takes a video sequence as input. First, it extracts image frames from the input video sequence, transforming the video input information into temporally discretized video image frames, facilitating the processing of video information using image processing knowledge. Firstly, this invention leverages the powerful learning capabilities of neural networks, training on massive image-depth map datasets of tens of millions of images to predict the absolute depth information of each image frame based on a single input image. Secondly, recognizing the inconsistencies between frames in the predicted absolute depth information, this invention uses the correlation between image frames and multi-view geometric consistency constraints to recover the scale and offset information in the absolute depth map predicted frame-by-frame by the neural network model, thereby optimizing camera and depth parameters. During the optimization phase, downsampling and keyframe matching are performed on the extracted image frames from the video, and a reasonable optimization objective is designed as a supervisory signal. Finally, the downsampled image frames, their corresponding absolute depths, optimized depth parameters, and camera parameters are post-processed to output a 3D scene patch representation.

[0012] Further, in step 1, the method for constructing the pre-trained monocular absolute depth estimation model is as follows: collect RGB-D image-depth ground truth matching data and divide them into three qualities: high, medium, and low. Use different supervision signals to learn and train the monocular absolute depth estimation model. During model training, set a standard camera space and, based on the proportion of the actual camera in the camera that took the image participating in the standard camera space, uniformly convert the captured images to the standard camera space for learning. During inference, convert back to the original camera space through the inverse transformation of this transformation to obtain the absolute depth.

[0013] This invention obtains basic depth information of image frames by training a monocular absolute depth estimation model. The amount and diversity of the collected data in this step should be as large as possible, and different loss functions should be used for supervision during training, depending on the quality of the data annotations.

[0014] Further, in step 2, the downsampling strategy is as follows: First, select the first frame image as the first downsampled image frame. For the i-th downsampled image frame, select image frames in the direction of increasing time axis. When the degree of change between the selected image frame and the i-th downsampled image frame is greater than 1 / 6, select the frame as the (i+1)-th downsampled image frame. i starts from 1 and iterates this process until all image frames are traversed.

[0015] Because the absolute depth predicted by the monocular absolute depth estimation model contains inconsistent scales and offsets between frames, it can cause ghosting or deformation in the reconstructed point cloud or surface, reducing the accuracy of 3D scene reconstruction. In step 2 of this invention, the optimizable depth parameters in the model-predicted absolute depth scale offset recovery module are used to recover the inconsistent scales and offsets between monocular predicted absolute depth maps.

[0016] Furthermore, in step 2, the optimizable depth parameters include global scale, global offset, and sparse weights. The global scale offset between monocular predicted absolute depth maps is recovered based on the global scale and global offset, and the local scale offset between monocular predicted absolute depth maps is recovered based on the sparse weights.

[0017] This invention employs least squares fitting and geographically weighted regression to recover global and local scale offsets, and recovers consistent absolute depth information between frames more effectively based on global scale, global offset, and sparse weights.

[0018] Furthermore, in step 2, the optimizable camera parameters are camera pose and camera intrinsic parameters, including the relative camera pose Euler angles, translation vectors, and camera focal length optimization coefficients between every two frames.

[0019] Based on absolute depth information and color image frames, 2D image pixels can be projected into a 3D space in the camera coordinate system to form a colored point cloud. This projection process requires knowledge of the camera's intrinsic parameters. Furthermore, differences in camera pose still exist between frames, necessitating the correction and alignment of camera pose variations using rotation and translation matrices. This invention proposes a camera parameter optimization scheme, setting the focal length of the camera's intrinsic parameters and the relative camera pose between each adjacent frame as optimizable variables. Utilizing the consistency between frames, the camera parameters are obtained through adaptive iterative optimization.

[0020] Furthermore, in step 3, the monitoring signal L is monitored by the optical flow between each keyframe. flow Geometric consistency L gc And regularization supervision of sparse weights L regu It consists of three parts:

[0021]

[0022]

[0023]

[0024] L=λ flow L flow +λ gc L gc +λ regu L regu

[0025] in, These represent the relative rotation Euler angle and relative pose translation between every two camera poses from frame i to frame j, respectively; δ is the optimizable coefficient of the camera intrinsic parameters; α i ,β i ω i Let $\frac{i}{i}$ be the global scale, global offset value, and sparse weight of the $i$-th frame, respectively; $V$ represents the set of effective projection point pixel coordinates from the $i$-th frame to the $j$-th frame; $p$ represents the global scale, global offset value, and sparse weight of the $i$-th frame. i The pixel coordinates of frame i are represented and obtained from V; K′ represents the selected set of keyframes, and (i,j) represents the matching pair between frame i and frame j; that is, sampling frame i and frame j. i→j Indicates the pixel coordinates p from the i-th frame i The pixel coordinates calculated by projecting onto the j-th frame; F i→j f represents the optical flow information from frame i to frame j obtained from the robust optical flow network RAFT prediction; sample (D j ,p i→j ) represents the consistent depth map D from the scale offset of the j-th frame. j Medium-sampled pixel p i→j Depth value at; f sample (D i→j ,p i () represents the projection depth map D calculated from the projection of frame i to frame j. i→j Medium-sampled pixel p i Depth value at; ω i,t ω represents the sparse weight of the i-th frame. i The t-th component; P represents the number of downsampled image frames; M represents the one-dimensional vector ω. i The length of λ; flow , λ gc and λ regu To balance the weight hyperparameters of each monitoring signal.

[0026] After the optimization parameter design in step 2 is completed, a suitable optimization objective needs to be adopted. Based on the consistency between image frames, the camera parameters and depth parameters are iteratively optimized to obtain them. In step 3 of this invention, a scheme combining optical flow supervision and geometric consistency supervision is adopted, which ensures both accurate matching between image frames and that the inter-frame projection relationship satisfies the multi-view geometric consistency constraint. The keyframe selection of this invention is divided into two parts: local keyframe selection and global keyframe selection. Local keyframes use the LoFTR algorithm to sample and optimize image keyframes, which can ensure the local consistency of the reconstruction results. The global keyframe selection strategy selects two suitable frames in the overlapping area as keyframe matching pairs based on the relative camera extrinsic parameters between each pair of frames, thereby optimizing the overall 3D scene reconstruction results. In addition, for cases where improper optimization of depth parameter scale and offset leads to excessive deformation of the reconstructed point cloud, this invention also adopts another regularization term to regularize and restrict the sparse weights in the local scale offset recovery, which can reconstruct a more accurate point cloud.

[0027] Furthermore, in step 4, a truncated signed distance function is used to fuse the obtained scale offset consistent depth map with the downsampled image frame and the optimized camera parameters to output the 3D scene reconstruction result.

[0028] This invention fuses the acquired scale offset consistent depth map with the downsampled image frame and the truncated signed distance function of the optimized camera parameter input, adaptively performs smoothing and denoising processing on the point cloud that is inconsistent between frames, and outputs a 3D scene patch representation.

[0029] The second aspect of this invention provides a method based on neural networks and multiple views. Figure 1 A consistent 3D scene reconstruction device, including:

[0030] The model training module is used to build a pre-trained monocular absolute depth estimation model.

[0031] The downsampling and model-predicted absolute depth scale offset recovery module extracts image frames from the video data to be processed and downsamples them to obtain downsampled image frames. The downsampled image frames are then input into a pre-trained monocular absolute depth estimation model to obtain a monocular predicted absolute depth map. Using the optimizable depth parameters in the model-predicted absolute depth scale offset recovery module, inconsistent scales and offsets between monocular predicted absolute depth maps are recovered to obtain a scale offset consistent depth map. The scale offset consistent depth map is then projected between downsampled image frames using optimizable camera parameters.

[0032] The keyframe selection and parameter optimization module is used to extract sparse features of each downsampled image frame using the LoFTR algorithm, match them with other downsampled image frames, select local keyframes, select global keyframes based on the relative rotation angle between two downsampled images, construct a supervision signal L based on the projection results, and train and optimize the depth parameters and camera parameters based on the supervision signal L.

[0033] The 3D scene reconstruction post-processing module is used to input the monocular predicted absolute depth map and the optimized depth parameters into the model predicted absolute depth scale offset recovery module to obtain a scale offset consistent depth map. The obtained scale offset consistent depth map, along with the downsampled image frame and the optimized camera parameters, is input into the image-depth fusion module to output the 3D scene reconstruction result.

[0034] A third aspect of the present invention provides a computer device, including a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the above-described neural network-based and multi-view... Figure 1 A consistent 3D scene reconstruction method.

[0035] A fourth aspect of the present invention provides a storage medium storing a program, which, when executed by a processor, implements the above-described neural network-based and multi-view... Figure 1 A consistent 3D scene reconstruction method.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) This invention utilizes neural networks to learn robust depth information, combined with multi-view... Figure 1 By optimizing the consistency geometric constraints, the reliance of existing methods on sensors is reduced, while the density and robustness of the 3D scene reconstruction method are improved, solving the problem that existing 3D scene reconstruction methods cannot achieve both robustness and density.

[0038] (2) This invention downsamples the acquired video image frames to obtain the downsampled image frames and their depth information. It uses least squares fitting and geographic weighted regression to restore global and local scale offsets. Based on global scale, global offset and sparse weights, it better restores the consistent absolute depth information between frames. This solves the problem that the absolute depth predicted by the monocular absolute depth estimation model contains inconsistent scale and offset between frames, which will cause ghosting or deformation of the reconstructed point cloud or patch, and reduce the accuracy of 3D scene reconstruction.

[0039] (3) The present invention adopts a scheme combining optical flow supervision and geometric consistency supervision to optimize depth parameters and camera parameters, which not only ensures accurate matching between image frames, but also ensures that the inter-frame projection relationship meets the multi-view geometric consistency constraint. At the same time, the optimization process uses a regularization term to regularize the sparse weights in the local scale offset recovery, which can reconstruct more accurate point clouds. Attached Figure Description

[0040] Figure 1 The embodiment is based on neural networks and multiple views Figure 1 A flowchart of a consistent 3D scene reconstruction method.

[0041] Figure 2 This is a schematic diagram of the model prediction absolute depth scale offset recovery module in the embodiment.

[0042] Figure 3 This is a schematic diagram of the three-dimensional reconstruction result in the embodiment. Detailed Implementation

[0043] The technical solution of the present invention will be further described below with reference to the accompanying drawings. The present invention is based on neural networks and multi-view... Figure 1 The flowchart of the consistent 3D scene reconstruction method is as follows: Figure 1 As shown, it includes the following steps:

[0044] Step 1: Construct a pre-trained monocular absolute depth estimation model.

[0045] A massive amount of monocular depth data was collected to train a monocular absolute depth estimation model, which exhibits high accuracy and precision. The training data was categorized into three quality levels: high-quality, medium-quality, and low-quality. High-quality data included accurate depth information from sensors such as radar; medium-quality data included depth information obtained through scheme matching calculations such as binocular matching; and low-quality data included pseudo-labels of depth information generated by optical flow. Different loss functions were applied to different data quality levels. High-quality data was supervised by pixel-wise normal vector regression, multi-scale gradient loss, and image ranking loss; medium-quality data was supervised by multi-scale gradient loss, image-level normal vector regression, and depth ranking loss; and low-quality data was supervised only by depth ranking loss. During model training, a standard camera space was set up, and images were uniformly transformed to the standard camera space based on the proportion of the actual camera's content participating in the standard camera space. During inference, the images were transformed back to the original camera space using the inverse transformation to obtain the absolute depth.

[0046] Step 2: Extract image frames from the video data to be processed and downsample them to obtain downsampled image frames. Input the downsampled image frames into the pre-trained monocular absolute depth estimation model in Step 1 to obtain monocular predicted absolute depth maps. Use the optimizable depth parameters in the model's predicted absolute depth scale offset recovery module to recover the inconsistent scale and offset between the monocular predicted absolute depth maps to obtain scale offset consistent depth maps. Project the scale offset consistent depth maps between the downsampled image frames using optimizable camera parameters.

[0047] Step 2-1: Extract N image frames from the input color video data. Downsampling is then performed. The downsampling strategy adopted in this embodiment is as follows: First, the first image frame is selected as the first downsampled image frame. For the i-th downsampled image frame, image frames are selected in the direction of increasing time. When the change between the selected image frame and the i-th downsampled image frame is greater than 1 / 6, this frame is selected as the (i+1)-th downsampled image frame. This process is iterated from i starting from 1 until all image frames are traversed, resulting in P downsampled image frames. Input P downsampled image frames into the monocular absolute depth estimation model pre-trained in step 1 to obtain the monocular predicted absolute depth map and obtain the corresponding monocular predicted absolute depth.

[0048] Step 2-2: Because inconsistencies in scale and offset between monocular predicted absolute depth maps can cause ghosting and distortion in the point cloud projected into 3D space, a model-predicted absolute depth scale offset recovery module is introduced to recover the scale and offset of multiple image frames and align them temporally. The model-predicted absolute depth scale offset recovery module is as follows: Figure 2 As shown, it includes two parts: global scale offset alignment and local scale offset alignment. Global alignment, for each monocular predicted absolute depth map, recovers the shared scale value α between pixels in each image frame. i and offset value β i And this value can be used for optimization:

[0049]

[0050] The depth map obtained by global alignment output in the above formula is... This is called the global alignment depth map. The scale A in local alignment... i ∈R H×W and offset B i ∈R H×W To generate two images pixel-by-pixel, with the same size as the image frame, the specific generation method is to uniformly sample m points from the globally aligned depth map. Then, for each sampling point, multiply it by the corresponding sparse weight point by point. As sparse anchor points, globally align the depth map. With sparse anchor points Perform geographic weighted regression alignment and output the scale map A. i With offset map B i Align scales with the global scale separately Perform pixel-by-pixel multiplication and addition:

[0051]

[0052]

[0053] Where M is 25, the symbol ⊙ represents pixel-wise multiplication, and f LWLR (·) is derived from the geographic weighted regression algorithm mentioned in the following paper: Towards 3d scene reconstruction from locally scale-aligned monocular video depth.arXiv preprint arXiv:2202.01470,2022.

[0054] After using the model-predicted absolute depth scale offset recovery module to recover the inconsistent scales and offsets between monocular predicted absolute depth maps, a depth map with consistent scale offsets is obtained.

[0055] Step 2-3: Project the scale-offset consistent depth map obtained in Step 2-2 onto the downsampled image frames using an optimizable camera pose and camera intrinsic parameters. The camera pose is initialized to the relative Euler angle r between every two adjacent image frames. i→(i+1) With pose translation t i→(i+1) The camera intrinsics are initialized to a simple pinhole model, the camera focal length is initialized to 1.2 times the longest side of the image frame, and an optimizable coefficient δ is set. In the actual optimization process, the relative rotation Euler angle r... i→(i+1) With pose translation t i→(i+1) It needs to be converted into a camera pose transformation matrix, camera pose P i The camera intrinsic parameter K is represented as follows:

[0056]

[0057]

[0058] Where P1 represents the camera pose in the first frame, E 4,4 Represents a 4x4 identity matrix. Represents Euler angles r from relative rotation i→(i+1) and pose translation t i→(i+1) To camera pose P i→(i+1)The mapping function. f0 is the camera focal length initialization parameter, and W and H represent the pixel width and pixel height of the image frame, respectively.

[0059] Camera pose P i With camera rotation matrix R i Translation matrix t i The relationship between them is:

[0060] Then, using camera intrinsic parameters, camera pose, and image frame depth, projection is performed between the downsampled image frames:

[0061] d i (p i )=f sample (D i p i (6)

[0062] d i→j (p i )=f sample (D i→j ,Pi), (7)

[0063]

[0064] Among them, f sample (D i ,p i ) indicates from depth map D i Medium-sampled pixel p i Depth value d at the location i (p i ), f sample (D i→j ,p i () represents the projection depth map D calculated from the projection from frame i to frame j. i→j Medium sampling pixel p i Depth value d at the location i→j (p i ); R i and t i Let P be the camera rotation and translation matrix for the i-th frame. i Components; p i→j The coordinates are p i The coordinates projected from frame i to frame j; t represents the transpose of the camera rotation matrix in frame j. j The denot represents the camera translation matrix for the j-th frame.

[0065] The optimizable variables in this invention are global scale, global offset, sparse weight values, relative Euler angles, relative pose translation, and camera focal length coefficient. After setting and initializing the optimizable parameters, corresponding keyframes need to be selected for inter-frame consistency optimization. This step involves optimizing the selection of keyframes and the selection of supervisory signals.

[0066] Step 3: Use the LoFTR algorithm to extract sparse features from each downsampled image frame and match them with other downsampled image frames to select local keyframes. Select global keyframes based on the relative rotation angle between two downsampled images. Construct a supervision signal L based on the projection results. Train the depth parameters and camera parameters based on the supervision signal L to optimize the depth parameters and camera parameters.

[0067] Step 3-1: Keyframe selection can be divided into two parts: local keyframe selection and global keyframe selection, which are divided into two stages. In the first optimization stage, the LoFTR algorithm is used to extract sparse features of each image and match them with other keyframes. For the obtained matching image pairs, they are uniformly sampled and batch optimized according to the supervision signal designed in formula (12) until the local geometry between keyframes is consistent. The sampling probability of local keyframes is as follows:

[0068]

[0069] Where, p l This represents the sampling probability of keyframes in the first stage.

[0070] In the second optimization stage, in addition to selecting local keyframes, global keyframes also need to be selected based on the relative rotation angle between two frames. Specifically, the rotation angle between each pair of frames is calculated. The sampling probability of the keyframe gradually increases and then gradually decreases as the rotation angle increases, reaching its maximum value at a rotation angle of φ. In the second stage, the global sampling probability and the local sampling probability each account for 1 / 2 of the total sampling probability, as shown in the following formula:

[0071]

[0072] Where, θ ij This represents the relative rotation angle between the i-th frame and the j-th frame, where φ is the angle threshold, set to... p is the global sampling probability, p g This represents the total sampling probability for the second stage.

[0073] Step 3-2: During the optimization process, for the keyframes i and j selected in Step 3-1, the i-th frame is projected onto the j-th frame, and the supervision signal L is supervised by the optical flow between the keyframes. flow Geometric consistency L gc and sparse weight regularization supervision Lregu It consists of three parts:

[0074]

[0075]

[0076]

[0077] L=λ flow L flow +λ gc L gc +λ regu L regu (14)

[0078] The optimizable parameters minimized in each iteration include the relative rotation Euler angles between the camera poses of every two frames from frame i to frame j. Relative pose translation The camera intrinsic parameters can be optimized by coefficient δ, and the global scale α of the i-th frame. i Global offset value β i and sparse weight ω i V represents the set of effective projection point pixel coordinates from frame i to frame j, where p is the pixel coordinates. i The pixel coordinates of the i-th frame are represented and obtained from V; K′ represents the set of keyframes selected in step 3-1, and (i,j) represents the matching pair between the i-th and j-th frames, i.e., sampling the i-th and j-th frames. i→j Represents the pixel coordinates p from the i-th frame i The pixel coordinates calculated by projecting onto the j-th frame. F i→j f represents the optical flow information from frame i to frame j obtained from the robust optical flow network RAFT prediction. sample (D j ,p i→j ) represents the consistent depth map D from the scale offset of the j-th frame. j Medium-sampled pixel p i→j The depth value at f sample (D i→j ,p i () represents the projection depth map D calculated from the projection of frame i to frame j. i→j Medium-sampled pixel p i Depth value at; ω i,t ω represents the sparse weight of the i-th frame. i The t-th component. P represents the number of downsampled image frames, and M represents the one-dimensional vector ω. i The length of λ. flow , λ gc and λ regu To balance the weight hyperparameters of each monitoring signal.

[0079] The overall optimization iterations totaled 6000, with each iteration sampling 50 keyframe matching pairs. The first stage consisted of 2000 iterations, with the supervision signal loss hyperparameter λ... flow , λ gc and λ regu The values ​​were set to 2, 0.5, and 0.01 respectively. The second stage consisted of 4000 iterations, with the first 2000 iterations monitoring the signal loss hyperparameter λ. flow , λ gc and λ regu The hyperparameter λ of the monitoring signal loss is set to 2, 1, and 0.01 respectively for the last 2000 iterations. flow , λ gc and λ regu Set them to 2, 0.1, and 0.01 respectively.

[0080] Step 4: Input the monocular predicted absolute depth map and optimized depth parameters into the model's predicted absolute depth scale offset recovery module to obtain a scale offset consistent depth map. Then, input the obtained scale offset consistent depth map, downsampled image frames, and optimized camera parameters into a truncated signed distance function-based fusion post-processing module (TSDF-Fusion) to output accurate and robust 3D scene reconstruction results (e.g., ...). Figure 3 (As shown).

[0081] This embodiment also provides a method based on neural networks and multi-view. Figure 1 A consistent 3D scene reconstruction device, including:

[0082] The model training module is used to build a pre-trained monocular absolute depth estimation model.

[0083] The downsampling and model-predicted absolute depth scale offset recovery module extracts image frames from the video data to be processed and downsamples them to obtain downsampled image frames. The downsampled image frames are then input into a pre-trained monocular absolute depth estimation model to obtain a monocular predicted absolute depth map. Using the optimizable depth parameters in the model-predicted absolute depth scale offset recovery module, inconsistent scales and offsets between monocular predicted absolute depth maps are recovered to obtain a scale offset consistent depth map. The scale offset consistent depth map is then projected between downsampled image frames using optimizable camera parameters.

[0084] The keyframe selection and parameter optimization module is used to extract sparse features of each downsampled image frame using the LoFTR algorithm, match them with other downsampled image frames, select local keyframes, select global keyframes based on the relative rotation angle between two downsampled images, construct a supervision signal L based on the projection results, and train and optimize the depth parameters and camera parameters based on the supervision signal L.

[0085] The 3D scene reconstruction post-processing module is used to input the monocular predicted absolute depth map and the optimized depth parameters into the model predicted absolute depth scale offset recovery module to obtain a scale offset consistent depth map. The obtained scale offset consistent depth map, along with the downsampled image frame and the optimized camera parameters, is input into the image-depth fusion module to output the 3D scene reconstruction result.

[0086] This embodiment also provides a computer device, including a processor and a memory for storing a processor-executable program. When the processor executes the program stored in the memory, it implements the above-mentioned neural network-based and multi-view... Figure 1 A consistent 3D scene reconstruction method.

[0087] This embodiment also provides a storage medium storing a program, which, when executed by a processor, implements the above-described neural network and multi-view-based approach. Figure 1 A consistent 3D scene reconstruction method.

Claims

1. A method for reconstructing a 3D scene based on neural networks and multi-view consistency, characterized in that, Includes the following steps: Step 1: Construct a pre-trained monocular absolute depth estimation model; Step 2: Extract image frames from the video data to be processed and downsample them to obtain downsampled image frames. Input the downsampled image frames into the pre-trained monocular absolute depth estimation model in Step 1 to obtain monocular predicted absolute depth maps. Use the optimizable depth parameters in the model's predicted absolute depth scale offset recovery module to recover the inconsistent scale and offset between monocular predicted absolute depth maps to obtain scale offset consistent depth maps. Project the scale offset consistent depth maps between downsampled image frames using optimizable camera parameters. Step 3: Use the LoFTR algorithm to extract sparse features from each downsampled image frame and match them with other downsampled image frames to select local keyframes. Select global keyframes based on the relative rotation angle between two downsampled images. Construct a supervision signal L based on the projection results. Train the depth parameters and camera parameters based on the supervision signal L to optimize the depth parameters and camera parameters. Step 4: Input the monocular predicted absolute depth map and the optimized depth parameters into the model predicted absolute depth scale offset recovery module to obtain a scale offset consistent depth map. Input the obtained scale offset consistent depth map, downsampled image frames, and optimized camera parameters into the image-depth fusion module to output the 3D scene reconstruction result.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that, In step 1, the method for constructing the pre-trained monocular absolute depth estimation model is as follows: collect RGB-D image-depth ground truth matching data and divide them into three quality levels: high, medium, and low. Use different supervision signals to learn and train the monocular absolute depth estimation model. During model training, set a standard camera space and, based on the proportion of the actual camera in the captured image participating in the standard camera space, uniformly transform the captured images to the standard camera space for learning. During inference, use the inverse transformation of this transformation to convert back to the original camera space for inference to obtain the absolute depth.

3. The three-dimensional scene reconstruction method according to claim 1, characterized in that, In step 2, the downsampling strategy is as follows: First, select the first frame image as the first downsampled image frame. For the i-th downsampled image frame, select image frames in the direction of increasing time axis. When the degree of change between the selected image frame and the i-th downsampled image frame is greater than 1 / 6, select the frame as the (i+1)-th downsampled image frame. i starts from 1 and iterates this process until all image frames are traversed.

4. The three-dimensional scene reconstruction method according to claim 1, characterized in that, In step 2, the optimizable depth parameters include global scale, global offset, and sparse weights. The global scale offset between monocular predicted absolute depth maps is recovered based on the global scale and global offset, and the local scale offset between monocular predicted absolute depth maps is recovered based on the sparse weights.

5. The three-dimensional scene reconstruction method according to claim 4, characterized in that, In step 2, the optimizable camera parameters are camera pose and camera intrinsic parameters, including the relative camera pose Euler angles, translation vectors, and camera focal length optimization coefficients between every two frames.

6. The three-dimensional scene reconstruction method according to claim 5, characterized in that, In step 3, the monitoring signal L is monitored by the optical flow between each keyframe. flow Geometric consistency L gc and sparse weighted regularization supervision L regu It consists of three parts: L=λ flow L flow +λ gc L gc +λ regu L regu in, These represent the relative rotation Euler angle and relative pose translation between every two camera poses from frame i to frame j, respectively; δ is the optimizable coefficient of the camera intrinsic parameters; α i ,β i ω i Let $\frac{i}{i}$ be the global scale, global offset value, and sparse weight of the $i$-th frame, respectively; $V$ represents the set of effective projection point pixel coordinates from the $i$-th frame to the $j$-th frame; $p$ represents the global scale, global offset value, and sparse weight of the $i$-th frame. i Let represent the pixel coordinates of the i-th frame, obtained from V; K′ represents the selected set of keyframes, and (i,j) represents the matching pair between the i-th and j-th frames; p i→j This indicates the pixel coordinates p from the i-th frame. i The pixel coordinates calculated by projecting onto the j-th frame; F i→j f represents the optical flow information from frame i to frame j obtained from the robust optical flow network RAFT prediction; sample (D j ,p i→j ) represents the consistent depth map D from the scale offset of the j-th frame. j Medium-sampled pixel p i→j Depth value at; f sample (D i→j ,p i () represents the projection depth map D calculated from the projection of frame i to frame j. i→j Medium-sampled pixel p i Depth value at; ω i,t ω represents the sparse weight of the i-th frame. i The t-th component; P represents the number of downsampled image frames; M represents the one-dimensional vector ω. i The length of λ; flow , λ gc and λ regu To balance the weight hyperparameters of each monitoring signal.

7. The three-dimensional scene reconstruction method according to claim 1, characterized in that, In step 4, a truncated signed distance function is used to fuse the obtained scale offset consistent depth map with the downsampled image frame and the optimized camera parameters to output the 3D scene reconstruction result.

8. A 3D scene reconstruction device based on neural networks and multi-view consistency, comprising: The model training module is used to build a pre-trained monocular absolute depth estimation model. The downsampling and model-predicted absolute depth scale offset recovery module extracts image frames from the video data to be processed and downsamples them to obtain downsampled image frames. The downsampled image frames are then input into a pre-trained monocular absolute depth estimation model to obtain a monocular predicted absolute depth map. Using the optimizable depth parameters in the model-predicted absolute depth scale offset recovery module, inconsistent scales and offsets between monocular predicted absolute depth maps are recovered to obtain a scale offset consistent depth map. The scale offset consistent depth map is then projected between downsampled image frames using optimizable camera parameters. The keyframe selection and parameter optimization module is used to extract sparse features of each downsampled image frame using the LoFTR algorithm, match them with other downsampled image frames, select local keyframes, select global keyframes based on the relative rotation angle between two downsampled images, construct a supervision signal L based on the projection results, and train and optimize the depth parameters and camera parameters based on the supervision signal L. The 3D scene reconstruction post-processing module is used to input the monocular predicted absolute depth map and the optimized depth parameters into the model predicted absolute depth scale offset recovery module to obtain a scale offset consistent depth map. The obtained scale offset consistent depth map, along with the downsampled image frame and the optimized camera parameters, is input into the image-depth fusion module to output the 3D scene reconstruction result.

9. A computer device, comprising a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the three-dimensional scene reconstruction method based on neural networks and multi-view consistency as described in any one of claims 1-7.

10. A storage medium storing a program, which, when executed by a processor, implements the three-dimensional scene reconstruction method based on neural networks and multi-view consistency as described in any one of claims 1-7.

Images