A real-time dynamic three-dimensional reconstruction method and system based on patch matching and lightweight monocular depth estimation fusion

Abstract

The application discloses a kind of real-time dynamic three-dimensional reconstruction method and system based on patch matching and lightweight monocular depth estimation fusion, the method is carried out PatchMatch matching to input multi-view image pair, obtains sparse depth information, and is fused with monocular depth estimation result;The depth map after fusion is used to construct three-dimensional Gaussian point cloud, realize high-quality three-dimensional scene reconstruction and new view rendering.The whole process uses end-to-end differentiable training framework, by optimizing pixel-level Gaussian parameter regression network and lightweight depth estimation model, the reconstruction accuracy and rendering efficiency of 3D Gaussian point cloud in dynamic scene are improved.The application can complete training and depth inference in a short time, suitable for real-time or near real-time application scenarios, can be widely used in virtual reality, augmented reality, film and television production and other fields, especially suitable for efficient three-dimensional reconstruction and new view synthesis of target in multi-view dynamic scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, and specifically discloses a real-time dynamic 3D reconstruction method and system based on patch matching and lightweight monocular depth estimation fusion. Background Technology

[0002] Multi-view 3D reconstruction has wide applications in computer vision, virtual reality, and film and television production. In recent years, Neural Radiance Field (NeRF) and its extended methods have made significant progress in novel perspective synthesis. However, NeRF typically requires extensive computation and optimization of sampling points in 3D space, making it difficult to balance rendering quality and real-time performance. Therefore, explicit representation methods such as point cloud rendering have regained popularity, with 3DGS representing a method that significantly improves rendering efficiency while maintaining visual quality. By representing the scene as learnable 3D Gaussian distributed particles, 3DGS demonstrates excellent performance and scalability in real-time rendering and 3D reconstruction tasks in complex scenes.

[0003] In the field of novel perspective synthesis, the GPS-Gaussian method proposes a "direct regression" approach for multi-view inputs. Instead of relying on continuous fine-tuning of individual scenes or characters, it learns generalizable priors for human bodies (or other targets) on large-scale multi-view data, directly outputting 3DGS point clouds frame-by-frame for rapid 3D reconstruction. This method leverages multi-view depth estimation to "lift" pixel-level Gaussian parameters from the 2D plane to 3D space. However, the Multi-View Stereo (MVS) matching used in GPS-Gaussian often employs an iterative depth estimation process, and for targets with complex self-occlusion, such as humans, it requires building large-scale feature or cost volumes for repeated optimization. This approach can lead to excessive computational overhead and long inference times in some dynamic scenes or real-time applications.

[0004] To reduce computational costs while maintaining high depth estimation accuracy, a feasible improvement strategy is to replace the original iterative MVS process with a combination of the PatchMatch algorithm and a lightweight monocular depth estimation network. PatchMatch is a commonly used algorithm for fast matching of local image patches. Through mechanisms such as neighborhood propagation and random search, PatchMatch can search for similar local patches among multiple view images in an approximate manner, quickly converging to a reasonable matching result. Compared with stereo matching methods based on global cost volumes, PatchMatch is more efficient in most texture-rich regions.

[0005] However, PatchMatch may exhibit unstable matching in regions with missing or repetitive textures, and the depth range or absolute scale information cannot be fully constrained by local matching alone. On the other hand, monocular depth estimation techniques have developed rapidly in recent years. Through supervised or self-supervised learning with large amounts of data, monocular models can learn a great deal of prior knowledge and output a reasonable depth distribution in most scenarios. However, due to the lack of geometric constraints between multiple viewpoints, monocular depth estimation often suffers from the "scale ambiguity" problem (i.e., it cannot guarantee that the predicted depth is strictly consistent with the actual physical scale). Summary of the Invention

[0006] Objective of the Invention: The technical problem to be solved by this invention is to address the shortcomings of existing technologies by providing a real-time dynamic 3D reconstruction method and system based on the fusion of patch matching and lightweight monocular depth estimation. By combining pixel-level Gaussian parameter regression of the source view with an end-to-end differentiable rendering framework, the reconstruction speed and rendering quality of 3DGS under sparse viewpoint conditions are significantly improved, achieving real-time reconstruction. The method includes the following steps:

[0007] Step 1: Prepare a monocular depth feature extraction model Initialize PatchMatch matching and monocular depth feature fusion convolutional neural network Encoder E required for preparing a pixel-by-pixel Gaussian parameter prediction model depth and decoders for predicting the rotation, scale, and opacity of Gaussian points, respectively.

[0008] Step 2: Input training image pair (I1, I2) and corresponding pose matrix (T1, T2), as well as camera intrinsic parameter matrix K; where I1 and I2 represent the first and second training images respectively, and T1 and T2 represent the pose matrix corresponding to the first training image I1 and the pose matrix corresponding to the second training image I2 respectively.

[0009] Step 3: Perform patch matching (PatchMatch) on the original pixel space of the image pair (I1, I2) to obtain sparse depth maps D of the matchable regions of I1 and I2 respectively. r1 D r2 and matching confidence S r1 ,S r2 ;

[0010] Step 4: Input the image pair (I1, I2) into the monocular depth prediction model respectively. In the process, the monocular depth feature map of I1 is obtained. Monocular depth feature map of I2 Where S is the dimension of the monocular depth feature map;

[0011] Step 5, convert the sparse depth map D r1 D r2 Matching confidence S r1 ,S r2 Monocular depth feature map The input is concatenated into a multi-channel input, forming a fused feature with dimensions of (S+2)×H×W, and then processed through a convolutional neural network. The fused features are subjected to confidence-based feature fusion, outputting the absolute depth maps D1 and D2 of the globally scale-aligned images I1 and I2; H and W are the height and width of the feature map, respectively, and the sparse depth map D... r1 D r2 And matching confidence S r1 ,S r2 The height and width of the monocular depth feature map need to be interpolated using bilinear interpolation for merging;

[0012] Step 6: Using absolute depth maps D1 and D2, the two-dimensional parameters are uplifted to three-dimensional space to construct a preliminary 3D Gaussian distribution, and then encoded by encoder E. depth and decoder Obtain the pixel-by-pixel position matrix M p Color matrix M c Rotation matrix M r Scaling matrix M s and the opacity matrix M a The color matrix of the Gaussian points is obtained from the training image pair (I1, I2);

[0013] Step 7: Obtain images C1 and C2 from the viewpoints of camera pose matrices T1 and T2 respectively through volume rendering;

[0014] Step 8: Repeat steps 2 to 7, iterating through image pairs at more than two time points, and updating the convolutional neural network. The parameters of the Gaussian parameter prediction model are calculated until the optimization converges;

[0015] After training, an end-to-end real-time dynamic 3D reconstruction model is obtained, which has the ability to generalize Gaussian distribution regression to any new scene image pair.

[0016] In step 4, monocular depth feature map and Using a monocular depth prediction model The calculation yields the following formula:

[0017]

[0018] In step 6, the two-dimensional parameters are elevated to three-dimensional space using the following formula:

[0019]

[0020] M c (i,j)=I(i,j)

[0021] Where M p (i,j) represents the position matrix M p The i-th row and j-th column; M c (i,j) represents the i-th row and j-th column of the color matrix; D(i,j) represents the i-th row and j-th column of the absolute depth map D; I(i,j) represents the i-th row and j-th column of the image I; I represents image I1 or I2; D represents absolute depth map D1 or D2.

[0022] In step 6, the expression for the Gaussian radiation sphere is:

[0023]

[0024] Among them G i (X) represents the distribution of the i-th Gaussian radiative sphere at spatial location X, μ i Let Σ be the center position of the i-th Gaussian radiation sphere, e be the natural constant, and Σ be the center position of the sphere. i From the rotation quaternion r i and scaling i Decision: Σ i =RGG T R T Where G is the scale s i The parameters obtained by diagonalization, R, are obtained by rotating the quaternion r. i The resulting rotation matrix; μ i ,r i ,s i They represent the matrix M respectively p M r M s The i-th element extracted from the previous step;

[0025] The parameter r of the Gaussian radiation sphere i s i and opacity α i By encoder E depth and decoder get:

[0026]

[0027]

[0028] Where I is the input image, Features at different scales are extracted for monocular depth, where D is the calculated monocular depth; Normalize is the normalization operation, and ClampMax is the maximum value truncation operation.

[0029] Step 7 includes:

[0030] Obtain the color C(i,j) through volume rendering:

[0031]

[0032] Where M is a subset of Gaussian points involved in pixel (i,j), and the elements of the subset are sorted by depth; c k Indicates from matrix M c The k-th element obtained from the decomposition; a j ,a k They represent the matrix M respectively a The j-th and k-th elements that were extracted from the decomposition.

[0033] In step 8, the following optimization loss function L is established:

[0034] L=αL color +βL ssim +γL d

[0035] Where α is the color L1 loss weight, β is the color structure similarity loss weight, γ is the depth L1 loss, and L... color Let L represent the color L1 loss function, L ssim L represents the structural similarity loss function. d The loss is L1 depth.

[0036] The loss function L is optimized to calculate the loss. Backpropagation is used to obtain the gradients of other parameters to be updated, and the gradient descent algorithm is used to update the parameters to be updated.

[0037] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0038] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described.

[0039] The present invention also provides a real-time reconstruction system for three-dimensional scenes, characterized in that it includes:

[0040] The input module is used to receive training image pairs (I1,I2), pose matrix (T1,T2), and camera intrinsic parameter matrix K;

[0041] The PatchMatch module is used to perform patch matching (PatchMatch) on the original pixel space of the image pair (I1,I2) to obtain a sparse depth map D. r1 D r2 ;

[0042] The monocular depth prediction module is used to input image pairs (I1, I2) into the monocular depth prediction model to obtain monocular depth feature maps.

[0043] The feature fusion module is used to concatenate the sparse depth map with the monocular depth feature map and then pass the convolutional neural network. Perform feature fusion and output the global scale-aligned absolute depth maps D1 and D2.

[0044] The 3D Gaussian distribution construction module is used to elevate two-dimensional parameters to three-dimensional space using absolute depth maps D1 and D2, and construct a preliminary 3D Gaussian distribution.

[0045] The rendering module is used to render images C1 and C2 from the viewpoint of the camera pose matrix T1 and T2.

[0046] The optimization module iterates through two or more image pairs, updating the weights of the convolutional neural network and the Gaussian parameter prediction network until the optimization converges.

[0047] The system also includes a training module, which enables the model to generalize its ability to render new perspectives on any new scene dynamic multi-view image pairs after training is completed.

[0048] This invention significantly improves the reconstruction accuracy and rendering efficiency of 3D Gaussian point clouds (3DGS) in dynamic scenes by introducing a multi-view depth acquisition strategy that combines PatchMatch with monocular depth estimation. Its advantages and beneficial effects are mainly reflected in the following aspects:

[0049] (1) Improved real-time performance and efficiency: Compared with the traditional iterative MVS method, this invention uses PatchMatch for fast matching of local features and monocular depth estimation to provide a complete depth distribution that may have scale bias. The two complement each other, enabling the entire depth estimation process to be completed in a short time, greatly reducing the dependence on large-scale cost volumes or long iterations, and providing a highly efficient solution for dynamic scenes or real-time applications.

[0050] (2) Geometric accuracy and robustness: By using the sparse depth obtained from PatchMatch as the basis for global correction, the inherent scale ambiguity problem of monocular depth estimation is effectively overcome. Even in complex scenes such as human figures with missing textures or severe self-occlusion, the fusion strategy of this invention can still obtain relatively accurate depth information, thereby significantly improving the geometric reconstruction accuracy of scenes or figures in 3DGS representation.

[0051] (3) End-to-end differentiable training framework: This invention maintains differentiability throughout the entire process from multi-view depth to 3DGS and then to multi-view rendering, supporting end-to-end optimization of pixel-level Gaussian parameter prediction networks and feature fusion CNNs. During training, all network modules are updated collaboratively, gradually improving the synthesis quality of 3D Gaussian point clouds under new perspectives, providing a solid generalization foundation for rapid inference on any new scene or character.

[0052] In summary, this invention, by comprehensively utilizing the advantages of PatchMatch and monocular depth estimation, achieves 3D reconstruction and new perspective synthesis of targets in dynamic scenes or sparse viewpoints in an end-to-end differentiable manner, effectively improving reconstruction speed and geometric accuracy, and providing an effective solution for real-time, high-quality 3DGS reconstruction. Attached Figure Description

[0053] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0054] Figure 1 This is a schematic diagram of the method of the present invention.

[0055] Figure 2 This is the result of implementing the present invention at 2K resolution. Detailed Implementation

[0056] This invention provides a real-time dynamic 3D reconstruction method based on patch matching and lightweight monocular depth estimation, applicable to dynamic scenes or real-time 3D reconstruction tasks. This embodiment achieves efficient and accurate 3D reconstruction through steps such as image data preprocessing, depth estimation, and the construction and rendering of 3D Gaussian point clouds. Specifically, this invention includes the following steps:

[0057] Step 1: Input data preparation and preprocessing;

[0058] First, input training image pairs (I1, I2) and their corresponding pose matrices (T1, T2), along with the camera's intrinsic parameter matrix K. The key to this step is collecting image pairs with sufficient viewpoint variations to ensure that subsequent 3D reconstruction and depth estimation have a rich source of information. Furthermore, prepare a monocular depth feature extraction model. Initialize PatchMatch matching and monocular depth feature fusion convolutional neural network This is used for subsequent depth estimation and feature fusion. Finally, the encoder E required to initialize the pixel-by-pixel Gaussian parameter prediction model is... depth and decoders for predicting the rotation, scale, and opacity of Gaussian points, respectively.

[0059] Step 2: Execute the PatchMatch algorithm;

[0060] like Figure 1 As shown in the module "PatchMatch", for image pairs (I1, I2), the PatchMatch algorithm is executed in the original pixel space to calculate the sparse depth map D of each image. r1 D r2 and matching confidence S r1 ,S r2 The PatchMatch algorithm efficiently provides depth estimation results for regions with obvious textures through rapid matching of local image patches. This step is particularly suitable for textured regions in complex scenes, enabling the rapid acquisition of relatively accurate local depth information.

[0061] Step 3: Monocular depth estimation;

[0062] Next, each image I1 and I2 will be input into the lightweight monocular depth feature extraction model. Obtain monocular depth feature map like Figure 1 The "Monocular Depth Prediction" module is shown in the figure. This model extracts spatial information from a single image using deep learning methods, and despite scale ambiguity, it can provide useful depth priors in most cases. Training must be performed completely for both images; the figure only shows training for one of the images.

[0063] Step 4: Feature fusion and depth correction;

[0064] In this step, the sparse depth map D output by the PatchMatch algorithm is... r1 D r2 Matching confidence S r1 ,S r2 and monocular depth feature map Concatenate the input feature maps to form a multi-channel input feature map with dimensions of (S+2)×H×W. Then, use... Feature fusion is performed to obtain globally scale-aligned absolute depth maps D1 and D2. This process effectively corrects the scale ambiguity of monocular depth and further improves the accuracy of depth estimation.

[0065] Step 5: Construct a 3D Gaussian point cloud;

[0066] Using the depth map obtained in step 4, the two-dimensional pixel depth information is "lifted" into three-dimensional space, thus constructing a preliminary 3D Gaussian point cloud. The depth information of each pixel is mapped to the position M of a Gaussian particle in three-dimensional space. p (i,j) and color M c (i,j), the formula is as follows:

[0067]

[0068] M c (i,j)=I(i,j)

[0069] Step 6: Gaussian distribution modeling;

[0070] Figure 1 The "3DGS parameters" in the text refer to the parametric modeling of a three-dimensional Gaussian radiant sphere. The expression for the three-dimensional Gaussian radiant sphere is:

[0071]

[0072] Among them G i (X) represents the distribution of the i-th Gaussian radiative sphere at spatial location X, μ i Let Σ be the center position of the i-th Gaussian radiation sphere, e be the natural constant, and Σ be the center position of the sphere. i From the rotation quaternion r i and scaling i Decision: Σ i =RGG T R T Where G is the scale s i The parameters obtained by diagonalization, R, are obtained by rotating the quaternion r. i The resulting rotation matrix; μ i ,r i ,s i They represent the matrix M respectively p M r M s The i-th element extracted from the previous step;

[0073] like Figure 1 As shown in the "3DGS Parameter Prediction Network" module, this module outputs the parameter r of the Gaussian radiation sphere. i s i and opacity α i Specifically, this module consists of encoder E depth and decoder The composition and calculation method of the Gaussian radiation sphere parameters are as follows:

[0074]

[0075] Step 7: View rendering;

[0076] correspond Figure 1 The rendering module uses the obtained 3D Gaussian point cloud to render the new viewpoints T1 and T2, resulting in rendered images C1 and C2. The formula for obtaining the color C(i,j) of pixel (i,j) in volume rendering is:

[0077]

[0078] Where M is a subset of Gaussian points involved in pixel (i,j), and the elements of the subset are sorted by depth. During the rendering process, the final rendered image is obtained by weighted summation of the color and transparency of all Gaussian points.

[0079] Step 8: Iterative optimization and training;

[0080] The above steps will be repeated, continuously updating the weights of the convolutional neural network and the pixel-level Gaussian parameter prediction network by traversing multiple image pairs until the optimization process converges. The overall optimization loss function L used in the training process consists of the following parts:

[0081] L=αL color +βL sdim +γL d

[0082] Where α is the color L1 loss weight, β is the color structure similarity loss weight, γ is the depth L1 loss, and L... cslor Let L represent the color L1 loss function, L ssim L represents the structural similarity loss function. d The loss is L1 depth.

[0083] After training, the resulting model can perform Gaussian distribution regression on new scenes and targets, thereby achieving 3D reconstruction and new perspective synthesis for any new scene. The trained model has strong generalization ability, can quickly respond to dynamic scene changes, and provide real-time 3D reconstruction results.

[0084] This invention proposes using PatchMatch to mine reliably matching multi-view feature point depths and using the results as a "scale reference" for lightweight monocular depth estimation, thereby achieving global correction of monocular depth. In pixel regions rich in texture or feature information, PatchMatch can obtain relatively accurate and stable multi-view matching depths. For regions where PatchMatch is difficult to match or where texture degradation exists, monocular depth estimation can provide reasonable initial depth values. By using the sparse depth obtained from PatchMatch as a constraint, the monocular depth output can be aligned on a global scale, avoiding large-scale scale drift generated by the monocular network. Compared to traditional iterative MVS, local feature search and monocular depth alignment of PatchMatch can complete global depth inference in a shorter time, which helps to achieve fast processing in dynamic scenes or real-time applications.

[0085] Finally, using the multi-view depth results obtained through the above fusion, the Gaussian particle parameters (position, color, opacity, variance matrix, etc.) learned in the view plane can be "lifted" into the 3D coordinate system, and the 3D Gaussian point clouds from multiple perspectives can be aggregated to complete high-fidelity 3D reconstruction and new perspective rendering of characters or scenes. Simultaneously, this process is differentiable throughout, from multi-view depth to 3D Gaussian Splatting and then to multi-view RGB rendering, laying the foundation for end-to-end training of pixel-level 3D Gaussian particle parameter prediction models. Because PatchMatch, combined with a lightweight monocular depth estimation scheme, balances speed and accuracy in depth information acquisition and has strong matching capabilities for key feature points in complex self-occlusion scenes, it can effectively replace the time-consuming MVS process in GPS-Gaussian, thus meeting the needs of real-time or near-real-time applications. In an Nvidia-A6000 graphics card environment, 13 FPS can be achieved when rendering at 2K resolution with an input resolution of 1K, and 33 FPS when rendering at 1K resolution with an input resolution of 512×512.

[0086] The embodiments of this invention are trained and validated on the THuman2.0 dataset, and the validation results are as follows: Figure 2 As shown, the left and right perspectives are the inputs to the model, and the rendering result of the new perspective is the output of the model.

[0087] In summary, this invention, by integrating PatchMatch multi-view matching and monocular depth estimation, effectively overcomes the scale ambiguity of monocular depth estimation while maintaining relatively reliable depth accuracy and significantly reducing computational load. Combined with the fast rendering capabilities of 3DGS, this invention can achieve real-time 3D reconstruction and new perspective synthesis of target characters or scenes in sparse multi-view scenarios, providing a novel and efficient technical solution for practical applications such as sports broadcasting, virtual events, and interactive entertainment.

[0088] This invention successfully solves the problems of high computational cost, slow speed, and inconsistent depth scales in multi-view depth estimation by combining the PatchMatch algorithm with lightweight monocular depth estimation technology. Simultaneously, it achieves a balance between real-time performance and high accuracy by utilizing the 3DGS method for efficient 3D reconstruction. This technology provides a practical solution for real-time 3D reconstruction in fields such as virtual reality, sports broadcasting, and interactive entertainment.

[0089] This invention provides a real-time dynamic 3D reconstruction method and system based on patch matching and lightweight monocular depth estimation fusion. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A real-time dynamic 3D reconstruction method based on patch matching and light-weight monocular depth estimation fusion, characterized in that, The method comprises the following steps: Step 1, preparing monocular depth prediction model Initializing patch matching PatchMatch and monocular depth feature fusion convolutional neural network Preparing encoder E needed for pixel-wise Gaussian parameter prediction model depth And decoder for predicting rotation, scale and opacity of Gaussian points respectively Step 2, inputting a training image pair (I1, I2) and corresponding pose matrices (T1, T2), and an intrinsic matrix K of a camera; wherein I1 and I2 respectively represent a first training image and a second training image inputted, and T1 and T2 respectively represent a pose matrix corresponding to the first training image I1 and a pose matrix corresponding to the second training image I2; Step 3. Perform patch matching PatchMatch in the original pixel space of image pair (I1, I2) to get sparse depth maps D r1 ,D r2 and matching confidence S r1 ,S r2 respectively. Step 4, input the image pair (I1, I2) into the monocular depth prediction model respectively to obtain the monocular depth feature map of I1 and the monocular depth feature map of I2 where S is the dimension of the monocular depth feature map; Step 5, sparse depth map I r1 ,D r2 , matching confidence S r1 ,S r2 , monocular depth feature map is spliced into a multi-channel input to form a fusion feature of a dimension of (S+2) x H x W, and the fusion feature is subjected to confidence-based feature fusion through a convolutional neural network to output absolute depth maps D1, D2 of the globally scale-aligned images I1, I2; H and W are respectively the height and width of the feature map, and the sparse depth map D r1 ,D r2 and the matching confidence S r1 ,S r2 need to be bilinearly interpolated to the height and width of the monocular depth feature map for merging; Step 6, using the absolute depth maps D1, D2, lift the 2D parameters to 3D space, construct a preliminary 3D Gaussian distribution, and pass through the encoder E depth and decoder to get the pixel-wise position matrix M p , color matrix M c , rotation matrix M r , scale matrix M s and opacity matrix M a ; the color matrix of the Gaussian points is obtained from the training image pair (I1, I2); Step 7, obtaining images C1 and C2 under the perspectives of the camera pose matrices T1 and T2 respectively through volume rendering; Step 8, repeat steps 2 to 7, traverse the image pairs of more than two time instants, update the convolutional neural network and the parameters of the Gaussian parameter prediction model until the optimization converges.

2. The method of claim 1, wherein, In step 4, monocular depth feature map and by the monocular depth prediction model is calculated, and the formula is:

3. The method of claim 2, wherein, In step 6, the two-dimensional parameters are lifted to a three-dimensional space by using the following formula: M c (i,j) = I(i,j) Where M p (i,j) represents the position matrix M p The i-th row and j-th column; M c (i,j) represents the i-th row and j-th column of the color matrix; D(i,j) represents the i-th row and j-th column of the absolute depth map D; I(i,j) represents the i-th row and j-th column of the image I; I represents image I1 or I2; D represents absolute depth map D1 or D2.

4. The method of claim 3, wherein, In step 6, the expression of the Gaussian radiosity sphere is as follows: where G i (X) is the distribution of the ith Gaussian radiating sphere at spatial location X, μ i is the center location of the ith Gaussian radiating sphere, e is the natural constant, Σ i is determined by the rotation quaternion r i and the scaling s i : Σ i = RGG T R T ; where G is the scale s i The resulting parameters from diagonalization, R is the quaternion rotation r i The resulting rotation matrix; μ i ,r i ,s i M p M r M s The ith element from the upper decomposition of M where the parameters r of the Gaussian radiance sphere i , s i and opacity a i are obtained by the encoder E depth and the decoder ​ where I is an input image, is the feature of different scales for monocular depth extraction, D is the calculated monocular depth; Normalize is a normalization operation, and ClampMax is a maximum truncation operation.

5. The method of claim 4, wherein, Step 7 comprises: Obtaining the color C(i, j) through volume rendering: where M is the subset of Gaussian points involved for pixel (i,j), the elements of the subset are sorted by depth; c k denotes the element of the matrix M c disassembled kth element;a j denotes the element of the matrix M k denotes the jth element and the kth element of the matrix M a disassembled respectively.

6. The method of claim 5, wherein, In step 8, the following optimization loss function L is established: L = aL color + βL ssim + γL d where a is the color L1 loss weight, b is the color structural similarity loss weight, g is the depth L1 loss weight, L color represents the color L1 loss function, L ssim represents the structural similarity loss function, L d is the depth L1 loss.

7. An electronic device, comprising: The system comprises a processor and a memory, and the memory stores program code, which, when executed by the processor, causes the processor to execute the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The system stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

9. A real-time three-dimensional scene reconstruction system implemented using the method of any one of claims 1 to 6. The system comprises: An input module configured to receive a training image pair (I1, I2), pose matrices (T1, T2) and an intrinsic matrix K of a camera; a PatchMatch module configured to perform patch matching PatchMatch in the original pixel space of the image pair (I1, I2) to obtain a sparse depth map D r1 ,D r2 ; a monocular depth prediction module configured to input the image pair (I1, I2) into a monocular depth prediction model to obtain a monocular depth feature map The feature fusion module is configured to splice the sparse depth map and the monocular depth feature map and pass them through a convolutional neural network The feature fusion is performed, and the global scale aligned absolute depth map D1, D2 is output. A three-dimensional Gaussian distribution construction module configured to lift two-dimensional parameters to a three-dimensional space by using absolute depth maps D1 and D2, and to construct a preliminary 3D Gaussian distribution; A rendering module configured to render images C1 and C2 under the perspectives of camera pose matrices T1 and T2; An optimization module configured to iterate through two or more image pairs, and to update the weights of the convolutional neural network and the Gaussian parameter prediction network until optimization converges.

10. The system of claim 9, wherein, The system further comprises a training module configured to, after training is completed, enable the model to have a generalization capability of rendering new perspectives for dynamic multi-perspective image pairs of any new scene.

Images