Three-dimensional scene reconstruction method, electronic equipment, storage medium and program product

Abstract

The invention discloses a three-dimensional scene reconstruction method, electronic equipment, a storage medium and a program product, and relates to the technical field of data processing, and the method comprises the steps: inputting each frame of multi-view image into a depth model, and obtaining a prediction depth map of each frame of multi-view image; for any target frame image in each frame of multi-view image, optimizing a local 3D Gaussian model and camera attitude information of the target frame image based on the predicted depth map of the target frame image and the depth information of the previous frame image to obtain an updated local 3D Gaussian model and updated camera attitude information of the target frame image; and optimizing the global 3D Gaussian model to obtain an updated global 3D Gaussian model based on the updated local 3D Gaussian model and the updated camera attitude information of each frame of multi-view image, and performing three-dimensional scene reconstruction based on the updated global 3D Gaussian model. According to the invention, the accuracy of a view synthesis result based on three-dimensional scene reconstruction is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a three-dimensional scene reconstruction method, an electronic device, a storage medium, and a program product. Background Art

[0002] The novel view synthesis task refers to rendering a target image corresponding to a target pose given a source image, a source pose, and a target pose. In the novel view synthesis task, the technology based on three-dimensional Gaussian reconstruction has shown significant advantages in fast reconstruction and real-time rendering. Among them, the CF-3DGS (Continuous Fusion-Three Dimensional Gaussian Splatting) technology realizes the functions of scene reconstruction and novel view synthesis without the need for precise pose input by leveraging the characteristics of local and global Gaussians.

[0003] However, the CF-3DGS technology lacks sufficient geometric constraints during the training process, resulting in an inability to fully and accurately grasp the geometric relationships in the scene, which causes certain deviations in the model during the learning process, thus affecting the accuracy of the synthesized images. In addition, the global Gaussian point cloud growth in the CF-3DGS technology adopts an incremental approach. As the point cloud is continuously added, there is a lack of an effective coordination and integration mechanism between the newly added point cloud and the existing point cloud, leading to artifacts and floating objects in the synthesized results obtained during view synthesis, thereby affecting the accuracy and quality of the synthesized images.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a three-dimensional scene reconstruction method, an electronic device, a storage medium, and a program product, aiming to solve the technical problem that the accuracy of the view synthesis results based on three-dimensional Gaussian reconstruction is affected.

[0006] To achieve the above objective, this application proposes a three-dimensional scene reconstruction method, and the method includes:

[0007] Inputting each frame of multi-view images into a depth model to obtain a predicted depth map for each frame of the multi-view images;

[0008] For any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimize the local 3D (Three Dimensional) Gaussian model and the camera pose information of the target frame image to obtain the updated local 3D Gaussian model and the updated camera pose information of the target frame image;

[0009] Based on the updated local 3D Gaussian models and the updated camera pose information of each frame of the multi-view images, optimize the global 3D Gaussian model to obtain an updated global 3D Gaussian model, wherein three-dimensional scene reconstruction is performed based on the updated global 3D Gaussian model.

[0010] In one embodiment, the step of, for any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimizing the local 3D Gaussian model and the camera pose information of the target frame image to obtain the updated local 3D Gaussian model and the updated camera pose information of the target frame image includes:

[0011] For any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimize the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and an updated depth model;

[0012] Input the target frame image into the updated depth model to obtain the updated depth map of the target frame image;

[0013] Determine the next frame image of the target frame image as the target frame image, determine the depth information of the previous frame image based on the updated depth map of the target frame image, execute the step of optimizing the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model and subsequent steps until all frames of the multi-view images are traversed, to obtain the updated local 3D Gaussian models and the updated camera pose information of each frame of the multi-view images.

[0014] In one embodiment, the step of, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimizing the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and an updated depth model includes:

[0015] Initialize the local 3D Gaussian model of the target frame image based on the depth information of the previous frame image to obtain an initialized local 3D Gaussian model;

[0016] Based on the predicted depth map of the target frame image and the initialized local 3D Gaussian model, update the initialized local 3D Gaussian model, the camera pose information of the target frame image, and the depth model to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and an updated depth model.

[0017] In one embodiment, the depth information includes a predicted depth map and affine transformation parameters;

[0018] The step of initializing the local 3D Gaussian model of the target frame image based on the depth information of the previous frame image to obtain an initialized local 3D Gaussian model includes:

[0019] Perform a depth alignment operation on the predicted depth map of the previous frame image and the affine transformation parameters of the previous frame image to obtain the aligned depth information of the previous frame image;

[0020] Initialize the local 3D Gaussian distribution of the target frame image through the aligned depth information of the previous frame image to obtain an initialized local 3D Gaussian model.

[0021] In one embodiment, the step of updating the initialized local 3D Gaussian model, the camera pose information, and the depth model based on the predicted depth map of the target frame image and the initialized local 3D Gaussian model to obtain an updated local 3D Gaussian model, updated camera pose information, and an updated depth model includes:

[0022] Render the predicted depth map of the target frame image based on the initialized local 3D Gaussian model and the camera pose of the target frame image to obtain a target frame rendered depth map, and render the target frame image based on the initialized local 3D Gaussian model to obtain a target frame rendered image;

[0023] Determine the depth loss corresponding to the target frame image based on the target frame rendered depth map and the predicted depth map of the target frame image, and determine the color value loss corresponding to the target frame image based on the target frame image and the target frame rendered image;

[0024] Based on the depth loss corresponding to the target frame image and the color value loss corresponding to the target frame image, update the initialized local 3D Gaussian model through backpropagation to obtain an updated local 3D Gaussian model, update the camera pose information of the target frame image to obtain the updated camera pose information of the target frame image, and update the parameters of the depth model to obtain an updated depth model.

[0025] In one embodiment, the step of optimizing the global 3D Gaussian model to obtain an updated global 3D Gaussian model based on the updated local 3D Gaussian models and updated camera pose information of each frame of the multi-view images includes:

[0026] Determine the initialization point of the global 3D Gaussian operation based on the aligned depth information of the first frame image and the last frame image in each frame of the multi-view images, the affine transformation parameters of the first frame image and the last frame image, the updated depth maps of each frame of the multi-view images, the updated local 3D Gaussian models of each frame of the multi-view images, and the updated camera pose information of each frame of the multi-view images;

[0027] Optimize the parameters of the global 3D Gaussian model based on the initialization point, the updated local 3D Gaussian models of each frame of the multi-view images, the updated camera pose information of each frame of the multi-view images, and the depth loss and color value loss corresponding to each frame of the multi-view images to obtain an updated global 3D Gaussian model.

[0028] In one embodiment, the depth model is a monocular depth network, and the monocular depth network includes a feature extraction block, an upsampling block, and an output layer;

[0029] The step of inputting each frame of the multi-view images into the depth model to obtain the predicted depth map of each frame of the multi-view images includes:

[0030] Traverse each frame of the multi-view images, and input the multi-view images into the feature extraction block to obtain first feature information;

[0031] Input the feature information into the upsampling block to obtain second feature information;

[0032] Input the second feature information into the output layer to obtain the predicted depth map of the multi-view images.

[0033] In addition, to achieve the above object, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the three-dimensional scene reconstruction method as described above.

[0034] In addition, to achieve the above object, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the three-dimensional scene reconstruction method as described above.

[0035] In addition, to achieve the above object, the present application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the three-dimensional scene reconstruction method described above.

[0036] In the present application, each frame of multi-view images is input into a depth model to obtain the predicted depth map of each frame of the multi-view images. For any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, the local 3D Gaussian model and the camera pose information of the target frame image are optimized to obtain the updated local 3D Gaussian model and the updated camera pose information of the target frame image. Compared with the lack of geometric constraints of inter-frame information in the CF-3DGS technology, by using the predicted depth map of the target frame image and the depth information of the previous frame image to optimize the local 3D Gaussian model and the camera pose information of the target frame image, geometric constraints can be provided for the update of the local 3D Gaussian model based on the coherence of depth information between adjacent frames, making the updated local 3D Gaussian model more in line with the actual scene structure. At the same time, the change in depth information can also reflect the movement or rotation of the camera in 3D space, thereby improving the accuracy of the camera pose information and providing a more accurate observation perspective for three-dimensional Gaussian reconstruction.

[0037] Based on the updated local 3D Gaussian model and the updated camera pose information of each frame of the multi-view images, the global 3D Gaussian model is optimized to obtain an updated global 3D Gaussian model, and three-dimensional scene reconstruction is performed based on the updated global 3D Gaussian model. The updated local 3D Gaussian model of each frame of image carries the local scene structure information optimized by depth information, and the updated camera pose information reflects the accurate information of each perspective. By integrating this information to optimize the global 3D Gaussian model, the accuracy of the global 3D Gaussian model can be improved.

[0038] Compared with CF-3DGS, the present application provides geometric constraints for model update by synthesizing depth information between frames, and optimizes the global 3D Gaussian model from local to global, which can avoid the unreasonable distribution problem caused by the lack of constraints during the growth of the global Gaussian point cloud, reduce the inconsistency and unreasonable overlap between local regions, thereby avoiding artifacts and floating objects in view synthesis, and improving the accuracy and integrity of the view synthesis result based on three-dimensional scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0040] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a schematic flowchart provided for the first embodiment of the three-dimensional scene reconstruction method of the present application;

[0042] Figure 2 It is a schematic flowchart provided for the second embodiment of the three-dimensional scene reconstruction method of the present application;

[0043] Figure 3 It is a schematic flowchart of model training in the three-dimensional scene reconstruction method provided for an embodiment of the present application;

[0044] Figure 4 It is a schematic module structure diagram of the three-dimensional scene reconstruction device in the embodiment of the present application;

[0045] Figure 5 It is a schematic device structure diagram of the hardware operating environment involved in the three-dimensional scene reconstruction method in the embodiment of the present application.

[0046] The implementation, functional features, and advantages of the object of the present application will be further described with reference to the accompanying drawings in combination with the embodiments. Detailed Embodiments

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] To better understand the technical solutions of the present application, the following will be described in detail in combination with the accompanying drawings of the specification and specific embodiments.

[0049] The main solution of the embodiment of the present application is: inputting each frame of multi-view images into a depth model to obtain the predicted depth map of each frame of the multi-view images; for any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimizing the local 3D Gaussian model and camera pose information of the target frame image to obtain the updated local 3D Gaussian model and updated camera pose information of the target frame image; based on the updated local 3D Gaussian model and updated camera pose information of each frame of the multi-view images, optimizing the global 3D Gaussian model to obtain an updated global 3D Gaussian model, wherein three-dimensional scene reconstruction is performed based on the updated global 3D Gaussian model.

[0050] In this embodiment, for the convenience of description, the following will be described with an electronic device as the execution subject.

[0051] In the novel view synthesis task, a model trained based on NeRF (Neural Radiance Field) can calculate the interaction between light rays and objects in the scene at the target pose, including absorption, reflection, and refraction of light rays, etc., based on the source image, source pose, and target pose, and then render a highly realistic target image. However, in real-world applications, accurate poses are not always feasible, and the preprocessing of camera parameters highly depends on offline methods such as SFM (Structure From Motion). This preprocessing is not only time-consuming but also often fails due to being sensitive to errors in feature extraction and difficulties in processing textureless or repetitive regions.

[0052] To reduce the dependence on camera parameters, a large amount of work focuses on directly integrating camera pose estimation into the neural radiance field framework. To solve the problems of 3D scene reconstruction and camera registration simultaneously, the optimization process usually involves additional constraints. In particular, NeRFMM first proposed an end-to-end pipeline to jointly optimize camera parameters and NeRF, but it is restricted by the scene; BARF analyzed the theoretical relationship between planar image alignment and neural radiance field reconstruction and proposed coarse-to-fine positional encoding, which requires an initial pose close to the ground truth scene; NOPE-NeRF integrated monocular depth maps into the joint estimation pipeline, improving the estimation quality, but it requires a long training time. In short, camera pose optimization in NeRF is achieved indirectly by updating ray casting of the camera position. The recently emerged 3D Gaussian reconstruction has stood out in the fields of novel view synthesis and 3D scene reconstruction due to the combination of high quality, fast reconstruction speed, and support for real-time rendering. The latest work, CF-3DGS, achieved a method for performing view synthesis without SFM preprocessing based on the continuity of the video and the display representation of 3DGS. However, the CF-3DGS technology lacks sufficient geometric constraints during the training process, resulting in an inability to fully and accurately grasp the geometric relationships in the scene, which causes certain biases in the model during the learning process, thus affecting the accuracy of the synthesized image. In addition, the global Gaussian point cloud growth in the CF-3DGS technology adopts an incremental approach. As the point cloud is continuously added, there is a lack of an effective coordination and integration mechanism between the newly added point cloud and the existing point cloud, resulting in artifacts and floating objects in the synthesized result obtained from view synthesis, thus affecting the accuracy and quality of the synthesized image.

[0053] The present application provides a solution. By integrating the depth information between each frame, geometric constraints are provided for model update, and the global 3D Gaussian model is optimized from local to global, which can avoid the unreasonable distribution problem caused by the lack of constraints during the growth process of the global Gaussian point cloud, reduce the inconsistency and unreasonable overlap between local regions, thereby avoiding artifacts and floating objects in view synthesis, and thus improving the accuracy and integrity of the view synthesis result based on three-dimensional scene reconstruction.

[0054] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, an electronic device, etc. that can implement the above functions. Hereinafter, an electronic device is taken as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the embodiment of the present application provides a three-dimensional scene reconstruction method, referring to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the three-dimensional scene reconstruction method of the present application.

[0056] In this embodiment, the three-dimensional scene reconstruction method includes steps S10 to S30:

[0057] Step S10, input each frame of multi-view image into a depth model to obtain the predicted depth map of each frame of the multi-view image;

[0058] The multi-view image is a sequence of images taken from different angles for capturing multiple perspective information of the scene. The predicted depth map is the depth map output by the depth model, representing the depth information of each pixel point in the image. Each frame of the multi-view image sequence is used as the input of the depth model, and the depth model converts the visual information in the multi-view image into a predicted depth map. It should be noted that the depth model can be a convolutional neural network or other neural networks suitable for depth estimation, and the specific architecture is not limited here. The depth model is pre-trained on a large amount of image data, and the training data includes images with known depth information and corresponding depth maps. During the training process, a loss function (such as the mean square error loss function) is used to measure the difference between the predicted depth map and the true depth map, and the parameters of the depth model are optimized through the backpropagation algorithm, so that the depth model can accurately predict the depth information from the input image. The specific training process is not elaborated here.

[0059] It can be understood that by predicting the depth map through the depth model, the depth information of the scene can be mined from the multi-view image, providing an accurate depth basis for subsequent 3D Gaussian model optimization and three-dimensional scene reconstruction.

[0060] Step S20. For any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimize the local 3D Gaussian model and the camera pose information of the target frame image to obtain the updated local 3D Gaussian model and the updated camera pose information of the target frame image;

[0061] The target frame image is the image frame currently being processed in the multi-view image sequence. The predicted depth map of the target frame image contains the scene depth information reflected by the target frame image. The depth information of the previous frame image of the target frame image provides the scene depth information under adjacent viewpoints. The depth information may specifically include information such as the predicted depth map and affine transformation parameters. The local 3D Gaussian model is a Gaussian distribution model used to represent the local geometric structure of the target frame image scene, including parameters such as mean, covariance, and weight. The camera pose information is used to characterize the camera position and orientation information of the target frame image.

[0062] In the embodiment, align the predicted depth map of the target frame with the depth information of the previous frame, optimize the relative pose of the target frame by minimizing the depth error to obtain the updated camera pose information (i.e., the updated camera pose information), and initialize the local 3D Gaussian model of the target frame based on the aligned depth information to obtain the updated local 3D Gaussian model (i.e., the updated local 3D Gaussian model).

[0063] It can be understood that by optimizing based on the depth information of adjacent frames, due to the coherence of the multi-view image sequence, the local 3D Gaussian model and the camera pose information can more accurately reflect the local details and viewpoint changes of the scene, avoiding the limitations of single-frame information, providing more accurate and continuous local information for the subsequent optimization of the global 3D Gaussian model, thereby improving the accuracy and stability of the entire three-dimensional scene reconstruction.

[0064] Step S30. Based on the updated local 3D Gaussian model and the updated camera pose information of each frame of the multi-view images, optimize the global 3D Gaussian model to obtain the updated global 3D Gaussian model, wherein three-dimensional scene reconstruction is performed based on the updated global 3D Gaussian model.

[0065] The global 3D Gaussian model is a Gaussian distribution model used to represent the global geometric structure of the entire scene, integrating the information of all frames. In this embodiment, integrate the local 3D Gaussian model and the camera pose information of each frame into the global 3D Gaussian model, adopt a random growth strategy, randomly select frames to participate in global optimization, avoid error accumulation caused by incremental methods, and optimize the parameters of the global 3D Gaussian model by minimizing the global error (such as depth error, photometric error, etc.) to obtain the updated global 3D Gaussian model.

[0066] Perform three-dimensional scene reconstruction based on the updated global 3D Gaussian model. Specifically, by mapping the Gaussian distribution in the global 3D Gaussian model to 3D space, construct the geometric structure and texture of the scene according to information such as the central position, shape, and color, and generate a complete three-dimensional scene representation. The specific process will not be elaborated here.

[0067] In a feasible implementation manner, the depth model is a monocular depth network, and the monocular depth network includes a feature extraction block, an upsampling block, and an output layer; the step S10: the step of inputting each frame of multi-view image into the depth model to obtain the predicted depth map of each frame of the multi-view image includes:

[0068] Step S101, traverse each frame of multi-view image, and input the multi-view image into the feature extraction block to obtain first feature information;

[0069] Process the multi-view images frame by frame in order. The specific order can be the time order, that is, traverse according to the order of image acquisition, or it can be adjusted according to specific algorithm requirements, such as giving priority to processing key frames or frames with rich features. The specific method is not limited here. Input each frame of multi-view image into the feature extraction block of the depth model to obtain first feature information. The structure of the feature extraction block can include multiple convolutional layers. For any convolutional layer, perform a convolutional operation on the image through the convolutional layer to obtain a convolutional result, input the convolutional result into an activation function (such as the ReLU function) to process and enhance the non-linear expression ability of the network to obtain an output result, and the output result of the last convolutional layer is the first feature information.

[0070] Step S102, input the feature information into the upsampling block to obtain second feature information;

[0071] Input the first feature information into the upsampling block. The upsampling block gradually restores the spatial resolution of the feature map to a resolution close to that of the input image through deconvolution operations or interpolation methods (such as bilinear interpolation, nearest neighbor interpolation). Skip connections splice the feature maps at the corresponding levels in the first feature information output by the feature extraction block with the upsampled feature map, and then perform fusion and optimization through a convolutional layer to obtain second feature information.

[0072] Step S103, input the second feature information into the output layer to obtain the predicted depth map of the multi-view image.

[0073] The second feature information is input into the output layer, and the convolutional kernel of the output layer performs a convolutional operation on the second feature information, finally obtaining a single-channel image, i.e., the predicted depth map. Each pixel value in the predicted depth map represents the depth information of the corresponding position in the original multi-view image. The representation method of the depth value can be set according to specific application requirements. For example, it can be a relative depth value normalized to the range [0, 1], or it can be a depth value converted to an actual physical distance through a specific conversion relationship, which is not limited here.

[0074] In this embodiment, each frame of the multi-view image is input into the depth model to obtain the predicted depth map of each frame of the multi-view image. For any target frame image in each frame of the multi-view image, based on the predicted depth map of the target frame image and the depth information of the previous frame image, the local 3D Gaussian model and the camera pose information of the target frame image are optimized to obtain the updated local 3D Gaussian model and the updated camera pose information of the target frame image. Compared with the CF-3DGS technology that lacks geometric constraints of inter-frame information, by using the predicted depth map of the target frame image and the depth information of the previous frame image to optimize the local 3D Gaussian model and the camera pose information of the target frame image, geometric constraints can be provided for the update of the local 3D Gaussian model based on the depth information coherence between adjacent frames, making the updated local 3D Gaussian model more conform to the actual scene structure. At the same time, the change of the depth information can also reflect the movement or rotation of the camera in the 3D space, thereby improving the accuracy of the camera pose information and providing a more accurate viewing angle for the three-dimensional Gaussian reconstruction.

[0075] Based on the updated local 3D Gaussian model and the updated camera pose information of each frame of the multi-view image, the global 3D Gaussian model is optimized to obtain the updated global 3D Gaussian model, where three-dimensional scene reconstruction is performed based on the updated global 3D Gaussian model. The updated local 3D Gaussian model of each frame of image carries the local scene structure information optimized by the depth information, and the updated camera pose information reflects the accurate information of each view. By integrating these information to optimize the global 3D Gaussian model, the accuracy of the global 3D Gaussian model can be improved.

[0076] Compared with CF-3DGS, this embodiment provides geometric constraints for model update by comprehensively considering the depth information between frames, and optimizes the global 3D Gaussian model from local to global, which can avoid the unreasonable distribution problem caused by the lack of constraints during the growth process of the global Gaussian point cloud, reduce the inconsistency and unreasonable overlap between local regions, thereby avoiding artifacts and floating objects in view synthesis, and improving the accuracy and integrity of the view synthesis result based on three-dimensional scene reconstruction.

[0077] Based on the first embodiment of this application, in the second embodiment of this application, for the content that is the same as or similar to the above-mentioned first embodiment, reference may be made to the above introduction and will not be elaborated hereinafter. On this basis, please refer to Figure 2 , step S20: For any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, the step of optimizing the local 3D Gaussian model and the camera pose information of the target frame image to obtain the updated local 3D Gaussian model and the updated camera pose information of the target frame image includes:

[0078] Step S201, for any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimize the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and the updated depth model;

[0079] Align the predicted depth map of the target frame with the depth information of the previous frame. According to the aligned depth information, adjust the parameters (mean, covariance, weight) of the local 3D Gaussian model to make it better fit the scene geometry to obtain the updated local 3D Gaussian model, and adjust the camera pose of the target frame by minimizing the depth error or photometric error to obtain the updated camera pose information, and update the weight of the depth model based on the depth error between the current frame and the previous frame to obtain the updated depth model.

[0080] Step S202, input the target frame image into the updated depth model to obtain the updated depth map of the target frame image;

[0081] Input the target frame image into the updated depth model. The updated depth model generates the updated depth map through feature extraction, upsampling, and the output layer, that is, the updated depth map. It can be understood that by improving the depth information of the multi-view images based on the updated depth model, more accurate data support is provided for the subsequent optimization process, ensuring that the subsequent optimization of the local 3D Gaussian model and the camera pose information can be carried out based on more accurate depth information, thereby improving the accuracy of 3D reconstruction.

[0082] Step S203, determine the next frame image of the target frame image as the target frame image, determine the depth information of the previous frame image based on the updated depth map of the target frame image, execute the step of optimizing the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model and subsequent steps until all frames of the multi-view images are traversed, to obtain the updated local 3D Gaussian model and the updated camera pose information of each frame of the multi-view images.

[0083] In this embodiment, the next frame image of the target frame image is determined as the new target frame image to achieve frame-by-frame traversal of the multi-view image sequence, and based on the updated depth map of the current target frame image, it is used as part of the previous frame depth information for the next optimization, ensuring that the depth information between adjacent frames can be coherently transmitted and based on, guaranteeing the continuity of the optimization process; for the new target frame image, the steps of optimizing the local 3D Gaussian model, camera pose information, and depth model are repeatedly executed until the local 3D Gaussian models and camera pose information of each frame of the multi-view image are updated, and the updated local 3D Gaussian models and updated camera pose information of each frame of the multi-view image are obtained. During this process, the depth model is continuously adjusted to better adapt to the characteristics of the entire multi-view image sequence.

[0084] It can be understood that by iteratively updating and optimizing each frame in the multi-view image sequence, the information of the entire sequence is guaranteed to be based on, avoiding the limitations of local optimization, and based on the updated depth map as the depth information transmission between adjacent frames, ensuring the coherence of information and the stability of the update process, making the entire optimization process more in line with the dynamic changes of the scene and the continuity of the three-dimensional structure, and improving the quality and reliability of three-dimensional scene reconstruction.

[0085] In a feasible implementation manner, the step S201: optimizing the local 3D Gaussian model, camera pose information, and depth model of the target frame image based on the predicted depth map of the target frame image and the depth information of the previous frame image to obtain the updated local 3D Gaussian model, updated camera pose information, and updated depth model of the target frame image includes:

[0086] Step S2011, initializing the local 3D Gaussian model of the target frame image based on the depth information of the previous frame image to obtain the initialized local 3D Gaussian model;

[0087] Align the depth information of the previous frame to the coordinate system of the target frame through camera pose transformation, and then according to the aligned depth information, select key points or regions in the scene of the target frame, and initialize a Gaussian distribution for each key point to obtain the initialized local 3D Gaussian model. Specifically, the mean of the Gaussian distribution is set to the depth position of the key point, the covariance of the Gaussian distribution is set to an initial value (for example, set according to the depth error range), and the weight of the Gaussian distribution is set to an initial value (for example, uniform distribution).

[0088] It can be understood that by initializing the local 3D Gaussian model of the target frame image based on the depth information of the previous frame image, the depth information between different frames is associated, providing a basis for the initialization of the local 3D Gaussian model of the target frame image based on sequence coherence, enabling the initialization of the local 3D Gaussian model to have certain prior information and improving the stability of the local 3D Gaussian model.

[0089] Step S2012: Based on the predicted depth map of the target frame image and the initialized local 3D Gaussian model, update the initialized local 3D Gaussian model, the camera pose information of the target frame image, and the depth model to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and the updated depth model.

[0090] Calculate the depth error between the predicted depth map and the depth information of the previous frame, and perform joint optimization on the local 3D Gaussian model, camera pose information, and depth model based on the depth error. For example, the parameters (mean, covariance, weight) of the local 3D Gaussian model can be adjusted by minimizing the depth error through a preset method, the camera pose of the target frame can be adjusted by minimizing the depth error or photometric error, and the weight of the depth model can be updated through the depth error. The specific optimization algorithm can be an algorithm such as gradient descent, which is not limited here.

[0091] In a feasible implementation, the depth information includes a predicted depth map and affine transformation parameters; the step S2011: the step of initializing the local 3D Gaussian model of the target frame image based on the depth information of the previous frame image includes:

[0092] Step S20111: Perform a depth alignment operation on the predicted depth map of the previous frame image and the affine transformation parameters of the previous frame image to obtain the aligned depth information of the previous frame image;

[0093] For each pixel point (u, v) in the previous depth map, calculate its corresponding position (u', v') in the target frame coordinate system, and map the depth value D t-1 (u, v) to the new position (u', v') to obtain the aligned depth information of the previous frame image. In the specific processing process, if the transformed coordinates may not be integers, the depth value of each pixel point in the target frame coordinate system can be calculated by interpolation (such as bilinear interpolation).

[0094] It can be understood that through the depth alignment operation, the predicted depth map of the previous frame image is combined with its affine transformation parameters to eliminate the depth information deviation caused by factors such as camera pose changes, ensuring that the depth information of the previous frame image is in the same reference system, so as to obtain more accurate and consistent depth information, and enhancing the consistency and coherence of the three-dimensional scene reconstruction system in processing depth information from different perspectives.

[0095] Step S20112: Initialize the local 3D Gaussian distribution of the target frame image through the aligned depth information of the previous frame image to obtain an initialized local 3D Gaussian model.

[0096] Select key points or regions in the target frame image (determined, for example, through a feature detection algorithm or uniform sampling). For each key point, initialize a Gaussian distribution according to the aligned depth information. Specifically, the mean of the Gaussian distribution is set to the depth position of the key point, the covariance of the Gaussian distribution can be set with an initial value according to the depth error range or the geometric characteristics of the scene (for example, a smaller covariance indicates a higher confidence level), and the weight of the Gaussian distribution can be set to a uniform distribution or assigned according to the importance of the key point. Combine all the initialized Gaussian distributions into a local 3D Gaussian model to obtain an initialized local 3D Gaussian model.

[0097] In a feasible implementation manner, the step S2012: Based on the predicted depth map of the target frame image and the initialized local 3D Gaussian model, update the initialized local 3D Gaussian model, camera pose information, and the depth model to obtain the steps of an updated local 3D Gaussian model, updated camera pose information, and updated depth model, include:

[0098] Step S20121: Render the predicted depth map of the target frame image based on the initialized local 3D Gaussian model and the camera pose of the target frame image to obtain a target frame rendered depth map, and render the target frame image based on the initialized local 3D Gaussian model to obtain a target frame rendered image;

[0099] According to the initialized local 3D Gaussian model and the camera pose of the target frame image, project the 3D Gaussian model into the perspective of the target frame image, calculate the depth information corresponding to each pixel position to generate a target frame rendered depth map, and project the 3D Gaussian model onto the target frame image plane according to the color and intensity information of the initialized local 3D Gaussian model to generate a target frame rendered image.

[0100] Step S20122: Determine the depth loss corresponding to the target frame image based on the target frame rendered depth map and the predicted depth map of the target frame image, and determine the color value loss corresponding to the target frame image based on the target frame image and the target frame rendered image;

[0101] Compare the rendered depth map of the target frame with the predicted depth map of the target frame image, and calculate the difference between the two as the depth loss. Specifically, loss functions suitable for depth information such as mean square error (MSE), mean absolute error (MAE), or others can be used, and there is no limitation here. Compare the target frame image with the rendered target frame, and calculate the color difference between them as the color value loss. Specifically, distance metrics in the color space, such as Euclidean distance or other perceptual color difference metrics, can be used.

[0102] It can be understood that by considering the losses in both the depth and color dimensions, the performance of the current model and pose can be comprehensively evaluated, avoiding the one-sidedness that may be brought by single-index evaluation, and contributing to more accurate optimization and better reconstruction results.

[0103] Step S20123, based on the depth loss corresponding to the target frame image and the color value loss corresponding to the target frame image, update the initialized local 3D Gaussian model through backpropagation to obtain an updated local 3D Gaussian model, and update the camera pose information of the target frame image to obtain the updated camera pose information of the target frame image, and update the parameters of the depth model to obtain an updated depth model.

[0104] Based on the depth loss and the color value loss, a total loss function is jointly constructed. The total loss function reflects the fitting degree of the current local 3D Gaussian model, camera pose, and depth model with the target frame image. The gradient of the total loss function is backpropagated to the parameters of the initialized local 3D Gaussian model, camera pose information, and depth model. Specifically, for the local 3D Gaussian model, parameters such as its center position, covariance matrix, color, and intensity are updated according to the gradient information to obtain an updated local 3D Gaussian model; for the camera pose information, parameters representing its position and orientation are updated according to the gradient information to obtain the updated camera pose information of the target frame image; for the depth model, parameters such as weights and biases in its network are updated to obtain the updated camera pose information of the target frame image.

[0105] In a feasible implementation manner, step S30: the step of optimizing the global 3D Gaussian model based on the updated local 3D Gaussian models and updated camera pose information of each frame of the multi-view images includes:

[0106] Step S301, based on the aligned depth information of the first frame image and the last frame image in each frame of the multi-view images, the affine transformation parameters of the first frame image and the last frame image, the updated depth maps of each frame of the multi-view images, the updated local 3D Gaussian models of each frame of the multi-view images, and the updated camera pose information of each frame of the multi-view images, determine the initialization point of the global 3D Gaussian operation;

[0107] The aligned depth information of the first frame image and the last frame image is the depth information after the depth alignment operation, which is used to characterize the depth information in the starting and ending states of the sequence and includes the depth structure of the scene at different time points. The aligned depth information can be a two-dimensional matrix, where each element represents the depth value at the corresponding pixel position. The affine transformation parameters describe the transformation of the first frame and the last frame in the 3D space, including translation, rotation, scaling, etc., and can be represented in matrix form, reflecting the position and pose changes of the camera at different times. The updated depth maps of each frame provide the depth information of each frame in the updated state, reflecting the depth structure of each frame after local optimization. The updated local 3D Gaussian models of each frame contain the 3D Gaussian distribution information of the local regions of each frame, such as the central position, shape (covariance matrix), color, and intensity, etc., and describe the 3D structure of the scene from a local perspective. The updated camera pose information of each frame reflects the position and orientation of the camera of each frame in the 3D space.

[0108] The initialization point of the global 3D Gaussian is the initial parameter information for starting the global 3D Gaussian operation, which is used to describe the basic information of the global 3D Gaussian and provides a starting point for subsequent optimization and adjustment processes, including but not limited to: position information, shape, color, and intensity information, etc. Specifically, the process of determining the initialization point of the global 3D Gaussian operation can be: based on the aligned depth information of the first frame and the last frame and the updated local 3D Gaussian models of each frame, determine the initial spatial position and shape of the global 3D Gaussian operation through spatial fusion or weighted average. For example, the central positions of the updated local 3D Gaussian models of each frame can be weighted averaged to determine the initial central position of the global 3D Gaussian, and the specific weights can be determined according to the confidence level of the depth or other factors, which are not limited here. For example, based on the covariance matrices of the updated local 3D Gaussian models of each frame, determine the shape of the initial global 3D Gaussian; extract the color and intensity information from the updated local 3D Gaussian models of each frame, and determine the initial color and intensity of the global 3D Gaussian according to preset rules. For example, the color and intensity of the updated local 3D Gaussian models of each frame can be averaged or weighted averaged to obtain the color and intensity information.

[0109] Step S302: Optimize the parameters of the global 3D Gaussian model based on the initialization point, the updated local 3D Gaussian models of the multi-view images of each frame, the updated camera pose information of the multi-view images of each frame, and the depth loss and color value loss corresponding to the multi-view images of each frame, to obtain an updated global 3D Gaussian model.

[0110] Taking the initialization point as the starting point of the global 3D Gaussian model, integrating the updated local 3D Gaussian model and camera pose information of each frame into the global 3D Gaussian model, using the depth loss and color value loss of each frame as constraints, optimizing the parameters of the global 3D Gaussian model by minimizing the global error (such as depth error, color error), and adopting a random growth strategy or a global optimization algorithm (such as gradient descent, Levenberg-Marquardt algorithm) for optimization.

[0111] In this embodiment, by synthesizing the depth information between each frame, geometric constraints are provided for model update, and the global 3D Gaussian model is optimized from local to global, which can avoid the unreasonable distribution problem caused by the lack of constraints during the growth process of the global Gaussian point cloud, reduce the inconsistency and unreasonable overlap between local regions, thereby avoiding artifacts and floating objects in view synthesis, and thus improving the accuracy and integrity of the view synthesis result based on three-dimensional scene reconstruction.

[0112] Exemplarily, to facilitate understanding of the implementation process of the three-dimensional scene reconstruction method obtained based on the above embodiments, please refer to Figure 3 , Figure 3 A flowchart of model training in a three-dimensional scene reconstruction method is provided. Specifically:

[0113] Given a set of images and a single depth map generated by an off-the-shelf monocular depth network (such as DPT), a learnable affine transformation is constructed to estimate the relative pose, learnable scale, and translation parameters to linearly adjust the single depth map. For the frame I at time step t t , the local 3D Gaussian model G is initialized using the depth information contained in the single depth map D t and therein. Then, this set of Gaussians with all attributes is trained by minimizing the photometric loss between the rendered image and the current image. Then, the trained Gt is used to learn the relative pose T of the current frame t . tand depth parameters. Depth is not only beneficial for depth alignment but also contains camera pose information. Due to potential noise in the relative pose, a step-by-step growing global 3DGS method can be adopted. By using the estimated relative pose and the newly observed frame as inputs, the 3D Gaussian function is gradually updated in iterations. However, since the densification process is consistent with the arrival speed of new frames, this means that the previous frames no longer provide supervision for the common area between adjacent frames. The corresponding Gaussian basis elements are affected by subsequent frames, so the reconstruction quality of the previously observed area may be affected. Experiments have demonstrated this process. Fix the camera pose and depth at the first frame (t = 1). Optimize the relative pose and distortion parameters of subsequent frames through local 3DGS using the above process. The optimized consistent depth and camera pose information are then used for the optimization of the global Gaussian. The aligned depths of the first and last frames are used to initialize the points.

[0114] Specifically, as Figure 3 shown, the model training in the three-dimensional scene reconstruction method is divided into two stages: local 3DGS (Local3DGaussian Splatting) and global 3DGS (Global3DGaussian Splatting). The specific process is as follows:

[0115] Stage 1: Local 3DGS (Local3DGaussian Splatting) stage, including:

[0116] 1. Input image and depth model: Input a series of images (such as I t+1 , I t , I t-1 , etc.), which are the original data to be processed. Based on the depth model (Depth Model), these input images are processed, and the depth model will generate a predicted depth map (Predicted Depth, such as D t ), which is a preliminary estimate of the depth of the objects in the image.

[0117] 2. Depth alignment and initialization: For the current frame t, perform a depth alignment operation on the depth information (D t-1 ) of the previous frame and the affine transformation parameters ({a t-1 , b t-1}) to obtain the aligned depth information (D t-1 ⊕{a t-1 , b t-1}). Initialize the local 3D Gaussian (G t ) based on the aligned depth information. This is the starting step of local 3D Gaussian reconstruction, providing an initial 3D Gaussian representation for subsequent optimization and rendering.

[0118] 3. Adaptive Depth Alignment (ADA) and Loss Calculation: Through the ADA strategy, compare the predicted depth map (D t ) with the rendered depth map to calculate the depth loss. The depth loss is used to measure the difference between the predicted depth and the rendered depth, and is an important basis for optimizing depth information. At the same time, compare the rendered image with the original image (Ground Truth, I t ) to calculate the RGB loss. The RGB loss is used to evaluate the difference between the rendered image and the original image in terms of color and visual effects, ensuring the accuracy and realism of the rendering result.

[0119] 4. Backpropagation and Optimization: According to the calculated depth loss and RGB loss, perform the backpropagation operation. Backpropagation will update the parameters of the local 3D Gaussian model (G t ) and the camera pose (T t ) and other relevant information according to the gradient of the loss function. By continuously iterating this process, the local 3D Gaussian is gradually optimized, and the depth information gradually reaches the consistent depth, that is, the depth information between different frames is more coordinated and accurate, providing a better basis for the subsequent global 3DGS stage.

[0120] II. Global 3DGS (Global 3D Gaussian Splatting) Stage, including:

[0121] 1. Initialization Point and Depth Information Integration: Based on the aligned depth (Consistent Depth) of the first frame and the last frame and the corresponding affine transformation parameters {a1, b1}, {a N , b N} to initialize the points (Initialize Points). These initialized points are the starting reference points of the global 3D Gaussian model, and they are closely related to the construction of the global Gaussian (G global ). Integrate the depth information (D x ⊕{a N , b N}) of all frames to provide comprehensive depth information support for the growth and optimization of the global Gaussian.

[0122] 2. Random Growth and Global Optimization: Adopt the Random Growth strategy to let all frames randomly participate in the training process. Different from the traditional incremental growth method, this random growth method can avoid the damage and error accumulation that the incremental method may cause to adjacent frames. During the random growth process, continue to be based on the ADA strategy, combine the depth loss and the RGB loss, and optimize the global Gaussian (G global ) and camera poses ({T1,…,T N}) and other parameters. Finally, through continuous optimization, obtain the optimized global Gaussian representation (Gglobal) and related parameters (c, r, s, α), which may include color (c), radius (7), scale (s), rotation angle (α), etc., to accurately describe the attributes and distributions of the global 3D Gaussian model, thereby achieving high-quality 3D reconstruction and novel view synthesis, improving the quality of view synthesis, and reducing problems such as artifacts.

[0123] The entire processing flow closely combines the concepts and steps represented by each noun appearing in the figure through operations such as local and global 3D Gaussian reconstruction, depth alignment, loss calculation, backpropagation, and random growth, realizing the complete process from the input image to high-quality 3D reconstruction and view synthesis.

[0124] Exemplarily, for the input multi-view image sequence, which includes at least the first frame to the Nth frame of images, where N is a positive integer. First, perform local 3D Gaussian training, and this local 3D Gaussian training includes: for each frame of image I t (where t = 1, 2,..., N), input each frame of image into the depth model, and the depth model performs depth prediction on this frame of image according to the texture, edge, color, and object structure features of the image to obtain the predicted depth map D t ; for the first frame, perform a depth alignment operation using the preset initial depth information and initial affine transformation parameters; for other frames, perform a depth alignment operation using the updated predicted depth map of the previous frame and the corresponding affine transformation parameters to obtain the aligned depth information, and initialize the local 3D Gaussian of this frame with the aligned depth information; generate a rendered depth map and a rendered image according to the local 3D Gaussian and the estimated camera pose; calculate the depth loss by comparing the predicted depth map with the rendered depth map, calculate the RGB loss by comparing the rendered image with the original image, based on the depth loss and the RGB loss, use the backpropagation algorithm to calculate the gradient, and update the parameters of the local 3D Gaussian and the camera pose according to this gradient, and at the same time update the parameters of the depth model to obtain the updated local 3D Gaussian and the optimized camera pose.

[0125] Then, perform global 3D Gaussian training, including: using the aligned depth information of the first frame and the last frame and their corresponding affine transformation parameters, combining the depth information of all frames and the optimized local 3D Gaussian and its camera pose to determine the initialization point of the global 3D Gaussian; adopting a random growth strategy, randomly selecting the information of frames, including the local 3D Gaussian and its related information, combining the depth loss and the RGB loss, and updating the parameters of the global 3D Gaussian, including position, shape, color, scale, rotation angle, etc., through backpropagation to obtain the optimized global 3D Gaussian and related parameters.

[0126] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the three-dimensional scene reconstruction method of the present application. Based on this technical concept, more forms of simple transformations are within the protection scope of the present application.

[0127] The present application also provides a three-dimensional scene reconstruction device. Please refer to Figure 4 , the three-dimensional scene reconstruction device includes:

[0128] A depth prediction module 10, configured to input multi-view images of each frame into a depth model to obtain a predicted depth map of each multi-view image of each frame;

[0129] A local 3D Gaussian optimization module 20, configured to, for any target frame image in the multi-view images of each frame, optimize the local 3D Gaussian model and camera pose information of the target frame image based on the predicted depth map of the target frame image and the depth information of the previous frame image, to obtain an updated local 3D Gaussian model and updated camera pose information of the target frame image;

[0130] A global 3D Gaussian optimization module 30, configured to optimize the global 3D Gaussian model based on the updated local 3D Gaussian models and updated camera pose information of each multi-view image of each frame to obtain an updated global 3D Gaussian model, wherein three-dimensional scene reconstruction is performed based on the updated global 3D Gaussian model.

[0131] Optionally, the local 3D Gaussian optimization module 20 is configured to:

[0132] For any target frame image in the multi-view images of each frame, optimize the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model based on the predicted depth map of the target frame image and the depth information of the previous frame image, to obtain an updated local 3D Gaussian model, an updated camera pose information, and an updated depth model of the target frame image;

[0133] Input the target frame image into the updated depth model to obtain an updated depth map of the target frame image;

[0134] Determine the next frame image of the target frame image as the target frame image, determine the depth information of the previous frame image based on the updated depth map of the target frame image, and perform the steps of optimizing the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model and subsequent steps until all frames of the multi-view images are traversed, to obtain the updated local 3D Gaussian model and updated camera pose information of each frame of the multi-view images respectively.

[0135] Optionally, the local 3D Gaussian optimization module 20 is used for:

[0136] Initialize the local 3D Gaussian model of the target frame image based on the depth information of the previous frame image to obtain an initialized local 3D Gaussian model;

[0137] Based on the predicted depth map of the target frame image and the initialized local 3D Gaussian model, update the initialized local 3D Gaussian model, the camera pose information of the target frame image, and the depth model, to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and an updated depth model.

[0138] Optionally, the depth information includes a predicted depth map and affine transformation parameters; the local 3D Gaussian optimization module 20 is used for:

[0139] Perform a depth alignment operation on the predicted depth map of the previous frame image and the affine transformation parameters of the previous frame image to obtain the aligned depth information of the previous frame image;

[0140] Initialize the local 3D Gaussian distribution of the target frame image through the aligned depth information of the previous frame image to obtain an initialized local 3D Gaussian model.

[0141] Optionally, the local 3D Gaussian optimization module 20 is used for:

[0142] Render the predicted depth map of the target frame image based on the initialized local 3D Gaussian model and the camera pose of the target frame image to obtain a target frame rendered depth map, and render the target frame image based on the initialized local 3D Gaussian model to obtain a target frame rendered image;

[0143] Determine the depth loss corresponding to the target frame image based on the target frame rendered depth map and the predicted depth map of the target frame image, and determine the color value loss corresponding to the target frame image based on the target frame image and the target frame rendered image;

[0144] Based on the depth loss corresponding to the target frame image and the color value loss corresponding to the target frame image, update the initialized local 3D Gaussian model through backpropagation to obtain an updated local 3D Gaussian model, update the camera pose information of the target frame image to obtain the updated camera pose information of the target frame image, and update the parameters of the depth model to obtain an updated depth model.

[0145] Optionally, the global 3D Gaussian optimization module 30 is configured to:

[0146] Based on the aligned depth information of the first frame image and the aligned depth information of the last frame image in each frame of the multi-view images, the affine transformation parameters of the first frame image and the affine transformation parameters of the last frame image, the updated depth maps of each frame of the multi-view images, the updated local 3D Gaussian models of each frame of the multi-view images, and the updated camera pose information of each frame of the multi-view images, determine the initialization point of the global 3D Gaussian operation;

[0147] Based on the initialization point, the updated local 3D Gaussian models of each frame of the multi-view images, the updated camera pose information of each frame of the multi-view images, and the depth loss and color value loss corresponding to each frame of the multi-view images, optimize the parameters of the global 3D Gaussian model to obtain an updated global 3D Gaussian model.

[0148] Optionally, the depth model is a monocular depth network, and the monocular depth network includes a feature extraction block, an upsampling block, and an output layer; the depth prediction module 10 is configured to:

[0149] Traverse each frame of the multi-view images, and input the multi-view images into the feature extraction block to obtain first feature information;

[0150] Input the feature information into the upsampling block to obtain second feature information;

[0151] Input the second feature information into the output layer to obtain the predicted depth map of the multi-view images.

[0152] The three-dimensional scene reconstruction device provided by the present application adopts the three-dimensional scene reconstruction method in the above embodiments, and can solve the technical problem that the accuracy of the view synthesis result based on the three-dimensional Gaussian reconstruction is affected. Compared with the prior art, the beneficial effects of the three-dimensional scene reconstruction device provided by the present application are the same as those of the three-dimensional scene reconstruction method provided by the above embodiments, and other technical features in the three-dimensional scene reconstruction device are the same as those disclosed in the method of the above embodiments, and will not be elaborated here.

[0153] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the three-dimensional scene reconstruction method in the above embodiments.

[0154] Reference is made below Figure 5 to, which shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functions and scope of use of the embodiments of the present application.

[0155] As Figure 5 shown, the electronic device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or include all the systems shown. More or fewer systems may be alternatively implemented or included.

[0156] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above functions defined in the method of the embodiments disclosed in the present application are executed.

[0157] The electronic device provided in the present application adopts the three-dimensional scene reconstruction method in the above embodiments, and can solve the technical problem that the accuracy of the view synthesis result based on three-dimensional Gaussian reconstruction is affected. Compared with the prior art, the beneficial effects of the electronic device provided in the present application are the same as those of the three-dimensional scene reconstruction method provided in the above embodiments, and other technical features in the electronic device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0158] It should be understood that each part disclosed in the present application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0159] As mentioned above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0160] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the three-dimensional scene reconstruction method in the above embodiments.

[0161] The computer-readable storage medium provided by the present application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted by any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0162] The above computer-readable storage medium can be included in an electronic device; it can also exist separately and not be assembled into the electronic device.

[0163] The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method of the above embodiment.

[0164] Computer program code for performing the operations of the present application can be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages - such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or can be connected to an external computer (for example, through the Internet based on an Internet service provider).

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0166] The modules involved in the embodiments described in the present application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0167] The readable storage medium provided in the present application is a computer-readable storage medium, and the computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned three-dimensional scene reconstruction method, which can solve the technical problem that the accuracy of the view synthesis result based on three-dimensional Gaussian reconstruction is affected. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as those of the three-dimensional scene reconstruction method provided in the above embodiments, and will not be elaborated here.

[0168] The present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the three-dimensional scene reconstruction method as described above.

[0169] The computer program product provided in the present application can solve the technical problem that the accuracy of the view synthesis result based on three-dimensional Gaussian reconstruction is affected. Compared with the prior art, the beneficial effects of the computer program product provided in the present application are the same as those of the three-dimensional scene reconstruction method provided in the above embodiments, and will not be elaborated here.

[0170] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation based on the technical concept of the present application and the content of the specification and drawings of the present application, or direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A three-dimensional scene reconstruction method, characterized in that, The three-dimensional scene reconstruction method includes: Inputting each frame of multi-view images into a depth model to obtain the predicted depth map of each frame of the multi-view images; For any target frame image in each frame of the multi-view images, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimizing the local 3D Gaussian model and camera pose information of the target frame image to obtain the updated local 3D Gaussian model and updated camera pose information of the target frame image; Based on the updated local 3D Gaussian models and updated camera pose information of each frame of the multi-view images, optimizing the global 3D Gaussian model to obtain an updated global 3D Gaussian model, wherein three-dimensional scene reconstruction is performed based on the updated global 3D Gaussian model.

2. The three-dimensional scene reconstruction method according to claim 1, wherein The step of, for any target frame image in each frame of the multi-view images, optimizing the local 3D Gaussian model and camera pose information of the target frame image based on the predicted depth map of the target frame image and the depth information of the previous frame image to obtain the updated local 3D Gaussian model and updated camera pose information of the target frame image includes: For any target frame image in each frame of the multi-view images, optimizing the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model based on the predicted depth map of the target frame image and the depth information of the previous frame image to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and an updated depth model; Inputting the target frame image into the updated depth model to obtain the updated depth map of the target frame image; Determining the next frame image of the target frame image as the target frame image, determining the depth information of the previous frame image based on the updated depth map of the target frame image, and performing the step of optimizing the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model and subsequent steps until all frames of the multi-view images are traversed to obtain the updated local 3D Gaussian models and updated camera pose information of each frame of the multi-view images.

3. The three-dimensional scene reconstruction method according to claim 2, wherein The step of, based on the predicted depth map of the target frame image and the depth information of the previous frame image, optimizing the local 3D Gaussian model of the target frame image, the camera pose information of the target frame image, and the depth model to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and an updated depth model includes: Initializing the local 3D Gaussian model of the target frame image based on the depth information of the previous frame image to obtain an initialized local 3D Gaussian model; Based on the predicted depth map of the target frame image and the initialized local 3D Gaussian model, updating the initialized local 3D Gaussian model, the camera pose information of the target frame image, and the depth model to obtain the updated local 3D Gaussian model of the target frame image, the updated camera pose information of the target frame image, and an updated depth model.

4. The three-dimensional scene reconstruction method according to claim 3, wherein The depth information includes a predicted depth map and affine transformation parameters; The step of initializing the local 3D Gaussian model of the target frame image based on the depth information of the previous frame image to obtain the initialized local 3D Gaussian model includes: Performing a depth alignment operation on the predicted depth map of the previous frame image and the affine transformation parameters of the previous frame image to obtain the aligned depth information of the previous frame image; Initializing the local 3D Gaussian distribution of the target frame image through the aligned depth information of the previous frame image to obtain the initialized local 3D Gaussian model.

5. The three-dimensional scene reconstruction method according to claim 4, characterized in that The step of updating the initialized local 3D Gaussian model, the camera pose information, and the depth model based on the predicted depth map of the target frame image and the initialized local 3D Gaussian model to obtain the updated local 3D Gaussian model, the updated camera pose information, and the updated depth model includes: Rendering the predicted depth map of the target frame image based on the initialized local 3D Gaussian model and the camera pose of the target frame image to obtain the rendered depth map of the target frame, and rendering the target frame image based on the initialized local 3D Gaussian model to obtain the rendered image of the target frame; Determining the depth loss corresponding to the target frame image based on the rendered depth map of the target frame and the predicted depth map of the target frame image, and determining the color value loss corresponding to the target frame image based on the target frame image and the rendered image of the target frame; Based on the depth loss corresponding to the target frame image and the color value loss corresponding to the target frame image, updating the initialized local 3D Gaussian model through backpropagation to obtain the updated local 3D Gaussian model, updating the camera pose information of the target frame image to obtain the updated camera pose information of the target frame image, and updating the parameters of the depth model to obtain the updated depth model.

6. The three-dimensional scene reconstruction method according to claim 5, characterized in that, The step of optimizing the global 3D Gaussian model based on the updated local 3D Gaussian models and the updated camera pose information of each frame of the multi-view images to obtain the updated global 3D Gaussian model includes: Determining the initialization point of the global 3D Gaussian operation based on the aligned depth information of the first frame image and the last frame image in each frame of the multi-view images, the affine transformation parameters of the first frame image and the last frame image, the updated depth maps of each frame of the multi-view images, the updated local 3D Gaussian models of each frame of the multi-view images, and the updated camera pose information of each frame of the multi-view images; Optimizing the parameters of the global 3D Gaussian model based on the initialization point, the updated local 3D Gaussian models of each frame of the multi-view images, the updated camera pose information of each frame of the multi-view images, and the depth loss and color value loss corresponding to each frame of the multi-view images to obtain the updated global 3D Gaussian model.

7. The three-dimensional scene reconstruction method according to any one of claims 1 to 6, characterized in that The depth model is a monocular depth network, and the monocular depth network includes a feature extraction block, an upsampling block, and an output layer; The step of inputting each frame of the multi-view images into the depth model to obtain the predicted depth maps of each frame of the multi-view images respectively includes: Traverse each frame of multi-view images, and input the multi-view images into the feature extraction block to obtain first feature information; Input the feature information into the upsampling block to obtain second feature information; Input the second feature information into the output layer to obtain the predicted depth map of the multi-view images.

8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 7 are implemented.

Images