A 3D reconstruction and rendering method based on multi-source optical sensors

Abstract

The present invention discloses a 3D reconstruction and rendering method based on a multi-source optical sensor, comprising the following steps: reconstructing multi-view dual-source images acquired by an infrared-visible light binocular camera to obtain a dual-source registration dataset, the dual-source registration dataset comprising an infrared dataset and a visible light dataset; rendering the infrared 3D Gaussian set and visible light camera parameters through a virtual perspective renderer to obtain an infrared image of a virtual perspective; simultaneously feeding the infrared image of the virtual perspective, the visible light dataset, and the infrared dataset into a 3D Gaussian feature fusion device to obtain an optimized fused 3D Gaussian set; and inputting the optimized fused 3D Gaussian set and visible light camera parameters into a generator to generate a fused video. This method solves the problem that in existing scenes with a lack of natural light and large light contrast, it is difficult to fully capture the detailed information of the scene using a single visible light sensor, thereby limiting the quality of 3D reconstruction and NVS.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction and new perspective generation, and in particular relates to a three-dimensional reconstruction and rendering method based on multi-source optical sensors. Background Art

[0002] 3D reconstruction and NVS are important research areas in computer vision and computer graphics, aiming to generate 3D models from 2D images or video data or synthesize images of the same scene from a new perspective. The development of this field can be seen as the development of 3D representation technology. In recent years, with the development of deep learning, in addition to explicit voxels, point clouds, and meshes, various implicit function representations have gradually emerged for scene representation. In November 2021, the introduction of Neural Radiance Field (NeRF) ushered in a new era of new perspective generation. It uses implicit neural networks to efficiently represent the color and lighting of the scene, and through supervised learning through a set of multi-view images with known camera poses, it achieves high-quality 3D scene reconstruction and new perspective synthesis. In November 2023, 3D Gaussian Splatting (3DGS) uses 3D Gaussian functions as the primitives for scene representation, greatly accelerating the speed of scene learning and achieving real-time and high-fidelity 3D rendering, while making explicit editing of the scene more possible.

[0003] However, in scenes with a lack of natural lighting and large lighting contrast, it is difficult to fully capture the detailed information of the scene through a single visible light sensor, which directly limits the quality of 3D reconstruction and NVS. Summary of the Invention

[0004] The purpose of the present invention is to provide a three-dimensional reconstruction and rendering method based on multi-source optical sensors to solve the problem that in existing scenes with insufficient natural lighting and large lighting contrast, it is difficult to fully capture the detailed information of the scene with a single visible light sensor, which limits the quality of three-dimensional reconstruction and NVS.

[0005] The present invention adopts the following technical solution: a three-dimensional reconstruction and rendering method based on a multi-source optical sensor, comprising the following contents:

[0006] Reconstructing multi-view dual-source images captured by an infrared-visible light binocular camera to obtain a dual-source registration dataset, wherein the dual-source registration dataset includes an infrared dataset and a visible light dataset; wherein the visible light dataset includes a visible light image, visible light camera parameters, and a visible light three-dimensional Gaussian set; and the infrared dataset includes an infrared image, infrared camera parameters, and an infrared three-dimensional Gaussian set;

[0007] Rendering the infrared three-dimensional Gaussian set and visible light camera parameters through a virtual perspective renderer to obtain an infrared image of a virtual perspective, and completely aligning the infrared image of the virtual perspective with the visible light image in the visible light dataset;

[0008] The virtual perspective infrared image, visible light dataset and infrared dataset are simultaneously fed into the 3D Gaussian feature fusion device to mine the geometric structure and texture information of the 3D scene and obtain the optimized fused 3D Gaussian set.

[0009] The optimized fused three-dimensional Gaussian set and visible light camera parameters are input into a generator to generate a fused video.

[0010] Furthermore, the fusion method of the three-dimensional Gaussian feature fuser is specifically as follows:

[0011] The visible light three-dimensional Gaussian set and the infrared three-dimensional Gaussian set are fused to obtain a fused three-dimensional Gaussian set;

[0012] Rendering the fused three-dimensional Gaussian set and visible light camera parameters by a virtual perspective renderer to obtain a fused image;

[0013] The fused image, the infrared image of the virtual perspective, and the visible light image are all input into a loss function calculation module. The loss function module calculates a function value. The function value obtains the gradient of each parameter in the fused three-dimensional Gaussian set through gradient backpropagation. Then, each parameter of the fused three-dimensional Gaussian set is updated and iterated according to the gradient direction. After multiple iterations, the optimized fused three-dimensional Gaussian set is obtained.

[0014] Furthermore, the virtual perspective renderer includes a projection module and a rasterization module.

[0015] Furthermore, the generator includes a projection module, a rasterization module and a frame merging module; the optimized fused three-dimensional Gaussian set and visible light camera parameters are input into the projection module, and after being rendered by the projection module and the rasterization module, they are input into the frame merging module to finally generate a fused video.

[0016] The beneficial effects of the present invention are:

[0017] 1. This paper discloses a method for 3D fusion and novel perspective rendering of 3DGS data generated from multi-source optical sensors. This is the first proposed 2D-supervised 3D fusion method for infrared and visible light images. We innovatively incorporate the loss function construction of 2D image fusion into our 3D Gaussian feature fuser, fixing the scene structure information and learning the scene texture information, ultimately achieving novel perspective images or smoothed videos given given camera parameters.

[0018] 2. This paper proposes a paradigm for transitioning from a dual-source registered 3D dataset to a fused view. First, a virtual view renderer is used to generate virtual view infrared images that are fully aligned with the visible light images in the dataset. These images serve as supervision to optimize the representation of the 3D fused scene in a 3D Gaussian feature fuser. Finally, a fused video generator is designed to visualize the fused video.

[0019] 3. This invention designs a 3D reconstruction and rendering method based on multi-source optical sensors, achieving 3D scene fusion and the generation of new perspectives. Compared with existing methods based on a single optical sensor, the 3D reconstruction in complex lighting scenarios, such as those with low light levels and high light contrast, is richer in detail, and the rendered new views are of higher quality. Compared with the fusion of visible light sensors with lidar and inertial navigation, the use of only visible and infrared binocular cameras offers lower costs and greater efficiency in terms of runtime and space usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an introduction diagram of the dataset of the present invention;

[0021] Figure 2 is a flow chart of the method of the present invention;

[0022] Figure 3 This is the overall architecture diagram of the algorithm of the present invention;

[0023] FIG4 (a) to FIG4 (c) are respectively a grassland visible light Gaussian set, a grassland infrared Gaussian set, and a grassland fused Gaussian set in an embodiment of the present invention;

[0024] FIG4 (d) to FIG4 (f) are respectively the visible light Gaussian set of the school bag, the infrared Gaussian set of the school bag and the fused Gaussian set of the school bag in an embodiment of the present invention;

[0025] FIG5(a) to FIG5(c) are respectively a visible light image of a school bag, an infrared image of grass, and a fused image of grass in an embodiment of the present invention;

[0026] FIG6( a ) to FIG6 ( c ) are respectively a visible light image of a grassland, an infrared image of a grassland, and a fused image of a grassland in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] This invention discloses a 3D reconstruction and rendering method based on multi-source optical sensors, belonging to the fields of 3D reconstruction and NVS technology. Compared with traditional visible light sensor-based 3DGS and its derivative generation methods, this method utilizes the 3DGS rendering pipeline to fuse complementary scene information from infrared and visible light images, achieving a 3D fusion reconstruction and rendering method under complex lighting conditions. Based on a dual-source registration dataset of visible light and infrared, this invention provides a virtual perspective image rendering method, as well as a method for 3D scene fusion reconstruction and new view fusion generation based on dual-source image and camera parameter supervision, achieving the generation of new fused views and videos for given camera poses.

[0029] The present invention provides a three-dimensional reconstruction and rendering method based on a multi-source optical sensor, comprising the following contents:

[0030] like Figure 1 As shown in the figure, the multi-view dual-source images collected by the infrared and visible light binocular camera are reconstructed by the COLMAP and 3DGS methods to obtain a dual-source registration dataset. The dual-source registration dataset includes an infrared dataset and a visible light dataset. Among them, the visible light dataset includes visible light images, visible light camera parameters and visible light three-dimensional Gaussian sets; the infrared dataset includes infrared images, infrared camera parameters and infrared three-dimensional Gaussian sets.

[0031] like Figure 2-Figure 3 As shown, the infrared three-dimensional Gaussian set and the visible light camera parameters are rendered through a virtual perspective renderer to obtain an infrared image of a virtual perspective, and the infrared image of the virtual perspective is completely aligned with the visible light image in the visible light dataset;

[0032] The virtual perspective infrared image, visible light dataset and infrared dataset are simultaneously fed into the 3D Gaussian feature fusion device to mine the geometric structure and texture information of the 3D scene and obtain the optimized fused 3D Gaussian set.

[0033] The optimized fused 3D Gaussian set and visible light camera parameters are input into the generator to generate a fused video.

[0034] In some embodiments, the fusion method of the three-dimensional Gaussian feature fuser is specifically as follows:

[0035] The visible light three-dimensional Gaussian set and the infrared three-dimensional Gaussian set are fused to obtain a fused three-dimensional Gaussian set;

[0036] The fused 3D Gaussian set and visible light camera parameters are rendered by a virtual perspective renderer to obtain a fused image;

[0037] The fused image, the infrared image of the virtual perspective, and the visible light image are all input into the loss function calculation module. The loss function module calculates a function value. The function value is back-propagated through the gradient to obtain the gradient of each parameter in the fused three-dimensional Gaussian set. Then, the various parameters of the fused three-dimensional Gaussian set are updated and iterated according to the gradient direction. After multiple iterations, when the function value calculated by the loss function drops to a very small value, the optimized fused three-dimensional Gaussian set is finally obtained.

[0038] In some embodiments, the virtual perspective renderer includes a projection module and a rasterization module.

[0039] In some embodiments, the generator includes a projection module, a rasterization module and a frame merging module; the optimized fused three-dimensional Gaussian set and visible light camera parameters are input into the projection module, and after rendering by the projection module and the rasterization module, they are input into the frame merging module to finally generate a fused video.

[0040] The overall architecture of the 3D reconstruction and rendering method based on multi-source optical sensors of the present invention consists of three parts: a virtual perspective renderer, a 3D Gaussian feature fuser, and a fused video generator.

[0041] 1. The virtual view renderer performs the pre-processing for 3D fusion reconstruction, obtaining fully aligned supervision images for subsequent loss function calculations. These supervision images are infrared images from a virtual view. To render a 3D Gaussian scene from an infrared source into an image from the visible camera's perspective, the 3DGS submodule diff-gaussian-rasterazation is used. This module loads visible camera parameters and an infrared 3D Gaussian set, rendering a virtual view infrared image that is fully aligned with the visible image in the dataset.

[0042] The renderer is divided into two parts: projection module and rasterization module.

[0043] The projection module projects the 3D Gaussian sphere in the 3D space into the 2D space for rendering. This process is mainly achieved by multiplying the covariance matrix representing the Gaussian sphere by the transformation matrix. The transformation formula from the 3D covariance matrix Σ to the 2D covariance matrix Σ is:

[0044] ,

[0045] Where W is the perspective transformation matrix, J k is the Jacobian matrix of the affine approximation of the projection transformation of the k-th Gaussian sphere. Through this transformation, a set of two-dimensional Gaussian circles on the projection plane is obtained.

[0046] The rasterization module first sorts the two-dimensional Gaussian circles by depth, and then divides the imaging plane into several 16×16 blocks. Next, for each block, a copy of the Gaussian circle with overlapping areas is made, and each copy is assigned an identifier in sequence. By rendering the ordered Gaussian, we can obtain the color value of each pixel, and the rendering process of each block is independent of each other. The single-pixel volume rendering formula in the imaging space is:

[0047] ,

[0048] where c i Is the color of each point, the color value is calculated by the SH coefficient of each RGB channel and the viewing direction. i It is obtained by computing a two-dimensional Gaussian distribution with covariance Σ multiplied by a learnable transparency parameter.

[0049] Through the above two modules, the function of virtual perspective renderer is realized.

[0050] 2. In the 3D Gaussian feature fuser, the key is learning a fused representation of a 3D scene. Using a neural network to learn a fusion strategy for a 3D Gaussian set requires a large amount of training data, and the model suffers from issues such as high training time and low generalization. Considering the physical properties, differentiability, and speed of the projection and rasterization modules, we can use them to generate a fused image. This is then combined with the visible light image and the virtual-view infrared image as a loss function. The gradient is then back-propagated to the Gaussian set of the fused scene to optimize the 3D Gaussian feature representation of the learned scene.

[0051] Specifically, we first use a Gaussian fusion module to merge the visible light Gaussian set and the infrared Gaussian set, both registered in the same world coordinate system, into a fused Gaussian set. This step aims to provide an initial value for the fused Gaussian set. Since the position and shape of the Gaussian spheres well reflect the spatial structure of the scene, while color and transparency vary significantly under different optical sources, we consider using all the position and shape information of the visible light Gaussian set and the infrared Gaussian set as the initial position and shape information for the fused Gaussian set. Furthermore, the SH coefficient representing color and the associated transparency are parameters to be learned and are set to the corresponding values ​​in the original dataset.

[0052] After obtaining the initial value of the fused Gaussian set, the fused image can be rendered through a projection module and a rasterization module, just like the renderer in the previous component. This is the forward pass process of the 3D Gaussian feature fuser. In the backward pass, the loss function is calculated using the real visible light image, the virtual perspective infrared image, and the fused rendered image. The formula is:

[0053] ,

[0054] The specific explanation is to take the 1 norm of the maximum intensity of the real visible light image and the virtual perspective infrared image and the intensity value of the fused rendering image, and then take the 1 norm of the maximum gradient and the gradient value of the fused rendering image, and then linearly combine them as the overall loss function.

[0055] The gradient is ultimately propagated back to the parameters of the three-dimensional Gaussian spheres in the scene. Specifically, we used the Adam optimizer for optimization. The optimization strategy fixed the position and shape of the Gaussian spheres, namely the center coordinates, scale coefficient, and rotation quaternion of each Gaussian sphere. We ensured that the gradients were only propagated back to the color-related SH coefficients and transparency, with initial learning rates of 0.0025 and 0.05, respectively. This allowed us to quickly obtain a fused Gaussian set using the spatial structure of the dual-source dataset as a strong prior. We can also relax the constraints on the spatial structure. For example, in subsequent verification experiments, we fixed the structural parameters for the first 3,000 iterations and then adjusted them for the next 4,000 iterations while optimizing the color-related parameters. The learning rates were 0.005 for the scale coefficient vector and 0.001 for the rotation quaternion vector.

[0056] 3. The final component is the Fused Gaussian Generator, whose purpose is to display the fusion results as a mosaic of fused image video frames. This component consists of a projection module, a rasterization module, and a frame merging module. It takes a controllable input camera parameter and the fused Gaussian set generated by the previous component as input to ultimately display the results. The projection module and rasterization module will not be described in detail here. The frame merging module merges the rendered images continuously acquired along the camera trajectory into a smooth virtual perspective video.

[0057] This invention uses infrared and visible light sensors to capture images separately. Leveraging loss functions from the field of 2D image fusion and the 3DGS scene representation method and rendering pipeline, this multi-source data is processed to fully exploit detailed information in scenes with low lighting conditions and high light contrast, achieving higher-quality 3D reconstruction and new perspective synthesis for fused rendering. Therefore, based on multi-source optical sensors, this invention mines scene geometry and texture information, combining the 3DGS architecture with image fusion training strategies to achieve high-quality 3D fused rendering under complex lighting conditions.

[0058] 3DGS is a technique for 3D reconstruction and NVS of scenes from multi-view images acquired by a single optical sensor. It has three key components. First, 3D Gaussian functions are introduced as scene representations. This method uses inputs similar to traditional NeRF methods: cameras calibrated using structured light measurement (SfM). The sparse point cloud generated during the SfM process is used to initialize a set of 3D Gaussian functions. Unlike most point-based solutions that require multi-view stereo (MVS) data, 3DGS achieves high-quality rendering using only SfM-generated points. 3D Gaussian functions are chosen as the scene representation because they are not only differentiable volumetric representations but also can be efficiently rasterized by projecting them into two dimensions and applying a standard blending process, which is consistent with the NeRF image formation model. Second, properties of the 3D Gaussian functions, including 3D position, opacity, anisotropic covariance, and spherical harmonic (SH) coefficients, are optimized using an adaptive density control step. 3D Gaussian functions are dynamically added and removed during the optimization process, resulting in a compact, unstructured, and accurate scene representation. Finally, 3DGS proposes a real-time rendering system that efficiently utilizes the GPU to perform anisotropic splashing in accordance with visibility order, and implements the backpropagation process more quickly and accurately through sorting and blending.

[0059] Existing 3D reconstruction and NVS technologies aim to accelerate running speed, improve rendering accuracy, reduce initial dependencies, and minimize memory usage based on architectures such as 3DGS. However, these efforts are based on visible light images and are limited by the physical properties of visible light sensors. Visible light images typically perform poorly in low light, changing lighting, or complex scenes, resulting in information loss during reconstruction and difficulty capturing the precise shape of objects. Furthermore, they are affected by motion blur and lighting changes during image acquisition, which can also lead to reduced reconstruction quality. There are also efforts that combine lidar and inertial devices to provide effective scene structure supervision, but these efforts are costly and are still limited by a single optical source, making it difficult to capture the richer optical details in the scene.

[0060] In this context, it is of great significance to integrate the complementary information of visible light sensors and infrared sensors to achieve high-quality 3D reconstruction and rendering of low-light and high-light-ratio scenes. Example

[0061] Since the existing datasets do not meet the requirements for fusion, we created a dataset of two scenes using images collected by an infrared and visible light binocular camera. The camera model is KYT-U130-60R3A. Since it uses near-wave infrared, an infrared fill light is used to highlight the objects in the scene. The dataset structure is introduced in Figure 1 , detailed data are shown in the table below:

[0062] Table 1 Basic information of the dataset used in the verification process

[0063]

[0064] The verification results are shown in Figures 4 to 6.

[0065] Figures 4(a)-4(e) compare the dual-source and fused 3D Gaussian sets of this embodiment. Figures 4(a)-4(c) show the full 3D Gaussian sets of the grass dataset under visible, infrared, and fused light sources, while Figures 4(d)-4(f) show the full 3D Gaussian sets of the schoolbag dataset under visible, infrared, and fused light sources. The scene is enveloped in darkness because the camera has a blind spot when capturing image data; the irregular edges are determined by the nature of the scene representation.

[0066] Figures 5(a) to 5(c) and 6(a) to 6(c) show image pairs from the schoolbag and grass datasets before fusion, and the fused, new-perspective synthetic images after fusion. This embodiment uses a near-wave infrared camera and infrared fill light to simulate the heat distribution in a scene observed by a thermal infrared camera. As a result, the fused scene retains rich textures while exhibiting a clearer spatial structure, with heated objects highlighted. This represents a breakthrough in generating new-perspective fused images from scratch, improving the quality of 3D fusion reconstruction and NVS based on dual-source optical sensors.

Claims

1. A three-dimensional reconstruction and rendering method based on multi-source optical sensors, characterized in that: Includes the following: Reconstructing multi-view dual-source images captured by an infrared-visible light binocular camera to obtain a dual-source registration dataset, wherein the dual-source registration dataset includes an infrared dataset and a visible light dataset; wherein the visible light dataset includes a visible light image, visible light camera parameters, and a visible light three-dimensional Gaussian set; and the infrared dataset includes an infrared image, infrared camera parameters, and an infrared three-dimensional Gaussian set; Rendering the infrared three-dimensional Gaussian set and visible light camera parameters through a virtual perspective renderer to obtain an infrared image of a virtual perspective, and completely aligning the infrared image of the virtual perspective with the visible light image in the visible light dataset; The virtual perspective infrared image, visible light dataset and infrared dataset are simultaneously fed into the 3D Gaussian feature fusion device to mine the geometric structure and texture information of the 3D scene and obtain the optimized fused 3D Gaussian set. Inputting the optimized fused three-dimensional Gaussian set and visible light camera parameters into a generator to generate a fused video; The fusion method of the three-dimensional Gaussian feature fuser is specifically as follows: The visible light three-dimensional Gaussian set and the infrared three-dimensional Gaussian set are fused to obtain a fused three-dimensional Gaussian set; The fused three-dimensional Gaussian set and visible light camera parameters are rendered by a projection module and a rasterization module to obtain a fused image; The fused image, the infrared image of the virtual perspective, and the visible light image are all input into a loss function calculation module. The loss function calculation module calculates a function value. The function value obtains the gradient of each parameter in the fused three-dimensional Gaussian set through gradient backpropagation. Then, each parameter of the fused three-dimensional Gaussian set is updated and iterated according to the gradient direction. After multiple iterations, the optimized fused three-dimensional Gaussian set is obtained.

2. The 3D reconstruction and rendering method based on multi-source optical sensors according to claim 1, wherein: The virtual perspective renderer includes a projection module and a rasterization module.

3. The three-dimensional reconstruction and rendering method based on a multi-source optical sensor according to claim 1 or 2, characterized in that: The generator includes a projection module, a rasterization module and a frame merging module; the optimized fused three-dimensional Gaussian set and visible light camera parameters are input into the projection module, rendered by the projection module and the rasterization module, and then input into the frame merging module to finally generate a fused video.

Images