Three-dimensional reconstruction and rendering method based on multi-source optical sensor

Abstract

The invention discloses a three-dimensional reconstruction and rendering method based on a multi-source optical sensor, and the method comprises the following steps: reconstructing a multi-view dual-source image collected by an infrared visible light binocular camera to obtain a dual-source registration data set which comprises an infrared data set and a visible light data set; rendering the infrared three-dimensional Gaussian set and the visible light camera parameters through a virtual view angle renderer to obtain an infrared image of a virtual view angle; sending the infrared image of the virtual view angle, the visible light data set and the infrared data set into a three-dimensional Gaussian feature fusion device at the same time to obtain an optimized fused three-dimensional Gaussian set; and inputting the optimized fused three-dimensional Gaussian set and visible light camera parameters into a generator to generate a fused video. The problems that in an existing scene with the lack of natural illumination and large illumination contrast, detail information of the scene is difficult to fully capture through a single visible light sensor, and the quality of three-dimensional reconstruction and NVS is limited are solved.

Description

[Technical field]

[0001] The present invention belongs to the technical field of three-dimensional reconstruction and new viewing angle generation, and in particular relates to a three-dimensional reconstruction and rendering method based on a multi-source optical sensor. [Background technology]

[0002] 3D reconstruction and NVS are important research areas in computer vision and computer graphics, aiming to generate 3D models from 2D images or video data or synthesize images of the same scene from a new perspective. The development of this field can be seen as the development of 3D representation technology. In recent years, with the development of deep learning, in addition to explicit voxels, point clouds, and grids, various implicit function representations have gradually emerged for scene representation. In November 2021, the introduction of Neural Radiance Field (NeRF) ushered in a new era of new perspective generation, using implicit neural networks to efficiently represent the color and lighting of the scene, and supervised learning through a set of multi-view images with known camera poses, achieving high-quality 3D scene reconstruction and new perspective synthesis. In November 2023, 3D Gaussian Splatting (3DGS) uses 3D Gaussian functions as primitives for scene representation, greatly accelerating the speed of scene learning, achieving real-time and high-fidelity 3D rendering, and making explicit editing of scenes more possible.

[0003] However, in scenes with a lack of natural lighting and large lighting contrast, it is difficult to fully capture the detailed information of the scene with a single visible light sensor, which directly limits the quality of 3D reconstruction and NVS. [Summary of the invention]

[0004] The purpose of the present invention is to provide a 3D reconstruction and rendering method based on multi-source optical sensors to solve the problem that in existing scenes with lack of natural lighting and large lighting contrast, it is difficult to fully capture the detailed information of the scene through a single visible light sensor, which limits the quality of 3D reconstruction and NVS.

[0005] The present invention adopts the following technical solution: a three-dimensional reconstruction and rendering method based on a multi-source optical sensor, comprising the following contents:

[0006] Reconstructing the multi-view dual-source images collected by the infrared-visible light binocular camera to obtain a dual-source registration data set, wherein the dual-source registration data set includes an infrared data set and a visible light data set; wherein the visible light data set includes a visible light image, visible light camera parameters, and a visible light three-dimensional Gaussian set; and the infrared data set includes an infrared image, infrared camera parameters, and an infrared three-dimensional Gaussian set;

[0007] The infrared three-dimensional Gaussian set and the visible light camera parameters are rendered by a virtual perspective renderer to obtain an infrared image of a virtual perspective, and the infrared image of the virtual perspective is completely aligned with the visible light image in the visible light data set;

[0008] The infrared image of the virtual perspective, the visible light data set and the infrared data set are simultaneously sent to the 3D Gaussian feature fusion device to mine the geometric structure and texture information of the 3D scene and obtain the optimized fused 3D Gaussian set.

[0009] The optimized fused three-dimensional Gaussian set and visible light camera parameters are input into a generator to generate a fused video.

[0010] Furthermore, the fusion method of the three-dimensional Gaussian feature fuser is specifically as follows:

[0011] The visible light three-dimensional Gaussian set and the infrared three-dimensional Gaussian set are fused to obtain a fused three-dimensional Gaussian set;

[0012] The fused three-dimensional Gaussian set and visible light camera parameters are rendered by a virtual perspective renderer to obtain a fused image;

[0013] The fused image, the infrared image of the virtual perspective and the visible light image are all input into a loss function calculation module. The loss function module calculates a function value. The function value obtains the gradient of each parameter in the fused three-dimensional Gaussian set through gradient back propagation. Then, each parameter of the fused three-dimensional Gaussian set is updated and iterated according to the gradient direction. After multiple iterations, an optimized fused three-dimensional Gaussian set is obtained.

[0014] Furthermore, the virtual perspective renderer includes a projection module and a rasterization module.

[0015] Furthermore, the generator includes a projection module, a rasterization module and a frame merging module; the optimized fused three-dimensional Gaussian set and visible light camera parameters are input into the projection module, and after being rendered by the projection module and the rasterization module, they are input into the frame merging module to finally generate a fused video.

[0016] The beneficial effects of the present invention are:

[0017] 1. The present invention discloses a 3D fusion and new perspective rendering method based on 3DGS data produced by multi-source optical sensors. This is the first time that a 3D fusion method under 2D supervision of infrared and visible light images has been proposed. We innovatively introduce the loss function construction method of 2D image fusion into our 3D Gaussian feature fuser, fix the scene structure information, learn the texture information of the scene, and finally realize the generation of new perspective images or smooth videos given camera parameters.

[0018] 2. This paper designs a paradigm from a dual-source registered 3D dataset to a fused view. This paper first uses a virtual view renderer to generate virtual view infrared images that are fully aligned with the visible light images in the dataset. Then, they are used as supervision to optimize the representation method of learning 3D fused scenes in a 3D Gaussian feature fuser. Finally, a fused video generator is designed to visualize the generated fused video as a display result.

[0019] 3. The present invention designs a 3D reconstruction and rendering method based on multi-source optical sensors, which realizes 3D scene fusion and new perspective view generation. Compared with the existing method based on a single optical sensor, the 3D reconstruction details in complex lighting scenes such as weak light and large light ratio are richer, and the quality of the rendered new view is higher. Compared with the work of integrating visible light sensors with lidar and inertial navigation, only visible light and infrared binocular cameras are needed, which has lower economic cost and higher efficiency in program running time and space occupation.

Brief Description of the Drawings

[0020] Figure 1 A diagram is provided to introduce the data set of the present invention;

[0021] Figure 2 is a flow chart of the method of the present invention;

[0022] Figure 3 It is the overall architecture diagram of the algorithm of the present invention;

[0023] Figure 4(a) to Figure 4(c) They are respectively a grassland visible light Gaussian set, a grassland infrared Gaussian set and a grassland fusion Gaussian set in the embodiment of the present invention;

[0024] Figure 4(d) to Figure 4(f) They are respectively a visible light Gaussian set of a school bag, an infrared Gaussian set of a school bag, and a fused Gaussian set of a school bag in an embodiment of the present invention;

[0025] Figure 5(a) to Figure 5(c) They are respectively a visible light image of a schoolbag, an infrared image of a grassland, and a fused image of a grassland in an embodiment of the present invention;

[0026] Figure 6(a) to Figure 6(c) They are respectively a grassland visible light image, a grassland infrared image and a grassland fused image in the embodiment of the present invention. [Specific implementation method]

[0027] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The present invention discloses a 3D reconstruction and rendering method based on multi-source optical sensors, which belongs to the field of 3D reconstruction and NVS technology. Compared with the traditional 3DGS based on visible light sensors and its derived generation method, we use the rendering pipeline of 3DGS to fuse the scene complementary information of infrared and visible light images, and realize the 3D fusion reconstruction and rendering method under complex lighting conditions. The present invention provides a virtual perspective image rendering method on a dual-source registration data set of visible light and infrared, as well as a method for 3D scene fusion reconstruction and new view fusion generation based on dual-source images and camera parameter supervision, which realizes the generation of new fusion views and videos for given camera poses.

[0029] The present invention provides a three-dimensional reconstruction and rendering method based on a multi-source optical sensor, comprising the following contents:

[0030] like Figure 1 As shown in the figure, the multi-view dual-source images collected by the infrared and visible light binocular cameras are reconstructed by the COLMAP and 3DGS methods to obtain a dual-source registration dataset, which includes an infrared dataset and a visible light dataset; wherein the visible light dataset includes visible light images, visible light camera parameters and visible light three-dimensional Gaussian sets; the infrared dataset includes infrared images, infrared camera parameters and infrared three-dimensional Gaussian sets.

[0031] like Figure 2-Figure 3 As shown, the infrared three-dimensional Gaussian set and the visible light camera parameters are rendered by a virtual view renderer to obtain an infrared image of a virtual view, and the infrared image of the virtual view is completely aligned with the visible light image in the visible light dataset;

[0032] The infrared image of the virtual perspective, the visible light data set and the infrared data set are simultaneously sent to the 3D Gaussian feature fusion device to mine the geometric structure and texture information of the 3D scene and obtain the optimized fused 3D Gaussian set.

[0033] The optimized fused 3D Gaussian set and visible light camera parameters are input into the generator to generate a fused video.

[0034] In some embodiments, the fusion method of the three-dimensional Gaussian feature fuser is specifically:

[0035] The visible light three-dimensional Gaussian set and the infrared three-dimensional Gaussian set are fused to obtain a fused three-dimensional Gaussian set;

[0036] The fused three-dimensional Gaussian set and visible light camera parameters are rendered by a virtual view renderer to obtain a fused image;

[0037] The fused image, the infrared image of the virtual perspective, and the visible light image are all input into the loss function calculation module. The loss function module calculates a function value. The function value is back-propagated through the gradient to obtain the gradient of each parameter in the fused three-dimensional Gaussian set. Then, each parameter of the fused three-dimensional Gaussian set is updated and iterated according to the gradient direction. After multiple iterations, when the function value calculated by the loss function drops to a very small value, the optimized fused three-dimensional Gaussian set is finally obtained.

[0038] In some embodiments, the virtual perspective renderer includes a projection module and a rasterization module.

[0039] In some embodiments, the generator includes a projection module, a rasterization module and a frame merging module; the optimized fused three-dimensional Gaussian set and visible light camera parameters are input into the projection module, and then input into the frame merging module after being rendered by the projection module and the rasterization module, and finally a fused video is generated.

[0040] The overall architecture of the three-dimensional reconstruction and rendering method based on multi-source optical sensors of the present invention consists of three parts: a virtual perspective renderer, a three-dimensional Gaussian feature fuser and a fused video generator.

[0041] 1. For the virtual view renderer, this component is the pre-work for 3D fusion reconstruction in order to obtain the fully aligned supervision image in the subsequent loss function calculation. The supervision image is the infrared image of the virtual view. In order to render the 3D Gaussian scene of the infrared source into the image of the visible light camera view, the 3DGS submodule diff-gaussian-rasterazation is used to load the visible light camera parameters and the infrared 3D Gaussian set, and render the virtual view infrared image that is fully aligned with the visible light image in the data set.

[0042] The renderer is divided into two parts: projection module and rasterization module.

[0043] The projection module projects the 3D Gaussian sphere in the 3D space into the 2D space for rendering. This process is mainly achieved by multiplying the covariance matrix representing the Gaussian sphere by the transformation matrix. The transformation formula from the 3D covariance matrix Σ to the 2D covariance matrix Σ is:

[0044]

[0045] Where W is the perspective transformation matrix, J k is the Jacobian matrix of the affine approximation of the projection transformation of the kth Gaussian sphere. Through this transformation, a set of two-dimensional Gaussian circles on the projection plane is obtained.

[0046] The rasterization module first sorts the two-dimensional Gaussian circles by depth, and then divides the imaging plane into several 16×16 blocks. Next, a copy of the Gaussian circle with overlapping areas is made for each block, and each copy is assigned an identifier in sequence. By rendering the ordered Gaussian, we can obtain the color value of each pixel, and the rendering process of each block is independent of each other. The single-pixel volume rendering formula of the imaging space is:

[0047]

[0048] where c i It is the color of each point, and the color value is calculated by the SH coefficient of each RGB channel and the direction of sight. i It is obtained by computing a 2D Gaussian distribution with covariance Σ multiplied by a learnable transparency parameter.

[0049] Through the above two modules, the function of virtual perspective renderer is realized.

[0050] 2. In the 3D Gaussian feature fuser, the focus is on how to learn the fusion representation of a 3D scene. If a neural network is used to learn the fusion strategy of a 3D Gaussian set, a large amount of training data is required, and the model has problems such as high training time cost and low generalization. Taking into account the physical properties, differentiability and speed of the projection module and rasterization module, we can use them to generate a fused image, and then use the visible light image and the virtual perspective infrared image as a loss function, and then back-propagate the gradient to the Gaussian set of the fused scene to optimize the 3D Gaussian feature representation of the learning scene. Since this method models the properties of the real physical world, it does not require the model to use a black box of a neural network to explore and learn. It is not "powerful bricks flying" but "ingenious".

[0051] Specifically, we first merge the visible light Gaussian set and the infrared Gaussian set registered in the same world coordinate system into a fused Gaussian set through a Gaussian fusion module. The purpose of this step is to provide an initial value for a fused Gaussian set. Since the position and shape of the Gaussian ball can well reflect the spatial structure information of the scene, and the color and transparency vary greatly under different optical sources, we consider using all the position and shape information of the visible light Gaussian set and the infrared Gaussian set as the initial position and shape information of the fused Gaussian set. In addition, the SH coefficient that characterizes the color and the related transparency are parameters to be learned, which can be set to the corresponding values ​​of the original data set.

[0052] After obtaining the initial value of the fused Gaussian set, the fused image can be rendered through a projection module and a rasterization module, just like the renderer of the previous component. This is the forward transfer process of the 3D Gaussian feature fuser. In the reverse transfer, the set loss function is calculated using the real visible light image, the virtual perspective infrared image, and the fused rendered image. The formula is:

[0053]

[0054] The specific explanation is to use the maximum intensity of the real visible light image and the virtual perspective infrared image and the intensity value of the fused rendering image to take the norm of 1, and then use the maximum gradient and the gradient value of the fused rendering image to take the norm of 1, and then linearly combine them as the overall loss function.

[0055] The gradient will eventually be back-propagated to the parameters of the three-dimensional Gaussian sphere in the scene. Specifically, we use the Adam optimizer for optimization. In the optimization strategy, the position and shape of the Gaussian sphere set, that is, the center coordinates, scale coefficient, and rotation quaternion of each Gaussian sphere are fixed; the gradient is only back-propagated to the color-related SH coefficient and transparency, and the initial learning rates are 0.0025 and 0.05, respectively. In this way, a fused Gaussian set with the spatial structure of the dual-source dataset as a strong prior can be quickly obtained. At the same time, we can also relax the constraint strength of the spatial structure. For example, in the subsequent verification experiment, we used the first 3000 iterations to fix the structural parameters. When optimizing the color-related parameters in the last 4000 iterations, the structural parameters were adjusted at the same time. The learning rates were 0.005 for the scale coefficient vector and 0.001 for the rotation quaternion vector.

[0056] 3. The last component is the fused Gaussian generator, which aims to display the fusion results in the form of fused image video frame splicing. This component consists of a projection module, a rasterization module and a frame merging module. It takes a controllable input camera parameter and a fused Gaussian set generated by the previous component as input, and finally obtains the display results. The projection module and the rasterization module are not repeated here, and the frame merging module merges the rendered images collected continuously on the camera trajectory into a smooth virtual perspective video.

[0057] The present invention uses infrared and visible light sensors to collect images respectively, and uses the loss function in the field of two-dimensional image fusion and the scene representation method and rendering pipeline of 3DGS to process multi-source data, fully mining the detailed information of scenes with weak lighting and large light ratio, and realizing higher-quality three-dimensional reconstruction and new perspective synthesis of fusion rendering. Therefore, the present invention is based on multi-source optical sensors, mining the geometric and texture information of the scene, combining the 3DGS architecture with the training strategy of image fusion, and realizing high-quality three-dimensional fusion rendering under complex lighting.

[0058] Among them, 3DGS is a technique for 3D reconstruction and NVS of scenes from multi-view images acquired by a single optical sensor. It has three important components. First, 3D Gaussian functions are introduced as scene representation. This method uses inputs similar to the traditional NeRF method, namely cameras calibrated by structured light measurement (SfM), and uses sparse point clouds generated in the SfM process to initialize a set of 3D Gaussian functions. Unlike most point-based solutions that require multi-view stereo matching (MVS) data, 3DGS only uses points generated by SfM to achieve high-quality rendering results. The reason for choosing 3D Gaussian functions as scene representation is that they are not only differentiable volume representations, but also can be efficiently rasterized by projecting into two dimensions and applying a standard mixing process, which is consistent with the image formation model of NeRF. Secondly, the properties of 3D Gaussian functions, including 3D position, opacity, anisotropic covariance, and spherical harmonic (SH) coefficients, are optimized, and an adaptive density control step is used to alternate. 3D Gaussian functions are dynamically added and removed when the optimization is performed, thereby achieving a compact, unstructured and accurate scene representation. Finally, 3DGS proposes a real-time rendering system that efficiently utilizes the GPU to perform anisotropic splashing in the visibility order, and implements the backpropagation process more quickly and accurately through sorting and blending.

[0059] Existing 3D reconstruction and NVS technologies are committed to speeding up the operation, improving rendering accuracy, reducing initial dependencies, and reducing memory usage based on architectures such as 3DGS, but these works are all based on visible light images and are limited by the physical properties of visible light sensors. Visible light images usually perform poorly in low light, changing lighting, or complex scenes, resulting in missing information during reconstruction and difficulty in capturing the precise shape of objects. At the same time, they are affected by motion blur and lighting changes when collecting images, which also leads to a decrease in reconstruction quality. There are also works that combine lidar and inertial devices to provide effective scene structure supervision, but the cost is high and is still limited by a single optical source, making it difficult to capture richer optical details in the scene.

[0060] In this context, it is of great significance to integrate the complementary information of visible light sensors and infrared sensors to achieve high-quality three-dimensional reconstruction and rendering of low-light and high-light-ratio scenes.

[0061] Example

[0062] Since the existing datasets do not meet the requirements of fusion, we created a dataset of two scenes using images collected by an infrared and visible light binocular camera. The camera model is KYT-U130-60R3A. Since it uses near-wave infrared, an infrared fill light is used to highlight the targets in the scene. See the dataset structure for an introduction. Figure 1, detailed data see the table below:

[0063] Table 1 Basic information of the dataset used in the verification process

[0064]

[0065] The verification results are shown in Figures 4 to 6.

[0066] Figure 4(a)-Figure 4(e) is a comparison diagram of the dual-source and fused three-dimensional Gaussian sets of this embodiment, where Figure 4(a) to Figure 4(c) The full picture of the three-dimensional Gaussian set of the grassland dataset under visible light source, infrared source and fusion source is shown. Figure 4(d) to Figure 4(f) The full picture of the three-dimensional Gaussian set of the schoolbag dataset under visible light source, infrared source and fusion source is shown. The scene is wrapped in darkness because there is a blind spot when the camera collects image data; the irregular edge is determined by the nature of the scene representation.

[0067] Figure 5(a) to Figure 5(c) as well as Figure 6(a) to Figure 6(c) They are the image pairs of the schoolbag and grass datasets before fusion, and the new perspective synthetic image after fusion. This embodiment uses a near-wave infrared camera and an infrared fill light to simulate the heat distribution in the scene observed by a thermal infrared camera. Therefore, the fused scene has a clearer spatial structure while retaining rich textures, and the hot targets are highlighted. A breakthrough from zero to one in the generation of new perspective fusion images has been achieved, improving the quality of 3D fusion reconstruction and NVS based on dual-source optical sensors.

Claims

1. A three-dimensional reconstruction and rendering method based on multi-source optical sensors, characterized in that: Includes the following: Reconstructing the multi-view dual-source images collected by the infrared-visible light binocular camera to obtain a dual-source registration data set, wherein the dual-source registration data set includes an infrared data set and a visible light data set; wherein the visible light data set includes a visible light image, visible light camera parameters, and a visible light three-dimensional Gaussian set; and the infrared data set includes an infrared image, infrared camera parameters, and an infrared three-dimensional Gaussian set; The infrared three-dimensional Gaussian set and the visible light camera parameters are rendered by a virtual perspective renderer to obtain an infrared image of a virtual perspective, and the infrared image of the virtual perspective is completely aligned with the visible light image in the visible light data set; The infrared image of the virtual perspective, the visible light data set and the infrared data set are simultaneously sent to the 3D Gaussian feature fusion device to mine the geometric structure and texture information of the 3D scene and obtain the optimized fused 3D Gaussian set. The optimized fused three-dimensional Gaussian set and visible light camera parameters are input into a generator to generate a fused video.

2. A three-dimensional reconstruction and rendering method based on a multi-source optical sensor as claimed in claim 1, characterized in that: The fusion method of the three-dimensional Gaussian feature fuser is specifically as follows: The visible light three-dimensional Gaussian set and the infrared three-dimensional Gaussian set are fused to obtain a fused three-dimensional Gaussian set; The fused three-dimensional Gaussian set and visible light camera parameters are rendered by a projection module and a rasterization module to obtain a fused image; The fused image, the infrared image of the virtual perspective and the visible light image are all input into a loss function calculation module. The loss function module calculates a function value. The function value obtains the gradient of each parameter in the fused three-dimensional Gaussian set through gradient back propagation. Then, each parameter of the fused three-dimensional Gaussian set is updated and iterated according to the gradient direction. After multiple iterations, an optimized fused three-dimensional Gaussian set is obtained.

3. A three-dimensional reconstruction and rendering method based on a multi-source optical sensor as claimed in claim 1 or 2, characterized in that: The virtual perspective renderer includes a projection module and a rasterization module.

4. A three-dimensional reconstruction and rendering method based on a multi-source optical sensor as claimed in claim 1 or 2, characterized in that: The generator includes a projection module, a rasterization module and a frame merging module; the optimized fused three-dimensional Gaussian set and visible light camera parameters are input into the projection module, and after being rendered by the projection module and the rasterization module, they are input into the frame merging module to finally generate a fused video.

Images