A three-dimensional reconstruction method based on fusion of visible light and near-infrared dual-mode information

Abstract

The application discloses a three-dimensional reconstruction method based on fusion of visible light and near-infrared dual-mode information, comprising: acquiring a pair of multi-view visible light and near-infrared images, performing motion structure-from-motion reconstruction on the visible light image to obtain camera pose and sparse point cloud; using a dense cross-modal matcher based on a Transformer to extract matching points and merge them into a global point set, and solving an external parameter matrix between two cameras to realize spatial alignment; using the visible light sparse point cloud as a geometric primitive to generate a Gaussian ellipsoid set, and assigning shared geometric parameters and independent appearance parameters of the corresponding dual-mode to each Gaussian ellipsoid; performing alternating joint iterative optimization on the Gaussian ellipsoid parameters as optimization variables, and accumulating the geometric parameter gradient of the two branches to make the near-infrared provide a supplementary gradient in the visible light limited area; and outputting a three-dimensional reconstruction result after training. The application improves the geometric integrity and rendering fidelity of three-dimensional reconstruction under poor imaging conditions.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, and in particular to a 3D reconstruction method based on fusing visible light and near-infrared dual-modal information. Background Technology

[0002] With the development of deep learning and 3D reconstruction technologies, the demand for scene reconstruction and new perspective synthesis based on multiple views is increasing for applications such as virtual reality, augmented reality, robot navigation, autonomous driving, and digital twins. 3D Gaussian sputtering, due to its advantages such as fast reconstruction speed, high rendering efficiency, and strong differentiability, has been widely used in recent years for real-time or near-real-time high-fidelity rendering tasks. However, existing 3D Gaussian reconstruction methods mainly rely on visible light images. Under poor imaging conditions such as overexposure and low texture, the quality of visible light images degrades severely, leading to a significant decrease in reconstruction quality. Specifically, current technology faces the following challenges: (1) Information bottleneck of single mode: Visible light cameras lose detail due to pixel saturation in overexposed areas, and produce confusion between reflection and penetration on glass or transparent surfaces. In these scenarios, it is difficult to extract sufficient geometric and texture clues by relying solely on visible light images, resulting in holes, blurring and geometric errors in the reconstruction results.

[0003] (2) Difficulty in utilizing cross-modal information: Near-infrared images are unaffected by visible light and maintain stable imaging even in overexposed areas. They also exhibit spectral response characteristics to materials such as glass, water, and vegetation that differ from those of visible light. However, fusing visible light and near-infrared information for 3D reconstruction faces two core challenges: First, there is a physical baseline offset and angular difference between the two cameras, requiring precise calibration of their positional transformation relationship; second, the brightness characteristics of the two modes are completely different in different spectral bands, and how to decouple and simultaneously model the two appearances under a unified 3D geometric representation is a problem that has not yet been well solved.

[0004] (3) The accuracy bottleneck of cross-modal extrinsic parameter calibration: Traditional methods rely on manually designed local feature descriptors, such as SIFT, to match between two modal images and then estimate extrinsic parameters. However, there is a gray-level nonlinear inversion caused by the difference in material spectral response between visible light and near-infrared images, which leads to inconsistent local gradient directions, extremely low inlier rate of feature matching, and large dispersion of extrinsic parameter estimation results between frames, which seriously restricts the accuracy of subsequent reconstruction.

[0005] In summary, existing 3D reconstruction methods face multiple challenges in cross-modal fusion scenarios, including extrinsic parameter calibration, information decoupling, and geometric preservation. Each step is independent, errors accumulate progressively, and a systematic solution for co-optimizing visible and near-infrared light is lacking. Especially when overexposed areas, low-texture areas, and glass or semi-transparent materials coexist in the scene, single-modal reconstruction methods struggle to recover the complete geometry, while traditional multimodal fusion strategies are limited by low extrinsic parameter calibration accuracy and intermodal information coupling. There is an urgent need for a 3D reconstruction method that can simultaneously address cross-modal spatial alignment, geometric appearance decoupling, and gradient complementary optimization within a unified framework, enabling high-quality 3D reconstruction using near-infrared information under conditions of limited visible light imaging. Summary of the Invention

[0006] This invention proposes a three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information to overcome the shortcomings of existing technologies, such as insufficient information of a single modality under harsh imaging conditions, low accuracy of cross-modal extrinsic parameter calibration, and difficulty in geometric and appearance coupling between modalities. It can obtain reconstruction and rendering results with complete geometric structure and high rendering fidelity in overexposed and low-texture scenes.

[0007] On the one hand, to achieve the above objectives, the present invention provides a three-dimensional reconstruction method based on fusing visible light and near-infrared dual-modal information, comprising: Acquire pairs of multi-view visible light and near-infrared images, perform motion recovery and structure reconstruction on the visible light images, and obtain the visible light camera pose and visible light sparse point cloud; A dense cross-modal feature matcher based on Transformer is constructed to extract cross-modal matching points from paired visible light and near-infrared images. The matching points of all frames are merged into a global matching point set. Essential matrix estimation and geometric consistency check are performed on the global matching point set to solve the position transformation extrinsic parameter matrix between the visible light camera and the near-infrared camera. The position transformation extrinsic parameter matrix is ​​used to map the pose of each visible light camera to the corresponding pose of the near-infrared camera, thereby achieving dual-modal spatial alignment. An initial set of Gaussian ellipsoids is generated using the visible light sparse point cloud as geometric primitives. Each Gaussian ellipsoid is assigned shared geometric parameters and independent appearance parameters. The shared geometric parameters include the center position, spatial scale, and rotation direction. The independent appearance parameters correspond to the visible light mode and the near-infrared mode, respectively. Using the parameters of the Gaussian ellipsoid set as the variables to be optimized, alternating joint iterative optimization is performed under the supervision of multi-view paired images. In each iteration, forward rendering and backpropagation of the visible light branch and the near-infrared branch are executed sequentially. Each branch calls the exclusive appearance parameters of the corresponding modality for rendering and calculates the appearance loss with the ground truth image of the corresponding modality. During backpropagation, each branch updates the exclusive appearance parameters of its own modality and accumulates the gradient of the shared geometric parameters. The backpropagation of the two branches completes the unified update of the shared geometric parameters. The near-infrared modality provides supplementary gradients for the shared geometric parameters in the visible light imaging-limited region. After training, the 3D reconstruction results fused with dual-modal information are output.

[0008] Preferably, the Transformer-based dense cross-modal feature matcher uses a LoFTR model pre-trained on a large-scale outdoor scene dataset, and establishes cross-modal feature correspondence between visible light and near-infrared images across the entire image range through a coarse-to-fine multi-scale cross-attention mechanism.

[0009] Preferably, the essential matrix estimation adopts an adaptive thresholding method, which automatically learns the interior point determination threshold based on the distribution of matching point data; The geometric consistency test uses the Cheirality test, which selects the solution with the most interior points visible in front of both cameras from the candidate solutions obtained by essential matrix decomposition as the final rotation matrix and translation vector.

[0010] Preferably, each of the independent appearance parameters includes the spherical harmonic function coefficients, opacity, and three-dimensional low-pass filter radius of the corresponding mode. The three-dimensional low-pass filter radius is calculated independently based on the camera focal length and resolution of the corresponding mode and is used to perform adaptive anti-aliasing correction on the Gaussian ellipsoid before rendering. The three-dimensional low-pass filter radius is calculated as follows: for each Gaussian ellipsoid, the three-dimensional low-pass filter radius is equal to the distance from the center of the corresponding Gaussian ellipsoid to the current rendering camera divided by the focal length of the corresponding modal camera, and then multiplied by the square root of the preset kernel scale coefficient; during rendering, the covariance matrix and opacity of the Gaussian ellipsoid under the corresponding modality are corrected using the corresponding filter radius to ensure that the integral value of the Gaussian ellipsoid in screen space remains unchanged before and after low-pass filtering.

[0011] Preferably, the shared geometric parameters are updated uniformly, including: A shared geometry optimizer, a visible light appearance optimizer, and a near-infrared appearance optimizer are set up to manage their respective parameters. After the backpropagation of each branch is completed in each iteration, the parameters of the shared geometry optimizer, the visible light appearance optimizer, and the near-infrared appearance optimizer are updated in sequence.

[0012] Preferably, the appearance loss of each branch is composed of a weighted average of L1 pixel loss and structural similarity loss.

[0013] Preferably, it further includes a density control step, including cloning operation, splitting operation and double-consistency pruning operation; The cloning and splitting operations are performed based on the gradient accumulation of the three-dimensional Gaussian primitives in the visible light mode in the screen space. The dual-consistency pruning operation is performed under the necessary condition that both the visible light opacity and the near-infrared opacity are below a preset threshold.

[0014] Preferably, during the training process, the order of the spherical harmonic function gradually increases from a low order to a preset maximum order with the number of iterations, and the opacity of each mode is periodically reset according to a preset period.

[0015] Preferably, the position transformation extrinsic parameter matrix is ​​used as a fixed constant for subsequent bimodal training after estimation, or it is incorporated into the training process as a learnable parameter and fine-tuned online through backpropagation of the gradient of the rendering loss.

[0016] On the other hand, to achieve the above objectives, the present invention also provides a three-dimensional reconstruction system that integrates visible light and near-infrared dual-modal information, comprising: The image acquisition and reconstruction module is used to acquire pairs of multi-view visible light images and near-infrared images, perform motion recovery structure reconstruction on the visible light images, and obtain the visible light camera pose and visible light sparse point cloud. The cross-modal extrinsic parameter estimation module is used to construct a dense cross-modal feature matcher based on Transformer. It extracts cross-modal matching points from paired visible light and near-infrared images, merges the matching points of all frames into a global matching point set, performs essential matrix estimation and geometric consistency check on the global matching point set, solves the position transformation extrinsic parameter matrix between the visible light camera and the near-infrared camera, and uses the position transformation extrinsic parameter matrix to map the pose of each visible light camera to the corresponding pose of the near-infrared camera, thereby achieving dual-modal spatial alignment. The three-dimensional scene initialization module is used to generate an initial set of Gaussian ellipsoids using the visible light sparse point cloud as geometric primitives, and to assign shared geometric parameters and independent appearance parameters to each Gaussian ellipsoid. The shared geometric parameters include center position, spatial scale and rotation direction, and the independent appearance parameters correspond to visible light mode and near-infrared mode, respectively. The alternating joint training module is used to perform alternating joint iterative optimization under the supervision of multi-view paired images, using the parameters of the Gaussian ellipsoid set as the variables to be optimized. In each iteration, forward rendering and backpropagation of the visible light branch and the near-infrared branch are executed sequentially. Each branch calls the exclusive appearance parameters of the corresponding modality for rendering and calculates the appearance loss with the ground truth image of the corresponding modality. During backpropagation, each branch updates the exclusive appearance parameters of its own modality and accumulates the gradient of the shared geometric parameters. The backpropagation of the two branches completes the unified update of the shared geometric parameters. The near-infrared modality provides supplementary gradients for the shared geometric parameters in the visible light imaging-limited region. The reconstruction output module is used to output the 3D reconstruction result that fuses bimodal information after training.

[0017] Compared with the prior art, the present invention has the following advantages and technical effects: (1) This invention addresses three types of problems simultaneously in the same training process: single-modal information bottleneck, low cross-modal extrinsic parameter calibration accuracy, and geometric appearance coupling between modalities. This avoids the accumulation of errors and information loss caused by separating multimodal fusion, extrinsic parameter calibration, and geometric reconstruction.

[0018] (2) In terms of extrinsic parameter calibration, this invention significantly improves the success rate of extrinsic parameter estimation by using the Transformer dense matcher and the global joint estimation strategy, effectively overcoming the problem of extremely low inlier rate caused by gray-scale nonlinear inversion in the traditional SIFT method, and avoiding the dispersion introduced by frame-by-frame estimation and averaging.

[0019] (3) In terms of geometric appearance decoupling, the present invention establishes independent appearance parameters for visible light and near infrared light under a unified three-dimensional geometric representation, so that the two modes share the physical structure while maintaining their true brightness response characteristics, and do not generate appearance interference while complementing geometric information across modes.

[0020] (4) In terms of training mechanism, the present invention accumulates the gradient of shared geometric parameters in alternating joint training, so that near-infrared provides supplementary gradient for shared geometry in dark and overexposed areas where visible light imaging quality is limited, effectively improving the integrity of dark structure and edge clarity, and reducing holes and blur caused by insufficient single modal information.

[0021] (5) In terms of rendering quality, the present invention uses a dual-modal three-dimensional low-pass filter independent calculation mechanism to adaptively compensate for the rendering aliasing caused by the resolution difference between the two modes, ensuring that both modes output high-fidelity rendering results at their respective resolutions.

[0022] (6) In terms of density control, the present invention protects the Gaussian ellipsoids that are only visible in the near-infrared mode from being accidentally deleted by a double consistency pruning strategy, so that the reconstruction results have a stronger geometric preservation ability for special material regions that are permeable to near-infrared light but reflect or block visible light. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a three-dimensional reconstruction method based on fusing visible light and near-infrared dual-modal information according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the training process according to an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0026] This embodiment proposes a three-dimensional reconstruction method based on fusing visible light and near-infrared dual-modal information, including: Acquire pairs of multi-view visible light and near-infrared images, perform motion recovery and structure reconstruction on the visible light images, and obtain the visible light camera pose and visible light sparse point cloud; A dense cross-modal feature matcher based on Transformer is constructed to extract cross-modal matching points from paired visible light and near-infrared images. The matching points of all frames are merged into a global matching point set. Essential matrix estimation and geometric consistency check are performed on the global matching point set to solve the position transformation extrinsic parameter matrix between the visible light camera and the near-infrared camera. The position transformation extrinsic parameter matrix is ​​used to map the pose of each visible light camera to the corresponding pose of the near-infrared camera, thereby achieving dual-modal spatial alignment. An initial set of Gaussian ellipsoids is generated using the visible light sparse point cloud as geometric primitives. Each Gaussian ellipsoid is assigned shared geometric parameters and independent appearance parameters. The shared geometric parameters include the center position, spatial scale, and rotation direction. The independent appearance parameters correspond to the visible light mode and the near-infrared mode, respectively. Using the parameters of the Gaussian ellipsoid set as the variables to be optimized, alternating joint iterative optimization is performed under the supervision of multi-view paired images. In each iteration, forward rendering and backpropagation of the visible light branch and the near-infrared branch are executed sequentially. Each branch calls the exclusive appearance parameters of the corresponding modality for rendering and calculates the appearance loss with the ground truth image of the corresponding modality. During backpropagation, each branch updates the exclusive appearance parameters of its own modality and accumulates the gradient of the shared geometric parameters. The backpropagation of the two branches completes the unified update of the shared geometric parameters. The near-infrared modality provides supplementary gradients for the shared geometric parameters in the visible light imaging-limited region. After training, the 3D reconstruction results fused with dual-modal information are output.

[0027] Specifically, such as Figure 1 ,include: Step 1: Input pairs of visible light and near-infrared multi-view images. Use the structure-from-motion technique to acquire camera pose and reconstruct sparse point clouds from the visible light images. Extract cross-modal matching points from the pairs of visible light and near-infrared images using a Transformer-based dense cross-modal feature matcher. Merge the matching points of all frames into a global point set. Through adaptive threshold essential matrix estimation and Cheirality geometric consistency test, solve the position transformation extrinsic parameter matrix between the visible light camera and the near-infrared camera in one step. Use the matrix to map the pose of the visible light camera to the pose of the near-infrared camera to achieve dual-modal spatial alignment. Step 2: Using the visible light sparse point cloud as the geometric primitive, generate an initial set of Gaussian ellipsoids; assign shared geometric parameters and two sets of independent appearance parameters to each Gaussian ellipsoid. The shared geometric parameters include the center position, spatial scale, and rotation direction. The two sets of independent appearance parameters correspond to the visible light mode and the near-infrared mode, respectively, including spherical harmonic function coefficients, mode-specific opacity, and three-dimensional low-pass filter radius. Step 3: In the alternating joint training, the visible light branch and the near-infrared branch each perform forward rendering in each iteration, calculate the appearance loss with the ground truth image of the corresponding modality, and update the exclusive appearance parameters of each modality with the gradients. At the same time, the gradients of the shared geometric parameters of the two branches are accumulated, so that the near-infrared branch provides supplementary gradients for the shared geometry in the region where the visible light imaging quality is limited, thereby achieving cross-modal geometric complementarity. Step 4: In the joint loss optimization unit, the appearance loss of the visible light branch and the appearance loss of the near-infrared branch are combined to form a joint target. The appearance loss is composed of L1 pixel loss and structural similarity loss weighted together. Step 5: Perform joint iterative updates on the 3D Gaussian parameters, including shared geometric parameters, visible light mode-specific appearance parameters, and near-infrared mode-specific appearance parameters; during the density control phase, clone and split the Gaussian ellipsoid based primarily on the visible light mode gradient; during pruning, delete is performed when both visible light opacity and near-infrared opacity are below a threshold; terminate if the maximum number of iterations is reached or the convergence criterion is met, otherwise return to step 3 to continue training.

[0028] Step 1, cross-modal extrinsic estimation and bimodal space alignment, includes: The input consists of paired visible light and near-infrared multi-view images. The visible light images are processed using motion reconstruction techniques to obtain sparse 3D point clouds involving both the camera interior and exterior. A Transformer-based dense feature matcher extracts cross-modal feature matching points from the paired visible light and near-infrared images. Matching points from all frames are merged into a global matching point set. An adaptive threshold essential matrix estimation is performed on this global point set to obtain the essential matrix and interior point set. Then, a Cheirality test is used to filter interior points visible in front of both cameras. The solution with the most in-front points is selected from four candidate solutions as the unique rotation matrix and translation vector. Combined with the physical baseline length, a metric rigid body transformation extrinsic parameter matrix is ​​obtained. The pose of the visible light camera is mapped to the pose of the near-infrared camera using the aforementioned position transformation extrinsic parameter matrix.

[0029] Furthermore, the Transformer-based dense cross-modal feature matcher uses a LoFTR model pre-trained on a large-scale outdoor scene dataset to establish cross-modal feature correspondences between visible light and near-infrared images across the entire image range through a coarse-to-fine multi-scale cross-attention mechanism.

[0030] Specifically, the Transformer-based dense feature matcher uses a LoFTR model pre-trained on a large-scale outdoor scene dataset. It searches for cross-modal correspondences across the entire image through a coarse-to-fine multi-scale cross-attention mechanism, unaffected by the nonlinear inversion of grayscale between visible light and near-infrared.

[0031] Furthermore, the essential matrix estimation employs an adaptive thresholding method, which automatically learns the threshold for determining interior points based on the distribution of matching point data. The geometric consistency test uses the Cheirality test, which selects the solution with the most interior points visible in front of both cameras from the candidate solutions obtained by essential matrix decomposition as the final rotation matrix and translation vector.

[0032] Specifically, the adaptive threshold essential matrix estimation uses the MAGSAC++ method to automatically learn interior points and determine the threshold based on the data distribution, without the need for manually setting a fixed threshold. The global matching point set strategy does not estimate each pair individually and then average them. Instead, it combines the matching points of all frames and solves the problem all at once. This statistically makes full use of the constraint information of all available matching points and avoids estimation instability caused by insufficient matching points or excessive noise in a single frame.

[0033] Furthermore, step 2, the bimodal Gaussian initialization, includes: Using sparse point clouds from visible light images as geometric primitives, an initial set of Gaussian ellipsoids is generated. Each Gaussian ellipsoid shares three sets of geometric parameters: center position, spatial scale, and rotation direction. Simultaneously, each Gaussian ellipsoid is assigned appearance parameters specific to the visible light mode and appearance parameters specific to the near-infrared mode. The appearance parameters for the visible light mode include visible light spherical harmonic function coefficients, visible light opacity, and the radius of the visible light three-dimensional low-pass filter. The appearance parameters for the near-infrared mode include near-infrared spherical harmonic function coefficients, near-infrared opacity, and the radius of the near-infrared three-dimensional low-pass filter. The two sets of appearance parameters encode the brightness response characteristics of the corresponding spectral bands.

[0034] The calculation method for the radius of the three-dimensional low-pass filter is as follows: For each Gaussian ellipsoid, its radius of the three-dimensional low-pass filter is equal to the distance from the center of the Gaussian ellipsoid to the current rendering camera divided by the focal length of the camera in that mode, and then multiplied by the square root of the preset kernel scale coefficient. During rendering, the covariance matrix and opacity of the Gaussian ellipsoid in the corresponding mode are corrected using this filter radius: the effective scale is the square root of the sum of the squares of the original scale and the filter radius, and the effective opacity is scaled according to the square root of the ratio of the determinant of the covariance matrix before and after correction, so as to ensure that the integral value of the Gaussian ellipsoid in screen space remains unchanged before and after low-pass filtering, thereby realizing dual-modal adaptive anti-aliasing rendering and eliminating rendering artifacts caused by the different resolutions of the two modal images.

[0035] In step 3, alternating joint training is performed, such as... Figure 2 ,include: Within each iteration, the visible light branch uses the visible light opacity, visible light spherical harmonic function coefficients, and the current shared geometric parameters for rasterization rendering. It compares the rasterization with the ground truth image to calculate the visible light loss, and backpropagates to update the visible light opacity, visible light spherical harmonic function coefficients, and accumulates gradients to the shared geometric parameters. The near-infrared branch uses the near-infrared opacity, near-infrared spherical harmonic function coefficients, and the same shared geometric parameters for rasterization rendering. It compares the near-infrared with the ground truth image to calculate the near-infrared loss, and backpropagates to update the near-infrared opacity, near-infrared spherical harmonic function coefficients, and continues to accumulate gradients to the shared geometric parameters. After backpropagation of each branch is completed, the three optimizer groups perform parameter updates in sequence; the gradient accumulation of the shared geometric parameters of the two branches enables the near-infrared mode to provide supplementary gradients for the shared geometry in overexposed areas and low-texture areas with poor visible light imaging quality.

[0036] Furthermore, the shared geometric parameters are updated uniformly, including: A shared geometry optimizer, a visible light appearance optimizer, and a near-infrared appearance optimizer are set up to manage their respective parameters. After the backpropagation of each branch is completed in each iteration, the parameters of the shared geometry optimizer, the visible light appearance optimizer, and the near-infrared appearance optimizer are updated in sequence.

[0037] Specifically, the shared geometry optimizer manages three sets of shared geometric parameters: center position, spatial scale, and rotation direction; the visible light appearance optimizer manages the visible light spherical harmonic function coefficients and visible light opacity; and the near-infrared appearance optimizer manages the near-infrared spherical harmonic function coefficients and near-infrared opacity.

[0038] The execution order of the three optimizer groups in each iteration is as follows: first, complete the forward rendering and backpropagation of each of the two branches, and then execute the parameter update of the shared geometry optimizer, the parameter update of the visible light appearance optimizer, and the parameter update of the near-infrared appearance optimizer in sequence, so that the shared geometry parameters can synchronously receive gradient information from the two modes.

[0039] Furthermore, it also includes density control steps, including cloning operations, splitting operations, and double-consistency pruning operations; The cloning and splitting operations are performed based on the gradient accumulation of the three-dimensional Gaussian primitives in the visible light mode in the screen space. The dual-consistency pruning operation is performed under the necessary condition that both the visible light opacity and the near-infrared opacity are below a preset threshold.

[0040] Specifically, the cloning and splitting operations are based on the combination of the gradient accumulation of the Gaussian ellipsoid in the screen space and its world space scale under the visible light mode; the double consistency pruning operation requires that the visible light opacity and the near-infrared opacity are both below the preset minimum threshold as a necessary condition, and deletion is only performed when both modes determine that a certain Gaussian ellipsoid is invisible, to prevent the accidental deletion of areas that are visible in the near-infrared but invisible in the visible light. The near-infrared visible but visible light invisible regions include: the glass back structure that is near-infrared permeable but visible light reflective, and the semi-transparent material interface with different penetration depths of the two modes of photons.

[0041] Furthermore, during the training process, the order of the spherical harmonic function gradually increases from a low order to a preset maximum order with the number of iterations, and the opacity of each mode is periodically reset according to a preset period.

[0042] After the position transformation extrinsic matrix is ​​estimated, it can be used as a fixed constant for subsequent bimodal training, or it can be incorporated into the training process as a learnable parameter and fine-tuned online through backpropagation of the gradient of the rendering loss.

[0043] Specifically, the method couples cross-modal extrinsic parameter estimation with bimodal 3D Gaussian training into a unified framework. The output of cross-modal extrinsic parameter estimation is directly used as the input of the near-infrared camera pose in bimodal 3D Gaussian training. After the position transformation extrinsic parameter matrix is ​​estimated, it can be used as a fixed constant for subsequent bimodal training, or it can be incorporated into the training process as a learnable parameter for online fine-tuning and refinement through backpropagation of the gradient of the rendering loss, thereby achieving closed-loop optimization from offline coarse calibration to online fine calibration.

[0044] This implementation also provides a three-dimensional reconstruction system that integrates visible light and near-infrared dual-modal information, including: The image acquisition and reconstruction module is used to acquire pairs of multi-view visible light images and near-infrared images, perform motion recovery structure reconstruction on the visible light images, and obtain the visible light camera pose and visible light sparse point cloud. The cross-modal extrinsic parameter estimation module is used to construct a dense cross-modal feature matcher based on Transformer. It extracts cross-modal matching points from paired visible light and near-infrared images, merges the matching points of all frames into a global matching point set, performs essential matrix estimation and geometric consistency check on the global matching point set, solves the position transformation extrinsic parameter matrix between the visible light camera and the near-infrared camera, and uses the position transformation extrinsic parameter matrix to map the pose of each visible light camera to the corresponding pose of the near-infrared camera, thereby achieving dual-modal spatial alignment. The three-dimensional scene initialization module is used to generate an initial set of Gaussian ellipsoids using the visible light sparse point cloud as geometric primitives, and to assign shared geometric parameters and independent appearance parameters to each Gaussian ellipsoid. The shared geometric parameters include center position, spatial scale and rotation direction, and the independent appearance parameters correspond to visible light mode and near-infrared mode, respectively. The alternating joint training module is used to perform alternating joint iterative optimization under the supervision of multi-view paired images, using the parameters of the Gaussian ellipsoid set as the variables to be optimized. In each iteration, forward rendering and backpropagation of the visible light branch and the near-infrared branch are executed sequentially. Each branch calls the exclusive appearance parameters of the corresponding modality for rendering and calculates the appearance loss with the ground truth image of the corresponding modality. During backpropagation, each branch updates the exclusive appearance parameters of its own modality and accumulates the gradient of the shared geometric parameters. The backpropagation of the two branches completes the unified update of the shared geometric parameters. The near-infrared modality provides supplementary gradients for the shared geometric parameters in the visible light imaging-limited region. The reconstruction output module is used to output the 3D reconstruction result that fuses bimodal information after training.

[0045] Specifically, the system's operation process organizes data transfer and parameter updates in batches. The cross-modal extrinsic parameter estimation module passes the position transformation extrinsic parameter matrix obtained in one iteration to subsequent modules to map the visible light camera pose to the near-infrared camera pose. The alternating joint training module first performs forward rendering and backward propagation of the visible light branch in each iteration, and then performs forward rendering and backward propagation of the near-infrared branch. The gradient of the shared geometric parameters is accumulated and updated by the two branches, and the two sets of exclusive appearance parameters are updated independently by their respective appearance optimizers. In the density control stage, the cloning and splitting of Gaussian ellipsoids are guided by the visible light modal gradient, and the corresponding Gaussian ellipsoid is deleted only when both the visible light opacity and the near-infrared opacity are below the threshold.

[0046] To more clearly illustrate the technical solution of the present invention, specific embodiments are provided below for description: Step 1: Cross-modal extrinsic estimation and bimodal space alignment.

[0047] Step 1.1: Input a pair of visible light and near-infrared multi-view images and the physical baseline length between the two cameras. First, unify the two modal images to the same effective resolution or establish a size correspondence, and correct the focal length and principal point of the near-infrared camera's intrinsic parameters to maintain pixel spatial correspondence.

[0048] Step 1.2: Sparse reconstruction of the visible light image sequence is performed using the structure-from-motion technique. SIFT feature points and 128-dimensional local descriptors are extracted from each visible light image. Feature matching between image pairs is performed through approximate nearest neighbor search, and ambiguous matching is filtered out using ratio testing. Based on the initial image pairs, dual-view geometric estimation and triangulation are performed. New images are registered frame by frame, and global optimization using bundle adjustment is performed. The output includes the intrinsic parameter matrix K_color (focal length f_color and principal point c_color), extrinsic parameter pose (rotation matrix R_color_i from world coordinate system to camera coordinate system and translation vector t_color_i) of each visible light camera viewpoint, as well as a sparse 3D point cloud.

[0049] Step 1.3: Cross-modal dense feature matching. Multiple pairs of visible-near-infrared images are sampled from the paired image set. For each pair, a Transformer-based LoFTR model is used for dense matching. LoFTR first uses a convolutional neural network to extract multi-scale feature maps from the visible and near-infrared images respectively. At the coarse-grained level, cross-modal feature correspondences are established across the entire image using Transformer self-attention and cross-attention mechanisms. Subsequently, at the fine-grained level, the coarse matching is locally refined, finally outputting the coordinates of densely matched points with sub-pixel precision. LoFTR does not rely on local gradient direction consistency, effectively overcoming the gray-level nonlinear inversion problem caused by the difference in material spectral response between visible and near-infrared light.

[0050] Step 1.4: Global Joint Extrinsic Parameter Estimation. All cross-modal matching points from all frames are merged into a single global matching point set. An adaptive threshold essential matrix estimation is then performed on this global point set. The estimator uses the MAGSAC++ method, which automatically learns the interior point threshold based on the data distribution without requiring manual setting of a fixed threshold. Human bias in the threshold hyperparameters is eliminated through a marginalization process. Singular value decomposition is performed on the estimated essential matrix to obtain four candidate solutions for the rotation matrix and translation vector.

[0051] Step 1.5: Cheirality Geometric Consistency Check and Metric Extrinsic Parameter Solution. For each candidate solution, the global matching point set is reconstructed into 3D points through triangulation. The number of 3D points simultaneously in front of both the visible light camera and the near-infrared camera is counted. The candidate solution with the most in front points is selected as the final rigid body transformation, yielding the rotation matrix and the unit direction translation vector. The unit direction translation vector is multiplied by the known physical baseline length B (meters) to obtain the metric position transformation extrinsic parameter matrix RT.

[0052] This global joint estimation strategy does not estimate each pair individually and then average them. Instead, it combines the matching information of all frames and solves the problem at once, making full use of all available constraints and avoiding estimation instability caused by insufficient matching points or excessive noise in a single frame.

[0053] Step 1.6: Camera pose mapping. The camera poses from each visible light viewpoint are mapped to the camera poses from each near-infrared viewpoint using the position transformation extrinsic matrix RT. This completes the dual-modal spatial alignment, unifying the poses of the two camera modes in the same world coordinate system.

[0054] Step 2: Bimodal Gaussian initialization.

[0055] Step 2.1: Initialize shared geometric parameters. Using the visible light sparse point cloud output in Step 1.2 as the geometric primitive, initialize a Gaussian ellipsoid for each 3D point. The center position is directly taken from the SfM 3D point coordinates; the spatial scale is taken as the natural logarithm of the average distance between the 3D point and its K nearest neighbors (K is 3 to 5), repeated to the three axes as the initial value of the anisotropic scaling vector; the rotation direction is initialized to the unit quaternion [1,0,0,0], indicating no initial rotation.

[0056] Step 2.2: Initialize the bimodal appearance parameters. Two independent sets of appearance parameters are assigned to each Gaussian ellipsoid simultaneously.

[0057] Visible light modal appearance parameters: The zero-order component of the visible light spherical harmonic function coefficients is taken as the RGB mean of the projected color of the three-dimensional point on all visible light images, and the higher-order components are initialized to zero; the visible light opacity is initialized to the preset value of 0.1 after mapping by the inverse Sigmoid function; the radius of the visible light three-dimensional low-pass filter is initialized to zero.

[0058] Near-infrared modal appearance parameters: The zero-order component of the near-infrared spherical harmonic function coefficients is taken as the mean of the projection intensity of the three-dimensional point on all near-infrared images, and the higher-order components are initialized to zero; the near-infrared opacity is initialized to the same value as visible light; the radius of the near-infrared three-dimensional low-pass filter is initialized to zero.

[0059] The covariance matrix of a three-dimensional Gaussian ellipsoid at a spatial point x is determined by its spatial scale and rotation direction: ; in, It is a diagonal scaling matrix. These represent the scaling factors of Gaussian along the three principal axes. It is a 3×3 orthogonal rotation matrix, obtained by parameterization of a unit quaternion.

[0060] Step 3: Dual-modal alternating joint training.

[0061] Step 3.1: Viewpoint Sampling. In each iteration, a pair of visible light and near-infrared training views are randomly sampled from the training set with the same index. Sampling is not repeated during the process. After the sample is exhausted, it is refilled, and with a 30% probability, a high-resolution viewpoint pair with an image width of not less than 800 pixels is used to enhance the model's ability to perceive details. The corresponding visible light camera pose, near-infrared camera pose, visible light ground truth image, and near-infrared ground truth image are obtained.

[0062] Step 3.2: Forward rendering of the visible light branch. Using the visible light opacity, visible light spherical harmonic coefficients, and current shared geometric parameters, forward rendering is performed based on the visible light camera pose using the CUDA differentiable rasterizer.

[0063] The rasterization process is as follows: a three-dimensional Gaussian ellipsoid is projected onto the screen space through the camera projection matrix, the screen space covariance matrix is ​​calculated, and after sorting by depth, an alpha blending of anisotropic Gaussian kernels is performed to obtain a visible light rendered image.

[0064] Before rendering, anti-aliasing correction is performed on the Gaussian ellipsoid using a modal-specific 3D low-pass filter radius. For each Gaussian ellipsoid, the filter radius is equal to the distance from its center to the current rendering camera divided by the focal length of the modal camera and then multiplied by the square root of a preset kernel scale coefficient. The effective scale is corrected using this filter radius to the square root of the sum of the squares of the original scale and the filter radius, and the effective opacity is corrected to the square root of the original opacity multiplied by the ratio of the determinant of the covariance matrix before and after correction, ensuring that the integral value of the Gaussian ellipsoid in screen space remains unchanged before and after low-pass filtering.

[0065] Step 3.3: Visible light loss calculation and backpropagation.

[0066] Calculate the appearance loss between the visible light rendered image and the ground truth image: ; in, For the overall training loss, To render the image, To train the true values, The SSIM loss weight is set to 0.2. This loss is backpropagated, and the gradients are used to update the visible light opacity and visible light spherical harmonic coefficients, while accumulating the gradients to the shared geometric parameters.

[0067] Step 3.4: Near-infrared branch forward rendering. Using the near-infrared opacity, near-infrared spherical harmonic coefficients, and the same shared geometric parameters, rasterization rendering is performed based on the near-infrared camera pose to obtain the near-infrared rendered image. The 3D low-pass filter correction method is the same as in Step 3.2, but uses the camera focal length and resolution parameters of the near-infrared mode.

[0068] Step 3.5: Near-infrared loss calculation and backpropagation. Calculate the appearance loss between the near-infrared rendered image and the ground truth image, backpropagate this loss, update the near-infrared opacity and near-infrared spherical harmonic function coefficients with gradients, and continue accumulating gradients towards the shared geometric parameters.

[0069] Step 3.6: Optimizer Parameter Update. After the backpropagation of the two branches is completed, the parameters of the three optimizer groups are updated in sequence: the shared geometry optimizer manages the center position, spatial scale, and rotation direction, and updates them uniformly after receiving the gradients accumulated from the two branches; the visible light appearance optimizer manages the visible light spherical harmonic coefficients and visible light opacity, and updates them independently; the near-infrared appearance optimizer manages the near-infrared spherical harmonic coefficients and near-infrared opacity, and updates them independently.

[0070] Step 4: Joint loss optimization and density control.

[0071] Step 4.1: The joint loss target is the sum of visible light loss and near-infrared loss: ; In the formula, For the purpose of joint loss target, For visible light loss, This is due to near-infrared loss.

[0072] Step 4.2: Density Control. Between iterations 500 and 15000, a density control operation is performed every 100 iterations, based on the cumulative gradient of the Gaussian ellipsoid in the screen space under the visible light mode.

[0073] The cloning operation is applicable to Gaussian ellipsoids whose screen spatial gradient norm exceeds a preset threshold of 0.0002 and whose world spatial scale is smaller than a preset scale threshold. It copies a copy in its high gradient direction to increase the local sampling density. The splitting operation is applicable to Gaussian ellipsoids whose screen spatial gradient norm exceeds a preset threshold and whose world spatial scale is larger than a preset scale threshold, splitting them into multiple smaller Gaussian ellipsoids. The dual-consistency pruning operation requires that both the visible light opacity and the near-infrared opacity be below a preset minimum threshold of 0.005. Deletion is only performed when both modes determine that a certain Gaussian ellipsoid is invisible.

[0074] Dual consistency pruning is a key design feature that distinguishes this embodiment from existing single-modal 3D Gaussian methods. Existing methods only check the opacity of a single mode, which may lead to the incorrect deletion of Gaussian ellipsoids that are clearly visible in the near-infrared but nearly transparent in the visible light due to dim lighting or occlusion. This embodiment requires that the opacities of both modes simultaneously meet a threshold condition before pruning is performed, effectively protecting the complementary geometric information across modes. It is particularly suitable for scenarios such as the surface of objects in the exposure area, structures behind glass that are translucent in the near-infrared but reflective in the visible light, and interfaces of semi-transparent materials where the photons of the two modes have different penetration depths.

[0075] Step 4.3: Asymptotically increasing the order of the spherical harmonic function. At the start of training, the order of the spherical harmonic function is 0, and the Gaussian ellipsoid behaves as an isotropic diffuse reflector. Every 1000 iterations, the order is increased by one level until a preset maximum order of 3 is reached. This strategy allows for rapid convergence to low-frequency geometry and appearance in the early stages of training, while gradually introducing view-dependent detailed appearance modeling capabilities in the later stages, achieving phased optimization from coarse to fine.

[0076] Step 4.4: Periodically reset opacity. Every 3000 iterations, the visible light opacity and near-infrared opacity of all Gaussian ellipsoids are reset to preset initial values ​​to eliminate the failure of Gaussian ellipsoids caused by premature opacity saturation during training, and to ensure that a sufficient number of active Gaussian ellipsoids participate in geometry optimization during the density control phase.

[0077] Step 4.5: Dynamic Update of 3D Low-Pass Filter Radius. After each density control operation causes a change in the Gaussian ellipsoid set, the 3D low-pass filter radii for visible light and near-infrared light are immediately recalculated. After the density control phase ends, the filter radius is updated every 100 iterations to maintain rendering accuracy until the end of training.

[0078] Step 5: Terminate judgment and output.

[0079] Step 5.1: Terminate training when the preset maximum number of iterations is reached or the convergence criterion is met; if neither the maximum number of iterations is reached nor the convergence criterion is met, return to step 4 to enter the next round of alternating joint training iterations.

[0080] Step 5.2: After training terminates, save the model file containing all Gaussian ellipsoid parameters. For each Gaussian ellipsoid, record the center position, spatial scale, rotation quaternion, visible light spherical harmonic coefficients, near-infrared spherical harmonic coefficients, visible light opacity, near-infrared opacity, visible light 3D low-pass filter radius, and near-infrared 3D low-pass filter radius. Given the camera pose and desired modality of the new viewpoint, select the corresponding appearance parameters and 3D low-pass filter radius, and call the rasterizer to output a high-quality new viewpoint rendering result for the corresponding modality.

[0081] This embodiment constructs a unified training process that includes cross-modal extrinsic parameter estimation, bimodal Gaussian initialization, alternating joint training, bimodal rendering, and joint loss optimization. Under paired visible light and near-infrared multi-view input, it first uses a Transformer-based dense matcher to perform global joint estimation of all frame matching points to solve the rigid body transformation extrinsic parameter matrix between the two cameras in one step, achieving bimodal spatial alignment. Then, using visible light sparse point clouds as geometric primitives, it assigns shared geometric parameters and two sets of independent appearance parameters to each Gaussian ellipsoid, forming a "shared geometry - independent appearance" bimodal parameter organization. In alternating joint training, within each iteration, it first... The system performs forward rendering and backpropagation of the visible light branch, followed by forward rendering and backpropagation of the near-infrared branch. A gradient accumulation mechanism using shared geometric parameters allows near-infrared to provide supplementary gradients in visible light imaging-limited regions. For density control, the visible light modal gradient guides the cloning and splitting of Gaussian ellipsoids, and a dual-consistency pruning strategy protects cross-modal visible Gaussian primitives. For 3D correction, the low-pass filter radius is independently calculated for each modality to compensate for rendering aliasing caused by resolution differences. Finally, the appearance losses of the two modalities are jointly optimized to improve the 3D Gaussian parameters, resulting in a 3D scene reconstruction with complete geometry and high rendering fidelity under low light, overexposure, and low-texture conditions.

[0082] This embodiment also provides a three-dimensional reconstruction system that integrates visible light and near-infrared dual-modal information, including: The image acquisition and reconstruction module is used to acquire pairs of multi-view visible light images and near-infrared images, perform motion recovery structure reconstruction on the visible light images, and obtain the visible light camera pose and visible light sparse point cloud. The cross-modal extrinsic parameter estimation module is used to construct a dense cross-modal feature matcher based on Transformer. It extracts cross-modal matching points from paired visible light and near-infrared images, merges the matching points of all frames into a global matching point set, performs essential matrix estimation and geometric consistency check on the global matching point set, solves the position transformation extrinsic parameter matrix between the visible light camera and the near-infrared camera, and uses the position transformation extrinsic parameter matrix to map the pose of each visible light camera to the corresponding pose of the near-infrared camera, thereby achieving dual-modal spatial alignment. The three-dimensional scene initialization module is used to generate an initial set of Gaussian ellipsoids using the visible light sparse point cloud as geometric primitives, and to assign shared geometric parameters and independent appearance parameters to each Gaussian ellipsoid. The shared geometric parameters include center position, spatial scale and rotation direction, and the independent appearance parameters correspond to visible light mode and near-infrared mode, respectively. The alternating joint training module is used to perform alternating joint iterative optimization under the supervision of multi-view paired images, using the parameters of the Gaussian ellipsoid set as the variables to be optimized. In each iteration, forward rendering and backpropagation of the visible light branch and the near-infrared branch are executed sequentially. Each branch calls the exclusive appearance parameters of the corresponding modality for rendering and calculates the appearance loss with the ground truth image of the corresponding modality. During backpropagation, each branch updates the exclusive appearance parameters of its own modality and accumulates the gradient of the shared geometric parameters. The backpropagation of the two branches completes the unified update of the shared geometric parameters. The near-infrared modality provides supplementary gradients for the shared geometric parameters in the visible light imaging-limited region. The reconstruction output module is used to output the 3D reconstruction result that fuses bimodal information after training.

[0083] Specifically, paired visible light and near-infrared multi-view images and their camera parameters are used as training data. The five processing modules mentioned above are called in batches. The paired loss of visible light rendering and ground truth, and the paired loss of near-infrared rendering and ground truth are combined into the overall objective. The three-dimensional Gaussian parameters are jointly updated through forward inference and backpropagation. When convergence is not met or the iteration limit is reached, the process returns to the alternating joint training and rendering stage to continue iterating. After the termination condition is met, a high-quality three-dimensional scene reconstruction result with fused dual-modal information is output.

[0084] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A three-dimensional reconstruction method based on fusing visible light and near-infrared dual-modal information, characterized in that, include: Acquire pairs of multi-view visible light and near-infrared images, perform motion recovery and structure reconstruction on the visible light images, and obtain the visible light camera pose and visible light sparse point cloud; A dense cross-modal feature matcher based on Transformer is constructed to extract cross-modal matching points from paired visible light and near-infrared images. The matching points of all frames are merged into a global matching point set. Essential matrix estimation and geometric consistency check are performed on the global matching point set to solve the position transformation extrinsic parameter matrix between the visible light camera and the near-infrared camera. The position transformation extrinsic parameter matrix is ​​used to map the pose of each visible light camera to the corresponding pose of the near-infrared camera, thereby achieving dual-modal spatial alignment. An initial set of Gaussian ellipsoids is generated using the visible light sparse point cloud as geometric primitives. Each Gaussian ellipsoid is assigned shared geometric parameters and independent appearance parameters. The shared geometric parameters include the center position, spatial scale, and rotation direction. The independent appearance parameters correspond to the visible light mode and the near-infrared mode, respectively. Using the parameters of the Gaussian ellipsoid set as the variables to be optimized, alternating joint iterative optimization is performed under the supervision of multi-view paired images. In each iteration, forward rendering and backpropagation of the visible light branch and the near-infrared branch are executed sequentially. Each branch calls the exclusive appearance parameters of the corresponding modality for rendering and calculates the appearance loss with the ground truth image of the corresponding modality. During backpropagation, each branch updates the exclusive appearance parameters of its own modality and accumulates the gradient of the shared geometric parameters. The backpropagation of the two branches completes the unified update of the shared geometric parameters. The near-infrared modality provides supplementary gradients for the shared geometric parameters in the visible light imaging-limited region. After training, the 3D reconstruction results fused with dual-modal information are output.

2. The three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information according to claim 1, characterized in that, The Transformer-based dense cross-modal feature matcher uses a LoFTR model pre-trained on a large-scale outdoor scene dataset. It establishes cross-modal feature correspondence between visible light and near-infrared images across the entire image range through a coarse-to-fine multi-scale cross-attention mechanism.

3. The three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information according to claim 1, characterized in that, The essential matrix estimation adopts an adaptive thresholding method, which automatically learns the interior point determination threshold based on the distribution of matching point data. The geometric consistency test uses the Cheirality test, which selects the solution with the most interior points visible in front of both cameras from the candidate solutions obtained by essential matrix decomposition as the final rotation matrix and translation vector.

4. The three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information according to claim 1, characterized in that, Each of the independent appearance parameters includes the spherical harmonic function coefficients, opacity, and three-dimensional low-pass filter radius of the corresponding mode. The three-dimensional low-pass filter radius is calculated independently based on the camera focal length and resolution of the corresponding mode and is used to perform adaptive anti-aliasing correction on the Gaussian ellipsoid before rendering. The three-dimensional low-pass filter radius is calculated as follows: for each Gaussian ellipsoid, the three-dimensional low-pass filter radius is equal to the distance from the center of the corresponding Gaussian ellipsoid to the current rendering camera divided by the focal length of the corresponding modal camera, and then multiplied by the square root of the preset kernel scale coefficient; during rendering, the covariance matrix and opacity of the Gaussian ellipsoid under the corresponding modality are corrected using the corresponding filter radius to ensure that the integral value of the Gaussian ellipsoid in screen space remains unchanged before and after low-pass filtering.

5. The three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information according to claim 1, characterized in that, The shared geometric parameters are updated uniformly, including: A shared geometry optimizer, a visible light appearance optimizer, and a near-infrared appearance optimizer are set up to manage their respective parameters. After the backpropagation of each branch is completed in each iteration, the parameters of the shared geometry optimizer, the visible light appearance optimizer, and the near-infrared appearance optimizer are updated in sequence.

6. The three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information according to claim 1, characterized in that, The appearance loss of each branch is composed of a weighted average of L1 pixel loss and structural similarity loss.

7. The three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information according to claim 1, characterized in that, It also includes density control steps, including cloning, splitting, and double-consistency pruning. The cloning and splitting operations are performed based on the gradient accumulation of the three-dimensional Gaussian primitives in the visible light mode in the screen space. The dual-consistency pruning operation is performed under the necessary condition that both the visible light opacity and the near-infrared opacity are below a preset threshold.

8. The three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information according to claim 1, characterized in that, During training, the order of the spherical harmonic function gradually increases from a low order to a preset maximum order with the number of iterations, and the opacity of each mode is periodically reset according to a preset period.

9. The three-dimensional reconstruction method based on the fusion of visible light and near-infrared dual-modal information according to claim 8, characterized in that, The position transformation extrinsic matrix, after estimation, can be used as a fixed constant for subsequent bimodal training, or incorporated into the training process as a learnable parameter for online fine-tuning and refinement through backpropagation of the gradient of the rendering loss.

10. A three-dimensional reconstruction system fusing visible light and near-infrared dual-modal information, used to implement the three-dimensional reconstruction method based on fusing visible light and near-infrared dual-modal information as described in any one of claims 1-9, characterized in that, include: The image acquisition and reconstruction module is used to acquire pairs of multi-view visible light images and near-infrared images, perform motion recovery structure reconstruction on the visible light images, and obtain the visible light camera pose and visible light sparse point cloud. The cross-modal extrinsic parameter estimation module is used to construct a dense cross-modal feature matcher based on Transformer. It extracts cross-modal matching points from paired visible light and near-infrared images, merges the matching points of all frames into a global matching point set, performs essential matrix estimation and geometric consistency check on the global matching point set, solves the position transformation extrinsic parameter matrix between the visible light camera and the near-infrared camera, and uses the position transformation extrinsic parameter matrix to map the pose of each visible light camera to the corresponding pose of the near-infrared camera, thereby achieving dual-modal spatial alignment. The three-dimensional scene initialization module is used to generate an initial set of Gaussian ellipsoids using the visible light sparse point cloud as geometric primitives, and to assign shared geometric parameters and independent appearance parameters to each Gaussian ellipsoid. The shared geometric parameters include center position, spatial scale and rotation direction, and the independent appearance parameters correspond to visible light mode and near-infrared mode, respectively. The alternating joint training module is used to perform alternating joint iterative optimization under the supervision of multi-view paired images, using the parameters of the Gaussian ellipsoid set as the variables to be optimized. In each iteration, forward rendering and backpropagation of the visible light branch and the near-infrared branch are executed sequentially. Each branch calls the exclusive appearance parameters of the corresponding modality for rendering and calculates the appearance loss with the ground truth image of the corresponding modality. During backpropagation, each branch updates the exclusive appearance parameters of its own modality and accumulates the gradient of the shared geometric parameters. The backpropagation of the two branches completes the unified update of the shared geometric parameters. The near-infrared modality provides supplementary gradients for the shared geometric parameters in the visible light imaging-limited region. The reconstruction output module is used to output the 3D reconstruction result that fuses bimodal information after training.

Images