Visible-thermal-infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge

Abstract

This invention discloses a visible-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics. The invention proposes a dual-opacity 3D Gaussian network model for visible-thermal infrared joint reconstruction, combining intrinsic radiation property decomposition, a dual-opacity rendering mechanism with spectral decoupling, and a dual-modal optimization objective function for visible-thermal infrared scenes. This significantly improves the positioning accuracy, structural consistency, and physical realism of visible-thermal infrared scene reconstruction. By introducing a dual-opacity rendering mechanism within a shared geometric framework, the method effectively shields high-frequency visible light textures from the influence of conflicting gradients in low-frequency thermal modes, significantly suppressing boundary blurring and ghosting artifacts caused by multimodal fusion. Furthermore, the intrinsic radiation property decomposition module elevates the modeling of thermal infrared signals from heuristic intensity regression to physics-based radiation measurement inference, enhancing the accuracy of temperature inversion and avoiding viewpoint-dependent temperature drift.

Description

Technical Field

[0001] This invention relates to the field of computer vision and multimodal 3D scene reconstruction technology, specifically to a physically based dual-opacity Gaussian splatting (RaViGS) visible-thermal infrared joint reconstruction method. This method employs a dual-opacity parallel architecture, combining radiation physics priors with a modal decoupling visibility mechanism, significantly improving the radiometric measurement accuracy and visual fidelity of synthesized dual-modal new perspectives. This invention is applicable to 3D spatial scene reconstruction in all weather and complex environments, providing high-fidelity visual reconstruction and physically consistent thermal field detection tools for applications such as digital twins, autonomous driving, disaster response, and industrial inspection. Background Technology

[0002] With the rapid development of digital twins and autonomous driving, all-weather and robust multimodal 3D scene reconstruction has become a core requirement in the field of computer vision. Although visible light-based reconstruction has made steady progress in terms of geometric detail and texture fidelity, visible light observation alone is often severely limited under extreme conditions (such as low light, strong camouflage, or smoke obscuration), where semantic cues and physical properties of the scene can only be partially observed. Thermal infrared (TIR) ​​images provide complementary measurements that are largely independent of ambient light by sensing radiation caused by surface temperature and reveal thermal distributions beyond the visible light spectrum.

[0003] Existing multimodal 3D reconstruction methods, particularly extensions of Neural Radiation Field (NeRF) and 3D Gaussian Splashing (3DGS), are typically limited to improving rendering quality within a single modality, failing to adequately model the imaging mechanisms of specific modalities. Visible light and thermal infrared differ inherently in their imaging principles and spectral characteristics: visible light relies on reflected illumination, while thermal infrared captures emitted radiation. Current techniques often use thermal infrared signals as auxiliary single-channel intensity signals and directly reuse them for optimization based on appearance models developed for the visible spectrum, causing their observations to deviate from their radiometric meaning.

[0004] Furthermore, due to cross-spectral visibility differences (e.g., ordinary glass is essentially transparent in the visible light spectrum but is highly absorptive in the long-wave thermal infrared spectrum, effectively making it opaque), forcing a single geometric opacity across different modes is physically ineffective. The conflict between the rich high-frequency textures of visible light images and the low-frequency, piecewise smoothed thermal modes of thermal infrared images can cause severe gradient interference during joint optimization. This coupling typically leads to boundary artifacts, local structure degradation, and overly smoothed thermal fields, resulting in a persistent trade-off between true-color detail and thermal fidelity.

[0005] Accurately locating geometric structures in three-dimensional space and recovering high-fidelity true-color textures and thermal radiation properties is crucial for all-weather autonomous driving and industrial inspection. However, existing visible light-thermal infrared joint reconstruction methods are insufficient in the face of complex lighting and thermal environments. They rely on a single geometric support and tightly coupled visibility parameters, making it difficult to cope with cross-spectral occlusion conflicts, resulting in reconstruction artifacts and thermal field distortion. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a visible-thermal-infrared joint 3D Gaussian reconstruction method based on radiation physics. This invention proposes a dual-opacity 3D Gaussian network model for visible-thermal-infrared joint reconstruction, combining intrinsic radiation property decomposition, a spectrally decoupled dual-opacity rendering mechanism, and a visible-thermal-infrared dual-modal joint optimization objective function, significantly improving the localization accuracy, structural consistency, and physical realism of visible-thermal-infrared scene reconstruction. The network adopts a dual-modal parallel architecture, integrating two core rendering mechanisms and an advanced multi-objective loss function. On a unified geometric framework, the visible light branch captures viewpoint-dependent appearance; while the intrinsic radiation property decomposition module decomposes the thermal radiation of the thermal infrared branch into emissivity, temperature, and reflectivity, explicitly considering emission and environmental reflection. The spectrally decoupled dual-opacity rendering mechanism allows primitives to maintain independent visibility and transmittance for each modality, effectively resolving cross-spectral structural conflicts. The joint optimization objective function integrates dual-modal reconstruction, structural regularization, Kirchhoff's laws-based physical priors, and edge-guided smoothing loss to address parameter ambiguity issues. This invention significantly improves the accuracy and robustness of multimodal scene analysis and establishes the first 3DGS framework that can simultaneously provide physically consistent thermal fields and high-fidelity visible light reconstruction.

[0007] The technical solution of this invention to solve its technical problem includes the following steps:

[0008] In a first aspect, embodiments of this application provide a visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge, specifically including the following steps:

[0009] Acquire synchronized and spatially aligned real multi-view visible light and thermal infrared image sequence data, and extract the corresponding camera pose.

[0010] The acquired image sequences are preprocessed, and the sparse point cloud of the scene is extracted by the structure of motion reconstruction (SfM) system to obtain a bimodal image set, which serves as the initial geometric input of the bi-opacity 3D Gaussian network.

[0011] A dual-opacity 3D Gaussian network model for joint reconstruction of visible light and thermal infrared is constructed, and the intrinsic radiation properties of the 3D Gaussian primitives in the network model are decomposed and processed based on the Stefan-Boltzmann law.

[0012] The first stage of joint optimization is performed on a dual-opacity 3D Gaussian network model using a preprocessed bimodal image set, and the shared geometric scaffold of the Gaussian network is stabilized by performing a specified number of iterations.

[0013] After the shared geometric support stabilizes, the training strategy is switched to optimize the decoupling of specific modal features. After iterative training and model pruning using a minimum gating strategy, the final model parameter weights are obtained.

[0014] The trained model is rendered from a new perspective, outputting a synthetic image that includes high-fidelity color texture details and a physically based, radiation-consistent thermal field.

[0015] In one possible implementation, the dual-opacity 3D Gaussian network model consists of a bottom-level shared geometric support and two independent feature representation branches mounted thereon, namely a visible light representation branch and a thermal infrared radiation branch. Finally, the visible light-thermal infrared joint reconstruction is achieved through the linkage mapping of the dual-modal decoupled rendering mechanism.

[0016] In one possible implementation, the shared geometric scaffold is constructed and initialized by instantiating a set of discrete anisotropic 3D Gaussian primitives using the acquired sparse point cloud. For each Gaussian primitive in the scene, its spatial distribution attribute is defined as its center position. With covariance matrix Covariance matrix The center position is determined by the rotation matrix R used to control the orientation of the Gaussian body and the scaling matrix S used to control its anisotropic shape. R, S) constitute the underlying geometric support for cross-modal sharing, namely the shared geometric support.

[0017] In one possible implementation, the visible light representation branch is constructed as follows: based on the shared geometric scaffold, a first set of independent learnable parameters is assigned to each Gaussian primitive to construct the visible light representation branch. These branch parameters include: spherical harmonic (SH) coefficients C for encoding view-related color features, and a scalar opacity parameter specifically controlling visible light occlusion relationships.

[0018] In one possible implementation, the thermal infrared radiation branch is constructed as follows: parallel to the visible light branch, a second set of independent learnable parameters is assigned to each Gaussian primitive to construct the thermal infrared radiation branch. These parameters include: a set of intrinsic physical parameters derived from the Stefan-Boltzmann law, and a scalar opacity parameter specifically controlling the relationship between thermal infrared radiation transmission and occlusion.

[0019] In one possible implementation, the linkage mapping of the dual-modal decoupled rendering mechanism is as follows: For a target pixel p on the image plane, the rendering mechanism extracts and projects N Gaussian primitives that cover the pixel region along the viewing direction, and sorts them strictly according to spatial depth. In the point-based forward blending accumulation algorithm, the two modal branches share the same geometric spatial depth sorting, but extract their own independent opacities and calculate the transmittance in parallel, thereby synthesizing visible light color and thermal infrared radiation intensity respectively.

[0020] In one possible implementation, the intrinsic radiation property decomposition and processing are specifically as follows:

[0021] The Intrinsic Radiative Attribute Factorization module is used to introduce three learnable parameters with explicit physical meanings, namely physical attributes, namely emissivity, normalized temperature, and reflectivity, for each 3D Gaussian primitive according to the Stefan-Boltzmann law. The physical attributes are parameterized by low-order spherical harmonic (SH) functions.

[0022] In one possible implementation, the joint optimization in the first phase specifically operates as follows:

[0023] By assigning two decoupled opacity fields to each Gaussian primitive, physical occlusion in different bands is modeled independently. The complete parametric representation of a single Gaussian primitive is then updated to the union of its geometric scaffold and bi-branch features. A bimodal decoupled rendering mechanism projects the 3D spatial properties of the preprocessed bimodal image set onto a 2D image plane. When calculating the cumulative transmittance along the light rays, each branch invokes its own independent modal opacity field for forward blending. After forward blending, the bi-opacity 3D Gaussian network undergoes a first-stage joint optimization, executing a specified number of iterations through a Gaussian supervision mechanism.

[0024] In one possible implementation, a non-uniform alternating optimization strategy is introduced during the iterative training process after the first-stage joint optimization. By periodically freezing specific sets of network parameters, geometric properties, visible light features, and thermal infrared physical parameters are alternately optimized, thus explicitly decoupling the dual-modal features.

[0025] During iterative training, Gaussian primitives are periodically pruned using a "minimum gating strategy." The core decision rule of this strategy is that every Gaussian primitive retained in the model must be physically supported by observations from both visible light and thermal infrared modes.

[0026] Meanwhile, a multi-objective joint optimization loss was formulated, which includes photometric reconstruction loss, gradient alignment loss, spherical harmonic coefficient regularization, physical prior loss based on Kirchhoff's laws, and edge-aware guided smoothing loss.

[0027] In one possible implementation, during training using a non-uniform alternating optimization strategy, specific parameters are periodically frozen by forcing the learning rate of non-current phase parameter groups to 0. Specifically, the aforementioned alternating scheduling mechanism dynamically divides the complete training iteration cycle into the following four alternating cyclic phase parameter groups: geometric phase (occupying 40% of the cycle): optimizing only spatial position, scaling, and rotation parameters; opacity decoupling phase (occupying 20% ​​of the cycle): optimizing only visible light and thermal infrared dual opacity parameters; visible light appearance phase (occupying 20% ​​of the cycle): optimizing only RGB spherical harmonic characteristic coefficients; and infrared physical phase (occupying 20% ​​of the cycle): optimizing only physical parameters.

[0028] Secondly, embodiments of this application provide a visible light-thermal infrared joint 3D Gaussian reconstruction system based on radiation physics knowledge, including the following modules:

[0029] Data acquisition module: Acquires synchronized and spatially aligned real multi-view visible light and thermal infrared image sequence data, and extracts the corresponding camera pose.

[0030] Preprocessing module: preprocesses the acquired image sequence and extracts the sparse point cloud of the scene through the structure for motion reconstruction (SfM) system to obtain a bimodal image set, which serves as the initial geometric input of the bi-opacity 3D Gaussian network.

[0031] Network building module: Constructs a dual-opacity 3D Gaussian network model for joint visible light-thermal infrared reconstruction.

[0032] Attribute decomposition and processing module: Performs intrinsic radiation attribute decomposition and processing on 3D Gaussian primitives in the network model based on Stefan-Boltzmann law.

[0033] Training module: The first stage of joint optimization is performed on the dual-opacity 3D Gaussian network using a preprocessed bimodal image set, and the shared geometric scaffold of the Gaussian network is stabilized by performing a specified number of iterations.

[0034] After the shared geometric support stabilizes, the training strategy is switched to optimize the decoupling of specific modal features. After iterative training and model pruning using a minimum gating strategy, the final model parameter weights are obtained.

[0035] Reconstruction module: Renders the trained model from a new perspective, outputting a synthetic image containing high-fidelity color texture details and a physically consistent thermal field based on radiation.

[0036] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory;

[0037] The memory is used to store computer programs.

[0038] When the processor executes the program stored in the memory, it implements any of the visible light-thermal infrared joint 3D Gaussian reconstruction methods described in this application.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the visible light-thermal infrared joint 3D Gaussian reconstruction methods described in this application.

[0040] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the visible light-thermal infrared joint 3D Gaussian reconstruction methods described in this application.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention's method effectively shields high-frequency visible light textures from the influence of conflicting gradients in low-frequency thermal modes by introducing a dual-opacity rendering mechanism within a shared geometric support, significantly suppressing boundary blurring and ghosting artifacts caused by multimodal fusion. Furthermore, the intrinsic radiation property decomposition module elevates the modeling of thermal infrared signals from heuristic intensity regression to physically based radiation measurement inference, enhancing the accuracy of temperature inversion and avoiding viewpoint-dependent temperature drift. The physical priors and edge-aware smoothing loss introduced in the multi-objective joint optimization make this method more adaptable to modeling the thermal radiation of complex material interfaces, ensuring thermodynamically consistent radiation recovery while preserving true-color complex details. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the overall process of the method in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further explained in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0045] like Figure 1 As shown, this invention provides a visible light-thermal infrared joint 3D Gaussian reconstruction method (RaViGS framework) based on radiation physics knowledge. This method aims to resolve the fundamental incompatibility between reflected radiation (visible light) and emitted radiation (thermal infrared) within 3D Gaussian representation. Its complete technical implementation process is detailed below:

[0046] Step S1: Acquisition of multimodal data and sparse geometry initialization

[0047] In this embodiment, firstly, multi-view visible light and thermal infrared image pairs of the scene are acquired, and the corresponding camera pose is extracted. Specifically, an image sequence of indoor and outdoor scenes is captured using a device equipped with a multimodal sensor (such as the FLIR One Pro LT stereo camera on an iPhone). The indoor and outdoor scenes encompass building facades and everyday heat-generating objects (such as hot water cups, Raspberry Pis, etc.), spanning different temperature ranges. The device needs to provide synchronized and strictly spatially aligned multi-view visible light and thermal infrared image pairs. In actual measurements, thermal accuracy is within... arrive The effective accuracy within the range is approximately The actual physical temperature value was used as the primary reference for evaluating the quality of the 3D thermal reconstruction. Subsequently, the captured multi-view visible light and thermal infrared images were input into the Structure-from-Motion (SfM) system for feature matching and bundle adjustment to extract the precise camera pose (including extrinsic and intrinsic parameters) of each image and generate the initial sparse point cloud of the scene as the initial geometric input to the dual-opacity 3D Gaussian network.

[0048] Step S2: Construct a dual-opacity 3D Gaussian network for joint visible light-thermal infrared reconstruction

[0049] In the 3D Gaussian splash framework, network construction involves the explicit parameterization and definition of the scene feature space. This invention constructs a dual-opacity parallel architecture, whose overall network architecture consists of a bottom-level "shared geometric scaffold" and two independent feature representation branches mounted on it (i.e., a visible light representation branch and a thermal infrared radiation branch). Finally, visible light-thermal infrared joint reconstruction is achieved through the linkage mapping of a dual-modal decoupled rendering mechanism. The specific construction process is as follows:

[0050] Construction and initialization of the shared geometric scaffold: Utilizing the acquired sparse point cloud, a set of discrete anisotropic 3D Gaussian primitives are instantiated and used as optimizable basic physical nodes for the network. For Gaussian primitives in the scene, their spatial distribution attribute is defined as the center position. With covariance matrix Covariance matrix The center position is determined by the rotation matrix R used to control the orientation of the Gaussian body and the scaling matrix S used to control its anisotropic shape. (R, S) constitute the underlying geometric framework shared across modalities. The spatial contribution of Gaussian primitives at position x in 3D space is quantified by the formula:

[0051]

[0052] In the formula, G(x) represents the basic spatial contribution of the Gaussian primitive to the x-coordinate position in three-dimensional space; x is any point position in three-dimensional space; The spatial center of the Gaussian primitive; The three-dimensional covariance matrix is ​​used to determine the influence range and shape of the Gaussian body.

[0053] Construction of the Visible Light Representation Branch: Based on the shared geometric scaffold, a first set of independent learnable parameters is assigned to each Gaussian primitive to construct the visible light representation branch. This branch parameter includes: spherical harmonic (SH) coefficients C for encoding view-related color features, and a scalar opacity parameter specifically controlling visible light occlusion relationships. This branch is responsible for capturing the true-color textures and lighting variations of the scene in the visible light spectrum.

[0054] Construction of the thermal infrared radiation branch: Parallel to the visible light branch, a second set of independent learnable parameters is assigned to each Gaussian primitive to construct the thermal infrared radiation branch. This branch parameter includes: a set of intrinsic physical parameters derived from the Stefan-Boltzmann law (specifically including spherical harmonic coefficients of properties such as emissivity, temperature, and reflectivity), and a scalar opacity parameter specifically controlling the relationship between thermal infrared radiation transmission and occlusion. This branch aims to explicitly constrain the generation of infrared radiation fields based on physical laws.

[0055] Establishing the linkage mapping of the dual-modal decoupled rendering mechanism: The dual-branch parameterized network structure constructed above directly determines the decoupled rendering physical logic during the model's forward propagation process. For a target pixel p on the image plane, the rendering mechanism extracts and projects N Gaussian primitives covering the pixel region along the viewing direction, and strictly sorts them according to spatial depth. In the point-based forward mixing accumulation algorithm, the two modal branches share the same geometric spatial depth sorting, but extract their own independent opacities ( and Parallel calculation of transmittance allows for the separate synthesis of visible light color and thermal infrared radiation intensity, thus completely resolving the technical conflict between the inconsistency between physical visibility and occlusion in transspectral imaging.

[0056] Step S3: Decomposition and processing of intrinsic radiation properties based on Stefan-Boltzmann law

[0057] Unlike visible light imaging, which is primarily dominated by external illumination reflection, thermal infrared sensors record thermal infrared signals that are a superposition of self-emitted thermal radiation and reflected ambient radiation. Directly using thermal infrared intensity as an additional color channel for standard 3DGS optimization would cause its representation to deviate from the laws of radiation physics. Therefore, this invention introduces an Intrinsic Radiative Attribute Factorization (IRGF) module. Based on the Stefan-Boltzmann law, three learnable parameters with explicit physical meaning—i.e., physical attributes—are introduced for each 3D Gaussian primitive: emissivity... Normalized temperature T and reflectivity Considering the non-Lambertian emission effects of complex scene surfaces (such as unpolished surfaces or microfaceted shading), these properties exhibit direction dependence. This invention parameterizes the physical properties using low-order spherical harmonic (SH) functions to capture the anisotropic radiation behavior of emissivity and reflectivity. For a 3D Gaussian primitive i viewed from the viewing direction d, its physical properties are calculated as follows:

[0058]

[0059] In the formula, Represents the SH evaluation function; These are the SH coefficients learned for the corresponding physical attributes; It is a sigmoid activation function, which ensures that the values ​​of these three physical properties are strictly kept within the physically reasonable range of [0,1].

[0060] Step S4: Stabilization of the shared geometric support

[0061] First, the parameterized representation of the dual-modal decoupled rendering mechanism is adapted before joint optimization: Since electromagnetic waves of different bands exhibit drastically different spectral transmittance and physical occlusion relationships on the surface of materials (such as ordinary glass), forcing the use of a single opacity parameter will cause gradient conflicts and manifest as geometric artifacts. Therefore, this invention assigns two decoupled opacity fields to each Gaussian primitive to independently model the physical occlusion of different bands. At this point, the complete parameterized representation of a single Gaussian primitive i is updated to the union of its geometric support and bi-branch features, i.e.:

[0062]

[0063] After establishing the complete parameters, a dual-modal decoupled rendering mechanism is used to project the 3D spatial attributes of the preprocessed dual-modal image set onto the 2D image plane. Given the target pixel p and the viewing direction d on the image plane, let N(p) be the set of Gaussian primitives that contribute to pixel p along the viewing direction. The visible light and thermal infrared characterization branches are strictly locked and share the underlying geometric framework (i.e., spatial location). Covariance This means they maintain absolute consistency in spatial projection and depth ordering; when calculating the cumulative transmittance along the ray, the two branches call their respective independent modal opacity fields ( and Forward mixing synthesis is performed.

[0064] The formula for visible light branch rendering output is expressed as follows:

[0065]

[0066] The formula for the thermal infrared branch rendering output is expressed as follows:

[0067]

[0068] Where C(p) is the final rendered color output value of the visible light branch of the target pixel p on the image plane. The thermal infrared branch represents the final rendered radiation intensity output value of the target pixel p on the image plane. It is a set of 3D Gaussian primitives strictly ordered by spatial depth from near to far along the line of sight ray passing through the target pixel p. Let be the visible light color value of the i-th Gaussian primitive in the set along the viewing direction d. The thermal infrared radiation intensity value of the i-th Gaussian primitive in the set along the observation direction d. and These represent the cumulative transmittance of visible light and thermal infrared light before reaching the i-th Gaussian primitive, respectively, which is obtained by multiplying the corresponding opacities of all Gaussian primitives in front of the i-th Gaussian primitive.

[0069] After forward mixing synthesis, the first stage of joint optimization is performed on the dual-opacity 3D Gaussian network. Under the parallel synthesis architecture of the dual-opacity 3D Gaussian network, the spectral differences of specific modes are accurately routed and fitted to their respective branch parameters. , , In ), and because the two branches share the same set of geometric projection logic, the rendering error gradients generated by the two modal images will simultaneously propagate back to the underlying geometric variables ( , This invention executes a specified number of iterations through a Gaussian supervision mechanism, effectively preventing geometric degradation caused by missing single-modal observations (such as extremely dark lighting or thermal blind spots), thereby optimizing and solidifying the basic geometric framework of the scene and providing extremely reliable physical support for decoupling and pruning in subsequent steps.

[0070] Step S5: Switch the training strategy to optimize feature decoupling. After iterative training and model pruning based on the minimum gating strategy, the final model parameter weights are obtained.

[0071] After completing joint optimization and initially stabilizing the shared geometric scaffold in step S4, this step switches the training strategy to optimize the decoupling of specific modal features to prevent cross-modal gradient interference caused by the significant differences in spectral signal characteristics between the visible light and thermal infrared modes. Specifically, a non-uniform alternating optimization strategy is introduced during the iterative training process following the first-stage joint optimization. By periodically freezing specific network parameter sets and alternately optimizing geometric properties, visible light features, and thermal infrared physical parameters, the dual-modal features are explicitly decoupled, ensuring stable convergence of the model in complex environments.

[0072] During iterative training, Gaussian primitives need to be pruned periodically to dynamically optimize the scene's geometric topology, ensuring that the model capacity accurately adapts to the scene's complexity. For bi-branch structures, this invention proposes and utilizes a "minimum gating strategy" for model pruning. The core decision rule of this strategy is: every Gaussian primitive retained in the model must simultaneously receive physical support from observations in both visible light and thermal infrared modes.

[0073] The specific trimming judgment logic is as follows: if the visible light mode opacity of the i-th Gaussian primitive ( ) and thermal infrared mode opacity ( If all values ​​in the binary representation are below a preset retention threshold (usually set to 0.005), the primitive is determined to lack physical saliency in both modalities, triggering a removal operation. The determination logic is formalized as follows:

[0074]

[0075] in, This is an indicator function that outputs 1 when the decision condition is met, indicating that the primitive is marked as a pruning object and removed from the current model parameters and optimizer state. This strategy forces each retained geometric skeleton to have joint physical support from both visible light and thermal infrared modal observations.

[0076] To address the ambiguities in cross-spectral alignment and intrinsic physical parameter decomposition, this invention proposes a multi-objective joint optimization loss, which includes the following key components:

[0077] (1) Photometric reconstruction loss : Calculations were performed separately for the visible light and thermal infrared branches, combined with Distance and structural similarity (D-SSIM) is used to ensure the fidelity of the base rendering.

[0078]

[0079] in, This is a weighting coefficient (e.g., 0.2). and These are the rendered image and the real image, respectively.

[0080] (2) Gradient alignment loss To suppress high-frequency noise in the visible light branch and ensure a smooth new perspective, gradients in the X and Y directions of the rendered and real images in the visible light are calculated. punish.

[0081]

[0082] in, and These represent the first-order spatial gradient operators of the image in the horizontal (X-axis) and vertical (Y-axis) directions, respectively.

[0083] (3) Regularization of spherical harmonic coefficients : Restrict higher-order SH coefficients in Gaussian primitives ( The energy of the model is used to reduce overfitting of the model to specific observation perspectives.

[0084]

[0085] Where i is the index of the Gaussian primitive; Let represent the higher-order visible light spherical harmonic (SH) coefficients with order l>0 in the appearance feature of the i-th Gaussian primitive.

[0086] (4) Physical prior loss based on Kirchhoff's laws Since thermal infrared intensity is affected by multiple intrinsic factors, physical boundary constraints need to be introduced. This applies to the 0th-order components of the emissivity and reflectivity SH coefficients (…). The following punishments will be imposed:

[0087]

[0088] Where K is the set of 3D Gaussian primitive indices visible within the current camera's view frustum, and i is the index of an individual primitive in the set. Boundary function This function ensures that the range of values ​​for the physical property m has practical physical meaning by penalizing values ​​outside the [0,1] interval; tolerance term. This is used to address slight transmission and nonlinearity in sensor response.

[0089] (5) Edge-aware guided smoothing loss Considering the low resolution and sparse texture of thermal infrared images, a high-resolution visible light gradient is used as a gating signal to regularize the thermal infrared rendering.

[0090]

[0091] in In pixel coordinate space; The pixel gradient of the thermal infrared rendering image; The gradient of the real visible light image corresponding to the viewpoint; To control the attenuation coefficient of the guiding intensity.

[0092] At the edges of visible light (where the gradient is large), the exponential term approaches 0, relaxing the smoothing constraint on thermal infrared to preserve clear boundaries in the thermal map; in flat areas of visible light, the exponential term approaches 1, forcibly applying smoothing to significantly suppress thermal infrared noise.

[0093] The final multi-objective joint optimization function is:

[0094]

[0095] in These are the balancing weight coefficients for the corresponding loss terms.

[0096] Step S6: Model Training Scheduling and Evaluation of Multimodal Novel Perspective Synthesis

[0097] In the specific hardware and software environment of implementation, the dual-opacity 3D Gaussian network model based on the RaViGS framework constructed in this invention is implemented using the PyTorch deep learning framework and trained on a single NVIDIA RTX 3090 GPU. The optimization process uses the Adam optimizer, performing a total of 30,000 forward and backward propagation iterations (first stage joint optimization: belonging to the early stage of macroscopic processes (iterations 0-15,000). Geometric phase 6000; opacity decoupling phase 3000; visible light appearance phase 3000; infrared physical phase 3000). The momentum parameter is fixed at... The learning rate for the geometric position parameters is set from... Exponential smooth decay to The learning rate for the physical parameters (emissivity, temperature, and reflectivity) is initialized to 0.0025. The weights of each regularization term in the total loss function are configured as follows: gradient alignment loss weights. Edge-guided smoothing loss weights Physical prior loss weights .

[0098] During training using a non-uniform alternating optimization strategy, the learning rate of non-current phase parameter groups is forced to 0, and specific parameters are periodically and alternately frozen. This alternating scheduling mechanism effectively cuts off direct gradient interference between spectral signals, ensuring efficient and stable convergence of the dual-opacity system. Specifically, the above-mentioned alternating scheduling mechanism dynamically divides the complete training iteration cycle into the following four alternating cyclic phase parameter groups: geometric phase (occupying 40% of the cycle): only optimizes spatial position, scaling, and rotation parameters; opacity decoupling phase (occupying 20% ​​of the cycle): only optimizes visible light and thermal infrared dual-opacity parameters; visible light appearance phase (occupying 20% ​​of the cycle): only optimizes RGB spherical harmonic characteristic coefficients; infrared physical phase (occupying 20% ​​of the cycle): only optimizes physical parameters such as emissivity, temperature, and reflectivity.

[0099] The trained model weights are input into a parallel rasterization renderer for synchronous rendering under either the test set viewpoint or a new viewpoint. During rendering, the renderer utilizes a dual-modal decoupled rendering mechanism to strictly lock the shared underlying geometric framework and render separately based on the visible light modal opacity. and thermal infrared mode opacity The forward blending and accumulation are performed independently. The final output includes a visible light composite image with high-fidelity color texture details, as well as a thermal infrared composite image with a radiation-consistent thermal field based on physical laws, enabling efficient and high-precision 3D reconstruction and evaluation of scenes under complex lighting and thermal environments.

[0100] The experimental results show that the multi-branch joint reconstruction framework of this invention has achieved highly competitive performance in all indicators.

[0101] (The table below shows the extensive validation results of this invention on the ThermoNeRF dataset)

[0102] Table 1. Comparison of overall thermal infrared reconstruction performance of different models on multimodal datasets.

[0103]

[0104] Table 1 presents the quantitative results of the model's thermal infrared rendering on the comprehensive scene set. RaViGS performed best across all key performance indicators. In particular, the MAE was significantly reduced to 0.472, significantly outperforming the channel stitching baseline (3DGS+IR). This demonstrates that intrinsic radiometric property decomposition effectively resolves the physical biases caused by simple grayscale regression.

[0105] Table 2. Detailed Comparison of Thermal Infrared Reconstruction Performance under Extreme Temperature Scenarios

[0106]

[0107] Table 2 provides a detailed comparison of extreme scenarios involving drastic temperature gradients (such as the melting of ice cups). The results show that this method, constrained by physical laws, exhibits strong robustness in reconstructing radiation fields at dynamic heat conduction and complex interfaces.

[0108] Table 3 Comparison of visible light overall reconstruction performance of different models on multimodal datasets.

[0109]

[0110] Table 3 validates the advantages of decoupling specific modal visibility within a shared geometry. High-frequency visible light textures are effectively shielded from conflicting gradients in low-frequency thermal modes, and our method ranks first in average PSNR at 22.011 dB.

[0111] Table 4 Comparison of Visible Light Reconstruction in Indoor Fine-Grained True-Color Texture Scenes

[0112]

[0113] Table 4 shows the reconstruction results on indoor objects with complex reflective and specular properties (such as display shelves and Raspberry Pi circuit boards). This framework successfully mitigates the blur artifacts caused by fusion while preserving subtle true-color details. To verify the effectiveness of the various mechanisms and loss functions in the proposed multi-branch augmentation network framework, we conducted extensive system ablation experiments.

[0114] This application also provides a visible light-thermal infrared joint 3D Gaussian reconstruction system based on radiation physics knowledge, including the following modules:

[0115] Data acquisition module: Acquires synchronized and strictly spatially aligned real multi-view visible light and thermal infrared image sequence data from smart mobile devices or datasets with external or built-in visible light-thermal infrared stereo cameras, and extracts the corresponding camera pose.

[0116] Preprocessing module: preprocesses the acquired image sequence and extracts the sparse point cloud of the scene through the structure for motion reconstruction (SfM) system to obtain a bimodal image set, which serves as the initial geometric input of the bi-opacity 3D Gaussian network.

[0117] Network building module: Constructs a dual-opacity 3D Gaussian network model for joint visible light-thermal infrared reconstruction.

[0118] Attribute decomposition and processing module: Performs intrinsic radiation attribute decomposition and processing on 3D Gaussian primitives in the network model based on Stefan-Boltzmann law.

[0119] Training module: The first stage of joint optimization is performed on the dual-opacity 3D Gaussian network using a preprocessed bimodal image set, and the shared geometric scaffold of the Gaussian network is stabilized by performing a specified number of iterations.

[0120] After the shared geometric support stabilizes, the training strategy is switched to optimize the decoupling of specific modal features. After iterative training and model pruning using a minimum gating strategy, the final model parameter weights are obtained.

[0121] Reconstruction module: Renders the trained model from a new perspective, outputting a synthetic image containing high-fidelity color texture details and a physically consistent thermal field based on radiation.

[0122] This application also provides an electronic device, including a processor and a memory.

[0123] The memory is used to store computer programs.

[0124] When the processor executes a program stored in the memory, it implements any of the methods described in this application.

[0125] In one possible implementation, the electronic device of this application embodiment further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0126] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0127] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0128] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0129] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0130] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the methods described in this application.

[0131] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in this application.

[0132] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0135] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics, characterized in that, Includes the following steps: Acquire synchronized and spatially aligned real multi-view visible light and thermal infrared image sequence data, and extract the corresponding camera pose; The acquired image sequences are preprocessed, and the sparse point cloud of the scene is extracted by the motion reconstruction structure system to obtain a bimodal image set, which is used as the initial geometric input of the bi-opacity 3D Gaussian network. A dual-opacity 3D Gaussian network model for joint reconstruction of visible light and thermal infrared is constructed, and the intrinsic radiation properties of the 3D Gaussian primitives in the network model are decomposed and processed based on the Stefan-Boltzmann law. The first stage of joint optimization of the dual-opacity 3D Gaussian network model is performed using a preprocessed bimodal image set, and the shared geometric scaffold of the Gaussian network is stabilized by performing a specified number of iterations. After the shared geometric support stabilizes, the training strategy is switched to optimize the decoupling of specific modal features. After iterative training and model pruning using a minimum gating strategy, the final model parameter weights are obtained. The trained model is rendered from a new perspective, outputting a synthetic image containing high-fidelity true-color texture details and a physically based, radiation-consistent thermal field.

2. The visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge according to claim 1, characterized in that, The dual-opacity 3D Gaussian network model consists of a bottom-level shared geometric support and two independent feature representation branches mounted on it: a visible light representation branch and a thermal infrared radiation branch. Finally, the visible light-thermal infrared joint reconstruction is achieved through the linkage mapping of the dual-modal decoupled rendering mechanism.

3. The visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge according to claim 2, characterized in that, Construction and initialization of shared geometric scaffolding: Using the acquired sparse point cloud, a set of discrete anisotropic 3D Gaussian primitives are instantiated; for Gaussian primitives in the scene, their spatial distribution attributes are defined as center position and covariance matrix; the covariance matrix is ​​jointly determined by the rotation matrix used to control the orientation of the Gaussian body and the scaling matrix used to control its anisotropic shape; the above center position and geometric shape parameters constitute the underlying geometric scaffolding shared across modalities.

4. The visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge according to claim 3, characterized in that, Construction of the visible light representation branch: Based on the shared geometric support, a first set of independent learnable parameters is assigned to each Gaussian primitive to construct the visible light representation branch; the branch parameters include: spherical harmonic coefficients for encoding view-related color features, and scalar opacity parameters specifically controlling visible light occlusion relationships; Construction of the thermal infrared radiation branch: Parallel to the visible light branch, a second set of independent learnable parameters is assigned to each Gaussian primitive to construct the thermal infrared radiation branch; the parameters of this branch include: a set of intrinsic physical parameters derived from the Stefan-Boltzmann law, and a scalar opacity parameter that specifically controls the relationship between thermal infrared radiation transmission and occlusion.

5. The visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge according to claim 4, characterized in that, The linkage mapping of the dual-modal decoupled rendering mechanism is as follows: For a target pixel p on the image plane, the rendering mechanism extracts and projects N Gaussian primitives that cover the pixel region along the viewing direction, and sorts them strictly according to spatial depth. In the point-based forward mixing and accumulation algorithm, the two modal branches share the same geometric space depth sort, but extract their own independent opacities to calculate the transmittance in parallel, thereby synthesizing visible light color and thermal infrared radiation intensity respectively.

6. The visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge according to claim 1, characterized in that, The decomposition and processing of intrinsic radiation properties are as follows: The intrinsic radiation property decomposition module is used to introduce three learnable parameters with explicit physical meaning, namely physical properties, emissivity, normalized temperature and reflectivity, for each 3D Gaussian primitive according to the Stefan-Boltzmann law. The physical properties are parameterized by low-order spherical harmonic functions.

7. The visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge according to claim 1, characterized in that, The specific steps for the joint optimization in the first phase are as follows: By assigning two decoupled opacity fields to each Gaussian primitive, physical occlusion in different bands is modeled independently. At this point, the complete parameterized representation of a single Gaussian primitive is updated to the union of its geometric scaffold and bi-branch features. A bimodal decoupled rendering mechanism is used to project the 3D spatial properties of the preprocessed bimodal image set onto the 2D image plane. When calculating the cumulative transmittance along the light ray, the two branches call their respective independent modal opacity fields for forward blending. After the forward blending is completed, the bi-opacity 3D Gaussian network is subjected to the first stage of joint optimization, and a specified number of iterations are performed through a Gaussian supervision mechanism.

8. The visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge according to claim 7, characterized in that, In the iterative training process after the first stage of joint optimization, a non-uniform alternating optimization strategy is introduced. By periodically freezing specific network parameter sets, geometric properties, visible light features and thermal infrared physical parameters are alternately optimized, thus explicitly decoupling the dual-modal features. During iterative training, Gaussian primitives are pruned periodically using a "minimum gating strategy" for model pruning. The core decision rule of this strategy is that each Gaussian primitive retained in the model must be physically supported by observations from both visible light and thermal infrared modes. Meanwhile, a multi-objective joint optimization loss was formulated, which includes photometric reconstruction loss, gradient alignment loss, spherical harmonic coefficient regularization, physical prior loss based on Kirchhoff's laws, and edge-aware guided smoothing loss.

9. The visible light-thermal infrared joint 3D Gaussian reconstruction method based on radiation physics knowledge according to claim 8, characterized in that, During training using a non-uniform alternating optimization strategy, the learning rate of non-current phase parameter groups is forced to 0, and specific parameters are periodically frozen alternately. Specifically, the above-mentioned alternating scheduling mechanism dynamically divides the complete training iteration cycle into the following four alternating cyclic phase parameter groups: geometric phase: only optimizes spatial position, scaling, and rotation parameters; opacity decoupling phase: only optimizes visible light and thermal infrared dual opacity parameters; visible light appearance phase: only optimizes RGB spherical harmonic characteristic coefficients; infrared physical phase: only optimizes physical parameters.

10. A visible light-thermal infrared joint 3D Gaussian reconstruction system based on radiation physics, characterized in that, Includes the following modules: Data acquisition module: Acquires synchronized and spatially aligned real multi-view visible light and thermal infrared image sequence data, and extracts the corresponding camera pose; Preprocessing module: preprocesses the acquired image sequence and extracts the sparse point cloud of the scene through the motion reconstruction structure system to obtain a bimodal image set, which serves as the initial geometric input for the bi-opacity 3D Gaussian network; Network building module: Constructs a dual-opacity 3D Gaussian network model for joint visible light-thermal infrared reconstruction; Attribute decomposition and processing module: Performs intrinsic radiation attribute decomposition and processing on 3D Gaussian primitives in the network model based on Stefan-Boltzmann law; Training module: The first stage of joint optimization is performed on the dual-opacity 3D Gaussian network using a preprocessed bimodal image set, and the shared geometric scaffold of the Gaussian network is stabilized by performing a specified number of iterations. After the shared geometric support stabilizes, the training strategy is switched to optimize the decoupling of specific modal features. After iterative training and model pruning using a minimum gating strategy, the final model parameter weights are obtained. Reconstruction module: Renders the trained model from a new perspective, outputting a synthetic image containing high-fidelity color texture details and a physically consistent thermal field based on radiation.

Images