A few-view thermal radiation field reconstruction method based on RGB-guided three-dimensional Gauss

Abstract

The application discloses a few-view thermal radiation field reconstruction method based on RGB guided three-dimensional Gaussian. The application decouples RGB geometry training from thermal infrared appearance update, and the RGB three-dimensional Gaussian geometry skeleton only needs to be trained once, and the subsequent thermal infrared recapture mainly updates thermal properties, which significantly reduces the repeated thermal field modeling time; the normal-based m=0 spherical harmonic thermal radiation parameterization is used, which conforms to the physical law that thermal radiation is mainly related to the incident angle, and is more stable than directly learning thermal color independently for each view under the supervision of few-view; the visibility-aware regularization closed-form fitting avoids long end-to-end optimization of all parameters, and can adaptively adjust the high-order spherical harmonic term of the under-constrained Gaussian according to the number of visible views; through RGB semantic / appearance clustering and KNN graph heat propagation, the thermal spherical harmonic coefficient is completed in the area which is not fully observed by a few thermal infrared views, the thermal field integrity is improved, and holes are reduced.

Description

Technical Field

[0001] This invention belongs to the fields of three-dimensional reconstruction, thermal infrared imaging, computer vision and multimodal neural rendering technology, and specifically relates to a method for reconstructing a few-view thermal radiation field based on RGB-guided three-dimensional Gaussian. Background Technology

[0002] Thermal infrared imaging reflects temperature-related information by sensing the infrared energy radiated by an object. It is characterized by its independence from visible light illumination and its ability to operate in nighttime, smoke, low light, or partially obscured conditions. Therefore, thermal infrared 3D reconstruction has significant application value in industrial equipment inspection, building energy efficiency diagnosis, fire rescue, security monitoring, environmental monitoring, and robot perception.

[0003] Traditional thermal infrared 3D modeling typically involves first acquiring geometry from RGB images, depth maps, or structured light, and then projecting the thermal infrared image onto a 3D surface to form texture. This type of method relies on accurate geometric registration and dense thermal infrared overlay. However, when faced with low-resolution thermal infrared images, weak textures, a limited number of viewpoints, or rapidly changing thermal fields, it is prone to issues such as blurred textures, holes, and unstable rendering of new viewpoints.

[0004] Neural radiation field methods can obtain relatively smooth thermal infrared reconstruction results through continuum rendering, but their training and rendering usually require a large number of samples and a long optimization time, making them unsuitable for tasks involving frequent and repetitive data acquisition in industrial inspections or thermal process monitoring. 3D Gaussian splashing technology has advantages such as explicit point element representation, fast training, and real-time rendering, making it suitable for expansion into multimodal scene representation.

[0005] Existing 3D Gaussian thermal infrared reconstruction methods typically assume a dense thermal infrared view and jointly optimize geometry and thermal appearance at each acquisition. When the scene geometry is relatively stable but the temperature distribution changes over time, repeatedly optimizing the complete geometry incurs unnecessary computational overhead. On the other hand, with limited-view thermal infrared input, some 3D Gaussian features are invisible or weakly visible, and directly optimizing thermal properties can lead to high noise in underconstrained regions, unstable boundaries, or incomplete thermal radiation fields.

[0006] Therefore, there is a need for a method for reconstructing three-dimensional thermal radiation fields that can reuse RGB three-dimensional geometric skeletons, quickly update thermal appearance with only a small number of thermal infrared images, and maintain spatial and semantic consistency in underconstrained regions. Summary of the Invention

[0007] To address the problems of existing thermal infrared 3D reconstruction methods, such as the need for dense thermal infrared views, the requirement for lengthy joint optimization of geometry and thermal appearance for each acquisition, and the instability of thermal properties in underconstrained regions under limited viewing angles, this invention proposes a limited-view thermal radiation field reconstruction method based on RGB-guided 3D Gaussian.

[0008] The technical solution adopted in this invention is as follows:

[0009] A few-view thermal radiation field reconstruction method based on RGB-guided 3D Gaussian, such as Figure 1 As shown, it includes the following steps:

[0010] S1. Acquire multi-view RGB images of the target scene;

[0011] S2. Train an RGB 3D Gaussian model based on the acquired RGB images to obtain a 3D Gaussian geometric skeleton containing Gaussian center, scale, rotation, transparency, RGB spherical harmonic color and normal information.

[0012] S3. During the thermal infrared recapture stage, only a small number of thermal infrared images and corresponding camera poses are acquired.

[0013] S4. Fix or weakly adjust the three-dimensional Gaussian geometric skeleton, and based on three-dimensional Gaussian differentiable rendering, represent the thermal infrared pixel as a weighted sum of multiple three-dimensional Gaussian thermal intensities covering the pixel, and estimate the view-by-view thermal intensity of each three-dimensional Gaussian under each thermal infrared view through gradient optimization.

[0014] S5. Based on the visibility of the three-dimensional Gaussian in the thermal infrared view and the incident angle between the line of sight of the thermal infrared camera and the normal, construct the design matrix and introduce an adaptive regularization term related to the number of visible view and the spherical harmonic order. Solve the closed-form solution of the visibility perception spherical harmonic fitting independently for each three-dimensional Gaussian to obtain the initial thermal spherical harmonic coefficients.

[0015] S6. For invisible or weakly visible underconstrained 3D Gaussians, cluster them based on semantic or visual features extracted from RGB images, construct a KNN graph within each cluster, and propagate the thermal sphere harmonic coefficients of significant 3D Gaussians to non-significant 3D Gaussians through graph Laplacian smoothing to complete the thermal radiation field.

[0016] S7 uses RGB self-consistency and thermal infrared observation supervision to quickly refine the geometry and thermal appearance in short iterations, and outputs a complete three-dimensional thermal radiation field and thermal infrared rendering results from a new perspective.

[0017] Furthermore, in S2, the total loss function used to train the RGB 3D Gaussian model is:

[0018] ,

[0019] in, Indicates the RGB image reconstruction loss. This represents the scaling regularization term. Indicates the normal supervision weight, This indicates the difference between the rendered normals and the monocular normal estimation results;

[0020] After training, the shortest scale axis of the three-dimensional Gaussian is approximated as the normal of the Gaussian to obtain the normal information;

[0021] Based on the obtained three-dimensional Gaussian geometric skeleton, thermal infrared spherical harmonic properties are added to each three-dimensional Gaussian to obtain a multimodal three-dimensional Gaussian representation:

[0022] ,

[0023] in, Indicates the first A multimodal three-dimensional Gaussian, Indicates the center of Gauss. Indicates scale. Indicates rotation, Indicates transparency. Represents RGB spherical harmonic colors. This indicates the thermal infrared spherical harmonic properties.

[0024] Furthermore, in S4, C thermal infrared images are collected. For pixel p in thermal infrared viewpoint c, let K be the number of Gaussians covering the pixel, and let the k-th Gaussian have a corresponding rendering weight and per-view thermal intensity. Then the predicted thermal intensity is:

[0025] ,

[0026] in, Indicates thermal infrared perspective medium pixel The predicted thermal intensity This represents the summation of three-dimensional Gaussian regions covering the pixel. Indicates the Gaussian index that participated in the rendering. Indicates the covered pixels Gaussian quantity, Indicates the first A Gaussian perspective medium pixel Rendering weights Indicates the first Gauss in thermal infrared perspective Thermal intensity per angle;

[0027] Per-view thermal intensity is obtained by minimizing the following formula:

[0028] ,

[0029] in, This indicates the estimated loss based on the heat intensity per angle. Indicates the number of thermal infrared images. This indicates the number of pixels in each thermal infrared image. Indicates the thermal infrared field of view index. Indicates pixel index, Indicates the predicted heat intensity. Indicates the observed thermal infrared intensity. This represents the square norm 2.

[0030] Furthermore, in S5, such as Figure 2 As shown, for a 3D Gaussian and thermal infrared camera, let the unit line-of-sight direction from the camera to the center of the Gaussian be , and let the Gaussian normal be . Then the cosine of the incident angle is defined as follows, and its value ranges from 0 to 1:

[0031] ,

[0032] in, Indicates thermal infrared camera Observing a 3D Gaussian cosine of the angle of incidence at time Represents a three-dimensional Gaussian unit normal, Indicates camera Pointing to a three-dimensional Gaussian The unit's line of sight direction, This represents the transpose of a vector. Indicates taking the absolute value;

[0033] Express the thermal infrared intensity as a real spherical harmonic expansion with respect to the cosine of the incident angle at m=0:

[0034] ,

[0035] in, Represents the three-dimensional Gaussian cosine at the incident angle The thermal infrared intensity below, This represents summation over spherical harmonic orders. Indicates the index of the spherical harmonic order. Represents the order of the maximum spherical harmonic. Represents a three-dimensional Gaussian In order The unknown thermal harmonic coefficients on the surface express Real spherical harmonic basis functions;

[0036] like Figure 3 As shown, the method for constructing the design matrix based on the visibility of a 3D Gaussian in a thermal infrared viewpoint is to define the visibility mask of the 3D Gaussian g in viewpoint c and construct the design matrix:

[0037] ,

[0038] in, Represents a three-dimensional Gaussian From the perspective Harmony order Design matrix elements at the location, Represents a three-dimensional Gaussian From the perspective Visibility mask in express real spherical harmonic basis functions, Represents the cosine of the corresponding incident angle;

[0039] An adaptive regularization term related to the number of visible viewpoints and the spherical harmonic order is introduced as follows:

[0040] ,

[0041] ,

[0042] in, Represents a three-dimensional Gaussian Visibility adaptive regularization strength, This indicates the number of visible viewpoints for the Gaussian. Indicates the visibility smoothing parameter; Represents a three-dimensional Gaussian In spherical harmonic order Regularity coefficients on Represents the basic regularity coefficient. Indicates the order penalty exponent, The visibility regularization intensity obtained by expression (8) Represents the minimum regularity coefficient;

[0043] Let the regularization matrix be a diagonal matrix composed of regularization coefficients of each order. Solve the following regularized least squares problem independently for each three-dimensional Gaussian:

[0044] ,

[0045] in, Represented by the thermosphere harmonic coefficient Minimize the variables. This represents the design matrix formed by equation (7). Represents a three-dimensional Gaussian The unknown thermosphere harmonic coefficient matrix This represents a matrix that stacks the heat intensity per view across all visible viewpoints. Denotes the square of the Frobenius norm. Represents the trace of a matrix. Indicates matrix transpose. Indicates by The resulting diagonal regular matrix; the closed-form solution is:

[0046] ,

[0047] in, This represents the thermal harmonic coefficient obtained by closed-form calculation. Design matrix transpose, Represents the data terms in the normal equation. Represents a regular matrix. This represents finding the inverse of a matrix. This represents the thermal intensity matrix per view.

[0048] Furthermore, the specific methods of S6 are as follows: Figure 4 As shown:

[0049] First, calculate the significance score:

[0050] ,

[0051] in, Indicates the first A three-dimensional Gaussian The significance score, Indicates all thermal infrared views and pixels Take the maximum value above. Represents a three-dimensional Gaussian From the perspective medium pixel Contribution to transparency This indicates the corresponding transmittance term; This represents the rendering contribution of a 3D Gaussian to a specified pixel within a given viewpoint. If the value is greater than a preset threshold, the Gaussian is considered a significant 3D Gaussian; otherwise, it is considered a non-significant 3D Gaussian.

[0052] Extract pixel-level semantic or visual features from RGB images and enhance the two-dimensional features into three-dimensional Gaussian features based on the rendering contribution of the pixels by three-dimensional Gaussians:

[0053] ,

[0054] ,

[0055] in, This represents the three-dimensional Gaussian features before normalization. Indicates Gaussian index, This indicates a weighted summation. Represents the RGB image index. This indicates the number of RGB images involved in feature enhancement. Indicates pixel index, Represents a three-dimensional Gaussian For the first Pixels in an RGB image Rendering contribution weight Indicates the first Pixels in an RGB image Two-dimensional semantic or visual features; This represents the normalized 3D Gaussian feature. express The vector norm is then used to form Gaussian clusters based on feature similarity, and a KNN graph is constructed within each cluster.

[0056] For two adjacent nodes i and j within the same Gaussian variety, the edge weights are:

[0057] ,

[0058] in, Represents Gaussian nodes With Gaussian nodes The graph edge weights between them This represents the Euclidean distance between two Gaussian centers. and Representing nodes respectively and nodes Corresponding to the center of a three-dimensional Gaussian, Represents the L2 norm;

[0059] Suppose that the graph Laplacian matrix consists of a degree matrix and an adjacency weight matrix, and that the nodes are rearranged according to the salient and insignificant sets:

[0060] ,

[0061] in, Represents the graph Laplace matrix. This represents the submatrix from salient nodes to salient nodes. This represents the submatrix from significant nodes to insignificant nodes. This represents the submatrix from non-significant nodes to significant nodes. This represents a submatrix from non-significant nodes to non-significant nodes. Represents the thermal harmonic coefficients of all nodes. Indicates the thermal harmonic coefficients of a significantly Gaussian sphere. Let represent the insignificant Gaussian thermosphere harmonic coefficients to be determined; by minimizing the graphical Dirichlet energy and fixing the significant Gaussian boundary conditions, we obtain:

[0062] ,

[0063] ,

[0064] in, Represents the Laplacian submatrix of a non-significant node subgraph. This represents the non-significant nodal thermosphere harmonic coefficient to be determined. This represents the coupling submatrix between non-significant nodes and significant nodes. Represents the known significant node thermal harmonic coefficients; express The inverse matrix; this process makes spatially similar and semantically similar three-dimensional Gaussians have smooth and consistent thermal radiation functions.

[0065] Furthermore, in S7, such as Figure 5 As shown, the joint loss for rapid refinement is:

[0066] ,

[0067] in, This represents the joint loss during the rapid refinement phase. This represents the RGB self-consistency loss caused by the pseudo-RGB target constraint. This represents the thermal infrared reconstruction loss generated under supervision by real thermal infrared images; and It is formed by combining L1 loss and structural similarity loss.

[0068] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method of the present invention.

[0069] The beneficial effects of this invention are:

[0070] 1) Decouple RGB geometry training from thermal infrared appearance update. The RGB 3D Gaussian geometry skeleton only needs to be trained once. Subsequent thermal infrared recapture mainly updates thermal properties, which significantly reduces the time for repeated thermal field modeling.

[0071] 2) Using m=0 spherical harmonic thermal radiation parameterization based on the normal conforms to the physical law that thermal radiation is mainly related to the incident angle, and is more stable than directly learning thermal colors independently for each viewpoint under supervision of fewer viewpoints;

[0072] 3) Visibility-aware regularized closed-form fitting avoids long-term end-to-end optimization of all parameters and can adaptively adjust the higher-order spherical harmonics of the underconstrained Gaussian based on the number of visible viewpoints.

[0073] 4) By using RGB semantic / appearance clustering and KNN graph thermal propagation, the thermal sphere harmonic coefficients are supplemented in areas not fully observed by a small number of thermal infrared views, thereby improving thermal field integrity and reducing porosity;

[0074] 5) In the rapid refinement stage, RGB self-consistency is used to suppress overfitting of thermal infrared in few-viewpoint thermal images, so that the thermal rendering results can maintain the temperature structure while having better geometric alignment and stability of new viewpoints. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0076] Figure 2 This is a schematic diagram of the three-dimensional Gaussian local coordinates and thermosphere harmonic representation in this invention.

[0077] Figure 3 This is a schematic diagram of the visibility-sensing spherical harmonic fitting process in this invention.

[0078] Figure 4 This is a schematic diagram of the semantic clustering and KNN graph heat propagation process in this invention.

[0079] Figure 5 This is a schematic diagram of the rapid refinement process for RGB self-consistency in this invention. Detailed Implementation

[0080] The technical solution of the present invention will be described below with reference to the accompanying drawings and embodiments.

[0081] Example:

[0082] This embodiment uses the TSplatter thermal infrared 3D reconstruction workflow as an example. The input includes multi-view RGB images of the target scene and a small number of thermal infrared images. The output is a 3D Gaussian set with thermal infrared spherical harmonic properties, which can be used for new-view thermal infrared image synthesis and 3D thermal radiation field analysis. Figure 1 As shown, it mainly includes:

[0083] RGB 3D Gaussian geometry skeleton training:

[0084] First, a standard 3D Gaussian splash model is trained using multi-view RGB images of the target scene. Each 3D Gaussian model includes at least a center, scale, rotation, transparency, and RGB spherical harmonic color. To ensure stable local surface orientation in subsequent thermal infrared radiation modeling, this embodiment approximates the shortest scale axis of the 3D Gaussian model as its normal.

[0085] During the RGB training phase, the Gaussian normal of each pixel can be rendered and supervised by the monocular normal estimation results. Simultaneously, a scale regularization term is added to encourage a locally patchy distribution of Gaussians. The training objective can be expressed as:

[0086]

[0087] in, This represents the total loss during RGB geometric skeleton training. Indicates the RGB image reconstruction loss. This represents the scaling regularization term. Indicates the normal supervision weight, This represents the difference between the rendered normals and the monocular normal estimation results. In one implementation, the RGB 3D Gaussian is trained for 30k iterations. Set the value to 0.1; this training is performed only once for each scenario.

[0088] Multimodal 3D Gaussian representation:

[0089] Based on the RGB geometric skeleton, this invention adds thermal infrared spherical harmonic properties to each three-dimensional Gaussian to obtain a multimodal three-dimensional Gaussian representation:

[0090]

[0091] In equation (2), represents the i-th multimodal 3D Gaussian, represents the Gaussian center, represents the scale, represents the rotation, represents the transparency, represents the RGB spherical harmonic color, and represents the thermal infrared spherical harmonic property. Figure 2 As shown, for a 3D Gaussian and thermal infrared camera, let the unit line-of-sight direction from the camera to the center of the Gaussian be , and let the Gaussian normal be . Then the cosine of the incident angle is defined as follows, and its value ranges from 0 to 1:

[0092]

[0093] In equation (3), Indicates thermal infrared camera Observing a 3D Gaussian cosine of the angle of incidence at time Represents a three-dimensional Gaussian unit normal, Indicates camera Pointing to a three-dimensional Gaussian The unit's line of sight direction, This represents the transpose of a vector. This indicates taking the absolute value.

[0094] Thermal infrared intensity can be expressed as a real spherical harmonic expansion with respect to the cosine of the incident angle, term m=0:

[0095]

[0096] In equation (4), Represents the three-dimensional Gaussian cosine at the incident angle The thermal infrared intensity below, This represents summation over spherical harmonic orders. Indicates the index of the spherical harmonic order. Represents the order of the maximum spherical harmonic. Represents a three-dimensional Gaussian In order The unknown thermal harmonic coefficients on the surface express Real spherical harmonic basis functions. In one implementation... Choose 4. Since the thermal infrared radiation of most surfaces mainly depends on the angle between the line of sight and the normal, and weakly depends on the azimuth angle in the tangent plane, adopt... This method can reduce the number of parameters to be estimated and improve stability under conditions with few viewing angles.

[0097] Per-view thermal intensity estimation:

[0098] Given C newly acquired thermal infrared images, this invention does not re-optimize the complete geometry, but instead first estimates the view-by-view thermal intensity of each 3D Gaussian at each thermal infrared viewpoint. For pixel p in thermal infrared viewpoint c, let K be the number of Gaussians covering the pixel, and let the k-th Gaussian have a corresponding rendering weight and view-by-view thermal intensity. Then, the predicted thermal intensity is:

[0099]

[0100] In equation (5), Indicates thermal infrared perspective medium pixel The predicted thermal intensity This represents the summation of three-dimensional Gaussian regions covering the pixel. Indicates the Gaussian index that participated in the rendering. Indicates the covered pixels Gaussian quantity, Indicates the first A Gaussian perspective medium pixel Rendering weights Indicates the first Gauss in thermal infrared perspective The thermal intensity at each angle.

[0101] Per-view thermal intensity is obtained by minimizing the following formula:

[0102]

[0103] In equation (6), This indicates the estimated loss based on the heat intensity per angle. Indicates the number of thermal infrared images. This indicates the number of pixels in each thermal infrared image. Indicates the thermal infrared field of view index. Indicates pixel index, Indicates the predicted heat intensity. Indicates the observed thermal infrared intensity. This represents the squared norm 2. The optimization process implicitly computes the sparse linear system through 3D Gaussian differentiable rendering, without explicitly constructing a huge pixel-Gaussian matrix. In one implementation, a maximum of 20 thermal infrared images are used for each thermal infrared recapture, with 25 iterations of view-by-view thermal intensity optimization.

[0104] Visibility-sensing spherical harmonic closed-form fitting:

[0105] like Figure 3 As shown, the view-by-view thermal intensity only describes the thermal appearance at the observed viewpoint. To obtain a three-dimensional thermal radiation field that can be rendered from any new viewpoint, this invention fits the view-by-view thermal intensity to the spherical harmonic coefficients shown in equation (4). Define the visibility mask of the three-dimensional Gaussian g in viewpoint c, and construct the design matrix:

[0106]

[0107] In equation (7), Represents a three-dimensional Gaussian From the perspective Harmony order Design matrix elements at the location, Represents a three-dimensional Gaussian From the perspective Visibility mask in express real spherical harmonic basis functions, This represents the cosine of the corresponding incident angle.

[0108] To avoid high-order oscillations from underconstrained Gaussians, this invention constructs regularization terms related to the number of visible viewpoints and the spherical harmonic order:

[0109]

[0110]

[0111] In equation (8), Represents a three-dimensional Gaussian Visibility adaptive regularization strength, This indicates the number of visible viewpoints for the Gaussian. Represents the visibility smoothing parameter; in equation (9), Represents a three-dimensional Gaussian In spherical harmonic order Regularity coefficients on Represents the basic regularity coefficient. Indicates the order penalty exponent, The visibility regularization intensity obtained by expression (8) This represents the minimum regularity coefficient.

[0112] Let the regularization matrix be a diagonal matrix composed of regularization coefficients of each order. Solve the following regularized least squares problem independently for each three-dimensional Gaussian:

[0113]

[0114] In equation (10), Represented by the thermosphere harmonic coefficient Minimize the variables. This represents the design matrix formed by equation (7). Represents a three-dimensional Gaussian The unknown thermosphere harmonic coefficient matrix This represents a matrix that stacks the heat intensity per view across all visible viewpoints. Denotes the square of the Frobenius norm. Represents the trace of a matrix. Indicates matrix transpose. Indicates by The resulting diagonal regular matrix.

[0115] This problem has a closed-ended solution:

[0116]

[0117] In equation (11), This represents the thermal harmonic coefficient obtained by closed-form calculation. Design matrix transpose, Represents the data terms in the normal equation. Represents a regular matrix. This represents finding the inverse of a matrix. Let represent the thermal intensity matrix per view. Since the maximum spherical harmonic order is relatively small, equation (11) can be solved in parallel on all three-dimensional Gaussians. In one implementation, Take 20, Take 0.01, Take 2.0, Take 0.000001.

[0118] Semantic clustering and KNN graph heat propagation (e.g.) Figure 4 (as shown)

[0119] Limited-view thermal infrared images typically cannot cover the entire scene, therefore some 3D Gaussians are weakly visible or invisible in the thermal infrared view. To compensate for the thermal spherical harmonic coefficients of these underconstrained Gaussians, this invention first calculates the saliency score:

[0120]

[0121] In equation (12), Indicates the first A three-dimensional Gaussian The significance score, Indicates all thermal infrared views and pixels Take the maximum value above. Represents a three-dimensional Gaussian From the perspective medium pixel Contribution to transparency This indicates the corresponding transmittance term. This represents the rendering contribution of a 3D Gaussian vector to a specified pixel within a given viewpoint. If... If the value is greater than a preset threshold, the Gaussian is considered a significant 3D Gaussian; otherwise, it is considered a non-significant 3D Gaussian. In one implementation, the threshold is set to 0.19.

[0122] To avoid the erroneous propagation of thermal properties between different objects or materials, this invention extracts pixel-level semantic or visual features from RGB images and enhances the two-dimensional features into three-dimensional Gaussian features based on the rendering contribution of three-dimensional Gaussians to the pixels.

[0123]

[0124]

[0125] In equation (13), This represents the three-dimensional Gaussian features before normalization. Indicates Gaussian index, This indicates a weighted summation. Represents the RGB image index. This indicates the number of RGB images involved in feature enhancement. Indicates pixel index, Represents a three-dimensional Gaussian For the first Pixels in an RGB image Rendering contribution weight Indicates the first Pixels in an RGB image The two-dimensional semantic or visual features; in equation (14), This represents the normalized 3D Gaussian feature. express The vector norm is then used to form Gaussian clusters based on feature similarity, and a KNN graph is constructed within each cluster.

[0126] For two adjacent nodes i and j within the same Gaussian variety, the edge weights can be:

[0127]

[0128] In equation (15), Represents Gaussian nodes With Gaussian nodes The graph edge weights between them This represents the Euclidean distance between two Gaussian centers. and Representing nodes respectively and nodes Corresponding to the center of a three-dimensional Gaussian, This represents the L2 norm.

[0129] Suppose that the graph Laplacian matrix consists of a degree matrix and an adjacency weight matrix, and that the nodes are rearranged according to the salient and insignificant sets:

[0130]

[0131] In equation (16), Represents the graph Laplace matrix. This represents the submatrix from salient nodes to salient nodes. This represents the submatrix from significant nodes to insignificant nodes. This represents the submatrix from non-significant nodes to significant nodes. This represents a submatrix from non-significant nodes to non-significant nodes. Represents the thermal harmonic coefficients of all nodes. Indicates the thermal harmonic coefficients of a significantly Gaussian sphere. Let represent the insignificant Gaussian thermal sphere harmonic coefficients to be determined. By minimizing the graphical Dirichlet energy and fixing the significant Gaussian boundary conditions, we can obtain:

[0132]

[0133]

[0134] In equation (17), Represents the Laplacian submatrix of a non-significant node subgraph. This represents the non-significant nodal thermosphere harmonic coefficient to be determined. This represents the coupling submatrix between non-significant nodes and significant nodes. Represents the known significant node thermal harmonic coefficients; in equation (18), express The inverse matrix. This process gives spatially similar and semantically similar three-dimensional Gaussians a smooth and consistent thermal radiation function. In one implementation, the KNN graph... Set the threshold to 50 for feature clustering similarity. Take 0.81.

[0135] RGB self-consistency rapid refinement:

[0136] like Figure 5As shown, after spherical harmonic fitting and graph propagation, the 3D Gaussian model possesses complete thermal infrared properties. To correct residual errors under limited viewpoint conditions, this invention performs short-iteration refinement. For each thermal infrared camera pose, a pseudo-RGB target is rendered using a pre-trained RGB 3D Gaussian model; the current multimodal 3D Gaussian model is simultaneously supervised by both the pseudo-RGB target and the real thermal infrared image.

[0137]

[0138] In equation (19), This represents the joint loss during the rapid refinement phase. This represents the RGB self-consistency loss caused by the pseudo-RGB target constraint. This represents the thermal infrared reconstruction loss generated by supervising real thermal infrared images. and Both can be formed by combining L1 loss and structural similarity loss. RGB self-consistency constraints prevent the model from destroying its geometry in order to fit a small number of thermal infrared images; thermal infrared loss directly corrects the temperature distribution of the observed area. In one implementation, the number of refinement iterations is 50 times the number of thermal infrared training images.

[0139] Application scenarios and implementation results:

[0140] This invention is applicable to scenarios where the geometry is relatively stable but the thermal state changes over time, such as thermal diagnostics of building envelopes, inspection of industrial furnaces or pipelines, temperature rise monitoring of storage and electrical equipment, and nighttime environmental perception for mobile robots. The system can train an RGB 3D Gaussian geometric skeleton during initial deployment, and thereafter quickly update the 3D thermal radiation field by collecting only a small number of thermal infrared images each time.

[0141] On real RGB-thermal infrared multi-view data, this invention can still obtain stable thermal infrared new view rendering results even with small thermal infrared view ratios of 5%, 10%, and 20%. Under 10% thermal infrared training data conditions, compared to existing methods that require hundreds to thousands of seconds of optimization, this invention can complete each thermal field update in tens of seconds. The above values ​​are illustrative of the examples and do not constitute a limitation on the scope of protection.

[0142] The above are merely preferred embodiments of the present invention. Those skilled in the art can, without departing from the principle of the present invention, replace the thermal infrared features with other long-wave infrared, near-infrared, or multispectral properties, replace the spherical harmonic basis functions with other angular basis functions, and replace the KNN graph solution with conjugate gradients, sparse Cholesky, or neural propagation modules. All such variations should fall within the protection scope of the present invention.

Claims

1. A method for reconstructing a few-view thermal radiation field based on RGB-guided three-dimensional Gaussian radiation, characterized in that, Includes the following steps: S1. Acquire multi-view RGB images of the target scene; S2. Train an RGB 3D Gaussian model based on the acquired RGB images to obtain a 3D Gaussian geometric skeleton containing Gaussian center, scale, rotation, transparency, RGB spherical harmonic color and normal information. S3. During the thermal infrared recapture stage, only a small number of thermal infrared images and corresponding camera poses are acquired. S4. Based on three-dimensional Gaussian differentiable rendering, the thermal infrared pixel is represented as a weighted sum of the thermal intensities of multiple three-dimensional Gaussians covering the pixel. The view-by-view thermal intensity of each three-dimensional Gaussian under each thermal infrared view is estimated through gradient optimization. S5. Using the three-dimensional Gaussian normal as the local coordinate system z-axis, calculate the cosine of the incident angle between the thermal infrared camera's line of sight and the normal, and transform the thermal intensity parameter into a spherical harmonic function with respect to this incident angle cosine. Based on the visibility of the three-dimensional Gaussian in the thermal infrared view, construct a design matrix, introduce an adaptive regularization term related to the number of visible viewpoints and the spherical harmonic order, and independently solve the closed-form solution of the visibility perception spherical harmonic fitting for each three-dimensional Gaussian to obtain the initial thermal spherical harmonic coefficients. S6. Cluster the three-dimensional Gaussian based on the semantic or visual features extracted from the RGB image, construct a KNN graph within each cluster, and use graph Laplacian smoothing to propagate the thermal sphere harmonic coefficients of the significant three-dimensional Gaussian in the visible region to the invisible or weakly visible non-significant three-dimensional Gaussian, thus completing the thermal radiation field. S7. Using the pseudo-RGB image of pre-trained RGB 3D Gaussian rendering as the self-consistency target, combined with thermal infrared observation images as supervision, the geometry and thermal appearance are rapidly refined with a small number of iterations, and the complete 3D thermal radiation field and new perspective thermal infrared rendering results are output.

2. The method for reconstructing a few-view thermal radiation field based on RGB-guided three-dimensional Gaussian radiation according to claim 1, characterized in that, In S2, the total loss function used to train the RGB 3D Gaussian model is: ， in, Indicates the RGB image reconstruction loss. This represents the scaling regularization term. Indicates the normal supervision weight, This indicates the difference between the rendered normals and the monocular normal estimation results; After training, the shortest scale axis of the three-dimensional Gaussian is approximated as the normal of the Gaussian to obtain the normal information; Based on the obtained three-dimensional Gaussian geometric skeleton, thermal infrared spherical harmonic properties are added to each three-dimensional Gaussian to obtain a multimodal three-dimensional Gaussian representation: ， in, Indicates the first A multimodal three-dimensional Gaussian, Indicates the center of Gauss. Indicates scale. Indicates rotation, Indicates transparency. Represents RGB spherical harmonic colors. This indicates the thermal infrared spherical harmonic properties.

3. The method for reconstructing a few-view thermal radiation field based on RGB-guided three-dimensional Gaussian radiation according to claim 2, characterized in that, In S4, C thermal infrared images were acquired. For pixel p in thermal infrared viewpoint c, let K be the number of Gaussians covering the pixel. The k-th Gaussian has a corresponding rendering weight and per-view thermal intensity. Then the predicted thermal intensity is: ， in, Indicates thermal infrared perspective medium pixel The predicted thermal intensity This represents the summation of three-dimensional Gaussian regions covering the pixel. Indicates the Gaussian index that participated in the rendering. Indicates the covered pixels Gaussian quantity, Indicates the first A Gaussian perspective medium pixel Rendering weights Indicates the first Gauss in thermal infrared perspective Thermal intensity per angle; Per-view thermal intensity is obtained by minimizing the following formula: ， in, This indicates the estimated loss based on the heat intensity per angle. Indicates the number of thermal infrared images. This indicates the number of pixels in each thermal infrared image. Indicates the thermal infrared field of view index. Indicates pixel index, Indicates the predicted heat intensity. Indicates the observed thermal infrared intensity. This represents the square norm 2.

4. The method for reconstructing a few-view thermal radiation field based on RGB-guided three-dimensional Gaussian radiation according to claim 3, characterized in that, In S5, for three-dimensional Gaussian... and thermal infrared camera Let the unit line of sight from the camera to the center of Gauss be denoted as . Let the Gaussian normal be... The cosine of the incident angle is defined as follows, and its value ranges from 0 to 1: ， in, Indicates thermal infrared camera Observing a 3D Gaussian cosine of the angle of incidence at time Represents a three-dimensional Gaussian unit normal, Indicates camera Pointing to a three-dimensional Gaussian The unit's line of sight direction, This represents the transpose of a vector. Indicates taking the absolute value; Express the thermal infrared intensity as a real spherical harmonic expansion with respect to the cosine of the incident angle at m=0: ， in, Represents the three-dimensional Gaussian cosine at the incident angle The thermal infrared intensity below, This represents summation over spherical harmonic orders. Indicates the index of the spherical harmonic order. Represents the order of the maximum spherical harmonic. Represents a three-dimensional Gaussian In order The unknown thermal harmonic coefficients on the surface express Real spherical harmonic basis functions; The method for constructing a design matrix based on the visibility of a 3D Gaussian in a thermal infrared viewpoint is to define a visibility mask for the 3D Gaussian g in viewpoint c and then construct the design matrix: ， in, Represents a three-dimensional Gaussian From the perspective Harmony order Design matrix elements at the location, Represents a three-dimensional Gaussian From the perspective Visibility mask in express real spherical harmonic basis functions, Represents the cosine of the corresponding incident angle; An adaptive regularization term related to the number of visible viewpoints and the spherical harmonic order is introduced as follows: ， ， in, Represents a three-dimensional Gaussian Visibility adaptive regularization strength, This indicates the number of visible viewpoints for the Gaussian. Indicates the visibility smoothing parameter; Represents a three-dimensional Gaussian In spherical harmonic order Regularity coefficients on Represents the basic regularity coefficient. Indicates the order penalty exponent. The visibility regularity intensity obtained by expression (8) Represents the minimum regularity coefficient; Let the regularization matrix be a diagonal matrix composed of regularization coefficients of each order. Solve the following regularized least squares problem independently for each three-dimensional Gaussian: ， in, Represented by the thermosphere harmonic coefficient Minimize the variables. This represents the design matrix formed by equation (7). Represents a three-dimensional Gaussian The unknown thermosphere harmonic coefficient matrix This represents a matrix that stacks the heat intensity per view across all visible viewpoints. Denotes the square of the Frobenius norm. Represents the trace of a matrix. Indicates matrix transpose. Indicates by The resulting diagonal regular matrix; the closed-form solution is: ， in, This represents the thermal harmonic coefficient obtained by closed-form calculation. Design matrix transpose, Represents the data terms in the normal equation. Represents a regular matrix. This represents finding the inverse of a matrix. This represents the thermal intensity matrix per view.

5. The method for reconstructing a few-view thermal radiation field based on RGB-guided three-dimensional Gaussian radiation according to claim 4, characterized in that, The specific method for S6 is as follows: First, calculate the significance score: ， in, Indicates the first A three-dimensional Gaussian The significance score, Indicates all thermal infrared views and pixels Take the maximum value above. Represents a three-dimensional Gaussian From the perspective medium pixel Contribution to transparency This indicates the corresponding transmittance term; This represents the rendering contribution of a 3D Gaussian to a specified pixel within a given viewpoint. If the value is greater than a preset threshold, the Gaussian is considered a significant 3D Gaussian; otherwise, it is considered a non-significant 3D Gaussian. Extract pixel-level semantic or visual features from RGB images and enhance the two-dimensional features into three-dimensional Gaussian features based on the rendering contribution of the pixels by three-dimensional Gaussians: ， ， in, This represents the three-dimensional Gaussian features before normalization. Indicates Gaussian index, This indicates a weighted summation. Represents the RGB image index. This indicates the number of RGB images involved in feature enhancement. Indicates pixel index, Represents a three-dimensional Gaussian For the Pixels in an RGB image Rendering contribution weight Indicates the first Pixels in an RGB image Two-dimensional semantic or visual features; This represents the normalized 3D Gaussian feature. express The vector norm is then used to form Gaussian clusters based on feature similarity, and a KNN graph is constructed within each cluster. For two adjacent nodes i and j within the same Gaussian variety, the edge weights are: ， in, Represents Gaussian nodes With Gaussian nodes The graph edge weights between them This represents the Euclidean distance between two Gaussian centers. and Representing nodes respectively and nodes Corresponding to the center of a three-dimensional Gaussian, Represents the L2 norm; Suppose that the graph Laplacian matrix consists of a degree matrix and an adjacency weight matrix, and that the nodes are rearranged according to the salient and insignificant sets: ， in, Represents the graph Laplace matrix. This represents the submatrix from salient nodes to salient nodes. This represents the submatrix from significant nodes to insignificant nodes. This represents the submatrix from non-significant nodes to significant nodes. This represents a submatrix from non-significant nodes to non-significant nodes. Represents the thermal harmonic coefficients of all nodes. Indicates the thermal harmonic coefficients of a significantly Gaussian sphere. Let represent the insignificant Gaussian thermosphere harmonic coefficients to be determined; by minimizing the graphical Dirichlet energy and fixing the significant Gaussian boundary conditions, we obtain: ， ， in, Represents the Laplacian submatrix of a non-significant node subgraph. This represents the non-significant nodal thermosphere harmonic coefficient to be determined. This represents the coupling submatrix between non-significant nodes and significant nodes. Represents the known significant node thermal harmonic coefficients; express The inverse matrix.

6. The method for reconstructing a few-view thermal radiation field based on RGB-guided three-dimensional Gaussian radiation according to claim 5, characterized in that, In S7, the joint loss for rapid refinement is: ， in, This represents the joint loss during the rapid refinement phase. This represents the RGB self-consistency loss caused by the pseudo-RGB target constraint. This represents the thermal infrared reconstruction loss generated under supervision by real thermal infrared images; and It is formed by combining L1 loss and structural similarity loss.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1 to 7.

Images