Infrared image reconstruction method based on thermal radiation parameter driving 3D Gaussian sputtering

Abstract

The application provides an infrared image reconstruction method based on thermal radiation parameter-driven 3D Gaussian sputtering, and the implementation steps are as follows: obtaining a training sample set and a test sample set; constructing an infrared image reconstruction network model based on thermal radiation parameter-driven 3D Gaussian sputtering; iteratively training the infrared image reconstruction network model; and obtaining an infrared image reconstruction result. The rendering module in the application performs 3D Gaussian sputtering on 3D Gaussian primitives by calculating thermal radiation parameters including band-pass emission, atmospheric attenuation coefficient and infrared opacity of each 3D Gaussian primitive based on geometric parameters and physical parameters of the 3D Gaussian primitive, and the temperature and emissivity in the physical parameters enable each 3D Gaussian primitive to have an interpretable thermal radiation attribute, thereby avoiding the defect that the Gaussian primitive is difficult to accurately represent the thermal radiation attribute in the infrared scene in the prior art only by considering the geometric parameters, improving the representation capability of the thermal radiation transmission law and the accuracy of the luminance distribution of the reconstructed image, and effectively improving the authenticity of the infrared image reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to an infrared image reconstruction method, specifically an infrared image reconstruction method based on thermal radiation parameters driven by 3D Gaussian sputtering, which can be applied to fields such as industrial inspection, agricultural inspection, medical health, and building diagnosis. Background Technology

[0002] Infrared image reconstruction refers to the process of recovering the infrared brightness distribution of a target scene from different viewpoints using observational images acquired by infrared thermal imaging equipment. 3D Gaussian sputtering is a technique based on explicit 3D Gaussian primitive representation, combined with real-time 3D reconstruction and novel viewpoint synthesis using a differentially rasterizable method. Its core principle is to use a large number of 3D Gaussian primitives as basic modeling units to represent a 3D scene. Each 3D Gaussian primitive is defined by parameters such as center coordinates, 3D covariance matrix, color, and opacity. Through the center coordinates, 3D covariance matrix, color, and opacity parameters of each 3D Gaussian primitive, rendering including 3D Gaussian sputtering and nonlinear mapping is performed. The attributes of the 3D Gaussian primitives are projected onto a 2D image plane and pixel-level fusion is completed, enabling the reconstruction of images from any viewpoint.

[0003] In recent years, 3D Gaussian sputtering has achieved breakthroughs in visible light new perspective synthesis and 3D reconstruction, and has been rapidly transferred to infrared scene reconstruction tasks. For example, Xi'an University of Electronic Science and Technology disclosed an infrared scene reconstruction method based on 3D Gaussian sputtering in its patent document "Infrared Scene Reconstruction and Rendering Method and Device Based on 3D Gaussian Sputtering" (Patent Application No.: CN202510826547.5, Publication No.: CN120765839A). This invention uses the SfM algorithm to process preprocessed infrared image data to obtain sparse 3D point clouds and camera pose, and sets a Gaussian ellipsoid at the center of the sparse 3D point cloud to construct a 3D Gaussian point cloud model; the camera pose is then input... A Gaussian differentiable rasterizer renderer is used to render a 3D Gaussian point cloud model, obtaining a 2D image from the corresponding viewpoint. The model is then trained by comparing the 2D image with the infrared image data of the camera pose using a loss function, resulting in a trained 3D Gaussian point cloud model. The loss function includes the temperature gradient field, the projection position of the Gaussian ellipsoid onto the pixel plane under the current camera viewpoint, and the projection direction of the principal axis. The trained 3D Gaussian point cloud model is then processed using a 3D graphics engine and a 3D Gaussian rasterizer renderer to obtain an infrared simulation image, resulting in a high degree of realism in the reconstruction and rendering of the infrared scene. However, since its Gaussian primitive parameter modeling still mainly follows the modeling approach of visible light 3D Gaussian sputtering, and does not explicitly construct physical parameters such as temperature and emissivity for infrared imaging characteristics, the Gaussian primitive is difficult to accurately characterize the thermal radiation properties in the infrared scene. At the same time, the rendering process mainly determines the projection position and weight of the Gaussian primitive on the image plane based on the center coordinates and 3D covariance matrix, and combines color parameters and opacity parameters to realize the sputtering of 3D Gaussian primitives. It does not consider the influence of thermal radiation parameters, including bandpass emission, atmospheric attenuation coefficient and infrared opacity, on the contribution of 3D Gaussian sputtering radiation, resulting in poor realism of infrared image reconstruction. Summary of the Invention

[0004] The purpose of this invention is to overcome the defects of the prior art and propose an infrared image reconstruction method based on thermal radiation parameters driven by 3D Gaussian sputtering, which solves the technical problem of poor realism of infrared reconstructed images in the prior art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0006] (1) Obtain the training sample set and the test sample set:

[0007] For different scenes captured by thermal imaging cameras, and each scene has multiple perspectives The infrared images are preprocessed, and a total of each scene is included. The training sample set is composed of 10 infrared images used as labels for the corresponding preprocessed infrared images. The remaining preprocessed infrared images are then used as labels. The test sample set consists of 10 infrared images, among which ;

[0008] (2) Construct an infrared image reconstruction network model based on thermal radiation parameters driven by 3D Gaussian sputtering:

[0009] An infrared image reconstruction network model is constructed, comprising a cascaded sparse point cloud and camera parameter acquisition module, a 3D Gaussian meta-parameter generation module, and a rendering module driven by thermal radiation parameters for 3D Gaussian sputtering. The output of the sparse point cloud and camera parameter acquisition module is connected to the input of the rendering module. The 3D Gaussian primitive parameter generation module generates geometric parameters, including the 3D covariance matrix of the center coordinates, and physical parameters, including temperature and emissivity, for each 3D Gaussian primitive. The thermal radiation parameter-driven 3D Gaussian sputtering rendering module renders the 3D Gaussian primitive using thermal radiation parameters, including bandpass emission, atmospheric attenuation coefficient, and infrared opacity, calculated from the geometric and physical parameters of each 3D Gaussian primitive.

[0010] (3) Iteratively train the infrared image reconstruction network model:

[0011] The infrared image reconstruction network model was trained using a sample set. Perform iterative training to obtain a well-trained reconstruction network model. ;

[0012] (4) Obtain the infrared image reconstruction results:

[0013] The test sample set is used as the trained reconstruction network model. The input is propagated forward to obtain A series of infrared reconstructed images.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] The rendering module in this invention performs 3D Gaussian sputtering on 3D Gaussian elements by calculating thermal radiation parameters, including bandpass emission, atmospheric attenuation coefficient, and infrared opacity, based on the geometric and physical parameters of each 3D Gaussian element. The temperature and emissivity in the physical parameters give each 3D Gaussian element interpretable thermal radiation properties. This avoids the shortcomings of existing technologies that only consider geometric parameters, which make it difficult for Gaussian elements to accurately represent the thermal radiation properties in infrared scenes. It improves the ability to represent the thermal radiation transmission law and the accuracy of the reconstructed image brightness distribution, effectively improving the realism of infrared image reconstruction. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0017] Figure 2 This is a schematic diagram of the infrared image reconstruction network model in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0019] Reference Figure 1 The present invention includes the following steps:

[0020] Step 1) Obtain the training sample set and the test sample set:

[0021] For different scenes captured by thermal imaging cameras, and each scene has multiple perspectives The infrared images are preprocessed, and a total of each scene is included. The training sample set is composed of 10 infrared images used as labels for the corresponding preprocessed infrared images. The remaining preprocessed infrared images are then used as labels. The test sample set consists of 10 infrared images. In this embodiment ;

[0022] The specific steps of image preprocessing are as follows: Median filtering is applied to each infrared image. This aims to suppress isolated bright spots, dark spots, and bad pixel noise by selecting a local window centered on the target pixel and replacing the target pixel's grayscale with the median grayscale value within the window. Non-uniform correction is then applied to the median-filtered infrared image to reduce fixed-pattern noise caused by differences in the responses of different detection units through pixel-level gain and bias correction. Finally, the non-uniformly corrected image is normalized to map the infrared image grayscale to a uniform numerical range, thereby improving the numerical consistency between different samples. Through this preprocessing, noise interference can be reduced while preserving the target thermal boundary and temperature gradient, making subsequent network training more stable.

[0023] Step 2) Construct an infrared image reconstruction network model based on thermal radiation parameters driven by 3D Gaussian sputtering. Its structure is as follows Figure 2 As shown:

[0024] An infrared image reconstruction network model is constructed, comprising a cascaded sparse point cloud and camera parameter acquisition module, a 3D Gaussian meta-parameter generation module, and a rendering module driven by thermal radiation parameters for 3D Gaussian sputtering. Among them, the 3D Gaussian primitive parameter generation module is used to generate the geometric and physical parameters of each 3D Gaussian primitive; the thermal radiation parameter driven 3D Gaussian sputtering rendering module is used to render the 3D Gaussian primitive by calculating thermal radiation parameters, including bandpass emission, atmospheric attenuation coefficient and infrared opacity, based on the geometric and physical parameters of each 3D Gaussian primitive, including 3D Gaussian sputtering and nonlinear mapping.

[0025] The rendering module includes a cascaded thermal radiation parameter calculation unit and a rendering unit; the rendering unit includes a cascaded radiation synthesis unit, an MLP unit, and an image reconstruction unit.

[0026] Step 3) Iteratively train the infrared image reconstruction network model:

[0027] (3a) Initialize the number of iterations Maximum number of iterations Reconstructing the network model The weights and biases are respectively and and order In this embodiment ;

[0028] (3b) The sparse point cloud and camera parameter acquisition module extracts the parameters of each training sample from the sparse point cloud and camera parameter acquisition module. A sparse point cloud composed of three-dimensional points in space and camera parameters; a 3D Gaussian meta-parameter generation module maps the sparse point cloud to obtain... Each 3D Gaussian element is assigned a coordinate system for a 3D point in space. As its corresponding 3D Gaussian element's center coordinates, then and the 3D covariance matrix of each 3D Gaussian element Geometric parameters that make up each 3D Gaussian element Simultaneously, physical parameters including temperature and emissivity are generated for each 3D Gaussian element through learnable parameters from infrared thermal radiation imaging. ;

[0029] (3b1) The sparse point cloud and camera parameter acquisition module extracts the sparse point cloud and camera parameters corresponding to each training sample. Since 3D Gaussian sputtering requires projecting Gaussian primitives in three-dimensional space onto a two-dimensional infrared image plane, if the camera intrinsic and extrinsic parameters are missing, it is impossible to determine the projection position, projection scale, and depth order of each Gaussian primitive under different viewpoints.

[0030] For the Each viewpoint, rotation matrix in camera parameters Translation vector The calculation formula is:

[0031] ;

[0032] ;

[0033] in, For the first The pose unit quaternion of the i-th viewpoint camera, representing the i-th viewpoint camera. The rotation attitude of the camera coordinate system relative to the world coordinate system describes the direction of the camera's optical axis and the rotation relationship of the three coordinate axes of the camera coordinate system relative to the world coordinate system. For the real part, , , It is an imaginary part and satisfies the unit constraint: , , , Representing the origin of the world coordinate system to the 1st point, respectively. The translation distance of the camera coordinate system in the three orthogonal directions of the X, Y, and Z axes. , This indicates the number of infrared image viewpoints captured in the same scene. and Used to transform three-dimensional points in the world coordinate system to the next... In a camera coordinate system with different viewpoints, the position, depth, and projection direction of 3D Gaussian elements can be determined at different viewpoints.

[0034] For the Each viewpoint, intrinsic parameter matrix in camera parameters for:

[0035] ;

[0036] in, and They represent the first The focal length of a camera in the horizontal and vertical directions in the image pixel coordinate system. and They represent the first The pixel coordinates of the principal point of the camera at each viewpoint in the horizontal and vertical directions of the image pixel coordinate system. After the extrinsic parameters have transformed the three-dimensional points in space to the camera coordinate system, the intrinsic parameter matrix... This is used to further project a three-dimensional point in the camera coordinate system onto the two-dimensional pixel coordinate system of the infrared image, obtaining the value of the three-dimensional point in the [missing information - likely a specific pixel coordinate system]. The pixel positions in infrared images from various perspectives; through the combined effect of extrinsic and intrinsic parameters, the projection position, projection scale, and depth order of each 3D Gaussian element under different perspectives can be determined, and a foundation is provided for subsequent rendering of 3D Gaussian sputtering driven by thermal radiation parameters.

[0037] (3b2) Since sparse point clouds only provide discrete three-dimensional spatial positions, while 3D Gaussian sputtering requires the use of Gaussian primitives with spatial extension ranges to continuously represent the scene, this invention maps each spatial three-dimensional point in the sparse point cloud to an initial 3D Gaussian primitive, resulting in a total of Each 3D Gaussian element is assigned a coordinate system for a 3D point in space. The coordinates of the corresponding 3D Gaussian primitive are used as the center coordinates. Since the density of point clouds may vary in different regions, if all 3D Gaussian primitives adopt the same scale, holes are likely to appear in sparse areas of the point cloud, and excessive overlap is likely to occur in dense areas. This invention adaptively determines the geometric parameters of each 3D Gaussian primitive based on the spacing between local neighboring points. 3D covariance matrix :

[0038] ;

[0039] ;

[0040] in, For the first Coordinates of a three-dimensional point in space of A set of three-dimensional point coordinates in a neighborhood space. for The Middle Coordinates of a three-dimensional point in space. This represents the 2-norm operation. for and The average spatial distance between the coordinates of all three-dimensional points in the space. The above operations can match the initial coverage of the 3D Gaussian units with the local point cloud density, thereby improving the stability of the initial geometric representation and reducing holes, overlaps or floating artifacts caused by unreasonable scale in subsequent training.

[0041] (3b3) Since the brightness of infrared images is mainly determined by the target temperature and the emissivity of the material, the color parameters of traditional 3D Gaussian elements alone cannot explain the differences in infrared brightness of different materials at the same temperature, or the brightness changes of the same material due to environmental reflection. This makes it difficult for Gaussian elements to accurately characterize the thermal radiation properties in infrared scenes. This invention explicitly generates color parameters for each 3D Gaussian element based on temperature... and emissivity physical parameters of composition ;

[0042] Specifically, no. Temperature of a 3D Gaussian element The calculation formula is:

[0043] ;

[0044] The temperature is a learnable parameter for each 3D Gaussian element, used to learn the thermal state of the local surface corresponding to each 3D Gaussian element. The Gaussian primitive can be initialized based on initial grayscale observations in infrared images from multiple training perspectives, or it can be initialized based on the average scene temperature or a preset temperature constant. The minimum temperature threshold, Represent the natural logarithm function, through The function operation ensures that the temperature is positive and within a reasonable temperature range;

[0045] No. The emissivity of a 3D Gaussian element The calculation formula is:

[0046] ;

[0047] The emissivity of each 3D Gaussian element is a learnable parameter. Represents the natural logarithm function. This represents an exponential function with the natural constant as its base. It can be initialized based on the material emission prior or a preset constant, and this calculation formula constrains the emissivity in the (0,1) interval;

[0048] By constructing physical parameters of temperature and emissivity, each 3D Gaussian element has interpretable thermal radiation properties. Furthermore, thermal radiation parameters are calculated based on the geometric and physical parameters of the 3D Gaussian elements, so that the pixel brightness of the reconstructed image is generated by an interpretable thermal radiation process, thereby improving the realism of the infrared image reconstruction results.

[0049] (3c) The thermal radiation parameter calculation unit in the rendering module is through , Bandpass emission in the thermal radiation parameters of each 3D Gaussian element is calculated using camera parameters. Atmospheric attenuation coefficient and infrared opacity The rendering unit renders each 3D Gaussian primitive using thermal radiation parameters to obtain the infrared reconstructed image of each training sample.

[0050] (3c1) The thermal radiation parameter calculation unit calculates the bandpass emission of each 3D Gaussian element. Atmospheric attenuation coefficient and infrared opacity Since traditional 3D Gaussian sputtering typically reconstructs images based on the center coordinates of 3D Gaussian elements, the 3D covariance matrix, color, and opacity parameters, while the pixel grayscale of infrared images does not originate from visible light color but is related to the infrared thermal radiation imaging process, mainly affected by the spontaneous emission of the target itself, the reflection of ambient radiation from the target surface, atmospheric propagation attenuation, and the occlusion and emission absorption relationships between different Gaussian elements, the rendering module in this invention does not directly render colors. Instead, it first calculates the thermal radiation parameters related to the infrared thermal radiation imaging process and then uses the thermal radiation parameters to perform pixel-level radiation synthesis and nonlinear mapping on the 3D Gaussian elements.

[0051] (3c11) Calculate bandpass exit ;

[0052] Because infrared cameras can only receive signals within their operating wavelength band. The thermal radiation within the object is not all of the thermal radiation of the object across the entire wavelength range, so this invention first calculates the spontaneous radiation that matches the working wavelength of the infrared camera.

[0053] ;

[0054] in, For each 3D Gaussian element at temperature Spontaneous radiation under, is Planck's constant. At the speed of light, Boltzmann's constant, and These represent the maximum and minimum values ​​of the infrared camera's operating wavelength band, respectively. Since performing Planck integration directly in each training iteration increases computation and may introduce numerical instability, in a preferred embodiment, values ​​are pre-constructed based on the temperature range. The lookup table of its derivatives is used to obtain spontaneous emission at the corresponding temperature through interpolation during the training process;

[0055] Because real infrared images contain not only the thermal radiation generated by the object itself due to temperature, but also the reflected infrared radiation from the surrounding environment such as walls, sky, ground, and equipment, the impact of environmental reflection on image brightness is particularly significant for low emissivity materials. Therefore, this invention further calculates the surface normal of each 3D Gaussian element. Ambient radiance in the direction ;

[0056] ;

[0057] in, The normalization coefficient is... , , , These are learnable spherical harmonic coefficients. For the Gaussian element normal The spherical harmonic function; through the above operations, the smooth-changing global environmental infrared radiation in the scene can be represented with fewer parameters, so that 3D Gaussian elements with different orientations can receive environmental reflection contributions of different intensities, thereby improving the accuracy of infrared brightness modeling in low emissivity areas and areas affected by environmental reflection.

[0058] Since the infrared brightness exhibited by an object in an infrared scene is typically determined by both spontaneous emission and ambient radiance, and the ratio of these two factors is related to the material's emissivity, this invention, based on emissivity, [details about emissivity and its properties]. and Weighted fusion is performed to obtain the bandpass output of each 3D Gaussian element:

[0059]

[0060] Through the above operations, materials with high emissivity can have their infrared brightness determined primarily by their own temperature, while materials with low emissivity can reflect the influence of environmental radiance on infrared brightness. This transforms 3D Gaussian elements from traditional color carriers into thermal radiation elements with a coupling relationship between temperature, material, and environment.

[0061] (3c12) Calculate the atmospheric attenuation coefficient ;

[0062] Infrared radiation experiences energy loss during its propagation from the target surface to the camera due to factors such as air absorption, humidity, aerosols, and propagation distance. This is especially true in outdoor, long-distance, or drone-based shooting scenarios. This physical process is called atmospheric propagation attenuation. If atmospheric propagation attenuation is ignored, the infrared radiation contribution of distant 3D Gaussian pixels can easily be overestimated, leading to problems such as overly bright distant scenes or inconsistent brightness across viewing angles in the reconstructed image. Therefore, this invention calculates an atmospheric attenuation coefficient related to propagation distance for each 3D Gaussian pixel. , used to characterize the atmospheric propagation attenuation of infrared radiation during the propagation process from the target to the camera;

[0063] Specifically, first, the coordinates of the 3D Gaussian center are... Transform to the In a camera coordinate system:

[0064] ;

[0065] Calculate the coordinates of each 3D Gaussian primitive in the camera coordinate system; then calculate the distance from that 3D Gaussian primitive to the camera optical center. :

[0066] ;

[0067] Finally, the atmospheric attenuation coefficient was calculated using an exponential model. :

[0068] ;

[0069] The learnable atmospheric extinction coefficient is used to describe the degree of loss of infrared radiation due to atmospheric attenuation as it propagates from the 3D Gaussian element to the camera.

[0070] Through the above operations, the radiation intensity reaching the camera can be adaptively adjusted according to the distance between the Gaussian element and the camera, so that the infrared brightness distribution of near and far targets is more in line with the real thermal radiation propagation law, thereby improving the realism of infrared reconstruction of far-distance scenes.

[0071] (3c13) Calculate infrared opacity ;

[0072] Since opacity in infrared imaging not only represents geometric occlusion but also reflects the target material's absorption of infrared radiation, spontaneous emission, and equivalent optical thickness along the line of sight, directly using the general opacity from traditional 3D Gaussian sputtering can easily lead to blurred thermal boundaries, floating artifacts, and aliasing of radiation contributions from preceding and following Gaussian elements. Therefore, this invention constructs infrared opacity parameters;

[0073] Specifically, the center coordinates of each 3D Gaussian element are first calculated using the pinhole camera projection function at the [missing information - likely a specific point or time period]. Projected coordinates on the image plane at each viewpoint:

[0074] ;

[0075] in, It is the perspective projection function of a standard pinhole camera; then according to and two-dimensional covariance matrix Calculate the pixel position of each 3D Gaussian primitive. Projection weights at the location :

[0076] ;

[0077] in, Used to describe the projection range, projection direction, and pixel coverage of 3D Gaussian pixels on the infrared image plane. Used to measure the position of each 3D Gaussian primitive relative to the image pixel. The coverage level at the location is further improved, according to and the unit vector along the camera's viewpoint direction Calculate the quadratic curvature of each 3D Gaussian element along the camera's viewpoint direction. :

[0078] ;

[0079] in, The 3D shape and viewing direction of the 3D Gaussian elements are transformed into an effective thickness along the line of sight, and further based on... and scene-level equivalent extinction factor Calculate the extinction coefficient of each 3D Gaussian element. :

[0080] ;

[0081] in, Used to quantify the radiation absorption intensity of the Gaussian element along the camera's viewing angle, ultimately based on and Obtain the infrared opacity of each 3D Gaussian element. :

[0082] ;

[0083] Through the above operations It can simultaneously reflect the projection coverage of Gaussian elements on the image plane and the radiation absorption intensity along the line of sight, making the 3D Gaussian sputtering process more in line with the infrared emission-absorption imaging mechanism, thereby reducing boundary blurring, floating artifacts and radiation contribution aliasing.

[0084] (3c2) The rendering unit renders each 3D Gaussian primitive using thermal radiation parameters. The specific method is as follows:

[0085] The radiation synthesis unit in the (3c21) rendering unit uses... , and Perform 3D Gaussian sputtering on each 3D Gaussian pixel to obtain the pixel position of the 3D Gaussian pixel on the two-dimensional plane of the reconstructed image. Radiation contribution at the location ;

[0086] Since the same pixel location may be affected by radiation contributions from multiple 3D Gaussian elements, and Gaussian elements closer to the camera may obstruct or absorb radiation from Gaussian elements behind them, this invention covers pixel locations in order of increasing distance from the camera. place Radiation accumulation is performed on 3D Gaussian elements to obtain pixel positions. Radiation contribution at the location:

[0087] ;

[0088] in, To reconstruct image plane pixels along the direction of the camera's optical center The number of Gaussian elements distributed along the spectral direction. , This indicates that the pixels located at the 1st position are sorted by distance from nearest to farthest along the viewpoint direction from the camera's optical center to the reconstructed image plane. The 3D Gaussian index before each 3D Gaussian element;

[0089] In the 3D Gaussian sputtering process, each 3D Gaussian element first generates an initial radiation contribution based on bandpass emission, then undergoes atmospheric attenuation based on propagation distance, and finally, the radiation contribution is determined based on the position of the 3D Gaussian element in front of it at the pixel location. The cumulative transmittance is calculated based on the infrared opacity at a given location, thereby determining the pixel position of that Gaussian pixel. The effective radiation contribution at the location can avoid the problem of excessive brightness caused by simply adding the radiation contributions of multiple Gaussian elements at the same pixel location. At the same time, it can correctly express the occlusion and emission-absorption relationship between Gaussian elements, making the infrared reconstructed image more consistent with the real infrared imaging law in terms of brightness distribution and thermal radiation transmission law, and effectively improving the realism of infrared image reconstruction.

[0090] (3c22) MLP unit pair A nonlinear mapping is performed, and the pixel brightness of each infrared reconstructed image is obtained through several "linear transformation + nonlinear activation" operations. :

[0091] ;

[0092] (3c23) The image reconstruction unit reconstructs the pixel brightness values ​​at all pixel locations into a two-dimensional image array according to the corresponding pixel coordinates, thereby obtaining the infrared reconstructed image of each training sample. :

[0093] ;

[0094] ;

[0095] in, The location of the infrared reconstructed image is The brightness value of the pixel. and These are the height and width of the image, respectively.

[0096] (3d) Calculate the absolute error loss of the infrared image reconstruction network model using each reconstructed infrared image and its corresponding real infrared image. And adopt the gradient descent method, based on For model parameters , The network model is updated to obtain the reconstruction network model for this iteration. ;

[0097] Absolute error loss of reconstructing the network model The calculation formula is:

[0098] ;

[0099] in, The location of the real infrared image is The brightness value of the pixel; since there may be local high temperature points, low temperature points or residual abnormal noise in the infrared image, compared with the squared error loss, the absolute error loss is less sensitive to extreme pixel errors and can reduce the excessive influence of a small number of abnormal pixels on the parameter update direction.

[0100] For model parameters as well as The update is performed using the following formulas:

[0101] ;

[0102] ;

[0103] in, , These represent the updated model weights and biases, respectively. , These represent the model weights respectively. Bias The learning rate , These represent the absolute error loss of the infrared image reconstruction network model, respectively. For model weights Bias The partial derivatives;

[0104] (3e) Judgment If true, then a well-trained reconstruction network model is obtained. Otherwise, let , Then proceed with step (3b).

[0105] Step 4) Obtain the infrared image reconstruction results:

[0106] The test sample set is used as the trained reconstruction network model. The input is propagated forward to obtain A series of infrared reconstructed images.

[0107] The technical effects of this invention will be explained below with reference to simulation experiments:

[0108] 1. Experimental conditions and contents:

[0109] The hardware platform for the simulation experiment of this invention is: an AMD Ryzen 9 5950X CPU processor, 64GB of memory, and a single NVIDIA GeForce RTX 3090 graphics card. The software platform for the simulation experiment is: Ubuntu 20.04 operating system, Python version 3.9, and PyTorch version 2.0.1.

[0110] This invention was validated on the TI-NSD and RGBT-Scenes datasets. The invention was compared with two existing 3D Gaussian sputtering-based thermal reconstruction methods in terms of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learned perceptual image similarity (LPIPS). The results are shown in Table 1.

[0111] Peak signal-to-noise ratio (PSNR) is used to evaluate the similarity between the reconstructed infrared image and the actual observed image at the pixel level. A higher PSNR value indicates less image reconstruction distortion. The calculation formula is as follows:

[0112] ;

[0113] in, The maximum value of the image pixels. This represents the mean square error between the predicted pixel and the actual pixel.

[0114] Structural similarity (SSIM) measures the similarity between two images based on three dimensions: brightness, contrast, and structure. The formula for its calculation is as follows:

[0115] ;

[0116] in, and These represent the mean and variance of the image, respectively. Represents covariance, and To maintain a stable constant.

[0117] Learning-Perception Image Similarity (LPIPS) is used to evaluate the realism of reconstructed images at the level of human visual perception. The lower the value, the more the reconstructed image matches the visual characteristics of infrared thermal imaging.

[0118] ;

[0119] in, For true infrared images, To reconstruct the infrared image, For the first The height and width of the layer feature map For the real image in the first Layer feature map The feature vector of the location, To reconstruct the image at the Layer feature map The feature vector of the location.

[0120] 2. Analysis of experimental results:

[0121] Table 1 Simulation Comparison Results

[0122]

[0123] Referring to Table 1, compared with the prior art, the present invention improves PSNR and SSIM and reduces LPIPS on the TI-NSD and RGBT-Scenes datasets. All three indicators are better, indicating that the present invention shows a significant performance improvement in infrared image reconstruction tasks.

Claims

1. An infrared image reconstruction method based on 3D Gaussian sputtering driven by thermal radiation parameters, characterized in that, Includes the following steps: (1) Obtain the training sample set and the test sample set: For different scenes captured by thermal imaging cameras, and each scene has multiple perspectives The infrared images are preprocessed, and a total of each scene is included. The training sample set is composed of 10 infrared images used as labels for the corresponding preprocessed infrared images. The remaining preprocessed infrared images are then used as labels. The test sample set consists of 10 infrared images, among which ; (2) Construct an infrared image reconstruction network model based on thermal radiation parameters driven by 3D Gaussian sputtering: An infrared image reconstruction network model is constructed, comprising a cascaded sparse point cloud and camera parameter acquisition module, a 3D Gaussian primitive parameter generation module, and a rendering module driven by thermal radiation parameters for 3D Gaussian sputtering. The output of the sparse point cloud and camera parameter acquisition module is connected to the input of the rendering module. The 3D Gaussian primitive parameter generation module generates geometric parameters, including the 3D covariance matrix of the center coordinates, and physical parameters, including temperature and emissivity, for each 3D Gaussian primitive. The thermal radiation parameter-driven 3D Gaussian sputtering rendering module performs 3D Gaussian sputtering and nonlinear mapping on the 3D Gaussian primitive using thermal radiation parameters, including bandpass emission, atmospheric attenuation coefficient, and infrared opacity, calculated from the geometric and physical parameters of each 3D Gaussian primitive. (3) Iteratively train the infrared image reconstruction network model: The infrared image reconstruction network model was trained using a sample set. Perform iterative training to obtain a well-trained reconstruction network model. ; (4) Obtain the infrared image reconstruction results: The test sample set is used as the trained reconstruction network model. The input is propagated forward to obtain A series of infrared reconstructed images.

2. The method according to claim 1, characterized in that, The preprocessing described in step (1) is implemented as follows: Median filtering is applied to each infrared image, and the filtered infrared images are then normalized after non-uniform correction to obtain preprocessed infrared images.

3. The method according to claim 1, characterized in that, The infrared image reconstruction network model described in step (2) ,in: The rendering module includes a cascaded thermal radiation parameter calculation unit and a rendering unit; the rendering unit includes a cascaded radiation synthesis unit, an MLP unit, and an image reconstruction unit.

4. The method according to claim 3, characterized in that, The infrared image reconstruction network model described in step (3) The iterative training process involves the following steps: (3a) Initialize the number of iterations Maximum number of iterations Reconstructing the network model The weights and biases are respectively and and order ; (3b) The sparse point cloud and camera parameter acquisition module extracts the parameters of each training sample from the sparse point cloud and camera parameter acquisition module. A sparse point cloud composed of three-dimensional points in space and camera parameters; a 3D Gaussian meta-parameter generation module maps the sparse point cloud to obtain... Each 3D Gaussian element is assigned a coordinate system for a 3D point in space. As its corresponding 3D Gaussian element's center coordinates, then and the 3D covariance matrix of each 3D Gaussian element Geometric parameters that make up each 3D Gaussian element Simultaneously, physical parameters including temperature and emissivity of each 3D Gaussian primitive are generated using learnable parameters of each primitive. ; (3c) The thermal radiation parameter calculation unit in the rendering module is through , Bandpass emission in the thermal radiation parameters of each 3D Gaussian element is calculated using camera parameters. Atmospheric attenuation coefficient and infrared opacity ; The rendering unit renders each 3D Gaussian primitive using thermal radiation parameters to obtain an infrared reconstructed image of each training sample. (3d) Calculate the absolute error loss of the infrared image reconstruction network model using each reconstructed infrared image and its corresponding real infrared image. And adopt the gradient descent method, based on For model parameters , The network model is updated to obtain the reconstruction network model for this iteration. ; (3e) Judgment If true, then a well-trained reconstruction network model is obtained. Otherwise, let , Then proceed with step (3b).

5. The method according to claim 4, characterized in that, The camera parameters described in step (3b), wherein the first Intrinsic parameter matrix of a camera with a single viewpoint Rotation matrix Translation vector They are respectively: ； ； ； in, and They represent the first The focal length of a camera in the horizontal and vertical directions in the image pixel coordinate system. and They represent the first The pixel coordinates of the principal point of the camera at each viewpoint in the horizontal and vertical directions of the image pixel coordinate system. , , , For the first The pose unit quaternion of each viewpoint camera, and , , , Representing the origin of the world coordinate system to the 1st point, respectively. The translation distance of the camera coordinate system in the three orthogonal directions of the X, Y, and Z axes. , This indicates the number of infrared image viewpoints captured in the same scene.

6. The method according to claim 5, characterized in that, Each 3D Gaussian element described in step (3b) has the following geometric parameters. 3D covariance matrix and physical parameters Temperature in and emissivity The calculation formulas are as follows: ； ； ； ； in For the first Coordinates of a three-dimensional point in space of A set of three-dimensional point coordinates in a neighborhood space. for The Middle Coordinates of a three-dimensional point in space. This represents the 2-norm operation. for and The average spatial distance between the coordinates of all three-dimensional points in the space. It is a 3-order identity matrix. The temperature is a learnable parameter for each 3D Gaussian element. The minimum temperature threshold, The emissivity of each 3D Gaussian element is a learnable parameter. Represents the natural logarithm function. This represents an exponential function with the natural constant as its base.

7. The method according to claim 6, characterized in that, The bandpass emission described in step (3c) Atmospheric attenuation coefficient and infrared opacity The calculation formulas are as follows: ； ； ； ； ； ； ； in, For each 3D Gaussian element at temperature Spontaneous radiation under, For each 3D Gaussian element in the normal The ambient radiance in the direction, The learnable atmospheric extinction coefficient. The distance between each 3D Gaussian element and the camera optical center. The coordinates of each 3D Gaussian element in the camera coordinate system. The extinction coefficient for each 3D Gaussian element. For each 3D Gaussian primitive, the pixel location on the reconstructed image Projection weights at that location, The two-dimensional covariance matrix obtained by projecting each 3D Gaussian element onto the two-dimensional plane of the image. The projection coordinates of each 3D Gaussian element on the reconstructed image. It is the perspective projection function of a standard pinhole camera.

8. The method according to claim 7, characterized in that, The specific method for rendering each 3D Gaussian primitive as described in step (3c) is as follows: The radiation synthesis unit in the rendering unit passes through , and Perform 3D Gaussian sputtering on each 3D Gaussian pixel to obtain the pixel position of the 3D Gaussian pixel on the two-dimensional plane of the reconstructed image. Radiation contribution at the location MLP unit pair A nonlinear mapping is performed to obtain the pixel brightness of each infrared reconstructed image. The image reconstruction unit reassembles the pixel brightness values ​​at all pixel locations into a two-dimensional image array according to their corresponding pixel coordinates, thus obtaining the infrared reconstructed image of each training sample. ,in: ； ； ； ； in, To reconstruct image plane pixels along the direction of the camera's optical center The number of Gaussian elements distributed along the spectral direction. , This indicates that the pixels located at the 1st position are sorted by distance from nearest to farthest along the viewpoint direction from the camera's optical center to the reconstructed image plane. The 3D Gaussian element index before each 3D Gaussian element. The location of the infrared reconstructed image is The brightness value of the pixel. and These represent the height and width of the image, respectively.

9. The method according to claim 8, characterized in that, The absolute error loss described in step (3d) The calculation formula is: ； in, The location of the real infrared image is The brightness value of the pixel.

10. The method according to claim 4, characterized in that, The step (3d) describes the adjustment of model parameters. as well as The update is performed using the following formulas: ； ； in, , These represent the updated model weights and biases, respectively. , These represent the model weights respectively. Bias learning rate, , These represent the absolute error loss of the infrared image reconstruction network model, respectively. For model weights Bias The partial derivatives of .

Images