Infrared scene reconstruction and rendering method and device based on three-dimensional Gaussian splashing

Abstract

The invention provides an infrared scene reconstruction and rendering method and device based on three-dimensional Gaussian splashing, and the method comprises the steps: carrying out the processing of preprocessed infrared image data through employing an SfM algorithm, obtaining a sparse three-dimensional point cloud and a camera pose, and setting a Gaussian ellipsoid at the center position of the sparse three-dimensional point cloud, so as to construct a three-dimensional Gaussian point cloud model; inputting the camera pose into a Gaussian microrasterized renderer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image of a corresponding view angle; the two-dimensional image and the infrared image data of the camera pose are compared through a loss function, the model is trained, a trained three-dimensional Gaussian point cloud model is obtained, and the loss function comprises a temperature gradient field, the projection position of a Gaussian ellipsoid in a pixel plane under the current camera visual angle and the main axis projection direction; and processing the trained three-dimensional Gaussian point cloud model by using a three-dimensional graphic engine and a 3D Gaussian rasterization renderer to obtain an infrared simulation image. And the authenticity of reconstruction and rendering of the infrared scene is relatively high.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared scene simulation, and in particular to a method and device for reconstructing and rendering an infrared scene based on three-dimensional Gaussian splattering. Background Art

[0002] 3D Gaussian Splatting (3DGS) is an emerging 3D scene representation and rendering technology. The core concept of 3DGS is to represent a 3D scene using a point cloud composed of 3D Gaussian functions. Each point can be considered a Gaussian ellipsoid, defined by parameters such as the position (mean), covariance matrix, and intensity. Volume rendering is used to calculate light attenuation and scattering, thereby generating a view from a specified perspective. Because the Gaussian distribution is continuous, 3DGS can represent complex surface textures and achieve smooth transitions between light and shadow in the visible light band, making it suitable for scene representation under complex lighting conditions. For 3D scenes in the infrared band, the core element of 3DGS is the spontaneous emission from surfaces with continuous thermal distributions, in addition to the influence of environmental reflections, atmospheric transmission, and other factors. At a 1080p resolution on a personal computer, 3DGS can often achieve rendering speeds exceeding 100 frames per second, enabling fast interactive response in infrared scene simulation applications. 3DGS is particularly important in infrared scene simulation.

[0003] Currently, traditional infrared scene simulation methods typically use a method that samples the temperature, radiance, or other grayscale textures of the target and background, and simplifies the coupling elements of the infrared radiation transmission characteristics at the real physical level to simulate infrared scenes. However, this simplification of the coupling elements of the infrared radiation transmission characteristics at the real physical level makes it difficult to achieve a simulation effect that approaches reality, resulting in poor realism in the reconstruction and rendering of infrared scenes. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method and device for reconstructing and rendering an infrared scene based on three-dimensional Gaussian splattering, so as to solve the problem of poor authenticity in the reconstruction and rendering of the infrared scene.

[0005] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of the present invention provides an infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering, comprising:

[0007] The infrared image data is preprocessed using Gaussian filtering method;

[0008] The pre-processed infrared image data is processed using the structure-from-motion algorithm to obtain a sparse 3D point cloud and camera pose. A Gaussian ellipsoid is set at the center of the sparse 3D point cloud to construct a 3D Gaussian point cloud model.

[0009] The camera pose is input into the Gaussian differentiable rasterizer to render the 3D Gaussian point cloud model and obtain a 2D image of the corresponding perspective.

[0010] The loss function is used to compare the two-dimensional image and the infrared image data corresponding to the camera pose, and the three-dimensional Gaussian point cloud model is trained to obtain a trained three-dimensional Gaussian point cloud model. The loss function is a function of the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera perspective, and the main axis projection direction of the Gaussian ellipsoid.

[0011] The trained 3D Gaussian point cloud model is processed using 3D graphics engine software and 3D Gaussian rasterizer to obtain an infrared simulation image.

[0012] A second aspect of the present invention provides an infrared scene reconstruction and rendering device based on three-dimensional Gaussian splattering, comprising:

[0013] A preprocessing module, used for preprocessing infrared image data using a Gaussian filtering method;

[0014] A construction module is used to process the preprocessed infrared image data using the motion recovery structure algorithm to obtain a sparse three-dimensional point cloud and camera pose, and set the Gaussian ellipsoid at the center of the sparse three-dimensional point cloud to construct a three-dimensional Gaussian point cloud model;

[0015] The rendering module is used to input the camera pose into the Gaussian differentiable rasterizer renderer to render the three-dimensional Gaussian point cloud model and obtain a two-dimensional image of the corresponding perspective;

[0016] A training module is used to train a 3D Gaussian point cloud model by comparing the 2D image with the infrared image data corresponding to the camera pose using a loss function. The loss function is a function of the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera perspective, and the projection direction of the principal axis of the Gaussian ellipsoid.

[0017] The processing module is used to process the trained 3D Gaussian point cloud model using 3D graphics engine software and 3D Gaussian rasterizer to obtain an infrared simulation image.

[0018] Compared with the prior art, the infrared scene reconstruction and rendering method and device based on three-dimensional Gaussian splattering provided by the present invention use a Gaussian filtering method to preprocess infrared image data; use a motion recovery structure algorithm to process the preprocessed infrared image data to obtain a sparse three-dimensional point cloud and camera pose, and set a Gaussian ellipsoid at the center of the sparse three-dimensional point cloud to construct a three-dimensional Gaussian point cloud model; input the camera pose into a Gaussian differentiable rasterizer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image of the corresponding perspective; use a loss function to compare the two-dimensional image and the infrared image data corresponding to the camera pose, train the three-dimensional Gaussian point cloud model, and obtain a trained three-dimensional Gaussian point cloud model, where the loss function is a function of the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera perspective, and the projection direction of the main axis of the Gaussian ellipsoid; use three-dimensional graphics engine software and a 3D Gaussian rasterizer to process the trained three-dimensional Gaussian point cloud model to obtain an infrared simulation image. In this way, during training, the loss function of the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle and the main axis projection direction of the Gaussian ellipsoid is used, as well as the infrared image data corresponding to the two-dimensional image and the camera posture is compared. The three-dimensional Gaussian point cloud model can be well trained to obtain a high-quality trained three-dimensional Gaussian point cloud model. By processing the well-trained three-dimensional Gaussian point cloud model, the quality of the obtained infrared simulation image is high, which makes the reconstruction and rendering of the infrared scene more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0020] Figure 1 The flowchart of the infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering is schematically shown;

[0021] Figure 2 A schematic diagram of infrared image data is schematically shown;

[0022] Figure 3 A schematic diagram of an infrared simulation image is schematically shown;

[0023] Figure 4 The structure of the infrared scene reconstruction and rendering device based on three-dimensional Gaussian splattering is schematically shown. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0025] It should be noted that, unless otherwise specified, the technical or scientific terms used in the present invention should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0026] The method in the embodiment of the present invention is described in detail below.

[0027] Figure 1 The flowchart of the infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering in an embodiment of the present invention is schematically shown. Figure 1 As shown, the infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering may include:

[0028] S101 , preprocessing the infrared image data using a Gaussian filtering method.

[0029] Before preprocessing the infrared image data using the Gaussian filtering method, the method further includes:

[0030] Acquire infrared image data, wherein the infrared image data is data obtained by photographing a scene target using a calibrated infrared camera.

[0031] Figure 2 Schematic diagram showing infrared image data, Figure 2 (a) and (b) are infrared image data taken from different angles, respectively, and both use a calibrated infrared camera to take infrared image data containing scene targets.

[0032] The acquired infrared image data can fully and meticulously display the scene and avoid motion blur.

[0033] Specifically, in order to reduce the artifacts caused by the inherent noise of the infrared detector and improve the quality of the subsequently trained three-dimensional Gaussian point cloud model, when the infrared camera imaging used has obvious sensor noise, the infrared image data is preprocessed using the Gaussian filtering method. When the infrared camera imaging used has no obvious sensor noise, the filtering parameters can be appropriately adjusted or the infrared image data can be not preprocessed.

[0034] S102. Process the preprocessed infrared image data using a structure-from-motion algorithm to obtain a sparse three-dimensional point cloud and camera pose, and set a Gaussian ellipsoid at the center of the sparse three-dimensional point cloud to construct a three-dimensional Gaussian point cloud model.

[0035] There are multiple infrared image data after preprocessing.

[0036] Specifically, the preprocessed infrared image data is processed using the Structure from Motion (SfM) algorithm to obtain a sparse 3D point cloud and camera pose. A Gaussian ellipsoid is set at the center of the sparse 3D point cloud to construct a 3D Gaussian point cloud model, including:

[0037] Feature global matching step: According to the Scale-Invariant Feature Transform (SIFT) algorithm, all feature points in the first preprocessed infrared image data are extracted, and the nearest neighbor search method is used to identify and match the key feature points of the same object in all preprocessed infrared image data among all feature points to obtain the marking information of the same feature points in the two processed infrared image data.

[0038] Initial modeling step: Select two image data from all preprocessed infrared image data, calculate the projection matrix of the two image data through the camera intrinsic parameters to estimate the initial camera pose and construct a three-dimensional point cloud model; among them, the projection matrix of the two image data is the projection matrix of the three-dimensional world coordinate space to the pixel space of the camera for the current infrared image data.

[0039] Global optimization step: Use bundle adjustment to optimize the initial camera pose and the 3D point coordinates in the 3D point cloud model to obtain the optimized camera pose and optimized 3D point coordinates;

[0040] Loop step: From the remaining preprocessed infrared image data except the first preprocessed infrared image data, select and add new infrared image data one by one, replace the first preprocessed infrared image data in the feature global matching step with the new infrared image data, repeat the feature global matching step and the initial modeling step to expand the three-dimensional Gaussian point cloud model until all feature points have been extracted for all preprocessed infrared image data, stop the loop operation, obtain the sparse three-dimensional point cloud and camera pose, and set the Gaussian ellipsoid at the center of the sparse three-dimensional point cloud to construct the three-dimensional Gaussian point cloud model.

[0041] S103 : Input the camera pose into a Gaussian differentiable rasterizer renderer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image of a corresponding perspective.

[0042] Specifically, using a Gaussian differentiable rasterizer renderer, the three-dimensional scene represented in the three-dimensional Gaussian point cloud model can be rendered by volume rendering to obtain a two-dimensional image of the corresponding perspective.

[0043] After inputting the camera pose into the Gaussian differentiable rasterizer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image corresponding to the view angle, the method further comprises:

[0044] Step A1: performing a discrete differential operation on the preprocessed infrared image data by using a Sobel operator to obtain a temperature gradient field.

[0045] The temperature gradient field includes a horizontal temperature gradient field and a vertical temperature gradient field.

[0046] The temperature gradient field is used to represent the spatial distribution characteristics of temperature changes.

[0047] The expression of the temperature gradient field is:

[0048]

[0049] wherein G x is the horizontal temperature gradient field, G y is the vertical temperature gradient field, I is a matrix corresponding to the preprocessed infrared image data, I∈R H×W , G x , G y ∈R H×W , R H×W is a matrix of the height H and the width W of the resolution of the infrared image data, S x is a convolution kernel of the horizontal Sobel operator, S y is a convolution kernel of the vertical Sobel operator, S y =S x T , * is a convolution operation, G x , G y ∈R H×W .

[0050] Step A2: determining a thermal gradient direction according to the horizontal temperature gradient field, the vertical temperature gradient field, a preset threshold value, and a center position corresponding to the Gaussian ellipsoid.

[0051] Specifically, step A2 comprises:

[0052] Step A21: determining a temperature gradient vector field according to the horizontal temperature gradient field and the vertical temperature gradient field, and performing a normalization calculation on the temperature gradient vector field to obtain a normalized temperature gradient vector field.

[0053] To avoid the influence of the amplitude on the scale of the direction vector, a temperature gradient vector field The gradient direction is normalized to a unit vector:

[0054]

[0055] in, The element in the i-th row and j-th column of the matrix corresponding to the normalized temperature gradient vector field, G x i,j is the element of the i-th row and j-th column of the matrix corresponding to the horizontal temperature gradient field, G y i,j is the element in the i-th row and j-th column of the matrix corresponding to the temperature gradient field in the vertical direction, is the vector field of normalized temperature gradient, R H×W×2 The height H, width W and length 2 matrix of the infrared image data resolution corresponding to the normalized temperature gradient vector field, is the modulus of the element in the i-th row and j-th column of the matrix corresponding to the normalized temperature gradient vector field.

[0056] Step A22: Determine a thermal gradient tensor according to the normalized temperature gradient vector field and a preset threshold.

[0057] Introducing a preset threshold criterion to eliminate pseudo gradients in low signal-to-noise ratio areas can improve the physical interpretability of the direction field.

[0058] The expression of the thermal gradient tensor is:

[0059]

[0060] Among them, u thm is the thermal gradient tensor, u thm ∈R H×W×2 , ε is the preset threshold, ε=10 -3 .

[0061] Step A23: Determine the thermal gradient direction according to the thermal gradient tensor and the corresponding center position of the Gaussian ellipsoid.

[0062] Substitute the center position corresponding to each Gaussian ellipsoid into the expression of the thermal gradient tensor to obtain the thermal gradient direction of the thermal gradient tensor at the i′th effective Gaussian ellipsoid projected to the current infrared camera pixel coordinate position

[0063] Step A3: Determine the projection position of the Gaussian ellipsoid in the normalized device coordinate space based on the world camera transformation matrix, the Gaussian ellipsoid position in the world coordinate system, and the principal axis direction with the largest scaling factor.

[0064] Specifically, step A3 includes:

[0065] Step A31: Use the world camera transformation matrix T view ∈R 4×4, the Gaussian ellipsoid position μ∈R in the world coordinate system 3 Convert to camera space and get the position of the Gaussian ellipsoid in the camera space coordinate system.

[0066] The expression of the position of the Gaussian ellipsoid in the camera space coordinate system is:

[0067]

[0068] Among them, P cam is the position of the Gaussian ellipsoid in the camera space coordinate system, which is represented by the three-dimensional space coordinate (x cam ,y cam ,z cam ) form homogeneous coordinates, x cam is the horizontal axis coordinate of the Gaussian ellipsoid in the camera space coordinate system, y cam is the vertical axis coordinate of the Gaussian ellipsoid in the camera space coordinate system, z cam is the depth axis coordinate of the Gaussian ellipsoid in the camera space coordinate system, T view is the world camera transformation matrix, μ is the Gaussian ellipsoid position in the world coordinate system, R 4×4 is a matrix with a height and width of 4, R 3×1 is a 3×1 real matrix.

[0069] Step A32: Use the world camera transformation matrix T view ∈R 4×4 , the main axis direction v with the largest scaling factor world ∈R 3×1 Convert to camera space and get the principal axis direction vector in the camera coordinate system.

[0070] The expression of the principal axis direction vector in the camera coordinate system is:

[0071]

[0072] Among them, v cam is the principal axis direction vector in the camera coordinate system, which is composed of the three-dimensional vector (v x ,v y ,v z ) form homogeneous coordinates, v x is the x-axis direction vector in the camera coordinate system, v y is the y-axis direction vector in the camera coordinate system, v z is the z-axis direction vector in the camera coordinate system, T view is the world camera transformation matrix, v world The direction of the principal axis with the largest scaling factor.

[0073] Step A33: Project the position of the Gaussian ellipsoid in the camera space coordinate system and the principal axis direction vector in the camera coordinate system to normalized device coordinates (NDC) to obtain the projection position of the Gaussian ellipsoid in the normalized device coordinate space.

[0074] The expression for the projected position of the Gaussian ellipsoid in the normalized device coordinate space is:

[0075]

[0076] Among them, p ndc is the projection position of the Gaussian ellipsoid in the NDC space, x ndc is the horizontal axis coordinate of the Gaussian ellipsoid projection in the NDC space, y ndc is the projection axis coordinate of the Gaussian ellipsoid in NDC space.

[0077] Step A4: According to the projection position of the Gaussian ellipsoid in the normalized device coordinate space, determine the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle.

[0078] Specifically, for the projection position of the Gaussian ellipsoid in the normalized device coordinate space, the direction vector v along the principal axis in the camera coordinate system is cam =(v x ,v y ,v z ) Take the derivative and get the Jacobian matrix:

[0079]

[0080] in, is the partial derivative of the projection position of the Gaussian ellipsoid in the normalized device coordinate space with respect to the small displacement s of the principal axis direction vector in the camera coordinate system, that is, the rate of change of the NDC coordinate caused by each unit displacement of the principal axis direction vector in the camera coordinate system, s is the small displacement of the principal axis direction vector in the camera coordinate system, is the partial derivative of the projection horizontal axis coordinate of the Gaussian ellipsoid in the NDC space with respect to the small displacement s of the principal axis direction vector in the camera coordinate system, It is the partial derivative of the projection axis coordinate of the Gaussian ellipsoid in the NDC space with respect to the small displacement s of the principal axis direction vector in the camera coordinate system.

[0081] NDC coordinates to pixel coordinates (x pixel ,y pixel ), that is, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle is:

[0082]

[0083] Among them, x pixel is the horizontal axis coordinate of the Gaussian ellipsoid projection on the pixel plane under the current camera perspective, y pixel is the projection vertical axis coordinate of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle, W is the width of the infrared image data resolution, and H is the height of the infrared image data resolution.

[0084] Step A5: Determine the main axis projection direction of the Gaussian ellipsoid according to the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle.

[0085] Specifically, the mapping relationship from NDC coordinates to pixel coordinates and the Jacobian matrix are used to transform the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle to the pixel coordinates, and the projection direction Δp of the small displacement s of the main axis direction vector in the camera coordinate system along the main axis direction in the pixel coordinates is obtained. pixel , and Δp pixel Normalized to obtain the principal axis projection direction of the Gaussian ellipsoid:

[0086]

[0087] Among them, u proj is the principal axis projection direction of a single Gaussian ellipsoid, Δp pixel is the projection direction of the small displacement s of the main axis direction vector in the camera coordinate system along the main axis direction vector in the pixel coordinate system, is the partial derivative of the projection horizontal axis coordinate of the Gaussian ellipsoid on the pixel plane under the current camera perspective with respect to the small displacement s of the principal axis direction vector in the camera coordinate system. It is the partial derivative of the projected vertical axis coordinate of the Gaussian ellipsoid on the pixel plane under the current camera perspective with respect to the small displacement s of the principal axis direction vector in the camera coordinate system.

[0088] Perform the above steps A1 to A5 on each pre-processed infrared image data to obtain the main axis projection direction of the Gaussian ellipsoid, that is, the main axis projection direction of the Gaussian ellipsoid of the i′th effective Gaussian main axis under the current infrared camera pixel coordinate projection.

[0089] Specifically, after step A5, the method further includes:

[0090] Step A6: Determine the absolute difference between the two-dimensional image and the infrared image data corresponding to the camera pose.

[0091] Step A7: Determine the cosine similarity loss of the thermal gradient according to the thermal gradient direction, the principal axis projection direction of the Gaussian ellipsoid, and the set of valid Gaussian ellipsoids.

[0092] Step A8: Determine a loss function based on the absolute difference, the cosine similarity loss of the thermal gradient, and the loss function based on the differentiable structural similarity index.

[0093] The expression of the loss function is:

[0094]

[0095] Among them, L is the loss function, λ D-SSIM is the weight coefficient for controlling the structural similarity loss, λ COS-SIM To control the weight coefficient of cosine similarity loss, L1 is the absolute difference between the two-dimensional image and the infrared image data corresponding to the camera pose, L D-SSIM is the loss function based on the differentiable structural similarity index, L COS-SIM is the cosine similarity loss of the thermal gradient, and U is the set of valid Gaussian ellipsoids is the principal axis projection direction of the Gaussian ellipsoid of the i′th effective Gaussian principal axis under the current infrared camera pixel coordinate projection, is the thermal gradient direction of the thermal gradient tensor projected from the i′th effective Gaussian ellipsoid to the current infrared camera pixel coordinate position, and ||·|| is the modulus.

[0096] L1 directly measures the difference in pixel values ​​between the predicted image (i.e., the two-dimensional image) and the real image (i.e., the infrared image data corresponding to the camera pose). D-SSIM is a loss function based on Differentiable Structural Similarity Index (D-SSIM), which is used to characterize the structural difference between the predicted image and the real image. D-SSIM The value of λ determines the importance of structural similarity over pixel differences when optimizing the 3D Gaussian point cloud model. D-SSIM The value of λ is usually taken as 0.2 in practice. COS-SIM In practice, a dynamic weight decay strategy is adopted, with an initial value of 0.2 and λ COS-SIM The value of is dynamically reduced with the number of optimization iterations. The purpose is to align the directions of the strongly constrained thermal gradients in the early stage of training and gradually release them in the later stage to avoid overfitting.

[0097] The loss function defines the alignment of the Gaussian principal axis projection direction with the thermal gradient, forcing the Gaussian ellipsoid to extend along the thermal gradient. This ensures a physically plausible fit for complex infrared scenes and improves rendering quality. The loss function combines the absolute difference between the 2D image and the infrared image data corresponding to the camera pose, the structural similarity of the images, and the thermal gradient difference to improve rendering quality.

[0098] S104. Using the loss function, compare the two-dimensional image and the infrared image data corresponding to the camera pose, train the three-dimensional Gaussian point cloud model, and obtain a trained three-dimensional Gaussian point cloud model.

[0099] The loss function is a function of the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle, and the main axis projection direction of the Gaussian ellipsoid.

[0100] During the training process, the 3D Gaussian point cloud model also needs to be optimized through density control and other optimizations, and the overall reconstruction quality is improved through densification and culling operations. The densification process focuses on areas where the Gaussian is too sparse. After a certain number of iterations (usually 100 times), the Gaussian whose view space position gradient exceeds a certain threshold is cloned or split. For under-reconstructed small-scale Gaussians, a copy is created and moved along the direction of its position gradient; for over-reconstructed large-scale Gaussians, it is replaced by two smaller Gaussians that are reduced by a specific scale factor. Cloning or splitting is adaptive during the optimization process, which can make the 3D Gaussian point cloud model better describe the details of the scene.

[0101] Culling primarily targets Gaussians with minimal or unreasonable impact. After a fixed number of iterations, Gaussians with transparency below a certain threshold are removed. This also removes Gaussians that are too large in view space to prevent an unreasonable increase in Gaussian density near the input camera pose. This culling operation ensures the accuracy and effectiveness of the Gaussians while conserving computational resources during the optimization process.

[0102] S105 , using 3D graphics engine software and a 3D Gaussian rasterizer to process the trained 3D Gaussian point cloud model to obtain an infrared simulation image.

[0103] Specifically, the trained 3D Gaussian point cloud model is processed using 3D graphics engine software and a 3D Gaussian rasterizer to obtain an infrared simulation image, including:

[0104] Step B1: Using 3D graphics engine software, the trained 3D Gaussian point cloud model is cropped to obtain a cropped 3D Gaussian point cloud model.

[0105] Step B2: Use a 3D Gaussian rasterizer to render the cropped 3D Gaussian point cloud model to obtain an infrared simulation image.

[0106] Figure 3 Schematic diagram of an infrared simulation image is shown schematically, wherein: Figure 3 (a) and (b) are infrared simulation images obtained at different angles, respectively. Figure 2The angles of (a) and (b) are the same. They are the rendering effects of the cropped 3D Gaussian point cloud model in the 3D engine after the trained 3D Gaussian point cloud model is cropped.

[0107] The present invention represents the infrared radiation field through a three-dimensional Gaussian point cloud model, and the infrared image data collected by the infrared camera trains the three-dimensional Gaussian point cloud model. The operation is simple and efficient. Compared with the traditional rasterization, ray tracing and other display methods for representing scenes, there is no need to invest costs in modeling, texture rendering, etc. The simulation effect of the present invention depends on the real scene data taken by the infrared camera, which is closer to the real imaging of the infrared sensor than the rendering method that only uses traditional rasterization. The present invention is suitable for real-time simulation rendering scenes, and has a faster rendering speed than ray tracing, path tracing and simulation methods based on the three-dimensional reconstruction and differentiable rendering technology of the Neural Radiance Fields (NeRF) series. The rendering strategy of the present invention is compatible with the traditional rasterization graphics pipeline, allowing the use of the same shader to render Gaussian models and mesh models. Compared with the simulation method based on the NeRF series and other implicit models to represent scenes, it has a faster model training speed and higher simulation flexibility.

[0108] Based on the above Figure 1 It can be seen from the implementation method that the embodiment of the present invention uses a Gaussian filtering method to preprocess the infrared image data; uses a motion recovery structure algorithm to process the preprocessed infrared image data to obtain a sparse three-dimensional point cloud and a camera pose, and sets a Gaussian ellipsoid at the center of the sparse three-dimensional point cloud to construct a three-dimensional Gaussian point cloud model; inputs the camera pose into a Gaussian differentiable rasterizer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image of the corresponding perspective; uses a loss function to compare the two-dimensional image and the infrared image data corresponding to the camera pose, and trains the three-dimensional Gaussian point cloud model to obtain a trained three-dimensional Gaussian point cloud model, where the loss function is a function of the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera perspective, and the projection direction of the principal axis of the Gaussian ellipsoid; uses a three-dimensional graphics engine software and a 3D Gaussian rasterizer to process the trained three-dimensional Gaussian point cloud model to obtain an infrared simulation image. In this way, during training, the loss function of the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle and the main axis projection direction of the Gaussian ellipsoid is used, as well as the infrared image data corresponding to the two-dimensional image and the camera posture is compared. The three-dimensional Gaussian point cloud model can be well trained to obtain a high-quality trained three-dimensional Gaussian point cloud model. By processing the well-trained three-dimensional Gaussian point cloud model, the quality of the obtained infrared simulation image is high, which makes the reconstruction and rendering of the infrared scene more realistic.

[0109] Based on the same inventive concept, as an implementation of the above-mentioned infrared scene reconstruction and rendering method based on three-dimensional Gaussian splashing, an embodiment of the present invention further provides an infrared scene reconstruction and rendering device based on three-dimensional Gaussian splashing. Figure 4 is a structural diagram of the device in the embodiment of the present invention, see Figure 4 As shown, the infrared scene reconstruction and rendering device based on three-dimensional Gaussian splashing may include:

[0110] The pre-processing module 401 is used to pre-process the infrared image data using a Gaussian filtering method;

[0111] A construction module 402 is configured to process the preprocessed infrared image data using a structure-from-motion algorithm to obtain a sparse three-dimensional point cloud and a camera pose, and to set a Gaussian ellipsoid at the center of the sparse three-dimensional point cloud to construct a three-dimensional Gaussian point cloud model.

[0112] A rendering module 403 is used to input the camera pose into a Gaussian differentiable rasterizer renderer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image of the corresponding perspective;

[0113] A training module 404 is configured to train a three-dimensional Gaussian point cloud model by comparing the two-dimensional image with the infrared image data corresponding to the camera pose using a loss function to obtain a trained three-dimensional Gaussian point cloud model, wherein the loss function is a function of the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane at the current camera perspective, and the projection direction of the principal axis of the Gaussian ellipsoid;

[0114] The processing module 405 is used to process the trained 3D Gaussian point cloud model using 3D graphics engine software and a 3D Gaussian rasterizer to obtain an infrared simulated image.

[0115] The device may also include:

[0116] The acquisition module is used to acquire infrared image data before preprocessing the infrared image data using the Gaussian filtering method. The infrared image data is data of a scene target photographed using a calibrated infrared camera.

[0117] The device may also include:

[0118] The temperature gradient field determination module is used to input the camera pose into the Gaussian differentiable rasterizer to render the three-dimensional Gaussian point cloud model. After obtaining the two-dimensional image of the corresponding perspective, the Sobel operator is used to perform discrete differential operations on the pre-processed infrared image data to obtain the temperature gradient field. The temperature gradient field includes the temperature gradient field in the horizontal direction and the temperature gradient field in the vertical direction.

[0119] A thermal gradient direction determination module is used to determine the thermal gradient direction based on the horizontal temperature gradient field, the vertical temperature gradient field, a preset threshold value, and the corresponding center position of the Gaussian ellipsoid;

[0120] A first projection position determination module is used to determine the projection position of the Gaussian ellipsoid in the normalized device coordinate space according to the world camera transformation matrix, the position of the Gaussian ellipsoid in the world coordinate system, and the direction of the principal axis with the largest scaling factor;

[0121] A second projection position determination module is used to determine the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle according to the projection position of the Gaussian ellipsoid in the normalized device coordinate space;

[0122] The projection direction determination module is used to determine the main axis projection direction of the Gaussian ellipsoid according to the projection position of the pixel plane of the Gaussian ellipsoid under the current camera viewing angle.

[0123] The device may also include:

[0124] an absolute difference determination module for determining an absolute difference between the two-dimensional image and the infrared image data corresponding to the camera pose after determining the principal axis projection direction of the Gaussian ellipsoid based on the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle and the projection position of the Gaussian ellipsoid in the normalized device coordinate space;

[0125] A cosine similarity loss determination module is used to determine the cosine similarity loss of the thermal gradient according to the thermal gradient direction, the principal axis projection direction of the Gaussian ellipsoid, and the set of valid Gaussian ellipsoids;

[0126] The loss function determination module is used to determine the loss function according to the absolute difference, the cosine similarity loss of the thermal gradient, and the loss function based on the differentiable structural similarity index.

[0127] The thermal gradient direction determination module is specifically used to determine the temperature gradient vector field based on the horizontal temperature gradient field and the vertical temperature gradient field, and normalize the temperature gradient vector field to obtain a normalized temperature gradient vector field; determine the thermal gradient tensor based on the normalized temperature gradient vector field and a preset threshold; and determine the thermal gradient direction based on the thermal gradient tensor and the corresponding center position of the Gaussian ellipsoid.

[0128] The first projection position determination module is specifically used to use the world camera transformation matrix to convert the position of the Gaussian ellipsoid in the world coordinate system to the camera space, so as to obtain the position of the Gaussian ellipsoid in the camera space coordinate system; use the world camera transformation matrix to convert the principal axis direction with the largest scaling factor to the camera space, so as to obtain the principal axis direction vector in the camera coordinate system; project the position of the Gaussian ellipsoid in the camera space coordinate system and the principal axis direction vector in the camera coordinate system to the normalized device coordinates, so as to obtain the projection position of the Gaussian ellipsoid in the normalized device coordinate space.

[0129] Loss function determination module, the expression of loss function is:

[0130]

[0131] Among them, L is the loss function, λ D-SSIM is the weight coefficient for controlling the structural similarity loss, λ COS-SIM To control the weight coefficient of cosine similarity loss, L1 is the absolute difference between the two-dimensional image and the infrared image data corresponding to the camera pose, L D-SSIM is the loss function based on the differentiable structural similarity index, L COS-SIM is the cosine similarity loss of the thermal gradient, U is the set of valid Gaussian ellipsoids, is the principal axis projection direction of the Gaussian ellipsoid of the i′th effective Gaussian principal axis under the projection of the current infrared camera pixel coordinates, is the thermal gradient direction of the thermal gradient tensor projected from the i′th effective Gaussian ellipsoid to the current infrared camera pixel coordinate position, and ||·|| is the modulus.

[0132] The processing module 405 is specifically used to use the three-dimensional graphics engine software to crop the trained three-dimensional Gaussian point cloud model to obtain a cropped three-dimensional Gaussian point cloud model; and use the 3D Gaussian rasterizer renderer to render the cropped three-dimensional Gaussian point cloud model to obtain an infrared simulation image.

[0133] It should be noted that the above description of the embodiment of the infrared scene reconstruction and rendering device based on 3D Gaussian splattering is similar to the description of the above-mentioned embodiment of the infrared scene reconstruction and rendering method based on 3D Gaussian splattering, and has similar beneficial effects as the embodiment of the infrared scene reconstruction and rendering method based on 3D Gaussian splattering. For any technical details not disclosed in the embodiment of the infrared scene reconstruction and rendering device based on 3D Gaussian splattering according to the present invention, please refer to the description of the embodiment of the infrared scene reconstruction and rendering method based on 3D Gaussian splattering according to the present invention for understanding.

[0134] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for infrared scene reconstruction and rendering based on three-dimensional Gaussian splattering, characterized in that: include: The infrared image data is preprocessed using Gaussian filtering method; The preprocessed infrared image data is processed using a structure-from-motion algorithm to obtain a sparse three-dimensional point cloud and camera pose, and a Gaussian ellipsoid is set at the center of the sparse three-dimensional point cloud to construct a three-dimensional Gaussian point cloud model; Inputting the camera pose into a Gaussian differentiable rasterizer renderer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image of the corresponding perspective; Using a loss function, the two-dimensional image and the infrared image data corresponding to the camera pose are compared to train the three-dimensional Gaussian point cloud model to obtain a trained three-dimensional Gaussian point cloud model, wherein the loss function is a function including the temperature gradient field of the preprocessed infrared image data, the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle, and the main axis projection direction of the Gaussian ellipsoid; The trained three-dimensional Gaussian point cloud model is processed using three-dimensional graphics engine software and a 3D Gaussian rasterizer to obtain an infrared simulation image.

2. The infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering according to claim 1, characterized in that: Before preprocessing the infrared image data using the Gaussian filtering method, the method further includes: The infrared image data is acquired, where the infrared image data is data obtained by photographing a scene target using a calibrated infrared camera.

3. The infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering according to claim 1, characterized in that: After inputting the camera pose into a Gaussian differentiable rasterizer renderer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image of a corresponding perspective, the method further includes: Performing discrete differential operation on the preprocessed infrared image data using a Sobel operator to obtain the temperature gradient field, wherein the temperature gradient field includes a horizontal temperature gradient field and a vertical temperature gradient field; Determining a thermal gradient direction according to the horizontal temperature gradient field, the vertical temperature gradient field, a preset threshold, and a center position corresponding to the Gaussian ellipsoid; Determine the projection position of the Gaussian ellipsoid in the normalized device coordinate space based on the world camera transformation matrix, the position of the Gaussian ellipsoid in the world coordinate system, and the direction of the principal axis with the largest scaling factor; Determining the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle according to the projection position of the Gaussian ellipsoid in the normalized device coordinate space; The main axis projection direction of the Gaussian ellipsoid is determined according to the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle.

4. The infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering according to claim 3, characterized in that: After determining the principal axis projection direction of the Gaussian ellipsoid according to the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle and the projection position of the Gaussian ellipsoid in the normalized device coordinate space, the method further includes: Determining an absolute difference between the two-dimensional image and the infrared image data corresponding to the camera pose; Determining a cosine similarity loss of the thermal gradient according to the thermal gradient direction, the principal axis projection direction of the Gaussian ellipsoid, and a set of valid Gaussian ellipsoids; The loss function is determined according to the absolute difference, the cosine similarity loss of the thermal gradient, and a loss function based on a differentiable structural similarity index.

5. The infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering according to claim 3, characterized in that: The determining of the thermal gradient direction according to the temperature gradient field in the horizontal direction, the temperature gradient field in the vertical direction, a preset threshold, and the center position corresponding to the Gaussian ellipsoid includes: Determining a temperature gradient vector field according to the temperature gradient field in the horizontal direction and the temperature gradient field in the vertical direction, and performing normalization calculation on the temperature gradient vector field to obtain a normalized temperature gradient vector field; determining a thermal gradient tensor according to the normalized temperature gradient vector field and the preset threshold; The thermal gradient direction is determined according to the thermal gradient tensor and the corresponding center position of the Gaussian ellipsoid.

6. The infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering according to claim 3, characterized in that: Determining the projection position of the Gaussian ellipsoid in the normalized device coordinate space according to the world camera transformation matrix, the Gaussian ellipsoid position in the world coordinate system, and the principal axis direction with the largest scaling factor includes: Using the world camera transformation matrix, the position of the Gaussian ellipsoid in the world coordinate system is converted to the camera space to obtain the position of the Gaussian ellipsoid in the camera space coordinate system; Using the world camera transformation matrix, transform the principal axis direction with the largest scaling factor into the camera space to obtain the principal axis direction vector in the camera coordinate system; The position of the Gaussian ellipsoid in the camera space coordinate system and the principal axis direction vector in the camera coordinate system are projected into the normalized device coordinates to obtain the projection position of the Gaussian ellipsoid in the normalized device coordinate space.

7. The infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering according to claim 4, characterized in that: The expression of the loss function is: Where L is the loss function, λ D-SSIM is the weight coefficient for controlling the structural similarity loss, λ COS-SIM is the weight coefficient for controlling the cosine similarity loss, L1 is the absolute difference between the two-dimensional image and the infrared image data corresponding to the camera pose, and L D-SSIM is the loss function based on the differentiable structural similarity index, L COS-SIM is the cosine similarity loss of the thermal gradient, U is the set of valid Gaussian ellipsoids, is the principal axis projection direction of the Gaussian ellipsoid of the i′th effective Gaussian principal axis under the current infrared camera pixel coordinate projection, is the thermal gradient direction of the thermal gradient tensor projected from the i′th effective Gaussian ellipsoid to the current infrared camera pixel coordinate position, and ||·|| is the modulus.

8. The infrared scene reconstruction and rendering method based on three-dimensional Gaussian splattering according to claim 1, characterized in that: The method of processing the trained three-dimensional Gaussian point cloud model using three-dimensional graphics engine software and a 3D Gaussian rasterizer to obtain an infrared simulation image includes: Using the three-dimensional graphics engine software, cropping the trained three-dimensional Gaussian point cloud model to obtain a cropped three-dimensional Gaussian point cloud model; The cropped three-dimensional Gaussian point cloud model is rendered using the 3D Gaussian rasterizer to obtain the infrared simulation image.

9. An infrared scene reconstruction and rendering device based on three-dimensional Gaussian splashing, characterized in that: include: A preprocessing module, used for preprocessing infrared image data using a Gaussian filtering method; A construction module is used to process the preprocessed infrared image data using a structure-from-motion algorithm to obtain a sparse three-dimensional point cloud and a camera pose, and to set a Gaussian ellipsoid at the center of the sparse three-dimensional point cloud to construct a three-dimensional Gaussian point cloud model; A rendering module, configured to input the camera pose into a Gaussian differentiable rasterizer renderer to render the three-dimensional Gaussian point cloud model to obtain a two-dimensional image of a corresponding perspective; a training module for training the three-dimensional Gaussian point cloud model by comparing the two-dimensional image with the infrared image data corresponding to the camera pose using a loss function to obtain a trained three-dimensional Gaussian point cloud model, wherein the loss function is a function of a temperature gradient field of the preprocessed infrared image data, a projection position of the Gaussian ellipsoid on a pixel plane at a current camera viewing angle, and a principal axis projection direction of the Gaussian ellipsoid; The processing module is used to process the trained three-dimensional Gaussian point cloud model using three-dimensional graphics engine software and a 3D Gaussian rasterizer to obtain an infrared simulation image.

10. The infrared scene reconstruction and rendering device based on three-dimensional Gaussian splattering according to claim 9, characterized in that: The device further comprises: a temperature gradient field determination module, configured to input the camera pose into a Gaussian differentiable rasterizer to render the three-dimensional Gaussian point cloud model, obtain a two-dimensional image of the corresponding viewing angle, and then use a Sobel operator to perform discrete differential operations on the preprocessed infrared image data to obtain the temperature gradient field, wherein the temperature gradient field includes a horizontal temperature gradient field and a vertical temperature gradient field; a thermal gradient direction determination module, configured to determine the thermal gradient direction based on the horizontal temperature gradient field, the vertical temperature gradient field, a preset threshold, and the center position corresponding to the Gaussian ellipsoid; A first projection position determination module is used to determine the projection position of the Gaussian ellipsoid in the normalized device coordinate space according to the world camera transformation matrix, the position of the Gaussian ellipsoid in the world coordinate system, and the direction of the principal axis with the largest scaling factor; a second projection position determining module, configured to determine a projection position of the Gaussian ellipsoid on a pixel plane under a current camera viewing angle according to the projection position of the Gaussian ellipsoid in the normalized device coordinate space; The projection direction determination module is used to determine the main axis projection direction of the Gaussian ellipsoid according to the projection position of the Gaussian ellipsoid on the pixel plane under the current camera viewing angle and the projection position of the Gaussian ellipsoid in the normalized device coordinate space.

Images