Three-dimensional temperature field reconstruction method based on 3D Gaussian splashing

Abstract

The invention discloses a three-dimensional temperature field reconstruction method based on 3D Gaussian splashing, and the method comprises the steps: firstly carrying out the feature point extraction of an input color image, obtaining a descriptor, and carrying out the feature matching of the extracted features; and then generating a camera pose and realizing reconstruction of sparse point clouds, and finally selecting corresponding strategies to realize three-dimensional temperature field reconstruction according to different requirements. According to the method, three different strategies can be formulated according to different application requirements of three-dimensional reconstruction on storage space, reconstruction speed, reconstruction quality and the like in the actual application process so as to realize reconstruction of the three-dimensional temperature field, and a user can select corresponding strategies to realize reconstruction of the three-dimensional temperature field according to different requirements. The on-demand selection strategy can flexibly adapt to various scene requirements, and the applicability and reconstruction efficiency of the model are improved, so that calculation resources are effectively optimized, the reconstruction precision is ensured, unnecessary calculation consumption is reduced, and the reconstruction effects of the color image and the temperature image are improved at the same time.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and to reconstruction of a three-dimensional temperature field of a scene, and in particular to a three-dimensional temperature field reconstruction method based on 3D Gaussian splashing. According to three different practical application requirements of reconstruction accuracy, reconstruction speed and memory required for reconstruction, three different training strategies are adopted to realize the reconstruction of the three-dimensional temperature field. Background Art

[0002] 3D reconstruction technology is one of the key applications in the field of computer vision and computer graphics. With the improvement of its reconstruction accuracy and speed and the reduction of memory required for 3D reconstruction, it is widely used in virtual reality (VR), augmented reality (AR), unmanned driving, medical imaging, game modeling, cultural heritage protection and other fields. In recent years, Neural Radiance Fields (NeRF) has become an important breakthrough in this field by reconstructing realistic 3D scenes from multi-view images. Its unique implicit scene representation capability has been highly praised, especially in generating realistic renderings from new perspectives. However, the main limitation of NeRF is that the rendering speed is slow, which is difficult to meet real-time requirements, and its implicit scene representation makes scene editing and operation relatively difficult.

[0003] 3D Gaussian Splatting (3DGS) is an emerging technology in the field of 3D reconstruction that aims to overcome the above limitations of NeRF. 3DGS represents the scene through millions of explicit Gaussians. It also allows fast and efficient rasterization rendering through sputtering, greatly improving the efficiency of real-time rendering and giving users the ability to interact and edit 3D scenes. This makes 3DGS an effective alternative to NeRF, especially for applications that require efficient rendering and fast operations, such as games, virtual reality, and augmented reality. However, despite its remarkable achievements in color image reconstruction, 3DGS still has shortcomings in multimodal data fusion, especially temperature field reconstruction.

[0004] Temperature field reconstruction involves recovering the three-dimensional temperature distribution of an object or scene from infrared thermal imaging images, which is crucial for applications such as industrial monitoring, fire detection, and building thermal analysis. Although 3DGS can successfully integrate color images and depth data for three-dimensional reconstruction, its performance in temperature field data processing has not yet reached the same level. Although the explicit Gaussian representation of 3DGS can effectively process RGB images and geometric forms, it faces the following challenges when fusing data from temperature images. Many challenges are mainly reflected in its limited support for multimodal data (especially temperature images). Although it has demonstrated strong capabilities in real-time rendering and color scene reconstruction, its reconstruction application for thermal scenes is still immature. At present, 3DGS still needs further research and technical improvements in temperature scene reconstruction to improve its adaptability and accuracy in multimodal scenes. Summary of the invention

[0005] Although the existing 3DGS technology performs well in the three-dimensional reconstruction of color images, it still has significant deficiencies when processing multimodal data, especially temperature images, and it is difficult to achieve the reconstruction of the three-dimensional temperature field. In view of this limitation, the present invention provides a three-dimensional temperature field reconstruction method based on 3D Gaussian splashing, which adopts three innovative strategies according to different needs, aiming to simultaneously improve the reconstruction effect of color images and temperature images and reduce storage space, thereby achieving a more comprehensive multimodal scene reconstruction.

[0006] A three-dimensional temperature field reconstruction method based on 3D Gaussian splashing includes the following steps:

[0007] Step 1: Extract feature points from the input color image and obtain descriptors.

[0008] Step 2: Perform feature matching on the extracted features.

[0009] Step 3: Generate camera pose and reconstruct sparse point cloud.

[0010] Step 4: Select corresponding strategies to achieve 3D temperature field reconstruction according to different requirements:

[0011] When there is a demand for reconstruction speed and a color model is already available, strategy 1 is selected to quickly reconstruct the temperature field with high fidelity: first, the Gaussian attributes of the color model are assigned to the newly created Gaussian model used to represent temperature, and then the input temperature image is trained to achieve reconstruction of the three-dimensional temperature field.

[0012] When there is a need for storage space for reconstruction, strategy 2 is selected to achieve high-fidelity 3D temperature field reconstruction while achieving high-fidelity 3D reconstruction of color scenes, and the storage space of the two models is greatly reduced compared to the other two strategies: two Gaussian models are initialized and used to train color scenes and temperature scenes at the same time.

[0013] When pursuing the reconstruction quality of the three-dimensional reconstruction of temperature scenes and color scenes, strategy three is selected: reconstruct both the color mode and the temperature mode into the same model, and modify the three-dimensional Gaussian sphere properties so that it can save both color and temperature modal information at the same time to achieve the reconstruction of the three-dimensional temperature field.

[0014] The beneficial effects of the present invention are as follows:

[0015] The method of the present invention can formulate three different strategies to achieve the reconstruction of the three-dimensional temperature field according to different application requirements of three-dimensional reconstruction in actual application, such as storage space, reconstruction speed, and reconstruction quality. Users can choose corresponding strategies to achieve the reconstruction of the three-dimensional temperature field according to different requirements. This on-demand selection strategy can flexibly adapt to the needs of various scenarios, improve the applicability and reconstruction efficiency of the model, thereby effectively optimizing computing resources, ensuring reconstruction accuracy while reducing unnecessary computing consumption, and improving the reconstruction effect of color images and temperature images at the same time. At the same time, different strategies have their own innovations:

[0016] Strategy 1: In order to quickly reconstruct the temperature field of the scene, we use the finetune strategy in the large model method on the basis of a visible light reconstruction that already has a good geometric structure of the scene, input the temperature map and iteratively optimize the 3D reconstruction results to quickly obtain the 3D temperature field. Compared with NeRF (Neural Radiance Fields), the training time of NeRF is usually more than 24 hours, while our method only takes 355 seconds on average. In addition, compared with strategies 2 and 3, the training time of this strategy is reduced by more than 250 seconds.

[0017] Strategy 2: To make full use of the information of the two modes, when one mode fails, the information of the other mode can be used to assist in the 3D reconstruction of multiple modes. For example, in a dark light scene, the visible light mode fails, but the thermal infrared mode remains the same. This strategy uses the information of the two modes as constraints for 3D reconstruction. At the same time, our experiments found that this reconstruction method will reduce the storage space of the two modes at the same time. By adjusting the hyperparameters, it can not only maintain a good 3D temperature field reconstruction effect, but also greatly reduce the storage space. Its storage space accounts for only 34% of strategy one and only 16% of strategy three.

[0018] Strategy 3: Although the same object may appear very different in different modes, the data of these modes all rely on the same geometric structure. Therefore, in Strategy 3, we apply different modal data to the same Gaussian, that is, each Gaussian has spherical harmonics of the visible light mode and spherical harmonics of the temperature mode. When rendering images of the visible light mode, the visible light mode spherical harmonics are used, and when rendering images of the temperature mode, the temperature mode spherical harmonics are used. Experiments have shown that although this strategy increases the storage space of the model, it achieves the highest reconstruction quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a flowchart of multi-modal fine-tuning Gaussian model reconstruction according to an embodiment of the present invention.

[0020] FIG. 2 is a flowchart of Gaussian model reconstruction of multiple single modalities according to an embodiment of the present invention.

[0021] FIG3 is a flow chart of a single multimodal Gaussian model reconstruction method according to an embodiment of the present invention.

[0022] FIG4 is a comparison diagram of the color image rendered by training and the real color image of scene 1 in the comparative experiment.

[0023] Figure 5 is a comparison of the temperature image rendered by training and the real temperature image in scene 1 in the comparative experiment.

[0024] FIG6 is a comparison diagram of the color image rendered by training and the real color image in the comparative experiment of scene 2.

[0025] Figure 7 is a comparison of the temperature image rendered by training and the real temperature image in scene 2 in the comparative experiment. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0027] A three-dimensional temperature field reconstruction method based on 3D Gaussian splashing includes the following steps:

[0028] Step 1: Extract feature points from the input color image and obtain descriptors;

[0029] For the input image Through the Gaussian kernel function Perform convolution to generate images of different scales:

[0030] ,

[0031] in, Is the image at scale The blurred image below, is a Gaussian function, which is obtained by applying different standard deviations to the original image. Gaussian filter, and get a series of images with different scales. Then calculate the difference Gaussian:

[0032] ,

[0033] Where k is the scale factor, and feature points are extracted by finding local extreme points in differential images at different scales.

[0034] Calculate the feature points Center, 3×1.5 is the magnitude and direction of the gradient of the radius, and the formula for calculating the magnitude of the gradient is as follows:

[0035] ,

[0036] The gradient direction is calculated as follows:

[0037] ,

[0038] The neighborhood of the key point is divided into 4×4 sub-regions, each of which contains gradient information in 8 directions, describing the sub-vector The eigenvalue combination obtained by partitioning the gradient direction histogram in the neighborhood of the key point is obtained: , forming a 4×4×8=128-dimensional descriptor.

[0039] Step 2: Perform feature matching on the extracted features;

[0040] Calculate the descriptor of a given feature point and the descriptor of the feature point set { }'s Euclidean distance:

[0041] ,

[0042] Among them, k represents the number of points for feature matching, and the nearest neighbor distance is calculated and next nearest neighbor distance ,calculate ,if <0.8, the nearest neighbor point is considered to be the matching point of the given feature point. Repeat the above steps until all feature points are matched.

[0043] Step 3: Generate camera pose and reconstruct sparse point cloud;

[0044] Randomly select 8 pairs of homogeneous coordinates of matching points between two images , ,in( ) is the two-dimensional pixel coordinate of the feature point in one of the images, ( ) is the two-dimensional pixel coordinate of the corresponding feature point in another image, and the matching points satisfy , where F is the basic matrix. The basic matrix F is calculated based on 8 pairs of matching points, and the essential matrix is ​​calculated. ,in and are the camera intrinsic parameters of the two images respectively.

[0045] The essential matrix E can be expressed as ,in is the antisymmetric matrix of the translation vector t, representing the outer product matrix of the translation vector, and R is the rotation matrix of the camera. Perform singular value decomposition on the essential matrix E , where U and V are orthogonal matrices, is the singular value matrix, and the rotation matrix is ​​obtained , and then calculate the centroid of the two images respectively and , solve for the translation vector , thereby obtaining the camera's pose. The two images are used as the starting scene to reconstruct the sparse point cloud, and the successfully matched feature points of the two images are added to the three-dimensional scene. The above steps are repeated by continuously adding new perspective images to gradually realize the reconstruction of the sparse point cloud. At the same time, bundle adjustment is used to continuously optimize the camera pose and the three-dimensional coordinates of the point cloud.

[0046] Step 4: Select the following corresponding strategies according to different needs;

[0047] If there is a greater demand for reconstruction speed, and based on the existing color model, such as in the case of fire rescue, 3D color reconstruction of visible light scenes has been achieved, and the fire scene needs to be reconstructed quickly, strategy 1 is used to quickly reconstruct the 3D temperature field with high fidelity. Among them, the color model can be reconstructed by the existing 3DGS method.

[0048] If there is a large demand for the storage space for reconstruction, such as the need to reconstruct the three-dimensional temperature field for large-scale scenes such as factories and cities, use strategy 2 to achieve high-fidelity three-dimensional reconstruction of color scenes and high-fidelity three-dimensional temperature field reconstruction at the same time, and the storage space of the two models is greatly reduced compared to the other two strategies.

[0049] If you pursue the reconstruction quality of the 3D reconstruction of temperature scenes and color scenes, such as needing to reconstruct the 3D temperature field of some devices that are sensitive to temperature and have strict requirements, use strategy three.

[0050] Strategy 1: Multimodal fine-tuning Gaussian model reconstruction method

[0051] The overall flow chart of strategy 1 is shown in Figure 1. A Gaussian model is created to represent the temperature scene and the Gaussian attributes of the existing color model are assigned to the Gaussian model as the initial training model. Each three-dimensional Gaussian ball of the Gaussian model contains four important attributes: the coordinates of the point, the covariance matrix, the opacity, and the spherical harmonic function. The three-dimensional Gaussian ball can be expressed as:

[0052] ;

[0053] Where μ represents the coordinates of the Gaussian sphere, and the initial coordinates of the point are represented by a sparse point cloud; x represents any point in the 3D scene; Σ represents the covariance matrix of the 3D Gaussian, which is defined using the scaling matrix S and the rotation matrix R:

[0054] ;

[0055] After obtaining the initial scene consisting of a three-dimensional Gaussian sphere, it is projected onto a two-dimensional plane and then placed into a fast differentiable rasterizer to render a temperature image:

[0056] ;

[0057] in represents the pixel position of the query, N represents the number of Gaussians in the two-dimensional plane corresponding to the pixel, represents the opacity of each Gaussian, and t represents the temperature color of the Gaussian sphere, which is determined by the spherical harmonic function. Finally, a temperature image with the same viewing angle as the input image is rendered. The temperature image rendered by the fast differentiable rasterizer is compared with the real temperature image at the same viewing angle to calculate the mean absolute error function , N is the total number of pixels in the image, is the first temperature image in the input The pixel value of pixels, The temperature image in the rendering The pixel value of pixels and the structural similarity loss function ,in is the mean pixel value of the input temperature image, is the mean pixel value of the rendered temperature image, is the pixel value variance of the input temperature image, is the pixel value variance of the rendered temperature image, is the covariance of the pixel values ​​of the input temperature image and the rendered temperature image, and is a very small constant introduced to avoid the denominator being 0. and Calculate the temperature scenario loss: , It indicates that the structural similarity loss coefficient can be adjusted according to the actual situation. , and update the gradient for back propagation.

[0058] In the back propagation, the Gaussian sphere is optimized and adjusted through adaptive density, which mainly includes: (1) deleting Gaussian spheres with opacity α less than the threshold; (2) for under-reconstructed areas, that is, when the Gaussian sphere in a certain area cannot fill the area, a Gaussian sphere of the same size is cloned and moved along the position gradient direction; (3) for over-reconstructed areas, that is, the current Gaussian sphere is larger than the area to be reconstructed, it is split. The covariance matrix, opacity, and spherical harmonics of the three-dimensional Gaussian sphere are all modifiable and continuously optimized during iteration. Finally, a more realistic temperature Gaussian model is generated. In this way, the geometric structure of the temperature model can be gradually adjusted to make it more consistent with the real scene.

[0059] Finally, the generated temperature Gaussian model can not only accurately represent the geometric shape of the scene, but also effectively reconstruct the three-dimensional temperature scene, successfully solving the problem of missing temperature image structure in the multimodal data fusion process. This method not only significantly improves the accuracy and detail of temperature field reconstruction, but also provides a new solution for multimodal three-dimensional reconstruction. At the same time, this method only needs to train the temperature scene based on the existing color model, meeting the needs of rapid reconstruction of the three-dimensional temperature field.

[0060] Strategy 2: Multiple single-modal Gaussian model reconstruction method

[0061] The overall process of strategy 2 is shown in Figure 2. After obtaining the camera pose and sparse point cloud in step 3, the sparse point cloud is initialized as a three-dimensional Gaussian sphere and stored in the two Gaussian models. At this point, the content and information in the two Gaussian models are completely consistent. However, in the next step, one Gaussian model is used as a color Gaussian model for training color images, and the other Gaussian model is used as a temperature Gaussian model specifically for training thermal temperature images.

[0062] Through this dual-model training strategy, the reconstruction effects of color images and thermal images in their respective modalities can be optimized separately, thereby achieving more accurate multimodal three-dimensional scene reconstruction.

[0063] Different from the method of sequentially training the color Gaussian model and the temperature Gaussian model in strategy one, strategy two adopts a parallel training method. In this strategy, the color Gaussian model and the temperature Gaussian model first project the three-dimensional Gaussian ball onto a two-dimensional plane at the same viewing angle. This step ensures the consistency of the viewing angle of the two projected images. Then, the two projections are rendered in the fast differentiable rasterizer respectively to generate a color image and a temperature image at the same viewing angle. Subsequently, the generated images are compared with the corresponding real color images and temperature images, and their respective losses are calculated.

[0064] In this strategy, the loss for color scenes is calculated separately and temperature scenario losses :

[0065] ,

[0066] ,

[0067] Combine the two into an overall loss function:

[0068] ),

[0069] Different from strategy 1, where each loss only affects the corresponding Gaussian model, this strategy applies the overall loss to both the color and temperature Gaussian models. The two models are iteratively optimized via adaptive density control so that each model is not only affected by its own mode but also benefits from the other mode.

[0070] This joint optimization method enhances the interaction between color and thermal modalities, thereby improving the training effect. In addition, since the two modalities jointly optimize the scene from different angles, the number of redundant points is further reduced, the point cloud is streamlined, and the storage requirements of the model are reduced. Ultimately, this strategy achieves 3D temperature scene reconstruction with lower storage costs while maintaining high accuracy.

[0071] Strategy 3: Single multimodal Gaussian model reconstruction method

[0072] This strategy extends the two single-modal Gaussian model reconstruction methods in Strategy 2. It not only retains the dual-modal loss constraints, but also integrates the two modalities of color and temperature into the same Gaussian model to obtain a multimodal three-dimensional Gaussian sphere. Each three-dimensional Gaussian sphere contains four important attributes: the coordinates of the point, the covariance matrix, the opacity, and the spherical harmonic function. On this basis, a spherical harmonic function for storing temperature information is added to each three-dimensional Gaussian sphere, so that a single Gaussian model can simultaneously render color images and temperature images. The complete flowchart is shown in Figure 3.

[0073] Similarly, the camera pose and sparse point cloud generated in step 3 are used to create a Gaussian model, and the sparse point cloud is initialized as a three-dimensional Gaussian sphere. At this stage, the generated three-dimensional Gaussian sphere not only has a unified pose, covariance matrix, and opacity parameter, but also includes two sets of spherical harmonics, which are used to represent the color of the color scene and the color of the temperature scene respectively. This design enables the same Gaussian model to simultaneously process and render diverse information from different modalities, improving the expressiveness and efficiency of the model.

[0074] Through this strategy, color and temperature data can be jointly expressed in the same model, allowing for more efficient reconstruction of multimodal scenes.

[0075] Similar to the first two strategies, this strategy first projects the 3D Gaussian sphere onto a 2D plane at a specified viewing angle, and then renders it through a fast differentiable rasterizer. For rendering of color images, the following rendering formula is followed:

[0076] ;

[0077] Among them, c is determined by the spherical harmonics representing color information. In this strategy, in addition to the color spherical harmonics, the model also contains another set of spherical harmonics for temperature images. Therefore, replacing c in the formula with t representing temperature information, the rendering formula for the temperature image is:

[0078] ;

[0079] Since the two Gaussian models share the same coordinates, covariance matrix, and opacity, the color image and temperature image are rendered at the same viewpoint and shape, ensuring the geometric consistency of the two. Next, the generated color image and temperature image are compared with their corresponding real images and their respective losses are calculated. and . And calculate the overall loss ), and continuously optimize the Gaussian model through back propagation and adaptive density control, and finally realize the reconstruction of the three-dimensional temperature field.

[0080] Since this strategy uses only one bimodal Gaussian model, it no longer needs to optimize the coordinates, covariance matrix, opacity and spherical harmonics of two sets of three-dimensional Gaussian balls separately as in Strategy 2. Instead, it only needs to optimize the coordinates, covariance matrix and opacity of two sets of spherical harmonics and one set of three-dimensional Gaussian balls. This greatly reduces the number of parameters that need to be optimized. This strategy ensures that the color scene and the temperature scene are strictly overlapped in geometry, improving the efficiency and accuracy of the model.

[0081] In one embodiment, the present invention further provides a three-dimensional temperature field reconstruction system based on 3D Gaussian splashing, including a feature extraction module, a feature matching module, a point cloud reconstruction module, a strategy selection module and a three-dimensional temperature field reconstruction module.

[0082] The feature extraction module is used to extract feature points from the input color image and obtain descriptors. The feature matching module is used to perform feature matching on the extracted features. The point cloud reconstruction module is used to generate camera poses and realize the reconstruction of sparse point clouds. The strategy selection module selects the three-dimensional temperature field reconstruction module corresponding to the corresponding strategy according to different requirements to realize the three-dimensional temperature field reconstruction. The strategy selection is as follows:

[0083] When there is a demand for reconstruction speed and a color model is already available, strategy 1 is selected to quickly reconstruct the temperature field with high fidelity. The corresponding three-dimensional temperature field reconstruction module operates as follows: first, the Gaussian attributes of the color model are assigned to the newly created Gaussian model used to represent temperature, and then the input temperature image is trained to achieve reconstruction of the three-dimensional temperature field.

[0084] When there is a demand for storage space for reconstruction, strategy 2 is selected to achieve high-fidelity three-dimensional temperature field reconstruction while achieving high-fidelity three-dimensional reconstruction of color scenes, and the storage space of the two models is greatly reduced compared to the other two strategies. The corresponding three-dimensional temperature field reconstruction module operates as follows: initialize two Gaussian models and use them to train color scenes and temperature scenes at the same time.

[0085] When pursuing the reconstruction quality of the three-dimensional reconstruction of temperature scenes and color scenes, strategy three is selected, and the corresponding three-dimensional temperature field reconstruction module operates as follows: both the color mode and the temperature mode are reconstructed into the same model, and the three-dimensional Gaussian sphere properties are modified so that it can save both color and temperature modal information at the same time to achieve the reconstruction of the three-dimensional temperature field.

[0086] In order to illustrate the advantages and applicability of the three proposed strategies, 10 multimodal scenes were selected for testing and comparison, and the three strategies were compared with the existing 3DGS method. Figure 4 is a comparison of the color image rendered by training and the real color image in scene 1 in the comparison experiment, where (a) represents the real color image, (b) represents the color image rendered by training using the 3DGS method, (c) represents the color image rendered by training using the strategy 2 method, and (d) represents the color image rendered by training using the strategy 3 method; Figure 6 is a comparison of the color image rendered by training and the real color image in scene 2 in the comparison experiment, where (a) represents the real color image, (b) represents the color image rendered by training using the 3DGS method, and (c) represents the color image rendered by training using the strategy 2 method. Figure 5 is a comparison of the temperature image and the real temperature image trained and rendered in the comparative experiment of scene 1, where (a) represents the real temperature image, (b) represents the temperature image trained and rendered using the strategy 1 method, (c) represents the temperature image trained and rendered using the strategy 2 method, and (d) represents the temperature image trained and rendered using the strategy 3 method; Figure 7 is a comparison of the temperature image and the real temperature image trained and rendered in the comparative experiment of scene 2, where (a) represents the real temperature image, (b) represents the temperature image trained and rendered using the strategy 1 method, (c) represents the temperature image trained and rendered using the strategy 2 method, and (d) represents the temperature image trained and rendered using the strategy 3 method. At the same time, the peak signal-to-noise ratio PSNR, structural similarity SSIM, perceptual difference LPIPS, model space occupied Mem and training time Time are selected for comparison and evaluation, as shown in Table 1:

[0087] Table 1

[0088]

[0089] Combining the qualitative comparison of images rendered by different training methods and the quantitative comparison of indicators, 3DGS only supports single-modal training. We train color scenes and temperature scenes separately. However, due to the lack of texture details in temperature images, it is impossible to generate effective camera poses and sparse point clouds in steps one to three. Therefore, the 3DGS method alone cannot achieve the reconstruction of the three-dimensional temperature field. Strategy one is different from 3DGS. It can achieve the reconstruction of the three-dimensional temperature field based on the trained color scene model. Compared with strategies two and three, which must train color scenes and temperature scenes at the same time, this strategy can train the temperature scene alone based on the color model. We trained the temperature scene based on the color model trained by the 3DGS method. The average training time is only 355 seconds, which can well meet the needs of fast reconstruction of the three-dimensional temperature field when the color model is already available. Compared with 3DGS and Strategy 1, the reconstruction effect of Strategy 2 has been further improved, and the trained temperature model only takes up 22MB of storage space, which is much lower than the storage space required by other strategies, and can meet the high demand for reconstruction storage space. In this comparative experiment, Strategy 3 can achieve the most excellent reconstruction effect, which can meet the needs of pursuing high-quality reconstruction of three-dimensional temperature fields and color scenes. This comparative experiment verifies that the reconstruction effects of these three strategies are improved compared with 3DGS, and reflects the relative advantages of each strategy. Users can choose the appropriate strategy according to their needs, which improves the applicability of the model and reduces the waste of resources.

[0090] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0091] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A three-dimensional temperature field reconstruction method based on 3D Gaussian splashing, characterized in that: The steps include: Step 1: Extract feature points from the input color image and obtain the descriptor. Step 2: Perform feature matching on the extracted features; Step 3: Generate camera pose and reconstruct sparse point cloud; Step 4: Select corresponding strategies to achieve 3D temperature field reconstruction according to different requirements: When there is a demand for reconstruction speed and a color model is already available, strategy 1 is selected to quickly reconstruct the temperature field with high fidelity: first, the Gaussian attributes of the color model are assigned to the newly created Gaussian model used to represent temperature, and then the input temperature image is trained to achieve reconstruction of the three-dimensional temperature field; When there is a need for storage space for reconstruction, strategy 2 is selected to achieve high-fidelity 3D reconstruction of color scenes and high-fidelity 3D reconstruction of temperature fields at the same time: initialize two Gaussian models and use them to train color scenes and temperature scenes at the same time; When pursuing the reconstruction quality of the three-dimensional reconstruction of temperature scenes and color scenes, strategy three is selected: reconstruct both the color mode and the temperature mode into the same model, and modify the three-dimensional Gaussian sphere properties so that it can save both color and temperature modal information at the same time to achieve the reconstruction of the three-dimensional temperature field.

2. The three-dimensional temperature field reconstruction method based on 3D Gaussian splashing according to claim 1, characterized in that: Step 1: For the input image Through the Gaussian kernel function Perform convolution to generate images of different scales: ， in, Is the image at scale The blurred image below, is a Gaussian function, which is obtained by applying different standard deviations to the original image. Gaussian filter is used to obtain a series of images with different scales; then the difference Gaussian is calculated: ， Where k is the scale factor, and feature points are extracted by finding local extreme points in differential images at different scales; Calculate the feature points Center, 3×1.5 is the magnitude and direction of the gradient of the radius, and the formula for calculating the magnitude of the gradient is as follows: ， The gradient direction is calculated as follows: ， The neighborhood of the key point is divided into 4×4 sub-regions, each of which contains gradient information in 8 directions, describing the sub-vector The eigenvalue combination obtained by partitioning the gradient direction histogram in the neighborhood of the key point is obtained: , forming a 4×4×8=128-dimensional descriptor.

3. The three-dimensional temperature field reconstruction method based on 3D Gaussian splashing according to claim 2 is characterized in that: Step 2: Calculate the descriptor of a given feature point and the descriptor of the feature point set { }'s Euclidean distance: ， Among them, k represents the number of points for feature matching, and the nearest neighbor distance is calculated and next nearest neighbor distance ,calculate ,if <0.8, the nearest neighbor point is considered to be the matching point of the given feature point; repeat the above steps until all feature points are matched.

4. The three-dimensional temperature field reconstruction method based on 3D Gaussian splashing according to claim 3 is characterized in that: Step 3: Randomly select 8 pairs of homogeneous coordinates of matching points between two images , in( ) is the two-dimensional pixel coordinate of the feature point in one of the images, ( ) is the two-dimensional pixel coordinate of the corresponding feature point in another image, and the matching points satisfy , where F is the basic matrix. The basic matrix F is calculated based on 8 pairs of matching points, and the essential matrix is ​​calculated. ,in and are the camera intrinsic parameters of the two images respectively; The essential matrix E can be expressed as ,in is the antisymmetric matrix of the translation vector t, representing the outer product matrix of the translation vector, R is the rotation matrix of the camera; perform singular value decomposition on the essential matrix E , where U and V are orthogonal matrices, is the singular value matrix, and the rotation matrix is ​​obtained , and then calculate the centroid of the two images respectively and , solve for the translation vector , thereby obtaining the camera's pose; and using these two images as the starting scene to reconstruct the sparse point cloud, adding the successfully matched feature points of the two images to the three-dimensional scene, and continuously adding images from new perspectives to repeat the above steps to gradually achieve the reconstruction of the sparse point cloud; at the same time, bundle adjustment is used to continuously optimize the camera pose and the three-dimensional coordinates of the point cloud.

5. The three-dimensional temperature field reconstruction method based on 3D Gaussian splashing according to claim 4, characterized in that: Strategy 1 is as follows: Create a Gaussian model for representing temperature scenes and assign the Gaussian attributes of the existing color model to the Gaussian model as the initial training model; each three-dimensional Gaussian ball of the Gaussian model contains four important attributes: the coordinates of the point, the covariance matrix, the opacity, and the spherical harmonic function; the three-dimensional Gaussian ball can be expressed as: ； Where μ represents the coordinates of the Gaussian sphere, and the initial coordinates of the point are represented by a sparse point cloud; x represents any point in the 3D scene; Σ represents the covariance matrix of the 3D Gaussian, which is defined using the scaling matrix S and the rotation matrix R: ； After obtaining the initial scene consisting of a three-dimensional Gaussian sphere, it is projected onto a two-dimensional plane and then placed into a fast differentiable rasterizer to render a temperature image: ； in represents the pixel position of the query, N represents the number of Gaussians in the two-dimensional plane corresponding to the pixel, represents the opacity of each Gaussian, t represents the temperature color of the Gaussian sphere, which is determined by the spherical harmonics; finally, a temperature image with the same viewing angle as the input image is rendered, and the temperature image rendered by the fast differentiable rasterizer is compared with the real temperature image at the same viewing angle to calculate the mean absolute error function And the structural similarity loss function ; Combine and Calculate the temperature scene loss and update the gradient for back propagation; Temperature scenario loss, as follows: ； in, represents the structural similarity loss coefficient; In the back propagation, the Gaussian sphere is optimized and adjusted through adaptive density, which mainly includes: (1) deleting Gaussian spheres with opacity α less than the threshold; (2) for under-reconstructed areas, that is, when the Gaussian sphere in a certain area cannot fill the area, a Gaussian sphere of the same size is cloned and moved along the position gradient direction; (3) for over-reconstructed areas, that is, the current Gaussian sphere is larger than the area to be reconstructed, it is split; the covariance matrix, opacity, and spherical harmonic function of the three-dimensional Gaussian sphere are continuously optimized during iteration; and the temperature Gaussian model is finally generated.

6. The three-dimensional temperature field reconstruction method based on 3D Gaussian splashing according to claim 4, characterized in that: Strategy 2 is as follows: After obtaining the camera's pose and sparse point cloud in step 3, the sparse point cloud is initialized as a three-dimensional Gaussian sphere and stored in the two Gaussian models respectively; at this time, the content and information in the two Gaussian models are completely consistent; one Gaussian model is used as a color Gaussian model for training color images, and the other Gaussian model is used as a temperature Gaussian model specifically for training thermal temperature images; A parallel training method is adopted. In this strategy, the color Gaussian model and the temperature Gaussian model first project the three-dimensional Gaussian sphere onto the two-dimensional plane at the same viewing angle. Then, the two projections are respectively rendered in the fast differentiable rasterizer to generate the color image and temperature image at the same viewing angle. Subsequently, the generated images are compared with the corresponding real color image and temperature image to calculate their respective losses. In this strategy, the loss for color scenes is calculated separately and temperature scenario losses : ， ， Combine the two into an overall loss function: ）， The overall loss is applied to both the color and temperature Gaussian models at the same time; during the back propagation process, the overall loss Both models are iteratively optimized via adaptive density control.

7. The three-dimensional temperature field reconstruction method based on 3D Gaussian splashing according to claim 6, characterized in that: Strategy 3 is as follows: Each 3D Gaussian sphere contains four important properties: the coordinates of the point, the covariance matrix, the opacity and the spherical harmonics. On this basis, a spherical harmonics for storing temperature information is added to each 3D Gaussian sphere, so that a single Gaussian model can render both color images and temperature images. Similarly, the camera pose and sparse point cloud generated in step 3 are used to create a Gaussian model, and the sparse point cloud is initialized as a three-dimensional Gaussian sphere. At this stage, the generated three-dimensional Gaussian sphere not only has a unified pose, covariance matrix and opacity parameter, but also includes two sets of spherical harmonic functions, which are used to represent the color of the color scene and the color of the temperature scene respectively. First, the three-dimensional Gaussian sphere is projected onto a two-dimensional plane according to the specified viewing angle, and then rendered by a fast differentiable rasterizer; for the rendering of color images, the following rendering formula is followed: ； Among them, c is determined by the spherical harmonics representing color information; in this strategy, in addition to the color spherical harmonics, the model also contains another set of spherical harmonics for temperature images; therefore, replacing c in the formula with t representing temperature information, the rendering formula for the temperature image is obtained as follows: ； Since the two Gaussian models share the same coordinates, covariance matrix, and opacity, the color image and temperature image are rendered at the same perspective and shape, ensuring the geometric consistency of the two; Next, the generated color image and temperature image are compared with their corresponding real images respectively, and their respective losses are calculated. and ; and calculate the overall loss ), the Gaussian model is continuously optimized through back propagation and adaptive density control, and finally the reconstruction of the three-dimensional temperature field is realized.

8. A three-dimensional temperature field reconstruction method based on 3D Gaussian splashing according to any one of claims 5 to 7, characterized in that: The mean absolute error function is as follows: ； Where N is the total number of pixels in the image. is the first temperature image in the input The pixel value of pixels, The temperature image in the rendering The pixel value of pixels; The structural similarity loss function is as follows: ； in, is the mean pixel value of the input temperature image, is the mean pixel value of the rendered temperature image, is the pixel value variance of the input temperature image, is the pixel value variance of the rendered temperature image, is the covariance of the pixel values ​​of the input temperature image and the rendered temperature image, and It is a very small constant introduced to avoid the denominator being 0.

Images