Three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothness constraint

Abstract

The invention discloses a three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraints, and belongs to the field of computer vision technology. The method first aligns the color-temperature image pairs under the same viewing angle of the same three-dimensional scene, and then generates an enhanced image by mixing or superimposing contour lines. The enhanced image is kinematically constructed to generate a point cloud and a camera pose. Two independent Gaussian models are then created, and point cloud data and camera pose are input to render a temperature image and a color image, respectively. A loss function is set for the Gaussian model of the temperature mode and the color mode, and a smoothing term is introduced in the loss function of the temperature mode. A regularization coefficient is set to combine the loss functions of the temperature mode and the color mode. The overall loss and gradient are calculated using the total loss function, and back propagated, and the various parameters of the model are optimized by adaptive density control. Finally, the trained model is used to complete the reconstruction of the three-dimensional scene.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and relates to three-dimensional reconstruction of scenes, and in particular to a three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraints. Background Art

[0002] In the field of computer vision and 3D reconstruction, the reconstruction of 3D temperature field is a very challenging technical problem. Its core goal is to generate accurate 3D temperature distribution through 2D temperature images, so as to provide authentic and reliable temperature information for virtual reality, thermal imaging monitoring, environmental monitoring and other fields. In building thermal analysis, industrial inspection, medical imaging and environmental simulation, 3D temperature field reconstruction can help better understand the temperature distribution on the surface of objects, discover potential abnormal hot spots, and improve overall safety and efficiency.

[0003] Currently, 3D temperature field reconstruction is usually modeled and rendered using point cloud, voxel or neural radiance field (NeRF) methods. Although these traditional methods have achieved good results in processing 3D geometric shapes, they have some significant limitations when processing temperature data. First, thermal images usually lack obvious textures and details, which makes it extremely challenging to extract accurate geometric information from them. In addition, since most objects are in thermal equilibrium, their temperature distribution is relatively smooth. Unlike the drastic changes that may exist in color images, the changes in temperature images are usually gradual. Traditional methods often ignore the smoothness characteristics of temperature data in the process of processing temperature images.

[0004] In addition, existing 3D temperature field reconstruction methods also face huge demands for storage space. Whether it is a point cloud or voxel-based reconstruction method, it is necessary to process massive 3D data. The increase in the amount of data not only increases the computational complexity, but also increases the burden of storage and transmission. Especially in some real-time systems or resource-constrained devices, how to reduce the demand for storage and computing resources while ensuring reconstruction accuracy has become a technical problem that needs to be solved urgently.

[0005] As an emerging 3D reconstruction method, 3D Gaussian Splatting (3DGS) has shown superior performance. It uses Gaussian distribution to represent points in a 3D scene, which not only provides efficient rendering effects, but also enables accurate reconstruction of the scene through explicit geometry and color distribution modeling. However, the existing 3DGS framework is mainly optimized for color mode, and the temperature mode is relatively less processed, making it difficult to effectively deal with the lack of texture in temperature images. Therefore, although 3DGS has made significant progress in the reconstruction of color scenes, it still has some shortcomings in the reconstruction of temperature fields. In addition, the 3DGS framework also faces challenges in the process of multimodal fusion and optimization. How to improve the reconstruction effect of the temperature mode without sacrificing the reconstruction accuracy of the color mode is still an urgent problem to be solved. Summary of the invention

[0006] In view of the shortcomings of the existing technology, the present invention proposes a 3D scene reconstruction method based on multimodal regularization and temperature smoothness constraint. By adding a smoothness loss function to improve the reconstruction accuracy and effect of the 3D temperature field, a multimodal regularization strategy is introduced to significantly reduce storage requirements, improve the overall performance of reconstruction, and optimize storage efficiency. This solves the problem that the current 3D temperature field reconstruction process takes up a lot of storage space and the reconstruction effect needs to be improved.

[0007] The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint specifically includes the following steps:

[0008] Step 1: Use a visible light camera and a thermal imaging camera to shoot the same three-dimensional scene from multiple perspectives to obtain color-temperature image pairs under multiple perspectives, and then align the color image and temperature image from the same image pair to eliminate the error caused by the imperfect overlap of the positions of the visible light camera and the thermal imaging camera.

[0009] Then the registered color image and temperature image To mix:

[0010]

[0011] in Represented as the mixed image, β∈[0,1], represents the opacity of the temperature image. Then the generated mixed image Perform motion construction to generate sparse point clouds and camera poses.

[0012] Preferably, the opacity β of the temperature image is set to 0.5, that is, the color image and the thermal image are mixed in a ratio of 50%:50%.

[0013] Preferably, edge contours in the color image are extracted and superimposed on the temperature image to generate a detail-enhanced thermal image, and motion construction (SfM, Structure from Motion) is performed on the detail-enhanced thermal image to generate a sparse point cloud and camera pose.

[0014] Step 2: Create two independent Gaussian models for training color scenes and temperature scenes respectively. Initialize the sparse point cloud obtained in step 2 as a Gaussian sphere and store it in the two Gaussian models.

[0015] Step 3: Design loss functions for color mode and temperature mode respectively. Loss function for color mode for:

[0016]

[0017] in is the mean absolute value error loss function between the real image and the rendered image, is the structural similarity loss function.

[0018] Temperature mode loss function for:

[0019]

[0020] in Represents the smoothness loss function:

[0021]

[0022] Represents the rendering temperature value at the pixel position (i, j), width represents the width of the image, height represents the height of the image, M=(width-2)×(height-2) represents the total number of pixels in the image that need to calculate the smoothness loss, , ∈[0,1], which are the weight coefficients of different loss terms.

[0023] Step 4: Define the multimodal regularization coefficient :

[0024]

[0025] in, and Represents the number of Gaussian spheres generated by temperature mode and color mode respectively.

[0026] Using multimodal regularization coefficients The loss function of the color scene and temperature scene designed in step 3 and The two Gaussian models are jointly optimized by backpropagation. The total loss function is for:

[0027]

[0028] Step 5: Using the camera pose obtained in step 1, project the two Gaussian balls corresponding to the color scene and the temperature scene onto a two-dimensional plane from the same viewing angle, and input the projection results of the two modes into the multimodal rasterizer for rendering to generate the corresponding color image. and temperature image :

[0029]

[0030]

[0031] in, represents the pixel position of the query, Represents the number of Gaussian balls in the two-dimensional plane corresponding to the pixel, represents the opacity of the kth Gaussian sphere, and They represent color information and temperature information respectively, and are determined by the spherical harmonics of the kth Gaussian sphere.

[0032] Then the rendered color image and temperature image They are compared with the real color image and the real temperature image under the same viewing angle, and the total loss function is used to calculate the The overall loss and gradient of the entire multimodal scene are calculated, back-propagated to the Gaussian model, and the parameters of the Gaussian ball are optimized through adaptive density control to complete the training of the multimodal Gaussian model.

[0033] Finally, the color-temperature image pair of a three-dimensional scene at one perspective is input into the trained Gaussian model to obtain the rendered color image and temperature image, thus achieving efficient reconstruction of the three-dimensional scene.

[0034] The present invention has the following beneficial effects:

[0035] 1. In view of the physical property that the temperature in an object is always conducting heat and making its own temperature consistent with the surrounding temperature, a smoothness loss function is designed to constrain the thermal modal image and assist in the 3D reconstruction of the temperature field, thereby improving the reconstruction effect. By introducing a multimodal regularization coefficient, it can not only avoid the problem of overfitting of one modality and underfitting of another modality, but also greatly reduce the storage cost of the model.

[0036] 2. In view of the fact that the temperature image taken of a scene lacks the texture features of the scene and it is difficult to achieve the camera positioning task alone, a multimodal image that fuses the color image and the temperature image is designed. The texture features of the color image are added while retaining the thermal infrared features of the scene. The fused multimodal image can more easily achieve camera positioning.

[0037] 3. In view of the fact that traditional three-dimensional scene reconstruction methods can only achieve single-modality scene reconstruction, a multi-modal joint training of temperature scenes and color scenes is designed to achieve three-dimensional scene reconstruction. The joint training enables the two modal scenes to be constrained by the other modality during the training process, thereby achieving better reconstruction effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The color image and temperature image of the black and white checkerboard calibration plate collected in the embodiment;

[0039] Figure 2 is a schematic diagram of an image pair after registration in an embodiment;

[0040] Figure 3 A schematic diagram of a hybrid image and a detail-enhanced thermal image generated in an embodiment;

[0041] Figure 4 The flowchart of the three-dimensional scene reconstruction method based on multi-modal regularization and temperature smoothness constraint is shown;

[0042] Figure 5 Schematic diagram of color images and temperature images of 10 scenes collected in the embodiment;

[0043] Figure 6 This is a comparison of the color images rendered after training the four methods in the ablation experiment;

[0044] Figure 7 Comparison of temperature images rendered after training using four different methods in the ablation experiment. DETAILED DESCRIPTION

[0045] The present invention will be further explained below with reference to the accompanying drawings;

[0046] The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint specifically includes the following steps:

[0047] Step 1: Image Registration

[0048] s1.1. Use an infrared heater to heat the black and white checkerboard calibration plate used for visible light registration, such as Figure 1As shown in the figure, the black checkerboard shows a relatively high temperature because it can absorb more thermal radiation. Before the calibration plate reaches thermal equilibrium with the surrounding environment, a visible light camera and a thermal infrared camera are used to simultaneously capture color images and temperature images at the same viewing angle.

[0049] s1.2. Get the coordinates of the pixel points at the corner points of the chessboard in the color image and the coordinates in the temperature image , according to the relationship between the coordinates and the essential matrix E , estimate the essential matrix E. Since the essential matrix E has only 5 degrees of freedom, the essential matrix E can be estimated by randomly selecting the coordinates of 8 corner points in the image and listing 8 equations.

[0050] s1.3. Perform singular value decomposition on the essential matrix E ,in is a singular value matrix, U and V are orthogonal matrices. , we know that the rotation matrix ,in is the antisymmetric matrix of the translation vector t, which represents the outer product matrix of the translation vector. Then calculate the centroid of the color image and the centroid of the temperature image , solve for the translation vector .

[0051] s1.4. Using the intrinsic parameters of the visible light camera , Thermal infrared camera internal parameters As well as the rotation matrix R and translation matrix t from the thermal infrared camera to the visible light camera, the coordinates of each pixel in the temperature image are Mapping onto a color image:

[0052]

[0053] The new coordinates calculated The pixels in the temperature image are mapped to the color image to achieve registration of the temperature image and the color image.

[0054] Step 2: Figure 2 As shown, firstly, a visible light camera and a thermal imaging camera are used to shoot the same scene from multiple perspectives to obtain color-temperature image pairs from multiple perspectives, and then the color-temperature image pairs are aligned according to the method in step one.

[0055] Due to the low texture and ghosting characteristics of pure temperature images, it is difficult to extract and match enough feature points in the subsequent process. Figure 2 The color image after registration shown in (a) and the temperature image shown in (b) To mix:

[0056]

[0057] in The mixed image is represented by β∈[0,1], which represents the opacity of the temperature image. In this embodiment, β=0.5 is set, that is, the color image and the thermal image are mixed in a ratio of 50%:50%, and the following is generated: Figure 3 The mixed image shown in (a).

[0058] The generated mixed image is then subjected to motion construction to generate sparse point clouds and camera poses, providing accurate scene geometry information and laying the foundation for subsequent Gaussian model initialization and projection rendering.

[0059] In one embodiment, edge contours in the color image are extracted and superimposed on the temperature image to generate Figure 3 The detail enhanced thermal image shown in (b) is subjected to motion construction.

[0060] Step 3: Figure 4 As shown in Figure 1, two independent Gaussian models are created for training color scenes and temperature scenes respectively. The sparse point cloud obtained in step 2 is initialized as a Gaussian sphere and stored in the two Gaussian models.

[0061] Each Gaussian sphere contains four key attributes: coordinates, covariance matrix, opacity, and spherical harmonics. The coordinates represent the position of the Gaussian sphere in the three-dimensional scene, the covariance matrix determines the shape of the Gaussian sphere, and the opacity and spherical harmonics work together to represent the color information of the Gaussian sphere, which is used to generate accurate color images and temperature images.

[0062] Step 4: Design loss functions for color mode and temperature mode respectively. Loss function for color mode for:

[0063]

[0064] in is the mean absolute value error loss function between the real image and the rendered image, is the structural similarity loss function.

[0065] Temperature mode loss function for:

[0066]

[0067] Due to the gentleness of heat conduction and radiation processes, temperature images usually do not change dramatically, and objects in thermal equilibrium have similar temperatures in most areas, which makes them appear smoother on the image. Therefore, the loss function of the temperature mode adds a smoothness loss function based on the loss function of the color mode. :

[0068]

[0069] The smoothness loss is obtained by calculating the average absolute value error between a pixel and the four surrounding pixels above, below, left and right. Represents the rendering temperature value at the pixel position (i, j), width represents the width of the image, height represents the height of the image, and M represents the total number of pixels in the image that need to calculate the smoothness loss. Since the pixels at the edge of the image are not calculated with this point as the center point, M=(width-2)×(height-2). , ∈[0,1], which are weight coefficients of different loss terms. In this embodiment, =0.2, =0.6.

[0070] Step 5: Loss function for color scene and temperature scene designed in step 4 and Based on , these two losses are combined for back propagation to jointly optimize the two Gaussian models. In order to avoid the problem of imbalanced weights between different modalities when training multimodal data, which causes one modality to dominate the training process and the other modality to be ignored, resulting in an imbalance in the number of generated Gaussian distributions, a dynamically adjusted multimodal regularization strategy is introduced. By dynamically monitoring the number of Gaussian balls generated by each modality during training, the weight of each modality loss is adjusted to balance the contribution of each modality during training. The multimodal regularization coefficient is defined :

[0071]

[0072] in, and Represents the number of Gaussian balls generated by temperature mode and color mode respectively. Introducing multimodal regularization coefficient The total loss function for:

[0073]

[0074] Through the multimodal regularization coefficient , so that when the number of Gaussians generated by a certain mode increases, the training weight of another mode increases accordingly, to ensure the balance between different modes, avoid the dominance of a single mode, effectively prevent the overfitting of a single mode, and ensure that both the color model and the temperature mode can be fully optimized, thereby improving the reconstruction effect of the entire model.

[0075] Step 6: Using the camera pose obtained in step 2, project the two Gaussian balls corresponding to the color scene and the temperature scene onto a two-dimensional plane from the same viewing angle to ensure that the viewing angle of the projection is consistent with that of the real image, and input the projection results of the two modes into the multimodal rasterizer for rendering to generate the corresponding color image. and temperature image :

[0076]

[0077]

[0078] in, represents the pixel position of the query, Represents the number of Gaussian balls in the two-dimensional plane corresponding to the pixel, represents the opacity of the kth Gaussian sphere, and They represent color information and temperature information respectively, and are determined by the spherical harmonics of the kth Gaussian sphere.

[0079] Then the rendered color image and temperature image They are compared with the real color image and the real temperature image under the same viewing angle, and the total loss function is used to calculate the The overall loss and gradient of the entire multimodal scene are calculated, back-propagated to the Gaussian model, and the various parameters of the Gaussian ball are optimized through adaptive density control. For areas that are not well-represented during the reconstruction process, that is, the volume of the Gaussian ball in the area cannot fill the area, one or more new Gaussian balls with the same properties as the original Gaussian ball are copied, and these new Gaussian balls are moved along the position gradient direction to fill the area. For over-reconstructed areas, that is, the current Gaussian ball volume is larger than the area to be filled, it is split into multiple small Gaussian balls of the same size, so that these split small Gaussian balls inherit the properties of the original Gaussian ball and are distributed around the location of the original Gaussian ball, thereby achieving a more detailed reconstruction of the area. In this way, by continuously iterating the above steps, the training of the multimodal Gaussian model is completed.

[0080] Finally, the color-temperature image pair of a three-dimensional scene at one perspective is input into the trained Gaussian model to obtain the rendered color image and temperature image, thus achieving efficient reconstruction of the three-dimensional scene.

[0081] In order to illustrate the positive impact of introducing the smoothness loss function and the multimodal regularization coefficient on the reconstruction results, an ablation experiment was carried out to compare the three-dimensional temperature field reconstruction results of this method with the original 3DGS in 10 different scenes. Figure 5 These are color images and temperature images of 10 scenes collected in this embodiment.

[0082] The color images and temperature images rendered by the four training methods are shown as follows: Figure 6 , 7 As shown, (a) represents the image rendered after training using the 3DGS method, (b) represents the image rendered after training using the multimodal joint training (MMGS) method, and (c) represents the use of MMGS to add temperature smoothing constraints The image rendered after the method is trained. (d) represents the method proposed in this application, that is, MMGS adds temperature smoothing constraints and the multimodal regularization coefficient The image rendered after the method is trained. At the same time, four indicators, including peak signal-to-noise ratio (PSNR), structural similarity (SSIM), perceptual difference (LPIPS), and model footprint (Mem), are selected for comparison and evaluation, as shown in Table 1:

[0083] Table 1

[0084]

[0085] Combining the images and quantitative indicators rendered by the four methods, it can be seen that this method can significantly improve the reconstruction effect of 3D color scenes and temperature scenes by adopting multimodal joint training. The PSNR of the color model is improved by 0.95dB, and the PSNR of the temperature model is improved by 0.85dB. When the temperature smoothing constraint is introduced, the PSNR of the color model is improved by 0.95dB. The three-dimensional temperature site reconstruction effect can be further improved. The PSNR of the temperature model is improved by 0.22dB. Then the multimodal regularization coefficient is increased. It can also significantly reduce the storage requirements of the Gaussian model with minimal reconstruction quality loss. For color scenes, the storage space is reduced by about 94% compared to the 3DGS method, and for temperature scenes, the storage space is reduced by about 86%.

Claims

1. A three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraints uses a visible light camera and a thermal imaging camera to shoot the same three-dimensional scene from multiple perspectives to obtain color-temperature image pairs under multiple perspectives, then aligns the color image and temperature image from the same image pair, and reconstructs the three-dimensional scene based on the aligned image pair, which is characterized by: The specific steps are as follows: Step 1: For the registered color image I RGB and temperature image I Th Mix them, and then generate the mixed image I mix Perform motion construction to generate sparse point clouds and camera poses; Step 2: Create two independent Gaussian models, one for training the color scene and the other for training the temperature scene; initialize the sparse point cloud obtained in step 2 as a Gaussian sphere and store it in the two Gaussian models; Step 3: Set the loss function for color mode and the temperature modal loss function They are: in is the mean absolute value error loss function between the real image and the rendered image, is the structural similarity loss function, λ, λ smooth ∈[0,1], respectively, are the weight coefficients of different loss terms; represents the smoothness loss function; Step 4: Define the multimodal regularization coefficient N Thermal and N RGB Represents the number of Gaussian balls generated by temperature mode and color mode respectively; sets the total loss function for: Step 5: Using the camera pose obtained in step 1, project the two Gaussian balls corresponding to the color scene and the temperature scene onto a two-dimensional plane from the same viewing angle, and input the projection results of the two modes into the multimodal rasterizer for rendering to generate the corresponding color image C(x ′ ) and temperature image T(x ′ ); then the rendered color image C(x ′ ) and temperature image T(x ′ ) are compared with the real color image and the real temperature image under the same viewing angle, and the total loss function is used to calculate the Calculate the overall loss and gradient of the entire multimodal scene, back-propagate to the Gaussian model, optimize the parameters of the Gaussian ball through adaptive density control, and complete the training of the multimodal Gaussian model; Finally, the color-temperature image pair of the three-dimensional scene at one perspective is input into the trained Gaussian model to obtain the rendered color image and temperature image, thus realizing the reconstruction of the three-dimensional scene.

2. The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint as claimed in claim 1, characterized in that: The method of image registration is: s1.

1. Use an infrared heater to heat the black and white checkerboard calibration plate used for visible light registration. Before the calibration plate reaches thermal equilibrium with the surrounding environment, use a visible light camera and a thermal infrared camera to simultaneously capture the color image and temperature image at the same viewing angle. s1.

2. Get the coordinates X of the pixel points at the corner points in the chessboard in the color image i =[x i ,y i ,1] and the coordinate X′ in the temperature image i =[x′ i ,y′ i ,1], according to the relationship between the coordinates and the essential matrix E Estimate the essential matrix E; s1.

3. Perform singular value decomposition of the essential matrix E = U∑V T , where Σ is a singular value matrix, U and V are orthogonal matrices; according to R = UV T Solve the rotation matrix and then calculate the center of mass of the color image and the centroid of the temperature image Solve for the translation vector t = p - Rp′; s1.

4. Using the intrinsic parameter K of the visible light camera RGB , the internal parameter K of the thermal infrared camera Th As well as the rotation matrix R and translation matrix t from the thermal infrared camera to the visible light camera, the pixel coordinates (u Th ,v th ) is mapped to a color image: According to the calculated new coordinates (u RGB ,v RGB ) The pixels in the temperature image are matched to the color image to achieve the registration of the temperature image and the color image.

3. The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint as claimed in claim 1, characterized in that: The mixed image I mix =βI Th +(1-β)I RGB , β∈[0,1], represents the opacity of the temperature image.

4. The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint as claimed in claim 3, characterized in that: Set the opacity of the temperature image β = 0.

5.

5. The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint as claimed in claim 1, characterized in that: In step one, edge contours in the color image are extracted and superimposed on the temperature image to generate a detail-enhanced thermal image, and motion construction is performed on the detail-enhanced thermal image to generate a sparse point cloud and camera pose.

6. The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint according to any one of claims 1 to 5, characterized in that: The smoothness loss function Where T i,j Represents the rendering temperature value at the pixel position (i, j), width represents the width of the image, height represents the height of the image, and M = (width-2) × (height-2) represents the total number of pixels in the image that need to calculate the smoothness loss.

7. The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint as claimed in claim 6, characterized in that: Set λ=0.2, λ smooth = 0.

6.

8. The three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothing constraint as claimed in claim 6, characterized in that: The method for optimizing the parameters of the Gaussian sphere through adaptive density control is: For the area that is not well represented in the reconstruction process, that is, the volume of the Gaussian sphere in the area cannot fill the area, copy one or more new Gaussian spheres with the same properties as the original Gaussian sphere, and move these new Gaussian spheres along the position gradient direction to fill the area; For the over-reconstructed area, that is, the volume of the current Gaussian sphere is larger than the area to be filled, the Gaussian sphere is split into multiple small Gaussian spheres of the same size. The split small Gaussian spheres inherit the properties of the original Gaussian sphere and are distributed around the location of the original Gaussian sphere. The training of the multimodal Gaussian model is completed by continuously iterating the steps of replication or splitting.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5.

Images