Three-dimensional point cloud infrared target generation method based on physical mechanism constraint

Abstract

The embodiment of the invention relates to the technical field of computer vision, in particular to a physical mechanism constraint-based three-dimensional point cloud infrared target generation method, which comprises the following steps of: performing sparse reconstruction, expansion and optimization on an infrared target based on a multi-view image and corresponding view position information thereof to obtain a sparse three-dimensional point cloud model; performing deep fusion and infrared radiation information mapping to obtain a three-dimensional model corresponding to the infrared target; creating a 3D Gaussian ellipsoid set based on the pose of the camera and the sparse three-dimensional point cloud model; based on the three-dimensional model corresponding to the infrared target and the 3D Gaussian ellipsoid set, 3D-to-2D rasterization rendering of temperature and emissivity is carried out by using a 3DIRGS algorithm, and a temperature characteristic pattern and an emissivity characteristic pattern are obtained; and inputting the temperature feature map and the emissivity feature map into a CNN network to obtain a radiation brightness field of the infrared target, and further obtaining infrared images of the infrared target in each view angle, thereby realizing the benign balance between the quality and the speed of the three-dimensional point cloud infrared target generation.

Description

Technical Field

[0001] The embodiments of this application relate to the field of computer vision technology, and in particular to a method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints. Background Technology

[0002] In recent years, 3D reconstruction technology has developed rapidly in computer vision and has been widely used in applications such as autonomous driving and intelligent perception. Traditional 3D reconstruction techniques mostly focus on motion structure recovery algorithms and multi-view stereo algorithms, which are complex and time-consuming, and perform poorly in complex scenes. Neural Radiance Field (NeRF) technology utilizes neural networks to achieve an implicit mapping from continuous 3D coordinates in space to image colors, effectively improving the quality of 3D reconstruction. However, NeRF technology also suffers from high computational overhead, slow rendering speed, and poor editability. To address this, 3D Gaussian Splatting (3DGS) technology has emerged. 3DGS technology uses a 3D Gaussian ellipsoid to represent objects in a scene, enabling scene editability and rapid reconstruction. During the rendering process, a scattering algorithm is applied to the Gaussian ellipsoid, thereby improving rendering speed.

[0003] Infrared imaging offers significant advantages in low-light and harsh environments, and is widely used in detection, reconnaissance, and nighttime security. However, significant gaps remain in infrared 3D reconstruction technology. Infrared images have sparse textures, unlike visible light images, making them unsuitable for direct application by existing algorithms. Furthermore, high-value infrared data is scarce, and the lack of diverse viewpoints leads to image distortion and impacts algorithm performance. Therefore, extending 3D reconstruction technology to infrared graphics to achieve high-precision infrared scene reconstruction and new viewpoint synthesis is of great significance and value.

[0004] Infrared 3D reconstruction is unique because its imaging mechanism differs fundamentally from that of visible light. Infrared sensors generate images by capturing thermal radiation from object surfaces rather than reflected light, resulting in infrared data exhibiting low texture resolution, high noise sensitivity, and dynamic thermal field variations. Traditional multi-view stereo methods perform poorly in infrared scenes due to the difficulty of feature matching and the sparse geometric information. NeRF, by modeling the characteristics of continuous radiation through implicit neural fields, can bypass explicit feature extraction and directly learn the relationship between thermal radiation fields and geometry from infrared images. However, the success of NeRF in the visible light domain cannot be directly applied to the infrared domain. Research shows that in cases of sparse infrared data, NeRF becomes more dependent on the input viewpoint, leading to artifacts or over-smoothing in the reconstruction results. As the inverse representation of NeRF, 3DGS relies heavily on explicit representation of Gaussian point clouds. While this can improve computational efficiency to some extent, due to the inherent characteristics of point clouds, this technique cannot directly model radiation and struggles to accurately assign radiation components and infrared parameters to each 3D Gaussian sphere (point in the point cloud). Therefore, how to effectively combine the infrared band with NeRF and 3DGS algorithms to form an algorithm for the vertical field of infrared 3D reconstruction is an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application propose a method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints. This method overcomes the challenge of balancing high-quality three-dimensional reconstruction of infrared images with the requirement to reduce computational complexity, achieving a good balance between the quality and speed of generating three-dimensional point cloud infrared targets.

[0006] To achieve the above objectives, embodiments of this application propose a method for generating 3D point cloud infrared targets based on physical mechanism constraints. The method includes the following steps: based on multi-view images and their corresponding view position information, sparse reconstruction of the infrared target is performed, and continuous expansion and optimization are carried out to obtain a sparse 3D point cloud model. Then, deep fusion and infrared radiation information mapping are performed to obtain a 3D model corresponding to the infrared target. Based on the camera pose and the sparse 3D point cloud model, a 3D Gaussian ellipsoid set is created, and spherical harmonic functions are used to control the extinction coefficient and infrared radiation brightness. Based on the 3D model corresponding to the infrared target and the 3D Gaussian ellipsoid set, the 3DIRGS algorithm is used to perform 3D to 2D rasterization rendering of temperature and emissivity respectively to obtain temperature feature maps and emissivity feature maps. The temperature feature maps and emissivity feature maps are input into a CNN network to simulate the infrared process of gray body radiation to obtain the radiation brightness field of the infrared target, and then the infrared image of the infrared target under various viewpoints is obtained.

[0007] To achieve the above objectives, embodiments of this application also propose a 3D point cloud infrared target generation system based on physical mechanism constraints. The system includes: a 3D model reconstruction module, used to sparsely reconstruct the infrared target based on multi-view images and their corresponding viewpoint position information, continuously expanding and optimizing to obtain a sparse 3D point cloud model, then performing depth fusion and infrared radiation information mapping to obtain a 3D model corresponding to the infrared target; a 3D Gaussian ellipsoid set creation module, used to create a 3D Gaussian ellipsoid set based on the camera pose and the sparse 3D point cloud model, and using spherical harmonic functions to control the extinction coefficient and infrared radiation brightness; a rasterization rendering module, used to perform 3D-to-2D rasterization rendering of temperature and emissivity respectively using the 3DIRGS algorithm based on the 3D model and 3D Gaussian ellipsoid set corresponding to the infrared target, obtaining temperature feature maps and emissivity feature maps; and an infrared image generation module, used to input the temperature feature maps and emissivity feature maps into a CNN network to simulate the infrared process of gray body radiation, obtaining the radiation brightness field of the infrared target, and thus obtaining infrared images of the infrared target from various viewpoints.

[0008] To achieve the above objectives, embodiments of this application also propose an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a three-dimensional point cloud infrared target generation method based on physical mechanism constraints as described above.

[0009] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints.

[0010] This application proposes a method for generating 3D point cloud infrared targets based on physical mechanism constraints. Utilizing Planck's law-constrained 3D point cloud fusion theory, it solves the problems of traditional methods neglecting the generation of small targets and lacking editable infrared parameters. First, based on multi-view images and their corresponding viewpoint position information, the infrared target is sparsely reconstructed, expanded, and optimized to obtain a sparse 3D point cloud model. Then, depth fusion and infrared radiation information mapping are performed to obtain the corresponding 3D model of the infrared target. Subsequently, based on the camera pose and the sparse 3D point cloud model, a 3D Gaussian ellipsoid set is created, and spherical harmonic functions are used to control the extinction coefficient and infrared radiance. Next, based on the 3D model of the infrared target and the 3D Gaussian ellipsoid set, the 3DIRGS algorithm is used to perform 3D-to-2D rasterization rendering of temperature and emissivity, respectively, to obtain temperature feature maps and emissivity feature maps. Finally, the temperature feature maps and emissivity feature maps are input into a CNN network to simulate the infrared process of gray body radiation, obtaining the radiance field of the infrared target, and thus the infrared images of the infrared target from various viewpoints. The method's 3D reconstruction results are significantly better than traditional methods in the visible light domain. It overcomes the challenge of balancing high-quality 3D reconstruction of infrared images with reduced computational complexity, achieving a good balance between quality and speed in generating 3D point cloud infrared targets. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints, provided in one embodiment of this application; Figure 2 This is a schematic diagram of the principle and model structure of 3DIRGS provided in one embodiment of this application; Figure 3 This is a schematic diagram of the COLMAP structure reconstruction process provided in one embodiment of this application; Figure 4 This is a schematic diagram comparing an infrared image generated by this method with an infrared image generated by a conventional method, provided in one embodiment of this application. Figure 5 This is a comparison diagram of the detailed target generation results of various algorithms provided in one embodiment of this application; Figure 6This is a schematic diagram of the structure of a three-dimensional point cloud infrared target generation system based on physical mechanism constraints provided in another embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the various embodiments of this application to enable readers to better understand this application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in this application can be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0014] One embodiment of this application proposes a method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints, applied to an electronic device. The electronic device can be a terminal or a server; this embodiment and subsequent embodiments will use a server as an example. The implementation details of the three-dimensional point cloud infrared target generation method based on physical mechanism constraints proposed in this embodiment are described below. The following implementation details are provided for ease of understanding only and are not essential for implementing this solution.

[0015] The specific process of the three-dimensional point cloud infrared target generation method based on physical mechanism constraints proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 11: Based on multi-view images and their corresponding view position information, sparse reconstruction of the infrared target is performed, and continuous expansion and optimization are carried out to obtain a sparse three-dimensional point cloud model. Then, deep fusion and infrared radiation information mapping are performed to obtain the three-dimensional model corresponding to the infrared target.

[0016] In the specific implementation, the server first needs to perform sparse reconstruction of the infrared target based on multi-view images and their corresponding view position information, and continuously expand and optimize the sparse reconstruction results to obtain a sparse three-dimensional point cloud model. Then, it performs deep fusion and infrared radiation information mapping to obtain the three-dimensional model corresponding to the infrared target.

[0017] In one example, the 3D point cloud infrared target generation method proposed in this embodiment is based on the 3DIRGS algorithm (a lightweight 3D Gaussian sputtering algorithm constrained by Planck's law and Beer-Lambert's law). The specific structure and principle of the 3DIRGS model can be described as follows: Figure 2 As shown.

[0018] In one example, during the data preparation phase, we need to obtain multi-view image data and their corresponding viewpoint position information so that the subsequent 3DIRGS model can learn comprehensive scene features. Real-world data can be reconstructed using COLMAP software. COLMAP is a widely used general-purpose stereo vision software for motion and multi-view applications, capable of reconstructing multi-view images into 3D models. The reconstruction process is as follows... Figure 3 As shown.

[0019] In one example, the server first needs to extract features from the multi-view images, detect feature points with unique properties in the multi-view images, and calculate a descriptor for each feature point. The descriptor is a numerical representation of the image features around the feature point, used for subsequent feature matching.

[0020] Subsequently, the server needs to compare the descriptors of feature points in different images, find matching feature point pairs, determine the correspondence between different images, start sparse reconstruction of infrared targets from a set of minimum number of image pairs, use the matching feature point pairs, calculate the camera pose (position and orientation) between images through principles such as epipolar geometry, initially estimate the position of three-dimensional spatial points, and construct the initial three-dimensional point cloud model.

[0021] Next, new images are gradually added to the initial 3D point cloud model. Based on the correspondence between the new images and the existing images in the initial 3D point cloud model, the camera pose of the new images and new 3D spatial points are calculated using methods such as triangulation. The initial 3D point cloud model is continuously expanded and optimized to obtain a sparse 3D point cloud model that contains more image information.

[0022] Then, based on the sparse 3D point cloud model, for multi-view images, using geometric and photometric consistency constraints, the depth value of each pixel is estimated to generate a depth map. All depth maps are then fused into the sparse 3D point cloud model to obtain a dense 3D point cloud model.

[0023] Then, by considering the consistency and overlapping areas of pixels under different viewpoints, incorrect matching points and noise points are removed to obtain a complete 3D point cloud model.

[0024] Finally, using methods such as Poisson reconstruction and moving cube algorithm, the complete 3D point cloud model is transformed into a triangular mesh model. The triangular mesh model is given a continuous surface, and the infrared radiation information of the image is mapped onto the triangular mesh model. Based on the projection of each triangular facet in the image, its corresponding texture coordinates are calculated, and texture is added to the triangular mesh model to complete the final reconstruction and obtain the 3D model corresponding to the infrared target.

[0025] It should be noted that for self-built simulation datasets, the motion structure should also be restored according to the above-mentioned process standards.

[0026] Step 12: Based on the camera pose and sparse 3D point cloud model, create a 3D Gaussian ellipsoid set and use spherical harmonic functions to control the extinction coefficient and infrared radiance.

[0027] In practice, after the reconstruction of the three-dimensional model corresponding to the infrared target is completed, the server can create a 3D Gaussian ellipsoid set based on the camera pose and the sparse three-dimensional point cloud model, and use spherical harmonic functions to control the extinction coefficient and infrared radiation brightness.

[0028] In one example, describing a 3D Gaussian ellipsoid requires four pieces of information: location, shape, infrared radiance, and extinction coefficient. The multidimensional Gaussian distribution expression for a 3D Gaussian ellipsoid is: ; ; ; in, The mean value represents the center point of the 3D Gaussian ellipsoid. When initializing the 3D Gaussian ellipsoid, this center point is the sparse point cloud obtained by the SfM algorithm. The covariance matrix is ​​symmetric and positive definite, representing the shape of the 3D Gaussian ellipsoid. It is an orthogonal matrix. For the reason A diagonal matrix composed of the eigenvalues ​​of . It is a positive diagonal matrix. The value on the diagonal is The square root of the eigenvalue, The multidimensional Gaussian distribution representing the 3D Gaussian ellipsoid.

[0029] Since diagonal matrices can be viewed as scaling transformations and orthogonal matrices as rotation transformations, therefore The geometric meaning is to stretch and rotate the orthogonal basis, and then project the random vector onto the new orthogonal basis.

[0030] It can be described by a three-dimensional vector, where each component of the three-dimensional vector corresponds to The diagonal components of the three-dimensional vector are expressed by the formula: ; in, , , All are diagonal components.

[0031] Through quadruples Represented as: ; in, Indicates the real part, , , Indicates the imaginary part.

[0032] In one example, the server uses spherical harmonic functions to control the extinction coefficient and its infrared radiance.

[0033] Spherical harmonics are a set of basis functions defined on the unit sphere, similar to basis vectors in space. Spherical harmonics possess orthogonality, completeness, and symmetry. They are defined as follows: ; ; ; in, Let be the order of the spherical harmonic function, that is, the quantum number of the total angular momentum of the spherical harmonic function. In physics, The energy level and angular momentum of the wave function are determined mathematically. It controls the complexity of the polynomial part of the spherical harmonic function. The order of the spherical harmonic function, i.e., the magnetic quantum number, is physics-wise... With angular momentum The components are related; mathematically, This determines the periodicity and phase of the spherical harmonic function along the azimuth direction. The normalization coefficient is... For about The related Legendre polynomial.

[0034] Any real function that is continuous on the surface of a sphere Both can be used Expanding into an average-convergent series, The expanded expression is: ; .

[0035] In one example, the 3DIRGS algorithm uses a third-order spherical function approximation. Due to the special nature of infrared images, the rendered object is an infrared grayscale image. To ensure consistency after display on the screen, three channels are still used for rendering, each channel consisting of an interval... A number is used to map to each channel to represent the infrared radiance.

[0036] Expanding each channel into a third-order spherical function requires 16 coefficients; therefore, to represent the radiance information of a 3D Gaussian ellipsoid, 48 parameters are needed.

[0037] When you get these 48 parameters and a direction vector Then, using The expansion expression is derived as follows: spherical harmonic function values ​​in the direction: ; in, This indicates three channels. express The spherical harmonic function value of the direction.

[0038] To facilitate synthesis and decomposition, and to make the expansion more symmetric, the 3DIRGS algorithm uses the following transformation to connect the real spherical harmonic function and the 3-channel parameters: .

[0039] During initialization, the 3DIRGS algorithm will Initialize to the 3-channel values ​​of a sparse 3D point cloud model, with other parameters initialized to 0, at a specific point. Its extinction coefficient is: ; in, The extinction coefficient is a parameter in the 3DIRGS algorithm. A pixel block is formed by the collective influence of multiple 3D Gaussian ellipsoids. Therefore... This represents the extinction coefficient components of multiple 3D Gaussian ellipsoids. The 3DIRGS algorithm initializes the transparency of each 3D Gaussian ellipsoid to 0.1 and continuously refines it during rendering.

[0040] Step 13: Based on the 3D model and 3D Gaussian ellipsoid set corresponding to the infrared target, the 3DIRGS algorithm is used to perform 3D to 2D rasterization rendering of temperature and emissivity respectively, to obtain temperature feature map and emissivity feature map.

[0041] In the specific implementation, after the server constructs the 3D Gaussian ellipsoid set, it can use the 3DIRGS algorithm to perform 3D to 2D rasterization rendering of temperature and emissivity based on the 3D model and 3D Gaussian ellipsoid set corresponding to the infrared target, respectively, to obtain temperature feature map and emissivity feature map.

[0042] In one example, the rendering process requires projecting a 3D Gaussian ellipsoid onto a 2D plane, with the 3D Gaussian ellipsoid having a multidimensional Gaussian distribution. Affine transformation Satisfying the first relation: .

[0043] After translating and rotating the 3D Gaussian ellipsoid from the world coordinate system to the camera coordinate system, and then transforming it to the image coordinate system through perspective transformation, the rotation transformation is expressed as: .

[0044] and Rotation matrix of the camera coordinate system and position coordinates get, , .

[0045] Based on the first relation, the transformed expected value is obtained. Covariance Matrix : ; .

[0046] This can be expressed by the formula: ; in, Represents the near plane, Represents the far plane. , , , These represent the top, bottom, left, and right borders of the camera viewfinder, respectively. axis, The field of view of the axis is represented as follows: ; ; .

[0047] In one example, since the 3D Gaussian ellipsoids are very small, therefore... , , The right end is At this point, using the Jacobian matrix Perform multivariate function expansion: .

[0048] The specific expression is: .

[0049] Based on this, perspective transformation is approximated as affine transformation, and combined with the first relation, we obtain... From the camera's perspective, it can be written as: .

[0050] Thus, the 3D Gaussian ellipsoid has been transformed into the image coordinate system. The algorithm for assigning infrared radiation characteristics to the transformed 3D Gaussian ellipsoid yields the extinction coefficient of the two-dimensional plane as follows: ; in, Indicates the first The extinction coefficient of a Gaussian ellipsoid.

[0051] In one example, for each 2D plane pixel after projection, it's equivalent to performing a ray traversal for all Gaussians, which would consume enormous computational resources and might not even produce good results. Therefore, a tiled rendering method is used. The idea is to divide the screen into blocks of 16×16 pixels each, and when rendering pixels in each block, only Gaussians that overlap with that block are considered. For Gaussian ellipsoids covering multiple blocks, the Gaussian ellipsoid is copied and assigned an identifier. For different blocks, parallel computation can be used, sorting the Gaussian ellipsoids by depth to speed up rendering.

[0052] Step 14: Input the temperature feature map and emissivity feature map into the CNN network to simulate the infrared process of gray body radiation, obtain the radiance field of the infrared target, and then obtain the infrared image of the infrared target from various viewpoints.

[0053] In the specific implementation, after obtaining the temperature feature map and the emissivity feature map, the server can input the temperature feature map and the emissivity feature map into the CNN network to simulate the infrared process of gray body radiation, obtain the radiance field of the infrared target, and then obtain the infrared image of the infrared target from various viewpoints.

[0054] In one example, the server concatenates the temperature feature map and the emissivity feature map, merging them into a single input vector (6-channel data). This merged input vector is then fed into a CNN network for processing. The CNN network consists of an encoder, an intermediate downsampling-upsampling structure, and a decoder.

[0055] The encoder's main function is to perform preliminary feature extraction and transformation on the input 6-channel data. The 6-channel input data contains information about temperature and emissivity. The encoder first uses a convolutional layer with a 3×3 kernel to transform the 6-channel input into a 64-channel feature map. The convolutional kernel slides across the image, extracting local features such as edges and textures through convolution operations. Padding = 1 ensures that the output feature map has the same size as the input, avoiding information loss. Subsequently, a batch normalization layer normalizes the output of the convolutional layer. During neural network training, the data distribution may change, leading to training instability. Batch normalization normalizes each batch of data, ensuring a mean of 0 and a variance of 1, thereby accelerating the model training process and improving model stability. Then, the PReLU activation function is used for activation. PReLU is a parameterized ReLU activation function with a learnable slope on the negative half-axis, adaptively adjusting the negative slope. Compared to the traditional ReLU function, PReLU increases the model's expressive power. By introducing the PReLU activation function, a non-linear factor is introduced, enabling the model to learn more complex feature representations. Finally, two ResBlock (64) residual blocks are used in the encoder. The core idea of ​​residual blocks is to directly add the input to the output of the convolutional layer through skip connections. This structure helps alleviate the gradient vanishing problem because during backpropagation, the gradient can be directly passed to the preceding layers through skip connections. At the same time, residual blocks can enhance feature extraction capabilities, enabling the model to learn higher-level features.

[0056] The intermediate downsampling-upsampling structure performs both downsampling and upsampling operations. The intermediate downsampling-upsampling structure halves the feature map size using a convolution operation with a stride of 2, while increasing the number of channels from 64 to 128. The purpose of downsampling is to reduce the spatial size of the feature map to better extract temperature and emissivity features, while increasing the number of channels to accommodate more feature information. After downsampling, ResBlock (128) residual blocks are used again to further enhance the feature representation. Then upsampling is performed. The deconvolution layer restores the feature map size to its original size through specific convolution operations, while reducing the number of channels from 128 to 64. The purpose of upsampling is to map the abstract features back to the original image size, preparing for subsequent decoding operations.

[0057] The decoder's task is to convert the intermediately processed feature maps into a final 3-channel radiometric image. The decoder first uses ResBlock (64) residual blocks to further process the features, enhancing their expressive power. Then, it progressively reduces the number of channels through two convolutional layers, ultimately outputting a 3-channel radiometric image.

[0058] Finally, the Sigmoid activation function is used to restrict the output pixel values ​​to the range of 0 to 1. The output is then linearly mapped to the RGB channels in the range of 0 to 255 to obtain infrared images of the infrared target at various viewing angles.

[0059] This embodiment proposes a 3D point cloud infrared target generation method based on physical mechanism constraints. Utilizing Planck's law-constrained 3D point cloud fusion theory, it solves the problems of traditional methods neglecting the generation of small targets and lacking editable infrared parameters. Based on multi-view images and their corresponding viewpoint position information, the infrared target is sparsely reconstructed, expanded, and optimized to obtain a sparse 3D point cloud model. Then, depth fusion and infrared radiation information mapping are performed to obtain the corresponding 3D model of the infrared target. Subsequently, based on the camera pose and the sparse 3D point cloud model, a 3D Gaussian ellipsoid set is created, and spherical harmonic functions are used to control the extinction coefficient and infrared radiance. Next, based on the 3D model of the infrared target and the 3D Gaussian ellipsoid set, the 3DIRGS algorithm is used to perform 3D-to-2D rasterization rendering of temperature and emissivity, respectively, to obtain temperature feature maps and emissivity feature maps. Finally, the temperature feature maps and emissivity feature maps are input into a CNN network to simulate the infrared process of gray body radiation, obtaining the radiance field of the infrared target, and thus the infrared images of the infrared target from various viewpoints. The method's 3D reconstruction results are significantly better than traditional methods in the visible light domain. It overcomes the challenge of balancing high-quality 3D reconstruction of infrared images with reduced computational complexity, achieving a good balance between quality and speed in generating 3D point cloud infrared targets.

[0060] The steps described above are for clarity only. In implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0061] In one embodiment, to verify the effectiveness of the proposed three-dimensional point cloud infrared target generation method based on physical mechanism constraints, we conducted relevant simulation experiments.

[0062] like Figure 4As shown, the 3DGS algorithm, as the basic method, cannot focus on minute details, such as a thin pole next to a house. The 3DGS algorithm fails to detect this target, and the generated new viewpoint does not include the thin pole. Our method (3DIRGS algorithm) can identify this, because our improved algorithm incorporates infrared parameters in two dimensions: temperature and emissivity. Texture information is implicitly contained in the emissivity channel, so separate feature extraction can focus on texture details, enhancing the generation effect. Overall comparisons also show that the images generated by the 3DIRGS algorithm are clearer and closer to the real-world dataset GT, validating the effectiveness of the 3DIRGS algorithm.

[0063] Figure 4 This visually demonstrates the generation details of each algorithm, providing enlarged images of the car targets generated by each algorithm. Figure 4 It is evident that the 3DIRGS algorithm has a significant advantage in generating infrared values. The targets generated by the 3DIRGS algorithm are closer to the real data in terms of radiance than those generated by 3DGS. Since the 3DGS series of algorithms have good attitude restoration capabilities, the improved 3DIRGS algorithm outperforms other algorithms in both attitude and radiance rendering, thus verifying the effectiveness of the 3DIRGS algorithm.

[0064] Accordingly, another embodiment of this application proposes a three-dimensional point cloud infrared target generation system based on physical mechanism constraints. The details of the three-dimensional point cloud infrared target generation system based on physical mechanism constraints proposed in this embodiment are described in detail below. The following content is only for the convenience of understanding and is not necessary for implementing this example.

[0065] Figure 6 This is a schematic diagram of the specific structure of a three-dimensional point cloud infrared target generation system based on physical mechanism constraints proposed in this embodiment, including: a three-dimensional model reconstruction module 21, a 3D Gaussian ellipsoid set creation module 22, a rasterization rendering module 23, and an infrared image generation module 24.

[0066] The 3D model reconstruction module 21 is used to sparsely reconstruct the infrared target based on multi-view images and their corresponding view position information, and continuously expand and optimize it to obtain a sparse 3D point cloud model. Then, it performs deep fusion and infrared radiation information mapping to obtain the 3D model corresponding to the infrared target.

[0067] The 3D Gaussian ellipsoid creation module 22 is used to create a 3D Gaussian ellipsoid based on the camera pose and a sparse 3D point cloud model, and to use spherical harmonic functions to control the extinction coefficient and infrared radiance.

[0068] The rasterization rendering module 23 is used to perform 3D to 2D rasterization rendering of temperature and emissivity based on the 3D model and 3D Gaussian ellipsoid set corresponding to the infrared target, respectively, using the 3DIRGS algorithm to obtain temperature feature map and emissivity feature map.

[0069] The infrared image generation module 24 is used to input the temperature feature map and emissivity feature map into the CNN network to simulate the infrared process of gray body radiation, obtain the radiance field of the infrared target, and then obtain the infrared image of the infrared target from various viewpoints.

[0070] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0071] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0072] Another embodiment of this application proposes an electronic device, the specific structure of which can be as follows: Figure 7 As shown, it includes: at least one processor 31; and a memory 32 communicatively connected to the at least one processor 31; wherein the memory 32 stores instructions executable by the at least one processor 31, the instructions being executed by the at least one processor 31 to enable the at least one processor 31 to perform a three-dimensional point cloud infrared target generation method based on physical mechanism constraints as described in the above method embodiment.

[0073] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0074] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0075] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a three-dimensional point cloud infrared target generation method based on physical mechanism constraints as described in the above method embodiments.

[0076] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disks, or optical disks.

[0077] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints, characterized in that, include: Based on multi-view images and their corresponding view position information, sparse reconstruction of infrared targets is performed, and continuous expansion and optimization are carried out to obtain a sparse three-dimensional point cloud model. Then, deep fusion and infrared radiation information mapping are performed to obtain the three-dimensional model corresponding to the infrared targets. Based on the camera pose and sparse 3D point cloud model, a 3D Gaussian ellipsoid set is created, and spherical harmonic functions are used to control the extinction coefficient and infrared radiance. Based on the 3D model and 3D Gaussian ellipsoid set corresponding to the infrared target, the 3DIRGS algorithm is used to perform 3D to 2D rasterization rendering of temperature and emissivity respectively, to obtain temperature feature map and emissivity feature map; Temperature feature maps and emissivity feature maps are input into a CNN network to simulate the infrared process of gray body radiation, thereby obtaining the radiance field of the infrared target and thus obtaining the infrared image of the infrared target from various viewpoints.

2. The method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints according to claim 1, characterized in that, Based on multi-view images and their corresponding viewpoint position information, sparse reconstruction of infrared targets is performed, and the reconstruction is continuously expanded and optimized to obtain a sparse 3D point cloud model. Then, deep fusion and infrared radiation information mapping are performed to obtain the corresponding 3D model of the infrared target, including: Feature extraction is performed on multi-view images to detect feature points with unique properties in the multi-view images, and a descriptor is calculated for each feature point; where the descriptor is a numerical representation of the image features around the feature point. By comparing the descriptors of feature points in different images, matching feature point pairs are found, the correspondence between different images is determined, and sparse reconstruction of infrared targets is performed starting from a set of minimum number of image pairs. Using the matching feature point pairs, the camera pose between images is calculated, the position of points in three-dimensional space is initially estimated, and an initial three-dimensional point cloud model is constructed. New images are gradually added to the initial 3D point cloud model. Based on the correspondence between the new images and the existing images in the initial 3D point cloud model, the camera pose of the new images and the new 3D spatial points are calculated. The initial 3D point cloud model is continuously expanded and optimized to obtain a sparse 3D point cloud model. Based on the sparse 3D point cloud model, for multi-view images, the depth value of each pixel is estimated using geometric and photometric consistency constraints to generate a depth map. All depth maps are then fused into the sparse 3D point cloud model to obtain a dense 3D point cloud model. By considering the consistency and overlapping areas of pixels under different viewpoints, incorrect matching points and noise points are removed to obtain a complete 3D point cloud model. The complete 3D point cloud model is converted into a triangular mesh model. A continuous surface is given to the triangular mesh model. The infrared radiation information of the image is mapped onto the triangular mesh model. Based on the projection of each triangular facet in the image, its corresponding texture coordinates are calculated. Texture is added to the triangular mesh model to obtain the 3D model corresponding to the infrared target.

3. The method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints according to claim 2, characterized in that, Based on the camera pose and a sparse 3D point cloud model, a 3D Gaussian ellipsoid set is created, including: Describing a 3D Gaussian ellipsoid requires four pieces of information: location, shape, infrared radiance, and extinction coefficient. The multidimensional Gaussian distribution expression for a 3D Gaussian ellipsoid is as follows: ； ； ； in, The mean value represents the center point of the 3D Gaussian ellipsoid. The covariance matrix is ​​symmetric and positive definite, representing the shape of the 3D Gaussian ellipsoid. It is an orthogonal matrix. For the reason A diagonal matrix composed of the eigenvalues ​​of . It is a positive diagonal matrix. The value on the diagonal is The square root of the eigenvalue, Represents the multidimensional Gaussian distribution of a 3D Gaussian ellipsoid; Described by a three-dimensional vector, each component of the three-dimensional vector corresponds to... The diagonal components of the three-dimensional vector are expressed by the formula: ； in, , , All are diagonal components; Through quadruples Represented as: ； in, Indicates the real part, , , Indicates the imaginary part.

4. The method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints according to claim 3, characterized in that, Using spherical harmonic functions to control the extinction coefficient and infrared radiance includes: Spherical harmonics are a set of basis functions defined on the unit sphere, similar to basis vectors in space. Spherical harmonics possess orthogonality, completeness, and symmetry. They are defined as follows: ； ； ； in, Let be the order of the spherical harmonic function, that is, the quantum number of the total angular momentum of the spherical harmonic function. In physics, The energy level and angular momentum of the wave function are determined mathematically. It controls the complexity of the polynomial part of the spherical harmonic function. The order of the spherical harmonic function, i.e., the magnetic quantum number, is physics-wise... With angular momentum The components are related; mathematically, This determines the periodicity and phase of the spherical harmonic function along the azimuth direction. The normalization coefficient is... For about The related Legendre polynomial; Real functions that are continuous on a sphere It can be used Expanding into an average-convergent series, The expanded expression is: ； 。 5. The method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints according to claim 4, characterized in that, The 3DIRGS algorithm uses a third-order spherical function approximation. Due to the special nature of infrared images, the rendered object is an infrared grayscale image. To ensure consistency after display on the screen, three channels are still used for rendering, each channel consisting of an interval... A number is used to map to each channel to represent the infrared radiance; Expanding each channel into a third-order spherical function requires 16 coefficients. Based on this, to represent the radiance information of a 3D Gaussian ellipsoid, 48 parameters are needed. When you get these 48 parameters and a direction vector Then, using The expansion expression is derived as follows: spherical harmonic function values ​​in the direction: ； in, This indicates three channels. express The spherical harmonic function value of the direction; To facilitate synthesis and decomposition, and to make the expansion more symmetric, the 3DIRGS algorithm uses the following transformation to connect the real spherical harmonic function and the 3-channel parameters: ; During initialization, the 3DIRGS algorithm will Initialize to the 3-channel values ​​of a sparse 3D point cloud model, with other parameters initialized to 0, at a specific point. Its extinction coefficient is: ； in, The extinction coefficient is a parameter in the 3DIRGS algorithm. A pixel block is formed by the collective influence of multiple 3D Gaussian ellipsoids. Therefore... This represents the extinction coefficient components of multiple 3D Gaussian ellipsoids. The 3DIRGS algorithm initializes the transparency of each 3D Gaussian ellipsoid to 0.1 and continuously refines it during rendering.

6. The method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints according to claim 5, characterized in that, Based on the 3D model and 3D Gaussian ellipsoid set corresponding to the infrared target, the 3DIRGS algorithm is used to perform 3D-to-2D rasterization rendering of temperature and emissivity, respectively, to obtain temperature feature maps and emissivity feature maps, including: The rendering process requires projecting a 3D Gaussian ellipsoid onto a 2D plane, and the 3D Gaussian ellipsoid has a multidimensional Gaussian distribution. Affine transformation Satisfying the first relation: ； After translating and rotating the 3D Gaussian ellipsoid from the world coordinate system to the camera coordinate system, and then transforming it to the image coordinate system through perspective transformation, the rotation transformation is expressed as: ; and Rotation matrix of the camera coordinate system and position coordinates get, , ; Based on the first relation, the transformed expected value is obtained. Covariance Matrix : ； ； This can be expressed by the formula: ； in, Represents the near plane, Represents the far plane. , , , These represent the top, bottom, left, and right borders of the camera viewfinder, respectively. axis, The field of view of the axis is represented as follows: ； ； 。 7. The method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints according to claim 6, characterized in that, Since 3D Gaussian ellipsoids are very small, therefore , , The right end is At this point, using the Jacobian matrix Perform multivariate function expansion: ； The specific expression is: ； Based on this, perspective transformation is approximated as affine transformation, and combined with the first relation, we obtain... From the camera's perspective, it can be written as: ; Thus, the 3D Gaussian ellipsoid has been transformed into the image coordinate system. The algorithm for assigning infrared radiation characteristics to the transformed 3D Gaussian ellipsoid yields the extinction coefficient of the two-dimensional plane as follows: ； in, Indicates the first The extinction coefficient of a Gaussian ellipsoid.

8. The method for generating three-dimensional point cloud infrared targets based on physical mechanism constraints according to claim 1, characterized in that, Temperature and emissivity feature maps are input into a CNN network to simulate the infrared process of gray-body radiation, obtaining the radiance field of the infrared target, and thus obtaining infrared images of the target from various viewpoints, including: The temperature feature map and the emissivity feature map are concatenated and merged into a single input vector, which is then fed into a CNN network for processing. A CNN network consists of three parts: an encoder, an intermediate downsampling-upsampling structure, and a decoder. The encoder performs preliminary feature extraction and transformation on the input vector. First, it uses a convolutional layer with a 3×3 convolutional kernel to convert the input vector into a 64-channel feature map. The 3×3 convolutional kernel slides on the input vector and extracts local features through convolution operations. Padding = 1 is set to ensure that the size of the output feature map is the same as that of the input vector. Then, the output of the convolutional layer is normalized through a batch normalization layer so that the mean of the data is 0 and the variance is 1. Then, the PReLU activation function is used to activate the data, which introduces non-linear factors. Finally, the input vector is directly added to the output of the convolutional layer through two ResBlock (64) residual blocks. The intermediate downsampling-upsampling structure halves the size of the feature map output by the encoder through a convolution operation with a stride of 2 and increases the number of channels from 64 to 128. After downsampling, ResBlock (128) residual blocks are used again to further enhance the feature representation. Then upsampling is performed, and the size of the enhanced feature map is restored to its original size using a deconvolution layer, while the number of channels is reduced from 128 to 64. The decoder first uses ResBlock (64) residual blocks to further process the feature map output by the intermediate downsampling-upsampling structure to enhance the expressive power of the features. Then, it gradually reduces the number of channels through two convolutional layers and finally outputs a 3-channel radiance image. Finally, the sigmoid activation function is used to limit the pixel values ​​of the 3-channel radiance image to the range of 0 to 1, and then linearly mapped to the RGB channels in the range of 0 to 255 to obtain the infrared image of the infrared target at each viewing angle.

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a three-dimensional point cloud infrared target generation method based on physical mechanism constraints as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a three-dimensional point cloud infrared target generation method based on physical mechanism constraints as described in any one of claims 1 to 8.

Images