Short-wave infrared small sample three-dimensional Gaussian three-dimensional reconstruction method based on knowledge assistance

Abstract

The invention discloses a short wave infrared small sample three-dimensional Gaussian three-dimensional reconstruction method based on knowledge assistance. The method comprises the following steps: firstly, converting a three-dimensional structure model into a point cloud, carrying out structure perception point cloud fusion on the point cloud and an original target point cloud, and then carrying out initial training to realize geometric construction based on structure prior; then generating a short-wave infrared simulation image based on the three-dimensional model, and performing target instance driven cross-domain adaptation training on a pre-trained diffusion model to obtain a cross-domain diffusion model; and finally, in combination with a cross-domain diffusion model and a structure prior driven geometric construction result, triple constraints are added to reconstruct a three-dimensional image. Through staged modeling under structure guidance, the fuzzy or artifact risk caused by directly introducing misaligned or error accumulated geometric information into the texture stage is avoided, and a stable structure guidance image with clear components is provided for the texture modeling stage; and the generation stability and the structural fidelity of the diffusion model under complex observation conditions are effectively enhanced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a short-wave infrared small sample three-dimensional Gaussian three-dimensional reconstruction method based on knowledge assistance, which is particularly suitable for the task of three-dimensional reconstruction and realistic rendering under the condition of small samples in a short-wave infrared imaging scene, and is used to solve the problems of overfitting, structure blur and texture distortion of traditional methods under the condition of insufficient samples. BACKGROUND

[0002] In recent years, three-dimensional reconstruction methods represented by NeRFs (Neural Radiance Fields) have made significant progress in high-fidelity image reconstruction tasks by implicitly modeling three-dimensional scenes. The extension method combining bounded implicit surfaces such as SDF-NeRFs further improves the geometric expression ability and becomes one of the current mainstream research directions.

[0003] With higher requirements for reconstruction efficiency, three-dimensional Gaussian projection method (3D Gaussian Splatting, 3DGS for short) is proposed, which uses explicit voxel modeling and high-efficiency rendering pipeline, not only speeds up the training speed of the model, but also maintains high visual quality. However, both NeRF-based methods and 3DGS extension schemes are highly dependent on multi-view images as input. In the case of extremely sparse view angles, the model is prone to overfitting at the training view angle, resulting in problems such as geometric structure collapse and texture blur at new view angles.

[0004] To improve the generalization ability and rendering quality under the condition of sparse view angles, existing research has introduced regularization constraints (such as weight normalization, frequency regularization) and diffusion models as texture priors. For example, by integrating a 2D pre-trained diffusion model into the texture generation process, the image texture distortion problem is effectively alleviated; in the 3DGS framework, a depth map is further introduced as a geometric prior, and a cross-view learning strategy based on score distillation sampling (Score Distillation Sampling, SDS) is combined, which significantly improves the consistency of geometry and texture under sparse input.

[0005] However, there are still the following technical difficulties: 1. Difficulty in integrating structure priors: Current structure priors (such as CAD models or 3D meshes) often have differences in pose, scale or morphology with actual image inputs, making point cloud alignment difficult, and even reducing the consistency and stability of geometric reconstruction when geometric information conflicts.

[0006] 2. Diffusion model lacks structure perception ability: Current diffusion models do not fully utilize three-dimensional geometric structure information in the process of texture or material modeling, lacking the ability to perceive the target structure, thereby affecting the accuracy and realism of the generated texture, resulting in insufficient texture divergence or missing details.

[0007] In addition, there is a target conflict between geometry modeling and texture generation in the three-dimensional reconstruction system: geometry reconstruction emphasizes accurate alignment and spatial consistency, and usually relies on sparse but accurate structure priors; while texture modeling relies on diversified image samples to adapt to the changes of materials in different thermal states or viewing angles. If an end-to-end modeling method is used to forcibly unify the two, the following problems may occur: 1. The geometry side strengthens the constraints (such as strict SDF regularization), which may inhibit the degree of freedom of texture generation, resulting in a decrease in texture quality; 2. The relaxation of the texture side may lead to the degradation of the geometry shape and even the collapse of the structure. SUMMARY

[0008] In order to overcome the shortcomings of the prior art, the present application provides a short-wave infrared small sample three-dimensional Gaussian three-dimensional reconstruction method based on knowledge assistance, to solve the problems of overfitting, structure blur and texture distortion of the traditional method 3DGS under insufficient samples. By effectively introducing structure priors into the 3DGS framework, the robustness of geometry reconstruction is improved, and the texture reconstruction quality is optimized by combining a structure-aware diffusion model, thereby realizing clear structure and realistic texture of short-wave infrared three-dimensional reconstruction under sparse viewing angles.

[0009] In a first aspect, the embodiments of the present application provide a short-wave infrared small sample three-dimensional Gaussian three-dimensional reconstruction method based on knowledge assistance, comprising the following steps: First, convert the three-dimensional structure model into a point cloud, and perform structure-aware point cloud fusion with the original target point cloud. Through joint optimization of the depth map and the signed distance function (SDF), the fused point cloud is initially trained to realize geometry construction based on structure priors.

[0010] Then, generate short-wave infrared simulation images based on the three-dimensional model, and perform target instance-driven cross-domain adaptation training on the pre-trained diffusion model to obtain a cross-domain diffusion model, realizing cross-domain migration of the diffusion model.

[0011] Finally, combine the cross-domain diffusion model and the structure prior-driven geometry construction result to increase three constraints in the basic 3DGS training process for three-dimensional image reconstruction.

[0012] In one possible implementation, the structure prior-based geometry construction is specifically as follows: Perform high-density Poisson disk sampling on the three-dimensional structure model, extract geometric surface feature points, including curved surfaces, edges and key vertices to obtain structure prior point clouds; and obtain visual hull point clouds through a visual hull algorithm from known sparse viewing angles.

[0013] The rigid body transformation of the structure prior point cloud is performed by the Blender software, including rotation, translation and scaling operation, to realize accurate alignment of the structure prior point cloud and the visual shell point cloud in a unified spatial coordinate system.

[0014] The average distance of each point in the aligned structure prior point cloud and visual shell point cloud is calculated by using the K-neighbor algorithm, and the repeated and boundary abnormal points are removed. The processed structure prior point cloud and visual shell point cloud are fused; and the structure consistency and spatial continuity of the fused point cloud are improved by using the K-neighbor algorithm.

[0015] In the training rendering process, in order to obtain a more complete geometric structure model, the original 3DGS loss is jointly constrained by depth and SDF, which is specifically implemented as follows: a monocular depth supervision is used to quickly establish a preliminary geometric structure, and the depth map is used to initially constrain the preliminary geometric structure.

[0016] The SDF constraint weight is gradually increased to strengthen the implicit geometric constraint, thereby improving the surface continuity and structure consistency, and obtaining the geometric construction result driven by the structure prior; wherein the weight parameter in the loss function is adaptively adjusted according to the training round number.

[0017] In one possible implementation, the cross-domain migration is specifically operated as follows: Based on the reconstructed three-dimensional structure model, combined with thermal radiation physical modeling, a simulation image with real geometric shape and short-wave infrared imaging characteristics is generated.

[0018] The structure control mechanism and the semantic adaptation strategy are introduced into the pre-trained diffusion model to complete the texture migration optimization from the simulation domain to the real image domain, and a cross-domain diffusion model is obtained.

[0019] In one possible implementation, the structure control mechanism introduces a ControlNet subnetwork, and the depth map generated by the simulation image is used as a conditional control signal c to guide the diffusion process to maintain geometric consistency. In addition, the semantic adaptation strategy is used to improve the adaptation and discrimination ability of the model to the structure guide data style, and the semantic adaptation strategy is specifically as follows: special prompt words are added to the text description of the real image based on the simulation image to distinguish the simulation and real targets.

[0020] In one possible implementation, the diffusion model uses Stable Diffusion.

[0021] In one possible implementation, the output result of the cross-domain diffusion model in the reconstruction process of the three-dimensional image is jointly optimized by the following constraint terms: (1) semantic consistency constraint: by introducing the matching loss between the cross-domain diffusion model prediction noise and the semantic features, the semantic content is kept stable.

[0022] (2) Depth consistency constraint: the negative Pearson correlation loss between the surface model depth map generated by the structure prior driven geometry construction result and the denoised image depth map aligns the geometry structure.

[0023] (3) Signal-to-noise ratio (SNR) physical consistency constraint: align the image noise variance of the target region with the noise level of the real SWIR image, and introduce a signal-to-noise ratio loss term to improve physical consistency.

[0024] Finally, the joint semantic consistency constraint, depth consistency constraint and SNR physical consistency constraint form an optimization objective function, and different weighting coefficients are assigned to realize multiple constraint optimization.

[0025] In a second aspect, the embodiments of the present application provide a short-wave infrared small sample three-dimensional Gaussian three-dimensional reconstruction system based on knowledge assistance, which includes the following modules: The geometry construction module converts the three-dimensional structure model into a point cloud, and performs structure-aware point cloud fusion with the original target point cloud. Through joint optimization of the depth map and the signed distance function (SDF), the fused point cloud is initially trained to realize structure prior-based geometry construction.

[0026] The cross-domain migration module generates a short-wave infrared simulation image based on the three-dimensional model, performs target instance driven cross-domain adaptive training on the pre-trained diffusion model to obtain a cross-domain diffusion model, and realizes cross-domain migration of the diffusion model.

[0027] The three-dimensional reconstruction module combines the cross-domain diffusion model and the structure prior driven geometry construction result, and adds triple constraints in the basic 3DGS training process to reconstruct the three-dimensional image.

[0028] In a third aspect, the embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, when the computer program is executed in a computer, the computer executes any of the methods described in the present application.

[0029] In a fourth aspect, the embodiments of the present application provide a computing device including a memory and a processor, the memory having executable code stored therein, and the processor executing the executable code to implement any of the methods described in the present application.

[0030] Compared with the prior art, the present application has the following advantages: This paper proposes a knowledge-assisted 3D Gaussian reconstruction method for small-sample shortwave infrared images. This method is divided into two aspects: First, as a fundamental constraint in the geometric modeling stage, a structural prior enhances spatial consistency under sparse input through structure-aware point cloud fusion and depth-SDF joint optimization. Second, the structural prior is also used to drive the simulated generation of SWIR imagery, thereby constructing structure-driven target domain training data for training a structure-aware diffusion model. This decoupling mechanism enables the separation and scheduling of structural information in geometric modeling and texture generation: the former ensures morphological accuracy and consistency, while the latter enhances the structural perception capability of texture reconstruction. This structure-guided staged modeling not only avoids the risk of blurring or artifacts caused by directly introducing misaligned or error-accumulated geometric information into the texture stage, but also provides a stable, clearly structured image for the texture modeling stage, effectively enhancing the generation stability and structural fidelity of the diffusion model under complex observation conditions such as cross-viewing angles and cross-thermal states. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 This is a schematic diagram of the overall scheme of the knowledge-assisted short-wave infrared small sample three-dimensional Gaussian three-dimensional reconstruction method according to an embodiment of the present invention.

[0033] Figure 2 This is a comparison diagram of the structural integrity of the three-dimensional structure prior point cloud sampling in an embodiment of the present invention.

[0034] Figure 3 This is a visualization diagram of the structure-driven SWIR simulation in an embodiment of the present invention.

[0035] Figure 4 Schematic diagram of the structure-aware diffusion adaptation strategy in an embodiment of the present invention.

[0036] Figure 5 This is a visualization comparison diagram of the gain of each module on the new perspective synthesis effect in an embodiment of the present invention.

[0037] Figure 6 This is a comparison chart of the four-view reconstruction of the mini3 drone using an embodiment of the present invention and other algorithms.

[0038] Figure 7 This is a comparison diagram of the SUV four-view reconstruction using an embodiment of the present invention and other algorithms. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0040] The embodiment of the present application provides a small sample image reconstruction method of neural radiance field based on combined diffusion texture constraint, as shown in Figure 1 The specific steps are as follows: Step one: convert the three-dimensional model into a point cloud, and perform structure-aware point cloud fusion with the original target point cloud, perform initial training on the fused point cloud through joint optimization of a depth map and a signed distance function (SDF), and realize geometric construction based on structure priori; In order to obtain an initial structure priori point cloud with a relatively complete geometric structure, first, the sparse SWIR input view , camera pose and target mask obtained through acquisition are used to construct a target rough geometric structure by using a visual hull algorithm, and a three-dimensional point cloud is randomly initialized inside the visual hull by using a rejection sampling strategy to generate a visual hull point cloud. However, due to the sparsity of the view and the mask error, the point cloud often has geometric defects in the unobserved area.

[0041] To make up for this defect, a three-dimensional structure model (such as SolidWorks) of the target is introduced, high-density Poisson disk sampling is performed on the geometric surface (such as curved surface, edge and vertex, etc.) of the three-dimensional structure model, the number n of sampling points is set, and a structure priori point cloud with complete structure and uniform distribution is generated. The generated structure priori point cloud has higher boundary detail resolution and more uniform distribution characteristics, and especially shows superior structure integrity and fidelity in the unobserved area, as shown in Figure 2 .

[0042] The structure priori point cloud is subjected to rigid transformation (rotation, translation, scaling) by using Blender software, and the accurate alignment of the structure priori point cloud and the visual hull point cloud in the spatial coordinate system is realized through a rotation matrix and a translation vector , which lays a geometric foundation for subsequent fusion.

[0043] In the point cloud fusion stage, in view of the redundancy problem of the overlapping area of the two point clouds, in the fusion process, the average distance of each point of the visual hull point cloud to the nearest neighbors in the structure priori point cloud is calculated based on the K-nearest neighbor distance, and an adaptive threshold where mean(d) is the mean distance between point clouds, std(d) is the standard deviation of the distance between point clouds, is an adaptive adjustable parameter; remove points whose average neighbor distance exceeds the threshold Fuse the processed structure prior point cloud and visual hull point cloud.

[0044] Convert the fused point cloud to a 3D Gaussian distribution representation: for each point cloud point, take its three-dimensional coordinates as the mean of the Gaussian distribution ; at the same time, encode the color information of the point cloud point into spherical harmonic coefficients by spherical harmonics. The scale of the Gaussian distribution is dynamically determined by the average distance between adjacent points, ensuring that the Gaussian distribution can adapt to the local geometric structure. The rotation is set to the unit quaternion, representing the direction of the Gaussian distribution in the initial state and aligning with the global coordinate system. The opacity is initialized to a constant value. This high-quality initialization provides a solid geometric foundation for subsequent joint optimization.

[0045] Further, the single-view depth estimation is combined with SDF regularization, and a joint geometry optimization strategy based on training process scheduling is proposed to dynamically adjust the loss weight of the two, realize the collaborative optimization of modeling efficiency and geometric accuracy, and finally realize the structure prior driven geometry construction result based on structure prior driven geometry construction. The joint geometry optimization strategy based on training process scheduling is as follows: Early training: mainly supervised by monocular depth, quickly establish the preliminary geometric structure based on the 3D Gaussian distribution representation converted by the fused point cloud, and constrain the preliminary geometric structure using monocular depth information : where represents the rendered depth value obtained by a monocular depth camera from an unknown view, and is the depth value predicted by the monocular depth estimation method. This stage uses monocular depth information to constrain the geometric structure, thereby quickly establishing a reasonable geometric framework.

[0046] Late training: predict the SDF value of each Gaussian sphere center through a lightweight two-layer MLP network. Based on the prediction result, adopt SDF regularization strategy to realize fine-grained optimization of the edge of Gaussian point cloud structure. The SDF regularization strategy includes: Geometric concave optimization: for the region where the absolute value of SDF is ≤0.001 and the gradient norm is ≥0.9, perform the splitting operation of Gaussian points, increase the point density along the gradient direction, and improve the expression ability of complex geometric details.

[0047] Geometric bulge optimization: Gaussian points with an absolute SDF value ≥ 0.1 and located outside the object are removed to eliminate artifacts and redundant interference outside the structure boundary.

[0048] The above mechanism effectively improves the fidelity of Gaussian point cloud in the edge area of ​​the structure, providing support for subsequent geometric refinement and optimization.

[0049] Gradually increase the SDF geometry loss Weights strengthen implicit geometric constraints and improve surface continuity and structural consistency: in, is the SDF value of the point cloud obtained through the network, is the gradient value of the point cloud.

[0050] Gradually increase the SDF constraint weight to strengthen surface continuity and structural consistency: in, is the 3DGS loss, is the SDF geometric loss, is the depth loss, weight parameter and As training progresses Adaptive Adjustment: , in is the current training round number, is the total number of training rounds, Controls the rate of change.

[0051] Step 2: Based on the short-wave infrared simulation image generated by the 3D model, the pre-trained diffusion model is trained in a target instance-driven cross-domain adaptation mode to achieve cross-domain migration of the diffusion model. Based on the reconstructed 3D structural model of the target and combined with thermal radiation physics modeling, a simulated image with realistic geometric morphology and SWIR imaging characteristics is generated. The specific process is as follows: In the scene simulation environment, set the camera node and target typical parameters to generate the target's trajectory relative to the camera ,in Indicates time. The target three-dimensional structure model is simulated by computational fluid dynamics (CFD) to obtain the target CFD temperature field model. ,in Indicates the spatial position of the target surface. Based on the target's CFD temperature field model and the camera-target trajectory , calculate the radiation attenuation during transmission and the target's radiation output . According to latitude and longitude, altitude, zenith angle and atmospheric type and other parameters to calculate the radiation of the sky background Finally, through the above thermal radiation physical modeling, high-fidelity multi-view SWIR simulation images are generated. Figure 3 Generate samples for the section.

[0052] In order to achieve the generation of SWIR texture with consistent structure and strong sense of reality under sparse viewing angle, this paper constructs a structure-aware cross-domain diffusion model. The specific process is as follows: Figure 4 Based on Stable Diffusion, a structural control mechanism and semantic adaptation strategy are introduced to achieve high-quality texture migration from the simulation domain to the real image domain.

[0053] In cross-domain transfer, the model fine-tunes the U-Net and CLIP in Stable Diffusion by injecting low-rank parameters (LoRA), thereby enhancing its ability to model differences in structural features. Furthermore, a structural control mechanism is introduced by introducing a ControlNet subnetwork, using the depth map generated by the simulated image as a conditional control signal c to guide the diffusion process to maintain geometric consistency. Furthermore, to effectively distinguish the styles of the simulated and real domains, a semantic adaptation strategy is employed to enhance the model's ability to adapt to and discriminate the style of structure-guided data. Specifically, the semantic adaptation strategy adds a special prompt word "synthetic" to the textual description of the real image to distinguish between simulated and real objects.

[0054] The combined effect of the above structural control mechanism and semantic adaptation strategy enables the cross-domain diffusion model to take into account semantic expression, structural consistency, and style translation capabilities during texture generation, significantly improving cross-modal reconstruction under sparse perspectives. The training loss function is as follows: in, represents a noisy image, k is a text description containing specific semantic information (such as "a SWIR image ofxxx" or "a synthetic SWIR image of xxx"), and c is the depth control information. is the denoising network, represents the input image at time step t.

[0055] Overall, the cross-domain diffusion model improves the generalization and matching ability of the model between simulation and real data through the joint optimization of structure control, semantic cues, and lightweight parameter migration, and also realizes the collaborative modeling of geometric consistency and texture quality in the cross-modal reconstruction task.

[0056] Step three: Based on the cross-domain diffusion model and the structure prior driven geometric construction results, a triple constraint is added to the basic 3DGS training process (based on the original loss) to reconstruct the three-dimensional image. Specifically, through SDS semantic consistency constraint, depth consistency constraint, and SNR physical consistency constraint, the triple constraint acts simultaneously, so that the target realizes high-fidelity three-dimensional reconstruction in the 3DGS training process.

[0057] To further improve the cross-view performance of the cross-domain diffusion model, a triple constraint mechanism is constructed to ensure balance between semantic, geometric, and physical consistency in the image generation process.

[0058] (1) SDS semantic consistency constraint: The SDS method is used to introduce semantic supervision in the rendering process, and the loss function is used for optimization: where is the weight function of the time step , which is used to balance the loss contribution under different noise levels. represents the noise predicted by the cross-domain diffusion model, is the semantic information, represents the noisy image after the diffusion model.

[0059] (2) Depth consistency constraint: To maintain the geometric structure consistency of the generated image, we introduce a depth consistency constraint. The depth map of the cross-domain diffusion model denoising image is calculated using a single-view depth estimator, and compared with the surface model depth map generated by the structure prior driven geometric construction result. The geometric similarity between the two is measured by negative Pearson correlation: where, represents the generated surface model depth map, is the depth map of the cross-domain diffusion model denoising image. The negative Pearson correlation penalizes the depth inconsistency, forcing the generated image to align with the target geometric structure.

[0060] (3) SNR physical consistency constraint Due to the high sensitivity of SWIR images to thermal radiation, the signal-to-noise ratio (SNR) characteristics of the images are relatively fixed. To ensure that the generated images meet the true infrared imaging characteristics, we introduce the SNR physical consistency constraint to maintain the physical consistency of the generated images through the noise variance model, so as to ensure that the noise level of the generated images is consistent with the true SWIR images, and the signal-to-noise ratio SNR constraint term is added to the loss function: Wherein: is the signal-to-noise ratio of the generated image; is the target signal-to-noise ratio, which is calculated based on the true infrared image.

[0061] The SDS semantic consistency constraint, the depth consistency constraint and the SNR physical consistency constraint are used to constrain the pictures generated by the cross-domain diffusion model: Wherein: , , are the weight coefficients of the SDS semantic consistency constraint, the depth consistency constraint and the SNR physical consistency constraint, respectively.

[0062] The optimized generated image is input into the 3DGS, and the original 3DGS reconstruction loss is used for joint optimization to combine the high-quality picture generated by the above optimization with the initial 3DGS rendering, and form the final optimization target.

[0063] The embodiment of the application also provides a short-wave infrared small sample three-dimensional Gaussian three-dimensional reconstruction system based on knowledge assistance, which comprises the following modules: A geometry construction module: convert a three-dimensional structure model into a point cloud, and perform structure-aware point cloud fusion with an original target point cloud, perform initial training on the fused point cloud through joint optimization of a depth map and a signed distance function (SDF), and realize geometry construction based on structure prior. The specific operation is as follows: Perform high-density Poisson disk sampling on the three-dimensional structure model, extract geometric surface feature points, including curved surfaces, edges and key vertices to obtain a structure prior point cloud; and obtain a visual hull point cloud through a visual hull algorithm based on known sparse views.

[0064] Perform rigid body transformation on the structure prior point cloud through Blender software, including rotation, translation and scaling operations, and realize accurate alignment of the structure prior point cloud and the visual hull point cloud in a unified spatial coordinate system.

[0065] Calculate the average distance of each point in the aligned structure prior point cloud and the visual hull point cloud by using a K-nearest neighbor algorithm, remove duplicate and boundary abnormal points, and fuse the processed structure prior point cloud and the visual hull point cloud; and improve the structure consistency and spatial continuity of the fused point cloud through the K-nearest neighbor algorithm.

[0066] In the training rendering process, in order to obtain a more complete geometric structure model, the original 3DGS loss is jointly constrained by depth and SDF, which is specifically implemented as follows: a monocular depth supervision is used to quickly establish a preliminary geometric structure, and the depth map is used to initially constrain the preliminary geometric structure.

[0067] The SDF constraint weight is gradually increased to strengthen the implicit geometric constraint, thereby improving the surface continuity and structural consistency, and obtaining a structure-prior-driven geometric construction result; wherein the weight parameter in the loss function is adaptively adjusted according to the training round number.

[0068] Cross-domain migration module: based on the three-dimensional model, short-wave infrared simulation images are generated, and the pre-trained diffusion model is trained to obtain a cross-domain diffusion model, realizing the cross-domain migration of the diffusion model. The specific operation is as follows: Based on the reconstructed target three-dimensional structure model, combined with thermal radiation physical modeling, simulation images with real geometric shape and short-wave infrared imaging characteristics are generated.

[0069] In the pre-trained diffusion model, a structure control mechanism and a semantic adaptation strategy are introduced to complete the texture migration optimization from the simulation domain to the real image domain, and a cross-domain diffusion model is obtained.

[0070] In one possible implementation, the structure control mechanism introduces a ControlNet subnetwork, and the depth map generated by the simulation image is used as a conditional control signal c to guide the diffusion process to maintain geometric consistency. In addition, a semantic adaptation strategy is used to improve the model's ability to adapt and distinguish the style of structure guidance data, and the semantic adaptation strategy is specifically as follows: special prompt words are added to the simulation image based on the text description of the real image to distinguish between simulation and real targets.

[0071] In one possible implementation, the diffusion model uses Stable Diffusion.

[0072] Three-dimensional reconstruction module: combined with the cross-domain diffusion model and the structure-prior-driven geometric construction result, a three-dimensional image is reconstructed in the basic 3DGS training process by adding a triple constraint.

[0073] In the reconstruction process of the three-dimensional image, the output result of the cross-domain diffusion model is jointly optimized by the following constraint terms: (1) semantic consistency constraint: by introducing a matching loss between the cross-domain diffusion model prediction noise and the semantic features, the semantic content is kept stable.

[0074] (2) Depth consistency constraint: The negative Pearson correlation loss between the surface model depth map generated by the geometry construction result driven by the structure prior and the denoised image depth map aligns the geometry structure.

[0075] (3) Signal-to-noise ratio (SNR) physical consistency constraint: Align the image noise variance of the target region with the noise level of the real SWIR image, and introduce the signal-to-noise ratio loss term to improve physical consistency.

[0076] Finally, the joint semantic consistency constraint, depth consistency constraint, and SNR physical consistency constraint form the optimization objective function, and different weighting coefficients are assigned to achieve multiple constraint optimization.

[0077] A. Experimental setup Our framework is based on GaussianObject. In the initial optimization phase, the geometric structure after fusing the structure prior point cloud and the visual hull is initialized as the model, and the 3DGS is trained for 30000 iterations, of which the first 15000 training focuses on depth constraint training to achieve coarse-grained optimization, and the last 15000 training focuses on SDF further fine-grained optimization of geometric structure.

[0078] Based on the given infrared camera simulation parameters (image resolution 1280x1024, optical system aperture 160mm, camera focal length 50mm, pixel size 5 , wavelength 0.95-1.7 , optical system transmittance 0.7, thermal sensitivity 15mK), first generate a large-scale target infrared image dataset through infrared image characteristic simulation. Then, use the LoRA method to fine-tune the ControlNet module of the Stable Diffusion v1.5 model based on the generated simulation infrared dataset. Specifically, by injecting low-rank adaptive weights into the text encoder and Transformer module, the fine-tuning training key parameter configuration is as follows: LoRA rank is set to 64, the initial learning rate is 10-3, and the total training step is 4000 steps.

[0079] Then, the fine-tuned diffusion model is trained for geometric texture optimization, and a total of 8000 iterations of optimization are performed. In the first 7000 iterations, the model generates new view images continuously and combines the diffusion model for progressive diffusion model detail optimization; in the last 1000 iterations, the reference view is introduced as an additional supervision signal to further enhance the generation consistency and detail fidelity of the model.

[0080] The entire training process takes about 1 hour on a single NVIDIA RTX 4090 GPU.

[0081] B. Ablation Experiment To systematically evaluate the effectiveness of each component of the proposed decoupling method, we designed a module-level ablation experiment. Building on the foundational GaussianObject algorithm, we sequentially introduced knowledge-assisted point cloud construction, SDF geometry optimization, infrared simulation-driven diffusion model fine-tuning, and structure-infrared joint rendering constraints. We analyzed the improvement in the quality of new-view synthesis under sparse viewports with each step. We prioritized PSNR, SSIM, and LPIPS as evaluation metrics, with PSNR measuring reconstruction quality, SSIM assessing structural similarity, and LPIPS reflecting the consistency of perceptual details.

[0082] like Figure 5 As shown in Table 1, the complete model achieves optimal performance in all three metrics: PSNR, SSIM, and LPIPS, validating the proposed invention's effective improvement in geometric fidelity and texture realism. The figure shows the gradual optimization of the synthesis effect after the introduction of each module: 1. Structure-aware point cloud fusion significantly improves the geometric integrity and consistency of point clouds (PSNR increased by 1.6445), making the geometric structure more realistic; 2. Joint optimization of the depth map and signed distance function (SDF) significantly enhances structural details, especially in boundary areas such as tires. 3. The structural control mechanism and semantic adaptation strategy enhance the perceptual consistency of texture modeling (LPIPS reduction by 14.4%), making material texture details more natural and credible; 4. The triple constraints ensure the fidelity of the geometric structure while achieving structurally and spectrally consistent infrared image rendering effects.

[0083] Table 1 Analysis of the impact of each module ablation on reconstruction quality indicators C. Comparative test To evaluate the performance of our invention in the sparse reconstruction task of shortwave infrared images, this study conducted systematic comparative experiments. The following representative models were selected as baseline comparison methods: (1) the basic model 3DGS; (2) the sparse view reconstruction method sparseGS; and (3) the GaussianObject method for high-quality reconstruction from very few views. All comparison models were implemented based on official open-source code and trained under the same experimental conditions to ensure fair comparison results.

[0084] Table 2 (mini3 UAV multi-view reconstruction index comparison) and Table 3 (SUV multi-view reconstruction index comparison) show the performance of our invention on SUV and mini3, and the results show that our invention has significant advantages on the data set. In particular, when the input view decreases from 12 views to 4 views: our invention reduces the PSNR on the SUV data set by 63.6% compared with GaussianObjeect, and all indicators are optimal at 4 views. It shows that the algorithm has stronger robustness to the reduction of view number.

[0085] To more intuitively evaluate the reconstruction effect, the present application respectively carries out multi-view reconstruction quality comparison on mini3 unmanned aerial vehicle and SUV model. As shown in Figure 6 (mini3 unmanned aerial vehicle four-view reconstruction quality comparison chart) Figure 7 (SUV four-view reconstruction quality comparison chart) shows the geometric structure and texture rendering details under 4-view.

[0086] The invention proposed in the present application is superior to the comparison algorithm in the reconstruction quality of mini3 unmanned aerial vehicle, SUV and other targets under the reference view and novel view.

[0087] Table 2 mini3 UAV multi-view reconstruction index comparison table Table 3 SUV multi-view reconstruction index comparison table The electronic device provided by the embodiment of the present application comprises a processor and a memory. The memory is used to store a computer program. The processor is used to execute the program stored on the memory to realize any method described in the present application.

[0088] In a possible implementation, the electronic device provided by the embodiment of the present application further comprises a communication interface and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0089] The communication bus mentioned in the above electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0090] The communication interface is used for communication between the above electronic device and other devices.

[0091] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0092] The aforementioned processor can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component.

[0093] In yet another embodiment provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method in any of the embodiments of the present application.

[0094] In yet another embodiment provided in the present application, a computer program product containing instructions, which, when run on a computer, causes the computer to execute the method in any of the embodiments of the present application.

[0095] In the embodiments described above, all or some of the steps can be implemented by using software, hardware, firmware or any combination thereof. When implemented by using software, all or some of the steps can be implemented in the form of one or more computer programs. The computer program is stored in a computer readable storage medium, and includes all the procedures which consider necessary to be executed by the computer so as to generate the flow or function as described in the embodiments of the present application. The computer readable storage medium includes: any tangible storage device which can store the data which can be read by the computer system. The computer readable storage medium can be the computer system's memory or the memory of another computer system to which the computer system is connected. The computer program which is embodied in the computer readable storage medium can be distributed over the network, computer system, other device, or any combination thereof, using any one or combination of transmission methods. The computer readable program can be stored in the computer readable storage medium used by the computer system. The computer readable program can also be loaded onto the computer system or network to cause one or more computers to perform a series of operations to generate the flow or function as described in the embodiments of the present application.

[0096] It should be noted that, in the present document, the terms such as first and second are used only to differentiate one entity or operation from another entity or operation, and do not necessarily require or imply there is any such actual relationship or order between these entities or operations. Also, the terms "include", "contain" or any other variants thereof are intended to cover the non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0097] Each of the embodiments in the present specification is described in a related manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0098] The above only describes the preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A knowledge-assisted SWIR small sample 3D Gaussian 3D reconstruction method, characterized by: The following steps are involved: First, the 3D structural model is converted into a point cloud and then structure-aware point cloud fusion is performed with the original target point cloud. The fused point cloud is initially trained by joint optimization of the depth map and the signed distance function (SDF) to achieve geometric construction based on structural priors. Then, a short-wave infrared simulation image is generated based on the 3D model, and the pre-trained diffusion model is trained through target instance-driven cross-domain adaptation to obtain a cross-domain diffusion model, thus achieving cross-domain migration of the diffusion model. Finally, combining the cross-domain diffusion model with the geometric construction results driven by structure prior, triple constraints are added to the basic 3DGS training process to reconstruct the three-dimensional image.

2. The knowledge-assisted shortwave infrared small sample 3D Gaussian 3D reconstruction method according to claim 1 is characterized in that: The geometric construction based on structural prior is as follows: Perform high-density Poisson disk sampling on the 3D structure model to extract geometric surface feature points to obtain a structural prior point cloud; use the known sparse viewpoint through the visual shell algorithm to obtain a visual shell point cloud; Use Blender software to perform rigid body transformation on the structural prior point cloud to achieve precise alignment with the visual shell point cloud in a unified spatial coordinate system; The K-nearest neighbor algorithm is used to calculate the average distance between each point in the aligned structural prior point cloud and the visual shell point cloud, and duplicate and boundary abnormal points are removed. The processed structural prior point cloud and the visual shell point cloud are then fused. In the training and rendering process, in order to obtain a more complete geometric structure model, the depth and SDF are used to constrain it based on the original 3DGS loss. The specific implementation is as follows: monocular depth supervision is used to establish a preliminary geometric structure, and the depth map is used to initially constrain the preliminary geometric structure; Gradually increase the SDF constraint weight and strengthen the implicit geometric constraints, thereby improving surface continuity and structural consistency, and obtaining a geometric construction result driven by structural priors; The weight parameters in the loss function are adaptively adjusted according to the number of training rounds.

3. The knowledge-assisted shortwave infrared small sample 3D Gaussian 3D reconstruction method according to claim 1 is characterized in that: The specific operations for cross-domain migration are as follows: Based on the reconstructed 3D structural model of the target and combined with thermal radiation physics modeling, a simulated image with realistic geometric morphology and short-wave infrared imaging characteristics is generated; A structural control mechanism and semantic adaptation strategy are introduced into the pre-trained diffusion model to complete the texture migration optimization from the simulation domain to the real image domain, and obtain a cross-domain diffusion model.

4. The knowledge-assisted shortwave infrared small sample 3D Gaussian 3D reconstruction method according to claim 3 is characterized in that: The structural control mechanism introduces a ControlNet sub-network, using the depth map generated by the simulated image as the conditional control signal c to guide the diffusion process to maintain geometric consistency; In addition, a semantic adaptation strategy is used to enhance the model's ability to adapt and discriminate the style of structure-guided data. Specifically, the semantic adaptation strategy is as follows: special prompt words are added to the text description of the real image to distinguish the simulated image from the real target.

5. The knowledge-assisted shortwave infrared small sample 3D Gaussian 3D reconstruction method according to claim 1 is characterized in that: The diffusion model adopts Stable Diffusion.

6. The knowledge-assisted shortwave infrared small sample 3D Gaussian 3D reconstruction method according to claim 1 is characterized in that: During the reconstruction of the 3D image, the output of the cross-domain diffusion model is optimized under the following constraints: (1) Semantic consistency constraint: By introducing a cross-domain diffusion model to predict the matching loss between noise and semantic features, the semantic content is kept stable; (2) Depth consistency constraint: The geometric structure is aligned using the negative Pearson correlation loss between the depth map of the surface model generated by the geometric construction result driven by the structure prior and the depth map of the denoised image; (3) Physical consistency constraint of signal-to-noise ratio (SNR): Align the image noise variance of the target area with the noise level of the real SWIR image, and introduce the signal-to-noise ratio loss term to improve physical consistency; Finally, the semantic consistency constraint, depth consistency constraint and SNR physical consistency constraint are combined to form the optimization objective function, and different weighting coefficients are assigned to each of them to achieve multi-constraint optimization.

7. Knowledge-assisted shortwave infrared small sample 3D Gaussian 3D reconstruction system, characterized by: Includes the following modules: Geometric construction module: Converts the 3D structure model into a point cloud and performs structure-aware point cloud fusion with the original target point cloud. Through the joint optimization of the depth map and the signed distance function (SDF), the fused point cloud is initially trained to achieve geometric construction based on structural priors. Cross-domain migration module: Generates short-wave infrared simulation images based on the 3D model, performs target instance-driven cross-domain adaptation training on the pre-trained diffusion model to obtain a cross-domain diffusion model, and realizes cross-domain migration of the diffusion model; 3D reconstruction module: Combining the cross-domain diffusion model with the geometric construction results driven by structural priors, triple constraints are added to the basic 3DGS training process to reconstruct 3D images.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

9. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

Images