An underwater scene three-dimensional reconstruction method and system

Abstract

The present application belongs to the field of computer vision and three-dimensional reconstruction technology, and particularly relates to a kind of underwater scene three-dimensional reconstruction method, and initial point cloud of scene is constructed based on motion recovery structure and multi-view stereo matching;Construct Gaussian radiation field, and Gaussian cell is decoupled into object Gaussian and medium Gaussian;In the Gaussian sputtering framework, a convolutional neural network is embedded, spatial detail features are extracted with a shallow branch to construct a depth confidence map, and adaptive geometric constraints are realized;Semantic features are extracted with a deep branch to construct a multi-view semantic consistency field, dynamic interference is automatically removed, and three-dimensional reconstruction of underwater scene is completed.The present application effectively improves the signal-to-noise ratio and reconstruction efficiency of underwater scene three-dimensional reconstruction, and can be used in various underwater application scenarios.The present application also provides a non-transitory readable recording medium storing the program of the method and a system containing the medium, which can call the program through a processing circuit to execute the above method.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, and discloses a method, recording medium and system for 3D reconstruction of underwater scenes. Background Technology

[0002] Underwater 3D scene reconstruction has significant application value in tasks such as seabed infrastructure inspection, autonomous vehicle navigation, and marine ecological monitoring. Optical vision-based 3D reconstruction is currently the main technique for acquiring high-fidelity visual details. However, in the underwater environment, light is severely absorbed and scattered by the participating medium, leading to severe degradation of images such as color decay and backscattering fogging.

[0003] Traditional Structure-from-Motion (SfM) techniques, due to the low-texture characteristics of underwater environments, typically only produce sparse or incomplete point clouds. In recent years, Neural Radiance Fields (NeRF) and its improved methods have enhanced reconstruction quality by embedding classical underwater imaging physics models into volumetric rendering equations; however, their reliance on multilayer perceptron rendering mechanisms results in enormous computational overhead, failing to meet the real-time rendering requirements of practical engineering applications. 3D Gaussian Splatting (3DGS) technology, through explicit point cloud parameterization and efficient differentiable rasterization, achieves high-quality real-time rendering and has rapidly become an advanced paradigm in the field. However, the standard Gaussian sputtering framework is based on the vacuum assumption and cannot be directly applied to water. Although some existing technologies attempt to incorporate underwater physics priors into the 3D Gaussian sputtering framework to improve its application in underwater environments, the following substantial drawbacks remain:

[0004] First, existing methods, whether relying on the homogeneous medium assumption or rigid voxel meshes, fail to completely decouple the target surface geometry from the non-homogeneous water medium at the physical level. During optimization training, the network is prone to stretching or expanding the object's surface geometry to simulate the forward scattering fog effect of water in order to minimize loss, resulting in significant geometric distortion in the reconstruction results.

[0005] Secondly, to alleviate the initialization problem caused by point cloud sparsity and to provide constraints for geometric optimization, existing methods often introduce pseudo-depth generated by pre-trained monocular depth estimation models for global supervision. However, these independent external depth models are not specifically designed for underwater degradation environments, and their parameters cannot be iteratively adjusted according to the specific underwater degradation scenario. Furthermore, the reliability of the pseudo-depth output in regions with strong backscattering decreases significantly. Applying unreliable pseudo-depth as a uniform global rigid constraint to the entire scene often leads to severe geometric fragmentation and structural distortion.

[0006] Finally, existing methods cannot effectively handle the floating artifacts caused by dynamic disturbances (such as schools of fish or suspended particles) in real underwater environments in the 3D reconstruction results. Summary of the Invention

[0007] To address the above problems, this invention provides a method for three-dimensional reconstruction of underwater scenes, comprising the following steps:

[0008] S1. For the acquired multi-view image sequence of the underwater scene, perform motion reconstruction structure and multi-view stereo matching in sequence to generate an initial point cloud;

[0009] S2. Construct a physics-driven Gaussian radiation field, decoupling the Gaussian elements into an object Gaussian set and a medium Gaussian set; the object Gaussian set is initialized based on the initial point cloud and used to model the inherent radiation of the underwater target surface; the medium Gaussian set is uniformly initialized within the underwater scene bounding box and used to approximate the backscattering integral term in the radiative transfer equation in a discretized manner.

[0010] S3. Embed a hierarchical convolutional neural network within the 3D Gaussian sputtering framework, inputting the rendered image generated in the current iteration; the network outputs a shallow feature map that preserves spatial details and local gradients through shallow branches, and outputs a single-channel depth estimation map through a depth prediction head, which serves as depth supervision information in iterative optimization and constructs a depth confidence map. Adaptive depth regularization is performed based on the depth confidence map, and the weight of the depth regularization loss is adaptively reduced for regions with low depth confidence.

[0011] S4. The hierarchical convolutional neural network is jointly iteratively optimized based on a joint loss function including photometric loss, confidence-aware depth regularization loss, and edge-aware smoothing loss. In each iteration, the semantic feature map containing scene semantic information is output through the deep branches of the network. Based on the multi-view semantic consistency constraint, the opacity gradient truncation loss is constructed to force the opacity of the Gaussian units of the fitted dynamic interference to approach zero, thereby realizing the automatic removal of dynamic interference and updating the input samples for the next iteration. After multiple iterations, the final three-dimensional reconstruction result of the underwater scene is output.

[0012] Preferably, modeling the inherent radiation of the underwater target surface includes the steps of decomposing the total received radiation into the direct transmission component of the target surface and the backscattering integral component of the participating medium through the radiative transfer equation; the discretization method is to approximate the radiative transfer equation by first-order discretization along the pixel ray, and physically decouple the total cumulative transmittance reaching the preset Gaussian element into the product of the object occlusion transmittance and the medium absorption transmittance; the rendering color is represented as the superposition of the surface geometric direct transmission component and the backscattering component of the suspended medium.

[0013] Preferably, the object Gaussian set has an optimizable spatial center position, covariance matrix, opacity, and higher-order spherical harmonic coefficients for modeling view-dependent surface reflections; and the medium Gaussian set has an optimizable spatial center position, isotropic variance, volume attenuation coefficient, and view-independent scattering color constrained to zero-order spherical harmonics.

[0014] The medium Gaussian is constrained to a zero-order spherical harmonic representation as a physical regularization strategy to prevent the network from using viewpoint-dependent color shifts as a shortcut to simulate water mist effects.

[0015] Preferably, the hierarchical convolutional neural network is a shared encoder backbone-dual-branch structure. The shared encoder backbone performs layer-by-layer convolution and downsampling on the input rendered image to extract multi-scale features. The shallow encoder branch is derived from the shallow features of the encoder backbone and outputs a shallow feature map that preserves local spatial details and gradient information. A depth prediction head is attached to the end of the shallow encoder branch. This depth prediction head consists of a 1×1 convolutional layer and maps the shallow feature map to a single-channel depth estimation map. The deep encoder branch is derived from the deep features of the encoder backbone. After further convolutional abstraction and upsampling, it outputs a deep semantic feature map that encodes global semantic information. The spatial resolution of the shallow feature map and the deep semantic feature map is consistent with that of the rendered image.

[0016] Preferably, the multi-view semantic consistency constraint process includes: taking advantage of the inherent characteristic that dynamic objects lack three-dimensional geometric consistency across multiple views, when a certain Gaussian primitive fits the semantic features of a dynamic object in one view, it will inevitably generate a huge feature space distance with the reference semantic features of the static background when projected in other views. The gradient of backpropagation forces the opacity of the primitive to approach zero, so that the automatic removal of dynamic interference can be achieved without manual annotation or explicit semantic labels.

[0017] Another aspect of the present invention is to provide a non-transient readable recording medium for storing one or more programs containing multiple instructions, which, when executed, cause the processing circuit to perform the above-described underwater scene three-dimensional reconstruction method.

[0018] Another aspect of the present invention provides an underwater scene three-dimensional reconstruction system, including a processing circuit and a memory electrically coupled thereto, the memory being configured to store at least one program, the program containing multiple instructions, the processing circuit running the program, and capable of executing the above-mentioned underwater scene three-dimensional reconstruction method.

[0019] Compared with existing technologies, the underwater scene three-dimensional reconstruction method, recording medium, and system provided by this invention have the following beneficial effects:

[0020] First, this invention constructs a physics-driven hybrid Gaussian representation based on the classical optical radiative transfer equation, explicitly decoupling the scene into two types of primitives: object Gaussians and medium Gaussians. This achieves a structural separation between the target surface geometry and the non-uniform suspended medium at the physical level. This measure effectively suppresses geometric expansion and distortion artifacts caused by the network's use of geometric stretching to simulate the fog effect of water, significantly improving the geometric accuracy of underwater scene 3D reconstruction.

[0021] Second, this invention embeds a hierarchical convolutional neural network within a 3D Gaussian sputtering framework, achieving unified extraction of depth and semantic information through a shared encoder backbone-dual-branch structure. This design enables end-to-end joint optimization of feature extraction and 3D reconstruction processes. The network can adaptively learn the optimal feature representation for the reconstruction task under underwater degradation conditions, avoiding the negative impact of inherent biases in external models on reconstruction accuracy.

[0022] Third, this invention utilizes the shallow branches of a hierarchical convolutional neural network to simultaneously output shallow feature maps and depth estimation maps. The channel variance of the shallow feature maps is used to construct a depth confidence map, while the depth estimation map serves as a depth supervision signal in iterative optimization; both are jointly optimized within the framework. The confidence map achieves adaptive coordination between depth regularization loss and photometric loss: in clear water regions, the gradient directions of the two losses are consistent, collaboratively constraining geometry and texture; in turbid water regions, the confidence map automatically decays the depth regularization loss weights, with photometric loss taking the lead to avoid texture being misled by erroneous geometric information, effectively preventing geometric collapse caused by global rigid depth constraints.

[0023] Fourth, this invention utilizes the semantic features output from deep branches of a hierarchical convolutional neural network to construct a multi-view semantic consistency field and designs an implicit opacity gradient truncation loss. This loss requires no manual annotation or semantic labeling; instead, it leverages the characteristic that dynamic objects generate projection conflicts in the multi-view semantic feature space due to a lack of consistency, driving the opacity of the corresponding Gaussian units to tend to zero, thus achieving automatic implicit removal of dynamic interference.

[0024] Fifth, this invention achieves high-fidelity 3D scene reconstruction and accurate color restoration while maintaining real-time rendering performance. Experiments show that compared with existing methods, it improves peak signal-to-noise ratio by at least 4.3%, structural similarity by 2.7%, and rendering speed by 127%, averaging 151.1 frames per second. It is suitable for applications such as underwater archaeology, coral ecological monitoring, submarine pipeline inspection, and shipwreck modeling. Attached Figure Description

[0025] Figure 1 This is a flowchart of the three-dimensional reconstruction framework in an embodiment of the present invention;

[0026] Figure 2This is a visualization of the Gaussian rendering of objects and Gaussian rendering of media in an embodiment of the present invention;

[0027] Figure 3 These are the different layer branch structures of the convolutional neural network embedded in the embodiments of the present invention;

[0028] Figure 4 This is the rendered depth feature map optimized by the aforementioned confidence-aware depth regularization constraint.

[0029] Figure 5 This is the result of removing dynamic interference and achieving clean scene reconstruction in the embodiments of the present invention.

[0030] Figure 6 This is a three-dimensional reconstruction effect diagram in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without innovative effort are within the scope of protection of the present invention.

[0032] See Figure 1 The present invention provides a method for 3D reconstruction of underwater scenes based on joint physical and semantic driving, which specifically includes steps P1 to P6:

[0033] P1. Initial Point Cloud Construction

[0034] Acquire multi-view image sequences of underwater scenes, sequentially perform motion reconstruction structure and multi-view stereo matching, estimate camera pose and generate initial point cloud;

[0035] Step 1: Input the acquired multi-view underwater image sequence, perform feature extraction, feature matching, and structure-of-motion (SfM) processes to estimate the camera intrinsic and extrinsic parameters (including camera position, attitude, and focal length) for each image, and output the initial sparse point cloud of the underwater scene. The three-dimensional spatial points in the sparse point cloud represent the physical locations in the scene with significant image features. In this embodiment, all images are downsampled to approximately 900×1400 pixel resolution before processing.

[0036] Step 2: Based on the camera pose and sparse point cloud estimated in Step 1, dense depth estimation is performed using the Multi-View Stereo Matching (MVS) module. This module calculates a pixel-by-pixel dense depth map by using the photometric consistency and geometric triangulation relationship between multi-view images, and then backprojects it into 3D space to generate an enhanced dense point cloud, compensating for the lack of sparse point cloud caused by the low-texture underwater environment.

[0037] It should be noted that multi-view stereo matching is a classic multi-view geometric method, and its depth estimation is entirely derived based on the geometric correspondence between multiple images. The depth confidence assessment and semantic feature extraction in subsequent iterative optimization processes are both completed by the hierarchical convolutional neural network proposed in this invention within a 3D Gaussian sputtering framework.

[0038] Step 3: Calculate the overall bounding box of the scene based on the spatial extrema of the dense point cloud to determine the spatial range for the subsequent uniform initialization of the medium Gaussian.

[0039] P2. Constructing a physics-driven hybrid Gaussian radiation field

[0040] To fundamentally resolve the interference of the underwater medium on geometric reconstruction at the physical level, this invention introduces the Radiative Transfer Equation (RTE) from classical optics. Within this physical framework, the total received radiation... The model is as follows:

[0041]

[0042] In the formula, The physical depth of the target surface from the camera; The inherent radiation of the target surface; The continuous direct transmittance from the surface to the camera; the integral term corresponds to the backscattered haze component, where Distance Continuous transmittance at that point The volume scattering coefficient varies spatially. It is a continuous scattering source function.

[0043] Wherein, scattering source function In the radiative transfer equation, it is physically defined as the incident light field. With phase function On a unit sphere Surface integral of the sphere:

[0044]

[0045] To discretize the aforementioned continuous physics model to fit modern differentiable rasterization pipelines, while avoiding the limitations of traditional rigid voxel meshes in fitting complex medium boundaries, this invention constructs a physics-driven hybrid Gaussian radiation field. This fundamentally achieves structural decoupling between the medium and the surface geometry.

[0046] Among them, the Gaussian set of objects Initialization is performed based on the dense point cloud augmented in step P1. Each object Gaussian has an optimizable spatial center position. Covariance matrix (Encoding scale and rotation direction), opacity and higher-order spherical harmonics (SH) coefficients This is used to model complex view-dependent surface reflections. In this embodiment, the maximum spherical harmonic order of the object's Gaussian is set to 3.

[0047] Medium Gaussian Set Uniform initialization is performed within the scene bounding box (calculated from the dense point cloud extrema in step P1), independent of surface features. In this embodiment, uniform sampling is performed within the scene bounding box. The medium Gaussian. Each medium Gaussian has an optimizable spatial center location. Isotropic variance Volume attenuation coefficient And the scattering colors are strictly constrained to zero-order spherical harmonics (i.e., isotropic approximation). .

[0048] It should be noted that this fixed number of medium Gaussians exhibits good generalization across different scene scales. This scalability stems from the spatial extent of each medium Gaussian. It is jointly optimized: in larger scenes, the medium Gaussian naturally expands to cover the volume space, thus maintaining representational power without needing to rely on hyperparameter tuning based on scene scale.

[0049] A first-order discretized approximation of the radiative transfer equation is performed along the pixel ray, reaching the... Total cumulative transmittance at each Gaussian unit Physically, it is decoupled into a transmittance component dominated by Gaussian occlusion of the object. With the continuous absorption transmittance component through a non-homogeneous medium The product of:

[0050]

[0051] In the formula, For the first Gaussian opacity of an object; For the first The volume attenuation coefficient of a medium in Gaussian form; For the effective cross step size. For each with a center and isotropic variance The medium Gaussian, and its effective cross step size with the camera ray. The Gaussian density distribution is derived analytically by performing a one-dimensional integral along the ray direction:

[0052]

[0053] In the formula, For the Gaussian center of the medium The perpendicular distance to the ray.

[0054] Ultimately, the rendering colors of the underwater scene The result is calculated by superimposing the direct transmission component of the surface geometry with the backscattering component of the suspended medium:

[0055]

[0056] In the formula, and These are the colors of the object's Gaussian and the medium's Gaussian, respectively, dynamically calculated based on the spherical harmonic coefficients and the viewing direction. and To control the three-dimensional covariance matrix of the two-dimensional projection process, the opacity weights at ray intersections are ultimately determined.

[0057] Figure 2 The visualization results of Gaussian rendering of the object and Gaussian rendering of the medium show that the method of the present invention separates the object and the medium based on the continuous radiation equation, thus achieving decoupled rendering of the object.

[0058] It should be noted that this decoupling method better reflects the actual physical mechanisms of optics. If the network attempts to simulate the mist effect of water by abnormally zooming in on the depth of the object's Gaussian, it will drastically change the cross step size of the medium's Gaussian. Thus in the color item This causes severe fluctuations, resulting in a huge penalty for photometric errors. This constraint forces the network to decouple the surface geometry from the suspended medium, effectively suppressing the ill-conditioned rendering problem caused by the simulation of fog effects by geometric stretching.

[0059] P3. Constructing a hierarchical convolutional neural network

[0060] See Figure 3 This invention embeds a hierarchical convolutional neural network within a three-dimensional Gaussian sputtering framework. It takes the currently rendered image as input and outputs both shallow feature maps and deep semantic feature maps, providing a unified feature-driven approach for subsequent depth confidence estimation and semantic consistency constraints.

[0061] The hierarchical convolutional neural network adopts a shared encoder backbone-dual-branch structure, specifically including:

[0062] Shared encoder: Rendering the input image Multi-scale features are extracted through layer-by-layer convolution and downsampling. The encoder consists of multiple convolutional blocks, each containing a convolutional layer, a batch normalization layer, and a ReLU activation function. Spatial downsampling is achieved through convolutions with a stride of 2. In this embodiment, the encoder contains four convolutional blocks, progressively reducing the spatial resolution to the original. , , and The corresponding number of output channels are 64, 128, 256 and 512 respectively.

[0063] Shallow encoder branch: This branch originates from the shallow features of the encoder. In this embodiment, it originates from the output of the second convolutional block, where the spatial resolution is the original. The number of channels is 128. After two layers of deconvolution upsampling, the spatial resolution is restored to full resolution, and a shallow feature map is output. In this embodiment, the number of shallow feature channels .

[0064] It should be noted that the shallow encoder branch originates from the shallow layer of the encoder backbone. The shallow convolutional features retain more spatial details, local gradients, and edge information, which are directly related to the local reliability of depth estimation. In underwater regions with strong scattering, local image details are severely degraded, and the corresponding shallow features exhibit high channel variance, effectively identifying regions where depth priors are unreliable.

[0065] Depth prediction head: Attached to the end of the shallow encoder branch, consisting of a... Convolutional layers consist of, The shallow feature map of a channel is mapped to a single-channel depth estimation map. The output depth value is then ensured to be positive using the Softplus activation function. This depth estimation map serves as a depth supervision signal during the iterative optimization process, replacing the role of the external depth model in the optimization loop.

[0066] It should be noted that the depth prediction head uses Convolution is used to achieve pixel-wise channel mapping without introducing additional spatial receptive fields, ensuring that depth prediction strictly preserves the spatial resolution and local details of shallow feature maps. Secondly, the parameters of the depth prediction head, as part of a hierarchical convolutional neural network, are jointly optimized with the entire 3D Gaussian sputtering framework, and its depth estimation capability continuously improves during the iteration process. In the initial optimization stage, rendering depth... Not yet accurate, CNN depth estimation A coarse depth perception is gradually established, primarily driven by indirect gradients with photometric loss. As optimization progresses, the rendered depth becomes more accurate, and the depth regularization loss further improves the consistency between the CNN's depth estimation and the rendered depth.

[0067] Deep encoder branch: This branch originates from deep features in the encoder backbone. In this embodiment, it originates from the output of the 4th convolutional block, where the spatial resolution is the original. The number of channels is 512. After further convolutional abstraction and multi-layer deconvolution upsampling to restore full resolution, a deep semantic feature map is output. In this embodiment, the number of deep semantic feature channels... .

[0068] It should be noted that the deep encoder branch is derived from the deep layer of the encoder. The deep convolutional features undergo multiple downsampling and nonlinear transformations to encode the global semantic structure and category-level abstract information of the scene. This effectively distinguishes objects of different semantic categories (such as coral reefs, schools of fish, and sandy areas), providing a high-quality reference for subsequent multi-view semantic consistency constraints.

[0069] P4. Constructing a confidence-aware deep regularization mechanism

[0070] Shallow feature maps based on the output of the shallow branches of the hierarchical convolutional neural network in step P3. Construct a pixel-level depth confidence map and then use it to estimate the depth map output by the shallow branch depth prediction head. With rendering depth Adaptive weighted supervision is applied to the consistency constraints between them.

[0071] Step 1: At pixel position At this point, for shallow feature maps Calculate the feature variance along the channel dimension This yields a scalar value that measures the uncertainty of local depth prediction.

[0072] It should be noted that the channel variance of shallow features reflects the reliability of depth estimation because of the following physical mechanism: In areas with clear water and well-defined scene structures, shallow convolutional kernels respond consistently and stably to the input, the activation values ​​of each channel tend to be consistent, and the variance is small, corresponding to high reliability of depth estimation. Conversely, in areas with turbid water and severe scattering, the input signal is subjected to strong random degradation interference, the response differences between different convolutional kernels increase, the activation values ​​of each channel are dispersed, the variance is large, and the depth estimation has low reliability. This is because shallow feature maps... With depth estimation map Feature representations sharing the same shallow encoder branch The channel variance directly reflects Prediction uncertainty at the corresponding pixel.

[0073] Step 2: Calculate the pixel position of the input image Spatial gradient magnitude at By introducing the inverse of the image spatial gradient magnitude into the exponential term, higher prior weights are preserved at clear physical boundaries (where the gradient is large and the penalty is small).

[0074] Step 3: Construct a composite probability distribution factor by combining the feature variance and the reciprocal of the spatial gradient magnitude. :

[0075]

[0076] In the formula, To prevent numerical instability of small constants; and is the scale alignment factor, used to safely normalize the combined confidence map to the [0,1] interval. In this embodiment, it is set to... , ,Right now .

[0077] It should be noted that the settings The motivation lies in the fact that underwater scattering disrupts low-level photometric gradients, making them inherently noisy; in contrast, jointly optimized shallow convolutional features exhibit greater robustness to such optical degradation. By assigning higher weights to the feature variance term, the confidence map prioritizes the optimized learnable features to determine depth reliability. Furthermore, the additive formula, rather than a multiplicative one, is chosen to avoid severe risk of zeroing out, ensuring stable depth guidance even when a particular metric locally weakens.

[0078] Step 4: Introduce a dynamic mask based on threshold gradient truncation using the cumulative transmittance of the medium. ,in For indicator functions, The preset threshold is set to $0.1$ in this embodiment. This mask explicitly cuts off invalid backpropagation in the Gaussian-dominated region of the medium, ensuring that geometric regularization is applied only to the solid surface region and avoiding the propagation of incorrect geometric constraints in the medium-dominated region.

[0079] Step 5: Multiply the composite probability distribution factor with the dynamic mask to generate the final pixel-level depth confidence map. :

[0080]

[0081] It should be noted that the depth confidence map implements an adaptive coordination mechanism between depth regularization loss and photometric loss in the joint optimization. Specifically: in areas where the water is clear and depth information is reliable, Approaching 1, depth regularization loss To obtain sufficient weights, the gradient direction is related to the photometric loss. Consistency: Convolutional Network Depth Estimation The Gaussian element is responsible for anchoring the Gaussian elements to the correct physical depth, while the photometric loss function constrains surface texture; the two work together. In areas where the water is turbid and depth information is unreliable... As the value approaches zero, the weights of the depth regularization loss are adaptively decayed, and the optimization process is dominated by the photometric loss, thus avoiding the texture being misled by incorrect geometric information. This mechanism prevents the gradient directions of the two losses from canceling each other out.

[0082] Figure 4 The rendering depth optimized by the aforementioned confidence-aware depth regularization constraint shows that the depth map rendered by the method of the present invention has strong continuity, and its depth information can reflect the depth distribution in the real scene.

[0083] P5. Constructing a multi-view semantic consistency field and an opacity gradient truncation loss.

[0084] The deep semantic feature map is based on the output of the deep branches of the hierarchical convolutional neural network in step P3. We construct a multi-view semantic consistency field and design an opacity gradient truncation loss to remove dynamic interference.

[0085] It should be noted that dynamic objects inherent in the underwater environment, such as schools of fish and suspended particles, severely degrade the quality of scene reconstruction. Because these objects violate multi-view geometric consistency, they introduce numerous artifacts into 3D space during the optimization process. This invention constructs a semantic consistency field within a 3D Gaussian sputtering framework and utilizes multi-view projection conflict signals to remove dynamic interference, eliminating the need for manual annotation or explicit semantic labels.

[0086] Step 1: Assign an additional high-dimensional learnable semantic feature vector to each object Gaussian. (Dimensions and Deep Semantic Feature Map) Consistent, in this embodiment This semantic feature follows the same volumetric rendering equation as color, and is used to generate a two-dimensional rendering semantic feature map by projecting along the camera ray. :

[0087]

[0088] in, For the first A Gaussian high-dimensional learnable semantic feature vector for an object. and These are its physical opacity and cumulative transmittance, respectively. This formula represents the high-dimensional semantic properties distributed in three-dimensional space. Projected onto a specific 2D camera plane, thus synthesizing a rendered semantic feature map of that viewpoint. .

[0089] Step 2: Output deep semantic feature maps from the deep encoder branches of the hierarchical convolutional neural network. The reference semantic feature map serves as the current perspective.

[0090] It should be noted that, due to the deep semantic feature map The output is from a convolutional neural network embedded within the framework, and this network is jointly optimized with the three-dimensional Gaussian radiation field using the same loss function. It can learn the optimal semantic representation for the current reconstruction task gradually during the optimization process.

[0091] Step 3: In the joint optimization stage, scene cleanup is achieved by leveraging the inherent characteristic of dynamic objects lacking 3D geometric consistency across multi-view images. Specifically, the semantic loss mechanism of this invention is category-independent, meaning it does not rely on the recognition and classification of specific objects, but is based on the conflict of high-dimensional feature vectors in spatial projection. If a certain 3D Gaussian primitive fits the deep local features of a dynamic disturbance (such as a swimming fish) at viewpoint A, because the disturbance is displaced in physical space, when the primitive with the same 3D coordinates is rendered from viewpoint B, the true reference features extracted by the visual base model at the corresponding pixel in viewpoint B will be lost. This is actually a static background at the lower level. In this case, the features rendered by this primitive... Compared with the true reference features of this location In high-dimensional latent variable spaces, alignment will fail, resulting in projection conflicts and large feature distances. (Error). To minimize this feature space discrepancy between multiple viewpoints, the backpropagated gradient signal automatically forces the opacity of the primitives. The process approaches zero. This process is based on the multi-view inconsistency of deep feature distribution, which is equivalent to applying an opacity gradient truncation to dynamic primitives, thus achieving automatic removal of dynamic interference without any manual annotation or explicit semantic labels.

[0092] Figure 5 This is the result of the present invention in removing dynamic interference and achieving clean scene reconstruction.

[0093] P6. Optimization based on joint loss function

[0094] Iterative optimization of a mixture of Gaussian radiation fields and a hierarchical convolutional neural network is performed based on a joint loss function. Total joint loss function. Includes a weighted sum of four terms:

[0095]

[0096] In the formula, , and These are the weighting coefficients used to balance the various loss components. In this embodiment, , , .

[0097] It should be noted that in this joint optimization framework, the parameters of the hierarchical convolutional neural network (including all convolutional weights of the shared encoder backbone, shallow encoder branches, and deep encoder branches) and the parameters of the mixed Gaussian radiation field (including the position, covariance, opacity, spherical harmonic coefficients, and learnable semantic feature vectors of the object Gaussian and medium Gaussian radiation fields) are all treated as optimizeable variables and processed by the same joint loss function. The gradients are updated collaboratively. This ensures deep coupling between feature extraction and 3D reconstruction, enabling the network to adaptively learn the optimal feature representation for the reconstruction task under underwater degradation conditions.

[0098] Mixed photometric loss Combined Absolute error and multi-scale structural dissimilarity measure D-SSIM:

[0099]

[0100] in, The actual input image, The color rendered by mixing the Gaussian radiation field in step P2. In this embodiment, .

[0101] Confidence-aware deep regularization loss Using the depth confidence map constructed in step P4 For rendering depth Depth estimation compared to the depth prediction head output of shallow branches in a hierarchical convolutional neural network Between Pixel-level adaptive weighting of distance:

[0102]

[0103] in, This is the rendered depth map obtained by accumulating the mixed Gaussian radiation field along the ray in the current iteration. This is a depth estimation graph output by the depth prediction head of the shallow branch of the hierarchical convolutional neural network in step P3.

[0104] Edge-aware smoothing loss Used to fill local geometric supervision gaps caused by confidence mask gradient truncation:

[0105]

[0106] In the formula, the edge-aware attenuation weight is derived jointly from the image gradient and the difference in shallow feature space: , ,in For rendering RGB images, The shallow feature map output by the shallow encoder branch in step P3 is in pixel... The value at that location.

[0107] It should be noted that although the confidence mask truncates the backpropagation of error depth in the gradient of the degenerate region, it inevitably creates gaps in the local surface geometry supervision, leading to geometric fragmentation. The edge-aware smoothing loss enables the model to maintain geometric continuity by forcing neighboring pixels with similar shallow features when faced with missing local geometric constraints, thereby promoting robust reconstruction of the local manifold.

[0108] Opacity gradient cutoff loss Used to minimize rendering semantic features With deep semantic reference features Between Distance forces the network to automatically reduce the opacity of the floating primitives that fit the dynamic object:

[0109]

[0110] in, For the complete two-dimensional spatial domain of image pixels, These are the pixel coordinates within this domain. This refers to the two-dimensional rendering semantic feature map generated by volumetric rendering in step S5. This is the deep semantic feature map output by the deep encoder branch in step P3.

[0111] This loss leverages the inherent characteristic that dynamic objects lack three-dimensional geometric consistency across multiple viewpoints. When a Gaussian primitive fits the semantic features of a dynamic object in one viewpoint, it will inevitably generate a huge feature space distance with the reference semantic features of the static background when projected in other viewpoints, i.e., multi-view semantic conflict. The gradient of backpropagation forces the opacity of the primitive to approach zero, which is equivalent to performing an implicit opacity gradient truncation operation on it.

[0112] During joint optimization, the location of the Gaussian center is accumulated. The absolute gradient norm guides the scene to perform adaptive cloning or split densification operations. Simultaneously, every 500 iterations, the Gaussian opacity of all objects is reset to 0.01, and Gaussian units with opacity below 0.05 are strictly pruned, thoroughly eliminating dynamic noise. All models are trained for 30,000 iterations under the same hardware conditions.

[0113] Figure 6 The reconstruction result is shown after performing steps P1 to P6 above. The reconstruction result has a complete geometric structure and high surface texture fidelity.

[0114] To verify the effectiveness of the method of this invention, experiments were conducted on the publicly available real underwater dataset SeaThru-NeRF. The experimental environment was configured with an RTX 4090 GPU, and the training iterations were 30,000. The experimental results show that the method of this invention successfully eliminates dynamic interference such as swimming fish and generates continuous and reasonable depth maps. Quantitative analysis shows that compared with existing methods, this invention improves the peak signal-to-noise ratio (PSNR) by an average of 4.3%, structural similarity (SSIM) by 2.7%, and rendering speed by 127%, averaging 151.1 FPS.

[0115] Assembling the above methods and steps into a program and storing it on a hard disk or other non-transitory storage medium constitutes an embodiment of the present invention's "a non-transitory readable recording medium"; while electrically connecting the storage medium to a computer processor and completing the three-dimensional reconstruction of an underwater scene through data processing constitutes an embodiment of the present invention's "a three-dimensional reconstruction system for an underwater scene".

[0116] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computers or available storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of underwater scenes, characterized in that, Includes the following steps: S1. For the acquired multi-view image sequence of the underwater scene, perform motion reconstruction structure and multi-view stereo matching in sequence to generate an initial point cloud; S2. Construct a physics-driven Gaussian radiation field, decoupling the Gaussian elements into an object Gaussian set and a medium Gaussian set; the object Gaussian set is initialized based on the initial point cloud and used to model the inherent radiation of the underwater target surface; the medium Gaussian set is uniformly initialized within the underwater scene bounding box and used to approximate the backscattering integral term in the radiative transfer equation in a discretized manner. S3. Embed a hierarchical convolutional neural network within the 3D Gaussian sputtering framework, and input the rendered image generated in the current iteration; The shallow feature map, which preserves spatial details and local gradients, is output through the shallow branch of the network. Then, a single-channel depth estimation map is output through the depth prediction head. This map is used as depth supervision information in the iterative optimization and a depth confidence map is constructed. Adaptive depth regularization is performed based on the depth confidence map. For regions with low depth confidence, the weight of the depth regularization loss is adaptively reduced. S4. The hierarchical convolutional neural network is jointly iteratively optimized based on a joint loss function including photometric loss, confidence-aware depth regularization loss, and edge-aware smoothing loss. In each iteration, the semantic feature map containing scene semantic information is output through the deep branches of the network. Based on the multi-view semantic consistency constraint, the opacity gradient truncation loss is constructed to force the opacity of the Gaussian units of the fitted dynamic interference to approach zero, thereby realizing the automatic removal of dynamic interference and updating the input samples for the next iteration. After multiple iterations, the final three-dimensional reconstruction result of the underwater scene is output.

2. The method for three-dimensional reconstruction of an underwater scene according to claim 1, characterized in that, Modeling the inherent radiation of an underwater target surface involves decomposing the total received radiation into the direct transmission component of the target surface and the backscattering integral component of the participating medium using the radiative transfer equation. The discretization method approximates the radiative transfer equation by first-order discretization along the pixel ray, and physically decouples the total cumulative transmittance at the preset Gaussian element into the product of the object occlusion transmittance and the medium absorption transmittance. The rendered color is represented as the superposition of the direct transmission component of the surface geometry and the backscattering component of the suspended medium.

3. The method for three-dimensional reconstruction of an underwater scene according to claim 2, characterized in that, Each object Gaussian in the object Gaussian set has an optimizable spatial center location, covariance matrix, opacity, and higher-order spherical harmonic coefficients for modeling view-dependent surface reflections; each medium Gaussian in the medium Gaussian set has an optimizable spatial center location, isotropic variance, volume attenuation coefficient, and view-independent scattering color constrained to zero-order spherical harmonics.

4. The method for three-dimensional reconstruction of an underwater scene according to claim 3, characterized in that, The hierarchical convolutional neural network is a shared encoder backbone-dual-branch structure. The shared encoder backbone performs layer-by-layer convolution and downsampling on the input rendered image to extract multi-scale features. The shallow encoder branch is derived from the shallow features of the encoder backbone and outputs a shallow feature map that preserves local spatial details and gradient information. At the end of the shallow encoder branch, a depth prediction head is attached, which consists of a 1×1 convolutional layer and maps the shallow feature map to a single-channel depth estimation map. The deep encoder branch is derived from the deep features of the encoder backbone, and after further convolutional abstraction and upsampling, outputs a deep semantic feature map that encodes global semantic information. The spatial resolution of the shallow feature map and the deep semantic feature map is consistent with that of the rendered image.

5. The method for three-dimensional reconstruction of an underwater scene according to claim 4, characterized in that, The multi-view semantic consistency constraint process includes: taking advantage of the inherent characteristic that dynamic objects lack three-dimensional geometric consistency across multiple views, when a certain Gaussian primitive fits the semantic features of a dynamic object in one view, it will inevitably produce a huge feature space distance with the reference semantic features of the static background when projected in other views. The gradient of backpropagation forces the opacity of the primitive to approach zero, so that the automatic removal of dynamic interference can be achieved without manual annotation or explicit semantic labels.

6. A non-transitory readable recording medium for storing one or more programs containing multiple instructions, characterized in that, When the instruction is executed, the processing circuit will perform the underwater scene three-dimensional reconstruction method according to any one of claims 1-5.

7. A three-dimensional reconstruction system for underwater scenes, comprising a processing circuit and a memory electrically coupled thereto, characterized in that, The memory is configured to store at least one program, the program containing multiple instructions, and the processing circuit runs the program to execute the underwater scene three-dimensional reconstruction method according to any one of claims 1-5.

Images