A 3D representation method for underwater scenes based on 3D Gaussian splashing

Abstract

The present invention discloses a 3D representation method for underwater scenes based on three-dimensional Gaussian splashing. The method comprises the following steps: obtaining a set of original underwater images captured from a single underwater scene; calibrating the intrinsic and extrinsic parameter matrices of the camera corresponding to each underwater image in the original underwater image set using the COLMAP algorithm, and simultaneously outputting a sparse point cloud; utilizing a pre-trained monocular depth estimator to derive a pseudo-depth map corresponding to each underwater image in the original underwater image set; initializing the properties of the water body Gaussian based on the sparse point cloud to obtain a corresponding water body Gaussian set; initializing the properties of the scene Gaussian based on the sparse point cloud and the pseudo-depth map to obtain a corresponding scene Gaussian set; and iteratively optimizing the attribute parameters in the water body Gaussian set and the scene Gaussian set to decouple and capture the corresponding water body properties and the scene's appearance, color, and geometric properties. This method can decouple complex water effects and the scene's appearance, color, and geometric information from underwater images.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional scene representation, and more particularly to a 3D representation method for underwater scenes based on three-dimensional Gaussian splashing. Background Art

[0002] 3D representation of underwater scenes plays a crucial role in numerous applications, including autonomous underwater vehicles (AUVs), marine ecological research, and underwater virtual reality systems. Effective underwater 3D representation requires effectively modeling both the water and the scene itself (i.e., the appearance, color, and geometry of the underwater scene in the absence of water). However, due to significant differences between underwater imaging and atmospheric imaging, existing vision-based 3D representation methods still face significant challenges in effectively representing underwater 3D scenes.

[0003] Next, the background and technology involved in this invention are introduced from four aspects: the characteristics of underwater imaging, traditional underwater scene 3D characterization technology, underwater scene 3D characterization technology based on Neural Radiance Field (NeRF), and 3D Gaussian Splatting (3DGS) technology.

[0004] Compared to imaging in atmospheric environments, underwater imaging exhibits three main characteristics: First, light is affected by attenuation during its propagation through water. This is because light reflected from objects is partially absorbed by water molecules on its way to the camera. The attenuation of red light is more significant than that of blue-green light, resulting in a bluish-green color shift in the appearance of the scene in underwater images. Second, light is affected by scattering, particularly backscattering, during its propagation through water. This is because ambient light is scattered by suspended particles in the water and then scatters into the camera, causing the appearance of the scene in the underwater image to exhibit haze appearance, low contrast, and other degradation phenomena. A third characteristic is that both the aforementioned attenuation and backscattering effects are highly correlated with line-of-sight distance (LOS distance), becoming more pronounced as the distance increases. This means that the same object photographed from different angles and distances will exhibit varying degrees of color degradation, no longer faithfully reflecting the true color of the scene itself, nor maintaining the cross-viewpoint consistency inherent in atmospheric imaging. In short, underwater images combine the scene's true color with complex water effects, posing a significant challenge to effective 3D representation of underwater scenes.

[0005] Traditional 3D representation methods for underwater scenes are primarily based on stereo vision technology. These methods typically adopt a two-stage implementation approach and can be further divided into two subcategories based on the technical approach employed. The first subcategory employs a "dewater first, then reconstruct" approach. These methods first utilize image restoration algorithms to remove water effects from underwater images one by one, restoring the scene's "true" appearance. They then utilize techniques such as stereo matching and Structure from Motion (SfM) to reconstruct the underwater 3D scene and its "true" appearance. The second subcategory employs a "reconstruct first, then dewater" approach. These methods first reconstruct the scene's geometric structure and degraded appearance with water effects using techniques such as stereo matching. They then introduce underwater imaging models to overlay the water effects, thereby decoupling the scene's "true" appearance from the underwater images through iterative optimization.

[0006] Neural Radiance Fields (NeRF) is an implicit underwater scene 3D representation technology. Neural Radiance Fields (NeRF) is the core technology of this type of method. Its principle is to use a deep neural network to implicitly model the scene. The network accepts the 3D coordinates and viewing direction of a scene as input, outputs the color and transparency of the point, and then achieves high-quality 3D scene rendering through ray marching strategies and volumetric rendering mechanisms. Correspondingly, NeRF-based underwater scene 3D representation technology extends NeRF's volumetric rendering framework to model the coupling mechanism between water effects and the scene's true appearance color during underwater imaging, thereby decoupling the scene's true appearance color, geometric information, and water properties. Taking SeaThru-NeRF as an example, this method assigns the transparency and color attributes of objects and the transparency and color attributes of water to each position in space. By modifying NeRF's volume rendering mechanism, which originally only models opaque objects, it is compatible with translucent water between the camera and the object, so as to model the attenuation and backscattering effects in the underwater imaging process. This makes it possible to directly learn the scene's true appearance color, geometry, and water parameters from underwater images, and realize 3D representation of underwater scenes.

[0007] The 3D Gaussian splatting technique is an explicit 3D scene representation technique that uses a set of learnable 3D Gaussian primitives to explicitly represent the scene. It also introduces a differentiable rasterization pipeline that outperforms the neural radiance field technique in terms of training time, rendering quality, and rendering speed. The following briefly introduces the specific implementation of the 3DGS technique. In the 3DGS technique, each 3D Gaussian primitive has a mean value. , covariance , opacity o, color In fact, the covariance property of each 3D Gaussian basis element is is a scale matrix that can be learned by elements and the rotation matrix According to the formula Calculated to ensure the covariance matrix The 3D Gaussian distribution corresponding to each 3D Gaussian basis element is expressed as ,here Refers to the coordinates of any point in the world coordinate system. In the setting of 3DGS technology, in order to model the non-Lambertian surface, a set of parameter-learnable spherical harmonic functions are used to calculate the color attributes that are finally involved in the rendering. In the rendering process, 3DGS technology introduces a tile-based differentiable rasterization pipeline. First, given the camera perspective to be rendered, each 3D Gaussian primitive It is projected into the corresponding image coordinate system and becomes a 2D Gaussian basis element. ; Then, these 2D Gaussian primitives The original 3D Gaussian primitives are sorted according to their depth values ​​in the corresponding camera coordinate system; then, the sorted 2D Gaussian primitives are sorted by The algorithm is used to render the color of each pixel. The specific rendering formula is as follows: , here Represents the coordinates of a given pixel in the image coordinate system, and S represents the set of all Gaussian primitives involved in rendering a given pixel. Indicates the opacity contribution of the i-th Gaussian primitive participating in the rendering at the current given pixel position. Traverse all pixels corresponding to the camera perspective to be rendered and complete the rendering to obtain the rendered image 3DGS technology usually uses and Calculate the rendered image using the equal color reconstruction loss and the original image The differences between them are analyzed and the learnable parameters of all Gaussian primitives are optimized through a differentiable rasterization pipeline to achieve accurate 3D representation of the scene.

[0008] However, traditional underwater scene 3D characterization technology, underwater scene 3D characterization technology based on neural radiation field, and three-dimensional Gaussian splashing technology still have some defects.

[0009] This section introduces the problems with traditional 3D representation methods for underwater scenes. First, a common problem with these two-stage methods is that the decoupling of scene appearance and the reconstruction of scene geometry are performed separately in the first and second stages, and the reconstruction effect of the second stage is heavily dependent on the reconstruction quality of the previous stage. Specifically, the geometric reconstruction quality of the "dewatering first, then reconstructing" type method depends on the quality of the image restoration in the previous stage. However, because image restoration is processed image by image and the restoration results of different images are independent, the consistency of the restored images cannot be guaranteed, which affects the accuracy of geometric reconstruction. The appearance restoration quality of the "reconstructing first, then dewatering" type method depends on the quality of the scene geometry output by stereo matching in the previous stage. However, because the geometric structure reconstructed directly from underwater images is inaccurate, especially in distant areas, it suffers from large depth map errors and reconstructs inaccuracies. This affects the accuracy of water effect superposition and the effectiveness of decoupling the scene's true appearance. Second, these methods use data-driven image restoration algorithms or predefined water parameter settings, which means that these methods lack detailed modeling of water parameters, making the learned 3D representation of the scene unable to well explain the water attribute information in the scene, making it difficult to carry out downstream tasks such as underwater new perspective synthesis.

[0010] This section describes the problems with neural radiance field-based 3D representation techniques for underwater scenes. First, the performance of these methods is limited by the inherent flaws of their underlying neural radiance field technology—the volume rendering framework and ray-line strategies they employ result in significant computational overhead and slow training and inference times, limiting their application in more efficient training and real-time rendering scenarios. Second, these methods employ implicit modeling of both the water and the scene itself, often leading to an incomplete decoupling of water effects from the scene's appearance geometry. The reconstructed scene itself often suffers from blurred details, color artifacts, and suspended objects.

[0011] This section describes the challenges faced by the 3D Gaussian splattering (3DGS) technique when representing underwater scenes in 3D. First, the original 3DGS technique assumes that images are captured in an atmospheric environment. In other words, the image set used to train 3DGS must have consistent content across viewpoints. However, distance-related water effects during underwater imaging cause the collected image set to no longer meet these requirements. Therefore, directly using 3DGS for 3D representation of underwater scenes will result in incorrect geometric structures and severe color artifacts, making it impossible to achieve 3D representation of underwater scenes. Second, there have been some recent works that extend 3DGS technology to underwater scenes. For example, WaterSpaltting introduces a volume rendering framework to model water effects while explicitly modeling the scene itself. Seasplat uses 3DGS to model the scene appearance and geometry while introducing a physically based underwater imaging model to superimpose water effects. UW-GS introduces a physically based color model for each Gaussian to model the influence of water effects. However, these methods have failed to achieve direct and explicit modeling of massive water media. The modeling quality of water effects is heavily dependent on the accuracy of the scene modeling itself, and it is difficult to accurately and effectively decouple the real appearance, geometry and water properties of underwater scenes.

[0012] Therefore, how to decouple the complex water effects and the appearance color and geometric information of the scene itself from underwater images, realize the rendering of the scene's own appearance color and underwater image rendering superimposed with water effects, so as to support the implementation of various downstream tasks such as underwater scene restoration, underwater new perspective synthesis, and underwater image restoration dataset construction, is a problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0013] In view of the above problems, the present invention provides a 3D characterization method for underwater scenes based on three-dimensional Gaussian splashing to at least solve some of the technical problems mentioned in the above background technology.

[0014] In order to achieve the above object, the present invention adopts the following technical solutions:

[0015] The present invention provides a 3D characterization method for underwater scenes based on 3D Gaussian splashing, comprising:

[0016] Obtain a set of original underwater images taken from a single underwater scene, use the COLMAP algorithm to calibrate the intrinsic and extrinsic parameter matrices of the camera corresponding to each underwater image in the original underwater image set, and output a sparse point cloud at the same time;

[0017] Using a pre-trained monocular depth estimator, we derive a pseudo-depth map corresponding to each underwater image in the original underwater image set.

[0018] Based on the sparse point cloud, the properties of the water body Gaussian are initialized to obtain the corresponding water body Gaussian set;

[0019] Based on the sparse point cloud and pseudo depth map, the properties of the scene Gaussian are initialized to obtain the corresponding scene Gaussian set;

[0020] The attribute parameters in the water body Gaussian set and the scene Gaussian set are iteratively optimized to decouple and capture the corresponding water body attributes and the appearance color and geometric attributes of the scene itself.

[0021] Furthermore, the properties of the water body Gaussian specifically include:

[0022] (1) Mean property of water body Gaussian:

[0023] For each water body Gaussian, its mean attribute is recorded as:

[0024]

[0025] in, is the mean attribute of the p-th water body Gaussian, representing the point coordinates in the world coordinate system; Represents the x-axis coordinate of the p-th water body Gaussian in the world coordinate system; Represents the y-axis coordinate of the p-th water body Gaussian in the world coordinate system; Represents the z-axis coordinate of the p-th water body Gaussian in the world coordinate system; parameter 、 and It is a non-learnable parameter and remains fixed after initialization;

[0026] (2) Covariance properties of water body Gaussian:

[0027] The covariance property of the water body Gaussian is modeled as isotropic corresponding to the spherical distribution; for each water body Gaussian, its covariance property is recorded as:

[0028]

[0029] in, represents the covariance property of the p-th water body Gaussian; Represents the shared parameter of the covariance attribute of all water body Gaussians, which is a non-learnable parameter and remains fixed after initialization; Represents the identity matrix of shape 3×3;

[0030] (3) Opacity property of water body Gaussian:

[0031] For each water body Gaussian, its opacity attribute is recorded as:

[0032]

[0033] in, represents the opacity attribute of the p-th water body Gaussian, which is a vector of length 3, corresponding to the three channels of RGB to model the wavelength selectivity of the water effect; Represents the shared parameters of the opacity attribute of all water body Gaussians; Indicates the opacity offset corresponding to the current p-th water body Gaussian; and are all vectors of length 3; Function is used to ensure The value range of each channel is (0,1); and All are learnable parameters;

[0034] (4) Color properties of water body Gauss:

[0035] For each water body Gaussian, its color attribute is recorded as:

[0036]

[0037] in, represents the color attribute of the p-th water body Gaussian; The shared parameter representing the color attribute of the Gaussian water body in the entire space is a vector of length 3, corresponding to the three channels of RGB; Indicates the color offset corresponding to the p-th water body Gaussian, which is a vector of length 3; Function is used to ensure The value range of each channel is (0,1); and is a learnable parameter.

[0038] Furthermore, the properties of the scene Gaussian include:

[0039] (1) Mean property of scene Gaussian:

[0040] For each scene Gaussian, its mean attribute is recorded as:

[0041]

[0042] in, is the mean attribute of the Gaussian of the rth scene, representing the point coordinates in the world coordinate system; Represents the x-axis coordinate of the r-th scene Gaussian in the world coordinate system; Represents the y-axis coordinate of the r-th scene Gaussian in the world coordinate system; Represents the z-axis coordinate of the r-th scene Gaussian in the world coordinate system; 、 and is a learnable parameter;

[0043] (2) Covariance properties of scene Gaussian:

[0044] The covariance property of the scene Gaussian is modeled as the anisotropy of the corresponding ellipsoidal distribution; for each scene Gaussian, its covariance property is recorded as:

[0045]

[0046] in, represents the covariance property of the r-th scene Gaussian; The scale matrix representing the ellipsoidal distribution; The rotation matrix representing the ellipsoidal distribution; and All are learnable parameters;

[0047] (3) Opacity property of scene Gaussian:

[0048] For each scene Gaussian, its opacity attribute is recorded as ; represents the opacity attribute of the r-th scene Gaussian, which is a scalar value and a learnable parameter with a value range of (0,1);

[0049] (4) Color properties of scene Gaussian:

[0050] For each scene Gaussian, its color attribute is recorded as:

[0051]

[0052] in, represents the color attribute of the r-th scene Gaussian; 、 and They correspond to the three channels of RGB and are learnable parameters with a value range of (0,1).

[0053] Furthermore, the water body Gaussian attributes are initialized based on the sparse point cloud to obtain the corresponding water body Gaussian set; specifically including:

[0054] (1) Initialize the mean attribute of the water body Gaussian:

[0055] According to the distribution range of the sparse point cloud, a rectangular bounding box is calculated and set as the scene boundary to limit the scene range;

[0056] Inside the scene boundary, a regular grid is constructed in the X, Y, and Z coordinate axes with a fixed step size. A water body Gaussian is placed at each grid node, and the position of the grid node in the world coordinate system is used as the mean attribute of the corresponding water body Gaussian.

[0057] (2) Initialize the covariance attribute of the water body Gaussian:

[0058] The covariance attribute shared parameters of all water body Gaussians , according to the formula Initialize; among them, represents the scale factor; Indicates a fixed step size;

[0059] (3) Initialize the opacity attribute of the water body Gaussian:

[0060] For each water body Gaussian, offset its opacity Initialized to [0,0,0];

[0061] Share the opacity attributes of all water body Gaussians Initialized to ;

[0062] (4) Initialize the color attributes of the water body Gaussian:

[0063] For each water body Gaussian, its color offset Initialized to [0,0,0];

[0064] Share the color attributes of all water body Gaussians is initialized to ;

[0065] (5) All water body Gaussians whose mean attributes, covariance attributes, opacity attributes and color attributes have been initialized are recorded as the water body Gaussian set.

[0066] Furthermore, the method of initializing the properties of the scene Gaussian based on the sparse point cloud and the pseudo depth map to obtain the corresponding scene Gaussian set includes:

[0067] Amplify the sparse point cloud according to the pseudo depth map to obtain an amplified point cloud;

[0068] Based on the augmented point cloud, the attributes of the scene Gaussian are initialized to obtain the corresponding scene Gaussian set.

[0069] Furthermore, the amplifying process of the sparse point cloud according to the pseudo depth map to obtain the amplified point cloud specifically includes:

[0070] (1) Based on the pseudo depth map corresponding to each underwater image in the original underwater image set, each underwater image is divided into multiple regions according to the preset depth value interval, and each region corresponds to a different depth interval on the pseudo depth map;

[0071] (2) Extract the sparse point cloud subset that can be observed by the camera perspective corresponding to each underwater image in the original underwater image set, calculate the depth value of all points in the sparse point cloud subset in the corresponding camera coordinate system, and obtain the projection points of all points in the sparse point cloud subset in the corresponding image coordinate system, which are recorded as original projection points;

[0072] (3) For each area of ​​each underwater image in the original underwater image set, calculate the number of pixels covered by the current area, determine the original projection point subset of the sparse point cloud subset corresponding to the current underwater image in the current area, and determine the number of original projection points contained in the current original projection point subset, calculate the number of projection points to be added in the current area, and randomly add the corresponding number of projection points in the current area, which are recorded as newly added projection points;

[0073] The number of projection points to be added in the current area is expressed as:

[0074]

[0075] in, Represents the original underwater image set The set of projection points to be added in the t-th region of the h-th underwater image; Representing a collection The number of projection points to be added in ; represents the minimum projection point density in each region; represents the tth region of the hth underwater image in the original underwater image set; express The number of pixels contained in ; represents the subset of original projection points of the h-th underwater image in the t-th region of the original underwater image set; Represents the original projection point subset The number of original projection points contained in ;

[0076] (4) For each newly added projection point in each area of ​​each underwater image in the original underwater image set, the depth value of the corresponding newly added spatial point in the camera coordinate system and color attributes Expressed as:

[0077]

[0078] Among them, m represents the mth newly added projection point; It represents the set of the nearest k original projection points of the mth newly added projection point found by the K nearest neighbor algorithm from the original projection point set of the current area; express The depth value of the original space point corresponding to the nth original projection point in the current camera coordinate system; express The color value of the original space point corresponding to the nth original projection point in the current camera coordinate system; represents weighted weight;

[0079] Among them, the coordinates of the mth newly added projection point in the image coordinate system are marked as , the coordinates of the nth original projection point in the image coordinate system are marked as , then the weighted weight Expressed as:

[0080]

[0081] (5) According to the camera projection transformation and the camera's external parameter matrix, the position of each newly added spatial point corresponding to the newly added projection point in the world coordinate system is obtained;

[0082] (6) Calculate the positions and color attributes of all newly added spatial points in all areas of all underwater images in the original underwater image set. The union of the set of these newly added spatial points and the set of original spatial points corresponding to the sparse point cloud is the augmented point cloud.

[0083] Furthermore, during each iterative optimization of the attribute parameters in the water body Gaussian set and the scene Gaussian set, an underwater image is selected from the original underwater image set. and its corresponding pseudo depth map , using the current iteration of the water body Gaussian primitive and the scene Gaussian primitive through the channel-by-channel multi-output rasterization rendering module to obtain the underwater image under the same camera perspective , the image of the scene itself , pure water background image and depth map By calculating the loss function and returning the gradient, the Gaussian properties of the water body and the scene are optimized.

[0084] The channel-by-channel multi-output rasterization rendering module includes:

[0085] (1) Underwater image rendering branch:

[0086] ① The set of all water body Gaussians is recorded as , the set of all scene Gaussians is recorded as ;

[0087] ②For the p-th water body Gaussian, its opacity attribute is re-expressed as ,in, 、 and for The three opacity components are independent of each other and correspond to the three RGB channels respectively;

[0088] ③For the rth scene Gaussian, its opacity attribute is reformulated as ,in, 、 and for The three opacity components correspond to the three RGB channels, and ;

[0089] ④ Given the camera perspective corresponding to the underwater image to be rendered, that is, the camera's intrinsic parameter matrix and extrinsic parameter matrix, find the water body Gaussian set and scene Gaussian set The union of , then the set All Gaussian primitives in are sorted by their depth values ​​in a given camera coordinate system for tile-based -blending rendering;

[0090] ⑤The color of each channel of any pixel of the underwater image to be rendered , expressed as:

[0091]

[0092] in, Represents the position coordinates of the pixel in the image coordinate system; Corresponding to different color channels; Represents the color value of the color channel of the i-th Gaussian primitive in the rendering process. Here, the color channel of the i-th Gaussian primitive is The color channel corresponds to represents the opacity contribution of the i-th Gaussian primitive to the corresponding color channel during the current pixel rendering process, and ; Represents the opacity component of the i-th Gaussian primitive in the corresponding color channel during rendering; Indicates the value of the 2D Gaussian distribution of the i-th Gaussian basis element at the current pixel in the current image coordinate system during the rendering process;

[0093] ⑥ Render each color channel of any pixel of the underwater image to be rendered independently;

[0094] ⑦Traverse all pixels of the underwater image to be rendered and complete the rendering, outputting the final underwater image ;

[0095] (2) Image rendering branch of the scene itself:

[0096] ①Extract scene Gaussian set All Gaussian primitives in ;

[0097] ② For the rth scene Gaussian, re-record its opacity attribute as a scalar ;

[0098] ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's intrinsic parameter matrix and extrinsic parameter matrix, the rasterization rendering pipeline in the original 3DGS technology is used to render the scene Gaussian set All Gaussian primitives in the image are processed and the final image of the scene itself is output. ;

[0099] (3) Pure water background image rendering branch:

[0100] ① Extract water body Gaussian set All Gaussian primitives in ;

[0101] ②For the p-th water body Gaussian, its opacity attribute is re-expressed as ,in The three opacity components are independent of each other;

[0102] ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's internal parameter matrix and external parameter matrix, the water body Gaussian set All Gaussian primitives in are sorted by their depth values ​​in a given camera coordinate system for tile-based Rendering;

[0103] ④ The color of each channel of any pixel of the pure water background image to be rendered Expressed as:

[0104]

[0105] in, Represents the position coordinates of the pixel in the image coordinate system; Corresponding to different color channels; Represents the color value of the color channel of the i-th Gaussian primitive in the rendering process. Here, the color channel of the i-th Gaussian primitive is The color channel corresponds to represents the opacity contribution of the i-th Gaussian primitive to the corresponding color channel during the current pixel rendering process, and ; Represents the opacity component of the i-th Gaussian primitive in the corresponding color channel during rendering; Indicates the value of the 2D Gaussian distribution of the i-th Gaussian basis element at the current pixel in the current image coordinate system during the rendering process;

[0106] ⑤ Render the color of each channel of any pixel of the pure water background image independently;

[0107] ⑥ Traverse all pixels of the pure water background image to be rendered and complete the rendering, outputting the final pure water background image ;

[0108] (4) Depth map rendering branch:

[0109] ①Extract scene Gaussian set All Gaussian primitives in participate in rasterization and rendering;

[0110] ② For the rth scene Gaussian, re-record its opacity attribute as a scalar ;

[0111] ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's intrinsic parameter matrix and extrinsic parameter matrix, the scene Gaussian set All Gaussian primitives in are sorted by their depth values ​​in a given camera coordinate system for tile-based Rendering;

[0112] ④The depth value of any pixel of the depth map to be rendered is expressed as:

[0113]

[0114] in, Represents the position coordinates of the pixel in the image coordinate system, represents the opacity contribution of the i-th Gaussian primitive in the current pixel rendering process, and ; Indicates the value of the 2D Gaussian distribution of the i-th Gaussian basis element at the current pixel in the current image coordinate system during the rendering process; Indicates the depth value of the i-th Gaussian primitive in the current camera coordinate system during rendering;

[0115] ⑤ When all pixels of the image to be rendered are rendered, the desired depth map is obtained .

[0116] Furthermore, in the process of iteratively optimizing the attribute parameters of the water body Gaussian set, the attribute optimization strategy of the water body Gaussian includes:

[0117] (1) The mean and covariance attributes of each water body Gaussian remain frozen after initialization and no longer participate in the optimization process;

[0118] (2) According to the number of parameter optimization iterations, the optimization process of the opacity attribute and color attribute of each water body Gaussian is divided into two stages:

[0119] During the first stage of optimization, the opacity offset of each water body Gaussian and color offset Keep frozen and do not participate in optimization, only optimize the opacity attribute shared parameters of all water body Gaussians Shared parameters with color attributes ;

[0120] During the second stage of optimization, unfreeze the opacity offset of each water body Gaussian and color offset , and shares parameters with the opacity attribute shared by all water body Gaussians Shared parameters with color attributes Optimize together.

[0121] Furthermore, in the process of iteratively optimizing the attribute parameters in the water body Gaussian set and the scene Gaussian set, the total loss function used is Expressed as:

[0122] In the first stage of optimization, ;

[0123] In the second stage of optimization, ;

[0124] in, represents the loss function based on image content; represents the coarse-grained depth loss function; Represents the local smoothing regularization term of the Gaussian opacity attribute of the water body; represents the local smoothing regularization term of the Gaussian color attribute of the water body; Represents the coarse-grained depth loss function The weight of Represents the local smoothing regularization term of the water body Gaussian opacity attribute weight; Represents the local smoothing regularization term of the water body Gaussian color attribute The weight of .

[0125] Further:

[0126] (1) Loss function based on image content The construction methods include:

[0127] ①Use the pseudo depth map of the current iteration round To create a background mask ; Given a predefined depth threshold ; Pseudo depth value is greater than The area composed of pixels is considered as the foreground area, and the corresponding background mask The pixel value at the corresponding position is 0; the pseudo depth value is less than or equal to The area composed of pixels is regarded as the background area, and the corresponding background mask The pixel value at the corresponding position is set to 1;

[0128] ② The underwater image in the original underwater image set is recorded as the true value of the underwater image, and the symbol is , construct a weight map with the same shape as the pseudo depth map Specifically, the weight graph The value of the lth pixel refers to the following requirements:

[0129] When the background mask corresponding to the lth pixel When the value is 0, the weight map The lth pixel corresponds to the foreground area, and the weight map corresponding to the lth pixel is expressed as:

[0130]

[0131] in, represents the gradient cutoff operator; It represents the color value corresponding to the lth pixel of the rendered underwater image;

[0132] When the background mask corresponding to the lth pixel When the value is 1, The lth pixel corresponds to the background area, and the weight map corresponding to the lth pixel is expressed as:

[0133]

[0134] Then, the foreground-enhanced underwater image can be obtained and foreground-enhanced underwater image ground truth Relationship:

[0135]

[0136] in, represents the element-wise multiplication operator;

[0137] ③Extract background mask All pixels with a value of 1 above are the background pixel set, which is recorded as , all pixels belonging to this set correspond to areas in the picture where there is no underwater scene and only pure water;

[0138] ④ Loss function based on image content Expressed as:

[0139]

[0140] in, Represents the loss function term The weight of Represents the loss function term The weight of Represents the loss function term The weight of Indicates the original 3DGS technology Loss function; Represents a pure water background image In the area No. The color value at each pixel; Represents the true value of underwater image In the area No. The color value at the pixel;

[0141] (2) Coarse-grained depth loss function The construction methods include:

[0142] The rendered depth map Converted into an approximate disparity map ,in represents the scale parameter;

[0143] The Pierce correlation coefficient with scale invariance is used as a similarity measure;

[0144] Coarse-grained depth loss function Expressed as:

[0145]

[0146] in, Represents the Pierce correlation coefficient calculation function; represents a pseudo depth map;

[0147] (3) Local smoothing regularization term for water body Gaussian opacity attribute , expressed as:

[0148]

[0149] (4) Local smoothing regularization term for water body Gaussian color attributes , expressed as:

[0150]

[0151] in, Represents the Gaussian set of water bodies involved in rendering; represents the set of k nearest neighboring water body Gaussians of the p-th water body Gaussian in the world coordinate system obtained by the KNN algorithm; represents the weight factor, and Expressed as ; represents the scale factor; Represents the mean attribute of the p-th water body Gaussian in the world coordinate system; Represents the mean attribute of the qth water body Gaussian in the world coordinate system.

[0152] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a 3D representation method for underwater scenes based on three-dimensional Gaussian splashing, which has the following beneficial effects:

[0153] The present invention can decouple complex water effects and the appearance color and geometric information of the scene itself from underwater images, and further can realize the rendering of the appearance color of the scene itself and the rendering of underwater images superimposed with water effects, thereby supporting the implementation of various downstream tasks such as underwater scene restoration, underwater new perspective synthesis, and underwater image restoration dataset construction.

[0154] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0155] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0156] Figure 1 Schematic diagram of the input and output of the channel-by-channel multi-output rasterization rendering module provided by an embodiment of the present invention.

[0157] Figure 2 Schematic diagram of the framework of the 3D characterization method for underwater scenes based on three-dimensional Gaussian splashing provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0158] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0159] The embodiment of the present invention discloses a 3D characterization method for underwater scenes based on 3D Gaussian splashing, comprising:

[0160] Obtain a set of original underwater images taken from a single underwater scene, use the COLMAP algorithm to calibrate the intrinsic and extrinsic parameter matrices of the camera corresponding to each underwater image in the original underwater image set, and output a sparse point cloud at the same time;

[0161] Using a pre-trained monocular depth estimator, we derive a pseudo-depth map corresponding to each underwater image in the original underwater image set.

[0162] Based on the sparse point cloud, the properties of the water body Gaussian are initialized to obtain the corresponding water body Gaussian set;

[0163] Based on the sparse point cloud and pseudo depth map, the properties of the scene Gaussian are initialized to obtain the corresponding scene Gaussian set;

[0164] The attribute parameters in the water body Gaussian set and the scene Gaussian set are iteratively optimized to decouple and capture the corresponding water body attributes and the appearance color and geometric attributes of the scene itself.

[0165] This embodiment of the present invention provides a 3D underwater scene representation method based on three-dimensional Gaussian splatting (3DGS). This method employs different 3D Gaussian primitive configuration strategies to explicitly represent the water and the scene itself in complex underwater scenes. A per-channel multi-output rasterization rendering module is introduced to implement rendering of different output types to meet the requirements of parameter optimization and various downstream tasks. The following sections first describe the proposed 3D Gaussian primitive configuration strategy, followed by a detailed description of the proposed per-channel multi-output rasterization rendering module, and then a detailed description of the implementation process of the proposed 3D underwater scene representation method.

[0166] 1. 3D Gaussian primitive setting strategy for water bodies and the scene itself

[0167] To address the unique characteristics of the translucent water medium and the opaque scene itself in underwater scenes, embodiments of the present invention design two different types of 3D Gaussian primitives to represent the water and the scene itself, respectively. Specifically, the 3D Gaussian primitive representing the water medium is referred to as the water Gaussian, and the 3D Gaussian primitive representing the scene itself is referred to as the scene Gaussian. The properties and characteristics of these two different types of Gaussian primitives are described below.

[0168] 1.1. Properties of the Water Gaussian Element

[0169] 1.1.1. Mean property of water Gaussian

[0170] The mean attribute setting of the 3D Gaussian basis element in the original 3DGS technology is the same. For each water body Gaussian, taking the p-th water body Gaussian as an example, its mean attribute is recorded as:

[0171]

[0172] in, is the mean attribute of the p-th water body Gaussian, representing the point coordinates in the world coordinate system; Represents the x-axis coordinate of the p-th water body Gaussian in the world coordinate system; Represents the y-axis coordinate of the p-th water body Gaussian in the world coordinate system; Represents the z-axis coordinate of the p-th water body Gaussian in the world coordinate system; parameter 、 and is a non-learnable parameter and remains fixed after initialization.

[0173] 1.1.2. Covariance properties of water Gaussian

[0174] Considering that the water medium in an underwater environment is relatively uniform overall, the main differences lie in the spatial non-uniformity caused by properties such as transmittance and color caused by local components and illumination direction. Therefore, unlike the anisotropic Gaussian distribution used in the original 3DGS technology, the distribution of the water Gaussian in the embodiment of the present invention is isotropic, corresponding to a spherical distribution. For each water Gaussian, taking the pth water Gaussian as an example, its covariance attribute is recorded as:

[0175]

[0176] in, represents the covariance property of the p-th water body Gaussian; Represents the shared parameter of the covariance attribute of all water body Gaussians, which is a non-learnable parameter and remains fixed after initialization; Represents the identity matrix of shape 3×3;

[0177] 1.1.3. Water Gaussian Opacity Property

[0178] Taking into account the spatial non-uniformity of water medium properties and the wavelength selectivity of water effects, for each water body Gaussian, taking the pth water body Gaussian as an example, its opacity attribute is recorded as:

[0179]

[0180] Different from the original 3DGS technology, in which the opacity attribute of each Gaussian primitive is a scalar, the opacity attribute of the water body Gaussian proposed in the embodiment of the present invention is It is a vector of length 3, corresponding to the three channels of RGB, to model the wavelength selectivity of the water effect. Represents the shared parameter of the opacity attribute of all water body Gaussians in the whole space; Indicates the opacity offset corresponding to the current p-th water body Gaussian; and are all vectors of length 3; The function guarantees The value range of each channel is (0,1). and are all learnable parameters.

[0181] 1.1.4. Water Gaussian Color Properties

[0182] Taking into account the spatial non-uniformity of water medium properties, different from the color attribute setting of each Gaussian basis element in the original 3DGS technology, for each water body Gaussian, taking the pth water body Gaussian as an example, its color attribute is recorded as:

[0183]

[0184] in, represents the color attribute of the p-th water body Gaussian; The shared parameter representing the color attribute of the Gaussian water body in the entire space is a vector of length 3, corresponding to the three channels of RGB; Indicates the color offset corresponding to the p-th water body Gaussian, which is a vector of length 3; Function is used to ensure The value range of each channel is (0,1); and is a learnable parameter.

[0185] 1.2. Attributes of Scene Gaussian Element

[0186] 1.2.1. Mean property of scene Gaussian

[0187] The mean attribute setting of the 3D Gaussian basis element in the original 3DGS technology is the same. For each scene Gaussian, taking the r-th scene Gaussian as an example, its mean attribute is recorded as:

[0188]

[0189] in, is the mean attribute of the Gaussian of the rth scene, representing the point coordinates in the world coordinate system; Represents the x-axis coordinate of the r-th scene Gaussian in the world coordinate system; Represents the y-axis coordinate of the r-th scene Gaussian in the world coordinate system; Represents the z-axis coordinate of the r-th scene Gaussian in the world coordinate system; 、 and is a learnable parameter.

[0190] 1.2.2. Covariance Properties of Scene Gaussians

[0191] The covariance attribute setting of the scene Gaussian in the embodiment of the present invention is the same as that of the 3D Gaussian basis element in the original 3DGS technology. The covariance attribute of the scene Gaussian in the embodiment of the present invention is anisotropic, corresponding to an ellipsoidal distribution. For each scene Gaussian, taking the rth scene Gaussian as an example, its covariance attribute is recorded as:

[0192]

[0193] in, represents the covariance property of the r-th scene Gaussian; The scale matrix representing the ellipsoidal distribution; The rotation matrix representing the ellipsoidal distribution; and are all learnable parameters.

[0194] 1.2.3. Opacity property of scene Gaussian

[0195] The opacity attribute setting of the 3D Gaussian primitives in the original 3DGS technology is the same. For each scene Gaussian, taking the rth scene Gaussian as an example, its opacity attribute is recorded as , which is a scalar value and a learnable parameter with a value range of (0,1).

[0196] 1.2.4. Color properties of scene Gaussians

[0197] Considering that underwater scenes typically exhibit Lambertian surface characteristics, unlike the original 3DGS technique, which models the color attributes of each Gaussian basis element using spherical harmonics, the color attributes of the scene Gaussian basis elements proposed in this embodiment are directly modeled using a vector of length 3. For each scene Gaussian, taking the rth scene Gaussian as an example, its color attribute is recorded as:

[0198]

[0199] in, represents the color attribute of the r-th scene Gaussian; 、 and They correspond to the three channels of RGB and are learnable parameters with a value range of (0,1).

[0200] 2. Channel-by-channel multi-output rasterization rendering module

[0201] In order to be compatible with the wavelength selectivity of water effects in underwater imaging, solve the problem that the opacity components of different channels of the water body Gaussian are independent of each other and are incompatible with the differentiable rasterization pipeline in the original 3DGS technology, and output the results to meet the needs of optimization and application, the embodiment of the present invention designs a channel-by-channel multi-output rasterization rendering module. Figure 1 As shown in the figure, the module includes four rendering branches: underwater image rendering branch, scene image rendering branch, pure water background image rendering branch and depth map rendering branch, which can realize underwater image , the image of the scene itself , pure water background image , Depth Map The following describes the rendering process of each rendering branch.

[0202] 2.1. Underwater Image Rendering Branch

[0203] ① The set of all water body Gaussians is recorded as , the set of all scene Gaussians is recorded as ;

[0204] ② For each water body Gaussian, taking the pth water body Gaussian as an example, its opacity attribute is re-expressed as ,in, 、 and for The three opacity components are independent of each other and do not affect each other, and correspond to the three RGB channels respectively;

[0205] ③For the rth scene Gaussian, its opacity attribute is reformulated as ,in, 、 and for The three opacity components correspond to the three RGB channels, and ;

[0206] ④ Given the camera perspective corresponding to the underwater image to be rendered, that is, the camera's intrinsic parameter matrix and extrinsic parameter matrix, find the water body Gaussian set and scene Gaussian set The union of , then the set All Gaussian primitives in the image are sorted by their depth values ​​in a given camera coordinate system for tile-based rendering. -blending rendering;

[0207] ⑤ The color of each channel of any pixel of the underwater image to be rendered (the position of the pixel in the image coordinate system is marked as x) It can be obtained according to the following formula:

[0208]

[0209] in, Represents the position coordinates of the pixel in the image coordinate system; Corresponding to different color channels; Represents the color value of the color channel of the i-th Gaussian primitive in the rendering process. Here, the color channel of the i-th Gaussian primitive is The color channel corresponds to represents the opacity contribution of the i-th Gaussian primitive to the corresponding color channel during the current pixel rendering process, and ; Represents the opacity component of the i-th Gaussian primitive in the corresponding color channel during rendering; Represents the value of the 2D Gaussian distribution of the i-th Gaussian primitive at the current pixel in the current image coordinate system during the rendering process. It should be noted that the i-th Gaussian primitive here refers to the i-th Gaussian primitive in the set of Gaussian primitives involved in the entire rendering process, and is different from the p-th water Gaussian primitive or the r-th scene Gaussian primitive mentioned earlier.

[0210] ⑥ It should be noted that unlike the original 3DGS technology, which uses the same set of opacity contributions for rendering all color channels of any pixel, in order to model the wavelength selectivity of water effects and address the issue of independent and distinct opacity components corresponding to different color channels of the water Gaussian, the "channel-by-channel multi-output rasterization rendering" module proposed in the embodiments of the present invention uses different opacity contributions for rendering different color channels of the same pixel. For any pixel of the underwater image to be rendered, each color channel is rendered independently. The color rendering process for a given pixel is not completed until all color channels have been rendered.

[0211] ⑦ Traverse all pixels of the underwater image to be rendered and complete the rendering to output the final underwater image .

[0212] 2.2. Scene image rendering branch

[0213] ①Extract scene Gaussian set All Gaussian primitives in ;

[0214] ② For each scene Gaussian, take the rth scene Gaussian as an example, and re-record its opacity attribute as a scalar ;

[0215] ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's intrinsic and extrinsic matrix, the rasterization rendering pipeline in the original 3DGS technology is used to All Gaussian primitives in the set are processed to output the final image of the scene itself .

[0216] 2.3. Pure water background image rendering branch

[0217] ①Extract water body Gaussian set All Gaussian primitives in ;

[0218] ② Same as the “Underwater Image Rendering Branch”, for each water body Gaussian, taking the p-th water body Gaussian as an example, its opacity attribute is re-expressed as ,here The three opacity components are independent of each other and do not affect each other;

[0219] ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's internal parameter matrix and external parameter matrix, the water body Gaussian set All Gaussian primitives in the image are sorted by their depth values ​​in a given camera coordinate system for tile-based rendering. -blending rendering.

[0220] ④ Any pixel of the pure water background image to be rendered (the position coordinate of the pixel in the image coordinate system is marked as ) of each channel It can be obtained according to the following formula:

[0221]

[0222] in, Represents the position coordinates of the pixel in the image coordinate system; Corresponding to different color channels; Represents the color value of the color channel of the i-th Gaussian primitive in the rendering process. Here, the color channel of the i-th Gaussian primitive is The color channel corresponds to represents the opacity contribution of the i-th Gaussian primitive to the corresponding color channel during the current pixel rendering process, and ; Represents the opacity component of the i-th Gaussian primitive in the corresponding color channel during rendering; Indicates the value of the 2D Gaussian distribution of the i-th Gaussian basis element at the current pixel in the current image coordinate system during the rendering process.

[0223] ⑤ Same as the “Underwater Image Rendering Branch”, any pixel of the pure water background image is rendered (the pixel’s position in the image coordinate system is marked as ) are rendered independently for each channel. Only when all color channels of the pixel are rendered, the color rendering process of the pixel is truly completed.

[0224] ⑥ Traverse all pixels of the pure water background image to be rendered and complete the rendering to output the final pure water background image .

[0225] 2.4. Depth Map Rendering Branch

[0226] ①Extract scene Gaussian set All Gaussian primitives in participate in rasterization and rendering.

[0227] ② Same as the “scene image rendering branch”, for each scene Gaussian, take the rth scene Gaussian as an example, and re-record its opacity attribute as a scalar .

[0228] ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's intrinsic parameter matrix and extrinsic parameter matrix, the scene Gaussian set All Gaussian primitives in the image are sorted by their depth values ​​in a given camera coordinate system for tile-based rendering. -blending rendering.

[0229] ④ Depth map to be rendered Any pixel (the pixel's position in the image coordinate system is marked as ) can be obtained according to the following formula:

[0230]

[0231] in, Represents the position coordinates of the pixel in the image coordinate system, represents the opacity contribution of the i-th Gaussian primitive in the current pixel rendering process, and ; Indicates the value of the 2D Gaussian distribution of the i-th Gaussian basis element at the current pixel in the current image coordinate system during the rendering process; Indicates the depth value of the i-th Gaussian primitive in the current camera coordinate system during rendering;

[0232] ⑤ When all pixels of the image to be rendered are rendered, the desired depth map can be obtained. .

[0233] 3. Implementation Process of the 3D Underwater Scene Characterization Method Based on 3D Gaussian Splashing Provided in the Embodiments of the Present Invention

[0234] Figure 2 This diagram illustrates a flow chart of a 3D underwater scene characterization method based on three-dimensional Gaussian splashing, provided by an embodiment of the present invention. Given a set of raw underwater images captured underwater, the method first undergoes a "COLMAP-based image preprocessing stage" and a "pseudo-depth map derivation stage." The "COLMAP-based image preprocessing stage" outputs the camera's intrinsic parameter matrix, extrinsic parameter matrix, and sparse point cloud required for subsequent stages. The "pseudo-depth map derivation stage" then outputs a pseudo-depth map corresponding to each image in the raw underwater image set, required for subsequent stages. Subsequently, the "water body Gaussian initialization stage" and the "scene Gaussian initialization stage" generate corresponding water body Gaussian sets and scene Gaussian sets. The "Gaussian basis learnable parameter optimization stage" then iteratively optimizes parameters to decouple and capture the corresponding water body properties and the scene's appearance, color, and geometric properties, meeting the requirements of the "downstream task application stage." The details of each stage are detailed below.

[0235] 3.1. COLMAP-based image preprocessing stage

[0236] Given a set of raw underwater images containing a single underwater scene , use the COLMAP algorithm to calibrate the intrinsic parameter matrix and extrinsic parameter matrix of the camera corresponding to each underwater image in the image set, and output a sparse point cloud (Sparse Point Cloud), recorded as , for the subsequent Gaussian primitive initialization and perspective rendering requirements.

[0237] 3.2. Pseudo depth map export stage

[0238] The coupled water effects in underwater images pose a huge challenge to the development of 3D characterization. Therefore, the embodiments of the present invention attempt to use the powerful geometric prior capabilities of the pre-trained monocular depth estimator to guide the initialization and optimization process of the underwater scene 3D characterization method.

[0239] Specifically, a pre-trained monocular depth estimator, such as the DepthAnything algorithm, is used to derive a pseudo depth map corresponding to each underwater image in the original underwater image set. Typically, the pseudo depth map derived by the monocular depth estimator reflects relative disparity information, and the pseudo depth value range is [0,1]. The pseudo depth value corresponding to the distant area is smaller, and the pseudo depth value corresponding to the nearby area is larger.

[0240] 3.3. Water body Gaussian initialization stage

[0241] Different from the attribute initialization strategy of the 3D Gaussian primitive in the original 3DGS technology, the initialization methods of various water body Gaussian attributes proposed in the embodiments of the present invention are introduced below.

[0242] (1) Initialize the mean attribute of the water body Gaussian:

[0243] Sparse point cloud derived from COLMAP The distribution range is calculated and set as a cuboid bounding box (AABB, axis-aligned bounding box) as the scene boundary to limit the scene range;

[0244] Inside the scene boundary, a regular grid is constructed in the X, Y, and Z coordinate axes with a fixed step size. A water body Gaussian is placed at each grid node, and the position of the grid node in the world coordinate system is used as the mean attribute of the corresponding water body Gaussian.

[0245] (2) Initialize the covariance attribute of the water body Gaussian:

[0246] The covariance attribute shared parameters of all water body Gaussians , according to the formula Initialize; among them, represents the scale factor; Indicates a fixed step size;

[0247] (3) Initialize the opacity attribute of the water body Gaussian:

[0248] For each water body Gaussian, take the pth water body Gaussian as an example, and set its opacity offset Initialized to [0,0,0];

[0249] Share the opacity attributes of all water body Gaussians Initialized to To ensure the opacity property of the water body Gaussian Initialized to [0.1, 0.1, 0.1];

[0250] (4) Initialize the color attributes of the water body Gaussian:

[0251] For each water body Gaussian, take the pth water body Gaussian as an example, and set its color offset Initialized to [0,0,0];

[0252] Share the color attributes of all water body Gaussians is initialized to To ensure the Gaussian color properties of the water is initialized to [0.1, 0.1, 0.1].

[0253] (5) All water body Gaussians whose mean attributes, covariance attributes, opacity attributes and color attributes have been initialized are recorded as the water body Gaussian set.

[0254] 3.4. Scene Gaussian Initialization Phase

[0255] The water effect present in underwater images causes the sparse point cloud output by COLMAP to be severely sparse. In particular, the imaging results of areas far from the camera are severely affected by the water effect. The number of spatial points corresponding to the corresponding distant areas is extremely small, which is not conducive to the initialization and optimization of 3D Gaussian primitives. To solve this problem, in an embodiment of the present invention, a "pseudo-depth map-guided sparse point cloud augmentation stage" is first used to augment and densify the overly sparse point cloud. Then, a "scene Gaussian initialization stage based on the augmented point cloud" is used to initialize the properties of the scene Gaussian to obtain the scene Gaussian set. The specific implementation of these two stages is described below.

[0256] 3.4.1. Pseudo-depth map guided sparse point cloud augmentation stage

[0257] In the embodiment of the present invention, the pseudo-depth map-guided sparse point cloud expansion stage aims to compensate for the problem of excessive sparseness of the COLMAP point cloud caused by severe underwater image degradation and difficulty in feature matching. Specifically:

[0258] (1) Based on the pseudo depth map corresponding to each underwater image in the original underwater image set, each underwater image is divided into multiple regions according to the preset depth value interval, and each region corresponds to a different depth interval on the pseudo depth map;

[0259] For example, suppose the original underwater image set have Take the hth underwater image as an example, and divide the image into 10 regions according to its corresponding pseudo depth map and the depth value interval of 0.1. , each area corresponds to a different depth interval on the pseudo depth map, for example, the hth The tth region of the underwater image The corresponding pixel pseudo depth value is in the range of [0.1*(t-1),0.1*t).

[0260] (2) Extract the sparse point cloud subset that can be observed by the camera perspective corresponding to each underwater image in the original underwater image set, calculate the depth value of all points in the sparse point cloud subset in the corresponding camera coordinate system, and obtain the projection points of all points in the sparse point cloud subset in the corresponding image coordinate system, which are recorded as original projection points;

[0261] Repeat the same operation for each original underwater image: Take the hth underwater image as an example, extract the COLMAP sparse point cloud subset that can be observed by the corresponding camera perspective , and calculate the sparse point cloud subset The depth values ​​of all points in the corresponding h-th camera coordinate system, as well as the projection points of these points on the corresponding h-th image coordinate system (referred to as original projection points), the set of all original projection points is referred to as the original projection point set ;

[0262] (3) For each area of ​​each underwater image in the original underwater image set, calculate the number of pixels covered by the current area, determine the original projection point subset of the sparse point cloud subset corresponding to the current underwater image in the current area, and determine the number of original projection points contained in the current original projection point subset, calculate the number of projection points to be added in the current area, and randomly add the corresponding number of projection points in the current area, which are recorded as newly added projection points;

[0263] Repeat the same operation for each region of each original underwater image: take the tth region of the hth underwater image For example, calculate the number of pixels covered by the current area , determine the original projection point set of the current underwater image The subset of original projected points in the tth region , and determine the number of original projection points contained in the original projection point subset , the minimum projection point density contained in each area is specified as , let the set of projection points to be added in the tth region be , then the number of projection points to be added in the tth region is The calculation formula is as follows:

[0264]

[0265] in, Represents the original underwater image set The set of projection points to be added in the t-th region of the h-th underwater image; Representing a collection The number of projection points to be added in ; if If the value is 0, it means no new projection points are needed; if , then the number of random additions in the tth region is The projection point of .

[0266] (4) Repeat the same operation for each newly added projection point of each region of each underwater image in the original underwater image set: take the tth region of the hth underwater image The mth newly added projection point in For example, the corresponding new spatial point Depth value in camera coordinate system and color attributes Expressed as:

[0267]

[0268] Wherein, m in the formula represents the mth newly added projection point; Indicates that from the current area The original projection point set The set of k original projection points closest to the mth newly added projection point found by the K-Nearest Neighbor (KNN) algorithm; express The depth value of the original space point corresponding to the nth original projection point in the current camera coordinate system; express The color value of the original space point corresponding to the nth original projection point in the current camera coordinate system; represents weighted weight;

[0269] Among them, the coordinates of the mth newly added projection point in the image coordinate system are marked as , the coordinate of the nth original projection point in the image coordinate system is recorded as , then the weighted weight Expressed as:

[0270]

[0271] (5) According to the camera projection transformation and the camera's external parameter matrix, the position of each newly added spatial point corresponding to the newly added projection point in the world coordinate system is obtained; specifically:

[0272] Let the width of the image taken by the current camera be w, the height be h, and the focal length be f. The corresponding transformation matrix of the camera coordinate system is (This matrix is ​​the inverse matrix of the camera's external parameter matrix). According to the basic principle of projection transformation, the mth newly added projection point The corresponding new spatial point The coordinates in the current camera coordinate system are , add a new space point The homogeneous coordinates in the world coordinate system can be obtained by the formula: get;

[0273] (6) Calculate the position attributes and color attributes of all newly added spatial points in all areas of all underwater images in the original underwater image set. The union of the set of these newly added spatial points and the set of original spatial points corresponding to the sparse point cloud is the desired augmented point cloud.

[0274] 3.4.2. Scene Gaussian initialization based on augmented point cloud

[0275] In this embodiment of the present invention, during the scene Gaussian initialization stage based on the augmented point cloud, the same Gaussian primitive initialization strategy as the original 3DGS is adopted. Specifically, the mean, covariance, color, and opacity attributes of each scene Gaussian primitive are initialized using the default 3DGS initialization method. The only difference is that the point cloud used to initialize the scene Gaussian is no longer the sparse point cloud output by COLMAP, but the "pseudo-depth map-guided augmented point cloud" provided by this embodiment of the present invention.

[0276] 3.5. Gaussian learnable parameter optimization stage

[0277] In the embodiment of the present invention, the learnable parameters of the Gaussian primitives are gradually optimized through multiple iterations to achieve accurate 3D representation of underwater scenes. Specifically, in each iteration, an underwater image is randomly selected from the original underwater image set. and its corresponding pseudo depth map , using the current iteration of the water body Gaussian primitives and the scene Gaussian primitives through the proposed "channel-by-channel multi-output rasterization rendering module" to obtain the underwater image under the same camera perspective , the image of the scene itself , pure water background image and depth map By calculating the loss function and returning the gradient, the water body Gaussian properties and the scene Gaussian properties are optimized. The following first introduces the water body Gaussian property optimization strategy and the scene Gaussian property optimization strategy, and then introduces the loss function used in the embodiment of the present invention.

[0278] 3.5.1. Water Gaussian property optimization strategy

[0279] During the optimization process, the mean attribute and covariance attribute of each water body Gaussian remain frozen after initialization and no longer participate in the optimization process. The optimization process of the opacity attribute and color attribute of each water body Gaussian is divided into two stages according to the number of iteration rounds. The optimization strategy of the opacity attribute and color attribute of each water body Gaussian is introduced in detail below.

[0280] The total number of iterations is divided into two stages. For example, if the total number of iterations is 30,000, the first 15,000 iterations are the first stage, and the last 15,000 iterations are the second stage.

[0281] During the first stage of optimization, the opacity offset of each water body Gaussian and color offset Keep frozen and do not participate in optimization, only optimize the opacity attribute shared parameters of all water body Gaussians Shared parameters with color attributes .

[0282] During the second stage of optimization, unfreeze the opacity offset of each water body Gaussian and color offset , and shares parameters with the opacity attribute shared by all water body Gaussians Shared parameters with color attributes Optimize together.

[0283] During the entire water body Gaussian optimization process, the Adaptive Density Control strategy in the original 3DGS technology is no longer implemented on the water body Gaussian.

[0284] 3.5.2. Attribute Optimization Strategy for Scene Gaussian

[0285] During the optimization process, the Gaussian attribute optimization strategy for all scenes adopts the conventional Gaussian basis element attribute optimization strategy used by the original 3DGS technology, and still adopts the adaptive density control strategy used by the original 3DGS technology.

[0286] 3.5.3. Loss Function

[0287] In an embodiment of the present invention, the loss functions used in the optimization process include four loss functions based on image content, coarse-grained depth loss function, local smoothing regularization term of water body Gaussian opacity attribute, and local smoothing regularization term of water body Gaussian color attribute. The specific principles and implementations of each loss function are introduced in detail below.

[0288] 3.5.3.1. Image Content-Based Loss Function

[0289] The original 3DGS technology used Loss and Color reconstruction loss functions such as loss face the following problems in the 3D representation of underwater scenes: the first problem is that the attenuation effect of the water body reduces the brightness of the seabed area in the underwater image, which will produce insufficient gradients for the 3D optimization of the corresponding scene area; the second problem is that these color reconstruction loss functions are only used in the original underwater image. and underwater images to be rendered The lack of direct supervision of the water body Gaussian optimization will lead to suboptimal characterization results. Therefore, in the embodiment of the present invention, a loss function based on image content is designed. On the one hand, it strengthens the representation of the seabed area (foreground) where the attenuation effect is serious, and on the other hand, it uses the color information of the pure water background area (background) to directly supervise the Gaussian properties of the water body. The following is a detailed introduction to the loss function based on image content. Calculation method.

[0290] ①Use the pseudo depth map of the current iteration round To create a background mask . Given a predefined depth threshold , the pseudo depth value is greater than The area composed of pixels is considered as the foreground region, and the corresponding background mask The pixel value at the corresponding position is 0; the pseudo depth value is less than or equal to The area composed of pixels is considered as the background region (Background Region), and the corresponding background mask The pixel value at the corresponding position is set to 1.

[0291] ② The underwater image in the original underwater image set is recorded as the true value of the underwater image, and the symbol is , construct a weight map with the same shape as the pseudo depth map Specifically, the weight graph The value of the lth pixel refers to the following requirements:

[0292] When the background mask corresponding to the lth pixel When the value is 0, the weight map The lth pixel corresponds to the foreground area. At this time, the weight map corresponding to the lth pixel is expressed as:

[0293]

[0294] in, represents the gradient cutoff operator; Represents the rendered underwater image The color value corresponding to the lth pixel;

[0295] When the background mask corresponding to the lth pixel When the value is 1, The lth pixel corresponds to the background area, and the weight map corresponding to the lth pixel is expressed as:

[0296]

[0297] Then, we get the foreground-enhanced underwater image. and foreground-enhanced underwater image ground truth Relationship:

[0298]

[0299] in, Represents the element-wise multiplication operator.

[0300] ③Extract background mask All pixels with a value of 1 above are the background pixel set, which is recorded as , all pixels belonging to this set correspond to areas in the picture where there is no underwater scene and only pure water.

[0301] ④ Loss function based on image content Expressed as:

[0302]

[0303] in, Represents the loss function term The weight of Represents the loss function term The weight of Represents the loss function term The weights of these three weights are used to balance the impact of different loss function items on the final optimization results, and their values ​​are ; Indicates the original 3DGS technology Loss function; Represents a pure water background image In the area No. The color value at the pixel; Represents the true value of underwater image In the area No. The color value at the pixel.

[0304] 3.5.3.2. Coarse-grained depth loss function

[0305] The inconsistency of perspectives between images in the original underwater image set will cause the geometric constraint capability to decrease during the optimization process, making it difficult for the optimized 3D representation to accurately reflect the geometric information of the scene. To solve this problem, the embodiment of the present invention designs a coarse-grained depth loss function. , using a pseudo depth map The geometric priors provided serve as additional coarse-grained geometric supervision.

[0306] First, due to the pseudo depth map It reflects relative disparity information rather than absolute depth information. In this embodiment of the present invention, the depth map obtained by rendering Converted into an approximate disparity map ,in is a scale parameter; approximate disparity map It has the value characteristics of "near is big and far is small" similar to the parallax space.

[0307] Secondly, in order to solve the problem of pseudo depth map and approximate disparity map To solve the scale ambiguity problem between the two, the embodiment of the present invention adopts the Pierce correlation coefficient with scale invariance as a similarity measure.

[0308] Finally, the coarse-grained depth loss function The calculation method is , this formula is the intermediate variable after converting the rendered depth map into the disparity space; Represents the Pierce correlation coefficient calculation function.

[0309] 3.5.3.3. Local Smoothing Regularization Term for Water Body Gaussian Opacity Attribute , and the local smoothing regularization term of the water body Gaussian color attribute ;

[0310] In the second stage of water body Gaussian attribute optimization, in order to ensure the optimization smoothness of the opacity offset attribute and color offset attribute of each water body Gaussian, the embodiment of the present invention designs the water body Gaussian opacity attribute local smoothing regularization term and the local smoothing regularization term of the water body Gaussian color attribute , the specific calculation method of these two regular terms is as follows:

[0311]

[0312]

[0313] in, Represents the Gaussian set of water bodies involved in rendering; Represents the water body Gaussian set The opacity offset of the p-th water body Gaussian; Represents the water body Gaussian set The color offset of the p-th water body Gaussian; Indicates the The world coordinate system obtained by the KNN algorithm for the water body Gauss The set of nearest neighbor water body Gaussians; the one mentioned here is a weighting factor. The calculation formula is , is a scaling factor, and They are A water body Gaussian and its The mean properties of the neighboring water body Gaussians in the world coordinate system.

[0314] 3.5.3.4. Final total loss function

[0315] The total loss function used in the final training The calculation settings are as follows:

[0316] In the first stage of optimization, , .

[0317] In the second stage of optimization, , , , ; Represents the coarse-grained depth loss function The weight of Represents the local smoothing regularization term of the water body Gaussian opacity attribute weight; Represents the local smoothing regularization term of the water body Gaussian color attribute The weight of 、 and Used to balance the impact of different loss function items on the final optimization results.

[0318] The entire optimization process uses the Adam optimizer to optimize the learnable parameters of all Gaussian primitives.

[0319] 3.6. Downstream Task Application Stage

[0320] The 3D representation method for underwater scenes proposed in this embodiment of the present invention can meet the needs of various downstream tasks. The following examples illustrate its application in three downstream tasks: underwater scene restoration, underwater new perspective synthesis, and underwater image restoration dataset construction.

[0321] 3.6.1. Example of underwater scene restoration task

[0322] In the first step, a set of corresponding original underwater images is collected for an underwater scene, and the 3D representation of the corresponding underwater scene is learned using the method proposed in the embodiment of the present invention.

[0323] The second step is to give each underwater image in the original underwater image set Corresponding to the camera perspective, the scene image rendering branch is enabled, and the scene image in the scene Gaussian set is Rendering. Since the scene itself is image The water body Gaussian set is not used in the rendering process, so it can reflect the real appearance color of the scene when there is no water effect. Can be considered as underwater image Image restoration results.

[0324] 3.6.2. Underwater New Perspective Synthesis Task Example

[0325] In the first step, a set of corresponding original underwater images is collected for an underwater scene, and the 3D representation of the corresponding underwater scene is learned using the method proposed in the embodiment of the present invention.

[0326] The second step is to enable the underwater image rendering branch given any camera perspective and render all Gaussian primitives in the union of the water body Gaussian set and the scene Gaussian set. The underwater image rendering branch can then be used to render the underwater image under the corresponding easy perspective. Because the underwater image rendering branch uses all Gaussian primitives in the union of the water body Gaussian set and the scene Gaussian set for rendering, the rendered underwater image is superimposed with the influence of the water body effect, reflecting the degraded appearance of the scene under the influence of the water body effect. Therefore, given any camera perspective, the 3D representation learned by the embodiment of the present invention can be used to render underwater images from the corresponding perspective, which means that the synthesis of new underwater perspectives is achieved.

[0327] 3.6.3. Example of underwater image restoration dataset construction task

[0328] In the first step, a set of corresponding original underwater images is collected for an underwater scene, and the 3D representation of the corresponding underwater scene is learned using the method proposed in the embodiment of the present invention.

[0329] In the second step, given a set of camera perspectives, for each camera perspective in the set, the scene image rendering branch and the underwater image rendering branch are enabled to render the scene image without water effects. and underwater images with water effects . For each camera perspective rendered and In terms of the image content of the two, they reflect the content of the same underwater scene shot from the same camera perspective. The difference is reflects the true appearance color of the scene itself, while It reflects the appearance of the scene with color degradation under the influence of water. Therefore, for each camera perspective, the rendered and They can be regarded as water-free true images and water-degraded images in the underwater image restoration dataset, respectively, to construct image pairs that can be used for supervised learning.

[0330] Compared with the existing methods, the embodiments of the present invention have the following advantages:

[0331] (1) Joint optimization of water parameters, scene geometry, and appearance color: In order to address the problem faced by traditional 3D representation methods of underwater scenes that it is difficult to achieve optimal representation of both scene geometry and appearance color through staged optimization, the embodiments of the present invention simultaneously set Gaussian primitives representing the water medium and Gaussian primitives representing the scene itself in space, and render underwater images through a customized differentiable rasterization pipeline. In this way, the parameters of the water medium and the color geometry parameters of the scene itself can be learned simultaneously in the end-to-end optimization process, thereby achieving better scene representation quality.

[0332] (2) Explicitly represent the water body and the scene itself to achieve high-speed and high-quality representation: In order to address the problems of huge training and rendering time and computing power overhead and unsatisfactory rendering quality caused by the implicit representation of the water body and the scene itself in the NeRF-based underwater scene 3D representation method, the embodiment of the present invention designs two types of attribute-customized Gaussian primitives - water body Gaussian and scene Gaussian - based on the characteristics of the water environment and the scene itself during underwater imaging, to explicitly represent the water body and the scene itself in the scene, respectively. This not only achieves faster training and rendering speeds, but also realizes fine-grained scene 3D representation and achieves richer rendering quality.

[0333] (3) Directly and explicitly modeling water bodies to avoid dependence on the modeling quality of the scene itself: In response to the existing method of using 3DGS technology to perform 3D representation of underwater scenes, the embodiment of the present invention directly models the influence of water effects during underwater imaging by designing water body Gaussian basis elements and distributing them in large quantities in space. The optimization process of water body parameters is changed from relying on the reconstruction quality of the Gaussian of the scene itself as adopted by the existing method to being parallel to the optimization of the Gaussian of the scene itself, thereby being able to more accurately and effectively decouple the real appearance, geometry and water properties of the underwater scene in the end-to-end joint optimization process.

[0334] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0335] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A 3D characterization method for underwater scenes based on three-dimensional Gaussian splashing, characterized in that: include: Obtain a set of original underwater images taken from a single underwater scene, use the COLMAP algorithm to calibrate the intrinsic and extrinsic parameter matrices of the camera corresponding to each underwater image in the original underwater image set, and output a sparse point cloud at the same time; Using a pre-trained monocular depth estimator, we derive a pseudo-depth map corresponding to each underwater image in the original underwater image set. Based on the sparse point cloud, the properties of the water body Gaussian are initialized to obtain the corresponding water body Gaussian set; Based on the sparse point cloud and pseudo depth map, the properties of the scene Gaussian are initialized to obtain the corresponding scene Gaussian set; Iteratively optimize the attribute parameters in the water body Gaussian set and the scene Gaussian set to decouple and capture the corresponding water body attributes and the scene's appearance color and geometric attributes; During each iterative optimization of the attribute parameters in the water body Gaussian set and the scene Gaussian set, an underwater image is selected from the original underwater image set. and its corresponding pseudo depth map The water body Gaussian primitives and scene Gaussian primitives of the current iteration are used to obtain the underwater image I and the scene image I under the same camera perspective through the channel-by-channel multi-output rasterization rendering module. s , pure water background image I w and depth map D; by calculating the loss function and returning the gradient, the Gaussian properties of the water body and the scene are optimized; In the process of iteratively optimizing the attribute parameters of the water body Gaussian set, the attribute optimization strategy of the water body Gaussian includes: (1) The mean and covariance attributes of each water body Gaussian remain frozen after initialization and no longer participate in the optimization process; (2) According to the number of parameter optimization iterations, the optimization process of the opacity attribute and color attribute of each water body Gaussian is divided into two stages: During the first stage of optimization, the opacity offset Δo of each water body Gaussian w，p and color shift Δc w，p Keep frozen and do not participate in the optimization, only optimize the opacity attribute shared parameters of all water body Gaussians o w，global Shares parameter c with color attribute w，global ; During the second stage of optimization, the opacity offset Δo of each water body Gaussian is unfrozen w，p and color shift Δc w，p , and shares the parameter o with the opacity attribute shared by all water body Gaussians w，global Shares parameter c with color attribute w，global Optimize together.

2. The 3D characterization method for underwater scenes based on 3D Gaussian splashing according to claim 1, characterized in that: The properties of the water body Gaussian specifically include: (1) Mean property of water body Gaussian: For each water body Gaussian, its mean attribute is recorded as: Among them, μ w，p is the mean attribute of the p-th water body Gaussian, representing the point coordinates in the world coordinate system; Represents the x-axis coordinate of the p-th water body Gaussian in the world coordinate system; Represents the y-axis coordinate of the p-th water body Gaussian in the world coordinate system; Represents the z-axis coordinate of the p-th water body Gaussian in the world coordinate system; parameter and It is a non-learnable parameter and remains fixed after initialization; (2) Covariance properties of water body Gaussian: The covariance property of the water body Gaussian is modeled as isotropic corresponding to the spherical distribution; for each water body Gaussian, its covariance property is recorded as: Among them, ∑ w，p represents the covariance property of the p-th water body Gaussian; σ w represents the shared parameter of the covariance attribute of all water body Gaussians, which is a non-learnable parameter and remains fixed after initialization; E represents the identity matrix with a shape of 3×3; (3) Opacity property of water body Gaussian: For each water body Gaussian, its opacity attribute is recorded as: zero w，p =Sigmoid(o w，global +Δo w，p ) Among them, w，p represents the opacity attribute of the p-th water body Gaussian, which is a vector of length 3, corresponding to the three channels of RGB, to model the wavelength selectivity of the water effect; w，global Represents the shared parameter of the opacity attribute of all water bodies; Δo w，p Indicates the opacity offset corresponding to the current p-th water body Gaussian; o w，global and Δo w，p are all vectors of length 3; the Sigmoid(·) function is used to ensure that o w，p The value range of each channel is (0,1); w，global and Δo w，p All are learnable parameters; (4) Color properties of water body Gauss: For each water body Gaussian, its color attribute is recorded as: c w，p =Sigmoid(c w，global +Δc w，p ) Among them, c w，p represents the color attribute of the p-th water body Gaussian; c w，global Represents the shared parameter of the color attribute of the Gaussian water body in the entire space, which is a vector of length 3, corresponding to the three channels of RGB; Δc w，p represents the color offset corresponding to the p-th water body Gaussian, which is a vector of length 3; the Sigmoid(·) function is used to ensure Δc w，p The value range of each channel is (0,1); C w，global and Δc w，p is a learnable parameter.

3. The 3D characterization method for underwater scenes based on 3D Gaussian splashing according to claim 1, characterized in that: The properties of the scene Gaussian include: (1) Mean property of scene Gaussian: For each scene Gaussian, its mean attribute is recorded as: Among them, μ s，r is the mean attribute of the Gaussian of the rth scene, representing the point coordinates in the world coordinate system; Represents the x-axis coordinate of the r-th scene Gaussian in the world coordinate system; Represents the y-axis coordinate of the r-th scene Gaussian in the world coordinate system; Represents the z-axis coordinate of the r-th scene Gaussian in the world coordinate system; and is a learnable parameter; (2) Covariance properties of scene Gaussian: The covariance property of the scene Gaussian is modeled as the anisotropy of the corresponding ellipsoidal distribution; for each scene Gaussian, its covariance property is recorded as: ∑ s，r =RSS T R T Among them, ∑ s，r represents the covariance property of the Gaussian of the rth scene; s represents the scale matrix of the ellipsoid distribution; R represents the rotation matrix of the ellipsoid distribution; s and R are both learnable parameters; (3) Opacity property of scene Gaussian: For each scene Gaussian, its opacity attribute is recorded as o s，r ;o s，r represents the opacity attribute of the r-th scene Gaussian, which is a scalar value and a learnable parameter with a value range of (0,1); (4) Color properties of scene Gaussian: For each scene Gaussian, its color attribute is recorded as: Among them, c s，r represents the color attribute of the r-th scene Gaussian; and They correspond to the three channels of RGB and are learnable parameters with a value range of (0,1).

4. The 3D characterization method for underwater scenes based on 3D Gaussian splashing according to claim 1, characterized in that: The method of initializing the properties of the water body Gaussian based on the sparse point cloud to obtain the corresponding water body Gaussian set specifically includes: (1) Initialize the mean attribute of the water body Gaussian: According to the distribution range of the sparse point cloud, a rectangular bounding box is calculated and set as the scene boundary to limit the scene range; Inside the scene boundary, a regular grid is constructed in the X, Y, and Z coordinate axes with a fixed step size. A water body Gaussian is placed at each grid node, and the position of the grid node in the world coordinate system is used as the mean attribute of the corresponding water body Gaussian. (2) Initialize the covariance attribute of the water body Gaussian: The covariance properties of all water bodies Gaussian share the parameter σ w , according to the formula σ w =λ σ Δs is initialized; where λ σ represents the scale factor; Δs represents the fixed step size; (3) Initialize the opacity attribute of the water body Gaussian: For each water body Gaussian, offset its opacity by Δo w,p Initialized to [0,0,0]; Share the opacity attribute of all water body Gaussians with parameter o w,global Initialized to [-ln9,-ln9,-ln9]; (4) Initialize the color attributes of the water body Gaussian: For each water body Gaussian, its color shift Δc w,p Initialized to [0,0,0]; The color attributes of all water bodies Gaussian share the parameter c w,global Initialized to [-ln9, -ln9, -ln9]; (5) All water body Gaussians whose mean attributes, covariance attributes, opacity attributes and color attributes have been initialized are recorded as the water body Gaussian set.

5. The 3D characterization method for underwater scenes based on 3D Gaussian splashing according to claim 1, characterized in that: The method of initializing the properties of the scene Gaussian based on the sparse point cloud and the pseudo depth map to obtain the corresponding scene Gaussian set includes: Amplify the sparse point cloud according to the pseudo depth map to obtain an amplified point cloud; Based on the augmented point cloud, the attributes of the scene Gaussian are initialized to obtain the corresponding scene Gaussian set.

6. The 3D characterization method for underwater scenes based on 3D Gaussian splashing according to claim 5, characterized in that: The amplifying process of the sparse point cloud according to the pseudo depth map to obtain the amplified point cloud specifically includes: (1) According to the pseudo depth map corresponding to each underwater image in the original underwater image set, each underwater image is divided into multiple regions according to the preset depth value interval, and each region corresponds to a different depth interval on the pseudo depth map; (2) Extract the sparse point cloud subset that can be observed by the camera perspective corresponding to each underwater image in the original underwater image set, calculate the depth value of all points in the sparse point cloud subset in the corresponding camera coordinate system, and obtain the projection points of all points in the sparse point cloud subset in the corresponding image coordinate system, which are recorded as original projection points; (3) For each area of ​​each underwater image in the original underwater image set, calculate the number of pixels covered by the current area, determine the original projection point subset of the sparse point cloud subset corresponding to the current underwater image in the current area, and determine the number of original projection points contained in the current original projection point subset, calculate the number of projection points to be added in the current area, and randomly add a corresponding number of projection points in the current area, which are recorded as newly added projection points; The number of projection points to be added in the current area is expressed as: in, Represents the original underwater image set The set of projection points to be added in the t-th region of the h-th underwater image; Representing a collection The number of new projection points in the region; ρ represents the minimum projection point density in each region; Ω h，t represents the tth region of the hth underwater image in the original underwater image set; |Ω h，t | represents Ω h，t The number of pixels contained in ; represents the subset of original projection points of the h-th underwater image in the t-th region of the original underwater image set; Represents the original projection point subset The number of original projection points contained in ; (4) For each newly added projection point in each area of ​​each underwater image in the original underwater image set, the depth value d of the corresponding newly added spatial point in the camera coordinate system m and color attribute c m Expressed as: Where m represents the mth newly added projection point; KNN(m, k) represents the set of the nearest k original projection points of the mth newly added projection point found by the K nearest neighbor algorithm from the original projection point set of the current area; d n represents the depth value of the original space point corresponding to the nth original projection point in KNN (m, k) in the current camera coordinate system; c n W represents the color value of the original space point corresponding to the nth original projection point in KNN (m, k) in the current camera coordinate system; m，n represents weighted weight; The coordinates of the mth newly added projection point in the image coordinate system are denoted as (u m , v m ), the coordinate of the nth original projection point in the image coordinate system is called (u n , v n ), then the weighted weight w m，n Expressed as: (5) According to the camera projection transformation and the camera's external parameter matrix, the position of each newly added spatial point corresponding to the newly added projection point in the world coordinate system is obtained; (6) Calculate the positions and color attributes of all newly added spatial points in all areas of all underwater images in the original underwater image set. The union of the set of these newly added spatial points and the set of original spatial points corresponding to the sparse point cloud is the augmented point cloud.

7. The method for representing underwater scenes in 3D based on 3D Gaussian splashing according to claim 5, characterized in that: The channel-by-channel multi-output rasterization rendering module includes: (1) Underwater image rendering branch: ① The set of all water body Gaussians is denoted as S w , the set of all scene Gaussians is recorded as S s ; ②For the p-th water body Gaussian, its opacity attribute is re-expressed as in, and for o w,p The three opacity components are independent of each other and correspond to the three RGB channels respectively; ③For the rth scene Gaussian, its opacity attribute is reformulated as in, and O s，r The three opacity components correspond to the three RGB channels, and ④ Given the camera perspective corresponding to the underwater image to be rendered, that is, the camera's intrinsic parameter matrix and extrinsic parameter matrix, find the water body Gaussian set S w and scene Gaussian set S s The union S of w ∪S s , then for the set S w ∪S s All Gaussian primitives in are sorted according to their depth values ​​in a given camera coordinate system in order to perform tile-based α-blending rendering; ⑤ The color I of each channel of any pixel of the underwater image to be rendered c (x), expressed as: Where x represents the position coordinate of the pixel in the image coordinate system; c∈{r, g, b} corresponds to different color channels; Represents the color value of the color channel of the i-th Gaussian primitive in the rendering process. Here, the color channel of the i-th Gaussian primitive is the same as I c (x) corresponds to the color channel; represents the opacity contribution of the i-th Gaussian primitive to the corresponding color channel during the current pixel rendering process, and Indicates the opacity component of the i-th Gaussian primitive in the corresponding color channel during the rendering process; G 2D (x) represents the value of the 2D Gaussian distribution of the i-th Gaussian basis element at the current pixel in the current image coordinate system during the rendering process; ⑥ Render each color channel of any pixel of the underwater image to be rendered independently; ⑦ Traverse all pixels of the underwater image to be rendered and complete the rendering, outputting the final underwater image I; (2) Scene image rendering branch: ① Extract scene Gaussian set S s All Gaussian primitives in ; ② For the rth scene Gaussian, re-record its opacity attribute as scalar O s，r ; ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's intrinsic parameter matrix and extrinsic parameter matrix, the rasterization rendering pipeline in the original 3DGS technology is used to render the scene Gaussian set S s All Gaussian primitives in are processed and the final scene image I is output s ; (3) Pure water background image rendering branch: ① Extract the water body Gaussian set S w All Gaussian primitives in ; ②For the p-th water body Gaussian, its opacity attribute is re-expressed as Among them w,p The three opacity components are independent of each other; ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's internal parameter matrix and external parameter matrix, the water body Gaussian set S w All Gaussian primitives in are sorted according to their depth values ​​in a given camera coordinate system in order to perform tile-based α-blending rendering; ④ The color of each channel of any pixel of the pure water background image to be rendered Expressed as: Where x represents the position coordinate of the pixel in the image coordinate system; c∈{r,g,b} corresponds to different color channels; Represents the color value of the color channel of the i-th Gaussian primitive in the rendering process. Here, the color channel of the i-th Gaussian primitive is The color channel corresponds to represents the opacity contribution of the i-th Gaussian primitive to the corresponding color channel during the current pixel rendering process, and Indicates the opacity component of the i-th Gaussian primitive in the corresponding color channel during the rendering process; G 2D (x) represents the value of the 2D Gaussian distribution of the i-th Gaussian basis element at the current pixel in the current image coordinate system during the rendering process; ⑤ Render the color of each channel of any pixel of the pure water background image independently; ⑥ Traverse all pixels of the pure water background image to be rendered and complete the rendering, outputting the final pure water background image I w ; (4) Depth map rendering branch: ①Extract scene Gaussian set S s All Gaussian primitives in participate in rasterization and rendering; ② For the rth scene Gaussian, re-record its opacity attribute as scalar o s,r ; ③ Given the camera perspective corresponding to the image to be rendered, that is, the camera's intrinsic parameter matrix and extrinsic parameter matrix, the scene Gaussian set S s All Gaussian primitives in are sorted according to their depth values ​​in a given camera coordinate system in order to perform tile-based α-blending rendering; ④The depth value of any pixel of the depth map to be rendered is expressed as: Among them, x represents the position coordinate of the pixel in the image coordinate system, α i Indicates the opacity contribution of the i-th Gaussian primitive in the current pixel rendering process, and α i =o i G 2D (x); G 2D (x) represents the value of the 2D Gaussian distribution of the i-th Gaussian basis element at the current pixel in the current image coordinate system during the rendering process; d i Indicates the depth value of the i-th Gaussian primitive in the current camera coordinate system during rendering; ⑤ When all pixels of the image to be rendered are rendered, the desired depth map D is obtained.

8. The 3D characterization method for underwater scenes based on 3D Gaussian splashing according to claim 7, characterized in that: The total loss function L used in the iterative optimization of the attribute parameters in the water body Gaussian set and the scene Gaussian set is final Expressed as: In the first stage of optimization, L final =L content +λ4L depth ; In the second stage of optimization, L final =L content +λ4L depth +λ5L o-smooth +λ6L c-smooth ; Among them, L content represents the loss function based on image content; L depth represents the coarse-grained depth loss function; L o-smooth represents the local smoothing regularization term of the Gaussian opacity attribute of the water body; L c-smooth represents the local smoothing regularization term of the water body Gaussian color attribute; λ4 represents the coarse-grained depth loss function L depth The weight of λ5 represents the local smoothing regularization term L of the water body Gaussian opacity attribute o-smooth Weight; λ6 represents the local smoothing regularization term L of the water body Gaussian color attribute c-smooth The weight of .

9. The method for representing underwater scenes in 3D based on 3D Gaussian splashing according to claim 8, characterized in that: (1) Loss function L based on image content content The construction methods include: ①Use the pseudo depth map of the current iteration round To create a background mask M bg ; Given a predefined depth threshold D thre ; The pseudo depth value is greater than D thre The area composed of pixels is regarded as the foreground area, and the corresponding background mask M bg The pixel value at the corresponding position is 0; the pseudo depth value is less than or equal to D thre The area composed of pixels is regarded as the background area, and the corresponding background mask M bg The pixel value at the corresponding position is set to 1; ② The underwater image in the original underwater image set is recorded as the true value of the underwater image, and the symbol is Construct a weight map W with the same shape as the pseudo depth map content ; Specifically, the weight graph W content The value of the lth pixel refers to the following requirements: When the background mask M corresponding to the lth pixel bg,l When the value is 0, the weight graph W content The lth pixel corresponds to the foreground area. At this time, the weight map corresponding to the lth pixel is expressed as: W content,l =(sg(I l )+0.001) -1 Where sg(·) represents the gradient cutoff operator; I l It represents the color value corresponding to the lth pixel of the rendered underwater image; When the background mask M corresponding to the lth pixel bg,l When the value is 1, W content The lth pixel corresponds to the background area, and the weight map corresponding to the lth pixel is expressed as: W content，l ＝1 Then, we get the foreground enhanced underwater image I′ and the foreground enhanced underwater image ground truth Relationship: I′=W conntennt ☉I Where ⊙ represents the element-wise multiplication operator; ③ Extract background mask M bg All pixels with a value of 1 are obtained, and the background pixel set is recorded as Ω bg , all pixels belonging to this set correspond to areas in the picture where there is no underwater scene and only pure water; ④ Loss function L based on image content content Expressed as: Among them, λ1 represents the loss function term The weight of λ2 represents the loss function term The weight of λ3 represents the loss function term The weight of L D-SSIM Represents L in the original 3DGS technology D-SSIM Loss function; Represents pure water background image I w In the region Ω bg The first bg The color value at the pixel; Represents the true value of underwater image In the region Ω bg The first bg The color value at the pixel; (2) Coarse-grained depth loss function L depth The construction methods include: Convert the rendered depth map D into an approximate disparity map D′=(γD+1) -1 , where γ represents the scale parameter; The Pierce correlation coefficient with scale invariance is used as a similarity measure; Coarse-grained depth loss function L depth Expressed as: Where Pearson(·) represents the Pierce correlation coefficient calculation function; represents a pseudo depth map; (3) Local smoothing regularization term L for the water body Gaussian opacity attribute o-smmoth , expressed as: (4) Local smoothing regularization term L for the Gaussian color attribute of water c-smooth , expressed as: Among them, S w represents the set of water body Gaussians involved in rendering; knn(p, k) represents the set of k nearest neighbor water body Gaussians in the world coordinate system obtained by the p-th water body Gaussian through the KNN algorithm; η p，q represents the weight factor, and η p，q Expressed as λ w represents the scale factor; μ p represents the mean attribute of the p-th water body Gaussian in the world coordinate system; μ q Represents the mean attribute of the qth water body Gaussian in the world coordinate system.

Images