Underwater three-dimensional scene reconstruction method and device based on three-dimensional Gaussian splashing and underwater imaging model

Abstract

The invention discloses an underwater three-dimensional scene reconstruction method and device based on a three-dimensional Gaussian splashing and underwater imaging model. The method comprises the following steps: constructing an underwater truth value image, corresponding camera parameters and sparse initialization point cloud into a data set; training a first underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model by using the data set and the three-dimensional reconstruction loss function, and training a second underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model by using the data set, the three-dimensional reconstruction loss function and the self-trimming supervision loss function; and generating a water-containing underwater image under the new visual angle by using the trained three-dimensional Gaussian splash model in the first underwater radiation field and the underwater imaging model, and generating a water-free underwater image under the new visual angle by using the trained three-dimensional Gaussian splash model in the second underwater radiation field. The method has the advantages that the training speed is high, the image rendering and three-dimensional reconstruction quality is high, and the underwater image with water or without water can be selectively rendered.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional scene reconstruction, and in particular relates to a method and device for underwater three-dimensional scene reconstruction based on three-dimensional Gaussian splashing and underwater imaging model. Background Art

[0002] As a key research direction in the field of computer vision, 3D reconstruction plays a vital role in the construction of the metaverse and the development of scene simulators required for reinforcement learning. In addition, it also shows broad application potential in the fields of digital twins, virtual reality, autonomous driving, and robotics. These fields have an urgent need for high-precision and high-efficiency 3D scene reconstruction to achieve accurate simulation and interaction with the real world.

[0003] Although traditional 3D reconstruction methods have met these needs to a certain extent, with the advancement of technology, 3D reconstruction technology based on Neural Radiance Fields (NeRF) has shown great advantages over traditional methods in rendering high-quality new perspective images and depth estimation. NeRF represents the 3D scene as a radiation field through a multilayer perceptron (MLP), that is, each point in the scene is associated with color and density information related to the viewing angle, and uses differentiable volume rendering technology to render the continuous radiation field into an image. By calculating the reconstruction loss of the training image and using gradient backpropagation to optimize the radiation field represented by the MLP, NeRF can generate realistic 3D scene rendering effects.

[0004] However, as an implicit 3D scene representation and rendering method, NeRF has significant disadvantages in training and rendering efficiency compared to explicit 3D reconstruction methods. To overcome this challenge, a new research progress proposes the 3D Gaussian Splatting (3DGS) technology. 3DGS uses an explicit scene representation based on point clouds and a matching fast rasterizer to achieve fast training and real-time rendering capabilities. By replacing the MLP with an explicit point cloud distribution that can be optimized through training gradient backpropagation, where each point is represented as a 3D Gaussian ellipsoid with color and opacity, and the 3D Gaussian body is projected to the image plane for color calculation through the fast rasterization method in computer graphics, a large amount of sampling and inference in volume rendering technology is avoided, significantly improving training and rendering efficiency.

[0005] Although radiation field-based 3D scene representation and rendering techniques such as NeRF and 3DGS have made significant progress in many aspects, they all assume that the given multi-view training images are clear and ignore the absorption and scattering effects of the medium on light. This results in the inability of these techniques to perform accurate 3D reconstruction when processing images containing fog or underwater environments. To address this problem, the underwater neural radiation field method (SeaThru-NeRF) came into being, which combines the underwater imaging model with the MLP-based NeRF to achieve underwater radiation field reconstruction. However, SeaThru-NeRF requires a large number of sample queries for volume rendering imaging during the training and rendering stages, which seriously affects its reasoning efficiency, thereby limiting its wide feasibility in practical applications. Therefore, for complex underwater environments, it is necessary to further improve the accuracy of 3D reconstruction methods as well as the training and rendering efficiency. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a method and device for reconstructing an underwater three-dimensional scene based on a three-dimensional Gaussian splash model and an underwater imaging model, which combines the three-dimensional Gaussian splash model and the underwater imaging model, simulates the influence of the water medium on the propagation of light through physical simulation approximation, and uses a three-dimensional reconstruction loss function to train the first underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model and ultimately use it to generate high-quality underwater images with water. At the same time, a combined loss function including a three-dimensional reconstruction loss function and a self-pruning supervision loss function is used to train the second underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model and ultimately use it to generate high-quality underwater images without water, thereby improving the accuracy and efficiency of underwater three-dimensional scene reconstruction, thereby having the ability to selectively render underwater images with water or underwater images without water from a new perspective.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a method for reconstructing an underwater three-dimensional scene based on a three-dimensional Gaussian splash and underwater imaging model, comprising the following steps:

[0009] The collected and preprocessed underwater true value images and their corresponding camera parameters and sparse initialization point clouds are constructed as a data set;

[0010] A first underwater radiation field based on a three-dimensional Gaussian splash model and an underwater imaging model is trained using a data set and a three-dimensional reconstruction loss function, and a second underwater radiation field based on a three-dimensional Gaussian splash model and an underwater imaging model is trained using a data set and a combined loss function including a three-dimensional reconstruction loss function and a self-pruning supervision loss function. During the training of the first underwater radiation field and the second underwater radiation field, a depth map and a water-free underwater image are generated by using the three-dimensional Gaussian splash model, and then input into the underwater imaging model to finally generate a water-containing underwater image.

[0011] The trained three-dimensional Gaussian splash model and underwater imaging model in the first underwater radiation field are used to generate underwater images with water under a new perspective, and the trained three-dimensional Gaussian splash model in the second underwater radiation field is used to generate underwater images without water under a new perspective.

[0012] Preferably, the process of training the first underwater radiation field or the second underwater radiation field includes:

[0013] A training perspective is randomly selected from the dataset and the corresponding camera parameters and underwater true image are obtained. In the three-dimensional Gaussian splash model, each point in the sparse initialization point cloud is initialized as a three-dimensional Gaussian body and the three-dimensional Gaussian body is rendered through a fast rasterization process to obtain a depth map and a water-free underwater image. In the underwater imaging model, the obtained depth map and water-free underwater image are used to generate an underwater image with water. The parameters of the three-dimensional Gaussian splash model and the underwater imaging model in the first underwater radiation field are optimized through iterative training using the underwater true image in the dataset and the three-dimensional reconstruction loss function. The parameters of the three-dimensional Gaussian splash model and the underwater imaging model in the second underwater radiation field are optimized through the underwater true image in the dataset and the combined loss function including the three-dimensional reconstruction loss function and the self-pruning supervision loss function.

[0014] Preferably, the underwater imaging model is expressed as:

[0015]

[0016] in, represents the underwater image with water generated by the underwater imaging model, represents the water-free underwater image generated by the three-dimensional Gaussian splash model, represents the attenuation rate of directly reflected light, represents the attenuation rate of backscattered light, represents the depth map generated by the 3D Gaussian splash model, Represents the mask light value, and are all learnable parameters.

[0017] Preferably, the 3D reconstruction loss function includes an L1 loss function and a structural similarity loss function, which are expressed as:

[0018]

[0019] in, represents the 3D reconstruction loss function, and λ represents an adjustable hyperparameter;

[0020] Represents the L1 loss function:

[0021]

[0022] Where w and h represent the width and height of the image respectively. represents the underwater image with water generated by the underwater imaging model, I WW Representation and The corresponding underwater ground truth image;

[0023] Represents the structural similarity loss function:

[0024]

[0025] Among them, SSIM(·) represents the structural similarity calculation function.

[0026] Preferably, the combined loss function including the 3D reconstruction loss function and the self-pruning supervision loss function is expressed as:

[0027]

[0028] in, represents the combined loss function, λ1, λ2 and λ3 represent adjustable hyperparameters;

[0029] represents the self-pruning supervision loss function, which is achieved by maintaining the consistency of the water-free underwater images before and after self-pruning, and is expressed as:

[0030]

[0031] in, represents the water-free underwater image rendered from the 3D Gaussian splash model before pruning, Represents a water-free underwater image rendered from a pruned 3D Gaussian splash model.

[0032] Preferably, the self-pruning process includes:

[0033] For the three-dimensional Gaussian volume obtained by initializing each point in the sparse initialization point cloud in the three-dimensional Gaussian splash model Remove 3D Gaussian A three-dimensional Gaussian volume used to simulate water quality in Then based on the removal The three-dimensional Gaussian volume is then subjected to a subsequent fast rasterization process.

[0034] Preferably, by removing each three-dimensional Gaussian volume The scaling matrix S in i Exceeding the predetermined threshold S τ Three-dimensional Gaussian To remove the The superscript i represents the index of the three-dimensional Gaussian volume and the scaling matrix.

[0035] In a second aspect, an embodiment of the present invention further provides an underwater three-dimensional scene reconstruction device based on a three-dimensional Gaussian splash and underwater imaging model, which is implemented using the above-mentioned underwater three-dimensional scene reconstruction method based on a three-dimensional Gaussian splash and underwater imaging model, and includes: a data set construction module, a radiation field training module and a new image generation module;

[0036] The data set construction module is used to construct the collected and preprocessed underwater true value images and their corresponding camera parameters and sparse initialization point clouds into a data set;

[0037] The radiation field training module is used to train a first underwater radiation field based on a three-dimensional Gaussian splash model and an underwater imaging model using a data set and a three-dimensional reconstruction loss function, and to train a second underwater radiation field based on a three-dimensional Gaussian splash model and an underwater imaging model using a data set and a combined loss function including a three-dimensional reconstruction loss function and a self-pruning supervision loss function. During the training of the first underwater radiation field and the second underwater radiation field, a depth map and a waterless underwater image are generated by using the three-dimensional Gaussian splash model, and then input into the underwater imaging model to finally generate a water-containing underwater image.

[0038] The new image generation module is used to generate an underwater image with water under a new perspective using the trained three-dimensional Gaussian splash model and underwater imaging model in the first underwater radiation field, and to generate an underwater image without water under a new perspective using the trained three-dimensional Gaussian splash model in the second underwater radiation field.

[0039] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer, the above-mentioned underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model is implemented.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) The version of the non-self-pruning supervised loss function in the present invention, by combining the three-dimensional reconstruction loss function to train the first underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model, can effectively simulate the influence of water on the propagation of light through the underwater imaging model, thereby improving the training speed of the underwater radiation field and the three-dimensional reconstruction quality of the underwater scene, and using the trained three-dimensional Gaussian splash model and underwater imaging model to generate high-quality underwater images with water.

[0043] (2) The present invention has a self-pruning supervised loss function version. By combining the three-dimensional reconstruction loss function and the self-pruning supervised loss function to train the second underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model, it is possible to remove the suspended and cluttered three-dimensional Gaussian body used to simulate the water medium, and use the trained three-dimensional Gaussian splash model to render a high-quality water-free underwater image, which can achieve high-quality water-free underwater image three-dimensional reconstruction effect while ensuring training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a flow chart of a method for reconstructing an underwater three-dimensional scene based on a three-dimensional Gaussian splash and underwater imaging model provided by an embodiment of the present invention;

[0046] Figure 2 3D Gaussian splashing and underwater imaging model provided by an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of rendering in combination with an underwater imaging model and three-dimensional Gaussian splashing provided by an embodiment of the present invention;

[0048] Figure 4 This is a comparison chart of the rendering results of underwater images with water using the non-self-pruning supervised loss function version provided by an embodiment of the present invention and other methods.

[0049] Figure 5 It is a comparison diagram of the waterless seabed image rendering results of different ablation experiment versions provided by the embodiment of the present invention and the existing three-dimensional Gaussian splashing method;

[0050] Figure 6 It is a structural schematic diagram of an underwater three-dimensional scene reconstruction device based on three-dimensional Gaussian splashing and underwater imaging model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0052] The inventive concept of the present invention is: in view of the problem of insufficient accuracy and efficiency of the three-dimensional reconstruction method of underwater scenes in the prior art, the embodiment of the present invention provides a method and device for underwater three-dimensional scene reconstruction based on three-dimensional Gaussian splashing and underwater imaging models, and combines the three-dimensional reconstruction loss function to train the first underwater radiation field based on the three-dimensional Gaussian splashing model and the underwater imaging model, which can effectively simulate the influence of water on light propagation through the underwater imaging model, improve the training speed of the underwater radiation field and the three-dimensional reconstruction quality of the underwater scene, and obtain high-quality underwater images with water. At the same time, by combining the three-dimensional reconstruction loss function and the self-pruning supervision loss function to train the second underwater radiation field based on the three-dimensional Gaussian splashing model and the underwater imaging model, the suspended and cluttered three-dimensional Gaussian body used to simulate the water medium can be removed, and the high-quality three-dimensional reconstruction effect of the water-free underwater image can be achieved while ensuring the training efficiency.

[0053] Figure 1 is a flow chart of a method for reconstructing an underwater three-dimensional scene based on a three-dimensional Gaussian splash and underwater imaging model provided by an embodiment of the present invention, Figure 2 Schematic diagram of the framework of the underwater 3D scene reconstruction method based on 3D Gaussian splashing and underwater imaging model provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the embodiment provides a method for reconstructing an underwater three-dimensional scene based on a three-dimensional Gaussian splash and an underwater imaging model, comprising the following steps:

[0054] S1, constructs the collected and preprocessed underwater true value images and their corresponding camera parameters and sparse initialized point clouds into a dataset.

[0055] In the embodiment, a series of preprocessing is performed on the collected multi-view underwater images, specifically including:

[0056] S1.1, sampling an underwater image of a single scene captured by a digital camera to a uniform resolution, such as 900×1400 resolution.

[0057] S1.2, white balance the image and remove extreme noise pixels at a ratio of 0.5% for each color channel.

[0058] S1.3, use the Structure from Motion (SfM) algorithm to obtain the camera internal and external parameters (including camera pose and focal length) corresponding to each image, and generate a sparse initialization point cloud

[0059] S1.4, divide the dataset containing all data and images into training set and test set.

[0060] S2, using the data set and the three-dimensional reconstruction loss function to train the first underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model, and using the data set and the combined loss function including the three-dimensional reconstruction loss function and the self-pruning supervision loss function to train the second underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model. During the training process of the first underwater radiation field and the second underwater radiation field, the depth map and the water-free underwater image are generated by the three-dimensional Gaussian splash model and then input into the underwater imaging model to finally generate the underwater image with water.

[0061] In the embodiment, a point-based primitive, namely an anisotropic three-dimensional Gaussian volume, is used. The three-dimensional Gaussian volume can be efficiently rendered by a fast rasterization method, and gradient back propagation can be used to update and optimize the three-dimensional Gaussian volume distributed in space during training. Attributes.

[0062] S2.1, in each iterative training, a training perspective is randomly selected from the training set to obtain the corresponding camera parameters and underwater true value image. First, the sparse initial point cloud Each point in is initialized as a three-dimensional Gaussian Each point in the point cloud Contains color and position information, and accordingly, Each corresponding three-dimensional Gaussian The position and color attributes of and color c i Property Definition. Angle-dependent color properties can be modeled using spherical harmonics (SH) with no more than three degrees to capture the effect of viewing angle-dependent radiation. In addition, each three-dimensional Gaussian Also contains the opacity α i and the covariance matrix The density and ellipsoid shape are defined separately.

[0063] Among them, the covariance matrix of a single three-dimensional Gaussian By scaling matrix S i and the rotation matrix R i The calculation results in:

[0064]

[0065] Among them, through the scaling matrix S i Implementing 3D Gaussian The scaling is done by rotating the matrix R i Changing the 3D Gaussian The superscript T indicates the matrix transpose.

[0066] S2.2, projecting the three-dimensional Gaussian volume onto the two-dimensional image plane through a fast rasterization process. It is expressed as:

[0067]

[0068] in, represents the mean value of the three-dimensional Gaussian body in the camera coordinate system, represents the covariance matrix in the camera coordinate system, π(·) represents the projection operation, T CW ∈SE(3) represents the camera position, SE(3) represents the three-dimensional special Euclidean group, μ W Represents the mean of the three-dimensional Gaussian body in the world coordinate system, J represents the Jacobian matrix of the affine approximation of the projection transformation, W represents the rotation matrix from the world coordinate system to the camera coordinate system, and the superscript T represents the matrix transpose.

[0069] S2.3, projecting the three-dimensional Gaussian volume onto the two-dimensional image plane Sort by depth and blend weighted by opacity to get the color of the corresponding pixel The calculation formula is as follows:

[0070]

[0071] Where n is the index in the total number N of three-dimensional Gaussian volumes overlapping a given pixel, the subscript j is the index belonging to the real number set [1, n-1], and T n What is calculated is the total opacity caused by the product of the opacity α of the first n-1 three-dimensional Gaussian bodies to the nth three-dimensional Gaussian body.

[0072] S2.4, rendering a 3D Gaussian volume to obtain depth values The depth value of the rendering is calculated by the following formula:

[0073]

[0074] The color c of the formula rendered in step S2.4 n Replaced by the depth z after three-dimensional Gaussian projection transformation n .

[0075] S2.5, according to the depth value obtained Get the corresponding depth map, according to the color Get the corresponding water-free underwater image.

[0076] In the embodiment, the fast rasterization method follows the method in 3D Gaussian splatting (3DGS). To improve rendering efficiency, the following processing is performed on a single thread in units of tiles: rasterization, depth sorting, and 3D Gaussian volume blending rendering according to opacity. The method includes:

[0077] (1) Rasterization and sorting. The screen is divided into 16×16 tiles, and the 3D Gaussian points in the viewing frustum of each tile are culled. Only the 3D Gaussian bodies that intersect the viewing frustum with 99% confidence are retained, and the 3D Gaussian bodies that are too close to the near and far planes of the viewing frustum are culled. The ID and depth of each tile are recorded for each 3D Gaussian body, and then the 3D Gaussian bodies of each tile are sorted by depth and rendered according to this sorting.

[0078] (2) 3D Gaussian blending rendering with tiles as threads. Start a thread for each tile, traverse the 3D Gaussian volumes sorted by depth in the tile from front to back, and blend the colors c of the 3D Gaussian volumes according to the opacity α to render. Preferably, when the opacity of the tile reaches saturation (α = 1 or T n =0), terminate the rendering of the block to improve rendering efficiency.

[0079] The above rendering method does not consider the influence of the possible medium on light propagation. Therefore, it is necessary to further introduce an underwater imaging model to simulate the influence of the water medium on light propagation. The existing underwater imaging model (source document Derya Akkaynak and Tali Treibitz. 2018. A revised underwater image formation model. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6723–6732) calculates the formula for the attenuated underwater image as follows:

[0080]

[0081] Among them, x is a point under water, J(x) is the water-free image after water removal, is the mask light value, t D (x) is the transmittance of the directly reflected light, t B (x) is the transmittance of backscattered light.

[0082] According to the Lambert-Beer empirical law, light decays exponentially in a medium, as shown below:

[0083]

[0084] Where d(x) is the distance between the object and the camera, β D is the attenuation rate of directly reflected light, β B is the attenuation rate of backscattered light, V D and V B Represents β D and β B Dependencies.

[0085] S2.6, such as Figure 3 As shown, in the embodiment, the calculation formula of the final underwater imaging model combined with the three-dimensional Gaussian splash model is as follows:

[0086]

[0087] in, It represents the underwater image with water finally rendered by the underwater imaging model. Represents the pixel color obtained by rendering the three-dimensional Gaussian splash model Composition of waterless underwater images, represents the attenuation rate of directly reflected light, represents the attenuation rate of backscattered light, Represents the depth value obtained by rendering the 3D Gaussian splash model The depth map composed of Represents the mask light value, and are all learnable parameters. The decay rate coefficient in formula (6) and Can be learned and optimized through gradient back propagation, with a decay rate of β D and β B Dependencies of V D and V B can therefore be simplified and not taken into account.

[0088] S2.7, iterative training is performed through the underwater true value images in the data set and the three-dimensional reconstruction loss function to optimize the parameters of the three-dimensional Gaussian splash model and the underwater imaging model in the first underwater radiation field.

[0089] In the embodiment, the 3D reconstruction loss function includes an L1 loss function and a structural similarity loss function, which are expressed as:

[0090]

[0091] in, represents the 3D reconstruction loss function, and λ represents an adjustable hyperparameter.

[0092] Represents the L1 loss function:

[0093]

[0094] Where w and h represent the width and height of the image respectively. represents the underwater image with water generated by the underwater imaging model, I WW Representation and The corresponding underwater ground truth image.

[0095] Represents the structural similarity loss function:

[0096]

[0097] Among them, SSIM(·) represents the structural similarity calculation function, which is used to measure the structural similarity between two images.

[0098] S2.8, optimizes the parameters of the three-dimensional Gaussian splash model and the underwater imaging model in the second underwater radiation field through the underwater true value image in the dataset and the combined loss function including the three-dimensional reconstruction loss function and the self-pruning supervision loss function.

[0099] In the embodiment, although the underwater imaging model used has taken into account the influence of water medium on light propagation, the learned three-dimensional Gaussian volume still contains some suspended and messy three-dimensional Gaussian volumes that simulate the water medium. In the embodiment, the self-pruning supervision loss function is introduced It is used to remove the suspended and cluttered 3D Gaussian bodies, thereby improving the quality of 3D reconstruction of underwater scenes. For the 3D reconstruction of the seabed, the reconstructed 3D Gaussian body It can be divided into two parts: a three-dimensional Gaussian volume simulating water medium and a 3D Gaussian volume simulating the seabed Since the underwater imaging model has taken into account the influence of water medium on light propagation, the embodiment introduces a self-pruning supervision loss function to remove the suspended cluttered three-dimensional Gaussian volume simulating the water medium. like Figure 4It can be observed from the reconstruction result of the existing 3D Gaussian splashing method shown in (a) that the 3D Gaussian volume simulating the water medium has a large scaling, which affects the rendering effect of the 3D Gaussian splashing. The scaling matrix S in i Exceeding the predetermined threshold S τ Three-dimensional Gaussian To achieve the purpose of self-pruning. The three-dimensional Gaussian volume after self-pruning is expressed as Rendering via fast rasterization Get relative Sharper underwater images after cropping without water

[0100] The combined loss function is expressed as:

[0101]

[0102] in, represents the combined loss function, and λ1, λ2, and λ3 represent adjustable hyperparameters.

[0103] represents the self-pruning supervision loss function, which is achieved by maintaining the consistency of the water-free underwater images before and after self-pruning, and is expressed as:

[0104]

[0105] in, represents the water-free underwater image rendered from the 3D Gaussian splash model before pruning, Represents a clearer water-free underwater image rendered from a trimmed 3D Gaussian splash model.

[0106] In an embodiment, the self-pruning supervision loss function can be enabled at different stages of the second underwater radiation field training. A three-dimensional Gaussian volume that simulates water media can be removed more thoroughly However, some details in the scene may be lost; on the contrary, the later activation Will reduce The removal effect is not obvious, but more scene details can be retained.

[0107] S2.9, start training. In this example, a configuration based on three-dimensional Gaussian splashing (3DGS) is used, that is, a training code based on PyTorch and CUDA is developed, a total of 40,000 training iterations are set, and a predetermined threshold S is set. τ Set to five times the median of the scaling matrix S of all current three-dimensional Gaussian bodies: S τ =5×MEDIAN({S i}), the order of the spherical harmonics is set to third order. Properties of the three-dimensional Gaussian, including opacity α i , scaling matrix S i and the rotation matrix R i , and their default learning rates are set to 0.05, 0.005, and 0.001, respectively. The segmentation and encryption operation of the three-dimensional Gaussian volume starts from the 500th iteration and continues until the end of training. Opacity α i The training is reset every 3000 iterations to remove redundant 3D Gaussian volumes. The training optimizer uses the Adam optimizer. In this example, the training is performed using a single NVIDIA Tesla V100 (32GB) GPU. After the training is completed, the test set is used for verification.

[0108] S3, using the trained three-dimensional Gaussian splash model and underwater imaging model in the first underwater radiation field to generate an underwater image with water under a new perspective, and using the trained three-dimensional Gaussian splash model in the second underwater radiation field to generate an underwater image without water under a new perspective.

[0109] S3.1, rendering high-quality underwater images with water. A virtual camera is established at a new perspective and the corresponding camera parameters are obtained. Based on the trained three-dimensional Gaussian splash model in the first underwater radiation field, the trained three-dimensional Gaussian volume is directly rendered to obtain a depth map and a waterless seabed image, and further rendered in combination with the underwater imaging model to obtain an underwater image with water at a new perspective.

[0110] S3.2, rendering high-quality water-free underwater images. Establish a virtual camera at a new perspective and obtain the corresponding camera parameters. Based on the trained three-dimensional Gaussian splash model in the second underwater radiation field, directly render the trained self-pruned three-dimensional Gaussian body to obtain a clearer water-free underwater image at a new perspective. It should be noted that for the trained second underwater radiation field, the three-dimensional Gaussian splash model and underwater imaging model can also be used to finally generate underwater images with water, but some scene details may be lost due to the introduction of the self-pruning supervision loss function to remove the three-dimensional Gaussian body simulating the water medium.

[0111] Experimental evaluation 1: Figure 4 As shown, Figure 4 (a), (b), (c), (d) and (e) represent the rendering results of the seabed image with water in the test set, respectively, including the true value image of the seabed, tensor decomposed radiation field (TensoRF), underwater neural radiation field (SeaThru-NeRF), three-dimensional Gaussian splashing (3DGS) and the version without self-pruning supervision loss function (trained first underwater radiation field). Figure 4The first to third rows show the visualization results from different scenes respectively. The results show that the visualization results of the version without self-pruning supervised loss function of the present invention have higher underwater image rendering quality than other methods.

[0112] Experimental evaluation 2: Figure 5 As shown, Figure 5 (a), (b) and (c) in the figure respectively show the comparison results of the waterless seabed image rendering of the existing three-dimensional Gaussian splashing (3DGS) method, the version of the present invention without self-pruning supervision loss function (trained first underwater radiation field), and the version of the present invention with self-pruning supervision loss function (trained second underwater radiation field). Figure 5 The first row shows the waterless seabed image rendered by the fast rasterization method using the unrendered 3D Gaussian volume shown in the second row. The results show that by introducing the self-pruning supervised loss function for training, the suspended cluttered 3D Gaussian volume simulating the water medium can be more effectively removed, thereby obtaining higher quality waterless seabed images.

[0113] In summary, the underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splash and underwater imaging model provided by the embodiment of the present invention can generate high-quality underwater images with water by using the trained three-dimensional Gaussian splash model and underwater imaging model based on the non-self-pruning supervised loss function version (the trained first underwater radiation field), and achieves the shortest time consumption of only 28 minutes in the training of a single scene, which is significantly shorter than the baseline method three-dimensional Gaussian splash (3DGS). At the same time, the training efficiency is about 100 times that of the similar underwater three-dimensional reconstruction method SeaThru-NeRF. Based on the self-pruning supervised loss function version (the trained second underwater radiation field), the trained three-dimensional Gaussian splash model can be used to render a higher quality underwater image than the existing method, showing excellent rendering performance and image quality performance. The two methods not only have a fast training speed, but also have high image rendering and three-dimensional reconstruction quality, and support selective rendering of underwater scene images with or without water.

[0114] Based on the same inventive concept, Figure 6 As shown, an embodiment of the present invention further provides an underwater three-dimensional scene reconstruction device 600 based on three-dimensional Gaussian splashing and underwater imaging model, including: a data set construction module 610, a radiation field training module 620 and a new image generation module 630.

[0115] The data set construction module 610 is used to construct the collected and pre-processed underwater true value images and their corresponding camera parameters and sparse initialization point clouds into a data set.

[0116] The radiation field training module 620 is used to train the first underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model using the data set and the three-dimensional reconstruction loss function, and to train the second underwater radiation field based on the three-dimensional Gaussian splash model and the underwater imaging model using the data set and the combined loss function including the three-dimensional reconstruction loss function and the self-pruning supervision loss function. During the training process of the first underwater radiation field and the second underwater radiation field, the depth map and the water-free underwater image are generated by the three-dimensional Gaussian splash model and then input into the underwater imaging model to finally generate the underwater image with water.

[0117] The new image generation module 630 is used to generate an underwater image with water at a new perspective using the trained three-dimensional Gaussian splash model and underwater imaging model in the first underwater radiation field, and to generate an underwater image without water at a new perspective using the trained three-dimensional Gaussian splash model in the second underwater radiation field.

[0118] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory and a processor, the memory is used to store a computer program, and the processor is used to implement the above-mentioned underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model when executing the computer program.

[0119] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model is implemented.

[0120] It should be noted that the underwater three-dimensional scene reconstruction device, electronic device and computer-readable storage medium based on three-dimensional Gaussian splashing and underwater imaging model provided in the above embodiments all belong to the same inventive concept as the underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model. The specific implementation process is detailed in the embodiment of the underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model, which will not be repeated here.

[0121] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for underwater three-dimensional scene reconstruction based on three-dimensional Gaussian splashing and underwater imaging model, characterized in that: The following steps are involved: The collected and preprocessed underwater true value images and their corresponding camera parameters and sparse initialization point clouds are constructed as a data set; A first underwater radiation field based on a three-dimensional Gaussian splash model and an underwater imaging model is trained using a data set and a three-dimensional reconstruction loss function, and a second underwater radiation field based on a three-dimensional Gaussian splash model and an underwater imaging model is trained using a data set and a combined loss function including a three-dimensional reconstruction loss function and a self-pruning supervision loss function. During the training of the first underwater radiation field and the second underwater radiation field, a depth map and a water-free underwater image are generated by using the three-dimensional Gaussian splash model, and then input into the underwater imaging model to finally generate a water-containing underwater image. The trained three-dimensional Gaussian splash model and underwater imaging model in the first underwater radiation field are used to generate underwater images with water under a new perspective, and the trained three-dimensional Gaussian splash model in the second underwater radiation field is used to generate underwater images without water under a new perspective.

2. The underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model according to claim 1 is characterized in that: The process of training the first underwater radiation field or the second underwater radiation field includes: A training perspective is randomly selected from the dataset and the corresponding camera parameters and underwater true image are obtained. In the three-dimensional Gaussian splash model, each point in the sparse initialization point cloud is initialized as a three-dimensional Gaussian body and the three-dimensional Gaussian body is rendered through a fast rasterization process to obtain a depth map and a water-free underwater image. In the underwater imaging model, the obtained depth map and water-free underwater image are used to generate an underwater image with water. The parameters of the three-dimensional Gaussian splash model and the underwater imaging model in the first underwater radiation field are optimized through iterative training using the underwater true image in the dataset and the three-dimensional reconstruction loss function. The parameters of the three-dimensional Gaussian splash model and the underwater imaging model in the second underwater radiation field are optimized through the underwater true image in the dataset and the combined loss function including the three-dimensional reconstruction loss function and the self-pruning supervision loss function.

3. The underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model according to claim 2 is characterized in that: The underwater imaging model is expressed as: in, represents the underwater image with water generated by the underwater imaging model, represents the water-free underwater image generated by the three-dimensional Gaussian splash model, represents the attenuation rate of directly reflected light, represents the attenuation rate of backscattered light, represents the depth map generated by the 3D Gaussian splash model, Represents the mask light value, and are all learnable parameters.

4. The underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model according to claim 2 is characterized in that: The 3D reconstruction loss function includes the L1 loss function and the structural similarity loss function, which is expressed as: in, represents the 3D reconstruction loss function, and λ represents an adjustable hyperparameter; Represents the L1 loss function: Where w and h represent the width and height of the image respectively. represents the underwater image with water generated by the underwater imaging model, I WW Representation and The corresponding underwater ground truth image; Represents the structural similarity loss function: Among them, SSIM(·) represents the structural similarity calculation function.

5. The underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model according to claim 4 is characterized in that: The combined loss function including the 3D reconstruction loss function and the self-pruning supervision loss function is expressed as: in, represents the combined loss function, λ1, λ2 and λ3 represent adjustable hyperparameters; represents the self-pruning supervision loss function, which is achieved by maintaining the consistency of the water-free underwater images before and after self-pruning, and is expressed as: in, represents the water-free underwater image rendered from the 3D Gaussian splash model before pruning, Represents a water-free underwater image rendered from a pruned 3D Gaussian splash model.

6. The underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model according to claim 5 is characterized in that: The self-pruning process includes: For the three-dimensional Gaussian volume obtained by initializing each point in the sparse initialization point cloud in the three-dimensional Gaussian splash model Remove 3D Gaussian A three-dimensional Gaussian volume used to simulate water quality in Then based on the removal The three-dimensional Gaussian volume is then subjected to a subsequent fast rasterization process.

7. The underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model according to claim 6 is characterized in that: By removing each three-dimensional Gaussian The scaling matrix S in i Exceeding the predetermined threshold S τ Three-dimensional Gaussian To remove the The superscript i represents the index of the three-dimensional Gaussian volume and the scaling matrix.

8. An underwater three-dimensional scene reconstruction device based on three-dimensional Gaussian splashing and underwater imaging model, which is implemented by the underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model according to any one of claims 1 to 7, characterized in that: include: Dataset construction module, radiation field training module and new image generation module; The data set construction module is used to construct the collected and preprocessed underwater true value images and their corresponding camera parameters and sparse initialization point clouds into a data set; The radiation field training module is used to train a first underwater radiation field based on a three-dimensional Gaussian splash model and an underwater imaging model using a data set and a three-dimensional reconstruction loss function, and to train a second underwater radiation field based on a three-dimensional Gaussian splash model and an underwater imaging model using a data set and a combined loss function including a three-dimensional reconstruction loss function and a self-pruning supervision loss function. During the training of the first underwater radiation field and the second underwater radiation field, a depth map and a waterless underwater image are generated by using the three-dimensional Gaussian splash model, and then input into the underwater imaging model to finally generate a water-containing underwater image. The new image generation module is used to generate an underwater image with water under a new perspective using the trained three-dimensional Gaussian splash model and underwater imaging model in the first underwater radiation field, and to generate an underwater image without water under a new perspective using the trained three-dimensional Gaussian splash model in the second underwater radiation field.

9. An electronic device comprising a memory and a processor, wherein the memory is used to store a computer program, wherein: The processor is used to implement the underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model as described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the underwater three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing and underwater imaging model described in any one of claims 1 to 7 is implemented.

Images