Underwater scene nerve implicit three-dimensional reconstruction method

Abstract

The invention belongs to the technical field of computer vision and computer graphics, and particularly relates to an underwater scene nerve implicit three-dimensional reconstruction method, which comprises the following steps of: shooting a group of optical images of an underwater scene from a plurality of visual angles by using a high-resolution underwater camera; calculating camera internal and external parameters corresponding to the corrected optical image; constructing a neural implicit three-dimensional reconstruction model of the underwater scene; taking the corrected image data and the corresponding internal and external parameters of the camera as training data, and constructing an underwater body rendering equation; training a neural implicit three-dimensional reconstruction model of the underwater scene; and performing three-dimensional reconstruction on the new underwater optical image data by using the trained underwater neural implicit model. According to the method, a special network model and a loss function are designed for the particularity of an underwater environment, geometric and structural information of a three-dimensional scene can be more accurately reconstructed from an underwater image or video, and the reconstruction precision is obviously higher than that of an existing general neural implicit three-dimensional reconstruction method.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and computer graphics, and in particular relates to a neural implicit three-dimensional reconstruction method for underwater scenes. Background Art

[0002] Three-dimensional reconstruction technology is an important research direction in the field of computer vision and computer graphics. By recovering the geometric and structural information of three-dimensional scenes from images or videos, it can provide support for many applications such as marine exploration, underwater archaeology, autonomous driving, and underwater robot navigation. Traditional three-dimensional reconstruction methods usually rely on clear geometric models and require fine parameter adjustment. In recent years, with the rise of deep learning technology, implicit three-dimensional reconstruction methods based on neural networks have gradually attracted attention. This method learns three-dimensional representations directly from images or videos in a data-driven manner without setting explicit geometric constraints, and has stronger versatility and robustness. However, existing neural implicit three-dimensional reconstruction methods are mainly aimed at simple or artificially synthesized scenes, and the reconstruction effect of complex underwater scenes is still limited. Underwater scenes have special characteristics such as light attenuation, blurred vision, and color distortion, which bring additional challenges to three-dimensional reconstruction. In order to solve the above problems, a new underwater three-dimensional reconstruction method that can combine color correction and neural implicit surface modeling is urgently needed to achieve high-precision and high-realism underwater surface reconstruction. Summary of the invention

[0003] In response to the technical problem of insufficient accuracy of underwater three-dimensional reconstruction caused by light refraction, scattering and color distortion in the above-mentioned prior art, the present invention provides a neural implicit three-dimensional reconstruction method for underwater scenes, which provides a reliable data basis for downstream underwater autonomous navigation, underwater target detection and seabed infrastructure maintenance.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A neural implicit 3D reconstruction method for underwater scenes comprises the following steps:

[0006] S1. Use a high-resolution underwater camera to capture a set of optical images of underwater scenes from multiple perspectives, ensuring extensive coverage of scene information at different angles and distances;

[0007] S2, removing extreme noise in each optical image data based on a white balance correction method, and calculating the camera internal and external parameters corresponding to the corrected optical image;

[0008] S3, constructing a neural implicit 3D reconstruction model of underwater scenes, including a parameter field for describing underwater scattering media and an SDF field and a color field for describing scene geometry information, wherein the 3D shape of the scene is implicitly represented by the SDF field;

[0009] S4, using the corrected image data and the corresponding internal and external parameters of the camera as training data to construct an underwater volume rendering equation;

[0010] S5. Based on back propagation, the SDF field, color field and parameter field of underwater scattering medium are jointly optimized through multiple constraints to train the neural implicit 3D reconstruction model of underwater scenes.

[0011] S6. Use the trained underwater neural implicit model to perform three-dimensional reconstruction of new underwater optical image data.

[0012] The method for calculating the camera internal and external parameters corresponding to the corrected optical image in S2 is: performing white balance correction processing on the collected underwater optical image, and performing 0.5% clipping on each color channel to remove extreme noise; performing feature point extraction, matching and three-dimensional point cloud construction on the corrected optical image, and calculating the camera internal and external parameters corresponding to each image, wherein the camera internal parameters include focal length and distortion coefficient, and the camera external parameters include position and posture.

[0013] The method for constructing the neural implicit 3D reconstruction model of the underwater scene in S3 is:

[0014] S31. Construct SDF field: The spatial position x∈R of any sampling point in three-dimensional space 3 Mapped to its corresponding SDF value sdf∈R 1 , intermediate features f∈R 256 , γ is the position code, is a multi-layer fully connected neural network. is the learnable parameter of the network;

[0015] S32, construct object density σ obj : where Φ s (x) = (1 + e -sx ) -1 is the sigmoid function, is the gradient information of the SDF field, and s is the learnable standard deviation;

[0016] S33. Constructing color field The spatial position x of any sampling point in three-dimensional space and the observation direction d∈R from the camera to the point 3 Mapped to its corresponding color value c obj ∈R 3 , F ω is a multi-layer fully connected neural network, ω is a learnable parameter of the network;

[0017] S34. Construct parameter field: Assuming that the scattering medium parameters are only related to the viewing angle and are uniformly distributed along the viewing direction of the camera, then σbs , σ attn ,c med =F θ (γ(d)), where a bs is the backscattering coefficient, σ attn is the attenuation coefficient, c med is the medium color, F θ is a multi-layer fully connected neural network, and θ is the learnable parameter of the network.

[0018] The method for constructing the underwater volume rendering equation in S4 is:

[0019] S41, ray parameterization: For a ray r(t)=o+td emitted from a camera, determine its starting point (camera position) o, ray direction d and sampling interval t;

[0020] S42, calculate sampling points: along the direction d of the light, sample the light uniformly to obtain N sampling points r(t i ), these sampling points will be used as input to the neural network later;

[0021] S43, obtaining color and density: each sampling point r(t i ) The color and density of the corresponding medium and scene;

[0022] S44, constructing scene rendering items;

[0023] S45, constructing a media rendering item;

[0024] S46. Build the total rendering item.

[0025] The method for constructing the scene rendering item in S44 is:

[0026]

[0027] in, represents the scene color corresponding to the i-th sampling interval, δ i =t i+1 -t i is the sampling interval between adjacent sampling points, Represents the i-th sampling point r(t i ) corresponds to the scene density, attenuation coefficient, and scene color, is defined as

[0028]

[0029] Represents the accumulated transparency of the scene before the ray r reaches the i-th sampling point.

[0030] The method for constructing the media rendering item in S45 is:

[0031]

[0032] in, represents the medium color corresponding to the i-th sampling interval, δ i =t i+1 -t i is the sampling interval between adjacent sampling points, Represents the i-th sampling point r(t i ) corresponds to the scene density, backscattering coefficient, and medium color, is defined as

[0033]

[0034] Represents the accumulated transparency of the medium before the ray r reaches the i-th sampling point.

[0035] The method for constructing the total rendering item in S46 is:

[0036]

[0037] in, Represents the rendering color obtained by simulating the real imaging process along ray r.

[0038] The method for training the neural implicit 3D reconstruction model of the underwater scene in S5 is:

[0039] S51. Construct SDF regularized loss function L eikonal :

[0040]

[0041] in, represents the gradient of the SDF field, and x is any point in three-dimensional space;

[0042] S52. Construct the square error photometric loss function L color :

[0043]

[0044] in, To render the color, C gt For the true color;

[0045] S53. Construct the total loss function L:

[0046] L=L color +λ·L eikonal

[0047] Here, λ is a hyperparameter and is set to 0.1.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. Aiming at the particularity of underwater environment, the present invention designs a special network model and loss function, which can more accurately reconstruct the geometric and structural information of the three-dimensional scene from underwater images or videos, and the reconstruction accuracy is significantly higher than the existing general neural implicit three-dimensional reconstruction method. It provides a reliable data foundation for downstream underwater autonomous navigation, underwater target detection and submarine infrastructure maintenance.

[0050] 2. Unlike supervised learning methods that rely on a large amount of finely labeled data, the present invention only requires unlabeled underwater images or video data to efficiently complete model training, reducing the cost and difficulty of obtaining high-quality labeled data.

[0051] 3. In addition to underwater scenes, the neural implicit 3D reconstruction framework proposed in this invention can be extended to other complex and special environments, such as smoke, dust, etc., and has good versatility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0053] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0054] Figure 1 is a flow chart of the steps of the present invention;

[0055] Figure 2 It is a schematic diagram of the network framework of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0057] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0058] See also Figure 1 The embodiment of the present invention provides a neural implicit 3D reconstruction method for underwater scenes, comprising the following steps:

[0059] S1. Use a high-resolution underwater camera to capture a set of optical images of underwater scenes from multiple perspectives, ensuring extensive coverage of scene information at different angles and distances;

[0060] S2. Remove extreme noise from each optical image data based on the white balance correction method, and calculate the camera internal and external parameters corresponding to the corrected optical image. The specific form is: perform white balance correction on the collected underwater optical images, and perform 0.5% cropping on each color channel to remove extreme noise; perform feature point extraction, matching and three-dimensional point cloud construction on the corrected optical images, and calculate the camera internal parameters (focal length, distortion coefficient) and external parameters (position, posture) corresponding to each image.

[0061] S3. Construct a neural implicit 3D reconstruction model of underwater scenes, including a parameter field for describing underwater scattering media and an SDF field and a color field for describing scene geometry information. The 3D shape of the scene is implicitly represented by the SDF field, such as Figure 2 The specific steps include:

[0062] S31. Construct SDF field: The spatial position x∈R of any sampling point in three-dimensional space 3 Mapped to its corresponding SDF value sdf∈R 1 , intermediate features f∈R 256 , γ is the position code, is a multi-layer fully connected neural network. is the learnable parameter of the network;

[0063] S32, construct object density σ obj : where Φ s (x) = (1 + e-sx ) -1 is the sigmoid function, is the gradient information of the SDF field, and s is the learnable standard deviation;

[0064] S33. Construct color field: The spatial position x of any sampling point in three-dimensional space and the observation direction d∈R from the camera to the point 3 Mapped to its corresponding color value c obj ∈R 3 , F ω is a multi-layer fully connected neural network, ω is a learnable parameter of the network;

[0065] S34. Construct parameter field: Assuming that the scattering medium parameters are only related to the viewing angle and are uniformly distributed along the viewing direction of the camera, then σ bs , σ attn , c med =F θ (γ(d)), where σ bs is the backscattering coefficient, σ attn is the attenuation coefficient, c med is the medium color, F θ is a multi-layer fully connected neural network, and θ is the learnable parameter of the network.

[0066] S4, using the corrected image data and the corresponding internal and external parameters of the camera as training data to construct an underwater volume rendering equation. The specific steps include:

[0067] S41, ray parameterization: for a ray r(t)=o+td emitted from the camera, determine its starting point (camera position) o, ray direction d and sampling interval t;

[0068] S42, calculate sampling points: along the direction d of the light, sample the light uniformly to obtain N sampling points r(t i ), these sampling points will be used as input to the neural network later;

[0069] S43, obtaining color and density: each sampling point r(t i ) The color and density of the corresponding medium and scene;

[0070] S44. Build scene rendering items:

[0071]

[0072] in, represents the scene color corresponding to the i-th sampling interval, δ i =t i+1 -ti is the sampling interval between adjacent sampling points, Represents the i-th sampling point r(t i ) corresponds to the scene density, attenuation coefficient, and scene color, is defined as

[0073]

[0074] Represents the cumulative transparency of the scene before the light r reaches the i-th sampling point;

[0075] S45. Build media rendering items:

[0076]

[0077] in, represents the medium color corresponding to the i-th sampling interval, δ i =t i+1 -t i is the sampling interval between adjacent sampling points, Represents the i-th sampling point r(t i ) corresponds to the scene density, backscattering coefficient, and medium color, is defined as

[0078]

[0079] Represents the cumulative transparency of the medium before the light r reaches the i-th sampling point;

[0080] S46. Build total rendering items:

[0081]

[0082] in, Represents the rendering color obtained by simulating the real imaging process along ray r.

[0083] S5. Using the gradient descent optimization algorithm, the specific steps of calculating the loss by combining the regularization loss related to SDF and the square error photometric loss function as the total loss function are as follows:

[0084] S51. Construct SDF regularized loss function L eikonal :

[0085]

[0086] in, represents the gradient of the SDF field, and x is any point in three-dimensional space;

[0087] S52. Construct the square error photometric loss function L color :

[0088]

[0089] in, To render the color, C gt For the true color;

[0090] S53. Construct the total loss function:

[0091] L=L color +λ·L eikonal

[0092] Here, λ is a hyperparameter and is set to 0.1.

[0093] S6. Use the trained underwater neural implicit model to perform three-dimensional reconstruction of new underwater optical image data.

[0094] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. A neural implicit 3D reconstruction method for underwater scenes, characterized in that: The following steps are involved: S1. Use a high-resolution underwater camera to capture a set of optical images of underwater scenes from multiple perspectives, ensuring extensive coverage of scene information at different angles and distances; S2, removing extreme noise in each optical image data based on a white balance correction method, and calculating the camera internal and external parameters corresponding to the corrected optical image; S3, constructing a neural implicit 3D reconstruction model of underwater scenes, including a parameter field for describing underwater scattering media and an SDF field and a color field for describing scene geometry information, wherein the 3D shape of the scene is implicitly represented by the SDF field; S4, using the corrected image data and the corresponding internal and external parameters of the camera as training data to construct an underwater volume rendering equation; S5. Based on back propagation, the SDF field, color field and parameter field of underwater scattering medium are jointly optimized through multiple constraints to train the neural implicit 3D reconstruction model of underwater scenes. S6. Use the trained underwater neural implicit model to perform three-dimensional reconstruction of new underwater optical image data.

2. The method for neural implicit 3D reconstruction of underwater scenes according to claim 1, characterized in that: The method for calculating the camera internal and external parameters corresponding to the corrected optical image in S2 is: performing white balance correction processing on the collected underwater optical image, and performing 0.5% clipping on each color channel to remove extreme noise; performing feature point extraction, matching and three-dimensional point cloud construction on the corrected optical image, and calculating the camera internal and external parameters corresponding to each image, wherein the camera internal parameters include focal length and distortion coefficient, and the camera external parameters include position and posture.

3. The method for neural implicit 3D reconstruction of underwater scenes according to claim 1, characterized in that: The method for constructing the neural implicit 3D reconstruction model of the underwater scene in S3 is: S31. Construct SDF field: The spatial position x∈R of any sampling point in three-dimensional space 3 Mapped to its corresponding SDF value sdf∈R 1 , intermediate features f∈R 256 , γ is the position code, is a multi-layer fully connected neural network. is the learnable parameter of the network; S32, construct object density σ obj : where Φ s (x) = (1 + e -sx ) -1 is the sigmoid function, is the gradient information of the SDF field, and s is the learnable standard deviation; S33. Construct color field: The spatial position x of any sampling point in three-dimensional space and the observation direction d∈R from the camera to the point 3 Mapped to its corresponding color value c obj ∈R 3 , F ω is a multi-layer fully connected neural network, ω is a learnable parameter of the network; S34. Construct parameter field: Assuming that the scattering medium parameters are only related to the viewing angle and are uniformly distributed along the viewing direction of the camera, then σ bs , σ attn , c med =F θ (γ(d)), where σ bs is the backscattering coefficient, σ attn is the attenuation coefficient, c med is the medium color, F θ is a multi-layer fully connected neural network, and θ is the learnable parameter of the network.

4. The method for neural implicit 3D reconstruction of underwater scenes according to claim 1, characterized in that: The method for constructing the underwater volume rendering equation in S4 is: S41, ray parameterization: For a ray r(t)=o+td emitted from a camera, determine its starting point (camera position) o, ray direction d and sampling interval t; S42, calculate sampling points: along the direction d of the light, sample the light uniformly to obtain N sampling points r(t i ), these sampling points will be used as input to the neural network later; S43, obtaining color and density: each sampling point r(t i ) The color and density of the corresponding medium and scene; S44, constructing scene rendering items; S45, constructing a media rendering item; S46. Build the total rendering item.

5. The method for neural implicit 3D reconstruction of underwater scenes according to claim 1, characterized in that: The method for constructing the scene rendering item in S44 is: in, represents the scene color corresponding to the i-th sampling interval, δ i =t i+1 ―t i is the sampling interval between adjacent sampling points, Represents the i-th sampling point r(t i ) corresponds to the scene density, attenuation coefficient, and scene color, is defined as Represents the accumulated transparency of the scene before the ray r reaches the i-th sampling point.

6. The method for neural implicit 3D reconstruction of underwater scenes according to claim 1, characterized in that: The method for constructing the media rendering item in S45 is: in, represents the medium color corresponding to the i-th sampling interval, δ i =t i+1 ―t i is the sampling interval between adjacent sampling points, Represents the i-th sampling point r(t i ) corresponds to the scene density, backscattering coefficient, and medium color, is defined as Represents the accumulated transparency of the medium before the ray r reaches the i-th sampling point.

7. The method for neural implicit 3D reconstruction of underwater scenes according to claim 1, characterized in that: The method for constructing the total rendering item in S46 is: in, Represents the rendering color obtained by simulating the real imaging process along ray r.

8. The neural implicit 3D reconstruction method for underwater scenes according to claim 1, characterized in that: The method for training the neural implicit 3D reconstruction model of the underwater scene in S5 is: S51. Construct SDF regularized loss function L eikonal : in, represents the gradient of the SDF field, and x is any point in three-dimensional space; S52. Construct the square error photometric loss function L color : in, To render the color, C gt For the true color; S53. Construct the total loss function L: L=L color +λ·L eikonal Here, λ is a hyperparameter and is set to 0.1.

Images