An underwater three-dimensional reconstruction method, system device and medium based on neural radiation field

Abstract

The present invention relates to a method, system, and medium for underwater three-dimensional reconstruction based on neural radiation fields. The method comprises: Step 1: establishing a neural radiation field for underwater targets and media, including a signed distance field and a color field for describing the target's geometric information, as well as a parameter field describing the backscattering and attenuation of the underwater media; Step 2: establishing an underwater volume rendering model to simulate the real imaging process and obtain a rendered image; and Step 3: implementing three-dimensional reconstruction of underwater objects by jointly optimizing the object's geometric color information, as well as the underwater scattering and attenuation coefficients, using multiple constraints. The present invention offers the advantages of low cost and high reconstruction accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an underwater three-dimensional reconstruction method, system device and medium based on neural radiation field. Background Art

[0002] With the development of my country's economy, the development and utilization of marine resources has gradually become a key sector of the economy. Compared to traditional manual labor, using robots to replace humans in tasks such as seafloor topography exploration, resource extraction, and archaeological excavations improves efficiency and reduces risks to human safety. To achieve automated robotic operations, high-precision 3D reconstruction of underwater scenes is essential. Therefore, building a high-precision underwater 3D reconstruction system is crucial.

[0003] Current methods for 3D reconstruction in air primarily use SFM (structure-from-motion) approaches to achieve high-precision multi-view reconstruction. These methods first extract image features and match them between different views to estimate depth. The predicted depth maps are then fused to obtain a dense point cloud. To obtain the surface, additional meshing steps, such as Poisson surface reconstruction, are required. However, these processes are complex and can accumulate errors at each stage, often resulting in incomplete 3D models, particularly for non-Lambertian surfaces, as they cannot handle view-dependent color, rendering stereo matching impossible in these regions. Recently, methods based on Neural Radiance Fields (NeRFs) have been proposed, which directly represent 3D models as implicit signed distance fields (SDFs), avoiding error accumulation while generating more complete surfaces. However, these methods are currently only used for 3D reconstruction in air. Furthermore, compared to 3D reconstruction in air, underwater environments suffer from image color distortion, reduced contrast, and loss of detail due to light attenuation and scattering, resulting in reduced feature matching rates across multiple views. Therefore, multi-view reconstruction methods in air cannot achieve high-precision dense reconstruction underwater. Summary of the Invention

[0004] The purpose of the present invention is to provide an underwater three-dimensional reconstruction method, system device and medium based on neural radiation field, to improve the three-dimensional reconstruction method based on neural radiation field NeRF in the air, so that it is suitable for underwater environment, so as to at least solve one of the problems existing in the prior art.

[0005] To achieve the purpose of the present invention, the present invention discloses an underwater three-dimensional reconstruction method based on neural radiation field, comprising the following steps:

[0006] Step 1: Establish the neural radiation field of underwater targets and media, including the signed distance field and color field used to describe the target's geometric information, and the parameter field describing the backscattering and attenuation of the underwater medium;

[0007] Step 2: Establish an underwater rendering model, simulate the real imaging process, and obtain a rendered image;

[0008] Step 3: Through multiple constraints, the geometric color information of the object, as well as the underwater scattering and attenuation coefficients are jointly optimized to achieve 3D reconstruction of the underwater 3D object.

[0009] In a further improvement of the present invention, in step 1, first constructing a neural radiation field comprises:

[0010] SDF field: s, f = F sdf (e(x)), maps a point x in space to its corresponding SDF value s∈R 1 , e is the multi-scale hash code, F sdf is a multi-layer perceptron, f is an intermediate feature;

[0011] Color field c=F c (enc(f, d)), the intermediate feature f and the light viewing angle d∈R 3 Mapped to color value c∈R 3 , enc is the position encoding function, F c is a multi-layer perceptron;

[0012] Parameter field, introducing the backscattering coefficient σ bs , attenuation coefficient σ atten and medium color c med , assuming that the coefficient is uniformly distributed underwater, so it is only related to the observation angle, three MLPs are introduced to model it: σ bs =F bs (d), σ atten =F atten (d), σ med =F med (d), F bs (d) F atten (d) F med (d) both represent multi-layer perceptrons.

[0013] For further improvement of the present invention, step 2 is specifically as follows:

[0014] For volume rendering, the point on the ray corresponding to each pixel is uniformly sampled: r(t) = o + td, r(t) is the sampling point, o∈R 3 is the starting point of the ray, d∈R 3 is the ray direction of the light viewing angle, t is the sampling interval;

[0015] Decompose the rendering process into object items and media items: in, and Respectively represent the object and medium color values ​​of each sampling interval;

[0016] The pixel color is obtained by rendering using the following underwater scattering attenuation model:

[0017]

[0018]

[0019] in,

[0020] Where, are the colors of the object and medium at the i-th interval on the light, δ i is the sampling interval, is the distance of the sampling point on the ray, σ attn and σ bs are the attenuation coefficient and backscattering coefficient, respectively, is the object density value on the i-th ray.

[0021] In a further improvement of the present invention, in step 3, the overall loss function during optimization training is

[0022] L=L rgb +λ*L sdf

[0023] Where, L rgb is the color loss function, L sdf is the loss for constraining the signed distance field;

[0024] After the model training is completed, dense sampling is performed in the reconstructed underwater neural radiation field model, and three-dimensional reconstruction of underwater three-dimensional objects is performed.

[0025] In a further improvement to the present invention, the color loss function is:

[0026] L rgb =1 / N t ||CC * ||

[0027] Where C * is the true color value, N t The number of pixels per sampling.

[0028] Furthermore, the loss of constraining the signed distance field is:

[0029]

[0030] Where, Represents the gradient of the signed distance field, where x is a point in space.

[0031] A further improvement of the present invention is to perform dense sampling in the reconstructed underwater neural radiation field model and to perform three-dimensional reconstruction of underwater three-dimensional objects, including:

[0032] Construct a 512×512×512 three-dimensional point matrix within the bounding box and feed it into the sdf network F sdf , get the sdf value of each point, and finally reconstruct the object mesh model through the surface reconstruction algorithm.

[0033] The present invention provides an underwater 3D reconstruction system based on neural radiation fields, which is used to implement the aforementioned method and includes the following modules:

[0034] An underwater neural radiation field construction module, wherein the underwater neural radiation field construction module is used to establish a neural radiation field of underwater targets and media, including a signed distance field and a color field for describing target geometric information, and a parameter field for describing backscattering and attenuation of the underwater medium;

[0035] A rendering module is used to establish an underwater body rendering model, simulate a real imaging process, and obtain a rendered image;

[0036] The reconstruction module optimizes the geometric color information of the object, as well as the underwater scattering and attenuation coefficients through multiple constraints to achieve three-dimensional reconstruction of underwater three-dimensional objects.

[0037] The present invention provides an underwater three-dimensional reconstruction device based on neural radiation fields, comprising:

[0038] at least one processor;

[0039] at least one memory for storing at least one program;

[0040] When the at least one program is executed by the at least one processor, the at least one processor implements the aforementioned method.

[0041] The present invention provides a storage medium storing a program executable by a processor, wherein the program executable by the processor is used to execute the aforementioned method when executed by the processor.

[0042] Compared with the prior art, the present invention has at least the following beneficial effects and advantages:

[0043] (1) By modeling the underwater body rendering process, the present invention can extend the 3D reconstruction model based on the neural radiation field to the underwater environment, thereby achieving low-cost and high-precision dense 3D reconstruction of underwater scenes, which can be applied to underwater mineral mining, archaeological excavation and other scenes.

[0044] (2) The present invention has the advantages of low cost and high reconstruction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0046] Figure 1 Schematic diagram of the neural radiation field network framework of underwater media and targets in an embodiment of the present invention.

[0047] Figure 2 This is a flowchart of the steps of an underwater three-dimensional reconstruction method based on neural radiation field provided by an embodiment of the present invention.

[0048] Figure 3 It is a schematic diagram of the underwater light imaging model. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] See also Figure 1 The present invention proposes an underwater three-dimensional reconstruction method based on neural radiation field, comprising the following steps:

[0051] Step 1: Use an underwater camera to capture RGB images of underwater targets from evenly distributed different viewing angles, and use the SFM method to obtain camera extrinsics.

[0052] Step 2: Establish the neural radiation field of underwater targets and media, including the signed distance (SDF) field used to describe the target geometric information, the color field, and the parameter field describing the backscattering and attenuation of the underwater medium.

[0053] The neural radiation field constructed in step 2 includes:

[0054] A signed distance field (SDF field): s, f = F o (e(x)), maps a point x in space to its corresponding SDF value o∈R 1 , e is the multi-scale hash code proposed in instant-ngp, F o is a multi-layer perceptron (MLP), f is an intermediate feature, R 1 is a one-dimensional real number. Based on the sdf value s, the density value σ is obtained. obj =φ(o), where φ is a bell-shaped function, such as the logistic function: φ(x) = βe -βx / (1+e -βx ) 2 , β is a function that is optimized along with the network and can control the steepness of the function.

[0055] Color field c=F c (enc(f, d)), the intermediate feature f and the light angle d∈R 3 Mapped to color value c∈R 3 . enc is a position encoding function, such as the common Fourier encoding, F c is another multi-layer perceptron.

[0056] In order to model the imaging process of light in underwater media, the backscattering coefficient σ is introduced bs , attenuation coefficient σ atten and medium color c med .

[0057] Assuming that the coefficients are uniformly distributed underwater, so they are only related to the observation angle, three multi-layer perceptrons (MLPs) are introduced to model the parameter field of the imaging process of light in underwater media: σ bs =F bs (d), σ atten =F atten (d), σ med =F med (d), where σ bs ,σ atten ,σ med Represent the backscattering coefficient, attenuation coefficient and medium density respectively, F bs (d) F atten (d) F med (d) represents three multi-layer perceptrons, which are used to predict the parameter field σ bs ,σ atten ,σ med .

[0058] Step 3: Based on the collected images, the constructed underwater neural radiation field is trained to obtain the geometric information of the object, and the camera parameters are fine-tuned to reduce the underwater camera positioning error.

[0059] Step 4: Establish an underwater body rendering model, simulate the real underwater imaging process, and obtain a rendered image.

[0060] The following introduces the underwater body rendering function. First, the final rendered pixel color C is decomposed into object items C obj and medium item C med :

[0061] To achieve volume rendering, first get the light corresponding to the pixel and uniformly sample the points on the light: r(t) = o + td. Where r(t) is the sampling point on the fiber, o∈R 3 is the starting point of the ray, d∈R 3 is the light viewing angle, t is the sampling position;

[0062] The discretized underwater rendering model is: in, and They represent the object color value and medium color value of each sampling interval respectively, and N is the number of optical fiber sampling points.

[0063] Specifically, the underwater scattering attenuation model ( Figure 3 ) The color of light r is derived, and the color of the i-th sampling interval is:

[0064]

[0065] in

[0066] Where, are the colors of the object and medium at the i-th interval on the light, δ i is the sampling interval, is the distance of the sampling point on the ray, σ attn and σ bs are the attenuation coefficient and backscattering coefficient predicted above, is the object density value on the i-th ray.

[0067] At this point, the rendering based on underwater neural radiation field is completed.

[0068] Step 5: Through multiple constraints, the geometric color information of the object, as well as the underwater scattering and attenuation coefficients are jointly optimized to achieve 3D reconstruction of the underwater 3D object.

[0069] The loss function and training process are introduced below:

[0070] The color loss function is:

[0071] L rgb =1 / N t ||CC * ||

[0072] Where C * is the true color value, N t is the number of pixels sampled each time. In some embodiments of the present invention, N t The value is 1024.

[0073] The loss for constraining the signed distance field is:

[0074]

[0075] in, Represents the gradient of the signed distance field, where x is a point in space.

[0076] In summary, the overall loss function is: L = L rgb +λ*L sdf ,λ is the balance coefficient.

[0077] In some embodiments of the present invention, the balance coefficient λ is 0.1.

[0078] Based on the overall loss function constructed above, the neural radiation field is optimized to reduce the gap between the rendered image and the real image, and gradient backpropagation is performed to optimize the geometric and color parameters;

[0079] In some embodiments of the present invention, the training adopts the Adam optimizer with a learning rate of 5×10 -4 , iterated 50,000 times.

[0080] After the model training is completed, dense sampling is performed in the reconstructed underwater neural radiation field model, and three-dimensional reconstruction of underwater three-dimensional objects is performed.

[0081] Perform dense sampling (construct a 512×512×512 three-dimensional point matrix within the bounding box) and feed it into the trained sdf network F sdf , get the sdf value of each point, and finally reconstruct the object mesh model through the Marching Cubes method or other surface reconstruction algorithms.

[0082] In some embodiments of the present invention, a neural radiation field-based underwater 3D reconstruction system is provided, which is used to implement the aforementioned method and includes the following modules:

[0083] An underwater neural radiation field construction module, wherein the underwater neural radiation field construction module is used to establish a neural radiation field of underwater targets and media, including a signed distance field and a color field for describing target geometric information, and a parameter field for describing backscattering and attenuation of the underwater medium;

[0084] A rendering module is used to establish an underwater body rendering model, simulate a real imaging process, and obtain a rendered image;

[0085] The reconstruction module optimizes the geometric color information of the object, as well as the underwater scattering and attenuation coefficients through multiple constraints to achieve three-dimensional reconstruction of underwater three-dimensional objects.

[0086] In some embodiments of the present invention, an underwater three-dimensional reconstruction device based on neural radiation field is provided, comprising:

[0087] at least one processor;

[0088] at least one memory for storing at least one program;

[0089] When the at least one program is executed by the at least one processor, the at least one processor implements the aforementioned underwater three-dimensional reconstruction based on neural radiation field.

[0090] In some embodiments of the present invention, a storage medium is provided, in which a program executable by a processor is stored. When the program is executed by the processor, it is used to perform the aforementioned underwater three-dimensional reconstruction based on neural radiation field.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. Regarding the apparatus and medium for underwater three-dimensional reconstruction based on neural radiation fields disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and relevant parts can be referred to the method description.

[0092] The aforementioned embodiments of the present invention provide an underwater three-dimensional reconstruction method, system device and medium based on the neural radiation field. First, an underwater camera is used to collect multi-perspective images of the target. Then, the points on the light corresponding to each pixel of the image are sampled, and the color value and density value of each point are predicted in the constructed neural radiation field. The color value corresponding to each light is rendered by an improved volume rendering model based on underwater scattering and attenuation. The neural radiation field is optimized based on the constructed loss function to reduce the gap between the rendered image and the real image, and gradient backpropagation is performed to optimize the geometric and color parameters. The reconstructed neural radiation field is densely sampled to obtain the SDF value of each point, and the object surface is extracted through the marching cubes method to complete the reconstruction of the object.

[0093] The step numbers in the above embodiments are only provided for the convenience of explanation and do not limit the order of the steps. The execution order of each step in the above embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0094] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules or units is not necessarily limited to those steps or modules or units clearly listed, but may include other steps or modules or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0095] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An underwater 3D reconstruction method based on neural radiation field, characterized in that: The steps include: Step 1: Establish the neural radiation field of underwater targets and media, including the signed distance field and color field used to describe the target's geometric information, and the parameter field describing the backscattering and attenuation of the underwater medium; Step 2: Establish an underwater volume rendering model, simulate the real imaging process, and obtain a rendered image; for volume rendering, uniformly sample the points on the light corresponding to each pixel: r(t) = o + td, r(t) is the sampling point, o∈R 3 is the starting point of the ray, d∈R 3 is the ray direction of the light viewing angle, t is the sampling interval; Decompose the rendering process into object items and media items: in and Respectively represent the object and medium color values ​​of each sampling interval; The pixel color is obtained by rendering using the following underwater scattering attenuation model: in Where, are the colors of the object and medium at the i-th interval on the light, δ i is the sampling interval, is the distance of the sampling point on the ray, σ attn and σ bs are the attenuation coefficient and backscattering coefficient, respectively, is the density value of the object on the i-th ray; Step 3: Through multiple constraints, the geometric color information of the object, as well as the underwater scattering and attenuation coefficients are jointly optimized to achieve 3D reconstruction of the underwater 3D object.

2. The underwater 3D reconstruction method based on neural radiation field according to claim 1, characterized in that: In step 1, the neural radiation field is first constructed, including: SDF field: s,f=F sdf (e(x)), maps a point x in space to its corresponding SDF value s∈R 1 , e is the multi-scale hash code, F sdf is a multi-layer perceptron, f is an intermediate feature; Color field c=F c (enc(f,d)), the intermediate feature f and the light angle d∈R 3 Mapped to color value c∈R 3 , enc is the position encoding function, F c is a multi-layer perceptron; Parameter field, introducing the backscattering coefficient σ bs , attenuation coefficient σ atten and medium color σ med , assuming that the coefficient is uniformly distributed underwater, so it is only related to the observation angle, three MLPs are introduced to model it: σ bs =F bs (d),σ atten =F atten (d),σ med =F med (d), F bs (d) F atten (d) F med (d) both represent multi-layer perceptrons.

3. The underwater 3D reconstruction method based on neural radiation field according to any one of claims 1-2, characterized in that: In step 3, the overall loss function during optimization training is L=L rgb +λ*L sdf Where, L rgb is the color loss function, L sdf is the loss for constraining the signed distance field; After the model training is completed, dense sampling is performed in the reconstructed underwater neural radiation field model, and three-dimensional reconstruction of underwater three-dimensional objects is performed.

4. The underwater 3D reconstruction method based on neural radiation field according to claim 3, characterized in that: The color loss function is: L rgb =1 / N t ||C-C * || Where C * is the true color value, N t The number of pixels per sampling.

5. The underwater 3D reconstruction method based on neural radiation field according to claim 3, characterized in that: The loss for constraining the signed distance field is: Where, Represents the gradient of the signed distance field, where x is a point in space.

6. The underwater 3D reconstruction method based on neural radiation field according to claim 3, characterized in that: Dense sampling is performed in the reconstructed underwater neural radiation field model, and 3D reconstruction of underwater 3D objects is performed, including: Construct a three-dimensional point matrix within the bounding box and feed it into the sdf network F sdf , get the sdf value of each point, and finally reconstruct the object mesh model through the surface reconstruction algorithm.

7. An underwater 3D reconstruction system based on neural radiation field, characterized in that: The method for implementing any one of claims 1 to 6 comprises the following modules: Underwater neural radiation field construction module, used to establish the neural radiation field of underwater targets and media, including the signed distance field and color field used to describe the target's geometric information, and the parameter field describing the backscattering and attenuation of the underwater medium; Rendering module, used to build underwater rendering model, simulate the real imaging process, and obtain rendered images; The reconstruction module jointly optimizes the geometric color information of objects, as well as underwater scattering and attenuation coefficients through multiple constraints to achieve three-dimensional reconstruction of underwater three-dimensional objects.

8. An underwater three-dimensional reconstruction device based on neural radiation field, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

9. A storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

Images