Underwater low-light static scene three-dimensional reconstruction and enhancement method based on neural radiation field
Through the method based on neural radiation field, combined with Retinex theory and volume rendering technology, the problem of three-dimensional reconstruction in underwater low-light static scenes is solved, high-quality three-dimensional reconstruction and image enhancement are achieved, and the optical characteristics of adapting to the underwater environment are improved. Reconstruction accuracy and efficiency.
Patent Information
- Application Number
- CN202411121705.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-15
AI Technical Summary
The prior art is difficult to efficiently and with high quality in underwater low-light static scenes, due to the complex optical characteristics and low image quality problems.
Using a method based on neural radiation field, physical modeling is carried out through Retinex theory and volume rendering technology, scene and media neural radiation field models are constructed, and supervision and training is carried out in combination with multi-exposure images and camera internal and external parameters to remove water media interference and brighten three-dimensional scenes. Model, achieving high-quality 3D reconstruction and rendering.
It improves the accuracy and efficiency of underwater three-dimensional reconstruction, can work effectively in low light conditions, and enhances the visual effect of the image, providing higher quality data support for subsequent applications.
Smart Images

Figure CN119068113A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of three-dimensional vision and image processing, and in particular to a method for three-dimensional reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields. Background Art
[0002] With the rapid development of computer vision and computational photography, 3D reconstruction technology has been widely used in many fields, such as virtual reality, augmented reality, cultural heritage protection, and medical imaging. However, the underwater environment has always been a major challenge in the field of 3D reconstruction due to its complex optical properties. The underwater lighting conditions are complex, and the scattering and absorption effects of light are significant, resulting in poor performance of traditional optical-based 3D reconstruction methods in underwater environments.
[0003] Traditional underwater 3D reconstruction methods mainly rely on technologies such as structured light, laser scanning, and multi-view stereo vision. These methods can achieve certain results in shallow water environments or under good lighting conditions, but their performance decreases significantly in low light, deep water, or turbid water. In addition, these methods usually require complex equipment and a large amount of computing resources, making them difficult to promote in practical applications.
[0004] In recent years, the emergence of Neural Radiance Fields (NeRF) technology has brought new opportunities to the field of 3D reconstruction. NeRF uses deep learning technology to train a neural network to represent the radiance field of a 3D scene, and can achieve remarkable results in high-quality reconstruction and rendering. The core idea of NeRF is to implicitly encode the 3D information of the scene in the weights of the neural network and reconstruct high-quality 3D scenes from sparse image views through an optimization process.
[0005] However, the application of NeRF technology in underwater low-light static scenes still faces many challenges. The lighting conditions in the underwater environment are extremely complex, and the propagation of light in water is affected by many factors, including scattering, absorption, and refraction. These factors make underwater images of low quality and bring difficulties to the training and optimization of NeRF. In addition, there are a large number of suspended particles and organisms in the underwater environment, which will also interfere with the acquisition and processing of images. Summary of the invention
[0006] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide a method for three-dimensional reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields, so as to solve the problem that the existing technology is difficult to reconstruct and enhance the three-dimensional model of underwater low-light static scenes efficiently and with high quality.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields, the method comprising the following steps:
[0008] Step a, obtaining multi-exposure visible optical image data of an underwater low-light static scene through a camera;
[0009] Step b, calculating the internal and external parameters of the camera corresponding to the captured visible optical image;
[0010] Step c, based on Retinex theory and volume rendering technology, the light field and reflection field are physically modeled to construct a scene neural radiation field model. Based on volume rendering technology, the water medium is physically modeled to construct a medium neural radiation field model, thereby constructing an overall neural radiation field model of an underwater low-light static scene.
[0011] Step d, inputting the single-exposure visible optical image collected by the camera and the corresponding camera internal and external parameters into the overall neural radiation field model for supervised training to obtain the reconstruction result of the underwater low-light static scene;
[0012] Step e: constructing a brightness enhancement model based on visual theory;
[0013] Step f: Based on the trained scene neural radiation field model, combined with the multi-exposure visible optical images collected by the camera and the corresponding internal and external parameters of the camera, supervised training is performed on the brightness enhancement model to output an underwater three-dimensional scene model after removing water medium interference and brightening, which can render a two-dimensional visible optical image at any viewing angle.
[0014] Furthermore, the step a is specifically as follows: evenly dividing the target scene into acquisition camera points; and acquiring at least two groups of visible optical images with different shutter times at each point.
[0015] Furthermore, in step b, the camera internal and external parameters corresponding to the visible optical image are calculated by the SFM algorithm, specifically:
[0016] Step ①: Use the image feature point extraction algorithm to detect key points and feature descriptors in each image, and find matching feature point pairs in different images through feature descriptors;
[0017] Step ②, estimate the basic matrix through random sampling consistency algorithm, and calculate the essential matrix using the calibrated camera intrinsic parameters;
[0018] Step ③, recover multiple sets of solutions of the camera relative rotation and translation matrices from the essential matrix through the triangulation verification method, and select the correct solution from them;
[0019] Step ④: Reconstruct 3D points from matching feature points using triangulation method;
[0020] Step 5: Starting from two views, gradually add new views, and incrementally optimize the camera extrinsics and 3D point cloud;
[0021] Step ⑥: Estimate the camera intrinsic parameters through pre-calibration or self-calibration methods.
[0022] Furthermore, the step c specifically includes the following sub-steps:
[0023] Step 1: Parameterize the light: Let the starting point of the light be o(x,y,z) and the direction be
[0024] Step ②, discrete sampling: uniformly sample N points r(t i ),i=1,…,N;
[0025] Step 3: Obtain the light field and reflection field: According to the Retinex theory, obtain each sampling point r(t i ) is L(r(t i ))=Φ1(r(t i ),d), the reflection field value is R(r(t i ))=Φ1(r(t i ), the object voxel density is
[0026] Step ④, obtain the water medium color and density: obtain each sampling point r(t i )'s medium color and the medium voxel density
[0027] Step 5: Calculate the cumulative transparency: Calculate the value of each sampling point r(t i )'s cumulative transparency and medium cumulative transparency
[0028] Step ⑥, differentiable volume rendering to obtain predicted images: The neural radiation field models Φ1 and Φ2 are used to predict each sampling point r(t i ) light field value L(r(t i ))、Reflection field value R(r(t i )), object voxel density Object cumulative transparency Medium Color Medium voxel density And the cumulative transparency of the medium The final color value of light r is obtained by differentiable volume rendering And generate a predicted image through the final color values of all lights on the image.
[0029] Furthermore, the differentiable volume rendering calculation formula is:
[0030]
[0031] in, Predicting the color for an object, Predict the color for the medium, δ i is the distance between adjacent sampling points,
[0032] Furthermore, in step d, supervised training is performed using a visible optical image dataset with exposure parameter ∈ 0 collected by the camera, and a gradient descent optimization strategy is adopted, with a training loss of Among them, λ1, λ2, λ3 are hyperparameters; To reconstruct the luminosity loss; Smoothing loss for the light field; The loss for medium voxel density optimization is used to constrain the density distribution of the water medium to make it as uniform as possible; The loss is optimized for the object voxel density to ensure that the scene neural radiation field model can construct the correct object geometry; after training, the neural radiation field model can be accessed from any perspective to obtain the corresponding two-dimensional medium / medium-free reconstruction view.
[0033] Further, Reconstructed photometric loss in The formula is: where sg(·) indicates gradient stop, ε = 10 -3 ,C0, The color values of the visible optical image collected by the camera under exposure parameter ∈ 0 and the image predicted by the neural radiation field models Φ1 and Φ2 respectively;
[0034] Light field smoothing loss The formula is: Where L is the two-dimensional light field image after the light field value L is rendered by differentiable volume, and Respectively represent the gradients of the two-dimensional light field image L in the horizontal and vertical directions;
[0035] Dielectric voxel density optimization loss Using the variance function and assuming that the medium radiation field stops at the intersection of the light and the object surface during volume rendering, the formula is: t i <Depth obj (r), where Var(·) is the sample variance, To predict the depth of the object, t i is the distance from the sampling point to the origin of the light;
[0036] Object voxel density optimization loss Make the distribution of object voxel density on ray r as dense as possible and penalize the non-dense distribution. Its integral form is Where u and v are any two points on the ray r, Its discrete form is where t i ,t i’ is the sampling point on ray r.
[0037] Furthermore, in step f, supervised training is performed on the brightness enhancement model, specifically:
[0038] In the training phase, the input of the brightness enhancement model Φ3 is the light field value L(r(t i )) and brightness adjustment coefficient Among them,∈0,∈ j ∈∈, j = 1, ..., M, ∈ is a set of exposure parameters corresponding to multi-exposure visible optical image data, M = |∈|-1; Φ3 output is the exposure parameter ∈ j Corresponding enhanced light field value After differentiable volume rendering and combining with the scene neural radiation field Φ1 output, the reflection field value R(r(t i )), get the exposure parameter ∈ j The corresponding predicted visible optical image The training adopts the gradient descent optimization strategy, and the training loss Among them, χ1 and χ2 are hyperparameters; To reconstruct the photometric loss, is the color correction loss based on the gray world theory, is the contrast loss;
[0039] In the inference phase, by controlling the brightness adjustment coefficient To obtain underwater low-light static scenes with enhanced brightness.
[0040] Further, Reconstructed photometric loss in The formula is: where sg(·) indicates gradient stop, ε = 10 -3 , C j , They are respectively the exposure parameters ∈ j The lower camera collects visible optical images and the color values of the images predicted by the neural radiation field models Φ1 and Φ2;
[0041] Color correction loss The formula is: Where p,q are RGB channels, and satisfy (p,q)∈{(R,G),(R,B),(G,B)};
[0042] Contrast loss The formula is: Where ζ is a hyperparameter.
[0043] Furthermore, the scene neural radiation field model is a neural network composed of fully connected layers, including: an 8-layer 256-dimensional shared fully connected layer, followed by three branch networks, namely, 1 layer of 128-dimensional fully connected layer, 1 layer of 128-dimensional fully connected layer, 4 layers of 128-dimensional, 128-dimensional, 1-dimensional, and 128-dimensional fully connected layers, respectively used to output object voxel density values, reflection field values, and light field values, respectively, using nonlinear activation function ReLU between layers, and the output layer using Softplus, Sigmoid, and Sigmoid activation functions respectively;
[0044] The medium neural radiation field model is a 4-layer fully connected network, in which the first 3 layers are 256-dimensional and the last layer is 128-dimensional, which is used to output the medium voxel density value and the medium color value of the sampling point. The nonlinear activation function ReLU is used between the layers, and the output layer uses Softpulus and Sigmoid activation functions respectively;
[0045] The brightness enhancement model is a neural network consisting of three fully connected layers with dimensions of 256, 128, and 128, respectively.
[0046] The present invention also provides a three-dimensional reconstruction and enhancement system for underwater low-light static scenes based on neural radiation fields, comprising an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the above-mentioned three-dimensional reconstruction and enhancement method for underwater low-light static scenes based on neural radiation fields.
[0047] The present invention also provides a computer-readable storage medium, in which a computer program is stored, characterized in that the computer program is used to be programmed or configured by a microprocessor to execute the above-mentioned neural radiation field-based underwater low-light static scene three-dimensional reconstruction and enhancement method.
[0048] The beneficial effects of the present invention are as follows: the method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields proposed in the present invention combines the advantages of NeRF and performs physical modeling for the special optical properties of the underwater environment, aiming to improve the accuracy and efficiency of underwater 3D reconstruction. Specifically, the present invention constructs an adaptive neural network structure and an effective training strategy based on Retinex theory and the physical properties of the underwater environment, thereby achieving high-quality underwater 3D reconstruction under low-light conditions. This method can not only work effectively under low-light conditions, but also enhance the visual effects of underwater images, thereby providing higher quality data support for subsequent applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0050] Figure 1 A schematic diagram of a flow chart of a method for 3D reconstruction and enhancement of an underwater low-light static scene based on a neural radiation field provided in an embodiment of the present invention;
[0051] Figure 2 A network structure diagram of the method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.
[0053] Combination Figure 1 The embodiment of the present invention provides a method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields, and the process includes:
[0054] Step a, obtaining multi-exposure visible optical image data of an underwater low-light static scene through a camera;
[0055] Step b, calculating the camera internal and external parameters corresponding to the visible optical image captured by the SFM (Structure from Motion) algorithm;
[0056] Step c, based on Retinex theory and volume rendering technology, the light field and reflection field are physically modeled to construct a scene neural radiation field model. Based on volume rendering technology, the water medium is physically modeled to construct a medium neural radiation field model, thereby constructing an overall neural radiation field model of an underwater low-light static scene.
[0057] Step d, inputting the single-exposure visible optical image collected by the camera and the corresponding camera internal and external parameters into the overall neural radiation field model for supervised training to obtain the reconstruction result of the underwater low-light static scene;
[0058] Step e: constructing a brightness enhancement model based on visual theory;
[0059] Step f: Based on the trained scene neural radiation field model, combined with the multi-exposure visible optical images collected by the camera and the corresponding internal and external parameters of the camera, supervised training is performed on the brightness enhancement model to output an underwater three-dimensional scene model after removing water medium interference and brightening, which can render a two-dimensional visible optical image at any viewing angle.
[0060] In one embodiment, a camera is used to acquire multi-exposure visible optical image data of an underwater low-light static scene, and the acquisition method is specifically as follows: 1) the target scene is evenly divided into acquisition camera points; 2) at each point, at least two groups of visible optical images with different shutter times (sensitivity, aperture and other parameters remain consistent).
[0061] In one embodiment, the camera internal and external parameters corresponding to the visible optical image are calculated by the SFM algorithm, specifically:
[0062] Step 1: Feature detection and feature matching: Use image feature point extraction algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features) or ORB (Oriented FAST and Rotated BRIEF) to detect key points and feature descriptors in each image, and find matching feature point pairs in different images through feature descriptors;
[0063] Step ②, basic matrix and essential matrix estimation: Estimate the basic matrix through the random sampling consensus algorithm (RANSAC, RandomSampleConsensus), and calculate the essential matrix using the calibrated camera intrinsic parameters;
[0064] Step 3, camera extrinsic parameter estimation: recover multiple sets of solutions of the camera relative rotation and translation matrices from the essential matrix through the triangulation verification method, and select the correct solution from them;
[0065] Step ④, triangulation reconstruction of 3D points: use triangulation method to reconstruct 3D points from matching feature points;
[0066] Step 5: Incremental reconstruction: Starting from two views, gradually adding new views, and incrementally optimizing camera extrinsics and 3D point clouds;
[0067] Step ⑥, obtain camera intrinsic parameters: estimate camera intrinsic parameters through pre-calibration or self-calibration method.
[0068] In one embodiment, the light field and the reflection field are physically modeled based on Retinex theory and volume rendering technology, and the water medium is physically modeled based on volume rendering technology, so as to construct a neural radiation field model of an underwater low-light static scene, which specifically includes the following steps:
[0069] Step 1: Parameterize the light: Let the starting point of the light be o(x,y,z) and the direction be Where x, y, z are the three-dimensional coordinates of the starting point of the light, θ, are the pitch angles of the light’s starting point, respectively;
[0070] Step ②, discrete sampling: uniformly sample N points r(t i ),i=1,…,N,r(t)=o+td, t is the sampling interval;
[0071] Step 3, obtain the light field and reflection field: According to the Retinex theory, the light field is the lighting information provided by the light source in the scene, which affects the intensity of light received by the surface of the object. The reflection field is the characteristic of the light reflected by the surface of the object, which is determined by the color and material of the object. Therefore, the light field value of the sampling point is related to the direction of the light, while the reflection field value is independent of the direction of the light. The scene neural radiation field model Φ1 is used to obtain each sampling point r(t i ) is L(r(t i ))=Φ1(r(t i ),d), the reflection field value is R(r(t i ))=
[0072] Φ1(r(t i ), the object voxel density is
[0073] Step ④, obtain the water medium color and density: obtain each sampling point r(t i )'s medium color and the medium voxel density
[0074] Step 5: Calculate the cumulative transparency: Calculate the value of each sampling point r(t i )'s cumulative transparency and medium cumulative transparency Specifically, the cumulative transparency calculation formula is:
[0075]
[0076]
[0077] Among them, δ k is the distance between adjacent sampling points.
[0078] Step ⑥, differentiable volume rendering to obtain predicted images: The neural radiation field models Φ1 and Φ2 are used to predict each sampling point r(t i ) light field value L(r(t i ))、Reflection field value R(r(t i )), object voxel density Object cumulative transparency Medium Color Medium voxel density And the cumulative transparency of the medium The final color value of light r is obtained by differentiable volume rendering And generate a predicted image through the final color values of all lights on the image.
[0079] Specifically, the calculation formula for differentiable volume rendering is:
[0080]
[0081] Among them, δ i is the distance between adjacent sampling points, Predicting the color for an object, Predict the color for the medium, The final cumulative transparency.
[0082] In one embodiment, the visible optical image captured by the camera and the corresponding camera internal and external parameters are input into the neural radiation field model for supervised training to obtain the reconstruction result of the underwater low-light static scene, specifically:
[0083] The supervised training is performed on the visible optical image dataset with exposure parameter ∈ 0 collected by the camera, and the gradient descent optimization strategy is adopted. Among them, λ1, λ2, λ3 are hyperparameters;
[0084] To reconstruct the photometric loss, where sg(·) indicates gradient stop, ε = 10 -3 ,C0, The color values of the visible optical image collected by the camera under exposure parameter ∈ 0 and the image predicted by the neural radiation field models Φ1 and Φ2 respectively;
[0085] is the light field smoothing loss, Among them, L is the two-dimensional light field image after the light field value L is rendered by differentiable volume, and Respectively represent the gradients of the two-dimensional light field image L in the horizontal and vertical directions;
[0086] The loss is optimized for the density of the medium voxel, which is mainly used to constrain the density distribution of the water medium to make it as uniform as possible. The variance function is used here, and it is assumed that the medium radiation field stops at the intersection of the light and the object surface during the volume rendering process. Thus, Where Var(·) is the sample variance, To predict the depth of the object, t i is the distance from the sampling point to the origin of the light;
[0087] To optimize the loss of object voxel density, in order to ensure that the scene neural radiation field can construct the correct object geometry, add Make the distribution of object voxel density on ray r as dense as possible and penalize non-dense distribution. Its integral form is Where u and v are any two points on the ray r, Its discrete form is where t i ,t i’ is the sampling point on the ray r.
[0088] After training, the neural radiation field model can be accessed from any perspective to obtain the corresponding two-dimensional medium / medium-free reconstruction view.
[0089] In one embodiment, based on the trained scene neural radiation field model, combined with the multi-exposure visible optical images collected by the camera and the corresponding camera internal and external parameters, supervised training of the brightness enhancement model is performed, specifically:
[0090] In the training phase, the input of the brightness enhancement model Φ3 is the light field value L(r(t i ))(Exposure parameter is ∈0) and brightness adjustment coefficient Among them,∈0,∈ j ∈∈,j=1,…,M,∈ is a set of exposure parameters corresponding to multi-exposure visible optical image data, M=|∈|-1. Φ3 outputs exposure parameters∈ j Corresponding enhanced light field value After differentiable volume rendering and combining with the scene neural radiation field Φ1 output, the reflection field value R(r(t i )), we can get the exposure parameter ∈ jThe corresponding predicted visible optical image
[0091] The training adopts the gradient descent optimization strategy, and the training loss Among them, χ1 and χ2 are hyperparameters;
[0092] To reconstruct the photometric loss, where sg(·) indicates gradient stop, ε = 10 -3 , C j , They are respectively the exposure parameters ∈ j The lower camera collects visible optical images and the color values of the images predicted by the neural radiation field models Φ1 and Φ2;
[0093] is the color correction loss based on the "gray world" theory, Among them, p,q are RGB channels, and satisfy (p,q)∈{(R,G),(R,B),(G,B)};
[0094] is the contrast loss, Among them, ζ is a hyperparameter.
[0095] In the inference phase, by controlling the brightness adjustment coefficient To obtain underwater low-light static scenes with enhanced brightness.
[0096] In one embodiment, the network structure is as follows Figure 2 As shown, specifically:
[0097] The above-mentioned scene neural radiation field model Φ1 is a neural network composed of fully connected layers, including: an 8-layer 256-dimensional shared fully connected layer, followed by three branch networks, namely 1 layer of 128-dimensional fully connected layer, 1 layer of 128-dimensional fully connected layer, and 4 layers of fully connected layers (128 dimensions, 128 dimensions, 1 dimension, and 128 dimensions, respectively) for outputting object voxel density values, reflection field values, and light field values, respectively. The nonlinear activation function ReLU is used between layers, and the output layer uses Softplus, Sigmoid, and Sigmoid activation functions, respectively.
[0098] The above-mentioned medium neural radiation field model Φ2 is a 4-layer fully connected network, in which the first 3 layers are 256-dimensional and the last layer is 128-dimensional, which is used to output the medium voxel density value and medium color value of the sampling point. The nonlinear activation function ReLU is used between the layers, and the output layer uses Softpulus and Sigmoid activation functions respectively.
[0099] The above brightness enhancement model Φ3 is a neural network composed of 3 fully connected layers (256 dimensions, 128 dimensions, and 128 dimensions respectively).
[0100] In addition, the present invention also provides an embodiment of a system for three-dimensional reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields, comprising an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the method for three-dimensional reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields.
[0101] In addition, the present invention also provides an embodiment of a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by a microprocessor to execute the method for three-dimensional reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields.
[0102] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields, characterized in that: The steps include: Step a, obtaining multi-exposure visible optical image data of an underwater low-light static scene through a camera; Step b, calculating the camera internal and external parameters corresponding to the captured visible optical image; Step c, based on Retinex theory and volume rendering technology, the light field and reflection field are physically modeled to construct a scene neural radiation field model. Based on volume rendering technology, the water medium is physically modeled to construct a medium neural radiation field model, thereby constructing an overall neural radiation field model of an underwater low-light static scene. Step d, inputting the single-exposure visible optical image collected by the camera and the corresponding camera internal and external parameters into the overall neural radiation field model for supervised training to obtain the reconstruction result of the underwater low-light static scene; Step e: constructing a brightness enhancement model based on visual theory; Step f: Based on the trained scene neural radiation field model, combined with the multi-exposure visible optical images collected by the camera and the corresponding internal and external parameters of the camera, supervised training is performed on the brightness enhancement model to output an underwater three-dimensional scene model after removing water medium interference and brightening, which can render a two-dimensional visible optical image at any viewing angle.
2. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to claim 1 is characterized in that: The step a specifically comprises: evenly dividing the target scene into acquisition camera points; and acquiring at least two groups of visible optical images with different shutter times at each point.
3. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to claim 1 is characterized in that: In step b, the camera internal and external parameters corresponding to the visible optical image are calculated by the SFM algorithm, specifically: Step ①: Use the image feature point extraction algorithm to detect key points and feature descriptors in each image, and find matching feature point pairs in different images through feature descriptors; Step ②, estimate the basic matrix through random sampling consistency algorithm, and calculate the essential matrix using the calibrated camera intrinsic parameters; Step ③, recover multiple sets of solutions of the camera relative rotation and translation matrices from the essential matrix through the triangulation verification method, and select the correct solution from them; Step ④: Reconstruct 3D points from matching feature points using triangulation method; Step 5: Starting from two views, gradually add new views, and incrementally optimize the camera extrinsics and 3D point cloud; Step ⑥: Estimate the camera intrinsic parameters through pre-calibration or self-calibration methods.
4. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to claim 1, characterized in that: The step c specifically includes the following sub-steps: Step 1: Parameterize the light: Let the starting point of the light be o(x,y,z) and the direction be Step ②, discrete sampling: uniformly sample N points r(t i ),i=1,…,N; Step 3: Obtain the light field and reflection field: According to the Retinex theory, obtain each sampling point r(t i ) is L(r(t i ))=Φ1(r(t i ),d), the reflection field value is R(r(t i ))=Φ1(r(t i ), the object voxel density is Step ④, obtain the water medium color and density: obtain each sampling point r(t i )'s medium color and the medium voxel density Step 5: Calculate the cumulative transparency: Calculate the value of each sampling point r(t i )'s cumulative transparency and medium cumulative transparency Step ⑥, differentiable volume rendering to obtain predicted images: The neural radiation field models Φ1 and Φ2 are used to predict each sampling point r(t i ) light field value L(r(t i ))、Reflection field value R(r(t i )), object voxel density Object cumulative transparency Medium Color Medium voxel density And the cumulative transparency of the medium The final color value of light r is obtained by differentiable volume rendering And generate a predicted image through the final color values of all lights on the image.
5. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to claim 4 is characterized in that: The differentiable volume rendering calculation formula is: in, Predicting the color for an object, Predict the color for the medium, δ i is the distance between adjacent sampling points, 6. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to claim 4, characterized in that: In step d, supervised training is performed using a visible optical image dataset with exposure parameter ∈ 0 collected by the camera, using a gradient descent optimization strategy, and the training loss Among them, λ1, λ2, λ3 are hyperparameters; To reconstruct the luminosity loss; Smoothing loss for the light field; The loss for medium voxel density optimization is used to constrain the density distribution of the water medium to make it as uniform as possible; The loss is optimized for the object voxel density to ensure that the scene neural radiation field model can construct the correct object geometry; after training, the neural radiation field model can be accessed from any perspective to obtain the corresponding two-dimensional medium / medium-free reconstruction view.
7. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to claim 6, characterized in that: Reconstructed photometric loss in The formula is: where sg(·) indicates gradient stop, ε = 10 -3 , They are the color values of the visible optical image collected by the camera under exposure parameter ∈ 0 and the image predicted by the neural radiation field models Φ1 and Φ2; Light field smoothing loss The formula is: Where L is the two-dimensional light field image after the light field value L is rendered by differentiable volume, and Respectively represent the gradients of the two-dimensional light field image L in the horizontal and vertical directions; Dielectric voxel density optimization loss Using the variance function and assuming that the medium radiation field stops at the intersection of the light and the object surface during volume rendering, the formula is: Where Var(·) is the sample variance, To predict the depth of the object, t i is the distance from the sampling point to the origin of the light; Object voxel density optimization loss Make the distribution of object voxel density on ray r as dense as possible and penalize the non-dense distribution. Its integral form is Where u and v are any two points on the ray r, Its discrete form is where t i ,t i’ is the sampling point on ray r.
8. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to claim 6, characterized in that: In step f, supervised training is performed on the brightness enhancement model, specifically: In the training phase, the input of the brightness enhancement model Φ3 is the light field value L(r(t i )) and brightness adjustment coefficient Among them,∈0,∈ j ∈∈, j = 1, ..., M, ∈ is a set of exposure parameters corresponding to multi-exposure visible optical image data, M = |∈|-1; Φ3 output is the exposure parameter ∈ j Corresponding enhanced light field value After differentiable volume rendering and combining with the scene neural radiation field Φ1 output, the reflection field value R(r(t i )), get the exposure parameter ∈ j The corresponding predicted visible optical image The training adopts the gradient descent optimization strategy, and the training loss Among them, χ1 and χ2 are hyperparameters; To reconstruct the photometric loss, is the color correction loss based on the gray world theory, is the contrast loss; In the inference phase, by controlling the brightness adjustment coefficient To obtain underwater low-light static scenes with enhanced brightness.
9. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to claim 8, characterized in that: Reconstructed photometric loss in The formula is: where sg(·) indicates gradient stop, ε = 10 -3 , They are respectively the exposure parameters ∈ j The lower camera collects visible optical images and the color values of the images predicted by the neural radiation field models Φ1 and Φ2; Color correction loss The formula is: Where p,q are RGB channels, and satisfy (p,q)∈{(R,G),(R,B),(G,B)}; Contrast loss The formula is: Where ζ is a hyperparameter.
10. The method for 3D reconstruction and enhancement of underwater low-light static scenes based on neural radiation fields according to any one of claims 1 to 9, characterized in that: The scene neural radiation field model is a neural network composed of fully connected layers, including: an 8-layer 256-dimensional shared fully connected layer, followed by three branch networks, namely, a 128-dimensional fully connected layer, a 128-dimensional fully connected layer, and 4 fully connected layers of 128 dimensions, 128 dimensions, 1 dimension, and 128 dimensions, respectively, for outputting object voxel density values, reflection field values, and light field values, respectively. The nonlinear activation function ReLU is used between the layers, and the output layer uses Softplus, Sigmoid, and Sigmoid activation functions respectively; The medium neural radiation field model is a 4-layer fully connected network, in which the first 3 layers are 256-dimensional and the last layer is 128-dimensional, which is used to output the medium voxel density value and the medium color value of the sampling point. The nonlinear activation function ReLU is used between the layers, and the output layer uses Softpulus and Sigmoid activation functions respectively; The brightness enhancement model is a neural network consisting of three fully connected layers with dimensions of 256, 128, and 128, respectively.
Citation Information
Patent Citations
Underwater three-dimensional reconstruction method, system device and medium based on neural radiation field
CN118334266A
Intelligent fishery underwater fish body three-dimensional reconstruction method based on neural radiation field
CN118397212A
3D rendering networks based on refractive neural radiance fields
WO2023086398A1
Cited By
Underwater scene nerve implicit three-dimensional reconstruction method
CN119863569A
