Underwater Scene Reconstruction Device and Method Based on Neural Radiance Field

Abstract

This application relates to a neural radiation field-based underwater scene reconstruction device and method, which relates to the field of 3D reconstruction technology. The device constructs the SP-SeaNeRF underwater 3D reconstruction model based on the neural radiation field. The SP-SeaNeRF model includes a neural sample filtering module, a neural radiation field module, a neural scattering field module, and a volume rendering module. The neural sample filtering module is used to obtain sample points distributed on the object surface along rays; the neural radiation field module is used to input the sampling point position and illumination embedding vector to output the color and density of the sampling point; the neural scattering field module is used to input the position-encoded ray direction to output water body parameters; and the volume rendering module is used to render the color, density, and water body parameters output by the neural radiation field module and the neural scattering field module into pixel colors. The SP-SeaNeRF model is designed to simulate the degradation process of underwater images, unsupervisedly learn scattering information, and render high-quality, scatter-free new perspective images.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to an underwater scene reconstruction device and method based on neural radiation fields. Background Art

[0002] As human exploration and development of the ocean continue to deepen, underwater scene reconstruction has attracted considerable attention. Fields such as ocean exploration, underwater archaeology, and seabed resource development all require accurate underwater scene reconstruction technology. Unlike scenes in the atmosphere, light is significantly absorbed and scattered in water, resulting in color casts and blurring in underwater optical images. Therefore, achieving high-quality reconstructions and restoring the scene's true colors typically requires the use of additional sensors such as lasers and sonar to acquire additional data, which incurs higher costs and more computational complexity. Furthermore, the movement and obstruction of fish, and the movement of aquatic plants can severely disrupt the consistency of the scene. These factors make underwater scene reconstruction a challenging problem.

[0003] The Neural Radiance Field (NeRF), proposed by Mildenhall et al., uses an MLP to learn the mapping between position and direction, color, and density, enabling the rendering of photorealistic images from new perspectives. As a neural implicit representation, NeRF can effectively represent large, unbounded scenes. Its high-quality rendering relies on precise sampling points. Common sampling methods are based on coarse-to-fine resampling, which simultaneously optimizes both the coarse and fine networks, using the coarse network to generate a probability density function. The finer points sampled from the probability density function are then fed into the fine network.

[0004] NeRF neural radiance fields (NeRFs) have been widely used in various applications of modeling light-matter interactions, including specular reflection, image dehazing, dark area enhancement, water surface refraction, and underwater scattering. Both WaterNeRF (a neural radiance field for underwater scenes) and SeaThru-NeRF (a neural radiance field for scattering media) model underwater scenes using continuous radiance fields. The difference between the two is that WaterNeRF uses a set of monocular underwater images and histogram-equalized images as input, estimating absorption and backscattering coefficients based on the histogram-equalized images. Its primary application is underwater robots with onboard point light sources. SeaThru-NeRF (a neural radiance field for scattering media) only requires a set of monocular underwater images as input. Based on the SeaThru imaging model, it develops a new rendering model to learn the 3D scene information and medium parameters. It considers underwater target image reconstruction as a combination of "underwater ground truth" and "backscattering," using two separate networks trained separately. This is similar to the classification of outdoor scene images into “static” buildings and “transient” occlusions in Neural Radiance Fields (NeRF-W) for unconstrained photo collections.

[0005] However, the neural radiation field of WaterNeRF for underwater scenarios was tested and verified in a relatively ideal laboratory underwater environment, and its adaptability to actual complex underwater environments is poor. Although the neural radiation field of SeaThru-NeRF in scattering media was tested and verified in real-world scenarios, it has poor versatility and performs poorly in clearer or more turbid water.

[0006] The water surface is typically not completely still. Waves cause light to refract and reflect irregularly, causing underwater illumination to change dynamically with the waves. When particles in the water are between 1 and 100 nm, the Tyndall effect occurs, scattering light and creating numerous "light pillars" that change with the influence of waves and currents. As a result, scene consistency is significantly reduced, resulting in numerous artifacts in the predicted underwater imagery and a large number of floating objects being incorrectly generated in the density field.

[0007] The NeRF neural radiation field method itself is computationally complex. Its coarse-to-fine sampling method requires multiple queries to the MLP multi-layer perceptron to obtain fine-grained sampling point locations, thus requiring significant computing resources and time for training and inference. Furthermore, underwater data acquisition and processing are more complex, further increasing the computational burden and making it difficult to meet real-time and high-efficiency requirements in practical applications. Summary of the Invention

[0008] This application relates to a device and method for underwater scene reconstruction based on neural radiation fields. The goal is to address the technical drawbacks of low-quality reconstruction of underwater images due to scattering and unstable illumination. By simulating the degradation process of underwater images, unsupervised learning of scattering information allows for rendering high-quality, scatter-free images from a new perspective. The technical solution is as follows:

[0009] On the one hand, a neural radiation field-based underwater scene reconstruction device is provided, and an SP-SeaNeRF neural radiation field-based underwater 3D reconstruction model is constructed. The SP-SeaNeRF neural radiation field-based underwater 3D reconstruction model includes a neural sample filtering module, a neural radiation field module, a neural scattering field module, and a volume rendering module.

[0010] The neural sample filtering module is used to obtain sampling points distributed on the surface of the object on the ray for each pixel of the input underwater image;

[0011] The neural radiation field module is used to input the sampling point position and illumination embedding vector of the neural sample filtering module to output the color and density of the sampling point;

[0012] The neural scattered field module is used to input the position-encoded ray directions to output water parameters;

[0013] The volume rendering module is used to render the color parameters and density parameters output by the neural radiation field module and the water parameters output by the neural scattering field module into pixel colors.

[0014] The neural sample filtering module includes: a multi-layer perceptron and a filter. The multi-layer perceptron is used to map rays into a set of sampling points, and the filter is used to perform sparse sampling on dense sampling points.

[0015] The neural radiance field module includes: two multi-layer perceptrons and illumination embedding vectors.

[0016] The illumination embedding vector is specifically expressed as follows:

[0017]

[0018] Among them, dynamic lighting l dynamic Represented as the set of transient illumination in the N input images Then take the average of these transient lighting as the static lighting of the scene static , each image P i Transient lighting Represented as this static lighting l static Added with a corresponding illumination bias, and for each image P i Allocate a length of n l The corresponding real-valued lighting embedding vector l i, these embedding vectors will be along with the network R Θ2 They are optimized together to learn the lighting bias in the corresponding image.

[0019] The Neural Scattering Field module includes: a multi-layer perceptron.

[0020] The volume rendering module includes: rendering equations for underwater images.

[0021] The rendering equation for underwater images is as follows:

[0022]

[0023] Among them, the ray will get K sampling points after passing through the neural sample filtering module, and the ray is divided into K-1 intervals, each interval L i =[s i ,s i+1 The length of ] is and J i Represents interval L i The positive transmission color, scattered color and true color of the object; Indicates that light is transmitted to s without a medium i The probability of being at Indicates that light is transmitted to s in water without any objects i The probability of Indicates that the light is in the interval L i The probability of being blocked inside; attn i bs =exp(-σ bs s i )and Represents the shading coefficient, which is accumulated along the ray for each interval and The color of the underwater image can be obtained;

[0024]

[0025] Multiply the object's true color J by the attenuation Then add Get underwater images

[0026] On the other hand, a method for underwater scene reconstruction based on neural radiation field is provided, which is applied to the above-mentioned underwater scene reconstruction device based on neural radiation field, and the method includes:

[0027] Each pixel of the input underwater image is filtered through the neural sample filtering module to obtain sampling points distributed on the surface of the object;

[0028] Input the sampling point position and illumination embedding vector of the neural sample filtering module through the neural radiation field module to output the color and density of the sampling point;

[0029] Based on the ray coding information, the neural scattered field module inputs the position-encoded ray direction to output water parameters;

[0030] The color parameters and density parameters output by the neural radiation field module and the water parameters output by the neural scattered field module are rendered into pixel colors through the volume rendering module.

[0031] The color and density parameters output by the neural radiance field module and the water parameters output by the neural scattering field module are rendered into pixel colors through the volume rendering module, which also includes:

[0032] Reconstruction loss of RawNeRF:

[0033]

[0034] in, is the predicted color, C is the true value, sg() means stop gradient, ∈=10 -3 .

[0035] Sample loss function:

[0036] L sample (t,w)=(u(t)-d r (w)) 2 +(e v(t) -b-1)

[0037] Among them, t is the position of the sampling point on the ray, w is the interval L i The weight of , u(t) and v(t) are the mean and variance of the sampling point positions respectively; d r (w) is the depth estimate, taken from the maximum weight interval L max The midpoint of b = 10 -4 ×(t f -t n ) -1 Represents the expected sampling point variance, introducing the hyperparameter near plane t n and far plane t f This allows sampling points to be relatively densely distributed in scenes of different scales.

[0038] The beneficial effects of the technical solution provided by this application include at least:

[0039] A neural radiation field-based underwater 3D reconstruction model, SP-SeaNeRF, is constructed. Through the neural sample filtering module, the network outputs the positions of the sampling points under the constraint of sample loss. The filter further screens the sampling points to obtain the sampling points distributed on the object surface. The neural radiation field module embeds the sampling point positions and illumination into vectors to predict the color and density of the sampling points. The neural scattering field module is used to predict water body parameters, and the volume rendering module is used to simulate the degradation process of underwater images. By establishing a direct connection between position and color, the SP-SeaNeRF model has strong scattering perception capabilities and can effectively decouple object color and scattered color. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 A schematic structural diagram of an underwater scene reconstruction device based on neural radiation field provided by an exemplary embodiment of the present application is shown;

[0042] Figure 2 Shown Figure 1 Schematic diagram of the structure of the neural sample filtering module;

[0043] Figure 3 Shown Figure 1 Schematic diagram of the structure of the neural radiation field module;

[0044] Figure 4 Shown Figure 1 Schematic diagram of the structure of the neural scattered field module;

[0045] Figure 5 Shown Figure 1 Schematic diagram of the structure of the middle body rendering module;

[0046] Figure 6 A schematic flow chart of a method for underwater scene reconstruction based on neural radiation fields provided by an exemplary embodiment of the present application is shown;

[0047] Figure 7 The predicted underwater image and acc image are shown, where Figure 7 a shows the underwater image and acc image predicted using the illumination embedding vector; Figure 7 b shows the underwater image and acc image predicted without using the illumination embedding vector;

[0048] Figure 8A schematic diagram of a degradation process provided by an exemplary embodiment of the present application is shown;

[0049] Figure 9 A comparison of the new view rendering of each model on the SeaThru-NeRF dataset is shown;

[0050] Figure 10 Shows a comparison of the new view rendering of each model on the Neuralsea dataset;

[0051] Figure 11 The figure shows the color restoration ability comparison with UIESS and TUDA on the 18-color DGK color chart in the Curasao scene. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0053] First, the terms used in the various embodiments of this application are explained:

[0054] MLP, short for Multilayer Perceptron, is a basic form of artificial neural network. It consists of an input layer, one or more hidden layers, and an output layer. Layers are connected by weights, but there are no connections within a layer, forming a feedforward network structure. Signals can only flow from one layer to the next, preventing loops.

[0055] SP-SeaNeRF, an underwater 3D reconstruction model based on neural radiance fields, which has strong scattering perception, is an underwater 3D reconstruction method based on neural radiance fields (NeRF), specially designed to improve underwater visual effects, especially combining 3D information and physical imaging models to alleviate color attenuation and blur caused by water bodies, thereby improving the clarity and color fidelity of underwater images or videos.

[0056] SeaThru-NeRF is an underwater image enhancement method based on Neural Radiance Fields (NeRF), specifically designed to improve underwater visual effects, especially by alleviating color attenuation and blur caused by water, thereby improving the clarity and color fidelity of underwater images or videos.

[0057] Coarse-to-fine is a sampling method that divides the sampling points when obtaining rays. The sampling is dense near the points with large color contribution, and sparse near the points with small contribution.

[0058] Positional Encoding (PE) is a method for extending low-dimensional information to high dimensions. It maps directions and coordinates into a high-dimensional space using sine and cosine functions to increase the model's expressiveness and generalization capabilities.

[0059] IPE, Integrated Positional Encoding, is a generalization of PE that allows a Gaussian region to be compactly characterized rather than a single point in space; it is intended to enhance the expressiveness and generalization capabilities of the model.

[0060] PSNR (Peak Signal-to-Noise Ratio) measures the degree of distortion of the reconstructed signal of an image or video relative to the original content. It measures the difference between the distorted signal and the original signal and is expressed in decibels after logarithmic transformation. Higher PSNR values indicate less distortion.

[0061] This application provides an underwater scene reconstruction device and method based on neural radiation fields. Through the self-built SP-SeaNeRF underwater three-dimensional reconstruction model based on neural radiation fields, the degradation process of underwater images is simulated, scattering information is learned in an unsupervised manner, and finally a high-quality scattering-free new perspective image is rendered.

[0062] SP-SeaNeRF, a neural radiance field-based underwater 3D reconstruction model, acquires 3D scene information and then learns water parameters based on a physical model of underwater imaging. After predicting the true color, the SP-SeaNeRF model degrades the true color to the color of the underwater image based on the water parameters. SP-SeaNeRF only takes the degraded underwater image as input and does not require any color distribution priors. Because the SP-SeaNeRF model is scatter-aware, it can reduce scattering interference during reconstruction, thereby reducing artifacts and floating objects. During rendering, scattering is removed based on the water parameters, resulting in high-quality, scatter-free images from a new perspective.

[0063] On the one hand, a device for reconstructing underwater scenes based on neural radiation fields is provided. Figure 1 A schematic diagram of the structure of an underwater scene reconstruction device based on neural radiation fields provided by an exemplary embodiment of the present application is shown. The device constructs an SP-SeaNeRF underwater 3D reconstruction model based on neural radiation fields, predicts water body parameters through a neural scattering field module, and simulates the degradation process of underwater images based on a volume rendering module. By establishing a direct connection between position and color, the SP-SeaNeRF model has strong scattering perception capabilities and can effectively decouple object color and scattered color.

[0064] The SP-SeaNeRF underwater 3D reconstruction model based on neural radiation field includes a neural sample filtering module 10, a neural radiation field module 20, a neural scattered field module 30 and a volume rendering module 40;

[0065] The neural sample filtering module 10 is used to obtain sampling points distributed on the surface of the object on the ray for each pixel of the input underwater image;

[0066] The neural radiation field module 20 is used to input the sampling point position and illumination embedding vector of the neural sample filtering module to predict the color and density of the sampling point;

[0067] The neural scattered field module 30 is used to input the position-encoded ray direction to obtain water parameters;

[0068] The volume rendering module 40 is used to render the color, density, and water parameters of the sampling points predicted by the neural radiation field module and the neural scattered field module into pixel colors.

[0069] Specific implementation process, such as Figure 1 As shown in Figure 1, first, a ray is emitted for each pixel of the input underwater image to obtain the ray ray(o,d). Then, the origin o and direction d of the ray are position-encoded PE respectively, and then spliced together to obtain the ray encoding information. The neural sample filtering module is used to output a set of sampling points on the ray according to the ray encoding information. Sampling point It is then encoded into a frustum by IPE and fed into the neural radiance field module along with the lighting embedding vector to predict the color of each point. and density At the same time, the ray encoding information is also input into the neural scattered field module to predict the shielding light c of each ray. bs , that is, the backscattered color at infinity and the positive transmission attenuation coefficient σ attn and the backscatter attenuation coefficient σ bs Finally, the parameter data predicted by the neural radiation field module and the neural scattering field module are brought into the rendering equation and then input into the volume rendering module, so that the color, density, and water parameters of the sampling point are rendered into the color of the pixel.

[0070] In summary, the apparatus provided in the embodiments of the present application constructs the SP-SeaNeRF underwater 3D reconstruction model based on neural radiance fields. Through the neural sample filtering module, the network outputs the locations of sampling points, subject to sample loss constraints. The filter further filters the sample points to obtain the distribution of sampling points on the object surface. The neural radiance field module embeds the sampling point locations and illumination into vectors to predict the color and density of the sampling points. The neural scattering field module predicts water parameters, and the volume rendering module simulates the degradation process of underwater images. By establishing a direct link between position and color, the SP-SeaNeRF model possesses strong scatter perception capabilities and can effectively decouple object color and scattered color. The SP-SeaNeRF model uses neural radiance field technology to implicitly reconstruct underwater scenes and integrates underwater imaging models to achieve unsupervised scatter removal. The neural sample filtering module uses a multilayer perceptron to achieve precise and dense sampling on rays. The neural radiance field module uses a multilayer perceptron to predict the density and color of sampling points and creates illumination embedding vectors using a generative implicit optimization method to model dynamic illumination. The neural scattering field uses a multilayer perceptron to predict water parameters. The volume rendering module simulates the underwater image degradation process through the underwater imaging model.

[0071] Figure 2 Shown Figure 1 Schematic diagram of the structure of the neural sample filtering module; Figure 2 As shown in Figure 2, the neural sample filtering module is used to obtain sampling points distributed on the surface of the object on the ray; its input is the ray origin o and direction d after position encoding, and the output is a set of sampling points on the ray The neural sample filtering module consists of a multi-layer perceptron and a filter.

[0072] Among them, the hidden layer of the multilayer perceptron consists of 8 linear layers with a width of 256, which uses the activation function of the leaky corrected linear unit, and there is a jump connection after the input layer and the fourth layer of the hidden layer; the output layer of the multilayer perceptron MLP is a linear layer with a width of 64, which uses the sigmoid activation function.

[0073] After the multi-layer perceptron MLP outputs a set of sampling points, the filter Densely distributed areas are sparsely sampled.

[0074] In the embodiment of the present application, for the ray origin o and direction d, the triplet (x o ,y o ,z o ) and bigrams Represented by, and both use position encoding γ k(·), k is the dimension of the encoding, and then the encoded data is input into the neural sample field S Θ , to output the sampling point locations:

[0075]

[0076] S Θ It has 8 fully connected (FC) layers with a width of 256, with jump connections between the input and the fourth layer. The hidden layer uses the LeakyReLU activation function and the output layer uses the Sigmoid activation function. After the neural radiation field module predicts the density, it can perform depth estimation and use the depth information to constrain S Θ The output of , thus obtaining the sampling points distributed near the surface of the object.

[0077] It should be noted that only a few key points play a decisive role in rendering, so overly dense points can be sparsely sampled; therefore, a filter similar to a sliding window is proposed to sample key points:

[0078]

[0079] The number of sampling points in the window is fixed to η, the step length of the window movement is τ, and the threshold is ε. According to the following formula, the sampling points t in the window can be filtered to

[0080]

[0081] Among them, ε s =ε / (t f -t n ), the filter will adjust the threshold according to the scale of the scene to adapt to scenes of different scales. In the filter, first calculate the length of n points in the window length = t i+η-1 -t i , if length < ε s , a point is generated in the middle of the window to replace the n points, otherwise the n points are directly copied, and then the window is slid back by τ points, and the process within the window is repeated until the window slides to the last sampling point.

[0082] Figure 3 Shown Figure 1 Schematic diagram of the structure of the neural radiation field module; Figure 3 As shown in Figure 1, the Neural Radiance Field module is used to obtain the color and density of each sampling point under stable lighting. Its input is the sampling point coordinates after integrated position encoding and the lighting embedding vector, and its output is the color and density of the sampling point. The Neural Radiance Field module includes two multi-layer perceptrons and the lighting embedding vector.

[0083] Among them, the first multi-layer perceptron is used to obtain density, and its input is the coordinates of the sampling points after integrated position encoding; its hidden layer consists of 8 linear layers with a width of 256, which uses the rectified linear unit activation function, and there is a jump connection after the fourth layer of the input layer and the hidden layer; the output layer is a linear layer with a width of 1, which uses the Softplus activation function.

[0084] The second multilayer perceptron is used to obtain color. Its input is the illumination embedding vector and the bottleneck vector output by the hidden layer of the first multilayer perceptron. Its hidden layer consists of a linear layer with a width of 128, which uses the activation function of the rectified linear unit. The output layer is a linear layer with a width of 3, which uses the sigmoid activation function.

[0085] This embodiment models dynamic lighting as a combination of static lighting and multiple lighting biases, and uses a learnable lighting embedding vector to learn these lighting biases, which are removed during image prediction to ensure consistent lighting in images rendered from different perspectives.

[0086] In the embodiment of the present application, after the ray obtains the sampling point in the neural sample filtering module, the integrated position encoding IPE, γ() is used to represent the ray as a frustum in Mip-NeRF to avoid aliasing problems. The neural radiation field module includes two multi-layer perceptrons MLP, which are instantiated as R Θ1 、R Θ2 , used to learn the position t^ of the point on the ray and the illumination embedding vector l to the object color c obj and density σ obj The example formula is as follows:

[0087]

[0088] c obj =R Θ2 (b,l) (5)

[0089] Where b is R Θ1 Output bottleneck vector, R Θ1 As the backbone network, its structure is similar to S Θ Same, R Θ2 It is a lightweight network with only one FC layer with a width of 128. The hidden layers all use the ReLU activation function. The Softplus activation function generates σ obj , the Sigmoid activation function generates c obj .

[0090] In a real underwater environment, dynamic lighting cannot be avoided. The embodiment of the present application uses the lighting embedding vector to effectively reduce the artifacts and floating objects caused by dynamic lighting. The lighting embedding vector is only used as RΘ2 This allows the model to change the lighting of the scene in a specific image while still ensuring that the 3D geometry of the static scene is shared across all images. Setting the lighting embedding vector to all zeros during inference renders images of underwater scenes with stable lighting.

[0091] The illumination embedding vector is specifically expressed as follows:

[0092]

[0093] Among them, dynamic lighting l dynamic Represented as the set of transient illumination in the N input images Then take the average of these transient lighting as the static lighting of the scene static , each image P i Transient lighting Represented as this static lighting l static Added with a corresponding illumination bias, and for each image P i Allocate a length of n l The corresponding real-valued lighting embedding vector l i , these embedding vectors will be along with the network R Θ2 They are optimized together to learn the lighting bias in the corresponding image.

[0094] Figure 7 a shows the underwater image and acc image predicted using the illumination embedding vector; Figure 7 b shows an underwater image and an acc image predicted without the illumination embedding vector. Comparing the red boxes, we can see that the underwater image without the illumination embedding vector has more artifacts, and the corresponding area in the acc image also has more floating objects. This is due to the dynamic underwater lighting. The illumination embedding vector can learn the transient illumination of each image, and then remove it during rendering to ensure that the reconstructed scene has stable lighting, thereby reducing artifacts and floating objects.

[0095] Figure 4 Shown Figure 1 The structural diagram of the neural scattered field module is as follows: Figure 4 As shown in Figure 1, the Neural Scattering Field module inputs position-encoded ray directions to obtain water parameters, namely, obscured light, forward transmission attenuation coefficient, and backscattering attenuation coefficient. Its input is the position-encoded ray directions, and its output is the water parameters. The Neural Scattering Field module includes a multilayer perceptron, which maps rays to a set of sampling points, and a filter that sparsely samples densely packed sampling points.

[0096] Among them, the hidden layer of the multilayer perceptron consists of a linear layer with a width of 128, which uses the Softplus activation function; the shielding light output layer uses the Sigmoid activation function; the forward transmission attenuation coefficient output uses the Softplus activation function, and the backscattering attenuation coefficient output layer uses the Softplus activation function. The three output layers are all linear layers with a width of 3.

[0097] In the embodiment of the present application, in order to simulate the degradation process of underwater images, a multi-layer perceptron MLP is used as the underwater scattering field R Θ3 , learn ray directions To the positive transmission attenuation coefficient σ attn , backscatter attenuation coefficient σ bs and shielding light c bs (backscattered color at infinity):

[0098]

[0099] Among them, R Θ3 It is a FC layer with a width of 128 and uses the Softplus activation function. The Softplus activation function is used to generate σ bs and σ attn , use Sigmoid activation function to generate c bs .

[0100] Figure 5 Shown Figure 1 The structural diagram of the body rendering module; Figure 5 As shown, the volume rendering module is used to render the color parameters and density parameters output by the neural radiation field and the water parameters output by the neural scattering field into pixel colors. The volume rendering module includes: a rendering equation for underwater images.

[0101] like Figure 5 As shown, the rendering process is an underwater image The degradation process of the object's true color J is the backscattered color process. Figure 8 A schematic diagram of a degradation visualization process provided by an exemplary embodiment of the present application is shown in FIG. Figure 8 As shown, the object's true color J is multiplied by the attenuation Then add You can get underwater images

[0102] The rendering process is divided into two steps. The first step is to render the color and density into a non-scattering underwater image. The rendering equation is:

[0103]

[0104] in:

[0105]

[0106] The second step is to degrade the non-scattered underwater image according to the water body parameters to obtain a degraded underwater image with color cast and blur. The rendering equation is:

[0107]

[0108] in:

[0109]

[0110] The ray passes through the neural sample filtering module and obtains K sampling points. The ray is divided into K-1 intervals, and each interval L i =[s i ,s i+1 ] has a length of δ i =s i+1 -s i ; The complete rendering equation is as follows:

[0111]

[0112] Figure 6 FIG. 1 shows a flowchart of a method for underwater scene reconstruction based on neural radiation field provided by an exemplary embodiment of the present application. Figure 6 As shown, on the other hand, a method for underwater scene reconstruction based on neural radiation field is provided, which is applied to the underwater scene reconstruction device based on neural radiation field as described above, and the method includes:

[0113] Each pixel of the input underwater image is filtered through the neural sample filtering module to obtain sampling points distributed on the surface of the object;

[0114] Input the sampling point position and illumination embedding vector of the neural sample filtering module through the neural radiation field module to output the color and density of the sampling point;

[0115] Based on the ray coding information, the neural scattered field module inputs the position-encoded ray direction to output water parameters;

[0116] The color and density parameters output by the neural radiation field module and the water parameters output by the neural scattered field module are rendered into pixel colors through the volume rendering module.

[0117] The color and density parameters output by the neural radiation field module and the water body parameters output by the neural scattered field module are rendered into pixel colors through the volume rendering module, which also includes a loss function.

[0118] The total loss function is as follows:

[0119]

[0120] Among them, λ1=0.0001 and λ2=0.001 are the balance coefficients of the regularization term, and their values are selected through cross-validation.

[0121] The standard L2 reconstruction loss is mainly dominated by the error in bright areas, and the brightness of the underwater environment is relatively low, so we use the reconstruction loss of RawNeRF:

[0122]

[0123] in, is the predicted color, C is the true value, sg() means stop gradient, ∈=10 -3 .

[0124] In order to make the objects in the density field have clearer boundaries, the transmission T obj Add a prior that is either 0 or 1, which can force a binary separation of water from the scene and convert the transmittance T obj Modeled as a mixture of two Laplace distributions and using negative log-likelihood loss:

[0125]

[0126] In order to constrain the output of the neural sample field, we first calculate the mean u(t) and variance v(t) of the sampling point position t on the ray, and then select the interval L with the largest weight w max The midpoint of the depth is used as the depth estimate d r (w), and for u(t) and d r (w) L2 regularization encourages sampling points to be d r For the central distribution, a penalty term about v is added at the end to make the sampling points more concentrated.

[0127] Sample loss function:

[0128] L sample (t,w)=(u(t)-d r (w)) 2 +(e v(t)-b -1) (22)

[0129] Among them, t is the position of the sampling point on the ray, w is the interval L i The weight of , u(t) and v(t) are the mean and variance of the sampling point positions respectively; d r (w) is the depth estimate, taken from the maximum weight interval L max The midpoint of b = 10 -4 ×(t f -tn ) -1 Represents the expected sampling point variance, introducing the hyperparameter near plane t n and far plane t f This allows sampling points to be relatively densely distributed in scenes of different scales.

[0130] In summary, the method provided in the embodiments of the present application uses a neural sample filtering module. Under the constraint of sample loss, the network outputs the position of the sampling point, and the filter further screens the sampling points to obtain the sampling points distributed on the surface of the object; the sampling point position and illumination are embedded into a vector through the neural radiation field module to predict the color and density of the sampling point; the neural scattering field module is used to predict water body parameters, and the volume rendering module is used to simulate the degradation process of underwater images. By establishing a direct connection between position and color, the SP-SeaNeRF model has strong scattering perception ability and can effectively decouple object color and scattered color.

[0131] The SP-SeaNeRF model uses neural radiance field technology to implicitly reconstruct underwater scenes and integrates it with an underwater imaging model to achieve unsupervised scatter removal. The neural sample filtering module uses a multi-layer perceptron to achieve precise and dense sampling of rays. The neural radiance field module uses a multi-layer perceptron to predict the density and color of sampling points and models dynamic lighting by creating illumination embedding vectors using a generative implicit optimization method. The neural scattering field uses a multi-layer perceptron to predict water parameters. The volume rendering module uses an underwater imaging model to simulate underwater image degradation.

[0132] Implementation effect:

[0133] The SeaThru-NeRF dataset and Neuralsea dataset are used as examples to evaluate the SP-SeaNeRF underwater 3D reconstruction model based on neural radiance fields. The information in the image of each scene is coherent and rich in details, which can well evaluate the SP-SeaNeRF model. Each scene also contains the camera pose obtained by COLMAP.

[0134] The SeaThru-NeRF dataset contains underwater scenes from four different sea areas: Curasao, Panama, IUI3-RedSea, and JG-RedSea, with 21, 18, 29, and 20 images, respectively. The Curasao and Panama scenes include an 18-color DGK color chart, which can be used to quantitatively evaluate the color correction of underwater images. The Neuralsea dataset contains only one underwater scene, with a total of 203 images. These images were acquired by an underwater robot in a laboratory tank using a stable LED light source, and all images were captured from a bird's-eye view.

[0135] The image resolutions in the SeaThru-NeRF and Neuralsea datasets were downsampled by a factor of 4 and 16, respectively, to meet the performance limits of the experimental equipment. 12.5% of the images were used for testing, and each scene was trained for 50,000 iterations, which takes approximately one hour on a single Nvidia GeForce RTX 3080 GPU.

[0136] The SP-SeaNeRF model is implemented in the JAX framework and uses the Adam optimizer with a batch size of 4096 rays. The learning rate decays from 2×10⁻³ to 2×10⁻⁵ and the number of sample points on the rays is 65. The dimension of the position encoding of the ray origin coordinates and direction in the neural sample field is 10, while the dimension of the position encoding of the direction in the neural radiation field is 4.

[0137] In the neural sample filtering module, the filter width is 3, the step size is 2, and the threshold is 10 -5 The length of the illumination embedding vector needs to be set according to the level of illumination clutter in the scene. In the five scenes of the two datasets used, the lengths of the illumination embedding vectors for Curasao, Panama, IUI3-RedSea, JG-RedSea, and Neuralsea are set to 16, 4, 4, 8, and 0, respectively.

[0138] Speed

[0139] Using a coarse-to-fine sampling method, the output layers of the coarse MLP and fine MLP are set to 128 and 32 respectively; the number of neural sample filtering parameters is 1174094, and the number of coarse-to-fine model parameters is 854542. Although the method of this application has more parameters, it is 46% faster, with the number of rays rendered per second being 50500 and 34500 respectively.

[0140] Quality

[0141] We selected SeaThru-NeRF, Mip-NeRF 360, and 3DGS as comparison methods, all of which are widely recognized cutting-edge technologies. On the SeaThru-NeRF dataset, our method improved the PSNR by 1.8856, 4.9279, and 5.9833, respectively, compared to the comparison methods. The improvement of our method on the Curasao scene was particularly significant. Figure 9As can be seen in the figure, the images rendered by the comparison methods in this scene exhibit numerous artifacts, which are caused by dynamic lighting. A similar phenomenon occurs in the JG-RedSea scene. However, the lighting in Panama and IUI3-RedSea is relatively stable, resulting in similar results compared to the comparison methods. On the Neuralsea dataset, our method still achieves the best performance, achieving PSNR improvements of 4.5456, 2.1810, and 10.2150, respectively, compared to the three comparison methods. Figure 10 As can be seen in the figure, the method of the present application can render more detailed textures of stones and starfish.

[0142] Image enhancement

[0143] UIESS and TUDA were selected as comparison methods, both of which are classic underwater image enhancement algorithms. The color cards in the scatter-free image rendered by the method of this application and the image enhanced by the comparison method were cut into separate color blocks one by one according to color, and then the mean of the color blocks was taken to draw the corresponding color cards. Finally, the color cards of the three models were compared with the standard color card and the PSNR was calculated. The method used in this application is based on a physical imaging model, so distant scattering can also be well removed, and color restoration is better, with the highest PSNR. Figure 11 Demonstrated specific effects.

[0144] Those skilled in the art will understand that all or part of the steps of implementing the above embodiments may be accomplished by hardware, or by programs instructing related hardware to accomplish the steps. The programs may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0145] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An underwater scene reconstruction device based on neural radiation field, characterized in that: The device constructs an SP-SeaNeRF underwater three-dimensional reconstruction model based on neural radiation field, and the SP-SeaNeRF underwater three-dimensional reconstruction model based on neural radiation field includes a neural sample filtering module, a neural radiation field module, a neural scattering field module and a volume rendering module; The neural sample filtering module is used to obtain sampling points distributed on the surface of the object on the ray for each pixel of the input underwater image; The neural radiation field module is used to input the sampling point position and illumination embedding vector of the neural sample filtering module to output the color and density of the sampling point; The neural scattered field module is used to input the position-encoded ray directions to output water parameters; The volume rendering module is used to render the color parameters and density parameters output by the neural radiation field module and the water parameters output by the neural scattering field module into pixel colors; The illumination embedding vector is a real-valued vector assigned to each input image. , whose length is ; The water body parameters are light shielding coefficient, forward transmission attenuation coefficient and backscattering attenuation coefficient.

2. The device according to claim 1, characterized in that The neural sample filtering module includes: a multi-layer perceptron and a filter, the multi-layer perceptron is used to map rays into a set of sampling points, and the filter is used to perform sparse sampling on dense sampling points.

3. The device according to claim 1, characterized in that The neural radiation field module includes: two multi-layer perceptrons and a light embedding vector.

4. The device according to claim 3, characterized in that The illumination embedding vector is specifically expressed as follows: Among them, dynamic lighting Represented as input The collection of transient lighting in the image , and then take the average of these transient lighting as the static lighting of the scene , each image Transient lighting Represented as static lighting And the corresponding lighting bias The sum of The length is The real-valued illumination embedding vector is connected to the second multi-layer perceptron network of the neural radiance field module Jointly optimizing,learning illumination biases in images.

5. The device according to claim 1, characterized in that The neural scattering field module includes: a multi-layer perceptron.

6. The device according to claim 1, characterized in that The volume rendering module includes: a rendering equation for underwater images.

7. The device according to claim 6, characterized in that The rendering equation of the underwater image is as follows: Among them, the ray will get through the neural sample filtering module sampling points, the ray is divided into intervals, each interval The length is , , and Respectively represent intervals The positive transmission color, scattered color and true color of the object; Indicates that light is transmitted to The probability of being at Indicates that light is transmitted to the water without any objects. The probability of Indicates that the light is in the interval The probability of being blocked; and Represents the shading coefficient, which is accumulated along the ray for each interval and , you can get the color of the underwater image; in, The sampling point color output by the neural radiance field module; The sampling point density output by the neural radiation field module; is the positive transmission attenuation coefficient; is the backscatter attenuation coefficient; is the backscattered color at infinity; Multiply the object's true color J by the attenuation Then add , get underwater image .

8. A method for underwater scene reconstruction based on neural radiation field, characterized in that: The method is applied to the underwater scene reconstruction device based on neural radiation field according to claim 1, and the method comprises: Each pixel of the input underwater image is filtered through the neural sample filtering module to obtain sampling points distributed on the surface of the object; Input the sampling point position and illumination embedding vector of the neural sample filtering module through the neural radiation field module to output the color and density of the sampling point; Based on the ray coding information, the neural scattered field module inputs the position-encoded ray direction to output water parameters; Based on the color parameters and density parameters output by the neural radiation field module and the water body parameters output by the neural scattered field module, the volume rendering module is used to render the pixels into colors; The illumination embedding vector is a real-valued vector assigned to each input image. , whose length is ; The water body parameters are light shielding coefficient, forward transmission attenuation coefficient and backscattering attenuation coefficient.

9. The method according to claim 8, characterized in that The color and density parameters output by the neural radiance field module and the water parameters output by the neural scattering field module are rendered into pixel colors through the volume rendering module, which also includes: Reconstruction loss of RawNeRF: in, is the predicted color, is the true value, Indicates stopping the gradient, .

10. The method according to claim 8, characterized in that The output color and density parameters and the water parameters output by the neural scattered field module are rendered into pixel colors through the volume rendering module, which also includes: Sample loss function: in, is the position of the sampling point on the ray, For interval The weight of and are the mean and variance of the sampling point locations respectively; For depth estimation, taken from the maximum weight interval midpoint; Represents the expected sampling point variance, introducing hyperparameters and This allows sampling points to be relatively densely distributed in scenes of different scales.

Images