Underwater scene three-dimensional reconstruction method, model training method, equipment and storage medium

Abstract

The embodiment of the invention provides an underwater scene three-dimensional reconstruction method, a model training method, equipment and a storage medium, and belongs to the technical field of artificial intelligence. According to the method, an image data set of an underwater scene is obtained, the image data set comprises a plurality of two-dimensional images describing the underwater scene, each two-dimensional image is marked with shooting position information and shooting direction information, ray sampling is carried out according to the image data set by adopting a multi-layer voxel sampling method, ray sampling information is obtained, and the ray sampling information is calculated. And predicting ray sampling information by using the initialized underwater scene reconstruction model to obtain an image pixel prediction value corresponding to a ray direction, and updating parameters of the underwater scene reconstruction model according to the image pixel prediction value and an image pixel true value in the image data set. According to the method, the model training efficiency can be improved, and the operation speed and the reconstruction accuracy during underwater scene reconstruction are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for three-dimensional reconstruction of underwater scenes, a model training method, a device and a storage medium. Background Art

[0002] The underwater environment covers more than 70% of the earth's surface and is rich in resources, which is crucial to ecological protection, resource development and historical heritage protection. Underwater scene 3D reconstruction technology plays a key role in underwater emergency management and rescue and salvage. It can provide accurate terrain and structure information in complex underwater environments, help locate lost objects and divers, and improve the accuracy of rescue plans. This technology also supports path planning for underwater robots and drones, improves the level of automation, reduces divers' risks, and provides data support for accident analysis and environmental monitoring, which greatly improves the speed and efficiency of underwater emergency response overall.

[0003] Early underwater detection mainly relied on sonar, which provided basic information, but had limited resolution and accuracy, making it difficult to meet the needs of water safety and emergency rescue. With the development of underwater photography, image processing is used for three-dimensional reconstruction, generating three-dimensional models through image stitching and algorithm processing to provide environmental support for rescue. However, complex underwater light and noise interference pose challenges to image processing, especially in precise positioning.

[0004] Advances in deep learning and multi-sensor fusion technologies have enabled modern algorithms to better cope with noise and distortion in underwater images, significantly improving the accuracy and efficiency of reconstruction. In related technologies, the NeRF (neural radiance field) method can be used for three-dimensional reconstruction. This method uses implicit neural networks to generate high-quality synthetic images from multi-view images and is widely used in the field of virtual reality. However, in underwater scenes, the application of the NeRF method requires consideration of the computational overhead caused by light scattering, absorption, and color distortion, which affects the efficiency of model training. At the same time, it is difficult to reconstruct underwater scenes at the same time. The speed of operation and the accuracy of reconstruction are difficult to reconstruct. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a three-dimensional reconstruction method, model training method, device and storage medium for underwater scenes, aiming to improve the efficiency of model training and improve the calculation speed and reconstruction accuracy during underwater scene reconstruction.

[0006] To achieve the above purpose, an embodiment of the present application provides a method for training a 3D reconstruction model of an underwater scene, comprising the following steps:

[0007] Acquire an image data set of an underwater scene, wherein the image data set includes a plurality of two-dimensional images describing the underwater scene, and each two-dimensional image is marked with shooting position information and shooting direction information;

[0008] Using a multi-layer voxel sampling method, ray sampling is performed according to the image data set to obtain ray sampling information, wherein the ray sampling information includes coordinate parameters of sampling data points on the ray and ray direction parameters;

[0009] The initialized underwater scene reconstruction model is used to predict the ray sampling information to obtain the image pixel prediction value corresponding to the ray direction;

[0010] According to the image pixel prediction values ​​and the image pixel true values ​​in the image data set, the parameters of the underwater scene reconstruction model are updated to obtain a trained underwater scene three-dimensional reconstruction model.

[0011] In some embodiments, the step of acquiring an image dataset of an underwater scene comprises the following steps:

[0012] Capture two-dimensional images from multiple perspectives through cameras;

[0013] Performing feature extraction and feature matching on the collected multiple two-dimensional images to obtain an essential matrix for describing the geometric relationship between the two-dimensional images;

[0014] The shooting position information and shooting direction information of each two-dimensional image are determined according to the essential matrix.

[0015] In some embodiments, the multi-layer voxel sampling method is used to perform ray sampling according to the image data set to obtain ray sampling information, including the following steps:

[0016] Determine, according to shooting position information, shooting direction information and image size of the two-dimensional image in the image data set, ray parameters between each pixel point in the two-dimensional image and the camera origin, and convert the ray parameters into a world coordinate system for representation, wherein the ray parameters include a ray direction and a ray origin;

[0017] A multi-layer voxel sampling method is adopted to perform sampling on each ray represented by the ray parameters to obtain ray sampling information.

[0018] In some embodiments, the multi-layer voxel sampling method is used to sample each ray represented by the ray parameter to obtain ray sampling information, including the following steps:

[0019] Performing sparse sampling on each ray represented by the ray parameter to obtain a plurality of first-level sampling points;

[0020] Determine a probability distribution function of the ray based on sparse sampling according to the plurality of sampling points of the first level;

[0021] determining a target area on the ray according to the probability distribution function;

[0022] Fine sampling is performed on the target area on the ray to obtain a plurality of second-level sampling points;

[0023] Ray sampling information is determined according to a plurality of the first-level sampling points and a plurality of the second-level sampling points.

[0024] In some embodiments, the method of using the initialized underwater scene reconstruction model to predict the ray sampling information to obtain the image pixel prediction value corresponding to the ray direction includes the following steps:

[0025] Encoding the ray sampling information in the form of sine and cosine functions to obtain first encoding information in a high-dimensional space;

[0026] Normalizing each sampling data point of the first coded information to obtain second coded information;

[0027] The second coded information is input into the underwater scene reconstruction model to obtain the image pixel prediction value corresponding to the ray direction.

[0028] In some embodiments, updating the parameters of the underwater scene reconstruction model according to the image pixel prediction values ​​and the true values ​​of the image pixels in the image data set to obtain a trained underwater scene 3D reconstruction model comprises the following steps:

[0029] Determining a loss value according to the image pixel prediction value and the image pixel true value in the image data set;

[0030] Updating the parameters of the underwater scene reconstruction model according to the loss value to obtain a trained underwater scene three-dimensional reconstruction model;

[0031] The loss value is calculated by a loss function, and the loss function is expressed as follows:

[0032]

[0033] in, is a ray set, C(r) is the true value of the image pixel on ray r, C c (r) is the image pixel prediction value based on sparse sampling points on the r ray, C f (r) is the predicted value of the image pixel based on dense sampling points on the r ray.

[0034] To achieve the above purpose, another aspect of the embodiment of the present application provides a method for 3D reconstruction of an underwater scene, comprising the following steps:

[0035] Acquiring target viewing angle information, wherein the target viewing angle information includes a target observation position and a target observation direction;

[0036] Using a multi-layer voxel sampling method, determining ray sampling information on a corresponding ray according to the target viewing angle information;

[0037] Inputting the ray sampling information into a three-dimensional reconstruction model of an underwater scene to obtain a rendered image at a target viewing angle;

[0038] The underwater scene three-dimensional reconstruction model is obtained by training using the underwater scene three-dimensional reconstruction model training method described in the above embodiment.

[0039] To achieve the above purpose, another aspect of the embodiment of the present application provides a training system for a 3D reconstruction model of an underwater scene, comprising:

[0040] A first module is used to obtain an image data set of an underwater scene, wherein the image data set includes a plurality of two-dimensional images describing the underwater scene, and each two-dimensional image is marked with shooting position information and shooting direction information;

[0041] The second module is used to adopt a multi-layer voxel sampling method to perform ray sampling according to the image data set to obtain ray sampling information, wherein the ray sampling information includes coordinate parameters of the sampling data points on the ray and ray direction parameters;

[0042] The third module is used to predict the ray sampling information by using the initialized underwater scene reconstruction model to obtain the image pixel prediction value corresponding to the ray direction;

[0043] The fourth module is used to update the parameters of the underwater scene reconstruction model according to the image pixel prediction values ​​and the image pixel true values ​​in the image data set to obtain a trained underwater scene three-dimensional reconstruction model.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program implements the method described in the above embodiment when executed by the processor.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the above embodiment.

[0046] The underwater scene three-dimensional reconstruction method, model training method, device and storage medium proposed in the present application first obtain an image data set of the underwater scene, the image data set includes multiple two-dimensional images describing the underwater scene, each two-dimensional image is marked with shooting position information and shooting direction information, and then a multi-layer voxel sampling method is used to perform ray sampling according to the image data set to obtain ray sampling information, the ray sampling information includes the coordinate parameters of the sampled data points on the ray and the ray direction parameters, and then the ray sampling information is predicted using the initialized underwater scene reconstruction model to obtain the image pixel prediction value corresponding to the ray direction, and the parameters of the underwater scene reconstruction model are updated according to the image pixel prediction value and the image pixel true value in the image data set. The present application adopts a multi-level voxel sampling method to sample on the ray to obtain more effective sampling data points, reduce the computational overhead caused by the subsequent sampling data points with sparse underwater scene content, thereby improving the model training efficiency, and further improving the operation speed and reconstruction accuracy during underwater scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a method for training a three-dimensional reconstruction model of an underwater scene provided in an embodiment of the present application;

[0048] Figure 2 It is a schematic diagram of the MLP neural network structure provided in the embodiment of the present application;

[0049] Figure 3 is a flow chart of a method for three-dimensional reconstruction of an underwater scene provided in an embodiment of the present application;

[0050] Figure 4 It is a conceptual schematic diagram of the three-dimensional reconstruction of an underwater scene provided in an embodiment of the present application;

[0051] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that, although the functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] First, some nouns involved in this application are analyzed:

[0056] Three-dimensional scene reconstruction is to restore the three-dimensional geometric structure of a scene from two-dimensional images or video data so as to observe and interact with the scene in a virtual environment.

[0057] COLMAP is an open source computer vision software with a wide range of applications in the field of 3D reconstruction. COLMAP is mainly used for 3D reconstruction from a series of 2D images, and can recover the geometric structure (point cloud) of the 3D scene and the camera pose corresponding to each image. COLMAP's technologies include feature detection and matching and incremental SfM (Structure-from-Motion). COLMAP can automatically detect key points from the input image, find matching key points between different images, establish connections between images through feature matching relationships, and gradually estimate the relative camera poses of all images to form a sparse point cloud.

[0058] Based on this, the embodiments of the present application provide a method for three-dimensional reconstruction of underwater scenes, a model training method, a device and a storage medium, aiming to improve the efficiency of model training and improve the computing speed and reconstruction accuracy during underwater scene reconstruction.

[0059] The underwater scene three-dimensional reconstruction method, model training method, device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the underwater scene three-dimensional reconstruction model training method and the underwater scene three-dimensional reconstruction method in the embodiments of the present application are described.

[0060] The underwater scene three-dimensional reconstruction model training method and underwater scene three-dimensional reconstruction method provided in the embodiment of the present application relate to the field of artificial intelligence technology. The underwater scene three-dimensional reconstruction model training method or underwater scene three-dimensional reconstruction method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the underwater scene three-dimensional reconstruction model training method or the underwater scene three-dimensional reconstruction method, etc., but is not limited to the above forms.

[0061] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0062] Figure 1 is an optional flowchart of the underwater scene 3D reconstruction model training method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0063] Step S101, obtaining an image dataset of an underwater scene, wherein the image dataset includes a plurality of two-dimensional images describing the underwater scene, and each two-dimensional image is marked with shooting position information and shooting direction information;

[0064] Step S102, using a multi-layer voxel sampling method to perform ray sampling according to the image data set to obtain ray sampling information, wherein the ray sampling information includes coordinate parameters of the sampled data points on the ray and ray direction parameters;

[0065] Step S103, using the initialized underwater scene reconstruction model to predict the ray sampling information, and obtain the image pixel prediction value corresponding to the ray direction;

[0066] Step S104, updating the parameters of the underwater scene reconstruction model according to the image pixel prediction values ​​and the image pixel true values ​​in the image data set, to obtain a trained underwater scene three-dimensional reconstruction model.

[0067] In step S101 of some embodiments, a camera can be used to shoot a video in an underwater scene, and then a plurality of two-dimensional images describing the underwater scene can be obtained by extracting frames from the video, and all the extracted two-dimensional images form an image data set. Each two-dimensional image in the image data set is marked with shooting location information and shooting direction information, and the information can be obtained based on a deep learning method, an object geometric relationship method, or an image metadata method. Taking the image metadata method as an example, when shooting, the camera will embed the camera model, focal length, aperture, shutter speed and other information into the EXIF ​​metadata of the image. By reading the EXIF ​​information, some of the camera's shooting parameters can be obtained. For cameras that support GPS functions, the EXIF ​​information also contains the latitude and longitude information of the shooting location. Specifically, this embodiment can also be obtained by using the three-dimensional reconstruction software COLMAP analysis.

[0068] In some embodiments, step S101 may include but is not limited to steps S201 to S203:

[0069] Step S201, collecting two-dimensional images of multiple perspectives through a camera;

[0070] Step S202, performing feature extraction and feature matching on the collected multiple two-dimensional images to obtain an essential matrix for describing the geometric relationship between the two-dimensional images;

[0071] Step S203: determining the shooting position information and shooting direction information of each two-dimensional image according to the essential matrix.

[0072] In this embodiment, an underwater camera is used to shoot the underwater scene to be reconstructed to obtain clear images at different viewing angles. If the video data is shot, it is extracted frame by frame as image data. The three-dimensional reconstruction software COLMAP is used to perform sparse reconstruction on the two-dimensional image to extract the shooting position information and shooting direction information of each image when it was shot. The shooting position information and shooting direction information are further converted into LLFF format through a script. The shooting position information and shooting direction information of the image recognized by the COLMAP software are as follows:

[0073] Create a project in COLMAP;

[0074] Perform feature extraction and matching on images, extract local features of images and match features between different images. Feature extraction is local features of images, usually key points and corresponding descriptors. These features are mainly used for matching between images for subsequent 3D reconstruction steps;

[0075] Determine the correspondence between the images based on the matching results, and estimate the position information and shooting direction information of the shooting camera corresponding to the two-dimensional image in the world coordinate system;

[0076] Perform sparse reconstruction on the image to generate a 3D point cloud model and the position information data and shooting direction information data of the shooting camera in the world coordinate system;

[0077] Save the camera's position information, shooting direction information data, and sparse point cloud data.

[0078] In one example, an underwater camera can be used to shoot the target scene from multiple angles with different viewing angles. If the shooting data is video data, the video data can be converted into a two-dimensional image by extracting frames. This example can use video shooting. The images extracted after video shooting are closely related, which facilitates feature extraction between subsequent images. To avoid redundant data or insufficient images, 5 images can be exported from the video per second. During the shooting process, a camera with higher resolution is used, the scene environment is clear, and dynamic objects are avoided to improve the reconstruction effect.

[0079] Use the 3D reconstruction software COLMAP to perform sparse reconstruction and extract the image pose data

[0080] Use COLMAP to extract the camera pose of the image (including shooting position information and shooting direction information). Specifically, create a new project and save it as database.db file. Select "Featureextraction" under "Processing" in the menu bar, set the camera model to SIMPLE_PINHOLE, then click "Extract" to extract features and feature matching, click "Start reconstruction" for sparse reconstruction, export the 3D point cloud model and camera pose through "Export model", and save them to the spare / 0 folder.

[0081] In COLMAP, the principle of acquiring the position and shooting direction data of the shooting camera mainly relies on the principle of multi-view geometry. The camera position and shooting direction corresponding to each image are estimated through feature matching, geometric verification, and optimization steps. Specifically, COLMAP will extract and match the images. Through feature matching, COLMAP can identify the corresponding feature points of the same object in different images. For each pair of matched images, COLMAP can calculate the fundamental matrix or essential matrix through the matched feature points. The essential matrix describes the geometric relationship between the two images, especially the position relationship and the perspective change relationship of the cameras. The essential matrix is ​​used for cameras with known intrinsic parameters, while the fundamental matrix is ​​used for uncalibrated cameras. Using the calculated essential matrix, COLMAP can preliminarily estimate the relative posture and position of the camera (i.e. the external parameters of the camera). This process can obtain the rotation and translation relationship between the cameras by solving the essential matrix. Through the corresponding feature points in multiple perspectives (multiple images), COLMAP calculates the position of these points in three-dimensional space through triangulation. According to the position of these three-dimensional points, the camera position and shooting direction estimation can be further optimized. After the initial estimation, COLMAP uses bundle adjustment to globally optimize the position and shooting direction of all cameras and the position of 3D points. Bundle adjustment adjusts the position of the camera, shooting and the coordinates of the 3D points by minimizing the projection error in all images, thereby improving the accuracy of the reconstruction of the entire scene. The optimized camera pose data reflects the specific position and direction of the camera when each image was taken. COLMAP can generate a complete camera trajectory, and the shooting camera position and shooting direction parameters corresponding to this trajectory are data obtained using the COLMAP software.

[0082] Further, the pose data is converted to LLFF format. Download and configure the LLFF script tool, and run imgs2poses.py to convert the format. After completion, delete the images that are not matched to the camera position and shooting direction, recalculate, and finally generate the poses_bounds.npy file in the working directory, that is, the camera position and shooting direction data. This embodiment uses the LLFF script to perform a series of operations such as normalization and coordinate system conversion on the camera position and shooting direction data, translate the scene center to the origin and scale it to a scale suitable for underwater scene reconstruction model training, and convert the right-handed coordinate system generated by COLMAP to the coordinate system used by the NeRF-based underwater scene reconstruction model to ensure that the camera's front direction is consistent (usually pointing to the -Z axis). In addition, LLFF can recalculate the focal length and field of view to adapt to different image resolutions, so that the sampling requirements of NeRF in the camera are matched. The above adjustments can make the NeRF model easier to train and improve the rendering effect.

[0083] In step S102 of some embodiments, a multi-layer voxel sampling method is used to perform ray sampling according to the image data set to obtain ray sampling information, and the ray sampling information includes coordinate parameters of the sampled data points on the rays and ray direction parameters. This embodiment applies a multi-layer voxel sampling method when sampling rays from the camera optical center to the image pixels. The multi-layer voxel sampling method divides and samples voxels (i.e., data points) at different levels or scales to obtain a richer point cloud representation. When sampling rays, this embodiment first performs large-scale sparse sampling, then identifies content-rich areas based on the sparse sampling, and then performs precise sampling at the scale of the area to obtain ray sampling information. Three-dimensional reconstruction based on ray sampling information can improve reconstruction efficiency and accuracy.

[0084] In some embodiments, step S102 may include but is not limited to steps S301 to S302:

[0085] Step S301, determining ray parameters between each pixel point in the two-dimensional image and the camera origin according to shooting position information, shooting direction information and image size of the two-dimensional image in the image data set, and converting the ray parameters to a world coordinate system for representation, wherein the ray parameters include a ray direction and a ray origin;

[0086] Step S302 : adopt a multi-layer voxel sampling method to sample each ray represented by the ray parameter to obtain ray sampling information.

[0087] In step S301 of some embodiments, the two-dimensional images in the image data set and the shooting position information, shooting direction information and image size corresponding to each two-dimensional image, that is, the length and width of the image are read. According to the shooting position information, shooting direction information and image size, the ray direction and ray origin between each pixel in the image and the camera origin are calculated, and then the ray parameters such as the ray direction and ray origin are converted from the camera coordinate system to the world coordinate system. When sampling data points on the ray, a bounding box with a sampling range of usually 2-6 meters is set.

[0088] In step S302 of some embodiments, step S302 may include but is not limited to steps S401 to S405:

[0089] Step S401, performing sparse sampling on each ray represented by the ray parameter to obtain a plurality of first-level sampling points;

[0090] Step S402, determining a probability distribution function of rays based on sparse sampling according to a plurality of first-level sampling points;

[0091] Step S403, determining the target area on the ray according to the probability distribution function;

[0092] Step S404, performing fine sampling on the target area on the ray to obtain a plurality of second-level sampling points;

[0093] Step S405 , determining ray sampling information according to the plurality of first-level sampling points and the plurality of second-level sampling points.

[0094] In this embodiment, NeRF's rendering strategy is based on ray sampling, and the color of each ray is obtained by weighted summation of the sampled data points on the ray. In order to reduce the computational overhead of sampling data points in the air part or the sampled data points in the occluded area, this embodiment adopts a "coarse to fine" multi-level voxel sampling method. Specifically, the sparse network inputs the relatively sparse first-level sampling points, and the dense network performs secondary fine sampling based on the coarse sampling data points and the output weights of the sparse network to obtain the second-level sampling points.

[0095] First, the stratified sampling method is used to uniformly collect the sparse P c The first-level sampling points are used to calculate the rendering results of the sparse network, as follows:

[0096] w i =T i ·(1-exp(-σ i ε i ));

[0097] in, For w i Normalize as follows:

[0098]

[0099] In the above formula, ε i is the distance between two adjacent sampling data points, σ i is the output weight value of the sparse network (that is, the volume density of the sampled data points). The normalized w i is the probability distribution function along the ray direction. Through the sparsely sampled probability distribution function, we can roughly get the distribution of objects in the ray direction, for example, which areas have denser object distribution.

[0100] Subsequently, based on the probability distribution function obtained by sparse sampling, Pf high-density second-level sampling points are sampled, so that more sampling data points can be obtained from the target area containing more visible content, so as to obtain more sampling data points in P c +P f The rendering result of the dense network is calculated on a set of sampled data points.

[0101] For example, take P c =64,P f=32. According to the ray direction parameters and the sampling range on the ray, 64 sparse sampling data points are obtained on each ray through layered sampling. Subsequently, 32 high-density sampling data points are obtained through fine sampling through each round of network output results. The sparse sampling data points and the high-density sampling data points are combined to obtain the coordinate parameters of the sampling data points on each ray. At the same time, the ray direction parameters are also used as observation angle parameters and will be input in the subsequent network.

[0102] In step S103 of some embodiments, the underwater scene reconstruction model is a neural network. For example, the underwater scene reconstruction model can adopt an MLP neural network structure, such as Figure 2 As shown in the figure, the MLP neural network used in the underwater scene reconstruction model has a total of 10 layers, the first 9 layers are 256×256 fully connected layers, and the last layer is 256×128. The underwater scene reconstruction model is used to predict the image pixel value (ie, RGB value) and weight value of the sampling data point based on the ray direction (ie, the observation angle direction) and the coordinate parameters of the sampling data point on the ray.

[0103] In some embodiments, step S103 may also include but is not limited to steps S501 to S503:

[0104] Step S501, encoding the ray sampling information in the form of sine and cosine functions to obtain first encoding information in a high-dimensional space;

[0105] Step S502, normalizing each sampling data point of the first coded information to obtain second coded information;

[0106] Step S503: input the second coding information into the underwater scene reconstruction model to obtain the image pixel prediction value corresponding to the ray direction.

[0107] In this embodiment, the input parameters of the model are the coordinates of the sampling data points and the observation angle information. Since the neural network has a weak learning ability for high-frequency information, directly inputting the coordinates of the sampling data points and the observation angle information will result in a low resolution of the rendering effect. By encoding the position information, the input can be mapped to high frequency, thereby improving the learning ability and rendering effect of the network. Therefore, before inputting the ray sampling information into the model, this embodiment needs to encode the ray sampling information. Specifically, the encoding uses the form of sine and cosine functions to map the input ray sampling information p to a high-dimensional space, as follows:

[0108] γ(p)=(sin(2 0 πp), cos(2 0 πp),...,sin(2 L-1 πp), coS(2 L-1 πp));

[0109] Then, the coordinate parameters of the sampled data points and the observation angle information are normalized to between [-1, 1]. For the 3D position, γ(x) is set to L=10; the viewing angle information γ(d) is set to L=4.

[0110] The encoded and normalized ray sampling information is input into the model for prediction. For example, please continue to refer to Figure 2 In the network structure of the model, the coordinate parameters of the sampled data points are input in the first and fifth layers respectively, the ninth layer outputs the volume density (i.e., the weight value σ), the observation perspective information is input in the ninth layer, and the last layer outputs the RGB pixel value corresponding to the sampled data point. In this way, the voxel density depends only on the spatial position, while the color depends on both the spatial position and the observation perspective. Through perspective-based color prediction, the neural network can simulate the lighting effect of the reconstructed object at different perspectives.

[0111] After the RGB value and weight value σ of the sampled data point are predicted by the model, the following rendering formula is used to calculate the RGB prediction value of each ray, that is, the image pixel prediction value, as follows:

[0112]

[0113] Among them, ε i is the distance between two adjacent sampling data points, c i and σ i is the RGB value and weight value of the sampled data point predicted by the model, where T(t) is rewritten as:

[0114]

[0115] In this embodiment, all the sampling data points (c i , σ i ) The method of summing the ray rendering color is differentiable and can be simplified to the traditional transparency blending algorithm, where the alpha value α i =1-exp(-σ i ε i ).

[0116] In step S104 of some embodiments, the parameters of the underwater scene reconstruction model are updated according to the image pixel prediction values ​​and the image pixel true values ​​of the images in the image data set to obtain a trained underwater scene three-dimensional reconstruction model.

[0117] In some embodiments, step S104 includes but is not limited to steps S601 to S602:

[0118] Step S601, determining a loss value according to the image pixel prediction value and the image pixel true value in the image data set;

[0119] Step S602: update the parameters of the underwater scene reconstruction model according to the loss value to obtain a trained underwater scene 3D reconstruction model.

[0120] In this embodiment, the loss function is the sum of the squared errors between each ray and the true value on the sparse network and the dense network. The loss function is expressed as follows:

[0121]

[0122] in, is the set of all rays in the two sets of sparse sampling and dense sampling samples, C(r) is the true value of the image pixel on the r ray, C c (r) is the volume rendering color of the sparse network on the r ray (i.e., the image pixel prediction value based on the sparse sampling points), C f (r) is the volume rendering color of the dense network on the r ray (i.e., the image pixel prediction value based on the dense sampling points).

[0123] The parameters of the underwater scene reconstruction model are continuously updated according to the loss value until the loss value is less than a certain expected value, thereby obtaining a trained underwater scene 3D reconstruction model.

[0124] Please refer to Figure 3 The embodiment of the present application also provides a method for 3D reconstruction of an underwater scene. The method for 3D reconstruction of an underwater scene in the embodiment of the present application may include but is not limited to steps S701 to S703:

[0125] Step S701, obtaining target viewing angle information, wherein the target viewing angle information includes a target observation position and a target observation direction;

[0126] Step S702, using a multi-layer voxel sampling method to determine ray sampling information on a corresponding ray according to target viewing angle information;

[0127] Step S703: inputting the ray sampling information into the 3D reconstruction model of the underwater scene to obtain a rendering image under the target viewing angle.

[0128] In this embodiment, the underwater scene 3D reconstruction model can be applied to generate an image under the new observation angle by inputting the position and shooting direction information of the new observation angle, and synthesize the images corresponding to the observation angle into a video file according to the observation order to complete the scene reconstruction. Figure 4, by using COLMAP software to extract the shooting posture and LLFF conversion of the collected images, an image data set is obtained, and the sampling point-related data in the image data set is obtained to repeatedly train the MLP network to obtain a trained underwater scene reconstruction model. The corresponding observation position and observation direction information are obtained along the new viewing path, and the sampling data points are obtained according to the above-mentioned hierarchical voxel sampling method. The coordinate parameters and observation methods of the sampling data points are processed by the above-mentioned encoding and normalization methods to obtain input data, and the input data is input into the trained model to obtain the RGB value and weight of each sampling data point. The RGB value of the pixel corresponding to each ray is weightedly calculated by the volume rendering formula to generate an image under a new viewing angle. Through a continuous viewing path, dynamic videos of underwater scenes can also be reconstructed.

[0129] The present application also provides a system for training a 3D reconstruction model of an underwater scene, including:

[0130] The first module is used to obtain an image data set of an underwater scene, wherein the image data set includes a plurality of two-dimensional images describing the underwater scene, and each two-dimensional image is marked with shooting position information and shooting direction information;

[0131] The second module is used to adopt a multi-layer voxel sampling method to perform ray sampling according to the image data set to obtain ray sampling information, wherein the ray sampling information includes coordinate parameters of the sampling data points on the ray and ray direction parameters;

[0132] The third module is used to predict the ray sampling information using the initialized underwater scene reconstruction model to obtain the image pixel prediction value corresponding to the ray direction;

[0133] The fourth module is used to update the parameters of the underwater scene reconstruction model according to the image pixel prediction values ​​and the true values ​​of the image pixels in the image data set to obtain a trained underwater scene three-dimensional reconstruction model.

[0134] It can be understood that the contents of the above-mentioned underwater scene three-dimensional reconstruction model training method embodiment are all applicable to the present system embodiment, the functions specifically implemented by the present system embodiment are the same as those in the above-mentioned underwater scene three-dimensional reconstruction model training method embodiment, and the beneficial effects achieved are also the same as the beneficial effects achieved by the above-mentioned underwater scene three-dimensional reconstruction model training method embodiment.

[0135] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and when the program is executed by the processor, the above-mentioned underwater scene 3D reconstruction model training method or underwater scene 3D reconstruction method is realized. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0136] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0137] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0138] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the underwater scene 3D reconstruction model training method or underwater scene 3D reconstruction method of the embodiments of this application;

[0139] Input / output interface 503, used to implement information input and output;

[0140] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0141] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0142] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via a bus 505 .

[0143] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned underwater scene three-dimensional reconstruction model training method or underwater scene three-dimensional reconstruction method.

[0144] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0146] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0147] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0148] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0149] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0152] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0155] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for training a 3D reconstruction model of an underwater scene, characterized in that: The following steps are involved: Acquire an image data set of an underwater scene, wherein the image data set includes a plurality of two-dimensional images describing the underwater scene, and each two-dimensional image is marked with shooting position information and shooting direction information; Using a multi-layer voxel sampling method, ray sampling is performed according to the image data set to obtain ray sampling information, wherein the ray sampling information includes coordinate parameters of sampling data points on the ray and ray direction parameters; The initialized underwater scene reconstruction model is used to predict the ray sampling information to obtain the image pixel prediction value corresponding to the ray direction; According to the image pixel prediction values ​​and the image pixel true values ​​in the image data set, the parameters of the underwater scene reconstruction model are updated to obtain a trained underwater scene three-dimensional reconstruction model.

2. The underwater scene 3D reconstruction model training method according to claim 1, characterized in that: The step of obtaining an image dataset of an underwater scene comprises the following steps: Capture two-dimensional images from multiple perspectives through cameras; Performing feature extraction and feature matching on the collected multiple two-dimensional images to obtain an essential matrix for describing the geometric relationship between the two-dimensional images; The shooting position information and shooting direction information of each two-dimensional image are determined according to the essential matrix.

3. The underwater scene 3D reconstruction model training method according to claim 1, characterized in that: The method of adopting a multi-layer voxel sampling method to perform ray sampling according to the image data set to obtain ray sampling information includes the following steps: Determine, according to shooting position information, shooting direction information and image size of the two-dimensional image in the image data set, ray parameters between each pixel point in the two-dimensional image and the camera origin, and convert the ray parameters into a world coordinate system for representation, wherein the ray parameters include a ray direction and a ray origin; A multi-layer voxel sampling method is adopted to perform sampling on each ray represented by the ray parameters to obtain ray sampling information.

4. The underwater scene 3D reconstruction model training method according to claim 3, characterized in that: The method of adopting a multi-layer voxel sampling method to sample each ray represented by the ray parameter to obtain ray sampling information includes the following steps: Performing sparse sampling on each ray represented by the ray parameter to obtain a plurality of first-level sampling points; Determine a probability distribution function of the ray based on sparse sampling according to the plurality of sampling points of the first level; determining a target area on the ray according to the probability distribution function; Fine sampling is performed on the target area on the ray to obtain a plurality of second-level sampling points; Ray sampling information is determined according to a plurality of the first-level sampling points and a plurality of the second-level sampling points.

5. The underwater scene 3D reconstruction model training method according to claim 4, characterized in that: The method of using the initialized underwater scene reconstruction model to predict the ray sampling information to obtain the image pixel prediction value corresponding to the ray direction includes the following steps: Encoding the ray sampling information in the form of sine and cosine functions to obtain first encoding information in a high-dimensional space; Normalizing each sampling data point of the first coded information to obtain second coded information; The second coded information is input into the underwater scene reconstruction model to obtain the image pixel prediction value corresponding to the ray direction.

6. The underwater scene 3D reconstruction model training method according to claim 5, characterized in that: The method of updating the parameters of the underwater scene reconstruction model according to the image pixel prediction values ​​and the image pixel true values ​​in the image data set to obtain a trained underwater scene 3D reconstruction model comprises the following steps: Determining a loss value according to the image pixel prediction value and the image pixel true value in the image data set; Updating the parameters of the underwater scene reconstruction model according to the loss value to obtain a trained underwater scene three-dimensional reconstruction model; The loss value is calculated by a loss function, and the loss function is expressed as follows: in, is a ray set, C(r) is the true value of the image pixel on ray r, C c (r) is the image pixel prediction value based on sparse sampling points on the r ray, C f (r) is the predicted value of the image pixel based on dense sampling points on the r ray.

7. A method for three-dimensional reconstruction of an underwater scene, characterized in that: The following steps are involved: Acquire target viewing angle information, wherein the target viewing angle information includes a target observation position and a target observation direction; Using a multi-layer voxel sampling method, determining ray sampling information on a corresponding ray according to the target viewing angle information; Inputting the ray sampling information into a three-dimensional reconstruction model of an underwater scene to obtain a rendered image at a target viewing angle; Wherein, the underwater scene three-dimensional reconstruction model is obtained by training the underwater scene three-dimensional reconstruction model training method according to any one of claims 1 to 6.

8. A 3D reconstruction model training system for underwater scenes, characterized in that: include: A first module is used to obtain an image data set of an underwater scene, wherein the image data set includes a plurality of two-dimensional images describing the underwater scene, and each two-dimensional image is marked with shooting position information and shooting direction information; The second module is used to adopt a multi-layer voxel sampling method to perform ray sampling according to the image data set to obtain ray sampling information, wherein the ray sampling information includes coordinate parameters of the sampling data points on the ray and ray direction parameters; The third module is used to predict the ray sampling information by using the initialized underwater scene reconstruction model to obtain the image pixel prediction value corresponding to the ray direction; The fourth module is used to update the parameters of the underwater scene reconstruction model according to the image pixel prediction values ​​and the image pixel true values ​​in the image data set to obtain a trained underwater scene three-dimensional reconstruction model.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the underwater scene three-dimensional reconstruction model training method as described in any one of claims 1 to 6 or the underwater scene three-dimensional reconstruction method as described in claim 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the underwater scene three-dimensional reconstruction model training method described in any one of claims 1 to 6 or the steps of the underwater scene three-dimensional reconstruction method described in claim 7.

Images