A method for reconstructing a laser depth map in a turbid water body based on deep learning

Abstract

The application provides a turbid water-based laser depth map reconstruction method based on deep learning, comprising the following steps: S1, placing observation targets in air medium and turbid water respectively, and collecting neural network training data; S2, based on the neural network training data, training a generative neural network based on a generative adversarial model to obtain a laser depth map reconstruction model; and S3, collecting laser radar point cloud data of the observation target, water body information and observation visual angle during measurement, inputting the laser depth map reconstruction model, and obtaining a reconstructed laser depth map. The application filters out the error influence caused by the multiple scattering effect of the water body through the deep learning method, thereby greatly improving the reconstruction accuracy of the underwater laser depth map.

Description

Technical Field

[0001] This invention relates to the field of lidar depth map calculation, and in particular to a method for reconstructing lidar depth maps in turbid water bodies based on deep learning. Background Technology

[0002] depth Figure 1 Depth maps are typically grayscale images, with pixel grayscale values ​​representing the distance of the real scene from the camera. Depth map information has important applications in computer vision, object detection, remote sensing mapping, and film and games. Most existing technologies use RGB-D systems to reconstruct depth maps, but in turbid underwater environments, the detectable depth of passive observation data is low, and the detectable distance is greatly limited. Compared to visible light cameras, lidar can quickly obtain high-resolution spatial information, thus being widely used in remote sensing mapping and autonomous driving. However, when depth map reconstruction based on lidar point clouds is applied to underwater detection, it is affected by turbid water, resulting in unavoidable laser multiple scattering and refraction effects, leading to significant deviations in the parameters of the reconstructed depth map. Summary of the Invention

[0003] The purpose of this invention is to provide a method for reconstructing laser depth maps in turbid water bodies based on deep learning. By combining visible light image data and lidar scattering cross-sectional data, and based on multimodal data fusion and deep learning technology, a framework for rapidly solving the lidar scattering interface of the target in the whole space is established, providing a feasible technical approach for obtaining the laser observation characteristics of the target in the whole space under limited measurement conditions.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A deep learning-based method for reconstructing laser depth maps in turbid water bodies, comprising the following steps:

[0006] S1. Place the observation target in air and turbid water respectively, and collect neural network training data;

[0007] S2. Based on the neural network training data, train the generative neural network based on the generative adversarial model to obtain the laser depth map reconstruction model;

[0008] S3. Collect lidar point cloud data of the observed target, as well as water body information and observation angle at the time of measurement, input the laser depth map reconstruction model, and obtain the reconstructed laser depth map.

[0009] Optionally, in step S1,

[0010] The neural network training data includes a matching dataset consisting of air medium depth maps and turbid water depth maps from the same observation perspective for different water bodies.

[0011] Optionally, step S1 includes:

[0012] S11. Place the observation target in the air medium, collect lidar point cloud data of the observation target from different observation angles, convert the lidar point cloud data of each observation angle into an air medium depth map, and store each air medium depth map and its corresponding observation angle.

[0013] S12. Place the observation target in turbid water, collect lidar point cloud data of the observation target from each observation angle in step S11, convert the lidar point cloud data from each observation angle into turbid water depth maps, and store each turbid water depth map and its corresponding water information and observation angle.

[0014] S13. Change the water body information of turbid water bodies;

[0015] S14. Repeat steps S11 to S13 until the distribution of the water body information covers the typical values ​​of the actual application scenario, then stop the loop and generate neural network training data.

[0016] Optionally, the conversion of the lidar point cloud data into a depth map is achieved by converting the time information in the lidar point cloud data into corresponding distance information based on the ToF method.

[0017] Optionally, the water body information is the absorption coefficient and scattering coefficient of turbid water.

[0018] Optionally, step S2 includes:

[0019] S21. The water body information and observation angle are used as condition vectors, the depth maps of each turbid water body are used as input images, and the depth maps of each air medium are used as target images, and then input into the generative neural network.

[0020] S22. The mean square error between the target image and the output image of the generative neural network is used as the target loss function;

[0021] S23. By synchronously training the generator and discriminator in the generative adversarial network, the target loss function is reduced until the error between the output image and the target image is less than a preset value, thereby obtaining the laser depth map reconstruction model.

[0022] Optionally, step S3 includes:

[0023] S31. For observation targets in turbid water bodies, conduct actual measurements and acquire measured lidar point cloud data using lidar.

[0024] S32. Convert the measured lidar point cloud data into a depth map with errors;

[0025] S33. Record the water body information and observation perspective at the time of the actual measurement, and convert them into a conditional vector;

[0026] S34. Input the depth map with errors and the condition vector into the laser depth map reconstruction model to generate a reconstructed laser depth map.

[0027] In summary, compared with existing technologies, the laser depth map reconstruction method for turbid water bodies based on deep learning provided by this invention has the following beneficial effects:

[0028] 1. This invention can effectively utilize the active detection capability of lidar point clouds and filter out the error caused by the multiple scattering effect of water through deep learning methods, thereby greatly improving the reconstruction accuracy of underwater lidar depth maps, which is beneficial to underwater detection and mapping-related research.

[0029] 2. The deep learning approach used in this method is simple to operate. By using the depth map obtained from the air as the ground truth, the black box model of deep learning is used to inversely remove the influence of noise black boxes. There is no need to build a complex calculation model or the water scattering effect. Only a large amount of sampling of laser detection point cloud data with and without water interference is required to achieve end-to-end depth map generation. At the same time, the deep learning model involved in the method is replaceable and can be updated as technology develops. Attached Figure Description

[0030] Figure 1 This is a flowchart of the laser depth map reconstruction method for turbid water bodies based on deep learning, according to the present invention. Detailed Implementation

[0031] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the deep learning-based laser depth map reconstruction method for turbid water bodies proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, intended only to facilitate and clarify the illustration of the embodiments of this invention, and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0032] It should be noted that, in this invention, relational terms such as "and" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the expressly listed elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0033] The principle of this invention is as follows: by combining laser point cloud data with artificial intelligence, based on the assumption that the scattering effect of turbid water on laser can be regarded as a noise black box, the depth map obtained in the air is used as the true value, and the influence of the noise black box is reversed by using a deep learning black box model, thereby improving the reconstruction accuracy of laser depth map under turbid water.

[0034] This invention provides a deep learning-based method for reconstructing laser depth maps in turbid water bodies, as shown in the attached figure. Figure 1 As shown, it includes the following steps:

[0035] S1. Place the observation target in air and turbid water respectively, and collect neural network training data; wherein, the neural network training data includes a matching dataset composed of air depth maps and turbid water depth maps from the same observation perspective for different water body information; including the following steps:

[0036] S11. Place the observation target in the air medium, collect lidar point cloud data of the observation target from different observation angles, convert the lidar point cloud data of each observation angle into an air medium depth map, and store each air medium depth map and its corresponding observation angle; wherein, the lidar point cloud data is converted into an air medium depth map by converting the time information in the lidar point cloud data into the corresponding distance information based on the ToF (Time of Flight) method;

[0037] S12. Place the observation target in turbid water. Collect lidar point cloud data of the observation target according to each observation angle in step S11. Convert the lidar point cloud data of each observation angle into turbid water depth maps. Store each turbid water depth map and its corresponding water information and observation angle. At this time, the turbid water depth map and the air medium depth map of the same observation angle constitute the different medium comparison data of the current water information. Among them, the water information is the absorption coefficient and scattering coefficient of the turbid water. The lidar point cloud data is converted into turbid water depth maps by converting the time information in the lidar point cloud data into the corresponding distance information based on the ToF method. The distribution of observation angles in steps S11 and S12 needs to cover the typical values ​​of the actual application scenario.

[0038] S13. Change the water body information of turbid water bodies;

[0039] S14. Repeat steps S11 to S13 until the distribution of water information covers the typical values ​​of the actual application scenario, then stop the loop from S11 to S13; at this point, all the stored data constitutes the neural network training data.

[0040] S2. Based on neural network training data, train a generative neural network based on a generative adversarial model to obtain a laser depth map reconstruction model; including the following steps:

[0041] S21. Using water body information and observation angle as condition vectors, the depth maps of each turbid water body as input images (input values), and the depth maps of each air medium as target images (true values), input them into the generative neural network.

[0042] S22. The mean square error between the target image and the output image of the generative neural network is used as the target loss function;

[0043] S23. By synchronously training the generator and discriminator in the generative adversarial network, the target loss function is reduced until the error between the output image and the target image is less than a preset value, thus obtaining the laser depth map reconstruction model.

[0044] S3. Collect lidar point cloud data of the observed target, as well as water body information and observation angle during the actual measurement. Input the data into the lidar depth map reconstruction model to obtain the reconstructed lidar depth map; including the following steps:

[0045] S31. For observation targets in turbid water bodies, conduct actual measurements and acquire measured lidar point cloud data using lidar.

[0046] S32. Convert the measured lidar point cloud data into a depth map with errors; wherein, the lidar point cloud data is converted into a depth map with errors by converting the time information in the lidar point cloud data into the corresponding distance information based on the ToF method;

[0047] S33. Record the water body information and observation perspective at the time of the actual measurement, and convert them into a conditional vector;

[0048] S34. Input the depth map with errors and the conditional vector into the laser depth map reconstruction model to generate a corrected depth map, i.e., reconstruct the laser depth map.

[0049] Furthermore, a specific embodiment is provided for illustration:

[0050] S1. By placing the observation target in air and turbid water respectively, neural network training data is collected; in this embodiment, the observation target is an ellipsoidal model with a length of 0.5 meters; including the following steps:

[0051] S11. Collect lidar point cloud data of the observed target from different observation angles, wherein the observation angle refers to the observation vector in a spherical coordinate system with the center of the observed target as the center. Where θ is the zenith angle. The azimuth angle is used as the reference angle, and data is collected in the upper hemisphere space at 5-degree intervals between the zenith angle and the azimuth angle. The lidar point cloud data from each observation angle are converted into air medium depth maps, and each air medium depth map and its corresponding observation angle are stored.

[0052] S12. Add inorganic salts and sediment to the water and stir thoroughly to form a turbid water body; place the observation target in the turbid water body, collect the lidar point cloud data of the observation target according to each observation angle in step S11, convert the lidar point cloud data of each observation angle into turbid water body depth maps, store each turbid water body depth map and its corresponding water body information and observation angle, at this time the turbid water body depth map and the air medium depth map of the same observation angle constitute the different medium comparison data of the current water body information.

[0053] S13. Adjusting the ratio of inorganic salts and sediment alters the water quality information of turbid water bodies;

[0054] S14. Repeat steps S11 to S13 until the distribution of water information covers the typical values ​​of the actual application scenario and the loop stops; at this point, all the stored data constitutes the neural network training data, which includes 5000 samples.

[0055] S2. Based on the neural network training data, train the generative neural network based on the generative adversarial model to obtain the laser depth map reconstruction model; wherein, the generative adversarial model adopts the pix2pixHD architecture; including the following steps:

[0056] S21. Using water body information and observation angle as condition vectors, the depth maps of each turbid water body as input images (input values), and the depth maps of each air medium as target images (true values), input them into the generative neural network.

[0057] S22. The mean square error between the target image and the output image of the generative neural network is used as the target loss function. In this embodiment, the L2 error loss function is adopted.

[0058] S23. By synchronously training the generator and discriminator in the generative adversarial network, the target loss function is reduced until the error between the output image and the target image is less than a preset value. In this embodiment, a total of 100 iterations are trained to obtain the laser depth map reconstruction model.

[0059] S3. Collect lidar point cloud data of the observed target, as well as water body information and observation angle during the actual measurement. Input the data into the lidar depth map reconstruction model to obtain the reconstructed lidar depth map; including the following steps:

[0060] S31. For observation targets in turbid water bodies, conduct actual measurements and acquire measured lidar point cloud data using lidar.

[0061] S32. Convert the measured lidar point cloud data into a depth map with errors; wherein, the lidar point cloud data is converted into a depth map with errors by converting the time information in the lidar point cloud data into the corresponding distance information based on the ToF method;

[0062] S33. Record the water body information and observation perspective at the time of the actual measurement, and convert them into a conditional vector;

[0063] S34. Input the depth map with errors and the conditional vector into the laser depth map reconstruction model to generate a corrected depth map, i.e., reconstruct the laser depth map.

[0064] In summary, the deep learning-based laser depth map reconstruction method for turbid water provided by this invention can effectively utilize the active detection capability of lidar point clouds and filter out the error caused by multiple scattering effects of water through deep learning methods, thereby greatly improving the reconstruction accuracy of underwater laser depth maps and benefiting underwater exploration and mapping research. The deep learning-based method is simple to operate. By using the depth map obtained in the air as the true value, the influence of noise black boxes is inversely removed using a deep learning black-box model. There is no need to build a complex calculation model. End-to-end depth map generation can be achieved by sampling a large amount of laser detection point cloud data with and without water interference.

[0065] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for reconstructing laser depth maps in turbid water bodies based on deep learning, characterized in that, Including the following steps: S1. Place the observation target in air and turbid water respectively, and collect neural network training data; S11. Place the observation target in the air medium, collect lidar point cloud data of the observation target from different observation angles, convert the lidar point cloud data of each observation angle into an air medium depth map, and store each air medium depth map and its corresponding observation angle. S12. Place the observation target in turbid water, collect lidar point cloud data of the observation target from each observation angle in step S11, convert the lidar point cloud data from each observation angle into turbid water depth maps, and store each turbid water depth map and its corresponding water information and observation angle. S13. Change the water body information of turbid water bodies; S14. Repeat steps S11 to S13 until the distribution of the water body information covers the typical value of the actual application scenario, then stop the loop and generate neural network training data. S2. Based on the neural network training data, train the generative neural network based on the generative adversarial model to obtain the laser depth map reconstruction model; S3. Collect lidar point cloud data of the observed target, as well as water body information and observation angle at the time of measurement, input the laser depth map reconstruction model, and obtain the reconstructed laser depth map.

2. The laser depth map reconstruction method in turbid water as described in claim 1, characterized in that, In step S1, The neural network training data includes a matching dataset consisting of air medium depth maps and turbid water depth maps from the same observation perspective for different water bodies.

3. The laser depth map reconstruction method in turbid water as described in claim 1, characterized in that, The conversion of lidar point cloud data into a depth map is achieved by converting the time information in the lidar point cloud data into corresponding distance information based on the Time-of-Flight (ToF) method.

4. The laser depth map reconstruction method in turbid water as described in claim 1, characterized in that, The water body information refers to the absorption coefficient and scattering coefficient of turbid water.

5. The laser depth map reconstruction method in turbid water as described in claim 2, characterized in that, Step S2 includes: S21. The water body information and observation angle are used as condition vectors, the depth maps of each turbid water body are used as input images, and the depth maps of each air medium are used as target images, and then input into the generative neural network. S22. The mean square error between the target image and the output image of the generative neural network is used as the target loss function; S23. By synchronously training the generator and discriminator in the generative adversarial network, the target loss function is reduced until the error between the output image and the target image is less than a preset value, thereby obtaining the laser depth map reconstruction model.

6. The laser depth map reconstruction method in turbid water as described in claim 2, characterized in that, Step S3 includes: S31. For observation targets in turbid water bodies, conduct actual measurements and acquire measured lidar point cloud data using lidar. S32. Convert the measured lidar point cloud data into a depth map with errors; S33. Record the water body information and observation perspective at the time of the actual measurement, and convert them into a conditional vector; S34. Input the depth map with errors and the condition vector into the laser depth map reconstruction model to generate a reconstructed laser depth map.

Images