Underwater target positioning and modeling method based on multispectral laser radar

Abstract

The invention provides an underwater target positioning and modeling method based on a multispectral laser radar, and the method comprises the steps: obtaining point cloud data and multispectral image data, carrying out the fusion of the point cloud data and the multispectral image data, obtaining multi-modal data, carrying out the denoising of the multi-modal data, carrying out the extraction of a dynamic object, obtaining the point cloud data with multispectral information, and carrying out the positioning and modeling of an underwater target. The method comprises the following steps: performing semantic segmentation on an underwater scene based on point cloud data of multispectral information according to an underwater surface feature of interest and a target, performing three-dimensional modeling on the underwater scene, and endowing a fine semantic tag to a model; according to the time-space reference construction method based on multi-sensor fusion of a multi-spectral laser radar, a multi-spectral camera, an inertial measurement unit and the like, the time-space reference is constructed through association of multi-spectral radar point cloud data and inertial data, underwater accurate reference construction is achieved, modeling of an underwater target is completed, and the precision of the underwater target is improved. And a new solution thought is provided for solving or relieving the problems of inaccurate detection precision, fuzzy characteristics and the like of the current underwater target.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater target detection, and in particular to an underwater target positioning and modeling method based on multi-spectral laser radar. Background Art

[0002] Underwater topography measurement is a basic marine surveying and mapping work, the purpose of which is to obtain the three-dimensional absolute or relative coordinates of the underwater topography, so as to obtain the relevant position, as well as additional information such as water depth, water level, salinity, and sound speed. Its core is underwater topography measurement. In addition to the business of topography measurement, related targets that cause certain obstacles or threats to surface / underwater navigation also need to be regularly detected and cleared.

[0003] Topographic surveying and target detection have undergone three major technological changes, from manual to automatic, sonar technology (from single beam to multi-beam), and airborne lidar. Early underwater topographic measurements were mainly achieved with the help of sounding rods or sounding hammers. The invention of sonar technology has brought new technical means to underwater topographic measurement. By detecting the two-way propagation time of sonar-transmitted sound waves from the transducer to the underwater, and then calculating the water depth in combination with the speed of sound, the ups and downs of underwater terrain are reflected. Based on sonar technology, the technical path from single beam to multiple beams, and from single ship-based measurement to stereo observation has been gradually iterated and developed, and the measured distance and data integrity have been continuously improved. The latest sonar detection system publicly available in the United States can achieve an underwater depth detection accuracy of 5 cm and a detection resolution of 1 meter * 1 meter.

[0004] In addition to sound waves, another underwater transmission window is the blue-green light band. Compared with sonar detection, lidar detection has the advantages of strong concealment, high resolution, and low false alarm rate. The reason is the shorter transmission wavelength of light waves. In addition, the spectrum contains a rich wavelength band, which can reflect the more subtle features of the object to be detected (such as material). At present, foreign spectral-based topographic measurement and target detection work mainly focuses on the use of airborne lidar for underwater mapping.

[0005] Since laser energy is severely attenuated during transmission in seawater, it is very difficult to detect underwater targets by a single means. Whether it is passive imaging or active imaging, the final image effect is not very good due to the interference of water on the detection process. The problems are as follows:

[0006] (1) Low model detection accuracy: The existing target detection algorithm has relatively low positioning accuracy for targets in underwater target images. Due to the complexity of the underwater environment and its significant differences from the terrestrial environment, the existing official model detection is directly used, and its single grid prediction box is difficult to accurately capture the position of the target, resulting in large positioning errors.

[0007] (2) Difficulty in detecting small targets: Underwater images involve a variety of small-sized targets, such as fish, sea urchins, scallops, etc., which poses a challenge to the detection accuracy of existing target detection algorithms. Existing target detection algorithms have poor performance in detecting small targets and are prone to missed detection and false detection.

[0008] (3) Difficulty in detecting blurred targets: Underwater images usually have low contrast and blurred details, which results in blurred features of some underwater targets. Existing technologies are difficult to capture the features of such targets, resulting in serious under-detection of blurred targets by the model. Therefore, it is very necessary to design an underwater target positioning and modeling method based on multispectral lidar. Summary of the invention

[0009] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide an underwater target positioning and modeling method based on multispectral laser radar.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] The present invention also provides an underwater target positioning and modeling method based on multispectral laser radar, comprising:

[0012] Obtain point cloud data and multispectral image data, and fuse them to obtain multimodal data;

[0013] De-noising and dynamic object extraction are performed on multimodal data to obtain point cloud data with multispectral information;

[0014] According to underwater objects of interest and targets, semantic segmentation of underwater scenes is performed based on point cloud data of multispectral information;

[0015] Perform 3D modeling of underwater scenes and give the models detailed semantic labels.

[0016] Preferably, point cloud data is obtained, specifically:

[0017] Construct underwater benchmark scenarios;

[0018] Determine the relative positioning of the underwater scene;

[0019] Analysis and perception of underwater scene echo waveform analysis;

[0020] Get point cloud data.

[0021] Preferably, constructing an underwater reference scene is specifically as follows:

[0022] Set up reply beacons and transmit and receive beacons, and calibrate reference beacons;

[0023] Calibrate underwater benchmarks and reference positions based on transmitting and receiving beacons and reply beacons;

[0024] Calculate spatial position.

[0025] Preferably, the relative positioning of the underwater scene is determined by:

[0026] A tightly coupled positioning framework based on multi-source sensor fusion of multispectral lidar and IMU, and a spatial geometric feature description algorithm based on non-repetitive scanning of three-dimensional point clouds are used to extract point features, line features and planes based on three-dimensional laser point clouds to obtain point cloud data.

[0027] Preferably, the analysis and perception of underwater scene echo waveform analysis is specifically as follows:

[0028] Perform environmental detection based on LiDAR, obtain echo signals, and extract multi-target information based on the echo signals.

[0029] Preferably, semantic segmentation of underwater scenes is performed based on point cloud data of multispectral information, specifically:

[0030] A network architecture based on supervoxels and Transformer is constructed and trained based on a preset data set. The trained network architecture is used to perform semantic segmentation of underwater scenes based on point cloud data of multispectral information.

[0031] Preferably, the underwater scene is three-dimensionally modeled and given fine semantic labels, specifically:

[0032] Based on the characteristics of laser radar in sensing the geometric information of underwater scenes, the texture information of underwater scenes is captured by visual sensors. Based on the characteristics of the geometric information of underwater scenes and the texture information of underwater scenes, the map is spliced ​​and updated using the information obtained by the same texture, spatial structure information and semantic segmentation in the map.

[0033] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0034] The present invention provides a method for underwater target positioning and modeling based on multispectral laser radar, the method comprising acquiring point cloud data and multispectral image data, fusing them to obtain multimodal data, performing denoising and dynamic object extraction on the multimodal data, obtaining point cloud data with multispectral information, semantically segmenting underwater scenes based on underwater objects of interest and targets based on point cloud data with multispectral information, performing three-dimensional modeling on underwater scenes, and giving fine semantic labels to the models. The present invention uses a spatiotemporal benchmark construction method of multi-sensor fusion such as multispectral laser radar, multispectral camera and inertial measurement unit, and constructs a spatiotemporal benchmark by associating multispectral radar point cloud data with inertial data, thereby realizing underwater precise benchmark construction and completing underwater target modeling; it provides new solutions to solve or alleviate the current problems of inaccurate underwater target detection accuracy and feature fuzziness. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 A schematic diagram of a flow chart of an underwater target positioning and modeling method based on a multispectral laser radar provided in an embodiment of the present invention;

[0037] Figure 2 It is a schematic diagram of the multi-sensor time synchronization scheme;

[0038] Figure 3 This is a schematic diagram of the semantic segmentation content of multispectral point cloud;

[0039] Figure 4 Schematic diagram of 3D modeling of underwater terrain point cloud;

[0040] Figure 5 This is a schematic diagram of an underwater target;

[0041] Figure 6 This is a schematic diagram of the modeling results;

[0042] Figure 7 Schematic diagram of the hydroacoustic positioning system. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The purpose of the present invention is to provide an underwater target positioning and modeling method based on multispectral lidar, which not only obtains multi-dimensional information of underwater scenes from the spatiotemporal spectrum level, but also realizes high-precision full-dimensional information restoration of underwater scenes through active and passive collaborative detection through data decoupling.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Figure 1 A schematic diagram of a flow chart of an underwater target positioning and modeling method based on a multispectral laser radar provided in an embodiment of the present invention, such as Figure 1As shown, the present invention provides an underwater target positioning and modeling method based on multispectral laser radar, comprising:

[0047] Step 100: Obtain point cloud data and multispectral image data, and fuse them to obtain multimodal data;

[0048] Step 200: performing denoising processing and dynamic object extraction on the multimodal data to obtain point cloud data with multispectral information;

[0049] Step 300: semantically segmenting the underwater scene based on the point cloud data of multispectral information according to underwater objects of interest and targets;

[0050] Step 400: Perform three-dimensional modeling of the underwater scene and assign fine semantic labels to the model.

[0051] In step 100, point cloud data is obtained, specifically:

[0052] (1) Construct an underwater benchmark scene, specifically:

[0053] 1) Set up the reply beacon and the transmitting and receiving beacon, and calibrate the reference beacon;

[0054] According to the benchmark setting methods under different environments and difficulty conditions, the benchmark construction based on fixed transponder beacons, the benchmark construction method based on floating beacons in indoor pool environment and the benchmark construction method based on floating beacons in outdoor pool conditions are studied respectively. The schematic diagram of the hydroacoustic positioning system is shown in Figure 7 As shown;

[0055] 2) Set up the reply beacon and the transmitting and receiving beacon, and calibrate the reference beacon;

[0056] The underwater carrier is equipped with a transmitting and receiving beacon, which is responsible for transmitting sound wave signals. By sending sound wave signals to the surrounding, the reply beacon will respond with sound wave signals after receiving the sound wave. The reply beacon function: after receiving the sound wave signal, the reply beacon will generate a specific response sound wave signal. This response signal contains information about the location or identity of the beacon and can be captured by the receiver of the underwater carrier. The research uses short baseline or ultra-short baseline positioning technology and "phased control measurement technology" to obtain the target distance. The measurement results are compared with laser ranging to conduct error analysis;

[0057] 3) Calibrate underwater benchmark and reference position based on transmitting and receiving beacons and reply beacons;

[0058] By processing the received signal, the underwater carrier performs distance calculation. Using the propagation time of sound waves or other distance measurement methods, the underwater carrier can calculate the distance to the reply beacon;

[0059] Spatial position calculation method: using spatial intersection, positioning algorithm based on sound wave arrival time difference, signal strength, and common underwater acoustic positioning systems often combine multiple algorithms to improve the accuracy and stability of position estimation and improve the accuracy of time and space reference

[0060] 4) Ensure the temporal and spatial consistency of multiple sensors;

[0061] The multi-source sensor time synchronization method and joint calibration technology based on positioning beacon / laser / vision / inertial navigation ensure the temporal and spatial consistency of sensors in the process of environmental perception;

[0062] The present invention provides a time-space consistency technical solution: in the synchronous positioning and composition algorithm of the fusion multi-sensor data, it is ensured that multiple sensors follow a certain frequency. In order to meet the requirements of consistency of the data collection information of multiple sensors, the present invention uses IMU and high-precision multi-channel time synchronization board to achieve time synchronization between multiple sensors, such as Figure 2 As shown;

[0063] The time synchronization scheme designed by the present invention consists of two parts: hard synchronization and soft synchronization. The beacon provides the most basic time reference for the entire system through satellite time. Multispectral lidar, IMU, multispectral camera and other modal sensors correct their own clocks based on this time reference to collect data. In the host computer software, the algorithm designed by the present invention organizes the data packets collected by all sensors and determines the timestamps to achieve soft synchronization;

[0064] The present invention studies the time deviation caused by the different clock accuracy of each sensor in the multi-source sensor, compensates the time error from both hardware and software aspects, synchronizes the time to a unified reference time scale, realizes hard synchronization and soft synchronization based on multi-source sensors, and hardware-oriented multi-source sensor hard synchronization. It adopts global time synchronization and local time synchronization to transfer GNSS time to the positioning beacon time, which is equivalent to the global time reference source of the whole system. Its PPS pulse output pin can be connected to the IMU and multi-spectral laser radar respectively, and is used to correct the hardware clock inside the IMU and multi-spectral laser radar. After the correction is completed, the physical acquisition time reference of these four sensors is unified. At the same time, the IMU has the function of setting the pulse output frequency, and its output pulse is connected to the external trigger pin of the camera, so that the physical acquisition time of the camera and the IMU is also unified.

[0065] Software-oriented multi-source sensor soft synchronization, the present invention focuses on the different frequency characteristics of each sensor, fuses the measurement values ​​of high-frequency sensors into virtual measurement values ​​and synchronizes and unifies the measurement values ​​of low-frequency sensors at a certain integer time to complete time alignment, develops time alignment algorithms based on the least squares criterion and the interpolation and extrapolation criterion, and quantifies the time synchronization accuracy and corresponding timeliness under different algorithms. Focusing on the adaptability of the interpolation and extrapolation method under different motion models, the present invention will re-derive the interpolation and extrapolation method based on different motion models of the sensor to optimize the time alignment algorithm;

[0066] (2) Determine the relative positioning of the underwater scene, specifically:

[0067] Aiming at the underwater satellite denial environment, a tightly coupled LIV (multispectral lidar) relative positioning framework based on multi-source sensor fusion of multispectral lidar / IMU / is designed. Based on the spatial geometric feature description algorithm of non-repetitive scanning of 3D point cloud, the point feature, line feature and plane extraction based on 3D laser point cloud is realized to obtain point cloud data.

[0068] (3) Analysis and perception of underwater scene echo waveform analysis;

[0069] The underwater environment is complex and the targets are rich and varied. When full-waveform lidar is used for environmental reconnaissance and detection, its laser echo signal usually contains echo information of multiple targets. It is necessary to use echo data processing methods to effectively extract multi-target information. For example, by obtaining the peak value in the echo information, the distance between the lidar and the target can be calculated through flight events;

[0070] Semantic segmentation of underwater scenes based on point cloud data of multispectral information is performed as follows:

[0071] The present invention adopts a new multi-Transformer architecture based on multi-scale supervoxels. First, the 3D point cloud is segmented into a hierarchical supervoxel structure, while adapting to the local features collected at multiple scales; then, a network architecture based on supervoxels and Transformer (SVT) is constructed. The architecture uses a sparse self-attention mechanism to learn the relationship between multi-scale supervoxels. This strategy can effectively learn the complex relationship feature distribution between point clouds in large-scale scenes. By treating the semantic segmentation of large point clouds as the classification of a few supervoxels, unlike traditional three-dimensional convolutional neural networks, this model can accurately classify millions of points at the same time without relying on sliding windows. Through this design, SVT can achieve higher accuracy and is lighter and faster to train than existing methods. Research techniques for semantic segmentation of multispectral point clouds such as Figure 3 As shown, it specifically includes point cloud feature design, point cloud multi-level segmentation design and semantic segmentation based on supervoxel and attention mechanism;

[0072] 3D modeling of underwater scenes and giving the models detailed semantic labels, specifically

[0073] In order to solve the problem of surface parameterization, a point cloud surface parameterization algorithm based on TSDF (truncated signed distance function) is used to triangulate and mesh the map surface. The visual sensor is used to capture the scene texture information, and the laser sensor is used to perceive the scene geometry information. An alignment algorithm based on the scene geometry position information measured by the laser sensor and the scene texture captured by the visual sensor is implemented. The map is spliced ​​and updated using the same texture, spatial structure information, and semantic information detected by deep learning in the map, and high-precision reconstruction of large scenes in complex underwater environments is achieved. For example, Figure 4 As shown;

[0074] To calculate the normal vector of the underwater terrain point cloud, the eigenvalue of each point is first solved, and the direction vector with the smallest eigenvalue is the normal vector of the point cloud. After obtaining the normal vector of each point in the point cloud, a 2.5-dimensional Delaunay triangulation is constructed for the point cloud to improve the efficiency of automatic reconstruction of triangular facets.

[0075] After generating a 2.5-dimensional triangulated mesh of the point cloud, attribute information is assigned to each point using the result of underwater scene semantic segmentation, and corresponding semantic attribute information is assigned to each triangular face according to the rules, ultimately obtaining a Delaunay triangulated mesh with refined semantic attributes.

[0076] To verify the functional effect of the present invention, an underwater target is set based on the present invention. Figure 5 As shown, the model is modeled based on the method described in the present invention, and the obtained model is as follows Figure 6 shown.

[0077] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0078] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An underwater target positioning and modeling method based on multispectral laser radar, characterized in that: include: Obtain point cloud data and multispectral image data, and fuse them to obtain multimodal data; De-noising and dynamic object extraction are performed on multimodal data to obtain point cloud data with multispectral information; According to underwater objects of interest and targets, semantic segmentation of underwater scenes is performed based on point cloud data of multispectral information; Perform 3D modeling of underwater scenes and give the models detailed semantic labels.

2. The method according to claim 1, characterized in that: Get point cloud data, specifically: Construct underwater benchmark scenarios; Determine the relative positioning of the underwater scene; Analysis and perception of underwater scene echo waveform analysis; Get point cloud data.

3. The method according to claim 2, characterized in that Construct an underwater benchmark scene, specifically: Set up reply beacons and transmit and receive beacons, and calibrate reference beacons; Calibrate underwater benchmarks and reference positions based on transmitting and receiving beacons and reply beacons; Calculate spatial position.

4. The method according to claim 2, characterized in that: Determine the relative positioning of the underwater scene, specifically: A tightly coupled LIV relative positioning framework based on multi-source sensor fusion of multispectral lidar and IMU, and a spatial geometric feature description algorithm based on non-repetitive scanning of three-dimensional point clouds are used to extract point features, line features and planes based on three-dimensional laser point clouds to obtain point cloud data.

5. The method according to claim 2, characterized in that: Analysis and perception of underwater scene echo waveform analysis, specifically: Perform environmental detection based on LiDAR, obtain echo signals, and extract multi-target information based on the echo signals.

6. The method according to claim 1, characterized in that Semantic segmentation of underwater scenes based on point cloud data of multispectral information is performed as follows: A network architecture based on supervoxels and Transformer is constructed and trained based on a preset data set. The trained network architecture is used to perform semantic segmentation of underwater scenes based on point cloud data of multispectral information.

7. The method according to claim 1, characterized in that The underwater scene is three-dimensionally modeled and given fine semantic labels. Specifically, the TSDF-based point cloud surface parameterization algorithm is used to triangulate and mesh the underwater scene map surface. The laser radar perceives the geometric information of the underwater scene and the visual sensor captures the texture information of the underwater scene. Based on the characteristics of the underwater scene's geometric information and the underwater scene's texture information, the map is spliced ​​and updated using the information obtained from the same texture, spatial structure information and semantic segmentation in the map.

Images