A method for constructing ship berthing and unberthing scene maps based on stereo perception
By combining a stereo perception method with laser radar, visual sensors, and inertial measurement units, a map of ship berthing and unberthing scenarios was constructed, solving the problem of low positioning accuracy in dynamic environments, achieving high-precision and intelligent environmental understanding, and improving the safety of ship berthing and unberthing.
Patent Information
- Application Number
- CN202410941763.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-07-15
AI Technical Summary
Existing SLAM technology struggles to build high-precision maps of ship berthing and unberthing scenarios in dynamic environments, resulting in low positioning accuracy and impacting the safety and automation level of ship berthing and unberthing.
A stereo perception-based approach is adopted, combining a laser detection radar, visual sensor, and inertial measurement unit on a small unmanned aerial vehicle. The U-Net model is used for semantic segmentation and depth information matching to construct a map of ship berthing and unberthing scenarios. The scanning angle of the laser detection radar is optimized by a generalized iterative closed-point method to eliminate cumulative errors and update the dynamic map in real time.
It enables the construction of high-precision maps for ship berthing and unberthing scenarios in dynamic environments, improves positioning accuracy and safety, provides rich environmental understanding information, and enhances the intelligence and safety of ship berthing and unberthing.
Smart Images

Figure CN118691997B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent sensing technology for waterway transportation, specifically relating to a method for constructing a map of ship berthing and departure scenarios based on three-dimensional perception. Background Technology
[0002] With globalization and the development of the domestic economy, the shipping industry is playing an increasingly important role in my country. The safety of berthing and unberthing is also a key concern in the shipping industry. Due to the increasing global demand for maritime transport and advancements in ship-related technologies, ships are developing towards higher speeds and larger sizes. The inertia of large hulls will significantly affect the maneuverability of ships, especially during berthing and unberthing. At lower speeds, wind and ocean currents have a greater impact on the ship's trajectory, greatly increasing the potential risks during berthing and unberthing. Therefore, accurately obtaining the relative position and speed information of the ship and the berth is crucial for improving the safety and automation / intelligence of berthing and unberthing processes. This requires the construction of real-time, comprehensive environmental understanding, and more precise and detailed berthing and unberthing scenario maps.
[0003] In coastal or estuary areas, the water level at port berths fluctuates with the cyclical changes of the tides. In some regions, the difference in water level between high and low tide can reach as high as 3 meters. These rapid changes in water level can lead to inaccurate environmental perception and low positioning accuracy for ships, preventing them from accurately berthing at designated berths or causing them to enter an unknown port berth. Without accurate maps, it is difficult for captains to have a precise understanding of the environment. Therefore, building a real-time and comprehensive map of berthing and unberthing scenarios is of great significance.
[0004] SLAM technology is a hot research topic in the field of perception and is widely used in various perception-related studies. In recent years, significant progress has been made in front-end technologies based on filtering or optimization, as well as back-end factor graph optimization and loop closure detection technologies, all based on a front-end plus back-end architecture. A number of related products, such as warehouse AGVs, unmanned sweeping vehicles, and low-speed unmanned delivery vehicles, have been implemented. However, many current SLAM technologies are based on static environment assumptions, and robust mapping and localization techniques in dynamic environments remain a pressing issue. Due to environmental changes, the actual environment during mapping often differs from the environment during localization, which can lead to localization errors and affect the final results. This drawback is even more pronounced during ship berthing and unberthing, as port berth areas often contain other vessels, buoys, surges, tides, and other dynamic factors. This complex environmental information affects the perception method's understanding of the environment and reduces localization accuracy, hindering robust mapping and localization in berthing and unberthing scenarios. Summary of the Invention
[0005] The purpose of this invention is to provide a method for constructing ship berthing and unberthing scene maps based on stereo perception, so as to achieve a more intelligent and comprehensive understanding of the environment and provide richer guidance information. This will help improve the safety of ship berthing and unberthing and provide practical value and significance for methods in the field of berthing and unberthing perception.
[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is: a method for constructing a ship berthing and unberthing scene map based on stereo perception, comprising the following steps: S1. Data is collected from ships within the preset berthing and unberthing range using video data acquisition devices mounted on small unmanned aerial vehicles (UAVs). This scans the surrounding environment to obtain video data, capturing the spatial position, velocity, and size of objects within the video. High-precision 3D point cloud data representing the ship and its surrounding environment is acquired via laser radar, including the position of each point in 3D space, reflection intensity, time of capture, and distance and depth information to each point. Inertial measurement units (IMUs) are installed on both the ship and the UAV to obtain inertial measurement data. This data includes the ship's and UAV's attitudes: roll angle, pitch angle, and heading angle, as well as the angular velocities and accelerations along the roll, pitch, and heading axes. Global position information, including longitude, latitude, and altitude, is obtained from the global navigation satellite system.
[0007] S2. Convert visual data, high-precision 3D point cloud data, inertial measurement data, and global location information into the northeast-northeast coordinate system for data modeling; S3. Under the SLAM framework, a ship berthing and departure map is constructed based on the high-precision 3D point cloud data of the laser radar. The high-precision 3D point cloud data is corrected by inertial measurement data to optimize the accuracy of the ship berthing and departure map constructed by the laser radar. The texture information of the ship berthing and departure map is supplemented by visual data collected by a small unmanned aerial vehicle to obtain the final ship berthing and departure scene map.
[0008] It also includes: S4, a visual inertial odometry subsystem composed of a video data acquisition device and an inertial measurement unit set on a small unmanned aerial vehicle, used to estimate the ship's attitude on the water surface and assist in the ship's precise positioning; A U-Net model based on a fully convolutional neural network is constructed on the visual inertial odometry subsystem. The model outputs segmented images of ship berthing and departure scene maps with various preset meaning information. The semantic segmentation and annotation of various preset meaning information are performed to obtain the semantic information of the segmented images.
[0009] It also includes: S5, acquiring depth information from high-precision 3D point cloud data, matching the semantic information of the segmented image with the high-precision 3D point cloud data, aligning the depth information with the dynamic object regions detected by the U-Net model, ensuring that each dynamic object region has corresponding depth information, preset a depth consistency threshold, for each pixel in the segmented image, calculating the difference between its depth value and the average depth value of the corresponding dynamic object region, when the difference of a pixel exceeds the depth consistency threshold, then the pixel is marked as a noise block and filtered out.
[0010] It also includes: S6, modeling the pixels of unfiltered dynamic objects in the dynamic object region detected by the U-Net model in the ship berthing and departure scene map, tracking the movement of dynamic objects, predicting their motion trajectory by combining the kinematic model, adding the predicted trajectory information to the ship berthing and departure scene map and continuously tracking it, and updating it in real time to obtain the ship berthing and departure scene dynamic map.
[0011] The specific method used in S3 to correct high-precision 3D point cloud data using inertial measurement data to optimize the accuracy of ship berthing and departure maps constructed by laser radar is as follows: The set of high-precision 3D point cloud data collected by a pre-defined laser radar is represented as P={ The pitch angle of the small unmanned aerial vehicle detected by the inertial measurement unit is... The scanning angle of the laser detection radar is The corrected laser detection radar scanning angle is Then the formulas for pitch angle compensation and scan angle correction are expressed as follows:
[0012]
[0013] When estimating a ship's position and attitude using the generalized iterative closed-point method, the observed data contains varying degrees of noise. This noise leads to a cumulative error in attitude estimation, which is expressed as:
[0014] in, It is an error function. It refers to the number of points in high-precision 3D point cloud data. It is the normal vector of the plane. It is the ship's rotation matrix. and These are the corresponding points in the previous and next frames, thus eliminating accumulated errors and further optimizing the accuracy of the ship berthing and departure maps constructed by the laser radar.
[0015] The U-Net model in S4 is an improved network structure based on a fully convolutional neural network. This network features a U-shaped symmetrical structure and consists of three parts: an encoder, skip connections, and a decoder. In the encoder stage, semantic features and contextual information of berthing and unberthing video images are extracted through downsampling operations using 3×3 convolutional layers, Re-LU activation function, and 2×2 max pooling layers. The decoder uses 2×2 deconvolution operations, combined with skip connection layers, to concatenate the deconvolution results with the corresponding feature maps to recover lost edge features and retain the positional information of the underlying layers. After upsampling, a 1×1 convolutional layer is used to adjust the number of channels and output a segmented image of the ship berthing and unberthing scene map with various preset meaning information.
[0016] S4 also includes: calculating the loss at each pixel location using pixel-wise cross-entropy loss, and then summing and averaging the results to obtain the loss for the entire segmented image. , represented as:
[0017] in, Indicates the number of samples. Indicates the number of categories. Indicates the true label, This indicates the predicted label.
[0018] The method used in S5 to calculate the difference between its depth value and the average depth value of the corresponding dynamic object region is as follows:
[0019] The depth value of the dynamic object region is... , The difference between its depth value and the average depth value of the dynamic object region is used; if the difference exceeds the depth consistency threshold, the pixel is marked as a noise block; only pixels whose depth information differs from the detected dynamic object region depth by less than or equal to the depth consistency threshold are retained, and the resulting new dynamic object region is represented as:
[0020] in, This is the set of pixels after filtering out noise blocks. This is the original set of pixels.
[0021] The sampling frequencies of the laser radar, inertial measurement unit, and global navigation satellite system are adjusted to be consistent, and the lowest value of the sampling frequencies of the three is taken as the unified sampling frequency.
[0022] A computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method as described in any of the preceding claims.
[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention extracts the correlation between dynamic information by using information from consecutive frames captured by a small unmanned aerial vehicle (UAV) and semantic information. It then combines this with scene depth information provided by a shipborne laser radar to filter out dynamic information in the environment, thus adapting to dynamic environments. Simultaneously, integrating semantic information into SLAM is valuable at the perception level, aiding in the identification of port facilities, ship markings, and dock structures. This enables a more intelligent and comprehensive understanding of the environment, providing richer guidance information. This will contribute to improving the safety of ship berthing and unberthing, providing practical value and real-world significance for methods in the field of berthing and unberthing perception. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the overall structure of an embodiment of the present invention; Figure 2 This is a schematic diagram of the SLAM method framework that integrates spatial stereo perception in an embodiment of the present invention; Figure 3(a) is a simplified schematic diagram of the actual working scenario in an embodiment of the present invention; Figure 3(b) is a simplified schematic diagram of the expected SLAM scanning scenario in an embodiment of the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0026] The present invention aims to provide a SLAM perception system based on laser detection radar-small unmanned aerial vehicle vision-IMU fusion and GNSS as an auxiliary system to solve the problems mentioned in the background art.
[0027] refer to Figure 1 - Figure 3(b) shows the following technical solution adopted by the present invention: a method for constructing a ship berthing and unberthing scene map based on stereo perception, the main steps of which are as follows: Visual data acquisition from S1, GNSS, IMU, laser radar, and small unmanned aerial vehicles. A small unmanned aerial vehicle (UAV) equipped with a high-resolution binocular optical camera (in this embodiment, the high-resolution binocular optical camera is used as a video data acquisition device) photographs ships within a predetermined berthing and unberthing range. The hovering position and altitude of the small UAV are adjusted so that the preset berth is centered in the image, obtaining high-resolution overhead visual data of the berthing and unberthing process. Simultaneously, high-precision 3D point cloud data is acquired using a laser radar to provide spatial information about the ship and its surrounding environment. Two IMU devices, one mounted on the ship and the other on the small UAV, record in real time the ship's attitude, angular velocities and accelerations along the roll, pitch, and yaw axes, and the small UAV's attitude, angular velocities along the roll, pitch, and yaw axes. The ship's global position information is obtained via GNSS.
[0028] S2, Online processing of multi-sensor data This invention primarily uses the NED coordinate system as the coordinate reference system for display and result analysis. It models the visual data of a small unmanned aerial vehicle (UAV) in a spatial coordinate system, calibrating the internal parameters of the camera mounted on the UAV, such as focal length and distortion, and the external parameters, such as the camera's position and attitude relative to the UAV. This ensures that the pixel coordinates in the images acquired by the UAV can be mapped to real-world coordinates. Visual data, 3D point cloud data, and IMU data are all converted to the NED coordinate reference system, with the origin of the NED coordinate reference system set to the initial position of the ship. During data processing, the Earth's inertial coordinate system also needs to be converted to the NED coordinate system. The conversion from the visual coordinate reference system of the small UAV to the NED coordinate reference system can be achieved by formula (1): (1) in, For rotation angle, The position of the small unmanned aerial vehicle in the visual coordinate system. This refers to the position in the NED coordinate system. Let be the translation matrix of a small unmanned aerial vehicle along three axes.
[0029] Ship appendage coordinate system The velocity components and rotation angles of the three axes are as follows: This information is obtained from the IMU. Motion in the ship's appendage coordinate system can be converted to motion in the NED coordinate system using the following formula: (2) (3) (4) (5) (6) in , , They are respectively around Three-axis rotation The rotation matrix of degrees; The Euler angle rotation matrix for rotating the attached coordinate reference frame to the NED coordinate reference frame; For ships in the NED coordinate system The position at that moment; Sampling time; for The ship's three-axis attitude angles at any given time; for The velocity components of the ship's three axes at any given time.
[0030] The installation positions and angles of the laser detection radar and the IMU will inevitably deviate. Therefore, it is necessary to transform the point cloud coordinate system data to the ship's appendage coordinate system, and then transform the point cloud data to the NED coordinate system using IMU data. The processing of the point cloud coordinate system to the appendage coordinate system is similar to that of the appendage coordinate system to the NED coordinate system. First, a rotation matrix needs to be defined. First, the Euler angle rotation matrix is used to rotate the point cloud coordinate system to the attached coordinate system; second, since the laser detection radar is relatively stationary with respect to the ship, only a translation matrix needs to be established. To correct this offset. The laser point cloud is in The coordinates of one of the points at time t are It can be calculated using equation (7): (7) in, This is the position of the point in the ship's appendage coordinate system. The parameters for rotation are used to convert the generated point cloud coordinates to the NED coordinate system through the transformation from the attached coordinate system to the NED coordinate system.
[0031] Because the sampling frequencies of the various sensors differ, temporal registration of the laser point cloud is also required. In this embodiment, the sampling frequency of the unmanned surface vessel's laser radar is 10 Hz, the IMU sampling frequency is 50 Hz, and the GNSS sampling frequency is 10 Hz. To process data at these different frequencies, 10 Hz is selected as the reference frequency for processing, and the IMU data is downsampled. Through interpolation, the original 50 Hz IMU data is adjusted to the same 10 Hz frequency as the laser radar and GNSS data, thereby ensuring that the data from each sensor are aligned in time.
[0032] S3. Real-time construction of ship berthing and departure maps. Visual SLAM methods are more sensitive to details such as color and texture, making them suitable for target recognition and scene understanding. However, their matching accuracy decreases and performance is limited in poor lighting conditions or large outdoor scenes. Furthermore, monocular cameras cannot directly measure the absolute scale of a scene. LiDAR SLAM methods can provide accurate distance and height information and are applicable to various lighting conditions, but they place higher demands on algorithm and hardware performance and are highly sensitive to rain, hindering their widespread applicability. Visual-LiDAR fusion SLAM methods complement the advantages of both types of data, achieving better localization and mapping capabilities. However, accuracy calibration between the two sensors is often difficult, and there is a lack of tightly coupled fusion algorithms between LiDAR and visual sensors. Current experiments combining LiDAR and visual sensors often only reach a low level of fusion, primarily utilizing the results of two measurement steps. This means that LiDAR detection and visual detection cannot closely assist each other, with decisions only made during relative displacement estimation, which underutilizes the capabilities of LiDAR.
[0033] Therefore, this invention employs a more tightly coupled SLAM framework combining LiDAR, IMU, and UAV vision fusion for berthing and unberthing scenarios. This framework is based on multiple sensors, integrating LiDAR, cameras carried by the small UAV, IMU, and GNSS. GNSS provides positioning for both the UAV's visual coordinate system and the LiDAR coordinate system. The remaining sensors constitute two subsystems: a Laser Inertial Odometry (LIO) subsystem composed of a shipborne LiDAR and IMU. The LIO subsystem constructs the geometry of the berthing and unberthing map by registering the point cloud measurements from each LiDAR scan. First, Gaussian interpolation is used to densify the LiDAR point cloud data. Then, the Generalized Iterative Closed Point (GICP) method is used to iteratively estimate the LiDAR attitude and other system states by minimizing the distance from each point in the scan to the plane fitted from the corresponding point in the map, estimating the ship's attitude changes between different time steps. Finally, the estimated state is used to append new points to the map, transforming continuous LiDAR scan data into a coherent map. When a ship is at sea, it is affected by natural factors such as waves, which may cause the hull to pitch. This change in scanning height can disturb the data acquired by the laser radar. More importantly, changes in the pitch angle can also cause distortion of the laser radar's scanning angle, leading to a mismatch between consecutive frames of laser point cloud data. This affects the accuracy and stability of SLAM mapping and localization. Therefore, it is necessary to correct the acquired laser point cloud data using the pitch angle information obtained from the IMU to ensure the consistency of the data in the horizontal direction. Assume the set of data points acquired by the laser radar is P = { }, pitch angle is The laser detection radar scanning angle is The corrected laser detection radar scanning angle is The formulas for pitch angle compensation and scan angle correction are as follows: (8) (9) When estimating the position and attitude of a ship using the generalized iterative closed-point method, the observed data will contain varying degrees of noise. Noise leads to cumulative errors in attitude estimation, and these errors increase over time. This requires the backend to eliminate cumulative errors and improve the accuracy of positioning and mapping through filtering or optimization algorithms. This invention requires the use of scan matching to estimate the pose transformation of the ship in adjacent frames. According to the pose transformation, it cannot be guaranteed that all laser radar data in the previous frame will be completely consistent with the laser data in the next frame. Therefore, the error is calculated as shown in formula (10), where, It is an error function. It is the number of points in the point cloud. It is the normal vector of the plane. It is the ship's rotation matrix. and These are the corresponding points in the previous and next frames, respectively, to eliminate accumulated errors and further optimize the map constructed by the laser radar.
[0034] (10) Another system is the Visual Inertial Odometry (VIO) subsystem, which consists of a small unmanned aerial vehicle (UAV) carrying a binocular camera and an IMU. The VIO subsystem can use visual information captured by the small UAV and data from its IMU to estimate the ship's pose on the water surface, assisting in accurate ship positioning. Simultaneously, the visual data captured by the small UAV can supplement texture information for maps constructed by lidar.
[0035] S4. Integrate the U-Net model into the VIO subsystem to realize the recognition of semantic information in berthing and unberthing scenarios and semantic modeling of the map. A U-Net model was constructed and used to extract semantic information about objects in the environment. U-Net is a network structure based on an improved fully convolutional neural network, characterized by a U-shaped symmetrical structure, consisting of an encoder, skip connections, and a decoder. In the encoder stage, semantic features and contextual information of berthing and unberthing video images were extracted through downsampling operations using 3×3 convolutional layers, Re-LU activation function, and 2×2 max pooling layers. The decoder on the right side used 2×2 deconvolution operations, combined with skip connection layers, to concatenate the deconvolution results with the corresponding feature maps to recover lost edge features and retain the underlying positional information. After upsampling, a 1×1 convolutional layer was used to adjust the number of channels, ultimately outputting segmented images with various semantic information for the berthing and unberthing scenes.
[0036] After the model is built, the collected image data is segmented and labeled with corresponding meanings, and each pixel is assigned to a corresponding meaning category: such as ship, dock, buoy, water surface, shoreline, wave, etc. The pre-labeled dataset is divided into training set, validation set and test set to train and evaluate the model. The loss function is minimized as shown in formula (11). The trained and validated U-Net model is used to perform semantic segmentation on the new input berthing and unberthing scene images. After U-Net recognition and classification, the distribution and state changes of various markers in the berthing and unberthing scene can be reflected more accurately in the semantic map. Due to the flexibility of the U-Net network structure and the effective capture of detailed features, it can realize the semantic feature recognition of complex water scene, which plays a supplementary role in the constructed three-dimensional map and provides a more comprehensive environmental perception.
[0037] (11) in, Indicates the number of samples. Indicates the number of categories. Indicates the true label, The predicted labels are represented here. Pixel-wise cross-entropy loss is used to calculate the loss at each pixel location, and then the average is summed to obtain the loss for the entire image. During training, minimizing the cross-entropy loss function allows the model to learn more accurate pixel-level predictions on the training data, thereby improving the accuracy of image segmentation. Leveraging the semantic information already extracted, objects are classified to distinguish between dynamic and static objects, filtering out dynamic information such as ships, buoys, and people.
[0038] S5. Filter out dynamic object regions using a strategy that combines scene semantic information and depth information. However, due to the complex environmental changes during berthing and unberthing scenarios, variations in lighting can cause the U-Net model to misclassify static objects or background clutter as dynamic objects. This results in the model's identified area either not completely covering moving objects or covering too much of the area, especially object boundaries. Boundary information leaks into the point cloud, forming numerous noise blocks and affecting the accuracy of mapping and localization. Since these noise blocks are typically long and narrow, distributed along object boundaries, but their depth information values often differ significantly from the actual object boundary regions, this invention employs a depth-based filtering strategy to remove noise blocks. The LIO and VIO subsystems are combined. The LIO subsystem mainly consists of a LiDAR sensor, an inertial measurement unit, and a computing unit; the VIO subsystem consists of a high-resolution binocular optical camera, an inertial measurement unit, and a computing unit. By combining two subsystems, map point cloud features are extracted from the data scanned by the LiDAR, and depth information of the scene content is obtained. Semantic information that has already been identified is selected from the image data. The LiDAR point cloud data features and visual semantic information are matched to align the depth information with the dynamic object regions detected by the U-Net model, ensuring that each dynamic object region has corresponding depth information. A depth consistency threshold is set. For each pixel, calculate the difference between its depth value and the average depth value of the corresponding dynamic object region. As shown in formula (12): (12) The depth value of the dynamic object region is... The difference between its depth value and the average depth value of the dynamic object region is If the difference exceeds the set threshold, the pixel is marked as a noise block. Only pixels whose depth information is consistent with the depth of the detected dynamic object region are retained, and pixels that are inconsistent with it are removed. The resulting new dynamic object region is shown in formula (13).
[0039] (13) in, It is the set of pixels after filtering out noise blocks. This is the original set of pixels. Through these two subsystems, a 3D environment map containing the meaning information of corresponding objects is constructed in real time and progressively, adapting to the dynamic environment.
[0040] S6, Dynamic Map Building The laser inertial odometry subsystem was used to construct a laser 3D point cloud map. The visual inertial odometry subsystem was used to construct a visual semantic map. Noise blocks were filtered out by combining the depth information of the laser point cloud map, making the map more accurate. By matching the pixel coordinates of the camera with the points on the laser point cloud, the pixel-level category information obtained by semantic segmentation was mapped onto the 3D point cloud to ensure that each point could get a corresponding label. According to the meaning information obtained by projection, a corresponding label was added to each point in the 3D point cloud. The semantic information of each point was stored as an additional attribute and the color in the point cloud was modified to represent different categories. At the same time, the point cloud with the labeled semantic information was fused with the original map to update the meaning information of each point in the map. Dynamic objects were detected and modeled in the map. The motion of the dynamic objects was tracked and their motion trajectory was predicted by combining the kinematic model, as shown in formula (14). The external forces and torques acting on the hull are shown in formula (15). Wherein, the subscript is These represent the forces and moments exerted on the ship's hull by static ocean fluids, propellers, rudders, wind, waves, and currents. Predicted trajectory information is added to the map and continuously tracked and updated. A three-dimensional environmental map containing semantic information is constructed in real time, adapting to the dynamic environment.
[0041] (14) (15) Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for constructing a ship berthing and unberthing scene map based on stereo perception, characterized in that, Includes the following steps: S1. Data is collected from ships within the preset berthing and unberthing range using video data acquisition devices mounted on small unmanned aerial vehicles (UAVs). This scanning process obtains video data information about the ships and their surrounding environment, including spatial position, speed, and size information. High-precision 3D point cloud data representing the spatial information of the ships and their surrounding environment is acquired using laser radar. This high-precision 3D point cloud data includes the position of each point in the image in 3D space, reflection intensity, time of capture, and distance-depth information to each point. Inertial measurement units (IMUs) are installed on both the ships and the small UAVs to obtain inertial measurement data. This inertial measurement data includes the attitude of the ships and the small UAVs: roll angle, pitch angle, and heading angle, as well as the angular velocities and accelerations along the roll, pitch, and heading axes. Global position information of the ships, including longitude, latitude, and altitude, is obtained through a global navigation satellite system. S2. Convert visual data, high-precision 3D point cloud data, inertial measurement data, and global location information into the northeast-northeast coordinate system for data modeling; S3. Under the SLAM framework, a ship berthing and departure map is constructed based on the high-precision 3D point cloud data of the laser radar. The high-precision 3D point cloud data is corrected by inertial measurement data to optimize the accuracy of the ship berthing and departure map constructed by the laser radar. The texture information of the ship berthing and departure map is supplemented by visual data collected by a small unmanned aerial vehicle to obtain the final ship berthing and departure scene map. S4. A visual inertial odometry subsystem is formed by a video data acquisition device and an inertial measurement unit installed on a small unmanned aerial vehicle. This subsystem is used to estimate the ship's attitude on the water surface and assist in the ship's precise positioning. A U-Net model based on a fully convolutional neural network is constructed on the visual inertial odometry subsystem. The model outputs segmented images of ship berthing and departure scene maps with various preset meaning information and performs semantic segmentation and annotation on the various preset meaning information to obtain the semantic information of the segmented images. S5. Obtain depth information from high-precision 3D point cloud data, match the semantic information of the segmented image with the high-precision 3D point cloud data, and align the depth information with the dynamic object regions detected by the U-Net model to ensure that each dynamic object region has corresponding depth information. A depth consistency threshold is preset. For each pixel in the segmented image, calculate the difference between its depth value and the average depth value of the corresponding dynamic object region. When the difference of a pixel exceeds the depth consistency threshold, the pixel is marked as a noise block and the pixel is filtered out. S6. Model the pixels of the unfiltered dynamic objects in the dynamic object region detected by the U-Net model in the ship berthing and departure scene map, track the movement of the dynamic objects, predict their motion trajectory by combining the kinematic model, add the predicted trajectory information to the ship berthing and departure scene map and continuously track it, and update it in real time to obtain the ship berthing and departure scene dynamic map.
2. The method for constructing a ship berthing and unberthing scene map based on stereo perception according to claim 1, characterized in that, The specific method used in S3 to correct high-precision 3D point cloud data using inertial measurement data to optimize the accuracy of ship berthing and departure maps constructed by laser radar is as follows: The set of high-precision 3D point cloud data collected by a pre-defined laser radar is represented as P={ The pitch angle of the small unmanned aerial vehicle detected by the inertial measurement unit is... The scanning angle of the laser detection radar is The corrected laser detection radar scanning angle is Then the formulas for pitch angle compensation and scan angle correction are expressed as follows: When estimating a ship's position and attitude using the generalized iterative closed-point method, the observed data contains varying degrees of noise. This noise leads to a cumulative error in attitude estimation, which is expressed as: in, It is an error function. It refers to the number of points in high-precision 3D point cloud data. It is the normal vector of the plane. It is the ship's rotation matrix. and These are the corresponding points in the previous and next frames, thus eliminating accumulated errors and further optimizing the accuracy of the ship berthing and departure maps constructed by the laser radar.
3. The method for constructing a ship berthing and unberthing scene map based on stereo perception according to claim 1, characterized in that, The U-Net model in S4 is a network structure based on an improved fully convolutional neural network. This network is characterized by a U-shaped symmetrical structure and consists of three parts: an encoder, skip connections, and a decoder. In the encoder stage, semantic features and contextual information of berthing and unberthing video images are extracted through downsampling operations using 3×3 convolutional layers, Re-LU activation function, and 2×2 max pooling layers. The decoder uses a 2×2 deconvolution operation, combined with a skip connection layer, to stitch the deconvolution result with the corresponding feature map to recover lost edge features and retain the positional information of the underlying layer. After upsampling, a 1×1 convolution layer is used to adjust the number of channels and output a segmented image of the ship berthing and departure scene map with various preset meaning information.
4. The method for constructing a ship berthing and unberthing scene map based on stereo perception according to claim 1, characterized in that, S4 also includes: calculating the loss at each pixel location using pixel-wise cross-entropy loss, and then summing and averaging the results to obtain the loss for the entire segmented image. , represented as: in, Indicates the number of samples. Indicates the number of categories. Indicates the true label, This indicates the predicted label.
5. The method for constructing a ship berthing and unberthing scene map based on stereo perception according to claim 1, characterized in that, The method used in S5 to calculate the difference between its depth value and the average depth value of the corresponding dynamic object region is as follows: The depth value of the dynamic object region is... , The difference between its depth value and the average depth value of the dynamic object region is used; if the difference exceeds the depth consistency threshold, the pixel is marked as a noise block; only pixels whose depth information differs from the detected dynamic object region depth by less than or equal to the depth consistency threshold are retained, and the resulting new dynamic object region is represented as: in, This is the set of pixels after filtering out noise blocks. For the original set of pixels, This is the depth consistency threshold.
6. The method for constructing a ship berthing and unberthing scene map based on stereo perception according to claim 1, characterized in that, The sampling frequencies of the laser radar, inertial measurement unit, and global navigation satellite system are adjusted to be consistent, and the lowest value of the sampling frequencies of the three is taken as the unified sampling frequency.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.
Citation Information
Patent Citations
Visual perception device and method for ship navigation environment
CN113705375A
Unmanned aerial vehicle indoor mapping and positioning method for ship detection
CN115902930A
