Berthing state estimation method based on shipborne laser radar target detection

Abstract

The invention discloses a berthing state estimation method based on shipborne laser radar target detection, and the method comprises the steps: constructing a ship three-dimensional target detection network model through an improved PointPill coding network, and introducing an average pool layer and an attention pooling layer into the coding network to carry out the fusion of the global perception information and the local perception information of a target, and performing model training according to the constructed model loss function, detecting and confirming whether an anchored ship exists in a preset berth through an optimal detection network model, and respectively acquiring whether the anchored ship exists or does not exist in the preset berth by introducing an RANSAC plane fitting algorithm. A berthing speed used for ship berthing state estimation is obtained by combining the required berthing front edge plane or the outer contour surface of the target ship, namely a berthing berthing boundary surface and the position of the shipborne laser radar; the defect that the berthing state estimation accuracy is poor due to the fact that a berthing line model obtained through a traditional method based on linear fitting serves as the berthing state estimation reference is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of ship berthing, and in particular to a berthing state estimation method based on shipborne laser radar target detection. Background Art

[0002] Ship berthing is a critical component of port operations, and its safety and efficiency directly impact port operating costs and navigation safety. Traditional berthing relies primarily on the crew's judgment and manual ranging (such as visual, radar, and cable-based ranging), which is subject to subjectivity, low accuracy, and insufficient real-time performance. Statistics show that berthing errors result in significant economic losses worldwide each year due to accidents such as ship collisions and dock damage. Furthermore, manual operations significantly decrease in reliability in inclement weather (such as fog and at night).

[0003] In addition, shortening the berthing time of ships is of great significance to improving the efficiency of ship construction and maintenance in shipyards. When ships under construction and maintenance berth in shipyard ports with severely restricted navigation waters, it is difficult for the pilot to accurately grasp the distance and speed of the ship approaching the berth, resulting in a long berthing process. Figure 23 As shown, taking the Dalian Shipyard as an example, the distance between the two docks is less than 400 meters, while the ships under construction are mostly large vessels at least 100 meters long, requiring extreme caution in berthing operations. Furthermore, as ships become larger, the driver's blind spot on the bridge increases, increasing the risk of collision during berthing. Therefore, precise sensing and control of the ship's approach, distance, and speed are essential to ensure safe berthing operations. Therefore, research on ship berthing state estimation in shipyard port waters has broad application prospects.

[0004] In recent years, the development of intelligent ship technology has provided new insights for berthing automation, with environmental perception being one of the core challenges. In restricted waters, such as shipyard harbors, Beidou and Global Positioning Systems (GPS) often have positioning errors of only meters due to obstruction by dock buildings, making it difficult to meet the precise positioning and ranging requirements of ships. Furthermore, maritime radars lack sufficient accuracy for short-range ranging, and visual sensors cannot directly measure distances. LiDAR (Light Detection and Ranging), with its high accuracy (centimeter-level), strong anti-interference capabilities (unaffected by light and haze), and ability to output three-dimensional point clouds, has become an ideal sensor for near-field perception of ships.

[0005] Existing research has made significant progress in LiDAR point cloud processing, but it still has obvious limitations for ship berthing scenarios. Previous research can be divided into two categories based on the location of the LiDAR sensor: berthing perception systems based on shore-based LiDAR and berthing perception systems based on ship-borne LiDAR. Chen and Li proposed a real-time ship tracking and dynamic berthing information extraction system based on shore-based two-dimensional LiDAR. Kim et al. developed an autonomous berthing system for unmanned boats based on symbol recognition, using cameras and LiDAR to detect berths and symbols to estimate relative heading angles and positions. Wang et al. introduced geographic information system spatial analysis technology to propose a method for monitoring the berthing and unberthing status of ships. Using geographic information system spatial theory, they extracted the optimal segmentation region of the ship hull from the point cloud data. Subsequently, key points were obtained from the optimal segmentation region to calculate the key parameters of the berthing and unberthing processes. Lan et al. deployed LiDAR on the ship lock and proposed a LiDAR-based ship positioning method. Point cloud preprocessing is achieved through steps such as point cloud reflection intensity filtering, coordinate transformation, and ship point cloud clustering extraction. Based on the characteristics of point cloud changes during ship navigation through locks, different segmentation strategies are proposed to extract the point cloud of each ship. Finally, real-time ship tracking is achieved using a Kalman filter. Wang et al. proposed a new method for calculating ship berthing angles using a shore-based lidar system. First, the point cloud data scanned by the lidar is preprocessed and projected onto a two-dimensional (2D) plane. The height values ​​are used as pixel values ​​to generate a bird's-eye view image. Finally, an improved Hough transform is proposed to detect straight lines representing the ship's nearshore outline in the bird's-eye view image. These detected lines are used to replace the ship's centerline for berthing angle calculation. Lu and Li proposed a method for estimating the multi-point ship pose information based on point cloud data. Compared with traditional pose estimation methods that treat the ship as a single point, this algorithm can effectively obtain ship pose information from multiple points, ultimately estimating real-time angle, distance, speed, and other information of multiple points on the ship. In subsequent research, the idea of ​​obtaining the angle information of the ship through point cloud data was used to determine the point cloud data within the hull range, and the angle information between the ship and the dock was finally estimated through these point cloud data. The study of berthing status estimation of berthed ships based on shore-based lidar can stably obtain various berthing parameters of the ship and transmit them to the port supervision department or the driver on the ship. However, this shore-based lidar-based method also has its limitations. First, lidars need to be deployed at standard port terminals, which is not applicable to some unsupervised inland docks. Secondly, the shore-based berthing perception system only has a relatively good effect on the first batch of ships berthing, and is not applicable to the situation where multiple ships are berthing.

[0006] To solve the problem of the inapplicability of the shore-based berthing perception system, many scholars have developed shipborne laser radar berthing perception systems. Hu et al. proposed a detailed calculation method for five berthing states, including the distance of the approaching ship to the wharf, transverse speed, longitudinal speed, approaching angle of the ship, and roll angle, based on the point cloud of the shipborne three-dimensional laser radar. Wang et al. proposed a new data fusion method combining multiple solid-state laser radars and millimeter wave radars to calculate the relative distance between the bow and stern of the ship and the berth, the berthing speed, and the approaching angle of the ship. Wang et al. proposed a berthing state estimation method that only relies on 3D laser radar, which can provide various types of berthing state information. This method uses a scan matching algorithm to estimate the position and attitude of the ship. In addition, a point cloud line fitting algorithm is used to establish the berth line equation. By integrating the fitted berth line equation with the ship motion data, the berthing state information of the ship is calculated. Wang et al. proposed a berthing perception system based on shipborne laser radar, which includes the following four different modules: positioning, navigation area acquisition, idle berth determination, and berthing state estimation. Among them, the idle berth determination module uses a coastline-constrained point cloud linear fitting algorithm to extract the parking area from the nearest point cloud and obtain the idle berth information. The berthing state estimation module uses the pre-established berthing line to calculate the basic berthing parameters, including the berthing distance, approaching angle, berthing speed, and yaw rate. By synthesizing the previous research, it can be found that the shipborne laser radar berthing perception system needs to face the problem of berth boundary extraction compared with the shore-based perception system. The previous research is based on berth line fitting to calculate the berthing parameters. In fact, the berth boundary is a plane, and this point-to-plane berth boundary processing method will undoubtedly increase the error of the berthing parameters. In addition, neither the shore-based nor the shipborne laser radar perception system considers the multi-ship berthing scenario, and in fact, multi-ship berthing is often required in shipyards or busy ports.

[0007] In addition, more and more researches on water transportation scenarios based on laser radar three-dimensional target detection. Yao et al. based on the clustering results of point cloud, a target position and volume estimation method is studied. Lin et al. proposed a PointPillar detection network based on deep learning, using laser radar as the main sensor, AIS as the auxiliary information source, realizing the three-dimensional target detection of ship, multi-target tracking and port static environment mapping. Xie et al. proposed a ship detection and tracking framework based on laser radar, which can switch between PointPillar, SECOND and PV-RCNN three detection networks by adopting modular network structure to meet various task requirements. This framework can be applied to busy marine environment and realizes the overall detection accuracy of 74.1%. However, when these point cloud detection networks are applied to ship target detection in marine scenarios, the data set true value label file format is different, as shown in Table 1, and the marine laser radar data set usually does not label the truncation degree, the occlusion degree, the observation angle and the two-dimensional target box. When training with a general target detection framework, the setting of fields not considered is usually set to a constant, which undoubtedly reduces the accuracy and precision of target detection, which also explains the phenomenon that the target detection accuracy of laser radar point cloud data set in marine scenarios is lower than that in the field of autonomous driving.

[0008] In summary, the existing technical solutions have the following problems:

[0009] Firstly, the method based on shore-based laser radar has its limitations. It is necessary to deploy laser radar on standard port wharf, which is not suitable for some unmanned wharf in inland river. Secondly, the berthing perception system based on shore has better effect on the first berth of the ship, but it is not suitable for the situation of multi-ship berthing. In addition, the berth boundary is a plane, and the previous research on berth boundary processing method with line instead of plane undoubtedly increases the error of berthing parameters. Therefore, neither shore-based nor ship-borne laser radar perception system considers the berthing scenario of multi-ship berthing. In fact, multi-ship berthing often occurs in shipyards or busy ports. In addition, when point cloud detection network is applied to ship target detection in marine scenarios, marine laser radar data set usually does not label the truncation degree, the occlusion degree, the observation angle and the two-dimensional target box. When training with a general target detection framework, the setting of fields not considered is usually set to a constant, which undoubtedly reduces the accuracy and precision of target detection. SUMMARY

[0010] The present application provides a berthing state estimation method based on ship-borne laser radar target detection to overcome the above technical problems.

[0011] In order to achieve the above purpose, the technical scheme of the present application is:

[0012] A berthing state estimation method based on shipborne laser radar target detection, comprising the following steps:

[0013] S1: collecting and acquiring shipborne laser radar point cloud;

[0014] and labeling the shipborne laser radar point cloud for ship anchor frame and ship type, obtaining preprocessed point cloud images, and randomly dividing them into training set and test set;

[0015] S2: improving the PointPillar coding network to build a ship three-dimensional target detection network model, which includes a point cloud coding network module, a point cloud feature extraction module and a detection head network;

[0016] The point cloud coding network module is used for point cloud coding processing of the shipborne laser radar point cloud, to obtain a point cloud pseudo image containing global perception point cloud features and local perception point cloud features of the fusion ship three-dimensional point cloud, the point cloud feature extraction module is used for extracting ship three-dimensional point cloud features in the preprocessed point cloud image to obtain a point cloud feature extraction image, and the detection head network is used for realizing ship three-dimensional target detection according to the point cloud feature extraction image;

[0017] S3: training the ship three-dimensional target detection network model based on the training set and the test set to obtain an optimal detection network model;

[0018] S4: according to the optimal detection network model, detecting the ship target of the laser radar point cloud at the preset berth to confirm whether there is an anchored ship at the preset berth according to the detection result;

[0019] If not, the RANSAC plane fitting algorithm is used to obtain a berth front plane for directly berthing the ship at the berth;

[0020] If there is, the target ship three-dimensional point cloud berthing at the preset berth is obtained, and the RANSAC plane fitting algorithm is used to obtain a target ship boundary plane to obtain a boundary fitting plane of the target berth, and then the outer contour surface of the target ship is confirmed according to the boundary fitting plane;

[0021] And the berth front plane and the outer contour surface are collectively referred to as the berthing berth boundary surface;

[0022] S5: based on the position of the shipborne laser radar, the laser radar berthing distance from the berthing berth boundary surface is obtained; and according to the laser radar berthing distance, the bow position and the stern position of the ship are obtained, and the bow berthing distance and the stern berthing distance from the berthing berth boundary surface are obtained;

[0023] Based on the bow berthing distance and the stern berthing distance, the berthing speed of the ship is obtained according to the sampling time of the shipborne laser radar; the berthing speed of the ship includes the bow berthing speed and the stern berthing speed, and then the berthing state of the ship is estimated according to the berthing speed of the ship.

[0024] Further, the point cloud encoding network module in S2 includes an input layer, a point cloud encoding layer, a maximum pooling layer, an average pooling layer, an attention pooling layer, and a point cloud pseudo image acquisition module.

[0025] The input layer is used to input the shipborne laser radar point cloud to the point cloud encoding layer.

[0026] The point cloud encoding layer is used to perform point cloud voxelization operation on the shipborne laser radar point cloud to obtain a point cloud stacked column graph, and perform grid division and encoding on the point cloud stacked column graph to obtain a pillar grid feature graph with the same size, and obtain an enhanced feature graph by calling a multi-layer perception MLP to map the pillar grid feature graph.

[0027] The maximum pooling layer is used to perform maximum pooling operation on the enhanced feature graph to obtain a local perception point cloud feature graph of the ship three-dimensional point cloud; and the average pooling layer is used to perform average pooling operation on the enhanced feature graph to obtain a global perception point cloud feature graph of the ship three-dimensional point cloud.

[0028] The attention pooling layer is used to predict and obtain local attention scores of each grid feature in the pillar grid feature graph based on a multi-layer perception MLP to obtain a local attention feature graph, and perform element-wise multiplication operation on the local attention feature graph and the pillar grid feature graph to obtain a fusion feature graph.

[0029] The point cloud pseudo image acquisition module is used to perform mean operation on the fusion feature graph, the global perception point cloud feature graph, and the local perception point cloud feature graph to obtain a 2D point cloud pseudo image containing the ship three-dimensional point cloud feature.

[0030] Further, the point cloud feature extraction module in S2 preferably uses a two-dimensional convolutional neural network 2D-CNN; and the detection head network preferably uses an SSD detection head of a target detection algorithm.

[0031] Further, the method for obtaining the optimal detection network model in S3 specifically includes:

[0032] S31: training the ship three-dimensional target detection network model constructed by using the training set to obtain a trained ship three-dimensional target detection network model.

[0033] S32: Based on the constructed model loss function, the trained ship three-dimensional target detection network model is evaluated by using the test set to determine whether the output of the trained ship three-dimensional target detection network model converges;

[0034] If yes, the trained ship three-dimensional target detection network model is the optimal detection network model;

[0035] Otherwise, based on the back propagation method, the parameter weight of the trained ship three-dimensional target detection network model is adaptively adjusted, and S31 is repeatedly executed.

[0036] Further, the construction formula of the model loss function in S32 is

[0037]

[0038] In the formula, L det represents the model loss function; N a represents the total number of ship anchor frames; L cls represents the target class loss; L loc represents the target position loss; L dir represents the target direction loss; L IoU represents the intersection over union prediction loss; L ref represents the reflectivity prediction loss; λ1, λ2, λ3, λ4, λ5 represent loss weights; L cls (p i ,c i ) represents an intermediate variable; p i ,c i respectively represent the regression of the target class prediction value and the true value from the i th anchor frame; a represents a balance factor; γ represents a focusing parameter; L reg (δ i ,t i ) represents an intermediate variable; l(c i =1) represents the position loss of the ship target; δ i ,t i respectively represent the predicted anchor frame and the true anchor frame of the ship target; x, y, z respectively represent the center coordinates in the three-dimensional space of the ship target; l, w, h respectively represent the dimensions of the ship target, i.e. length, width and height; θ represents the orientation of the ship target; δ ij ,t ij respectively represent the predicted value and the true value of the i th ship target in the j th dimension; L sm (δ ij -t ij ) is L sm (g) represents an intermediate variable; L dir (h d ,ρ i ) represents an intermediate variable; hd represents a two-dimensional vector of the ship target heading classification; p i represents a classification probability of the ship target heading; p i,k represents the probability of the i-th ship target on the k-th class; k represents the index of the ship target heading class; h d,k represents the predicted value of the ship target heading being classified into the k-th class; q i represents the IOU predicted value; IoU(δ i ,t i ) represents the intersection over union between the predicted anchor frame δ i and the real anchor frame t i ; V IoU (δ i ,t i ) represents the actual intersection over union of the predicted anchor frame and the real anchor frame; L sm (·) represents the Smooth Ll loss function; r i represents the reflectance intensity predicted value; represents the reflectance intensity true value.

[0039] Further, the S4 specifically comprises the following steps:

[0040] S41: According to the optimal detection network model, the laser radar point cloud at the preset berth is detected for ship target, so as to confirm whether there is an anchored ship at the preset berth according to the detection result;

[0041] If not, the target ship three-dimensional point cloud parked at the preset berth is obtained, and the target ship boundary surface point cloud is obtained based on the RANSAC plane fitting algorithm, so as to obtain the boundary fitting plane of the target berth, and then the outer contour surface of the target ship is confirmed according to the boundary fitting plane;

[0042] Specifically, the following steps are included:

[0043] S100: According to the relative position between the target ship and the target ship, the three-dimensional point cloud close to the side of the target ship is defined as the boundary surface point cloud set of the target ship;

[0044] S101: Randomly extract three sample point clouds in the boundary surface point cloud set;

[0045] And according to the randomly extracted sample point cloud, the model parameters in the spatial plane equation are solved to obtain the fitting spatial plane model, and the expression of the spatial plane equation is

[0046] Ax+By+Cz+D=0

[0047] In the formula: A, B, C represent three coordinate component values of the unit normal vector of the plane to be solved, which satisfy A 2 +B 2 +C2 = 1 ; D represents the distance from the coordinate origin to the spatial plane; x, y, z represent the position coordinates of the sample point cloud;

[0048] S102: Traverse the remaining point clouds in the boundary surface point cloud set after extracting the sample point cloud, and calculate the distance of each remaining point cloud to the fitted spatial plane model respectively, and the calculation formula is

[0049]

[0050] In the formula: d i represents the distance of the remaining point cloud to the fitted spatial plane model; x i , y i , z i represent the position coordinates of the i-th remaining point cloud;

[0051] S103: Set the distance threshold d t , and record the remaining point cloud that satisfies d i > d t as an outlier; continue to execute step S104 for the remaining point cloud that satisfies d i ≤ d t ;

[0052] S104: Obtain the normal vector n j of the remaining point cloud and the normal vector n p of the fitted spatial plane based on the fitted spatial plane model, and calculate the angle θ j between the normal vectors, and the calculation formula of the angle between the normal vectors is

[0053]

[0054] S105: Set the angle threshold β, and the remaining point cloud corresponding to θ j > β is an outlier, and the remaining point cloud corresponding to θ j ≤ β is an inner point, and then obtain the inner point set;

[0055] S106: Repeat steps S101 to S105 until the random extraction times reach the preset iteration threshold, and obtain a plurality of inner point sets;

[0056] The iteration threshold of the RANSAC plane fitting algorithm is determined by the probability of extracting qualified samples, and the setting formula of the preset iteration threshold is

[0057]

[0058] In the formula: ω represents the probability that the three initial sample point clouds selected in the extraction process are all plane points; ε represents the probability of selecting a plane point from the boundary surface point cloud set; N Ranrepresents the iteration number threshold value;

[0059] S107: The number of inner points in each inner point set is counted and obtained, and the fitting space plane model corresponding to the inner point set with the largest number of inner points is taken as the optimal fitting space plane model, so as to obtain the boundary fitting plane of the target berth, and then the outer contour surface of the target ship is confirmed according to the boundary fitting plane;

[0060] If not, the ship can be directly berthed in the berth;

[0061] And the method of directly berthing the ship in the berth, specifically comprises:

[0062] S200: The latitude and longitude coordinates of the target berth are obtained by the preset navigation system, and the latitude and longitude coordinates of the position of the ship are subtracted to obtain the coordinate difference;

[0063] S201: The coordinate difference is converted into a meter coordinate, i.e. the laser radar position coordinate of the target berth, and the target berth point cloud set is obtained by neighborhood search with the laser radar position coordinate of the target berth as the center and with a preset search radius r. It is judged whether the number of cloud points of the target berth point cloud set meets the preset number threshold value;

[0064] If yes, the plane model of the berth front edge is obtained based on the RANSAC plane fitting algorithm according to the target berth point cloud set;

[0065] If not, the search radius is enlarged for neighborhood search until the target berth point cloud set obtained by search meets the preset number threshold value, and the plane model of the berth front edge is obtained based on the RANSAC plane fitting algorithm according to the target berth point cloud set, and then the plane of the berth front edge is confirmed;

[0066] S42: The plane of the berth front edge and the outer contour surface are collectively referred to as the berthing berth boundary surface.

[0067] Further, the S5 specifically comprises the following steps:

[0068] S51: Based on the position of the ship-borne laser radar, the laser radar berthing distance from the berthing berth boundary surface is obtained, and the formula for obtaining the laser radar berthing distance is:

[0069] d lidar =f(0,0,0);

[0070] In the formula, d lidar represents the laser radar berthing distance; f(0,0,0) represents the distance from the point (0,0,0) in the laser radar coordinate system to the berthing berth boundary surface;

[0071] S52: Obtain the bow berthing distance and the stern berthing distance of the ship according to the laser radar berthing distance;

[0072] and the formula for obtaining the bow berthing distance and the stern berthing distance is:

[0073] d bow =f(Δ B ,Δ L1 ,0)

[0074] d stern =f(Δ B ,Δ L2 ,0)

[0075] In the formula, d bow represents the bow berthing distance; d stern represents the stern berthing distance; Δ B represents the included angle between the straight line of the ship body along the bow and stern directions and the line segment on which the laser radar berthing distance lies; Δ L1 represents the distance from the laser radar center to the bow; and Δ L2 represents the distance from the laser radar center to the stern.

[0076] S53: Obtain the berthing speed of the ship according to the point cloud sampling time of the shipborne laser radar based on the bow berthing distance and the stern berthing distance.

[0077] In the formula, the berthing speed of the ship includes the bow berthing speed and the stern berthing speed.

[0078] and the expression of the bow berthing speed and the stern berthing speed is

[0079]

[0080] In the formula, Δt represents the time variable obtained by subtracting the time stamp of the i+1 frame point cloud from the time stamp of the i frame point cloud; v bow ,v stern respectively represent the bow berthing speed and the stern berthing speed. respectively represent the bow berthing distance and the stern berthing distance corresponding to the i frame point cloud. respectively represent the bow berthing distance and the stern berthing distance corresponding to the i+1 frame point cloud.

[0081] Further, the berthing state of the ship is estimated according to the berthing speed of the ship.

[0082] Beneficial effects: the application provides a berthing state estimation method based on shipborne laser radar target detection, improves a ship three-dimensional target detection network model of a PointPillar coding network, introduces an average pooling layer and an attention pooling layer in the coding network to improve the fusion of global perception and local perception information of the target, ensures that the model can more comprehensively capture the point cloud features in the pillar, and according to the model loss function constructed, the model is trained, and then the performance and detection accuracy of target detection are effectively improved; the optimal detection network model is used for detecting and confirming whether the preset berth exists an anchored ship, and the RANSAC plane fitting algorithm is introduced to obtain the berth front plane or the target ship's outer contour surface, that is, the berthing berth boundary surface, when the preset berth exists or does not exist an anchored ship, and the berthing speed used for ship berthing state estimation is obtained according to the position of the shipborne laser radar and the berthing berth boundary surface, which solves the defect that the berthing state estimation accuracy is poor when the berth line model obtained by using the traditional linear fitting method is used as the reference of the berthing state estimation, greatly improves the berthing precision of the berthing perception system, and enhances the applicability and expansibility of the berthing perception system. BRIEF DESCRIPTION OF DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, below will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0084] Figure 1 The flow chart of the berthing state estimation method based on shipborne laser radar target detection of the application;

[0085] Figure 2 The design block diagram of the berthing perception system architecture based on shipborne laser radar in the embodiment;

[0086] Figure 3 The schematic diagram of the improved Point Pillar coding network in the embodiment;

[0087] Figure 4 The schematic diagram of the point cloud data collection module structure in the embodiment;

[0088] Figure 5 The schematic diagram of the relationship between the point cloud and the fitting plane normal vector in the embodiment;

[0089] Figure 6 The schematic diagram of the shipyard port berthing operation case in the embodiment;

[0090] Figure 7A schematic diagram of the closest distance between the bow and stern of the ship to the outer contour surface of the target ship in this embodiment;

[0091] Figure 8 A top view of Lingshu Port and a wharf of a certain section of Tianjin Haihe River in this embodiment;

[0092] Figure 9 A schematic diagram of Zhilong No. 1 test boat in this embodiment;

[0093] Figure 10 A schematic diagram of the scene of the disclosed laser point cloud data set in this embodiment;

[0094] Figure 11 A schematic diagram of the sailing track of the ship in Lingshu Port in this embodiment;

[0095] Figure 12 A visualization result diagram of ship three-dimensional target detection of the ship in Lingshu Port in this embodiment;

[0096] Figure 13 An image of the ship in Lingshu Port at 30 seconds in this embodiment;

[0097] Figure 14 A top view of berth area identification in this embodiment;

[0098] Figure 15 An effect diagram of berth area ship point cloud extraction in this embodiment;

[0099] Figure 16 An effect diagram of boundary plane fitting of berth front point cloud in this embodiment;

[0100] Figure 17 A sailing track diagram of berthing empty berth test in this embodiment;

[0101] Figure 18 A distance curve diagram of the ship to the target ship in the first group of tests in this embodiment;

[0102] Figure 19 A simulation curve diagram of the distance of the ship to the front of the wharf in the first group of tests in this embodiment;

[0103] Figure 20 An error diagram of the distance of the laser radar to the berth boundary surface and the RTK true value in this embodiment;

[0104] Figure 21 A sailing track curve diagram of the ship in Tianjin Haihe berthing a parked ship in this embodiment;

[0105] Figure 22 A distance and speed curve diagram of the ship to the outer contour surface of the target ship in the second group of tests in this embodiment;

[0106] Figure 23 Figure 1 is a schematic diagram of the berthing of a ship to be repaired at two docks of a shipyard in Dalian. DETAILED DESCRIPTION

[0107] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0108] The embodiment provides a berthing state estimation method based on shipborne laser radar target detection, as shown in Figures 1-2 The method comprises the following steps:

[0109] S1: Collect and acquire shipborne laser radar point clouds, and use Supervisely to label the shipborne laser radar point clouds for a ship anchor frame and a ship type, acquire a pretreated point cloud graph, and randomly divide the pretreated point cloud graph into a training set and a test set;

[0110] Specifically, the laser point cloud data collection module is designed as shown in Figure 4 and contains a power supply module, laser radar original data reading, GPS-RTK original data reading, point cloud data publishing and data storage functions. The power supply module pre-installed power supply supplies power to the laser radar, GPS-RTK and host computer (Intel NUC). The laser radar (RFans-16) is connected with the host computer through an Ethernet interface. The LiDAR driver node converts the original scanning data into a "sensor_msgs / PointCloud2" message in the Robot Operating System (ROS) format, and publishes it to the " / lidar / raw" topic. The GPS-RTK is connected with the host computer through a USB data line. The GPS-RTK driver node converts the original positioning data into a "sensor_msgs / NavSatFix" message in the ROS format, and the published topic is saved as a rosbag file, supporting subsequent model training and optimization. At the same time, the point cloud data topic is subscribed by other downstream tasks of the berthing auxiliary system. The whole process runs stably at a frequency of 10 Hz.

[0111] S2: improving the Point Pillar coding network to build a ship three-dimensional target detection network model, which comprises a point cloud coding network module, a point cloud feature extraction module and a detection head network; wherein the point cloud feature extraction module is preferably a two-dimensional convolutional neural network 2D-CNN; and the detection head network is preferably an SSD detection head of a target detection algorithm;

[0112] The point cloud coding network module is used to perform point cloud coding processing on the shipborne laser radar point cloud to obtain a point cloud pseudo image containing global perception point cloud features and local perception point cloud features of the ship three-dimensional point cloud; the point cloud feature extraction module is used to extract ship three-dimensional point cloud features in the preprocessed point cloud image to obtain a point cloud extraction feature map; and the detection head network is used to realize ship three-dimensional target detection according to the point cloud extraction feature map.

[0113] In specific embodiments, the point cloud coding network module comprises an input layer, a point cloud coding layer, a maximum pooling layer, an average pooling layer, an attention pooling layer and a point cloud pseudo image acquisition module.

[0114] The input layer is used to input the shipborne laser radar point cloud to the point cloud coding layer.

[0115] The point cloud coding layer is used to perform point cloud voxelization operation on the shipborne laser radar point cloud to obtain a point cloud stacked column graph, and perform grid division and coding on the point cloud stacked column graph to obtain a pillar grid feature map with the same size, and perform feature mapping on the pillar grid feature map by calling a multi-layer perceptron MLP to obtain an enhanced feature map.

[0116] The maximum pooling layer is used to perform maximum pooling operation on the enhanced feature map to obtain a local perception point cloud feature map of the ship three-dimensional point cloud; and the average pooling layer is used to perform average pooling operation on the enhanced feature map to obtain a global perception point cloud feature map of the ship three-dimensional point cloud.

[0117] The attention pooling layer is used to predict and obtain local attention scores of each grid feature in the pillar grid feature map based on a multi-layer perceptron MLP to obtain a local attention feature map, and perform element-wise multiplication operation on the local attention feature map and the pillar grid feature map to obtain a fusion feature map.

[0118] The point cloud pseudo image acquisition module is used to perform mean operation on the fusion feature map, the global perception point cloud feature map and the local perception point cloud feature map to obtain a 2D point cloud pseudo image containing ship three-dimensional point cloud features.

[0119] The ship three-dimensional target detection network model constructed in the embodiment projects the original point cloud on the X-Y plane through a simplified Point Net to generate a sparse 2D pseudo image, and uses a top-down network based on a two-dimensional convolutional neural network to process the pseudo image to extract multi-scale features and use the multi-scale features for target detection. However, the original pillar feature network may lose important local geometric information and position information when randomly sampling each pillar, thereby affecting the detection accuracy of the target. Therefore, the embodiment introduces average pooling and attention pooling into the encoding network, and increases the weight of the reflection intensity when calculating the attention score, thereby strengthening the influence of the point cloud reflection intensity on the detection accuracy. In the embodiment, the point cloud encoding network module continues the voxelization method of PointPillar, and divides the 3D space into pillar grids of the same size by using a WxH size grid on the X-Y plane. The original pillar encoding network is as shown in Figure 5 The upper half (in the blue box), during the voxelization process, D is the feature dimension, which represents the extension of the feature dimension of each point in the pillar to 9 dimensions to include the coordinates, reflection intensity, relative offset and center coordinates of each point; C represents the mapping of the features of each pillar to 64-dimensional features through a multilayer perceptron (MLP) to enhance the feature representation.

[0120] The improved PointPillar encoding network in the embodiment is as shown in Figure 3 The upper half is a general network, and the lower half is an improved part (in the red box). Through the maximum pooling encoding module, i.e., the maximum pooling layer, and the average pooling encoding module, i.e., the average pooling layer, the significant features and average features in the pillar are extracted, i.e., the maximum feature value in each pillar is extracted through the maximum pooling operation, and the average value of all point features is calculated through the average pooling, so that a more comprehensive feature representation is obtained. In addition, through the attention pooling encoding module, i.e., the attention pooling layer, the attention scores of all points in the pillar are predicted through the MLP, the important local feature information is retained, and then the attention features of each pillar are obtained through weighted summation, so that the target detection effect is improved. Finally, the average value of the attention pooling features, the maximum pooling features and the average pooling features is calculated to obtain the final pillar features. The feature fusion strategy in the embodiment combines the global perception and local perception information, so that the model can more comprehensively capture the point features inside the pillar, thereby effectively improving the performance of target detection.

[0121] S3: training the ship three-dimensional target detection network model based on the training set and the test set to obtain an optimal detection network model;

[0122] In specific embodiments, the method for obtaining the optimal detection network model comprises:

[0123] S31: training the constructed ship three-dimensional target detection network model using the training set to obtain a trained ship three-dimensional target detection network model:

[0124] S32: based on the constructed model loss function, evaluating the trained ship three-dimensional target detection network model using the test set to determine whether the output of the trained ship three-dimensional target detection network model converges;

[0125] If yes, the trained ship three-dimensional target detection network model is the optimal detection network model;

[0126] Otherwise, based on the back propagation method, the parameter weight of the trained ship three-dimensional target detection network model is adaptively adjusted, and S31 is repeatedly executed;

[0127] The construction formula of the model loss function is

[0128]

[0129] In the formula, L det represents the model loss function; N a represents the total number of ship anchor frames; L cls represents the target category loss; L loc represents the target position loss; L dir represents the target direction loss; L IoU represents the intersection over union prediction loss; L ref represents the reflectivity prediction loss; λ1, λ2, λ3, λ4, λ5 represent loss weights, which are set to 1.0, 2.0, 0.2, 1.0, and 0.8 in this embodiment;

[0130] Specifically, the model loss function includes the following losses:

[0131] (1) Target category loss:

[0132] The target category loss in this embodiment is defined using Focal Loss, and its expression is:

[0133]

[0134] In the formula: Na represents a constant, representing the total number of anchor frames, i.e. the number of anchor frames preset by the model on the feature map; p i and c i respectively represent the target category prediction value and the true value regressed from the i th anchor frame, wherein the true value of 0 represents background and the true value of 1 represents a ship; in fact, p irepresents the class probability and p i The closer it is to 1, the higher the probability that the current target is a ship; conversely, p i The closer it is to 0, the higher the probability that the current target is the background. The hyperparameter a represents a balancing factor to address class imbalance, used to balance the ratio of positive and negative samples. In this embodiment, a is set to 0.25. γ represents a focusing parameter, which aims to make the model pay more attention to difficult and misclassified samples. In this embodiment, the γ value is set to 2.

[0135] (2) Target position loss:

[0136] This embodiment uses the Smooth L1 function L sm (·) Calculate the position loss, which is expressed as:

[0137]

[0138]

[0139] Where: l(c i =1) means only calculating the position loss of the ship target; i With t i They represent the predicted value and true value of the ship target, respectively. Both include 7 dimensions (x, y, z, l, w, h, θ), where x, y, z are the center coordinates of the ship target in the three-dimensional space, l, w, h represent the size of the target, i.e., length, width, and height; θ represents the direction of the target; δ ij With t ij are the predicted value and true value of the jth dimension of the i-th ship target respectively;

[0140] (3) Direction loss:

[0141] This embodiment solves the equation L by introducing directional loss reg (δ i ,t i ) formula, the sine function used to calculate the directional deviation cannot distinguish ±π, and its expression is

[0142]

[0143] Where: h d is a two-dimensional vector, indicating that the direction of the ship target is classified into two categories; ρ i Represents the classification probability of the ship target direction. If the target’s true direction θ i If ρ is non-negative, i is [0, 1], if the target is actually heading towards θ i If ρ is negative, i is [1, 0]; ρ i,kPi,k represents the probability of the ith ship target on the kth class; k represents the index of the direction class; h d,k Pi,k represents the predicted value of the ship target i, which depends on the class of k and is used to represent different possibilities of the target orientation.

[0144] (4) Intersection over Union loss:

[0145] In this embodiment, the Smooth L1 function is used to calculate the Intersection over Union loss, and the expression is as follows:

[0146]

[0147] In the formula, q i represents the IOU predicted value; IoU (δ i ,t i ) represents the Intersection over Union between the predicted anchor frame δ i and the real anchor frame t i , which is an index for evaluating the overlap degree of the predicted frame and the real frame, and the calculation method is to divide the intersection area of the predicted frame and the real frame by the union area of them; V IoU (δ i ,t i ) is the actual Intersection over Union of the target predicted anchor frame and the true value anchor frame; L sm (·) represents the Smooth Ll loss function;

[0148] (5) Reflection intensity loss:

[0149] In this embodiment, the Smooth L1 function is used to calculate the reflection intensity loss, and the expression is as follows:

[0150]

[0151] In the formula, r i represents the reflection intensity predicted value; represents the reflection intensity true value; L sm (·) represents the Smooth Ll loss function;

[0152] S4: According to the optimal detection network model, the ship target is detected for the laser radar point cloud at the preset berth, so as to confirm whether there is an anchored ship in the preset berth according to the detection result;

[0153] If not, the RANSAC plane fitting algorithm is used to obtain the berth front plane for directly parking the ship at the berth;

[0154] If the target ship is present, acquire the three-dimensional point cloud of the target ship parked in the preset berth, and acquire the boundary surface point cloud of the target ship based on the RANSAC plane fitting algorithm to obtain the boundary fitting plane of the target berth, and then confirm the outer contour surface of the target ship according to the boundary fitting plane;

[0155] The berth front edge plane and the outer contour surface are collectively referred to as a parking berth boundary surface.

[0156] Specifically, in busy port waters or shipyards, ships often face the situation of multiple ships and berthing. At this time, the ship needs to judge whether the berth has a ship anchored. If there is a ship anchored, the ship needs to be berthed on one side of the anchored ship, and if it is an empty berth, it can be directly berthed in the berth. The embodiment discusses the target berth boundary plane extraction method based on the laser radar for the two cases, and in order to make up for the defect that the previous research uses a linear fitting-based method to obtain a berth line model as a benchmark for berthing state estimation, thereby causing poor berthing state estimation accuracy, the embodiment introduces a target berth boundary plane model based on the RANSAC plane fitting algorithm to solve this problem.

[0157] Specifically, the following steps are included:

[0158] S41: According to the optimal detection network model, the laser radar point cloud at the preset berth is detected for ship targets to determine whether there is an anchored ship in the preset berth according to the detection result.

[0159] If the target ship is not present, acquire the three-dimensional point cloud of the target ship parked in the preset berth, and acquire the boundary surface point cloud of the target ship based on the RANSAC plane fitting algorithm to obtain the boundary fitting plane of the target berth, and then confirm the outer contour surface of the target ship according to the boundary fitting plane.

[0160] Specifically, the following steps are included:

[0161] S100: According to the relative position of the ship and the target ship, define the three-dimensional point cloud close to the ship side as the boundary surface point cloud set of the target ship.

[0162] S101: Randomly extract three sample point clouds in the boundary surface point cloud set.

[0163] And according to the randomly extracted sample point cloud, solve the model parameters in the spatial plane equation to obtain the fitting spatial plane model, and the expression of the spatial plane equation is

[0164] Ax+By+Cz+D=0

[0165] In the formula: A, B, and C represent three coordinate component values of the unit normal vector of the plane to be solved, and satisfy A 2 +B 2 +C2 = 1 ; D represents the distance from the coordinate origin to the spatial plane; x, y, z are coordinate variables in three-dimensional space, representing the position coordinates of any point on the plane, i.e. the position coordinates of the coordinate sample point cloud;

[0166] S102: Then use all the remaining points (x i ,y i ,z i ) in the ship boundary surface point cloud set to verify the plane model, that is, traverse the remaining point clouds in the boundary surface point cloud set after extracting the sample point cloud, and calculate the distance of each remaining point cloud to the fitted spatial plane model respectively, and the calculation formula is

[0167]

[0168] In the formula: d i represents the distance of the remaining point cloud to the fitted spatial plane model; x i , y i , z i represents the position coordinates of the i-th remaining point cloud;

[0169] S103: Set the distance threshold d t , and mark the remaining point cloud that satisfies d i > d t as an outlier; continue to execute step S104 for the remaining point cloud that satisfies d i ≤ d t ;

[0170] S104: Obtain the normal vector n j of the remaining point cloud and the normal vector n p of the fitted spatial plane obtained based on the fitted spatial plane model, wherein the method for obtaining the normal vector of the fitted spatial plane is a known technology, which will not be described in detail here; and calculate the angle θ j between the normal vectors, and the calculation formula of the angle between the normal vectors is

[0171]

[0172] In the formula: |n p | and |n j | represent the modulus of the normal vector, and the values are both 1;

[0173] S105: Set the angle threshold β, and the remaining point cloud corresponding to θ j > β is an outlier, and the remaining point cloud corresponding to θ j ≤ β is an inner point, and then obtain the inner point set;

[0174] The angle between the normal vectors of the remaining point cloud and the fitted plane in this embodiment is as follows Figure 5As shown, the straight line represents the ideal fitting plane, the straight red arrow represents the normal vector direction of the ideal fitting plane, and the dotted cyan arrow represents the normal vector of the inlier, which is similar to the normal vector direction of the ideal fitting plane. The dotted orange arrow represents the normal vector of the outlier, which is quite different from the normal vector direction of the ideal fitting plane; in theory, the point on the plane, the normal vector of which is parallel to the plane normal vector, is set as an angle threshold β, and the point θ j ≤β is regarded as an inlier; the inlier determination criterion of the method described in the embodiment is that the point which meets the distance threshold requirement of the point to the plane and the angle threshold requirement of the cloud point and the normal vector of the fitting plane is regarded as an inlier, so as to reduce the probability of misidentification;

[0175] S106: repeatedly performing steps S101 to S105 until the random extraction times reach a preset iteration number threshold, to obtain a plurality of inlier sets;

[0176] The iteration number threshold of the RANSAC plane fitting algorithm is determined by the probability of extracting qualified samples, in the embodiment, the model in which the number of inliers meets the requirement is recorded, and the above steps are repeated until the sampling times reach the iteration number threshold, and the iteration is ended; wherein the iteration number threshold of the RANSAC algorithm is determined by the probability of extracting qualified samples, that is, the point on the plane model is a qualified sample, the higher the probability of the three initial sample points selected in the embodiment being qualified samples, the smaller the iteration number, and the preset iteration number threshold is set as

[0177]

[0178] In the formula, ω represents the probability that the three initial sample points selected in the extraction process are all plane points, which can be determined according to the number of the ship outer contour point set; ε represents the probability of selecting a plane point from the boundary surface point cloud set; N Ran represents the iteration number threshold;

[0179] S107: counting and obtaining the number of inliers in each inlier set, and taking the inlier set corresponding to the largest number of inliers as the optimal fitting space plane model, to obtain the boundary fitting plane of the target berth, and then confirming the outer contour surface of the target ship according to the boundary fitting plane;

[0180] If not, the ship can be directly docked at the berth;

[0181] The method of directly docking the ship at the berth specifically includes:

[0182] S200: obtaining the latitude and longitude coordinates of the target berth by the preinstalled navigation system, and obtaining the coordinate difference by subtracting the latitude and longitude coordinates of the position of the ship;

[0183] S201: convert the coordinate difference into a metric coordinate, i.e., a laser radar position coordinate of the target berth, perform neighborhood search with the laser radar position coordinate of the target berth as the center and a preset search radius r to obtain a target berth point cloud set, and determine whether the number of cloud points of the target berth point cloud set meets a preset number threshold;

[0184] If yes, a plane model of a berth front edge is obtained based on a RANSAC plane fitting algorithm according to the target berth point cloud set;

[0185] If no, the search radius is expanded for neighborhood search until the target berth point cloud set obtained by the search meets the preset number threshold, and a plane model of a berth front edge is obtained based on a RANSAC plane fitting algorithm according to the target berth point cloud set, thereby confirming the berth front edge plane;

[0186] In the embodiment, if no ship is docked in the berth area, the latitude and longitude coordinates of the target berth provided by the navigation system are subtracted by the latitude and longitude coordinates of the position of the ship, and then the difference is converted into a metric coordinate, i.e., the X and Y axis coordinates of the laser radar of the target berth, and neighborhood search is performed with the laser radar coordinate of the target berth as the center and a radius r to obtain a target berth point cloud; if the number of target berth point clouds is less than N berth , the search radius is continuously expanded to ensure the model quality of the plane fitting of the target berth boundary, and finally after obtaining the target berth point cloud set, a plane model of a berth front edge is obtained through a RANSAC plane fitting algorithm to obtain the berth front edge plane, and the principle of the berth front edge plane is the same as that of the outer contour surface, which will not be described in detail here;

[0187] S42: the berth front edge plane and the outer contour surface are collectively referred to as a docking berth boundary surface;

[0188] S5: based on the position of the ship-borne laser radar, a laser radar docking distance from the docking berth boundary surface is obtained, and the bow position and the stern position of the ship are obtained according to the laser radar docking distance, and the bow docking distance and the stern docking distance from the docking berth boundary surface are obtained;

[0189] Based on the bow docking distance and the stern docking distance, the docking speed of the ship is obtained according to the sampling time of the ship-borne laser radar, and the docking speed of the ship includes the bow docking speed and the stern docking speed, thereby realizing estimation of the docking state of the ship.

[0190] Specifically, when berthing in the shipyard port area, there are many berthing state parameters worth paying attention to, including sailing speed, berthing angle and berthing distance, etc. In this embodiment, by combining the practice case of shipyard berthing operation, the berthing state parameters most concerned by the driver are summarized as the distance and speed of approaching the berth. And during the maintenance of large ships in the shipyard, the draft is particularly shallow, which can easily cause the propeller to be not fully submerged underwater, so the maneuverability of the ship itself is poor. As shown in Figure 6 , most of the ships need two tugboats to assist during berthing, and the positions of the two tugboats are generally located at the bow and stern of the ship, respectively. Therefore, the driver frequently pays attention to the distance and speed of the bow and stern approaching the berth during berthing in order to command the tugboat to push or pull the large ship;

[0191] Specifically, the method comprises the following steps:

[0192] S51: Based on the position of the ship-borne laser radar, the laser radar berthing distance from the boundary surface of the berthing berth is obtained, and the acquisition formula of the laser radar berthing distance is:

[0193] d lidar =f(0,0,0);

[0194] In the formula, d lidar represents the laser radar berthing distance; f(0,0,0) represents the distance from the point (0,0,0) in the laser radar coordinate system to the boundary surface of the berthing berth;

[0195] S52: According to the laser radar berthing distance, the bow berthing distance and the stern berthing distance of the bow and the stern of the ship from the boundary surface of the berthing berth are obtained;

[0196] And the acquisition formula of the bow berthing distance and the stern berthing distance is:

[0197] d bow =f(Δ B ,Δ L1 ,0)

[0198] d stern =f(Δ B ,Δ L2 ,0)

[0199] In the formula, d bow represents the bow berthing distance; d stern represents the stern berthing distance; Δ B represents the included angle between the straight line of the ship body passing through the center of the laser radar and the line segment of the laser radar berthing distance along the bow and stern directions; Δ L1 represents the distance from the laser radar center to the bow; and Δ L2 represents the distance from the laser radar center to the stern;

[0200] When the ship gradually approaches the target ship in this embodiment, the distances d bow and d stern from the bow and the stern of the ship to the target ship's outer contour surface can be calculated according to the target ship's outer contour surface obtained by the berth boundary plane extraction module Figure 7 , as shown in the schematic diagram; the distance d lidar from the position of the ship-borne laser radar to the target ship's outer contour surface bow and d stern need to be calculated once again, where f (Δ B , Δ L1 , 0) represents the distance from the point (Δ B , Δ L1 , 0) in the laser radar coordinate system to the plane model; f (Δ B , Δ L2 , 0) represents the distance from the point (Δ B , Δ L2 , 0) in the laser radar coordinate system to the plane model; Δ B , Δ L1 , Δ L2 is the external parameter of the laser radar, which needs to be calibrated according to the installation position of the laser radar on the ship and the size of the ship body;

[0201] S53: based on the bow approaching distance and the stern approaching distance, the ship approaching speed is obtained according to the point cloud sampling time of the ship-borne laser radar;

[0202] wherein the ship approaching speed includes the bow approaching speed and the stern approaching speed;

[0203] and the expressions of the bow approaching speed and the stern approaching speed are

[0204]

[0205] In the formula: Δt represents the time variable obtained by subtracting the time stamp of the i+1th frame from the i th frame, that is, the time stamp of the i+1th frame is subtracted from the i th frame and the unit is converted into seconds; v bow , v stern represent the bow approaching speed and the stern approaching speed, respectively; represent the bow approaching distance and the stern approaching distance corresponding to the i th frame of point cloud, respectively; represent the bow approaching distance and the stern approaching distance corresponding to the i+1th frame of point cloud, respectively;

[0206] Further, the ship approaching state is estimated according to the ship approaching speed.

[0207] The application examples of the embodiment specifically include:

[0208] (1) Test environment and test equipment

[0209] The berthing perception system proposed in this embodiment is tested by Zhi Long 1 unmanned ship in port waters (Ling Shui Port of Dalian Maritime University) and inland waters (a wharf in Haihe River, Tianjin, China). The test site is shown in Figure 8 , wherein Figure 8 (a) represents a top view of a certain section of Ling Shui Port;

[0210] Figure 8 (b) represents a top view of the wharf (b) in the section of the Haihe River in Tianjin. As shown in Figure 9 , the length of Zhi Long 1 is 1.75 m and the width is 1.5 m. The ship is equipped with RFans-16 mechanical laser radar, RTK, and binocular camera. The berthing perception system of this embodiment is developed based on ROS1 on a computer with 32 GB RAM and NVIDIA Ge Force GTX4060Ti (16 GB) graphics card of Intel Core i5-11600KF. The test of this embodiment is divided into two parts: berth boundary plane extraction test and berthing state estimation test. Before introducing the specific test results, several professional terms in berthing operation need to be explained. Figure 8 There are three wharfs in (b), of which No. 2 and No. 3 are empty, and No. 1 has berthed two ships. If the ship is moved from No. 3 to No. 2, such operation from empty berth to empty berth is not different from berthing. Due to the need to repair the ship, the second ship is moved from No. 2 to the outer edge of No. 1, which is different from the conventional berthing operation, and the point cloud of the second ship needs to be extracted from the laser point cloud and the outer contour plane is extracted as the target berth boundary plane. The target ship described in the experimental analysis refers to the second ship in Figure 8 , and the target berth refers to the berth area of the outer edge of the second ship, i.e. Figure 8 the red box of the outer edge of the second ship in (b).

[0211] (2) Test results of three-dimensional target detection of ship

[0212] The target berth boundary plane extraction depends on the ship three-dimensional target detection algorithm, and the ship point cloud three-dimensional target detection is completed based on the improved PointPillar detection network in the embodiment. Although the training depending on the virtual data set has been able to achieve good detection accuracy, the real ship data set is still indispensable for improving the generalization performance of the model. In terms of data set construction, the intelligent dragon No. 1 is used to carry out 16-line laser radar in Lingshui port and Tianjin Haihe inland navigation, and real ship three-dimensional point cloud data is collected. The collected laser point cloud data set needs to go through three-dimensional labeling to generate a true value label file. Since most point cloud-based detection networks are developed based on the KITTI data set. Therefore, in order to facilitate the use of the data set of the present application to conduct comparative experiments, the Supervisely platform is used to perform supervised target labeling according to the labeling format of the KITTI data set. In addition, the present application also introduces the existing public real ship data set into the training set of the embodiment, and the data set scene includes the Thames River and the port, which is described in detail in Figure 10 . It is worth mentioning that the above data sets are collected by 16-line laser radar and the labeling format and target classification are the same, and the data set division details are shown in Table 1;

[0213] Table 1. Data set details

[0214]

[0215] Since the marine laser radar data set is usually not labeled with the truncation degree, the occlusion degree, the observation angle and the two-dimensional target frame. When using the improved PointPillar detection network for training, the field usually not considered is set to a constant. This processing method will undoubtedly reduce the accuracy and precision of target detection. In order to improve the detection accuracy of the marine laser radar data set in the improved PointPillar detection network, the present application introduces the point cloud reflection intensity in the true value label file. First, the point cloud in the frame is extracted according to the size and position of the target three-dimensional detection frame in the true value label file, and then the average reflection intensity of the point cloud in the frame is calculated. The average reflection intensity value is used as a field of the true value label file.

[0216] Based on the method proposed in the embodiment, the accuracy of target berth area recognition depends on the accuracy of ship three-dimensional target recognition in the case of having a ship in the berth. Therefore, the ship three-dimensional target recognition module based on the improved Pointpillar detection network is verified first. The experiment is implemented based on the Open PC Det framework, which can train the improved PointPillar algorithm. The software environment is: Ubuntu 20.04LTS, Python 3.8, Cuda11.1, PyTorch1.10; the network structure is implemented using the PyTorch framework, training and testing are performed using GPU, the end-to-end training is implemented using the Adam optimizer, the batch size is 8, the weight decay value is set to 0.01, the momentum value is set to 0.9, the learning rate decay value is set to 0.1, and the maximum iteration number is set to 120. Then, the improved PointPillar network is trained using the combination of the training data set in Table 1, and the detection performance of the improved PointPillar detection network is verified on the test data set as shown in Table 2. The results show that compared with the results reported in the original paper introducing the PointPillar detection network, the detection performance is improved by 18.7% and 14.3% respectively.

[0217] Table 2. Evaluation results of PointPillar detection network

[0218]

[0219] To verify the accuracy of the ship three-dimensional target detection model proposed in the method, a ship three-dimensional target detection experiment is performed in the Lingshui Port. The sailing trajectory of the ship in the experiment is shown by the red trajectory line in Figure 11 . During the voyage, 1 large ship target, 6 small ship targets and several small ship targets on the shore are encountered on the water surface. Due to the high frequency output of the laser radar, the change across adjacent frames is small. Therefore, the target detection results in 6 frames of point clouds with equal time intervals are selected to visualize the performance of the model, and the visualization results are as shown in Figure 12 . The information of the point cloud intensity is used as the color of the point cloud, and blue and green have been used as the color of the point cloud, so red is used as the color of the three-dimensional target detection box. Taking the large ship in the red circle in Figure 13 as an example to analyze the effect of the target detection model. Before 60 seconds, since the ship is far away from the large ship, fewer point clouds are collected, and the point clouds are all the head point clouds of the large ship, and the model detects the point clouds as small ships; after 60 seconds, as the ship approaches the large ship, the point clouds of the large ship are gradually displayed, and the model identifies them as large ships. In addition, it can be found in Figure 12 that the model can also identify some small ships parked on the shore, and in fact the ships on the shore are not labeled.

[0220] (3) Target berth boundary plane extraction

[0221] The above experiments fully verify the accuracy of the ship target detection model. Next, based on the model, a target berth boundary plane extraction experiment is conducted on a certain section of the Tianjin Haihe River to verify the generalization of the target detection model and the effect of the berth boundary plane extraction. The experiment described in this embodiment is divided into two parts: to verify the effect of the ship on the boundary plane extraction of the anchored ship and the empty berth.

[0222] A. Berth boundary plane extraction based on anchored ship:

[0223] Figure 14 The process of obtaining the berth boundary plane extraction in the experiment is shown. In Figure 14 , when the ship approaches the target berth, the point cloud shows the profile of the wharf front (black contour line), the first ship (green contour line) and the second ship (blue contour line) parked at the wharf. The purpose of the ship is to berth to the outer edge of the second ship. The first thing to do is to obtain the boundary plane point cloud of the second ship, and then solve the boundary plane model based on the RANSAC plane fitting algorithm. Figure 15 The target ship point cloud obtained based on the ship three-dimensional target detection model is shown. Since there are direction errors and position errors in the three-dimensional detection frame of the ship, the three-dimensional detection frame cannot be directly used as the berth boundary plane of the ship. The method described in this embodiment extracts the plane model of the ship outer contour surface according to the ship point cloud obtained in the detection frame, and uses the extracted plane model as the berth boundary plane. According to the relative position of the ship and the berth, the ship outer contour point cloud close to the ship side in the ship point cloud is extracted, and the target berth boundary plane model of the ship can be determined based on the extracted ship outer contour point cloud using the RANSAC plane fitting algorithm.

[0224] B. Berth boundary plane extraction based on empty berth:

[0225] Compared with the occupied berth in A, the target berth area identification of the empty berth is relatively simple. Only a part of the point cloud of the berth needs to be obtained according to the berth coordinates, and then the RANSAC plane fitting algorithm is used to obtain the berth boundary plane model. Taking the navigation experiment of a certain section of the Tianjin Haihe River as an example, the target berth boundary plane extraction of the empty berth of the ship is analyzed. During the process of the ship approaching berth No. 2, the ship three-dimensional target detection model does not detect a ship near the berth coordinates, which can be determined as an empty berth. Then, the berth coordinate point cloud is extracted with the berth coordinates as the center, and the berth front point cloud is extracted according to the relative position of the berth and the ship. The approximate boundary plane model of the berth is obtained based on the RANSAC plane fitting algorithm and the berth point cloud. An example of empty berth boundary plane extraction is shown in Figure 16As shown in the figure, the gray plane is the plane model of the front of berth No. 2, based on which the berthing status information can be accurately calculated.

[0226] (4) Berthing status estimation

[0227] by Figure 17 Taking the navigation test of the berthing state estimation test as an example, the ship departs from the berth at position 3 (38.999401, 117.701291) and returns to the berth at position 2 (38.999184, 117.701431) after making a circle. The distance between the bow and stern of the target ship and the berth boundary surface near berth 1 is as follows: Figure 18 .in, Figure 18 The distance between the starboard bow and stern of the own ship and the outer contour of the target ship is recorded when the own ship approaches and leaves the target ship at berth 1. When the own ship just departs from berth 3, the berth area cannot be identified. When approaching berth 1, the outer contour of the target ship is identified and the distance between the starboard bow and stern of the own ship and the outer contour of the target ship is calculated, until the own ship turns left and moves away from berth 1 and the target ship is no longer identified. Figure 17 It can be found that when approaching target ship No. 1, the distance between the ship and the target ship is about 10 meters to 15 meters, and then gradually moves away from the target ship. When the distance is about 46 meters, the target ship cannot be identified. This is consistent with the Figure 17 The trajectory diagram between the central horizontal coordinates (-130, -220) is very consistent.

[0228] Figure 19 Displays the distance between the bow, stern and laser radar and the berth plane when the ship is berthing at berth No. 2. The final berthing distance is stable at 0.3 meters, and the ship has completed berthing. In addition, Figure 19 The corresponding Figure 17 The trajectory diagram between the horizontal coordinates (-40, 10) shows that the ship approaches the berth at a large angle and finally becomes parallel to the berth. This berthing state will cause the distance between the stern and the berth to be greater than the bow. Later, the distance between the bow and the berth is the same. Figure 19 Got confirmed.

[0229] In order to verify the accuracy of the berthing parameter calculation based on LiDAR in this implementation, Figure 19 Comparison of the berthing distance d calculated based on RTK data rtk After zooming in locally, it can be found that the distance measurement based on lidar and the distance measurement based on RTK are very close. Figure 20 The errors of LiDAR ranging and RTK ranging were further evaluated, and the root mean square error (RMSE) of the two was calculated to be 0.1354 meters, which can meet the needs of ship berthing operations. Figure 20The error curve of the second group of tests can be found that when the ship is far away from the target berth, the error is greater than 0.2 meters, and when approaching the berth, the error gradually decreases to less than 0.1 meters. This phenomenon may be that the ship approaches the berth front with a small angle, at which time the berth boundary recognition is not stable, and as the ship approaches the berth front, the berth area recognition tends to be stable, and the ranging error also tends to be stable. In order to verify the generalization of the berthing perception system of the method described in this embodiment, the second group of tests, the ship sets off from No. 3 berth to berth on the outside of the target ship (second-class ship) in No. 1 berth, the sailing track is as shown in Figure 21 The sailing track diagram of the ship setting off from No. 3 berth to berth on the side of the red ship in No. 1 berth. The starting point and the berth coordinates of the test are (38.999401, 117.701291) and (39.000259, 117.698883) respectively, and since the distance is short, the scale in the track diagram has been converted to meter coordinates. Figure 22 The distance and speed curves of the ship and the berth boundary surface calculated after the ship recognized the berth area are shown respectively. Figure 21 It can be known that the ship sets off directly to the target ship, turns left to berth on the outer contour of the target ship after approaching the target ship, and adjusts the sailing attitude of the ship. Figure 22 As shown in Figure 22 (a) represents the distance curve of the ship from the outer contour of the target ship in the second group of tests, Figure 22 (b) represents the speed curve of the ship from the outer contour of the target ship in the second group of tests, and it can also be seen that the ship has turned several times during approaching the target ship, and at 160 seconds, it is almost parallel to the outer contour of the target ship, at which time the distance from the outer contour of the target ship is about 0.75 meters, and the speed has decreased to 0.

[0230] The method has the beneficial effects that: the ship three-dimensional target detection network model constructed by improving the PointPillar coding network, through the introduction of the maximum pooling layer, the average pooling layer and the attention pooling layer, improves the fusion of the information of the global perception and the local perception of the target, ensures that the model can more comprehensively capture the point cloud features inside the pillar, and according to the model loss function constructed, the model is trained, and then the performance and detection precision of the target detection are effectively improved; through the optimal detection network model, whether the preset berth exists the anchored ship is detected and confirmed, and through the introduction of the RANSAC plane fitting algorithm, the required berth front plane or the outer contour plane of the target ship, i.e. the berthing berth boundary plane, when the preset berth exists or does not exist the anchored ship is obtained, and through the position of the shipborne laser radar and the berthing berth boundary plane, the berthing speed used for the berthing state estimation is obtained; the defect that the berthing state estimation accuracy is poor caused by the traditional method of using the linear fitting based berth line model as the berthing state estimation reference is solved, the berthing precision of the berthing perception system is greatly improved, and the applicability and expansibility of the berthing perception system are enhanced.

[0231] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for estimating berthing state based on shipborne laser radar target detection, characterized in that: The specific steps include: S1: Collect and obtain shipborne lidar point cloud; The ship-borne lidar point cloud is annotated with ship anchor frames and ship categories to obtain a pre-processed point cloud map, which is then randomly divided into training and test sets. S2: Improve the PointPillar encoding network to build a 3D ship target detection network model, which includes a point cloud encoding network module, a point cloud feature extraction module, and a detection head network; The ship-borne lidar point cloud is processed through the point cloud coding network module to obtain a point cloud pseudo-image that contains the global perception point cloud features and local perception point cloud features of the fused ship's 3D point cloud. The point cloud feature extraction module extracts the ship's 3D point cloud features from the pre-processed point cloud image to obtain a point cloud extraction feature map. The detection head network detects the 3D ship target based on the point cloud extraction feature map. S3: Train the constructed ship 3D target detection network model based on the training set and the test set to obtain the optimal detection network model; S4: Detecting ship targets on the LiDAR point cloud at the pre-berth according to the optimal detection network model, and confirming whether there is an anchored ship at the pre-berth based on the detection results; If it does not exist, the berth front plane for docking the ship directly at this berth is obtained based on the RANSAC plane fitting algorithm; If it exists, the 3D point cloud of the target ship docked at the preset berth is obtained, and the boundary surface point cloud of the target ship is obtained based on the RANSAC plane fitting algorithm to obtain the boundary fitting plane of the target berth, and then the outer contour surface of the target ship is confirmed according to the boundary fitting plane; The berth front plane and the outer contour surface are collectively referred to as the berth boundary surface; S5: Based on the position of the ship's onboard laser radar, obtain the laser radar berthing distance from the berthing space boundary surface; and obtain the bow and stern positions of the ship, and the bow and stern berthing distances from the berthing space boundary surface based on the laser radar berthing distance; Based on the bow berthing distance and the stern berthing distance, the berthing speed of the ship is obtained according to the sampling time of the ship-borne lidar; and the berthing speed of the ship includes the bow berthing speed and the stern berthing speed, and then the berthing status of the ship is estimated according to the berthing speed of the ship.

2. The method for estimating berthing state based on shipborne laser radar target detection according to claim 1, characterized in that: The point cloud coding network module in S2 includes input layer, point cloud coding layer, maximum pooling layer, average pooling layer, attention pooling layer and point cloud pseudo image acquisition module; The input layer is used to input the shipborne lidar point cloud into the point cloud encoding layer; The point cloud encoding layer is used to perform a point cloud voxelization operation on the shipborne lidar point cloud to obtain a point cloud stacked column map, and then grid the point cloud stacked column map and encode it to obtain a pillar grid feature map of the same size. The pillar grid feature map is feature mapped by calling a multi-layer perceptron (MLP) to obtain an enhanced feature map. The maximum pooling layer is used to perform a maximum pooling operation on the enhanced feature map to obtain a local perception point cloud feature map of the ship's three-dimensional point cloud; the average pooling layer is used to perform an average pooling operation on the enhanced feature map to obtain a global perception point cloud feature map of the ship's three-dimensional point cloud; The attention pooling layer is used to predict and obtain the local attention score of each grid feature in the pillar grid feature map based on the multi-layer perceptron MLP to obtain a local attention feature map, and perform an element-by-element multiplication operation on the local attention feature map and the pillar grid feature map to obtain a fused feature map; The point cloud pseudo image acquisition module is used to perform a mean operation on the fusion feature map, the global perception point cloud feature map and the local perception point cloud feature map, thereby obtaining a 2D point cloud pseudo image containing the three-dimensional point cloud features of the ship.

3. The method for estimating berthing state based on shipborne laser radar target detection according to claim 2, characterized in that: The point cloud feature extraction module in S2 is preferably a two-dimensional convolutional neural network 2D-CNN; the detection head network is preferably an SSD detection head of the target detection algorithm.

4. The method for estimating berthing state based on shipborne laser radar target detection according to claim 3, characterized in that: The method for obtaining the optimal detection network model in S3 specifically includes: S31: Use the training set to train the constructed ship 3D target detection network model to obtain the trained ship 3D target detection network model: S32: Based on the constructed model loss function, the trained ship 3D object detection network model is evaluated using the test set to determine whether the output of the trained ship 3D object detection network model converges; If so, the trained ship 3D target detection network model is the optimal detection network model; Otherwise, based on the back propagation method, the parameter weights of the trained ship three-dimensional target detection network model are adaptively adjusted, and S31 is repeatedly executed.

5. The method for estimating berthing state based on shipborne laser radar target detection according to claim 3, characterized in that: The formula for constructing the model loss function in S32 is: Where: L det Represents the model loss function; N a Indicates the total number of ship anchor frames; L cls represents the target category loss; L loc represents the target position loss; L dir Indicates target direction loss; L IoU represents the intersection-over-union prediction loss; L ref Represents the reflection intensity prediction loss; λ1,λ2,λ3,λ4,λ5 represent the loss weights; L cls (p i ,c i ) represents an intermediate variable; p i ,c i They represent the target category prediction value and true value regressed from the i-th anchor box respectively; a represents the balance factor; γ represents the focus parameter; L reg (δ i ,t i ) represents an intermediate variable; l(c i =1) indicates the position loss of the ship target; δ i ,t i Represent the predicted anchor frame and the real anchor frame of the ship target respectively; x, y, z represent the center coordinates of the ship target in the three-dimensional space; l, w, h represent the size of the ship target, i.e., length, width, and height; θ represents the direction of the ship target; δ ij ,t ij They represent the predicted value and true value of the i-th ship target in the j-th dimension respectively; L sm (δ ij -t ij ) is L sm (g) represents the intermediate variable; L dir (h d ,ρ i ) represents an intermediate variable; h d Represents the two-dimensional vector of the ship target direction being classified; ρ i Indicates the classification probability of the ship target direction; ρ i,k represents the probability of the i-th ship target in the k-th category; k represents the index of the ship target direction category; h d,k Indicates the predicted value of the ship target’s direction being divided into k categories; q i Represents the IOU prediction value; IoU(δ i ,t i ) represents the predicted anchor box δ i and the real anchor box t i The intersection-over-union ratio between IoU (δ i ,t i ) represents the actual intersection-over-union ratio between the predicted anchor box and the true anchor box; L sm (·) represents the Smooth Ll loss function; r i represents the predicted value of reflection intensity; Indicates the true value of reflection intensity.

6. The method for estimating berthing state based on shipborne laser radar target detection according to claim 5, characterized in that: The S4 specifically includes the following steps: S41: Detecting ship targets on the lidar point cloud at the pre-positioned berth according to the optimal detection network model, so as to confirm whether there is an anchored ship at the pre-positioned berth according to the detection results; If it does not exist, the 3D point cloud of the target ship docked at the preset berth is obtained, and the boundary surface point cloud of the target ship is obtained based on the RANSAC plane fitting algorithm to obtain the boundary fitting plane of the target berth, and then the outer contour surface of the target ship is confirmed according to the boundary fitting plane; The specific steps include: S100: Based on the relative positions of the own ship and the target ship, a three-dimensional point cloud close to the own ship is defined as a boundary surface point cloud set of the target ship; S101: Randomly extract three sample point clouds from the boundary surface point cloud set; And according to the randomly selected sample point cloud, the model parameters in the space plane equation are solved to obtain the fitting space plane model, and the expression of the space plane equation is Ax+By+Cz+D=0 Where: A, B, C represent the three coordinate components of the plane unit normal vector to be solved, satisfying A 2 +B 2 +C 2 =1; D represents the distance from the coordinate origin to the spatial plane; x, y, z represent the position coordinates of the sample point cloud; S102: traverse the remaining point clouds in the boundary surface point cloud set after extracting the sample point cloud, and calculate the distance between each remaining point cloud and the fitting space plane model. The calculation formula is: Where: d i Indicates the distance from the remaining point cloud to the fitted space plane model; x i ,y i ,z i Represents the position coordinates of the i-th remaining point cloud; S103: Setting distance threshold d t , and will satisfy d i >d t The remaining point cloud is recorded as the external point; i ≤d t The remaining point clouds continue to execute step S104; S104: Get the normal vector n of the remaining point cloud j and the normal vector n of the fitting space plane obtained based on the fitting space plane model p , and calculate the normal vector angle θ j And the calculation formula of the normal vector angle is S105: Set the angle threshold β and set the angle that satisfies θ j >β The remaining point cloud corresponding to is the outlier, which will satisfy θ j The remaining point cloud corresponding to ≤β is the inlier point, and then the inlier point set is obtained; S106: Repeat steps S101 to S105 until the number of random sampling reaches a preset iteration threshold, and obtain a number of interior point sets; The iteration threshold of the RANSAC plane fitting algorithm is determined by the probability of extracting qualified samples, and the setting formula of the preset iteration threshold is: Where: ω represents the probability that the three initial sample point clouds selected in the extraction process are all plane points; ε represents the probability of selecting a plane point from the boundary surface point cloud set; N Ran Indicates the iteration number threshold; S107: Count and obtain the number of inliers in each inlier set, and use the fitting space plane model corresponding to the inlier set with the largest number of inliers as the optimal fitting space plane model to obtain the boundary fitting plane of the target berth, and then determine the outer contour surface of the target ship based on the boundary fitting plane; If it does not exist, the ship can be docked directly at this berth; The method of directly berthing the vessel at this berth includes: S200: Obtain the longitude and latitude coordinates of the target berth by using a preset navigation system, and subtract the longitude and latitude coordinates of the own ship's location to obtain a coordinate difference; S201: Convert the coordinate difference into metric coordinates, i.e., the laser radar position coordinates of the target berth. Perform a neighborhood search with a preset search radius r, centered on the laser radar position coordinates of the target berth, to obtain a target berth point cloud set. Determine whether the number of cloud points in the target berth point cloud set meets a preset threshold. If the conditions are met, the plane model of the berth front is obtained based on the target berth point cloud set based on the RANSAC plane fitting algorithm; If not, the search radius is expanded to perform neighborhood search until the target berth point cloud set obtained by the search meets the preset number threshold. Based on the RANSAC plane fitting algorithm, the plane model of the berth front is obtained according to the target berth point cloud set, and the berth front plane is confirmed. S42: The berth front plane and the outer contour surface are collectively referred to as the berth boundary surface.

7. The method for estimating berthing state based on shipborne laser radar target detection according to claim 6, characterized in that: The S5 specifically includes the following steps: S51: Based on the location of the ship's onboard laser radar, obtain the laser radar berthing distance from the berthing boundary surface, and the formula for obtaining the laser radar berthing distance is: d lidar =f(0,0,0); Where: d lidar Indicates the berthing distance of the laser radar; f(0,0,0) represents the distance from the (0,0,0) point in the laser radar coordinate system to the boundary surface of the berthing space; S52: Obtaining the bow berthing distance and the stern berthing distance of the own ship from the berthing space boundary surface respectively based on the laser radar berthing distance; The formula for obtaining the bow berthing distance and the stern berthing distance is: d bow =f(Δ B ,D L1 ,0) d stern =f(Δ B ,D L2 ,0) Where: d bow Indicates the bow berthing distance; d stern Indicates the stern berthing distance; Δ B Δ represents the angle between the straight line along the bow and stern direction passing through the center of the lidar and the line segment of the lidar berthing distance; L1 Indicates the distance between the center of the laser radar and the bow; Δ L2 Indicates the distance from the center of the lidar to the stern; S53: Based on the bow berthing distance and the stern berthing distance, the berthing speed of the own ship is obtained according to the point cloud sampling time of the ship-borne laser radar; Among them, the berthing speed of the ship includes the bow berthing speed and the stern berthing speed; The expressions of bow berthing speed and stern berthing speed are: Where: Δt represents the time variable obtained by subtracting the timestamps of the i+1th and i-th frame point cloud samples; v bow ,v stern They represent the bow berthing speed and the stern berthing speed respectively; They represent the bow berthing distance and stern berthing distance corresponding to the i-th frame point cloud respectively; They represent the bow berthing distance and stern berthing distance corresponding to the i+1th frame point cloud respectively; Then, the berthing status of the ship is estimated according to the berthing speed of the ship.

Images