Multi-sensor fused ship berthing sensing auxiliary method and system

Abstract

The invention provides a multi-sensor fusion ship berthing sensing auxiliary method and system, and the method comprises the steps: obtaining ship-borne laser radar point cloud data and ship-borne IMU odometer data, carrying out the point cloud motion compensation of the ship-borne laser radar point cloud data, and issuing the compensation; feature point clouds of the point cloud data are extracted, and shipborne laser radar point cloud data integrating the feature point clouds are issued; acquiring shipborne IMU data and shipborne laser radar odometer data, integrating the shipborne IMU data in real time, and issuing a shipborne IMU odometer; acquiring shipborne RTK data, detecting and judging a satellite signal state, and issuing shipborne RTK odometer data; acquiring shipborne RTK odometer data and shipborne laser radar point cloud data integrated with feature point cloud, calculating and optimizing the current state pose of a ship, issuing a shipborne laser radar odometer and reconstructing a three-dimensional navigation scene; a hardware external parameter calibration result is obtained, a ship-borne IMU odometer coordinate system, a ship-borne laser radar odometer coordinate system and a ship-borne RTK odometer coordinate system are unified, and a three-dimensional navigation scene and all odometer data are visualized.

Description

Technical Field

[0001] The present invention relates to the field of ship berthing perception and the field of ship autonomous positioning technology. Specifically, it particularly relates to a method and system for ship berthing perception assistance based on multi-sensor fusion. Background Art

[0002] Perceiving the navigation scene information is the primary link for a ship to perform autonomous berthing operations. The multi-sensor fusion perception method is very suitable for application in the scenarios of ship berthing and inshore navigation. In recent years, the equipment of high-precision perception devices has promoted the process of ships achieving autonomous driving and remote control. The data processing method will directly affect the ship's ability to perceive the surrounding targets during the berthing stage. During the ship's autonomous berthing process, first, it is necessary to perceive the ship's own state and the information of surrounding obstacles. Second, it is necessary to construct the navigation scene information required for the collision avoidance system and the navigation system. Finally, it is necessary to input the pose information and positioning information into the decision-making system for autonomous berthing control.

[0003] In the existing ship berthing assistance systems, most rely on a single perception device to complete the berthing task, and are easily affected by external factors such as the environment and weather. The distance between the ship and the berth is directly or indirectly calculated, and the ship's entry port attitude is measured, but the relative state picture between the ship and the dock cannot be intuitively obtained. Ships mainly use the electronic chart system to obtain navigation environment information and cannot construct a three-dimensional navigation scene in real time. Ship positioning mainly relies on marine radars and GNSS devices, and there is a problem of time sequence synchronization between the obtained geographical location information and the self-attitude data collected by the gyroscope. Therefore, to achieve ship autonomous berthing, the current perception means can no longer meet the actual needs of the ship berthing assistance system. Summary of the Invention

[0004] In view of the above-mentioned technical problems in the prior art, such as limited perception accuracy, single type of equipment, insufficient positioning ability, accumulation of positioning errors, and inability to meet the requirements of multi-dimensional data fusion, a method and system for ship berthing perception assistance based on multi-sensor fusion are provided.

[0005] The technical means adopted by the present invention are as follows:

[0006] A method for ship berthing perception assistance based on multi-sensor fusion includes:

[0007] S1. Obtain the on-board lidar point cloud data and the on-board IMU odometer data, perform point cloud motion compensation on the on-board lidar point cloud data, and publish the corrected on-board lidar point cloud data;

[0008] S2. Obtain the published on-board lidar point cloud data, extract the feature point cloud, and publish the on-board lidar point cloud data integrated with the feature point cloud;

[0009] S3. Obtain the on-board IMU data and the on-board lidar odometer data, integrate the on-board IMU data in real time, and publish the on-board IMU odometer;

[0010] S4. Obtain the on-board RTK data, detect and judge the satellite signal status, and publish the on-board RTK odometer data;

[0011] S5. Obtain the on-board RTK odometer data and the on-board lidar point cloud data integrating feature points, calculate and optimize the current state pose of the ship, and publish the on-board lidar odometer and the reconstructed three-dimensional navigation scene;

[0012] S6. Obtain the calibration result of the hardware extrinsic parameters, unify the coordinate systems of the on-board IMU odometer, the on-board lidar odometer and the on-board RTK odometer, and visualize the three-dimensional navigation scene and all the odometer data.

[0013] Further, step S1 specifically includes:

[0014] S11. Obtain the on-board lidar point cloud and the start and end timestamps, traverse the current on-board IMU data in real time and integrate it, initialize the attitude angle of the current on-board lidar point cloud frame, and calculate the ship rotation compensation matrix using the spherical interpolation algorithm;

[0015] S12. Obtain the on-board IMU odometer data, calculate the ship translation compensation matrix using the bilinear interpolation algorithm, and perform motion compensation on the on-board lidar point cloud data;

[0016] S13. Store and publish the effective on-board lidar point cloud data and position index after motion compensation.

[0017] Further, in step S11, the spherical interpolation algorithm used is specifically as follows:

[0018]

[0019] In the formula, \(R(t)\) is the ship rotation compensation matrix at time \(t\), \(R i is the point cloud rotation matrix at the \(i\)-th moment, and \(\theta\) is the three-axis rotation component obtained by pre-integration.

[0020] Further, in step S12, the bilinear interpolation algorithm used is specifically as follows:

[0021]

[0022] In the formula, \(T(t)\) is the ship translation compensation matrix at time \(t\), \(T i is the point cloud displacement matrix at the \(i\)-th moment, \(p\) is the three-axis displacement component obtained by pre-integration, \(r F and \(r B are the interpolation ratios, and \(t\) is the point cloud timestamp.

[0023] Further, step S2 specifically includes:

[0024] S21. Obtain the on-board lidar point cloud data released in step S1, calculate the curvature θ of each laser point, and judge the feature point cloud, including edge points and plane points;

[0025] S22. If θ≥0.6, it is determined as an edge point; otherwise, it is determined as a plane point;

[0026] S23. Release the on-board lidar point cloud data integrated with feature point cloud.

[0027] Further, step S3 specifically includes:

[0028] S31. Obtain the current on-board IMU data and perform initialization operations;

[0029] S32. Obtain the timestamp and pose of the on-board lidar odometer and perform IMU pre-integration;

[0030] S33. Release the on-board IMU odometer data.

[0031] Further, step S4 specifically includes:

[0032] S41. Obtain the serial port signal and perform initialization operations;

[0033] S42. Convert the original NMEA data collected by the on-board RTK receiver into ROS standard format data;

[0034] S43. Detect and judge the satellite signal status, including detecting the satellite signal status and judging the position covariance; use the status detection algorithm to detect whether there is enhanced positioning information and judge whether the position covariance meets the system requirements, specifically as follows:

[0035]

[0036] In the formula, RTK fix is the RTK signal status, status = 0 indicates no enhanced positioning; status = 1 indicates satellite enhanced positioning; status = 2 indicates ground-based enhanced positioning; status = -1 indicates unable to position;

[0037] S44. If the satellite signal status detection is qualified, release the on-board RTK odometer; otherwise, reject the release and wait for the differential correction data according to the preset time If it is determined that the system has not reached the ideal state for a long time, and a warning is sent.

[0038] Further, step S5 specifically includes:

[0039] S51. Obtain the on-ship RTK odometer and the on-ship lidar point cloud data integrated with feature point clouds;

[0040] S52. Perform point cloud matching and calculate the current state pose of the ship;

[0041] S53. Perform backend optimization to optimize the current state pose of the ship;

[0042] S54. Perform loop closure detection to optimize the global state pose of the ship;

[0043] S55. Publish the on-ship lidar odometer and reconstruct the three-dimensional navigation scene.

[0044] Further, step S6 specifically includes:

[0045] S61. Obtain the calibration result of the hardware external parameters and unify the coordinate systems of the on-ship IMU odometer, the on-ship lidar odometer, and the on-ship RTK odometer;

[0046] S62. Convert the on-ship IMU odometer to the real-time roll, pitch, and yaw of the ship;

[0047] S63. Convert the on-ship lidar odometer to the real-time speed and azimuth of the ship;

[0048] S64. Convert the RTK odometer to the historical reference trajectory of the ship in the geodetic coordinate system;

[0049] S65. Visualize the three-dimensional navigation scene and all odometer data.

[0050] The present invention also provides a multi-sensor fusion ship berthing perception assistance system implemented based on the above-mentioned multi-sensor fusion ship berthing perception assistance method, including: a data preprocessing module, a feature extraction module, an IMU pre-integration module, an RTK signal output module, a mapping module, and a coordinate publishing module, where:

[0051] The data preprocessing module is used to obtain the on-ship lidar point cloud data and the on-ship IMU odometer data, perform point cloud motion compensation on the on-ship lidar point cloud data, and publish the corrected on-ship lidar point cloud data;

[0052] The feature extraction module is used to obtain the published on-ship lidar point cloud data, extract feature point clouds, and publish the on-ship lidar point cloud data integrated with feature point clouds;

[0053] The IMU pre-integration module is used to obtain the on-ship IMU data and the on-ship lidar odometer data, perform real-time integration of the on-ship IMU data, and publish the on-ship IMU odometer;

[0054] The RTK signal output module is used to obtain on-board RTK data, detect and judge the satellite signal status, and publish on-board RTK odometer data;

[0055] The mapping module is used to obtain on-board RTK odometer data and on-board lidar point cloud data integrated with feature points, calculate and optimize the current state and pose of the ship, and publish on-board lidar odometer and reconstructed three-dimensional navigation scenarios;

[0056] The coordinate publishing module is used to obtain the calibration result of the external parameters of the hardware, unify the coordinate systems of the on-board IMU odometer, on-board lidar odometer and on-board RTK odometer, and visualize the three-dimensional navigation scenario and all odometer data.

[0057] Compared with the prior art, the present invention has the following advantages:

[0058] 1. A multi-sensor fusion-based ship berthing perception assistance method and system provided by the present invention solve the problem of limited positioning distance caused by the use of a single sensor in traditional ship auxiliary driving systems.

[0059] 2. A multi-sensor fusion-based ship berthing perception assistance method and system provided by the present invention, whose on-board lidar has the ability to collect navigation scenario information all-weather, including target azimuth, target distance, reflection intensity, etc., which is beneficial to accurately measure the relative azimuth between the ship and the navigation environment.

[0060] 3. A multi-sensor fusion-based ship berthing perception assistance method and system provided by the present invention accurately obtains the reference trajectory of the ship in the geodetic coordinate system by fusing on-board RTK, enhances the ship's own positioning ability, and avoids the problem of accumulated positioning errors.

[0061] 4. A multi-sensor fusion-based ship berthing perception assistance method and system provided by the present invention accurately obtains the real-time roll, pitch and yaw of the ship by fusing on-board IMU, enhancing the ship's own state perception ability.

[0062] 5. A multi-sensor fusion-based ship berthing perception assistance method and system provided by the present invention can completely reconstruct the three-dimensional navigation scenario of the ship, providing a strong guarantee for the ship's autonomous berthing and safe berthing.

[0063] For the above reasons, the present invention can be widely promoted in the fields of ship berthing perception and ship autonomous positioning. Description of the Drawings

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0065] Figure 1 This is the flowchart of the method of the present invention.

[0066] Figure 2 This is the schematic diagram of the data preprocessing module provided by the embodiment of the present invention.

[0067] Figure 3 This is the schematic diagram of the feature extraction module provided by the embodiment of the present invention.

[0068] Figure 4 This is the schematic diagram of the IMU pre-integration module provided by the embodiment of the present invention.

[0069] Figure 5 This is the schematic diagram of the RTK signal output module provided by the embodiment of the present invention.

[0070] Figure 6 This is the schematic diagram of the mapping module provided by the embodiment of the present invention.

[0071] Figure 7 This is the schematic diagram of the coordinate publishing module provided by the embodiment of the present invention.

[0072] Figure 8 This is the block diagram of the invention system.

[0073] In the figure: 1. IMU pre-integration module; 2. Data preprocessing module; 3. Feature extraction module; 4. RTK signal output module; 5. Mapping module; 6. Coordinate publishing module; 7. Shipborne IMU; 8. Shipborne lidar; 9. Shipborne RTK; 10. Hardware extrinsic calibration result. Detailed implementation manners

[0074] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0075] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0076] As Figure 1 shown, the present invention provides a multi-sensor fusion-based ship berthing perception assistance method, including:

[0077] S1. Obtain the on-board lidar point cloud data and the on-board IMU odometer data, perform point cloud motion compensation on the on-board lidar point cloud data, and publish the corrected on-board lidar point cloud data;

[0078] S2. Obtain the published on-board lidar point cloud data, extract the feature point cloud, and publish the on-board lidar point cloud data integrated with the feature point cloud;

[0079] S3. Obtain the on-board IMU data and the on-board lidar odometer data, integrate the on-board IMU data in real time, and publish the on-board IMU odometer;

[0080] S4. Obtain the on-board RTK data, detect and judge the satellite signal status, and publish the on-board RTK odometer data;

[0081] S5. Obtain the on-board RTK odometer data and the on-board lidar point cloud data integrated with the feature point cloud, calculate and optimize the current state pose of the ship, and publish the on-board lidar odometer and the reconstructed three-dimensional navigation scene;

[0082] S6. Obtain the hardware extrinsic calibration result, unify the coordinate systems of the on-board IMU odometer, the on-board lidar odometer and the on-board RTK odometer, and visualize the three-dimensional navigation scene and all odometer data.

[0083] Specifically, as a preferred embodiment of the present invention, as Figure 2 shown, step S1 specifically includes:

[0084] S11. Obtain the on-board lidar point cloud and the start and end timestamps, traverse the current on-board IMU data in real time and integrate it, initialize the attitude angle of the current on-board lidar point cloud frame, and calculate the ship rotation compensation matrix using the spherical interpolation algorithm;

[0085] In this embodiment, in step S11, the spherical interpolation algorithm used is specifically as follows:

[0086]

[0087] In the formula, R(t) is the ship rotation compensation matrix at time t, and R i is the point cloud rotation matrix at the i-th moment, and θ is the three-axis rotation component obtained by pre-integration.

[0088] S12. Obtain the on-board IMU odometer data, calculate the ship translation compensation matrix using the bilinear interpolation algorithm, and perform motion compensation on the on-board lidar point cloud data;

[0089] In this embodiment, in step S12, the bilinear interpolation algorithm used is specifically as follows:

[0090]

[0091] In the formula, T(t) is the ship translation compensation matrix at time t, and T i is the point cloud displacement matrix at the i-th moment, p is the three-axis displacement component obtained by pre-integration, r F and r B are the interpolation ratios, and t is the point cloud timestamp.

[0092] S13. Store and publish the effective on-board lidar point cloud data and position index after motion compensation.

[0093] Specifically, as a preferred embodiment of the present invention, as Figure 3 shown, step S2 specifically includes:

[0094] S21. Obtain the on-board lidar point cloud data published in step S1, calculate the curvature θ of each laser point, and judge the feature point cloud, including edge points and plane points;

[0095] S22. If θ≥0.6, it is determined as an edge point, otherwise it is determined as a plane point;

[0096] S23. Publish the on-board lidar point cloud data integrated with the feature point cloud.

[0097] Specifically, as a preferred embodiment of the present invention, as Figure 4 shown, step S3 specifically includes:

[0098] S31. Obtain the current on-board IMU data and perform initialization operations;

[0099] S32. Obtain the timestamp and pose of the on-board lidar odometer, and perform IMU pre-integration;

[0100] S33. Publish the on-ship IMU odometer data.

[0101] In specific implementation, as a preferred implementation manner of the present invention, as Figure 5 shown, step S4 specifically includes:

[0102] S41. Obtain the serial port signal and perform initialization operations;

[0103] S42. Convert the original NMEA (National Marine Electronics Association) data collected by the on-ship RTK receiver into ROS (Robot Operating System) standard format data;

[0104] S43. Detect and judge the satellite signal status, including detecting the satellite signal status and judging the position covariance; use the status detection algorithm to detect whether there is enhanced positioning information, and judge whether the position covariance meets the system requirements, specifically as follows:

[0105]

[0106] In the formula, RTK fix is the RTK signal status, status = 0 indicates no enhanced positioning; status = 1 indicates satellite enhanced positioning; status = 2 indicates ground-based enhanced positioning; status = -1 indicates unable to position;

[0107] S44. If the satellite signal status detection is qualified, publish the on-ship RTK odometer; otherwise, reject the publication and wait for the differential correction data according to the preset time ; if then it is determined that the system has not reached the ideal state for a long time, and a warning is sent.

[0108] In specific implementation, as a preferred implementation manner of the present invention, as Figure 6 shown, step S5 specifically includes:

[0109] S51. Obtain the on-ship RTK odometer and the on-ship lidar point cloud data integrated with feature points;

[0110] S52. Perform point cloud matching and calculate the current state pose of the ship;

[0111] S53. Perform backend optimization to optimize the current state pose of the ship;

[0112] S54. Perform loop detection to optimize the global state pose of the ship;

[0113] S55. Publish the on-board lidar odometer and reconstruct the three-dimensional navigation scene.

[0114] During specific implementation, as a preferred implementation mode of the present invention, as Figure 7 shown, step S6 specifically includes:

[0115] S61. Obtain the calibration result of the hardware extrinsic parameters, and unify the coordinate systems of the on-board IMU odometer, the on-board lidar odometer, and the on-board RTK odometer;

[0116] S62. Convert the on-board IMU odometer to the roll, pitch, and yaw of the ship in real time;

[0117] S63. Convert the on-board lidar odometer to the speed and azimuth of the ship in real time;

[0118] S64. Convert the RTK odometer to the historical reference trajectory of the ship in the geodetic coordinate system;

[0119] S65. Visualize the three-dimensional navigation scene and all odometer data.

[0120] Corresponding to the multi-sensor fusion-based ship berthing perception assistance method in this application, this application also provides a multi-sensor fusion-based ship berthing perception assistance system, as Figure 8 shown, including: a data preprocessing module 2, a feature extraction module 3, an IMU (Inertial Measurement Unit) pre-integration module 1, an RTK (Real-time kinematic) signal output module 4, a mapping module 5, and a coordinate publishing module 6, where:

[0121] The data preprocessing module 2 is used to obtain the point cloud data of the on-board lidar 8 and the odometer data of the on-board IMU 7, perform point cloud motion compensation on the point cloud data of the on-board lidar 8, and publish the corrected point cloud data of the on-board lidar 8;

[0122] The feature extraction module 3 is used to obtain the published point cloud data of the on-board lidar 8, extract the feature point cloud, and publish the point cloud data of the on-board lidar 8 integrated with the feature point cloud;

[0123] The IMU pre-integration module 1 is used to obtain the data of the on-board IMU 7 and the odometer data of the on-board lidar 8, perform real-time integration on the data of the on-board IMU 7, and publish the odometer of the on-board IMU 7;

[0124] The RTK signal output module 4 is used to obtain the data of the on-board RTK 9, detect and judge the satellite signal status, and publish the odometer data of the on-board RTK 9;

[0125] The mapping module 5 is used to obtain the odometer data of the shipborne RTK 9 and the point cloud data of the shipborne lidar 8 integrating feature point clouds, calculate and optimize the current state pose of the ship, and publish the odometer of the shipborne lidar 8 and the reconstructed three-dimensional navigation scene;

[0126] The coordinate publishing module 6 is used to obtain the hardware extrinsic calibration result 10, unify the odometer coordinate systems of the shipborne IMU 7, the shipborne lidar 8, and the shipborne RTK 9, and visualize the three-dimensional navigation scene and all odometer data.

[0127] For the embodiments of the present invention, since they correspond to the above embodiments, the description is relatively simple. For relevant similarities, please refer to the description in the above embodiments. Details are not repeated here.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assisting ship berthing perception with multi-sensor fusion, characterized in that, Including: S1. Obtain the point cloud data of the shipborne lidar and the odometer data of the shipborne IMU, perform point cloud motion compensation on the point cloud data of the shipborne lidar, and publish the corrected point cloud data of the shipborne lidar; S2. Obtain the published point cloud data of the shipborne lidar, extract the feature point cloud, and publish the point cloud data of the shipborne lidar integrated with the feature point cloud; S3. Obtain the shipborne IMU data and the odometer data of the shipborne lidar, integrate the shipborne IMU data in real time, and publish the shipborne IMU odometer; S4. Obtain the shipborne RTK data, detect and judge the satellite signal status, and publish the shipborne RTK odometer data; S5. Obtain the shipborne RTK odometer data and the point cloud data of the shipborne lidar integrated with the feature point cloud, calculate and optimize the current state pose of the ship, and publish the shipborne lidar odometer and the reconstructed three-dimensional navigation scene; S6. Obtain the calibration result of the hardware extrinsic parameters, unify the coordinate systems of the shipborne IMU odometer, the shipborne lidar odometer, and the shipborne RTK odometer, and visualize the three-dimensional navigation scene and all odometer data.

2. A method for assisting ship berthing perception by multi-sensor fusion according to claim 1, characterized in that, Step S1 specifically includes: S11. Obtain the point cloud of the shipborne lidar and the start and end timestamps, traverse the current shipborne IMU data in real time and integrate it, initialize the attitude angle of the current shipborne lidar point cloud frame, and calculate the ship rotation compensation matrix using the spherical interpolation algorithm; S12. Obtain the shipborne IMU odometer data, calculate the ship translation compensation matrix using the bilinear interpolation algorithm, and perform motion compensation on the point cloud data of the shipborne lidar; S13. Store and publish the effective point cloud data of the shipborne lidar and the position index after motion compensation.

3. A multi-sensor fusion-based ship berthing perception assistance method according to claim 2, characterized in that, In step S11, the spherical interpolation algorithm used is as follows: where \(R(t)\) is the ship rotation compensation matrix at time \(t\), \(R\) i is the point cloud rotation matrix at the \(i\)-th moment, and \(\theta\) is the three-axis rotation component obtained by pre-integration.

4. A method for assisting ship berthing perception by multi-sensor fusion according to claim 2, characterized in that, In step S12, the bilinear interpolation algorithm used is as follows: where \(T(t)\) is the ship translation compensation matrix at time \(t\), \(T\) i is the point cloud displacement matrix at the \(i\)-th moment, \(p\) is the three-axis displacement component obtained by pre-integration, \(r\) F and \(r\) B are the interpolation ratios, and \(t\) is the point cloud timestamp.

5. A method for assisting ship berthing perception by multi-sensor fusion according to claim 1, characterized in that, Step S2 specifically includes: S21. Obtain the point cloud data of the shipborne lidar published in step S1, calculate the curvature θ of each laser point, and judge the feature point cloud, including edge points and plane points; S22. If θ≥0.6, it is determined as an edge point, otherwise it is determined as a plane point; S23. Publish the point cloud data of the shipborne lidar integrated with the feature point cloud.

6. A method for assisting ship berthing perception by multi-sensor fusion according to claim 1, characterized in that Step S3 specifically includes: S31. Obtain the current shipborne IMU data and perform initialization operations; S32. Obtain the timestamp and pose of the shipborne lidar odometer, and perform IMU pre-integration; S33. Publish the shipborne IMU odometer data.

7. A method for assisting ship berthing perception with multi-sensor fusion according to claim 1, characterized in that Step S4 specifically includes: S41. Obtain the serial port signal and perform initialization operations; S42. Convert the original NMEA data collected by the shipborne RTK receiver into ROS standard format data; S43. Detect and judge the satellite signal status, including detecting the satellite signal status and judging the position covariance; use the status detection algorithm to detect whether there is enhanced positioning information, and judge whether the position covariance meets the system requirements, as follows: where RTK fix is the RTK signal status, status = 0 indicates no enhanced positioning; status = 1 indicates satellite-based enhanced positioning; status = 2 indicates ground-based enhanced positioning; status = -1 indicates unable to position; S44. If the satellite signal status detection is qualified, the on-board RTK odometer is released; otherwise, the release is refused and the differential correction data is awaited according to the preset time. If it is determined that the system has not reached the ideal state for a long time, a warning is sent.

8. A method for assisting ship berthing perception by multi-sensor fusion according to claim 1, characterized in that Step S5 specifically includes: S51. Obtain the shipborne RTK odometer and the point cloud data of the shipborne lidar integrated with the feature point cloud; S52. Perform point cloud matching to calculate the current state pose of the ship; S53. Perform backend optimization to optimize the current state pose of the ship; S54. Perform loop detection and optimize the global pose of the ship; S55. Publish the on-board lidar odometer and reconstruct the 3D navigation scene.

9. A method for assisting ship berthing perception with multi-sensor fusion according to claim 1, characterized in that Step S6 specifically includes: S61. Obtain the calibration result of the hardware external parameters and unify the coordinate systems of the on-board IMU odometer, on-board lidar odometer, and on-board RTK odometer; S62. Convert the on-board IMU odometer into the roll, pitch, and yaw of the ship in real time; S63. Convert the on-board lidar odometer into the speed and azimuth of the ship in real time; S64. Convert the RTK odometer into the historical reference trajectory of the ship in the geodetic coordinate system; S65. Visualize the 3D navigation scene and all odometer data.

10. A multi-sensor fusion ship berthing perception assistance system implemented by the multi-sensor fusion ship berthing perception assistance method according to any one of claims 1-9, characterized in that, It includes: A data preprocessing module, a feature extraction module, an IMU pre-integration module, an RTK signal output module, a mapping module, and a coordinate publishing module, where: The data preprocessing module is used to obtain the on-board lidar point cloud data and on-board IMU odometer data, perform point cloud motion compensation on the on-board lidar point cloud data, and publish the corrected on-board lidar point cloud data; The feature extraction module is used to obtain the published on-board lidar point cloud data, extract feature point clouds, and publish the on-board lidar point cloud data integrated with feature point clouds; The IMU pre-integration module is used to obtain the on-board IMU data and on-board lidar odometer data, perform real-time integration of the on-board IMU data, and publish the on-board IMU odometer; The RTK signal output module is used to obtain the on-board RTK data, detect and judge the satellite signal status, and publish the on-board RTK odometer data; The mapping module is used to obtain the on-board RTK odometer data and the on-board lidar point cloud data integrated with feature point clouds, calculate and optimize the current pose of the ship, and publish the on-board lidar odometer and reconstruct the 3D navigation scene; The coordinate publishing module is used to obtain the calibration result of the hardware external parameters, unify the coordinate systems of the on-board IMU odometer, on-board lidar odometer, and on-board RTK odometer, and visualize the 3D navigation scene and all odometer data.

Images