A multi-modal obstacle avoidance system for multi-sensor fusion of an unmanned ship based on ROS

Abstract

The present invention provides a multi-modal obstacle avoidance system for multi-sensor fusion of an unmanned ship based on ROS, which is characterized by including three parts: sensors, a controller, and an actuator. In this system, a lidar, an industrial camera, an inertial measurement unit, and a GPS module are used to sense the surrounding environment and obtain attitude and position information, belonging to the sensors of the system; an artificial intelligence development board and a single-chip microcomputer are the main control modules, which make decisions according to the input through built-in algorithms, belonging to the controller of the system; a drive controller and a drive mechanical device are used to execute the decisions of the controller, belonging to the actuator of the system; the present invention uses a multi-sensor distributed modular architecture, enabling various functions to be coupled together, allowing tasks to be executed simultaneously in multiple threads, greatly reducing computing power, making the control more flexible, and equipped with a high-efficiency and high-accuracy vision-lidar fusion obstacle avoidance algorithm, enabling the unmanned ship to flexibly handle various complex situations that occur on the water surface and having the capabilities of autonomous cruising and autonomous obstacle avoidance.

Description

Technical Field

[0001] This invention relates to the field of aquatic equipment, specifically a multimodal obstacle avoidance system for unmanned vessels based on ROS and multi-sensor fusion. Background Technology

[0002] Currently, the obstacle avoidance methods used on unmanned vessels are mainly single-sensor obstacle avoidance. For example, only lidar (1) is used to detect surrounding obstacle information, or only a camera is used to capture images, and then the images or radar feedback information are processed and identified before obstacle avoidance is performed based on the identification results.

[0003] The advantage of lidar lies in its ability to detect distant obstacles, as well as obstacles within a 360-degree horizontal range. It boasts a long effective detection range and a wide coverage area. However, lidar can only detect a single plane, with almost no vertical detection extension, and due to its installation height, it cannot effectively detect obstacles floating on the water surface. Cameras, on the other hand, offer a recognition range of approximately 100 degrees both horizontally and vertically, effectively identifying obstacles floating on the water surface. However, their detection range is very short, making them ineffective at detecting distant obstacles.

[0004] Currently, the overall control system of unmanned surface vessels (USVs) mostly uses embedded development boards for direct control, such as the STM32 series, ESP series, and Raspberry Pi series. This control method is limited by the computing power of the development board and cannot perform some complex functions that require high computing power, such as path planning and visual recognition. Furthermore, it is difficult to develop and has a long development cycle. With the rise of many advanced technologies and algorithms in recent years, simply using embedded development boards is no longer suitable for the development and application of current USVs.

[0005] The Robot Operation System (ROS) is a highly flexible robot software architecture. This architecture offers many advantages:

[0006] a) Distributed peer-to-peer computing

[0007] b) Software reuse

[0008] c) Quick Visual Debugging

[0009] d) Loosely coupled modular development

[0010] Our multi-sensor fusion obstacle avoidance system is developed and tested based on the ROS architecture. Under this architecture, each algorithm node can execute its process independently, and can also communicate efficiently and modularly through ROS's communication methods to achieve collaborative cooperation between algorithms.

[0011] While existing single-sensor obstacle avoidance systems are relatively easy to apply, they cannot effectively handle the complex obstacle situations in real-world waterways. In real-world waterways, two types of obstacles may coexist: one type is floating obstacles with little elevation above the water, such as garbage bags, water bottles, small driftwood, and aquatic plants; the other type is larger obstacles that need to be detected and avoided from a greater distance, such as protruding islands, riverbanks, bridge arches, and ships traveling in opposite or the same direction. Especially when one or more sub-types of these two main types of obstacles are present in the surrounding waters, obstacle avoidance using a single sensor becomes extremely complex and difficult, significantly reducing its effectiveness and making practical application impossible. Summary of the Invention

[0012] The purpose of this invention is to propose a multimodal obstacle avoidance system based on ROS for unmanned surface vessels, using multi-sensor fusion to address the limitations of single-sensor obstacle avoidance systems in effectively handling complex obstacles in real-world water environments. These obstacles primarily include: one type is floating obstacles with low elevation above the water, such as garbage bags, water bottles, small driftwood, and aquatic plants; the other type is large obstacles that require detection and avoidance from a greater distance, such as protruding islands, riverbanks, bridge arches, and ships traveling in opposite or the same direction. Especially when one or more sub-types of these two types of obstacles appear simultaneously in the surrounding waters, obstacle avoidance using a single sensor becomes extremely complex and difficult, significantly reducing its effectiveness and hindering practical application.

[0013] To achieve the above objectives, the present invention provides the following technical solution:

[0014] A multimodal obstacle avoidance system for unmanned surface vessels (USVs) based on ROS, characterized by comprising three parts: sensors, a controller, and actuators. This system is developed and tested based on the ROS architecture. The sensors include a LiDAR 1, an industrial camera 2, an IMU (Inertial Measurement Unit) 3, and a GPS module 4 for sensing the surrounding environment and acquiring attitude and position information. The controller, comprising an AI development board 5 and a microcontroller 6, is the main control module for USV operation, making decisions based on input using a built-in algorithm. The actuators include a drive controller and a drive mechanism 7 for executing the controller's decisions. The LiDAR 1, industrial camera 2, IMU, and GPS module 4 are all connected to the AI ​​development board 5 via cables. The drive controller and AI development board 5 are both connected to the microcontroller 6 via cables. The drive controller and drive mechanism 7 are also connected via cables. In use, the IMU and GPS module 3... Module 4 is installed on the connecting bridge. The IMU inertial measurement unit 3 and the GPS module 4 use the currently common IMU+GPS data fusion algorithm to achieve high-precision positioning and path calculation for the unmanned vessel in the field environment. They are further combined with the navigation function package to achieve point-to-point navigation, which is performed synchronously with the obstacle avoidance function. The lidar 1 and the industrial camera 2 are installed at the bridgehead of the unmanned vessel connecting bridge, aligned with the central axis of the connecting bridge. The industrial camera 2 is placed in front of the lidar 1 to obtain radar point cloud data and images from the industrial camera 2. After filtering and analysis, filtered radar information and water surface obstacle image information are obtained. These are packaged and encapsulated into "messages" in the ROS architecture and published. The constructed central obstacle avoidance algorithm node subscribes to the above-mentioned water surface obstacle image information and filtered radar information. The obstacle avoidance algorithm determines the direction and speed of movement required for the vessel to reach the target point and iteratively updates based on the collected real-time data.

[0015] To achieve the localization and path calculation capabilities of unmanned surface vessels in wilderness environments, and further integrate them with the navigation function package in the ROS architecture, the following specific steps are included:

[0016] The fusion algorithm measures the vessel's current latitude and longitude position, attitude, speed, and acceleration. Using a common IMU+GPS fusion algorithm, this information is converted into odometer data, which records the vessel's motion. The operator can use a personal computer within the control system, employing a distributed framework built on the Robot Operating System (ROS) and a Wi-Fi local area network formed by the communication router 8, to send multiple unmanned vessel cruise point coordinates to the AI ​​development board 5 sequentially. The navigation function package combines the current position and speed information to calculate the required attitude and speed to reach the target point, converting this into speed parameters which are then sent to the microcontroller 6.

[0017] The lidar 1 and the industrial camera 2 perform identification simultaneously, but the two sensors have different identification areas and identify different targets. However, the identification results will eventually be unified under a coordinate system to facilitate obstacle avoidance calculation. Specifically, the lidar 1 is responsible for detecting obstacles raised in the range of 1m-40m and 360 degrees laterally. The industrial camera 2 is responsible for detecting floating obstacles on the water surface in the range of 100 degrees laterally and 80 degrees longitudinally in front of the ship.

[0018] The radar point cloud data directly obtained by the lidar 1 is large in volume and is subject to a lot of noise interference on the water surface. Therefore, it is necessary to establish a radar filter to simplify the radar data information and filter out interference points and noise points. This filtering algorithm is based on the degree of dispersion of the radar point cloud data; in areas where scattered points are relatively clustered, they can be considered as obstacles that need to be avoided, and vice versa.

[0019] The images directly acquired by the industrial camera 2 need to be analyzed and processed before the obstacles on the water surface can be identified. The specific analysis and processing requires the use of a convolutional neural network Yolov5 for identification and processing. It uses a pre-trained dataset, which includes a series of common floating objects on the water surface, such as mineral water bottles, plastic garbage bags, floating logs, and wooden barrels. The dataset will be continuously upgraded in the future to add other floating objects that may appear in the wild. The neural network reads each frame captured by the industrial camera 2, analyzes and processes it, and finally identifies the obstacles on the water surface, recording the multiple coordinate corner points (u,v), area S, and obstacle type H of the relevant obstacles in the image.

[0020] The central obstacle avoidance algorithm mainly includes the following:

[0021] S1. Calculate the image information published by the Yolov5 node; convert the 2D image information of multiple coordinate corner points (u,v), area S, and obstacle type H of the water surface obstacle in the image obtained from the industrial camera 2 into fuzzy coordinates (x,v) of the water surface obstacle in the ship coordinate system.m ,y m This is called "water surface obstacle coordinates";

[0022] S2. Calculate the radar information released by the radar filter node; convert the "angle-distance" information contained in the ranges array of the radar information into the coordinates (x, y) of the obstacle in the ship's coordinate system. n ,y n This is called "ordinary obstacle coordinates";

[0023] S3. Based on the coordinates of all surface obstacles and ordinary obstacles, derive the Level 1 threat coefficient Th1 and Level 2 threat coefficient Th2 for each obstacle; Th1 is the distance of the obstacle coordinates relative to the ship's coordinate system, and Th2 is the square of the obstacle's x-coordinate. 2 The specific calculation formula is as follows:

[0024]

[0025] Th2 = x 2

[0026] Level 1 threat coefficient describes the straight-line distance between the obstacle and the ship, while Level 2 threat coefficient describes the lateral distance between the obstacle and the ship. Level 1 threat coefficient is established to allow ships to avoid obstacles based on their straight-line distance to the obstacle, and it is the most basic obstacle avoidance reference data. Level 2 threat coefficient is mainly for obstacles directly opposite the ship's direction of travel, and such obstacles require a significant change in course to avoid. Both Level 1 and Level 2 threat coefficients are applied to the obstacle avoidance process.

[0027] S4. All obstacles detected within the ship's detection range will be calculated as level 1 or 2 threat coefficients. However, the ship will establish a "surrounding safety zone" because some obstacles are not on the ship's path of travel, so there is no need to avoid such obstacles. The safety zone is a rectangular area extending 3.5m to the left and right of the ship and 30m forward and backward.

[0028] S5. To facilitate description within a two-dimensional coordinate system, the parameters describing the ship's two-dimensional motion—linear velocity V and angular velocity ω—are transformed into x-axis velocity and y-axis velocity, denoted as (V... x V y The calculation formula is as follows:

[0029] V x =Vcos(ωΔt)

[0030] V y =Vsin(ωΔt)

[0031] Where Δt is the sampling time.

[0032] Based on the coordinates of all currently detected obstacles, calculate (V) x V y The formula is as follows:

[0033] V x =K1(∑ left Th1-∑ right Th1)+K2(∑ left Th2-∑ right Th2)

[0034] V y =K3V x +K4(∑ left Th2-∑ right TH2)

[0035] The coordinate system is divided into two regions, the left half and the right half, with the y-axis as the dividing line; V x It is calculated by multiplying the difference in the first-level coefficients of all obstacle coordinates in the left and right halves by a scaling factor K1, and then multiplying the difference in the second-level coefficients of all obstacle coordinates in the left and right halves by a scaling factor K2. This is done to give the ship better obstacle avoidance maneuverability towards obstacles directly in front of it and close to its sides, and to handle complex situations with multiple obstacles. x This formula allows ships to dynamically maintain distances from various obstacles, and when encountering two or more obstacles that are close together, it also allows the ship to maintain an optimal safe distance from each obstacle. y It is V x Multiply by a scaling factor K3, then multiply the difference of the level 2 coefficients of all obstacle coordinates in the left and right halves by a scaling factor K4; the reason for doing this is to make V y To some extent, it is based on V x It can also achieve the same dynamic speed adjustment function.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] This invention uses a multi-sensor distributed modular architecture, which allows various functions to be easily coupled together and can execute tasks simultaneously in multiple threads, greatly reducing computing power and making control more flexible. It enables unmanned vessels to have autonomous cruising and obstacle avoidance capabilities, and is equipped with a highly efficient and accurate fusion obstacle avoidance algorithm, allowing unmanned vessels to flexibly cope with various complex situations on the water surface, representing a significant leap forward in intelligence. Attached Figure Description

[0038] Figure 1 This is a schematic diagram illustrating the obstacle avoidance monitoring principle of the present invention;

[0039] Figure 2This is a physical image of the ship used in this invention;

[0040] Figure 3 This is a diagram of the rectangular safety area used in the obstacle avoidance system of this invention;

[0041] Figure 4 This is a schematic diagram of the first type of obstacle distribution and obstacle avoidance route in an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the second type of obstacle distribution and obstacle avoidance route in an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the third type of obstacle distribution and obstacle avoidance route in an embodiment of the present invention;

[0044] Figure 7 This is a flowchart of the central obstacle avoidance algorithm of the present invention;

[0045] Figure 8 This is a basic structural diagram of the operation of the present invention;

[0046] In the diagram: 1. LiDAR; 2. Industrial camera; 3. IMU (Inertial Measurement Unit); 4. GPS module; 5. Artificial intelligence development board; 6. Microcontroller; 7. Drive mechanism. Detailed Implementation

[0047] To clarify the technical problems, technical solutions, implementation processes, and performance demonstrations, the present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative. The present invention is not intended to limit the scope of the invention. Various exemplary embodiments, features, and aspects of this disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0048] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0049] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0050] Example 1

[0051] like Figure 1 , Figure 2 and Figure 8As shown, a multimodal obstacle avoidance system for unmanned surface vessels (USVs) based on ROS, characterized by comprising three parts: sensors, a controller, and actuators. This system is developed and tested based on the ROS architecture. The sensors include a lidar 1, an industrial camera 2, an IMU (Inertial Measurement Unit) 3, and a GPS module 4 for sensing the surrounding environment and acquiring attitude and position information. The controller, comprising an AI development board 5 and a microcontroller 6, is the main control module for USV operation, making decisions based on input using a built-in algorithm. The actuators include a drive controller and a drive mechanism 7 for executing the controller's decisions. The lidar 1, the industrial camera 2, the IMU 3, and the GPS module 4 are described. Camera 2, the IMU inertial navigation module, and GPS module 4 are all connected to the artificial intelligence development board 5 via cables. The drive controller and the artificial intelligence development board 5 are both connected to the microcontroller 6 via cables. The drive controller and the drive mechanical device 7 are also connected via cables. In use, the IMU inertial measurement unit 3 and the GPS module 4 are mounted on the connecting bridge. The IMU inertial measurement unit 3 and the GPS module 4 employ the currently common IMU+GPS data fusion algorithm to achieve high-precision positioning and path calculation functions for the unmanned vessel in the field environment, and are further integrated with the navigation function package. The system enables point-to-point navigation, which is performed synchronously with obstacle avoidance. The lidar 1 and industrial camera 2 are installed at the head of the connecting bridge of the unmanned vessel, aligned with the bridge's central axis. The industrial camera 2 is placed in front of the lidar 1 to obtain radar point cloud data and images from the camera. After filtering and analysis, filtered radar information and water surface obstacle images are obtained, packaged and encapsulated into "messages" within the ROS architecture, and published. The constructed central obstacle avoidance algorithm node then subscribes to the aforementioned water surface obstacle images and filtered radar information, using the obstacle avoidance algorithm to infer the time required for the vessel to reach the target point. The system tracks the direction and speed of movement, continuously updating the data based on real-time data collection. Real-time messages from the sensors are constantly subscribed to by the central obstacle avoidance algorithm node. The sensor node then determines whether to continue executing the obstacle avoidance method from the previous moment or interrupt the previous process to initiate a new obstacle avoidance action based on the current situation. In this obstacle avoidance system, considering the respective advantages and disadvantages of LiDAR 1 and industrial camera 2, the LiDAR is responsible for detecting obstacles raised within a 1m-40m range and a 360-degree lateral radius, such as riverbanks, reefs, and passing ships. Industrial camera 2 is responsible for detecting floating obstacles on the water surface within a 100-degree lateral and 80-degree longitudinal radius in front of the ship, such as water bottles, plastic bags, and driftwood. The core idea of ​​this obstacle avoidance system is to complement the strengths and weaknesses of these two sensors.

[0052] The drive controller can be an electronic speed controller, and the drive mechanical device 7 can be an underwater thruster.

[0053] To achieve the localization and path calculation capabilities of unmanned surface vessels in wilderness environments, and further integrate them with the navigation function package in the ROS architecture, the following specific steps are included:

[0054] The fusion algorithm measures the vessel's current latitude and longitude position, attitude, speed, and acceleration. Using a common IMU+GPS fusion algorithm, this information is converted into odometer data, which records the vessel's motion. The operator can use a personal computer within the control system, employing a distributed framework built on the Robot Operating System (ROS) and a Wi-Fi local area network formed by the communication router 8, to send multiple unmanned vessel cruise point coordinates to the AI ​​development board 5 sequentially. The navigation function package combines the current position and speed information to calculate the required attitude and speed to reach the target point, converting this into speed parameters which are then sent to the microcontroller 6.

[0055] The lidar 1 and the industrial camera 2 perform identification simultaneously, but the two sensors have different identification areas and identify different targets. However, the identification results will eventually be unified under a coordinate system to facilitate obstacle avoidance calculation. Specifically, the lidar 1 is responsible for detecting obstacles raised in the range of 1m-40m and 360 degrees laterally. The industrial camera 2 is responsible for detecting floating obstacles on the water surface in the range of 100 degrees laterally and 80 degrees longitudinally in front of the ship.

[0056] The radar point cloud data directly obtained by the lidar 1 is large in volume and subject to significant noise interference on the water surface. Therefore, a radar filter algorithm is needed to simplify the radar data and remove interference and noise points. This filtering algorithm is based on the discreteness of the radar point cloud data; areas where scattered points are relatively clustered are considered obstacles to be avoided, while areas where they are scattered are considered noise. Thanks to the performance of the radar, even if the method deviates in the processing of a few frames, subsequent processes can successfully correct this.

[0057] The images directly acquired by the industrial camera 2 need to be analyzed and processed before the obstacles on the water surface can be identified. The specific analysis and processing requires the use of a convolutional neural network Yolov5 for identification and processing. It uses a pre-trained dataset, which includes a series of common floating objects on the water surface, such as mineral water bottles, plastic garbage bags, floating logs, and wooden barrels. The dataset will be continuously upgraded in the future to add other floating objects that may appear in the wild. The neural network reads each frame captured by the industrial camera 2, analyzes and processes it, and finally identifies the obstacles on the water surface, recording the multiple coordinate corner points (u,v), area S, and obstacle type H of the relevant obstacles in the image.

[0058] like Figure 7As shown, the central obstacle avoidance algorithm mainly includes the following:

[0059] S1. Calculate the image information published by the Yolov5 node; convert the 2D image information of multiple coordinate corner points (u,v), area S, and obstacle type H of the water surface obstacle in the image obtained from the industrial camera 2 into fuzzy coordinates (x,v) of the water surface obstacle in the ship coordinate system. m ,y m This is called "water surface obstacle coordinates";

[0060] S2. Calculate the radar information released by the radar filter node; convert the "angle-distance" information contained in the ranges array of the radar information into the coordinates (x, y) of the obstacle in the ship's coordinate system. n ,y n This is called "ordinary obstacle coordinates";

[0061] S3. Based on the coordinates of all surface obstacles and ordinary obstacles, derive the Level 1 threat coefficient Th1 and Level 2 threat coefficient Th2 for each obstacle; Th1 is the distance of the obstacle coordinates relative to the ship's coordinate system, and Th2 is the square of the obstacle's x-coordinate. 2 The specific calculation formula is as follows:

[0062]

[0063] Th2 = x 2

[0064] Level 1 threat coefficient describes the straight-line distance between the obstacle and the ship, while Level 2 threat coefficient describes the lateral distance between the obstacle and the ship. Level 1 threat coefficient is established to allow ships to avoid obstacles based on their straight-line distance to the obstacle, and it is the most basic obstacle avoidance reference data. Level 2 threat coefficient is mainly for obstacles directly opposite the ship's direction of travel, and such obstacles require a significant change in course to avoid. Both Level 1 and Level 2 threat coefficients are applied to the obstacle avoidance process.

[0065] S4. All obstacles detected within the ship's detection range will be calculated with a threat level of 1 or 2. However, the ship will establish a "surrounding safety zone" because some obstacles are not located on the ship's path and therefore do not require obstacle avoidance. The safety zone is a rectangular area extending 3.5m to the left and right of the ship and 30m forward and backward. Converted to the ship's coordinate system as follows: Figure 3 As shown, this rectangular area represents the optimal safety zone determined through actual testing, taking into account factors such as hull size, turning radius, speed, and the performance of onboard sensors.

[0066] S5. To facilitate description within a two-dimensional coordinate system, the parameters describing the ship's two-dimensional motion—linear velocity V and angular velocity ω—are transformed into x-axis velocity and y-axis velocity, denoted as (V... x V y The calculation formula is as follows:

[0067] V x =Vcos(ωΔt)

[0068] V y =Vsin(ωΔt)

[0069] Where Δt is the sampling time.

[0070] Based on the coordinates of all currently detected obstacles, calculate (V) x V y The formula is as follows:

[0071] V x =K1(∑ left Th1-∑ right Th1)+K2(∑ left Th2-∑ right Th2)

[0072] V y =K3V x +K4(∑ left Th2-∑ right Th2)

[0073] The coordinate system is divided into two regions, the left half and the right half, with the y-axis as the dividing line; V x It is calculated by multiplying the difference in the first-level coefficients of all obstacle coordinates in the left and right halves by a scaling factor K1, and then multiplying the difference in the second-level coefficients of all obstacle coordinates in the left and right halves by a scaling factor K2. This is done to give the ship better obstacle avoidance maneuverability towards obstacles directly in front of it and close to its sides, and to handle complex situations with multiple obstacles. x This formula allows ships to dynamically maintain distances from various obstacles, and when encountering two or more obstacles that are close together, it also allows the ship to maintain an optimal safe distance from each obstacle. y It is V x Multiply by a scaling factor K3, then multiply the difference of the level 2 coefficients of all obstacle coordinates in the left and right halves by a scaling factor K4; the reason for doing this is to make V y To some extent, it is based on V x It can also achieve the same dynamic speed adjustment function.

[0074] Using the obstacle avoidance system described above, obstacle avoidance experiments were conducted. The following are the instructions the ship would execute for several different obstacle distribution scenarios:

[0075] ① The distribution of obstacles near the ship is as follows Figure 4 When indicated, the ship will navigate normally according to navigation instructions;

[0076] ② Distribution of obstacles near the ship, such as Figure 5 As shown, the vessel will quickly turn right and will continue along its original navigation route once the threat has disappeared.

[0077] ③ The distribution of obstacles near the ship is as follows Figure 6 As shown, the ship first turns right, then quickly turns left, maneuvering flexibly between the two obstacles while maintaining the optimal safe distance between each.

[0078] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A ROS-based multi-sensor fusion multi-modal obstacle avoidance system for unmanned ships, characterized in that, The system comprises three parts: sensors, controllers, and actuators. Developed and tested based on the ROS architecture, the sensors include a lidar (1), an industrial camera (2), an IMU (inertial measurement unit) (3), and a GPS module (4) for sensing the surrounding environment and acquiring attitude and position information. The controller includes an AI development board (5) and a microcontroller (6), serving as the main control module for unmanned vessel operations, making decisions based on input. The actuators include a drive controller and a drive mechanism (7) for executing the controller's decisions. The lidar (1), industrial camera (2), IMU, and GPS module (4) are all connected to the AI ​​development board (5) via cables. The drive controller and AI development board (5) are connected to the microcontroller (6) via cables. The drive controller and drive mechanism (7) are connected via cables. During use, the IMU (inertial measurement unit) and GPS module (4) are installed... The IMU (Inertial Measurement Unit) (3) and the GPS module (4) use the IMU+GPS data fusion algorithm to realize the positioning and path calculation functions of the unmanned ship in the field environment, and further combine with the navigation function package to realize the fixed-point navigation function. This function is performed synchronously with the obstacle avoidance function. The lidar (1) and the industrial camera (2) are installed at the bridgehead of the unmanned ship connecting bridge, aligned with the central axis of the connecting bridge. The industrial camera (2) is placed in front of the lidar (1) to obtain the radar point cloud data and the industrial camera (2) image. After filtering and analysis, the filtered radar information and water surface obstacle image information are obtained. The information is packaged and encapsulated into a message in the ROS architecture and published. Then, the constructed central obstacle avoidance algorithm node subscribes to the above-mentioned water surface obstacle image information and filtered radar information. The central obstacle avoidance algorithm determines the attitude and speed required for the unmanned ship to reach the target point next, and iterates and updates continuously based on the collected real-time data. To achieve the localization and path calculation capabilities of unmanned surface vessels in outdoor environments, and further integrate them with the navigation function package in the ROS architecture, the following specific steps are included: The IMU+GPS data fusion algorithm can measure the latitude and longitude coordinates, attitude orientation, speed and acceleration of the current unmanned vessel, convert them into odometer information, and record the movement process of the unmanned vessel. The operator can use a personal computer in the control system to send multiple latitude and longitude coordinates of the unmanned vessel's cruising points to the artificial intelligence development board (5) in turn through the distributed framework built by the Robot Operating System (ROS) and the WIFI local area network formed by the communication router (8). The navigation function package will combine the current location information and speed information to calculate the attitude and speed information required to reach the target point, and convert it into speed information and send it to the microcontroller (6). The lidar (1) and the industrial camera (2) perform identification simultaneously, and unify the identification results under a coordinate system; specifically, the lidar (1) is responsible for detecting obstacles raised in the range of 1m-40m and 360 degrees laterally; the industrial camera (2) is responsible for detecting floating obstacles on the water surface in the range of 100 degrees laterally and 80 degrees longitudinally in front of the unmanned vessel. The lidar (1) needs to establish a radar filter. The filtering algorithm is based on the degree of dispersion of the radar point cloud data. Areas where scattered points are relatively clustered are obstacles that need to be avoided, while areas where they are not clustered are considered noise. The images directly acquired by the industrial camera (2) require the use of a convolutional neural network Yolov5 for identification and processing during specific analysis and processing. The dataset used is a pre-trained dataset that includes floating objects on the water surface. The dataset will be continuously upgraded in the future to include other floating objects that appear in the wild environment. The neural network reads each frame of the image captured by the industrial camera (2) and performs analysis and processing to finally identify the obstacles on the water surface and record the multiple coordinate corner points (u,v), area S, and obstacle type H of the relevant obstacles in the image.

2. The ROS-based multi-sensor fusion multi-modal obstacle avoidance system of claim 1, wherein, The central obstacle avoidance algorithm mainly includes the following: S1. Solve the image information published by the Yolov5 node; convert the 2D image information of multiple coordinate corner points (u,v), area S, and obstacle type H of the water surface obstacle in the image obtained from the industrial camera (2) into fuzzy coordinates of the water surface obstacle in the coordinate system of the unmanned vessel. , is called the water surface obstacle coordinate; S2. Calculate the radar information released by the radar filter node; convert the angle-distance information contained in the ranges array of the radar information into the coordinates of the obstacle in the UAV coordinate system. These are called ordinary obstacle coordinates; S3. Based on the current coordinates of all surface obstacles and ordinary obstacles, derive the Level 1 threat coefficient for each obstacle. and Level 2 threat level ; It is the distance between the obstacle's coordinates and the unmanned vessel's coordinate system. It is the square of the x-coordinate of the obstacle. The specific calculation formula is as follows: ， ， Level 1 threat coefficient describes the straight-line distance between the obstacle and the unmanned vessel, while Level 2 threat coefficient describes the lateral distance between the obstacle and the unmanned vessel. Level 2 threat coefficient applies to obstacles directly opposite the direction of travel of the unmanned vessel, and both Level 1 and Level 2 threat coefficients apply to the obstacle avoidance process. S4. All obstacles detected within the detection range of the unmanned vessel will be calculated as level 1 or 2 threat coefficients, but the unmanned vessel will establish a surrounding safety zone; the surrounding safety zone is a rectangular area with 3.5m on each side of the unmanned vessel and 30m in front and behind it. S5. To facilitate description in a two-dimensional coordinate system, the parameters describing the two-dimensional motion of the unmanned vessel are: linear velocity. and angular velocity Convert to x-axis velocity and y-axis velocity, denoted as... The calculation formula is as follows: ， ， in Sampling time Based on the coordinates of all currently detected obstacles, calculate The formula is as follows: ， ， The coordinate system is divided into two regions, the left half and the right half, with the y-axis as the dividing line. It is the difference of the level 1 coefficients of all obstacle coordinates in the left and right halves multiplied by a scaling factor. Add the difference between the level 2 coefficients of all obstacle coordinates in the left and right halves, multiplied by a scaling factor. ; yes Multiply by a scaling factor Add the difference between the level 2 coefficients of all obstacle coordinates in the left and right halves, multiplied by a scaling factor. ; Calculated and The data is transmitted to the drive controller, which controls the drive mechanical device (7) to ensure that the unmanned vessel can avoid obstacles according to the central obstacle avoidance algorithm.

Images