A ship navigation intelligent obstacle avoidance method based on multi-modal perception

Abstract

The application discloses a kind of ship navigation intelligent obstacle avoidance methods based on multi-modal perception, it is related to ship intelligent navigation technical field, including the following steps: acquisition multi-modal perception data and standardization, generate ship navigation perception dataset;Arrange perception node, generate ship multi-node perception layout;Establish ship circumferential perception coordinates and generate ship circumferential layered covering relationship;Divide far distance early warning area, near distance identification area and berthing monitoring area, generate ship zoned perception state;Process far distance radar data and near distance millimeter wave radar data, generate radar target perception result;Process visual image data, generate visual target perception result and ship look-around panoramic image;Match radar target perception result and visual target perception result, combined with ship body attitude change is directed to compensation and collision risk judgment, generate ship navigation intelligent obstacle avoidance result.The application has the advantages of wide perception range, target fusion accuracy and obstacle avoidance judgment stability.

Description

Technical Field

[0001] This invention relates to the field of intelligent ship navigation technology, and in particular to an intelligent obstacle avoidance method for ship navigation based on multimodal perception. Background Technology

[0002] In recent years, with the development of intelligent navigation, radar detection, video surveillance, image recognition, and multi-sensor fusion technologies, environmental perception and obstacle avoidance warning in scenarios such as long-distance voyages, port entry, and berthing have gradually become important directions in navigation safety technology. Existing technologies typically equip ships with millimeter-wave radar, security radar, visible light cameras, infrared illumination devices, or attitude sensors to detect buoys, small boats, fishing vessels, port facilities, and other obstacles around the ship. Radar ranging and speed measurement, video target recognition, image stitching, and attitude compensation are used to assist the navigator in assessing the surrounding environment, thereby reducing the risk of collisions at night, in heavy fog, in low visibility conditions, and during berthing.

[0003] However, existing ship obstacle avoidance perception solutions still suffer from insufficient multi-source perception coordination capabilities. On the one hand, a single radar or vision device cannot simultaneously handle long-range early warning, close-range identification, and near-field monitoring during berthing. While radar can acquire target distance, azimuth, and speed, its target category identification capability is limited. Although vision can identify target areas and target categories, it is easily affected by visibility, field of view, and hull obstruction. On the other hand, existing solutions lack a unified approach to the installation location, horizontal pointing, pitch operating mode, and perception range of perception nodes, making it difficult to establish a circumferential layered coverage relationship for the ship and to perform zoned perception according to long-range early warning areas, close-range identification areas, and berthing monitoring areas. Furthermore, the lack of continuous fusion and compensation processing between radar target perception results, visual target perception results, ship panoramic images, and changes in ship attitude leads to insufficient accuracy and stability in collision risk assessment. Summary of the Invention

[0004] One objective of this invention is to propose an intelligent obstacle avoidance method for ship navigation based on multimodal perception. This invention makes full use of multimodal perception, ship circumferential layered coverage, and hull attitude compensation technology. It describes in detail the method for ships to achieve zoned perception, target fusion, and collision risk assessment in scenarios of long-distance voyage, port entry, and berthing. It has the advantages of wide perception range, accurate target recognition, strong adaptability to low visibility, and high stability of obstacle avoidance judgment.

[0005] According to an embodiment of the present invention, a ship navigation intelligent obstacle avoidance method based on multimodal perception includes the following steps: Collect multimodal perception data during ship navigation and perform standardization processing to generate a ship navigation perception dataset; Organize the installation locations, horizontal orientation, pitch working modes, and sensing range of the sensing nodes to generate a multi-node sensing layout for the ship. Based on the multi-node perception layout of the ship, the ship's circumferential perception coordinates are established, and the radar detection range, visual imaging range and pitch working mode are associated to generate the ship's circumferential layered coverage relationship. Based on the circumferential layered coverage relationship of the ship, the long-range early warning area, the close-range identification area and the berthing monitoring area are divided to generate the ship's zoned perception status; Based on the ship's zoned perception state processing, long-range radar data and short-range millimeter-wave radar data are used to identify the distance, bearing, speed and track of targets around the ship and generate radar target perception results. Based on the ship's zone perception state processing visual image data, the target area and target category of the target around the ship are identified, adjacent field-of-view images are stitched together, and visual target perception results and ship's panoramic view images are generated. The radar target perception results and visual target perception results are matched to generate multimodal fusion target results. The pointing compensation of the multimodal fusion target results and the ship's panoramic image is performed according to the ship's attitude change. The collision risk is judged by combining the target distribution in the ship's panoramic image, and the intelligent obstacle avoidance result of ship navigation is generated.

[0006] Optionally, the multimodal sensing data includes long-range radar data, short-range millimeter-wave radar data, visual image data, ship attitude data, sensing node layout information, pitch working mode data, radar visual calibration parameters, and ship navigation status data. The standardization processing includes data acquisition time synchronization, unified sensing node identification, unified data format, unified ship coordinate reference, unified radar data dimensions, visual image frame rate adjustment, pitch working mode marking, attitude data alignment, abnormal data removal, and missing data completion.

[0007] Optionally, the generation of the ship's multi-node sensing layout includes: Based on the ship navigation perception dataset, the installation positions of each perception node at the bow, midship and stern are determined, and the installation positions are sorted according to the forward and backward direction and the left and right direction of the hull. The horizontal orientation of each sensing node is determined based on its installation location, and then the horizontal orientation is associated with its installation location. Mark the pitch working mode of each sensing node and associate the pitch working mode with the horizontal direction of each sensing node; The sensing range is organized according to the radar detection range and visual imaging range of each sensing node, and the sensing range is combined with the installation position, horizontal pointing and elevation working mode. Based on the combined installation location, horizontal orientation, pitch working mode, and sensing range, a multi-node sensing layout for the ship is generated.

[0008] Optionally, the generation of the ship's circumferential layered coverage relationship includes: Using the bow direction of the ship as the starting direction for angle calculation and the centerline of the ship as the orientation reference, the circumferential sensing coordinates of the ship are established based on the installation position and horizontal direction of each sensing node in the multi-node sensing layout of the ship. The installation location of each sensing node is written into the ship's circumferential sensing coordinates, and the coverage orientation of each sensing node in the ship's circumferential sensing coordinates is determined according to the horizontal direction. According to the elevation working mode of each sensing node, the radar detection range and visual imaging range are respectively associated with the coverage azimuth, generating the layered coverage content of each sensing node. The layered coverage content of all sensing nodes is combined according to the arrangement order of bow, midship and stern to generate the circumferential layered coverage relationship of the ship.

[0009] Optionally, the generation of the ship's zone awareness state includes: Based on the radar detection range in the circumferential layered coverage relationship of the ship, the area covered by long-range radar data and where the visual imaging range does not form a target category judgment is determined, and a long-range early warning area is generated. Based on the overlapping part of the radar detection range and visual imaging range in the circumferential layered coverage relationship of the ship, the area that can simultaneously perform target distance identification, target orientation identification and target category judgment is determined, and a close-range identification area is generated. Based on the pitch working mode in the circumferential layered coverage relationship of the ship, the radar detection range and visual imaging range near the hull in the berthing mode are determined, and the berthing monitoring area is generated. The long-range early warning area, the close-range identification area, and the berthing monitoring area are combined according to the ship's circumferential perception coordinates to generate the ship's zone perception status.

[0010] Optionally, the generation of the radar target perception results includes: Determine the location of the long-range early warning area, the close-range identification area, and the berthing monitoring area in the ship's circumferential sensing coordinates according to the ship's zone sensing status. Match long-range radar data to the long-range early warning area, identify the distance, bearing and speed of targets around the ship in the long-range early warning area, and generate long-range radar identification content. The close-range millimeter-wave radar data is matched to the close-range identification area and the berthing monitoring area. The distance, bearing and speed of targets around the ship in the close-range identification area and the berthing monitoring area are identified to generate close-range radar identification content. The data collected from long-range and short-range radars are continuously processed according to the time of collection to form the tracks of targets around the ship. The distance, bearing, speed, and trajectory of targets around the ship are combined to generate radar target perception results.

[0011] Optionally, the generation of the visual target perception results and the ship's panoramic surround view image includes: Determine the location of the near-field identification area and the berthing monitoring area in the ship's circumferential sensing coordinates according to the ship's zone sensing status. The visual image data is matched to the near-field recognition area and the berthing monitoring area according to the coverage direction of each sensing node. The matched visual image data is then corrected and the image clarity is improved to generate regional visual images. Contour extraction and location marking of targets around ships in regional visual images are performed to generate target regions; The target area is classified, a target category is generated, and the target area, target category and acquisition time are combined to generate a visual target perception result; Based on the coverage orientation of each sensing node, adjacent field-of-view images are extracted. The overlapping areas of the adjacent field-of-view images are registered, the brightness is adjusted, and the boundaries are fused to generate a panoramic image of the ship.

[0012] Optionally, the generation of the intelligent obstacle avoidance results for ship navigation includes: The radar target perception results and visual target perception results are matched according to the collection time and the ship's circumferential perception coordinates. The distance, bearing, speed, track, target area and target category of the same target around the ship are combined to generate multimodal fusion target results. Based on the changes in the ship's attitude, the pitch and roll changes are processed to generate a pointing compensation amount; The target orientation in the multimodal fusion target result and the target display position in the ship's panoramic image are corrected by using the pointing compensation amount, and the compensated multimodal fusion target result and the compensated ship's panoramic image are generated. Collision risk assessment is performed based on the compensated multimodal fusion target results and the target distribution in the compensated ship surround panoramic image, identifying dangerous targets, dangerous locations and dangerous levels, and generating intelligent obstacle avoidance results for ship navigation.

[0013] The beneficial effects of this invention are: This invention collects and standardizes multimodal perception data during ship navigation, unifying long-range radar data, short-range millimeter-wave radar data, visual image data, ship attitude data, perception node layout information, pitch working mode data, radar visual calibration parameters, and ship navigation status data into a unified ship navigation perception dataset. This enables data from different perception sources to participate in subsequent processing under a unified time, unified node identification, and unified ship coordinate reference, reducing information loss in low visibility, long-range detection, short-range identification, and ship occlusion scenarios using a single radar or single vision device, and improving the data integrity and consistency of ship navigation environment perception.

[0014] This invention generates a multi-node sensing layout for ships by organizing the installation positions, horizontal orientation, pitch working modes, and sensing ranges of sensing nodes. It further establishes the ship's circumferential sensing coordinates and circumferential layered coverage relationship, enabling sensing nodes arranged at the bow, midship, and stern to form a circumferential coverage structure according to radar detection range, visual imaging range, and pitch working mode. This facilitates the division of the ship's surrounding environment into long-range early warning areas, short-range identification areas, and berthing monitoring areas. It also enables long-range radar data, short-range millimeter-wave radar data, and visual image data to undertake corresponding sensing tasks according to different navigation scenarios, improving the zoned sensing capabilities during long voyages, port entry, and berthing.

[0015] This invention processes radar data and visual image data separately based on the ship's zoned perception state to generate radar target perception results, visual target perception results, and a ship's panoramic view image. It then generates a multimodal fusion target result by matching the radar and visual target perception results, enabling the related representation of the distance, bearing, speed, track, target area, and target category of targets around the ship, thus improving the reliability of target recognition and positioning. Simultaneously, it compensates for the pointing of the multimodal fusion target result and the ship's panoramic view image based on changes in the ship's attitude, and combines this with the target distribution in the panoramic view image to assess collision risk. This reduces the impact of ship pitch and roll on the perception pointing and target display position, improving the stability of the judgment of dangerous targets, dangerous bearings, and dangerous levels. This makes the intelligent obstacle avoidance results for ship navigation more applicable to complex navigation scenarios such as nighttime, heavy fog, low visibility, and berthing near-field conditions. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a ship navigation intelligent obstacle avoidance method based on multimodal perception proposed in this invention; Figure 2This is a schematic diagram of the visual target perception results and the generation of a panoramic image of a ship's surrounding view, based on a multimodal perception-based intelligent obstacle avoidance method for ship navigation proposed in this invention. Figure 3 This is a schematic diagram illustrating the generation of intelligent obstacle avoidance results for a ship navigation intelligent obstacle avoidance method based on multimodal perception proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-3 A method for intelligent obstacle avoidance in ship navigation based on multimodal perception includes the following steps: Collect multimodal perception data during ship navigation and perform standardization processing to generate a ship navigation perception dataset; Organize the installation locations, horizontal orientation, pitch working modes, and sensing range of the sensing nodes to generate a multi-node sensing layout for the ship. Based on the multi-node perception layout of the ship, the ship's circumferential perception coordinates are established, and the radar detection range, visual imaging range and pitch working mode are associated to generate the ship's circumferential layered coverage relationship. Based on the circumferential layered coverage relationship of the ship, the long-range early warning area, the close-range identification area and the berthing monitoring area are divided to generate the ship's zoned perception status; Based on the ship's zoned perception state processing, long-range radar data and short-range millimeter-wave radar data are used to identify the distance, bearing, speed and track of targets around the ship and generate radar target perception results. Based on the ship's zone perception state processing visual image data, the target area and target category of the target around the ship are identified, adjacent field-of-view images are stitched together, and visual target perception results and ship's panoramic view images are generated. The radar target perception results and visual target perception results are matched to generate multimodal fusion target results. The pointing compensation of the multimodal fusion target results and the ship's panoramic image is performed according to the ship's attitude change. The collision risk is judged by combining the target distribution in the ship's panoramic image, and the intelligent obstacle avoidance result of ship navigation is generated.

[0019] In this embodiment, the multimodal sensing data includes long-range radar data, short-range millimeter-wave radar data, visual image data, ship attitude data, sensing node layout information, pitch working mode data, radar visual calibration parameters, and ship navigation status data. The standardization processing includes data acquisition time synchronization, unified sensing node identification, unified data format, unified ship coordinate reference, unified radar data units, visual image frame rate adjustment, pitch working mode marking, attitude data alignment, abnormal data removal, and missing data completion.

[0020] In this embodiment, the generation of the ship's multi-node sensing layout includes: Based on the ship navigation perception dataset, the installation positions of each perception node at the bow, midship and stern are determined, and the installation positions are sorted according to the forward and backward direction and the left and right direction of the hull. Based on the installation location of each sensing node, the horizontal orientation of each sensing node is organized and associated with its installation location. Specifically, the bow direction of the ship is used as the reference direction for horizontal orientation, and the arrangement orientation of each sensing node relative to the bow direction is determined by the ship's centerline and the installation boundary on the outer side of the hull. For sensing nodes located at the bow, their installation location is marked as the forward observation position, and the horizontal orientation is organized to be a forward orientation along the bow direction. For sensing nodes located at the stern, their installation location is marked as the aft observation position, and the horizontal orientation is organized to be an aft orientation along the stern direction. For sensing nodes located in the middle of the ship, based on their position on the port or starboard side, and their position fore and aft near the bow or stern, the horizontal orientation towards port forward, port aft, starboard forward, or starboard aft is organized. After the horizontal orientation is organized, the node identifier, installation location, and horizontal orientation of each sensing node are written into the same node arrangement record. The pitch operating mode of each sensing node is marked, and the pitch operating mode is associated with the horizontal direction of each sensing node. Specifically, according to the ship's navigation state, the pitch operating modes are divided into long-distance mode and berthing mode. Long-distance mode is the horizontal observation state, and berthing mode is the downward observation state. For each sensing node, the installation position and horizontal direction of the sensing node are retrieved, and the long-distance mode and berthing mode are written into the node layout record. The pitch angle in the long-distance mode is bound to the horizontal direction of the sensing node, so that the sensing node can perform long-distance observation along the predetermined horizontal direction in the long-distance mode. The pitch angle in the berthing mode is bound to the horizontal direction of the sensing node, so that the sensing node can perform near-field observation along the predetermined horizontal direction in the berthing mode. After the binding is completed, a fixed association content between the sensing node, installation position, horizontal direction, long-distance mode and berthing mode is formed in the node layout record, and the fixed association content is written into the ship's multi-node sensing layout. The sensing range is organized according to the radar detection range and visual imaging range of each sensing node, and then combined with the installation location, horizontal pointing, and elevation operating mode. Specifically, for each sensing node, the installation location, horizontal pointing, and elevation operating mode in the node layout record are retrieved to obtain the radar detection range and visual imaging range of that sensing node in long-range mode and parking mode. The radar detection range is organized according to the detection distance, horizontal coverage angle, and vertical coverage angle, and the visual imaging range is organized according to the imaging distance, horizontal field of view, and vertical field of view. Taking the installation location of the sensing node as the starting point of the sensing range, the horizontal pointing as the center direction of the sensing range, and the elevation operating mode as the vertical pointing state of the sensing range, the radar detection range and visual imaging range are mapped to the corresponding directional areas. For the same sensing node, the mapped radar detection range and visual imaging range are merged to form the sensing range of that sensing node. The sensing range, along with the installation location, horizontal pointing, and elevation operating mode, is written into the same node layout record. Based on the combined installation location, horizontal orientation, pitch working mode, and sensing range, a multi-node sensing layout for the ship is generated.

[0021] In this embodiment, the generation of the circumferential layered coverage relationship of the ship includes: Using the bow direction of the ship as the starting direction for angle calculation and the centerline of the ship as the orientation reference, the circumferential sensing coordinates of the ship are established based on the installation position and horizontal direction of each sensing node in the multi-node sensing layout of the ship. The establishment of the ship's circumferential sensing coordinates is as follows: taking the ship's bow as the zero-degree direction, the ship's bow is extended along the horizontal plane to form the angle calculation direction; taking the ship's centerline as the bearing adjustment reference, the installation positions on both sides of the ship's centerline are respectively arranged as port and starboard bearings; for each sensing node in the ship's multi-node sensing layout, the installation position and horizontal direction of the sensing node are obtained, the installation position is converted into a bearing position relative to the ship's centerline, and the horizontal direction is converted into a circumferential angle relative to the ship's bow; for sensing nodes located at the bow, their circumferential angles are arranged as forward angles; for sensing nodes located at the stern, their circumferential angles are arranged as aft angles; for sensing nodes located in the middle of the ship, their circumferential angles are arranged according to port and starboard bearings and positions near the bow or stern; the installation position, bearing position, and circumferential angle of each sensing node are written into the ship's circumferential sensing coordinates, so that each sensing node has a fixed spatial position and a fixed horizontal direction in the ship's circumferential sensing coordinates. The installation location of each sensing node is written into the ship's circumferential sensing coordinates, and the coverage orientation of each sensing node in the ship's circumferential sensing coordinates is determined according to the horizontal direction. The determination of the coverage orientation is as follows: the installation position of each sensing node in the ship's multi-node sensing layout is extracted one by one, the installation position is converted into longitudinal coordinates according to the ship's fore-and-aft direction, the installation position is converted into lateral coordinates according to the ship's left-and-right direction, and the longitudinal and lateral coordinates are written into the position record of the corresponding sensing node in the ship's circumferential sensing coordinates; when determining the coverage orientation of each sensing node in the ship's circumferential sensing coordinates according to the horizontal direction, the horizontal direction of each sensing node is converted into a circumferential angle relative to the ship's bow, and the circumferential angle is used as the center of the coverage orientation of the sensing node. The sensing node located at the bow is marked as the forward coverage orientation, the sensing node located at the stern is marked as the aft coverage orientation, and the sensing node located in the middle of the ship is marked as the coverage orientation according to the port forward, port aft, starboard forward, or starboard aft orientation, and the coverage orientation is bound to the position record of the sensing node; According to the elevation working mode of each sensing node, the radar detection range and visual imaging range are respectively associated with the coverage azimuth, generating the layered coverage content of each sensing node. The generation of layered coverage content is specifically as follows: The coverage azimuth and elevation operating mode of each sensing node are acquired sequentially; the radar detection range in long-range mode is associated with the coverage azimuth to form the radar coverage content of that sensing node in long-range mode; the visual imaging range in long-range mode is associated with the coverage azimuth to form the visual coverage content of that sensing node in long-range mode; the radar detection range in parking mode is associated with the same coverage azimuth to form the radar coverage content of that sensing node in parking mode; the visual imaging range in parking mode is associated with the same coverage azimuth to form the visual coverage content of that sensing node in parking mode; the radar coverage content in long-range mode, the visual coverage content in long-range mode, the radar coverage content in parking mode, and the visual coverage content in parking mode are merged according to the sensing nodes to generate the layered coverage content of each sensing node. The layered coverage content of all sensing nodes is combined according to the arrangement order of bow, midship and stern to generate the circumferential layered coverage relationship of the ship.

[0022] In this embodiment, the generation of the ship's zone sensing status includes: Based on the radar detection range in the circumferential layered coverage relationship of the ship, the area covered by long-range radar data and where the visual imaging range does not form a target category judgment is determined, and a long-range early warning area is generated. The generation of the long-range early warning area is specifically as follows: The coverage azimuth and radar detection range of each sensing node are acquired sequentially, and the radar detection range is expanded along the coverage azimuth to form the radar coverage area; within the same coverage azimuth, the area within the visual imaging range capable of determining the target category is used as the near-end reference area; along the outer extension direction of the radar coverage area, the far-end radar coverage area extending beyond the near-end reference area is extracted; the far-end radar coverage areas of each sensing node are stitched together and deduplicated according to the ship's circumferential sensing coordinates to form a long-range coverage area distributed around the ship's circumference; the long-range coverage area is marked as the long-range early warning area. Based on the overlapping part of the radar detection range and visual imaging range in the circumferential layered coverage relationship of the ship, the area that can simultaneously perform target distance identification, target orientation identification and target category judgment is determined, and a close-range identification area is generated. The generation of the close-range identification area is specifically as follows: the coverage azimuth, radar detection range, and visual imaging range of each sensing node are acquired one by one; the radar detection range is expanded along the coverage azimuth to form the radar coverage area, and the visual imaging range is expanded along the same coverage azimuth to form the visual coverage area; within the same coverage azimuth, the overlapping part of the radar coverage area and the visual coverage area is truncated to form the overlapping coverage area of ​​the same sensing node; within the overlapping coverage area, the area that can provide the target distance and target azimuth from the radar target perception results, and the target area and target category from the visual target perception results are retained; the overlapping coverage areas of each sensing node are stitched together and deduplicated according to the ship's circumferential sensing coordinates to generate a close-range identification area distributed around the ship's circumference. Based on the pitch working mode in the circumferential layered coverage relationship of the ship, the radar detection range and visual imaging range near the hull in the berthing mode are determined, and the berthing monitoring area is generated. The generation of the berthing monitoring area is specifically as follows: The coverage azimuth, pitch operating mode, radar detection range, and visual imaging range of each sensing node are acquired one by one; the radar detection range and visual imaging range with the pitch operating mode set to berthing mode are selected; the radar detection range in berthing mode is expanded along the coverage azimuth to form the berthing radar coverage area, and the visual imaging range in berthing mode is expanded along the same coverage azimuth to form the berthing visual coverage area; taking the installation position of each sensing node as the starting point of the area, the berthing radar coverage area and berthing visual coverage area are extracted along the direction closest to the ship's perimeter; the berthing radar coverage area and berthing visual coverage area within the same coverage azimuth are merged to form the berthing coverage area of ​​each sensing node; the berthing coverage areas of each sensing node are spliced ​​and deduplicated according to the ship's circumferential sensing coordinates to generate the berthing monitoring area. The long-range early warning area, the close-range identification area, and the berthing monitoring area are combined according to the ship's circumferential perception coordinates to generate the ship's zone perception status.

[0023] In this embodiment, the generation of radar target perception results includes: Determine the location of the long-range early warning area, the close-range identification area, and the berthing monitoring area in the ship's circumferential sensing coordinates according to the ship's zone sensing status. Match long-range radar data to the long-range early warning area, identify the distance, bearing and speed of targets around the ship in the long-range early warning area, and generate long-range radar identification content. The generation of long-range radar identification content specifically involves: acquiring the detection point distance, detection point angle, radial velocity, and acquisition time frame by frame from the long-range radar data; obtaining the coverage azimuth and installation position of the sensing node where the long-range radar is located in the ship's circumferential sensing coordinates; superimposing the detection point angle onto the coverage azimuth of the sensing node to obtain the circumferential angle of the detection point in the ship's circumferential sensing coordinates; calculating the position of the detection point in the ship's circumferential sensing coordinates according to the detection point distance, circumferential angle, and installation position of the sensing node; comparing the calculated position with the long-range warning area, and retaining the detection points that fall within the long-range warning area. Detection points not falling within the long-range early warning area are removed; detection points with similar distances, circumferential angles, and radial velocities within the same acquisition time are aggregated to form radar target points for targets around the ship; the distances of all detection points within the radar target points are averaged to obtain the distances of targets around the ship; the circumferential angles of all detection points within the radar target points are averaged to obtain the bearings of targets around the ship; the radial velocities of all detection points within the radar target points are averaged to obtain the velocities of targets around the ship; the distances, bearings, velocities, and acquisition times of targets around the ship are then written into the long-range radar identification data. The close-range millimeter-wave radar data is matched to the close-range identification area and the berthing monitoring area. The distance, bearing and speed of targets around the ship in the close-range identification area and the berthing monitoring area are identified to generate close-range radar identification content. The generation of near-range radar identification content specifically involves: acquiring the detection point distance, detection point angle, radial velocity, and acquisition time frame by frame from the near-range millimeter-wave radar data; obtaining the coverage azimuth, installation position, and pitch operating mode of the sensing node where the near-range millimeter-wave radar is located; superimposing the detection point angle onto the coverage azimuth of the sensing node to obtain the circumferential angle of the detection point in the ship's circumferential sensing coordinates; calculating the position of the detection point in the ship's circumferential sensing coordinates according to the detection point distance, circumferential angle, and installation position of the sensing node; matching detection points with pitch operating mode in long-range navigation mode with the near-range identification area, and matching detection points with pitch operating mode in berthing mode with the berthing monitoring area. The system matches and retains detection points that fall within the close-range identification area or berthing monitoring area; it aggregates detection points that are close in distance, circumferential angle, and radial velocity within the same acquisition time to form close-range radar target points around the ship; it averages the distances of each detection point within the close-range radar target point to obtain the distance of the target around the ship; it averages the circumferential angles of each detection point within the close-range radar target point to obtain the bearing of the target around the ship; it averages the radial velocities of each detection point within the close-range radar target point to obtain the velocity of the target around the ship; and it writes the distance, bearing, velocity, acquisition time, and region of the target around the ship into the close-range radar identification data. The data collected from long-range and short-range radars are continuously processed according to the time of collection to form the tracks of targets around the ship. The formation of the track of targets around the ship is as follows: Long-range and short-range radar identification data are arranged from earliest to latest according to acquisition time, and the target distance, bearing, and speed at the same acquisition time are compiled into a target record; target records from adjacent acquisition times are compared, and target records with continuous changes in target distance, bearing, and speed are grouped into the same target around the ship; when a target record from the long-range radar identification data enters the short-range identification area or berthing monitoring area, this target record is connected with target records from the short-range radar identification data that are close in position and have continuous speed changes; target records grouped into the same target around the ship are arranged sequentially according to acquisition time, retaining the target distance, bearing, and speed at each acquisition time; the arranged target records are then connected to form the track of targets around the ship. The distance, bearing, speed, and trajectory of targets around the ship are combined to generate radar target perception results.

[0024] In this embodiment, the generation of visual target perception results and ship surround panoramic images includes: Determine the location of the near-field identification area and the berthing monitoring area in the ship's circumferential sensing coordinates according to the ship's zone sensing status. The visual image data is matched to the near-field recognition area and the berthing monitoring area according to the coverage direction of each sensing node. The matched visual image data is then corrected and the image clarity is improved to generate regional visual images. The generation of regional visual images specifically involves: acquiring visual image data collected by each sensing node and retrieving the coverage orientation of each sensing node in the ship's circumferential sensing coordinates; marking the visual image data into the near-range recognition area or berthing monitoring area under the same coverage orientation; performing region matching according to the near-range recognition area for visual image data in long-distance navigation mode, and performing region matching according to the berthing monitoring area for visual image data in berthing mode; after completing region matching, performing lens distortion correction on the matched visual image data to remove image edge curvature and target contour offset; unifying the frame rate and screen size of the corrected visual image data to ensure that the visual images of different sensing nodes maintain the same processing scale; performing brightness adjustment, contrast adjustment, and noise suppression on the unified visual image data to enhance target boundaries and target contours; and filtering the visual image data according to image clarity, retaining visual images with continuous target boundaries and complete target contours to generate regional visual images. Contour extraction and location marking of targets around ships in regional visual images are performed to generate target regions; The target area is classified, a target category is generated, and the target area, target category and acquisition time are combined to generate a visual target perception result; The generation of visual target perception results is specifically as follows: Target regions are cropped one by one from the visual image, preserving their boundaries, contours, area, aspect ratio, brightness distribution, and edge direction. The cropped target regions are adjusted to a uniform image size, and invalid backgrounds within the target regions are masked to obtain target region images. Contour integrity checks are performed on the target region images, marking regions with broken contours or severe target occlusion as areas to be reviewed. Target regions with continuous contours and clear boundaries are input into the target category judgment process. During the target category judgment process, the shape contour, size ratio, edge density, and brightness distribution in the target region images are comprehensively matched to classify the target regions into buoys, patrol boats, civilian fishing vessels, or unconfirmed targets, generating target categories. The location, target category, acquisition time, and associated perception node of the same target region are combined to form a single visual target record. All visual target records acquired within the same time period are arranged according to the ship's circumferential perception coordinates to generate visual target perception results. Based on the coverage orientation of each sensing node, adjacent field-of-view images are extracted, and overlapping areas of adjacent field-of-view images are registered, brightness is adjusted, and boundaries are fused to generate a panoramic image of the ship. The generation of a ship's panoramic view image is specifically as follows: The regional visual images collected by each sensing node are arranged in order of coverage orientation, and two adjacent regional visual images in the coverage orientation are selected as adjacent field-of-view images; the overlapping area located at the coverage boundary is extracted from the adjacent field-of-view images, and the target edges, fixed structure contours, and texture features within the overlapping area are matched to obtain the image position offset between the two adjacent field-of-view images; the adjacent field-of-view images are translated, rotated, and scaled according to the image position offset to ensure that the same target or the same environmental structure in the overlapping area falls into a consistent position; the brightness of the registered adjacent field-of-view images is adjusted to ensure that the brightness, contrast, and hue on both sides of the overlapping area are continuous; boundary fusion is performed at the boundary of the overlapping area to smoothly stitch together the edge transition areas of the two adjacent field-of-view images; the overlapping area registration, brightness adjustment, and boundary fusion of all adjacent field-of-view images are completed sequentially according to the coverage orientation to generate a ship's panoramic view image unfolding around the ship's circumference.

[0025] In this embodiment, the generation of intelligent obstacle avoidance results for ship navigation includes: The radar target perception results and visual target perception results are matched according to the collection time and the ship's circumferential perception coordinates. The distance, bearing, speed, track, target area and target category of the same target around the ship are combined to generate multimodal fusion target results. The generation of multimodal fusion target results is specifically as follows: The target range, bearing, speed, track, and acquisition time from the radar target perception results are arranged according to the ship's circumferential perception coordinates; the target area, target category, and acquisition time from the visual target perception results are arranged according to the ship's circumferential perception coordinates; During the same acquisition time or adjacent acquisition times, radar target perception results and visual target perception results whose target bearings fall within the same coverage area are selected; the target range from the radar target perception results is projected along the target bearing to the target position in the ship's circumferential perception coordinates; the center position of the target area in the visual target perception results is converted to the visual position in the ship's circumferential perception coordinates; the proximity between the target position and the visual position is compared; radar target perception results and visual target perception results with close positions and continuous acquisition times are marked as targets around the same ship; for targets around the same ship, the range, bearing, speed, and track from the radar target perception results are merged with the target area and target category from the visual target perception results to generate multimodal fusion target results; Based on the changes in the ship's attitude, the pitch and roll changes are processed to generate a pointing compensation amount; The generation of the pointing compensation amount is as follows: The pitch and roll angles in the ship's attitude data are acquired sequentially according to the acquisition time. The pitch angle at the current acquisition time is subtracted from the pitch angle at the previous acquisition time to obtain the pitch angle change. The roll angle at the current acquisition time is subtracted from the roll angle at the previous acquisition time to obtain the roll angle change. The pitch angle changes over multiple consecutive acquisition times are averaged to obtain the continuous pitch change. The roll angle changes over multiple consecutive acquisition times are averaged to obtain the continuous roll change. The positive or negative direction of the pitch angle change determines the bow pitch or bow sinking state, and the positive or negative direction of the roll angle change determines the ship's port or starboard list state. The continuous pitch change is converted into a pitch compensation angle according to the pitch mode of the sensing node. The continuous roll change is converted into a horizontal compensation angle according to the horizontal pointing direction of the sensing node, with opposite compensation directions for the port and starboard directions. The pitch compensation angle and the horizontal compensation angle are combined according to the acquisition time to generate the pointing compensation amount. The target orientation in the multimodal fusion target result and the target display position in the ship's panoramic image are corrected by using the pointing compensation amount, and the compensated multimodal fusion target result and the compensated ship's panoramic image are generated. The generation of the compensated multimodal fusion target result is as follows: the horizontal compensation angle in the pointing compensation quantity is superimposed with the target bearing in the multimodal fusion target result to obtain the corrected target bearing; the pitch compensation angle in the pointing compensation quantity is associated with the coverage bearing to which the target bearing belongs to correct the bearing assignment of the target in the long-range warning area, the close-range identification area, or the berthing monitoring area; the corrected target bearing of the target around the same ship during the continuous acquisition time is smoothed and sorted, eliminating bearing jumps caused by short-term hull rolling, and retaining the continuously changing target bearing; the corrected target bearing, target distance, target speed, track, target area, and target category are recombined to generate the compensated multimodal fusion target result; The generation of the compensated ship panoramic image is specifically as follows: the horizontal compensation angle is converted into the image lateral offset, the pitch compensation angle is converted into the image longitudinal offset, and the target display position in the ship panoramic image is moved according to the image lateral and longitudinal offsets to make the target display position consistent with the corrected target orientation in the compensated multimodal fusion target result; the moved target display position is then subjected to boundary cropping and overlapping area smoothing to generate the compensated ship panoramic image. Collision risk assessment is performed based on the compensated multimodal fusion target results and the target distribution in the compensated ship surround panoramic image, identifying dangerous targets, dangerous locations and dangerous levels, and generating intelligent obstacle avoidance results for ship navigation. The determination of hazardous targets, hazardous bearings, and hazardous levels involves the following steps: First, acquiring the target distance, bearing, speed, trajectory, and category from the compensated multimodal fusion target results based on the acquisition time. Then, acquiring the target distribution in the compensated ship-wide panoramic image at the same acquisition time. Based on the ship's navigation status data, determining the ship's navigation direction and berthing movement direction, comparing the target bearing with the ship's navigation direction, and filtering targets located ahead, to the side, and along the berthing movement direction of the ship's navigation direction. For the filtered targets around the ship, comparing changes in target distance over continuous acquisition time; if the target distance continuously decreases, it is marked as an approaching target. Finally, comparing the target distance over continuous acquisition time... If the bearing of a target continues to approach the ship's sailing direction or berthing direction, it is marked as a bearing convergence target. Based on the approach trend of the target speed and track, the approaching target, the bearing convergence target, and the dense target distribution in the compensated ship's panoramic image are combined for judgment. Targets around the ship that simultaneously have the following characteristics are identified as dangerous targets: close target distance, high target speed, track pointing towards the ship's sailing direction or berthing direction, and target distribution located in the close-range identification area or berthing monitoring area. The bearing of the dangerous target in the ship's circumferential sensing coordinates is identified as the dangerous bearing. The danger level is determined according to the target distance from near to far, the target speed from high to low, and the track proximity from strong to weak.

[0026] Example 1: To verify the feasibility of this invention in practice, it was applied to the navigation scenario of a cargo ferry in a coastal port area. This vessel needs to navigate continuously in the outer harbor channel, the turning area of ​​the harbor basin, and the berthing area. During navigation, it is prone to encountering situations such as insufficient lighting at night, morning sea fog, small vessels passing through the port area, dense distribution of buoys, and limited near-field visibility during berthing. Traditional methods relying on a single radar or single-channel video observation, while able to assist the operator in detecting obstacles to some extent, still suffer from problems such as unclear target types at long distances, unstable orientation of close-range targets, and discontinuous observations from the side and stern. Especially during berthing, the operator needs to repeatedly switch between radar and video feeds, making it difficult to form a unified obstacle avoidance judgment in a timely manner.

[0027] In this scenario, the ship deploys sensing nodes at its bow, midship, and stern to collect long-range radar data, short-range millimeter-wave radar data, visual image data, and hull attitude data. This data is then standardized by combining sensing node deployment information, pitch mode data, radar visual calibration parameters, and ship navigation status data to generate a ship navigation sensing dataset. The system organizes the installation locations, horizontal pointing, pitch modes, and sensing ranges of the sensing nodes to generate a multi-node sensing layout for the ship. Based on this layout, a circumferential sensing coordinate system is established, creating a hierarchical circumferential coverage relationship. The waters surrounding the ship are then divided into a long-range warning zone, a short-range identification zone, and a berthing monitoring zone. When the ship is navigating in the outer harbor channel, long-range radar data is used to identify the distance, bearing, speed, and track of distant targets. After entering the harbor basin, short-range millimeter-wave radar data and visual image data are used together for target area and target category identification. After entering the berthing area, the system focuses on processing targets around the hull within the berthing monitoring zone and stitches adjacent field-of-view images to generate a panoramic image of the ship.

[0028] In practical applications, the system matches radar target perception results with visual target perception results to generate multimodal fusion target results, enabling the unified presentation of the distance, bearing, speed, track, target area, and target category of targets around the same vessel. When the vessel is affected by surges, turning, or berthing operations, the system compensates for the direction of the multimodal fusion target results and the vessel's panoramic image based on changes in hull attitude, ensuring that the target bearing displayed on the bridge interface remains consistent with the actual circumferential bearing of the vessel. The helmsman can view the vessel's panoramic image, hazardous targets, hazardous bearings, and hazardous levels on the same screen, and accordingly perform actions such as deceleration, turning, waiting, or adjusting berthing.

[0029] During low-visibility navigation periods in spring, nighttime berthing periods in summer, and regular port operations, the system continuously retains ship navigation perception datasets, radar target perception results, visual target perception results, ship surround panoramic images, multimodal fusion target results, and bridge handling records. Reviewing and verifying this data reveals that the long-range warning area can provide target tracks before visual target classification is difficult, the close-range identification area can supplement target areas and categories, and the berthing monitoring area can stably present the distribution of targets around the ship. Therefore, this invention can improve the continuity of target perception around the ship, the accuracy of fusion judgment, and the stability of obstacle avoidance results in complex scenarios such as nighttime, heavy fog, port entry, and berthing.

[0030] Table 1. Comparison of Overall Performance of Intelligent Obstacle Avoidance Methods for Ship Navigation

[0031] In terms of the effective detection rate of long-range targets, the method of this invention achieves 91.8%, higher than the 87.6% of single-millimeter-wave radar and fixed video split-screen obstacle avoidance methods. This is because this invention does not simply display radar targets through a single path, but rather incorporates long-range radar data into the ship's navigation perception dataset and divides the long-range warning area based on the ship's circumferential layered coverage relationship. This allows long-range targets to be identified by radar based on distance, bearing, speed, and trajectory even before visual images can stably determine their category. This processing method makes long-range perception more targeted and reduces the impact of long-range target limitations imposed by ordinary video line-of-sight.

[0032] In terms of accuracy for near-range target category recognition, the method of this invention achieves 85.9%, while the single millimeter-wave radar and fixed video split-screen obstacle avoidance method achieves 80.7%. In traditional methods, video footage is typically displayed independently, requiring drivers to manually determine whether the video target matches the radar location. Target category recognition is significantly affected by lighting, fog, viewing angle, and occlusion. This invention processes visual image data within both the near-range recognition area and the parking monitoring area, matching it with radar target perception results. This allows the target area and category to corroborate with the target distance and orientation, resulting in more stable target category recognition.

[0033] In terms of radar visual target matching accuracy, the method of this invention achieves 84.2%, while the traditional method achieves 78.4%. This improvement mainly stems from the ship's circumferential sensing coordinates and the ship's circumferential layered coverage relationship. In the traditional method, the radar interface and the video split-screen interface are usually independent of each other, and the position of the same target in different interfaces requires manual matching, which is prone to mismatch or duplicate judgments. This invention unifies the installation position, horizontal pointing, radar detection range, and visual imaging range of the sensing nodes into the ship's circumferential sensing coordinates, and then matches the radar target perception results with the visual target perception results, thereby improving the fusion accuracy of targets around the same ship.

[0034] The average error in target bearing display is 3.1°, lower than the 4.2° of the traditional method. Traditional fixed video and radar display methods often struggle to adequately compensate for target display offsets caused by ship pitch and roll, especially during turning, swell, and low-speed maneuvering at berth, where the target's displayed position in the video frame easily deviates from its actual bearing. This invention generates a pointing compensation amount based on changes in ship attitude and corrects the target bearing in the multimodal fusion target result and the target display position in the ship's panoramic view image, making the compensated target display closer to the actual circumferential bearing of the ship.

[0035] Regarding the near-field target false negative rate during berthing, the method of this invention achieves 8.3%, lower than the 11.6% of the traditional method. Single-millimeter-wave radar and fixed video split-screen obstacle avoidance methods are easily affected by hull obstruction, camera field of view edges, and near-field blind spots during berthing, resulting in insufficient continuous perception capability for small targets on the ship's side, stern, and near the berth. This invention organizes near-range millimeter-wave radar data and visual image data through berthing modes and berthing monitoring areas, enabling targets near the ship's perimeter to be included in the zoned perception state, thus reducing the risk of near-field false negatives during berthing.

[0036] In terms of collision risk assessment accuracy and false alarm rate, the method of this invention achieves 88.1% and 10.2% respectively, while the single millimeter-wave radar and fixed video split-screen obstacle avoidance methods achieve 83.5% and 13.7% respectively. The advantage of this invention lies in its risk assessment, which does not solely rely on radar distance or video observation, but rather combines multimodal fusion target results, compensated target orientation, target distribution in the ship's panoramic view, and ship navigation status for a comprehensive judgment. Therefore, for targets truly located in the ship's navigation direction, to its side, or in its berthing movement direction, this invention can more stably identify dangerous targets, dangerous orientations, and dangerous levels, while reducing false alarms for ordinary surrounding targets. Overall, the performance improvement of this invention mainly comes from three aspects: zoned perception, radar visual fusion, and attitude compensation.

[0037] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A smart obstacle avoidance method for ship navigation based on multimodal perception, characterized in that, Includes the following steps: Collect multimodal perception data during ship navigation and perform standardization processing to generate a ship navigation perception dataset; Organize the installation locations, horizontal orientation, pitch working modes, and sensing range of the sensing nodes to generate a multi-node sensing layout for the ship. Based on the multi-node perception layout of the ship, the ship's circumferential perception coordinates are established, and the radar detection range, visual imaging range and pitch working mode are associated to generate the ship's circumferential layered coverage relationship. Based on the circumferential layered coverage relationship of the ship, the long-range early warning area, the close-range identification area and the berthing monitoring area are divided to generate the ship's zoned perception status; Based on the ship's zoned perception state processing, long-range radar data and short-range millimeter-wave radar data are used to identify the distance, bearing, speed and track of targets around the ship and generate radar target perception results. Based on the ship's zone perception state processing visual image data, the target area and target category of the target around the ship are identified, adjacent field-of-view images are stitched together, and visual target perception results and ship's panoramic view images are generated. The radar target perception results and visual target perception results are matched to generate multimodal fusion target results. The pointing compensation of the multimodal fusion target results and the ship's panoramic image is performed according to the ship's attitude change. The collision risk is judged by combining the target distribution in the ship's panoramic image, and the intelligent obstacle avoidance result of ship navigation is generated.

2. The intelligent obstacle avoidance method for ship navigation based on multimodal perception according to claim 1, characterized in that, The multimodal sensing data includes long-range radar data, short-range millimeter-wave radar data, visual image data, ship attitude data, sensing node layout information, pitch working mode data, radar visual calibration parameters, and ship navigation status data. The standardization processing includes data acquisition time synchronization, unified sensing node identification, unified data format, unified ship coordinate reference, unified radar data units, visual image frame rate adjustment, pitch working mode marking, attitude data alignment, abnormal data removal, and missing data completion.

3. The intelligent obstacle avoidance method for ship navigation based on multimodal perception according to claim 1, characterized in that, The generation of the ship's multi-node sensing layout includes: Based on the ship navigation perception dataset, the installation positions of each perception node at the bow, midship and stern are determined, and the installation positions are sorted according to the forward and backward direction and the left and right direction of the hull. The horizontal orientation of each sensing node is determined based on its installation location, and then the horizontal orientation is associated with its installation location. Mark the pitch working mode of each sensing node and associate the pitch working mode with the horizontal direction of each sensing node; The sensing range is organized according to the radar detection range and visual imaging range of each sensing node, and the sensing range is combined with the installation position, horizontal pointing and elevation working mode. Based on the combined installation location, horizontal orientation, pitch working mode, and sensing range, a multi-node sensing layout for the ship is generated.

4. The intelligent obstacle avoidance method for ship navigation based on multimodal perception according to claim 1, characterized in that, The generation of the ship's circumferential layered coverage relationship includes: Using the bow direction of the ship as the starting direction for angle calculation and the centerline of the ship as the orientation reference, the circumferential sensing coordinates of the ship are established based on the installation position and horizontal direction of each sensing node in the multi-node sensing layout of the ship. The installation location of each sensing node is written into the ship's circumferential sensing coordinates, and the coverage orientation of each sensing node in the ship's circumferential sensing coordinates is determined according to the horizontal direction. According to the elevation working mode of each sensing node, the radar detection range and visual imaging range are respectively associated with the coverage azimuth, generating the layered coverage content of each sensing node. The layered coverage content of all sensing nodes is combined according to the arrangement order of bow, midship and stern to generate the circumferential layered coverage relationship of the ship.

5. The intelligent obstacle avoidance method for ship navigation based on multimodal perception according to claim 1, characterized in that, The generation of the ship's zone sensing status includes: Based on the radar detection range in the circumferential layered coverage relationship of the ship, the area covered by long-range radar data and where the visual imaging range does not form a target category judgment is determined, and a long-range early warning area is generated. Based on the overlapping part of the radar detection range and visual imaging range in the circumferential layered coverage relationship of the ship, the area that can simultaneously perform target distance identification, target orientation identification and target category judgment is determined, and a close-range identification area is generated. Based on the pitch working mode in the circumferential layered coverage relationship of the ship, the radar detection range and visual imaging range near the hull in the berthing mode are determined, and the berthing monitoring area is generated. The long-range early warning area, the close-range identification area, and the berthing monitoring area are combined according to the ship's circumferential perception coordinates to generate the ship's zone perception status.

6. The intelligent obstacle avoidance method for ship navigation based on multimodal perception according to claim 1, characterized in that, The generation of the radar target perception results includes: Determine the location of the long-range early warning area, the close-range identification area, and the berthing monitoring area in the ship's circumferential sensing coordinates according to the ship's zone sensing status. Match long-range radar data to the long-range early warning area, identify the distance, bearing and speed of targets around the ship in the long-range early warning area, and generate long-range radar identification content. The close-range millimeter-wave radar data is matched to the close-range identification area and the berthing monitoring area. The distance, bearing and speed of targets around the ship in the close-range identification area and the berthing monitoring area are identified to generate close-range radar identification content. The data collected from long-range and short-range radars are continuously processed according to the time of collection to form the tracks of targets around the ship. The distance, bearing, speed, and trajectory of targets around the ship are combined to generate radar target perception results.

7. The intelligent obstacle avoidance method for ship navigation based on multimodal perception according to claim 1, characterized in that, The generation of the visual target perception results and the ship's panoramic surround view image includes: Determine the location of the near-field identification area and the berthing monitoring area in the ship's circumferential sensing coordinates according to the ship's zone sensing status. The visual image data is matched to the near-field recognition area and the berthing monitoring area according to the coverage direction of each sensing node. The matched visual image data is then corrected and the image clarity is improved to generate regional visual images. Contour extraction and location marking of targets around ships in regional visual images are performed to generate target regions; The target area is classified, a target category is generated, and the target area, target category and acquisition time are combined to generate a visual target perception result; Based on the coverage orientation of each sensing node, adjacent field-of-view images are extracted. The overlapping areas of the adjacent field-of-view images are registered, the brightness is adjusted, and the boundaries are fused to generate a panoramic image of the ship.

8. The intelligent obstacle avoidance method for ship navigation based on multimodal perception according to claim 1, characterized in that, The generation of the intelligent obstacle avoidance results for ship navigation includes: The radar target perception results and visual target perception results are matched according to the collection time and the ship's circumferential perception coordinates. The distance, bearing, speed, track, target area and target category of the same target around the ship are combined to generate multimodal fusion target results. Based on the changes in the ship's attitude, the pitch and roll changes are processed to generate a pointing compensation amount; The target orientation in the multimodal fusion target result and the target display position in the ship's panoramic image are corrected by using the pointing compensation amount, and the compensated multimodal fusion target result and the compensated ship's panoramic image are generated. Collision risk assessment is performed based on the compensated multimodal fusion target results and the target distribution in the compensated ship surround panoramic image, identifying dangerous targets, dangerous locations and dangerous levels, and generating intelligent obstacle avoidance results for ship navigation.

Images