Ship information identification method and system based on multi-modal data fusion

Abstract

The invention relates to a ship information identification method and system based on multi-modal data fusion, belongs to the field of ship traffic, and solves the problem of low accuracy of single-modal data in the prior art. The method comprises the following specific steps: inputting video data obtained by a ship-mounted camera into a trained ship plane detection model to obtain a 2D image frame of a ship contained in the video data; inputting point cloud data obtained by a ship carrying laser radar into the trained ship 3D point cloud detection model to obtain a ship 3D detection frame included in the point cloud data; acquiring AIS information of the ship and other ships in a world coordinate system by using a ship-borne AIS; matching and fusing every two ship information of the 2D image frame and the AIS information, the 3D detection frame and the AIS information, and the 3D detection frame and the 2D image frame; and the matching and fusion results are integrated to obtain a final ship identification result, so that more comprehensive and accurate ship identification is realized.

Description

Technical Field

[0001] The present invention relates to the field of ship traffic, and in particular to a ship information recognition method and system based on multimodal data fusion. Background Art

[0002] Target detection in the marine environment (such as ships, buoys, channel markers, etc.) is crucial for navigation safety, maritime rescue and environmental monitoring. However, the marine environment has unique challenges, including: Light changes: The difference in light between day and night is huge, and targets at sea will also change significantly under different weather conditions (such as fog, rain, strong light, etc.); Complex background: Sea surface fluctuations, sunlight reflections, cloud changes, etc. may make it difficult to detect visual targets; Occlusion and interference: Occlusion between ships, splashing of waves, etc. may affect the recognition of targets.

[0003] Image target detection technology is widely used in the maritime field in scenarios such as monitoring, navigation assistance and autonomous ships. The image data obtained by the camera can detect and identify ships, buoys and other maritime targets. Image target detection is easily affected by factors such as lighting, weather conditions and occlusion, resulting in reduced detection accuracy, and it provides two-dimensional information, and cannot directly obtain the three-dimensional position and distance of the target; AIS is a system for ship identification and positioning. It transmits VHF (very high frequency) radio signals to provide dynamic information of ships, such as position, speed, heading and identification code. It plays a vital role in shipping safety, traffic management and collision avoidance systems, but AIS signals may be interfered with, shielded or maliciously tampered with, resulting in data loss or inaccuracy, and AIS information may not be effectively transmitted in the open sea or in places with poor radio signals. AIS information is mainly point data and cannot provide detailed information such as the appearance and size of the ship; LiDAR uses laser pulses to measure the distance and three-dimensional shape of ships and other maritime targets. LiDAR can provide stable data under various lighting conditions, so it has obvious advantages in maritime applications, especially in autonomous ships and navigation systems. However, the ship information identification method based on the above technology has limitations and shortcomings, especially in harsh marine environments, complex traffic conditions or signal loss, it is unable to provide accurate and comprehensive ship information. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a ship information identification method and system based on multimodal data fusion, so as to solve the problem of low accuracy of single modal data in the prior art.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] On the one hand, an embodiment of the present invention provides a method for identifying ship information based on multimodal data fusion, comprising the following steps:

[0007] Input the video data obtained by the camera on board the ship into the trained ship plane detection model to obtain a 2D image frame of the ship contained in the video data;

[0008] Input the point cloud data obtained by the laser radar on board the ship into the trained ship 3D point cloud detection model to obtain the 3D detection frame of the ship contained in the point cloud data;

[0009] Use shipborne AIS to obtain AIS information of own ship and other ships in the world coordinate system;

[0010] Matching and fusing the ship information of each pair between the 2D image frame and the AIS information, between the 3D detection frame and the AIS information, and between the 3D detection frame and the 2D image frame;

[0011] The matching and fusion results are integrated to obtain the final ship recognition result.

[0012] Furthermore, the specific steps of matching and fusing the 2D image frame with the AIS information for ship information include:

[0013] Based on the 2D image frame, a relative azimuth angle from the corresponding ship to the camera coordinate system of the own ship is calculated;

[0014] Based on the AIS information of the own ship and the other ship, the relative azimuth of the other ship to the own ship in the world coordinate system is calculated, and the relative azimuth of the other ship to the own ship in the camera coordinate system is calculated by combining the mapping relationship between the relative azimuth in the camera coordinate system and the relative azimuth in the world coordinate system;

[0015] Determine whether the error between the two relative azimuths is less than the threshold value; if yes, the match is successful; if no, the match fails;

[0016] The successfully matched 2D image frame is merged and fused with the AIS information.

[0017] Further, the mapping relationship is determined based on the following steps:

[0018] Selecting several groups of physical points as reference points in the image captured by the camera on board the ship, and measuring the relative azimuth angles of the several groups of reference points and the field of view of the camera on board the ship in the camera coordinate system;

[0019] Determine several groups of reference points at the same position in the world coordinate system map, and obtain the relative azimuth angles between the corresponding points in the world coordinate system and the camera carried by the ship;

[0020] The mapping relationship is determined by fitting based on the relative azimuths from several groups of reference points to the camera coordinate system of the ship and the relative azimuths in the world coordinate system.

[0021] Furthermore, the specific steps of matching and fusing the 3D detection frame with the AIS information include:

[0022] Based on the 3D detection frame, the ship distance and relative azimuth in the world coordinate system from the corresponding ship to the own ship are calculated;

[0023] Determine whether the error between the ship distance in the world coordinate system and the ship distance in the AIS information of the other ship is less than a threshold value, if yes, the distance matching is successful, otherwise, the distance matching fails;

[0024] Matching the relative azimuth calculated based on the AIS information of the own ship and the other ship with the relative azimuth in the world coordinate system, and judging whether the error between the two relative azimuths is less than the threshold value; if yes, the relative azimuth matching is successful; if no, the relative azimuth matching fails;

[0025] When the ship distance and relative azimuth in the world coordinate system are successfully matched, the 3D detection frame is successfully matched with the AIS information; otherwise, the matching fails;

[0026] The successfully matched 3D detection frame and the AIS information are merged and fused.

[0027] Furthermore, the specific steps of calculating the ship distance and relative azimuth from the corresponding ship to the ship in the world coordinate system include:

[0028] Performing algorithm analysis on the point cloud data obtained by the laser radar carried by the ship;

[0029] The parsed point cloud data is input into the trained ship 3D point cloud detection model to obtain the estimation result of the ship's 3D detection frame;

[0030] Calculating the coordinates of the center of mass of the corresponding ship according to the estimation result of the 3D detection frame;

[0031] Based on the coordinates of the center of mass point and the position of the laser radar carried by the ship, the ship distance and relative azimuth in the world coordinate system of the corresponding ship to the ship are obtained.

[0032] Furthermore, the specific steps of matching and fusing the 3D detection frame and the 2D image frame include:

[0033] Mapping the 3D detection frame into a 2D detection frame of the ship;

[0034] Matching the mapped 2D detection frame with the 2D image frame for ship information;

[0035] The information of the successfully matched 3D detection frame and the 2D image frame is merged and fused.

[0036] Further, the ship plane detection model is trained based on the YOLOv8 network model, and the steps include:

[0037] Use the cameras on board ships to collect video data of various ships sailing in offshore ports;

[0038] Parsing the various types of ship video data into ship images and cleaning them;

[0039] Performing data annotation on the ship image on a visual data annotation platform; the annotation is a 2D outer frame of the ship;

[0040] The annotated ship images are organized into a ship image dataset, and the dataset is divided into a training set, a validation set, and a test set;

[0041] Based on the ship image dataset and the YOLOv8 network model, a ship plane detection model is trained to obtain an estimation result of the 2D image frame of the ship.

[0042] Furthermore, based on the ship image dataset and the YOLOv8 network model, a ship plane detection model is trained, including:

[0043] The YOLOv8 network model is improved, the BiFormer module is introduced into the original convolutional network of the YOLOv8 backbone network, and an efficient pyramid network architecture is constructed. Then, a ship plane detection model is obtained by training based on the ship image dataset.

[0044] Furthermore, the specific steps of training the ship 3D point cloud detection model based on CenterPoint include:

[0045] Use the laser radar on board ships to collect point cloud data of various ships sailing in offshore ports;

[0046] Parsing the ship point cloud data into a pcd file and cleaning it;

[0047] Carry out 3D data annotation of the ship point cloud on the point cloud data annotation platform;

[0048] The annotated ship point cloud data are organized into a ship point cloud dataset, and the dataset is divided into a training set, a validation set, and a test set;

[0049] Based on the ship point cloud dataset and the CenterPoint model, ship 3D point cloud detection is trained to obtain an estimation result of the ship 3D detection frame.

[0050] On the other hand, an embodiment of the present invention provides a ship information identification system based on multimodal data fusion, comprising:

[0051] Ships are equipped with cameras to collect video data of various ships sailing in offshore ports;

[0052] The ship is equipped with LiDAR to collect point cloud data of various ships sailing in offshore ports;

[0053] The ship detection module is used to train a ship plane detection model and a ship 3D point cloud detection model respectively using various data collected during navigation in the sea port; and based on the trained model and the video data and point cloud data collected in real time, obtain a 2D image frame and a 3D detection frame of the ship contained in the video data and the point cloud data respectively;

[0054] Shipborne AIS, used to obtain AIS information of own ship and other ships;

[0055] An information fusion module is used to match and fuse the ship information of the ship 2D image frame and the AIS information, the ship 3D detection frame and the AIS information, and the 3D detection frame and the 2D image frame respectively; and integrate the matching and fusion results to obtain a final ship recognition result;

[0056] The display module is used to display the final ship identification result.

[0057] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0058] 1. The present invention proposes to extract effective information from each mode through the information complementarity of multimodal data, and combine the ship identity and position data provided by AIS with the visual information of the image and the three-dimensional information of the lidar, so as to achieve more comprehensive and accurate ship identification, especially when a single mode cannot provide sufficient information, to enhance the target identification and tracking capabilities;

[0059] 2. By integrating data sources from different modes, it can maintain a high level of recognition performance in the case of bad weather, complex sea conditions or signal interruption, reduce recognition errors caused by single data failure, have strong environmental adaptability, and enhance the robustness of the system;

[0060] 3. Through advanced algorithms and data processing technologies, multi-modal data can be quickly integrated and analyzed to ensure that the system can identify ship information in a short time, meet the needs of real-time monitoring and emergency response, and realize real-time monitoring and response;

[0061] 4. The overall performance of the multimodal data fusion ship information identification system provided by the embodiment of the present invention is significantly improved; at the same time, it is adaptable to various application scenarios, not only suitable for maritime traffic management, but also can be applied to maritime rescue, military monitoring, marine environment monitoring and other scenarios; it provides a universal and efficient ship information identification solution.

[0062] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0064] Figure 1 It is a flow chart of a method for ship information identification by multimodal data fusion according to an embodiment of the present invention;

[0065] Figure 2 Schematic diagram of a system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0067] Example 1

[0068] A specific embodiment of the present invention discloses a ship information recognition method based on multimodal data fusion, such as Figure 1 As shown, the following steps are included:

[0069] Step S1, inputting the video data obtained by the camera on board the ship into the trained ship plane detection model to obtain the 2D image frame of the ship contained in the video data;

[0070] Step S2: input the point cloud data obtained by the laser radar carried by the ship into the trained ship 3D point cloud detection model to obtain the 3D detection frame of the ship contained in the point cloud data;

[0071] Step S3, using the shipborne AIS to obtain the AIS information of the own ship and other ships in the world coordinate system;

[0072] Step S4, matching and fusing the ship information between the 2D image frame and the AIS information, between the 3D detection frame and the AIS information, and between the 3D detection frame and the 2D image frame;

[0073] Step S5: Integrate the matching and fusion results to obtain a final ship recognition result.

[0074] Through the above method, data is acquired using shipborne cameras and lidar, and ship 2D and 3D detection frame data are obtained based on the ship detection model. The fused ship information is obtained by matching and fusing image data, AIS information and lidar data, which can significantly improve the accuracy and robustness of marine target detection.

[0075] Exemplarily, in step S1, a camera mounted on a ship is used to capture images of the surrounding environment; the captured images are input into a trained ship plane detection model through an rtsp video stream for ship recognition, and a 2D image frame containing pixel coordinates of the four corner points of the ship is obtained.

[0076] Furthermore, the 2D image frame can be combined with tracking technology to obtain a real-time ship tracking ID, including using DeepSORT technology to achieve multi-target ship tracking. The ship tracking ID information can also be used as fused ship identification information.

[0077] Exemplarily, based on the YOLOv8 deep learning network model, a ship plane detection model is trained, and the specific steps include:

[0078] Step S11, using a ship-mounted camera to collect video data of various ships sailing in the sea port;

[0079] Step S12, parsing various types of ship video data into images and cleaning them;

[0080] Step S13, annotating the ship image on the visual data annotation platform; the annotation is 2D outer frame data of the ship;

[0081] Step S14: organize the annotated data into a ship image dataset, and divide the dataset into a training set, a validation set, and a test set; wherein the validation set is used to evaluate the performance of the model on unseen data and perform parameter tuning; and the test set is used to test the performance of the final model;

[0082] Step S15: Based on the ship image dataset and the YOLOv8 network model, a ship plane detection model is trained to obtain an estimation result of a 2D image frame of the ship.

[0083] Furthermore, the YOLOv8 network model is improved by introducing the BiFormer module into the original convolutional network of the Backbone part of YOLOv8. After constructing an efficient pyramid network architecture based on dynamic sparse attention, a ship plane detection model is obtained based on the ship image dataset training, so that the network can more effectively process small-scale targets, dense targets and occluded targets.

[0084] Specifically, in step S2, the point cloud data obtained by the laser radar carried by the ship is analyzed by algorithm; the analyzed point cloud data is input into the trained ship 3D point cloud detection model for ship identification, and the 3D detection boundary box (referred to as 3D detection box) of the three-dimensional laser point box of the other ship is obtained.

[0085] Exemplarily, the point cloud data is processed in the Robot Operating System (ROS) based on the rosbag package.

[0086] By adding lidar sensors, high reliability can be maintained in different environments. In scenes such as insufficient light, rainy days, and foggy days, lidar sensors can supplement the deficiencies of camera sensors.

[0087] Exemplarily, training the ship 3D point cloud detection model based on the CenterPoint deep learning network includes the following steps:

[0088] Step S21, collecting point cloud data of various types of ships sailing in sea ports;

[0089] Step S22, parsing the ship point cloud data into a pcd file and cleaning it;

[0090] Step S23, performing 3D data annotation on the ship point cloud on the point cloud data annotation platform;

[0091] Step S24, organizing the annotated ship point cloud data into a ship point cloud data set, and dividing the data set into a training set, a validation set, and a test set;

[0092] Step S25: Based on the ship point cloud dataset and the CenterPoint model, a ship 3D point cloud detection model is trained to obtain a ship 3D detection frame estimation result.

[0093] Specifically, in step S3, the shipborne AIS obtains the AIS information of the ship and other ships, including the ship identification code (mmsi), longitude (Lon), latitude (Lat), ship heading (track to the ground cog), bow heading (heading), true bearing of other ships (BRG) and straight-line distance of other ships relative to the ship (RNG) and other information.

[0094] Using AIS information, the system can still identify the target through its identity characteristics even when the image or lidar data is blurred or unclear. For example, ship information transmitted through AIS can help the system accurately identify and track the ship even when the image data is unclear.

[0095] Specifically, in step S4, the specific steps of matching and fusing the 2D image frame with the AIS information include:

[0096] Step S411, based on the 2D image frame, calculating the relative azimuth angle from the corresponding ship to the camera coordinate system of the own ship;

[0097] Step S412: Based on the AIS information of the own ship and the other ship, the relative azimuth of the other ship to the own ship in the world coordinate system is calculated, and the relative azimuth of the other ship to the own ship in the camera coordinate system is calculated by combining the mapping relationship between the relative azimuth in the camera coordinate system and the relative azimuth in the world coordinate system; wherein the specific steps of determining the mapping relationship include:

[0098] Selecting several groups of physical points as reference points in the image captured by the ship's onboard camera, and measuring the relative azimuths of the several groups of reference points and the field of view of the ship's onboard camera in the camera coordinate system;

[0099] Determine several groups of reference points at the same position in the world coordinate system map, and obtain the relative azimuth angles between the corresponding points in the world coordinate system and the camera points on board the ship;

[0100] The mapping relationship is determined by fitting based on the relative azimuths from several groups of reference points to the ship's camera coordinate system and the relative azimuths in the world coordinate system.

[0101] Step S413, determine whether the relative azimuth information error obtained by the two methods is less than the threshold value; if yes, the match is successful, otherwise, the match fails; wherein the threshold value is adjusted according to the specific application needs, such as setting the error threshold value to 10 degrees, and determining that the match is successful when the error is less than 10 degrees.

[0102] Step S414: merge and fuse the successfully matched 2D image frames with the AIS information, and retain the unmatched 2D image frames.

[0103] Specifically, the specific steps of matching and fusing the 3D detection frame with the AIS information include:

[0104] Step S421, based on the 3D detection frame, calculate the ship distance and relative azimuth information from the corresponding ship to the own ship in the world coordinate system, the specific steps include:

[0105] According to the estimation result of the ship's 3D detection frame, the coordinates of the center of mass of the other ship corresponding to the 3D detection frame are calculated;

[0106] Based on the coordinates of the centroid point and the position of the ship's onboard laser radar, the ship distance and relative azimuth information in the world coordinate system of the corresponding ship and the ship are obtained.

[0107] Step S422, judging whether the error between the ship distance in the world coordinate system and the ship distance in the AIS information of other ships is less than a threshold value; if yes, the distance matching is successful, otherwise, the distance matching fails;

[0108] Step S423, matching the relative azimuth calculated based on the AIS information of the own ship and the other ship with the relative azimuth in the world coordinate system, and judging whether the error between the two relative azimuths is less than a threshold value; if yes, the relative azimuth matching is successful; if no, the relative azimuth matching fails;

[0109] Step S424: When the ship distance and the relative azimuth in the world coordinate system are successfully matched, the 3D detection frame is successfully matched with the AIS information; otherwise, the matching fails;

[0110] Step S425: merge and fuse the successfully matched 3D detection frame with the AIS information; and retain the unmatched 3D detection frame.

[0111] It should be noted that, under normal circumstances, each ship will transmit AIS information. Among the AIS information of multiple other ships received by the ship, the identity of the ship can only be determined by matching the 2D image frame or 3D detection frame. Therefore, when the matching fails, only the 2D image frame or 3D detection frame is retained.

[0112] Specifically, the specific steps of matching and fusing the ship information between the 3D detection frame and the 2D image frame include:

[0113] Step S431, mapping the 3D detection frame to a 2D detection frame of the ship;

[0114] Step S432: matching the mapped 2D detection frame with the 2D image frame for ship information based on whether the area intersection over union (IOU) is within a certain threshold;

[0115] Step S433: merge and fuse the information of the successfully matched 3D detection frame and 2D image frame.

[0116] Specifically, in step S5, the three fusion results are integrated through strategy processing to obtain the final ship recognition result, including the 2D image detection frame detected by all sensors, AIS information, 3D detection frame, and ship distance calculated by the centroid point of the 3D detection frame and other information.

[0117] It should be noted that ship-mounted cameras, ship-mounted laser radars and ship-borne AIS constitute multiple sensors, which will collect ship data of different modes. During each matching and fusion process, each sensor may contain information of multiple ships. The ship identity is identified through the matching process, and the ship data of different modes of each ship are matched, and then the multi-modal data of the same ship is fused.

[0118] Exemplarily, the strategy processing includes each time the match is successful, the merged information is used as the fusion result of this time; when the match fails, the corresponding information is retained as the fusion result of this time. Among them, when the ship data of the two modes are unsuccessfully matched, including when the modal information corresponds to different ships, the matching fusion process will not end, waiting for the matching process of the next ship until all the ship information is matched; it also includes when a certain modal data is lost, retaining the ship data under another modality as the fusion result. When both matches are successful, the ship identification result after the fusion of all modalities will be obtained. When the match is unsuccessful, the ship identification information of at least one modality will be obtained.

[0119] Compared with the prior art, the present embodiment provides a ship information identification method based on multimodal data fusion, which integrates data from different sensors, such as radar, optical images, infrared images, AIS (automatic identification system) data, etc., and performs matching fusion processing, and finally realizes the fusion of multimodal information, which can significantly improve the accuracy of target detection. This multimodal fusion can reduce the false detection rate and missed detection rate of a single sensor, thereby providing more accurate and rich ship identification information. On the one hand, the method can improve the recognition accuracy and speed of ships by using the improved YOLOv8 as a ship recognition model; on the other hand, the CenterPoint 3D point cloud ship recognition model detection method is used to obtain the three-dimensional laser point frame of the ship, which can be used as a supplement to camera ship recognition, and can realize multi-sensor automatic monitoring of other ship information around the ship.

[0120] Example 2

[0121] Another specific embodiment of the present invention discloses a ship information identification system based on multimodal data fusion, comprising:

[0122] Ships are equipped with cameras to collect video data of various ships sailing in offshore ports;

[0123] The ship is equipped with LiDAR to collect point cloud data of various ships sailing in offshore ports;

[0124] The ship detection module is used to train a ship plane detection model and a ship 3D point cloud detection model respectively using various data collected during navigation in the sea port; and based on the trained model and the video data and point cloud data collected in real time, obtain a 2D image frame and a 3D detection frame of the ship contained in the video data and the point cloud data respectively;

[0125] Shipborne AIS, used to obtain AIS information of own ship and other ships;

[0126] An information fusion module is used to match and fuse the ship information of the ship 2D image frame and the AIS information, the ship 3D detection frame and the AIS information, and the 3D detection frame and the 2D image frame respectively; and integrate the matching and fusion results to obtain a final ship recognition result;

[0127] The display module is used to display the final ship identification result.

[0128] Among them, the system can apply the identification method described in any one of the schemes in Example 1 to identify ship information.

[0129] Compared with the prior art, the present embodiment provides a ship information identification system based on multimodal data fusion, which obtains the image visual information and three-dimensional information of the ship through the ship-mounted camera and the ship-mounted laser radar, obtains the ship detection frame information in combination with the ship detection module, and fuses it with the AIS information through the information fusion module, and finally realizes a highly accurate ship identification system. On the one hand, the fusion of data sources from different modes enhances the robustness of the system; on the other hand, the system adapts to a variety of application scenarios, not only suitable for maritime traffic management, but also can be applied to maritime rescue, military monitoring, marine environment monitoring and other scenarios.

[0130] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0131] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A ship information identification method based on multimodal data fusion, characterized in that: The steps include: Input the video data obtained by the camera on board the ship into the trained ship plane detection model to obtain a 2D image frame of the ship contained in the video data; Input the point cloud data obtained by the laser radar on board the ship into the trained ship 3D point cloud detection model to obtain the 3D detection frame of the ship contained in the point cloud data; Use shipborne AIS to obtain AIS information of own ship and other ships in the world coordinate system; Matching and fusing the ship information of each pair between the 2D image frame and the AIS information, between the 3D detection frame and the AIS information, and between the 3D detection frame and the 2D image frame; The matching and fusion results are integrated to obtain the final ship recognition result.

2. A ship information identification method based on multimodal data fusion according to claim 1, characterized in that: The specific steps of matching and fusing the 2D image frame with the AIS information for ship information include: Based on the 2D image frame, a relative azimuth angle from the corresponding ship to the camera coordinate system of the own ship is calculated; Based on the AIS information of the own ship and the other ship, the relative azimuth of the other ship to the own ship in the world coordinate system is calculated, and the relative azimuth of the other ship to the own ship in the camera coordinate system is calculated by combining the mapping relationship between the relative azimuth in the camera coordinate system and the relative azimuth in the world coordinate system; Determine whether the error between the two relative azimuths is less than the threshold value; if yes, the match is successful; if no, the match fails; The successfully matched 2D image frame is merged and fused with the AIS information.

3. A ship information identification method based on multimodal data fusion according to claim 2, characterized in that: The mapping relationship is determined based on the following steps: Selecting several groups of physical points as reference points in the image captured by the camera on board the ship, and measuring the relative azimuth angles of the several groups of reference points and the field of view of the camera on board the ship in the camera coordinate system; Determine several groups of reference points at the same position in the world coordinate system map, and obtain the relative azimuth angles between the corresponding points in the world coordinate system and the camera carried by the ship; The mapping relationship is determined by fitting based on the relative azimuths from several groups of reference points to the camera coordinate system of the ship and the relative azimuths in the world coordinate system.

4. The ship information identification method based on multimodal data fusion according to claim 1 is characterized in that: The specific steps of matching and fusing the 3D detection frame with the AIS information include: Based on the 3D detection frame, the ship distance and relative azimuth in the world coordinate system from the corresponding ship to the own ship are calculated; Determine whether the error between the ship distance in the world coordinate system and the ship distance in the AIS information of the other ship is less than a threshold value, if yes, the distance matching is successful, otherwise, the distance matching fails; Matching the relative azimuth calculated based on the AIS information of the own ship and the other ship with the relative azimuth in the world coordinate system, and judging whether the error between the two relative azimuths is less than the threshold value; if yes, the relative azimuth matching is successful; if no, the relative azimuth matching fails; When the ship distance and relative azimuth in the world coordinate system are successfully matched, the 3D detection frame is successfully matched with the AIS information; otherwise, the matching fails; The successfully matched 3D detection frame and the AIS information are merged and fused.

5. The ship information identification method based on multimodal data fusion according to claim 4 is characterized in that: The specific steps of calculating the ship distance and relative azimuth in the world coordinate system from the corresponding ship to the own ship include: Performing algorithm analysis on the point cloud data obtained by the laser radar carried by the ship; The parsed point cloud data is input into the trained ship 3D point cloud detection model to obtain the estimation result of the ship's 3D detection frame; Calculating the coordinates of the center of mass of the corresponding ship according to the estimation result of the 3D detection frame; Based on the coordinates of the center of mass point and the position of the laser radar carried by the ship, the ship distance and relative azimuth in the world coordinate system of the corresponding ship to the ship are obtained.

6. A ship information identification method based on multimodal data fusion according to claim 1, characterized in that: The specific steps of matching and fusing the 3D detection frame and the 2D image frame include: Mapping the 3D detection frame into a 2D detection frame of the ship; Matching the mapped 2D detection frame with the 2D image frame for ship information; The information of the successfully matched 3D detection frame and the 2D image frame is merged and fused.

7. The ship information identification method based on multimodal data fusion according to claim 1 is characterized in that: The ship plane detection model is trained based on the YOLOv8 network model, and the steps include: Use the cameras on board ships to collect video data of various ships sailing in offshore ports; Parsing the various types of ship video data into ship images and cleaning them; Performing data annotation on the ship image on a visual data annotation platform; the annotation is a 2D outer frame of the ship; The annotated ship images are organized into a ship image dataset, and the dataset is divided into a training set, a validation set, and a test set; Based on the ship image dataset and the YOLOv8 network model, a ship plane detection model is trained to obtain an estimation result of the 2D image frame of the ship.

8. The ship information identification method based on multimodal data fusion according to claim 7 is characterized in that: Based on the ship image dataset and the YOLOv8 network model, a ship plane detection model is trained, including: The YOLOv8 network model is improved, the BiFormer module is introduced into the original convolutional network of the YOLOv8 backbone network, and an efficient pyramid network architecture is constructed. Then, a ship plane detection model is obtained by training based on the ship image dataset.

9. A ship information identification method based on multimodal data fusion according to any one of claims 1 to 8, characterized in that: The specific steps of training the ship 3D point cloud detection model based on CenterPoint include: Use the laser radar on board ships to collect point cloud data of various ships sailing in offshore ports; Parsing the ship point cloud data into a pcd file and cleaning it; Carry out 3D data annotation of the ship point cloud on the point cloud data annotation platform; The annotated ship point cloud data are organized into a ship point cloud dataset, and the dataset is divided into a training set, a validation set, and a test set; Based on the ship point cloud dataset and the CenterPoint model, ship 3D point cloud detection is trained to obtain an estimation result of the ship 3D detection frame.

10. A ship information identification system based on multimodal data fusion, characterized in that: include: Ships are equipped with cameras to collect video data of various ships sailing in offshore ports; The ship is equipped with LiDAR to collect point cloud data of various ships sailing in offshore ports; The ship detection module is used to train a ship plane detection model and a ship 3D point cloud detection model respectively using various data collected during navigation in the sea port; and based on the trained model and the video data and point cloud data collected in real time, obtain a 2D image frame and a 3D detection frame of the ship contained in the video data and the point cloud data respectively; Shipborne AIS, used to obtain AIS information of own ship and other ships; An information fusion module is used to match and fuse the ship information of the ship 2D image frame and the AIS information, the ship 3D detection frame and the AIS information, and the 3D detection frame and the 2D image frame respectively; and integrate the matching and fusion results to obtain a final ship recognition result; The display module is used to display the final ship identification result.

Images