A ship monitoring method and system fusing unmanned aerial vehicle data

Abstract

The application discloses a ship monitoring method and system fusing unmanned aerial vehicle data. The method comprises the following steps: S1, acquiring AIS data of all ships in a current section of a channel, and determining a ship AIS track set according to a ship code in the AIS data; S2, acquiring three-dimensional point cloud images of all ships in the current section of the channel by using a laser radar, and determining a ship laser radar track set according to the three-dimensional point cloud images; S3, acquiring two-dimensional images of all ships in the current section of the channel by using an unmanned aerial vehicle, and determining a ship unmanned aerial vehicle track set according to the two-dimensional images; and S4, performing multi-source matching on the ship laser radar track set, the ship unmanned aerial vehicle track set and the ship AIS track set in combination with a reference center line, and if the matching is successful, performing subsequent tracking and illegal detection on the ships by using the unmanned aerial vehicle. The method simultaneously integrates three types of data, namely, AIS data, fixed laser radar data and unmanned aerial vehicle image data, sufficiently makes up for defects of single-source data, and improves the accuracy of ship monitoring.

Description

Technical Field

[0001] This invention relates to the field of ship monitoring technology, and more specifically, to a ship monitoring method and system that integrates unmanned aerial vehicle (UAV) data. Background Technology

[0002] Currently, waterway vessel monitoring mainly relies on fixed monitoring points (such as lidar and cameras) and Automatic Identification Systems (AIS). Fixed monitoring points are typically deployed at intervals of several kilometers, resulting in blind spots. Furthermore, the lateral perspective makes it difficult to accurately match vessel and AIS information when multiple vessels are converging. AIS data updates infrequently, has limited accuracy, and there are instances where AIS is not activated or the signal is lost, leading to monitoring loopholes.

[0003] Due to their high maneuverability and wide field of view, unmanned aerial vehicles (UAVs) are increasingly being used for waterway inspection. However, the positioning accuracy of single UAV image recognition is limited by factors such as weather, lighting, and vibration, making it difficult to directly match with AIS or radar data. Existing technologies lack effective multi-source data fusion mechanisms, hindering collaborative operations between UAVs and fixed monitoring systems.

[0004] Therefore, there is an urgent need for an intelligent ship monitoring method and system that can integrate AIS, LiDAR, and UAV image data to improve the accuracy, real-time performance, and automation level of ship supervision. Summary of the Invention

[0005] This invention provides a ship monitoring method and system that integrates unmanned aerial vehicle (UAV) data to address the problem of low accuracy in ship monitoring in the prior art.

[0006] To achieve the above objectives, the present invention provides a ship monitoring method integrating unmanned aerial vehicle (UAV) data. The method includes: S1, acquiring AIS data of all ships within the current channel segment, and determining a set of ship AIS trajectories based on the ship codes in the AIS data; S2, acquiring three-dimensional point cloud images of all ships within the current channel segment using lidar, and determining a set of ship lidar trajectories based on the three-dimensional point cloud images; S3, acquiring two-dimensional images of all ships within the current channel segment using UAVs, and determining a set of ship UAV trajectories based on the two-dimensional images; S4, performing multi-source matching by combining the ship lidar trajectories, ship UAV trajectories, and ship AIS trajectories with a reference centerline. If the matching is successful, the UAV is used for subsequent tracking and violation detection of the ship.

[0007] Optionally, before step S1, the process includes: dividing the current waterway into multiple segments; establishing an electronic fence for each segment; calculating the corresponding midpoint and forming an original centerline based on the latitude and longitude sampling points of the two side boundaries of each segment; and thinning the original centerline using the DP algorithm to obtain a reference centerline.

[0008] Optionally, S3 includes: performing target detection on each frame of the two-dimensional image to obtain the first bounding box of all ships and the ship type; extracting feature points in the first bounding box of the current ship in two adjacent frames using a sparse optical flow algorithm, and calculating the motion speed of the current ship in the two adjacent frames based on the extracted feature points; predicting the first bounding box of the current ship in the next frame in two adjacent frames based on the first bounding box of the current ship in the previous frame and the motion speed of the current ship in the two adjacent frames; updating the predicted first bounding box of the current ship in the next frame in two adjacent frames according to the motion direction of the current ship; obtaining the bow latitude and longitude, stern latitude and longitude, length and width of the current ship based on the position of the updated first bounding box of the current ship relative to the edge of the current channel segment and the latitude and longitude conversion of the UAV; and collecting the updated first bounding boxes of all ships in all frames to obtain the ship UAV trajectory set.

[0009] Optionally, S4 includes: S41, matching the current ship lidar trajectory in the ship lidar trajectory set with the drone trajectory of each ship in the ship drone trajectory set; if the match is successful, proceed to S42; S42, matching the current ship lidar trajectory in the ship lidar trajectory set with the AIS trajectory of each ship in the ship AIS trajectory set; if the match is successful, proceed to S43; S43, determining whether the current ship type in the successfully matched current ship AIS trajectory is consistent with the current ship type in the successfully matched current ship drone trajectory; if they are consistent, the match is successful; otherwise, proceed to S44; S44, matching the successfully matched current ship AIS trajectory with the successfully matched current ship drone trajectory; if the match is successful, using the drone to perform subsequent tracking and violation detection on the current ship.

[0010] Optionally, after S4, the method further includes: if the current ship UAV trajectory has a matching result in the ship lidar trajectory set, then use the latitude and longitude of the current ship in the lidar trajectory as the current ship's latitude and longitude; otherwise, use the latitude and longitude of the current ship in the UAV trajectory as the current ship's latitude and longitude; calibrate the current ship's latitude and longitude using the latitude and longitude in the AIS data so that the UAV can subsequently track the current ship based on the calibrated latitude and longitude.

[0011] Optionally, S41 includes: S411, selecting the current frame from all images captured by the lidar, performing target bounding box selection on the 3D point cloud image of the current frame to obtain the second bounding box of all ships; simultaneously, selecting a frame from all images captured by the UAV that matches the lidar time, and obtaining the updated first bounding box of all ships in that frame; S412, generating a matching score between any two ship bounding boxes in the current frame by combining them with a reference centerline; S413, in frames 0 to k, using the Hungarian algorithm based on the matching score to determine the optimal pairwise ship bounding box matching result among multiple ship bounding boxes; S414, if the matching score of the current two ship bounding boxes is greater than 0 in consecutive frames j to k, then it is determined that the current two ship bounding boxes are successfully matched, and the current ship lidar trajectory and the current ship UAV trajectory are successfully matched; otherwise, the matching is recalculated in frames 1 to k+1 until a match is successful.

[0012] Optionally, S44 includes: S441, in frames 0 to k, taking the bow position of the first ship bounding box of the currently matched ship in each frame, forming a UAV trajectory polyline segment m. i On the currently matched AIS trajectory of the vessel, take k trajectory points at the corresponding time to form an AIS trajectory polyline segment n. i S442. According to the captain, m i The bow position is translated point by point along the ship's direction of travel with a preset step size to obtain multiple translation positions; S443, each translation position is compared with n i For the corresponding trajectory points, calculate all error values ​​based on the least squares method, and record the minimum error value and the corresponding translation distance; S444, if the minimum error value is within the preset range, then determine that the current ship's AIS trajectory is successfully matched, and the current ship's UAV trajectory is successfully matched, and use the UAV to perform subsequent tracking and violation detection on the current ship; otherwise, continue to recalculate in frames 1 to k+1 until a match is successful.

[0013] On the other hand, the present invention provides a ship monitoring system that integrates UAV data. The system includes: a first determining unit, configured to acquire AIS data of all ships within the current segment of the waterway and determine a set of ship AIS trajectories based on the ship codes in the AIS data; a second determining unit, configured to acquire three-dimensional point cloud images of all ships within the current segment of the waterway using lidar and determine a set of ship lidar trajectories based on the three-dimensional point cloud images; a third determining unit, configured to acquire two-dimensional images of all ships within the current segment of the waterway using UAVs and determine a set of ship UAV trajectories based on the two-dimensional images; and a multi-source matching unit, configured to perform multi-source matching by combining the ship lidar trajectory set, the ship UAV trajectory set, and the ship AIS trajectory set with a reference centerline. If the matching is successful, the UAV is used for subsequent tracking and violation detection of the ship.

[0014] Optionally, the third determining unit includes: a detection subunit, used to perform target detection on each frame of two-dimensional image to obtain the first ship bounding box and ship type of all ships; a calculation subunit, used to extract feature points in the first ship bounding box of the current ship in two adjacent frames using a sparse optical flow algorithm, and calculate the motion speed of the current ship in the two adjacent frames based on the extracted feature points; a prediction subunit, used to predict the first ship bounding box of the current ship in the next frame in two adjacent frames based on the first ship bounding box of the current ship in the previous frame and the motion speed of the current ship in the two adjacent frames; an update subunit, used to update the predicted first ship bounding box of the current ship in the next frame in two adjacent frames according to the motion direction of the current ship; a transformation subunit, used to obtain the bow latitude and longitude, stern latitude and longitude, ship length and ship width of the current ship based on the position of the updated first ship bounding box of the current ship relative to the edge of the current channel segment, and the latitude and longitude of the UAV; and an aggregating subunit, used to aggregate the updated first ship bounding boxes of all ships in all frames to obtain a ship UAV trajectory set.

[0015] Optionally, the multi-source matching unit includes: a first matching subunit, used to match the current ship lidar trajectory in the ship lidar trajectory set with the UAV trajectory of each ship in the ship UAV trajectory set; if the match is successful, it proceeds to the second matching subunit; the second matching subunit is used to match the current ship lidar trajectory in the ship lidar trajectory set with the AIS trajectory of each ship in the ship AIS trajectory set; if the match is successful, it proceeds to the judgment subunit; the judgment subunit is used to determine whether the current ship type in the successfully matched current ship AIS trajectory is consistent with the current ship type in the successfully matched current ship UAV trajectory; if they are consistent, the match is successful; otherwise, it proceeds to the third matching subunit; the third matching subunit is used to match the successfully matched current ship AIS trajectory with the successfully matched current ship UAV trajectory; if the match is successful, the UAV is used to perform subsequent tracking and violation detection on the current ship.

[0016] The beneficial effects of this invention are:

[0017] This invention provides a method and system for ship monitoring that integrates unmanned aerial vehicle (UAV) data. The method includes: acquiring AIS data of all ships within the current waterway segment; determining a set of ship AIS trajectories based on the ship codes in the AIS data; acquiring three-dimensional point cloud images of all ships within the current waterway segment using lidar; determining a set of ship lidar trajectories based on the three-dimensional point cloud images; acquiring two-dimensional images of all ships within the current waterway segment using UAVs; determining a set of ship UAV trajectories based on the two-dimensional images; and performing multi-source matching by combining the ship lidar trajectories, ship UAV trajectories, and ship AIS trajectories with a reference centerline. If a match is successful, the UAV is used for subsequent tracking and violation detection of the ship. This method integrates three types of data—AIS, fixed lidar, and UAV images—to effectively compensate for the deficiencies of single-source data and improve the accuracy of ship monitoring. Attached Figure Description

[0018] Figure 1 This is a flowchart of a ship monitoring method that integrates unmanned aerial vehicle (UAV) data, provided by an embodiment of the present invention.

[0019] Figure 2 This is a flowchart of determining the set of ship-based unmanned aerial vehicle (UAV) trajectories provided in an embodiment of the present invention;

[0020] Figure 3 This is a flowchart of multi-source matching provided in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the structure of a ship monitoring system that integrates UAV data according to an embodiment of the present invention;

[0022] Figure 5This is a schematic diagram of the structure of the third determining unit provided in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the structure of the multi-source matching unit provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] Figure 1 This is a flowchart of a ship monitoring method that integrates unmanned aerial vehicle (UAV) data, as provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0026] S1. Obtain AIS data of all vessels in the current section of the waterway, and determine the set of vessel AIS trajectories based on the vessel codes in the AIS data;

[0027] In an optional implementation, prior to S1, the following is included:

[0028] Divide the current waterway into multiple segments; establish an electronic fence for each segment;

[0029] Based on the latitude and longitude sampling points of both sides of each flight segment, calculate the corresponding midpoint and form the original centerline;

[0030] The original centerline is thinned using the DP algorithm to obtain the reference centerline.

[0031] Specifically, the current channel segment is obtained and divided into multiple segments; an electronic fence is constructed for each segment, defined by its two side boundaries; the latitude and longitude of the two endpoints of the left boundary and the two endpoints of the right boundary of each segment are recorded; for each pair of adjacent left and right boundary endpoints, their geographic center point is calculated; all center points are connected sequentially to form the original centerline of the current channel segment. The original centerline contains many redundant points, which is detrimental to efficient subsequent calculations. To obtain a reference line that maintains the channel shape while being sufficiently simple, the Douglas-Peucker Algorithm (DP algorithm) is used to thin the original centerline, resulting in the reference centerline P. R .

[0032] When a vessel passes through the current segment of the channel, it typically sends AIS data to the base station at regular intervals. This AIS data includes: vessel name, call sign, MMSI (Main Dimension Machine Identifier), vessel type, navigation status, latitude and longitude, and draft. An electronic fence is used to initially filter out AIS data outside the current segment of the channel. Then, the AIS data of all vessels within the current segment of the channel is compared with the database content, and some non-vessel AIS data, such as that from navigation devices, is filtered out based on the database content. The continuous trajectory of a single vessel is extracted from the MMSI code in the filtered vessel AIS data. The continuous trajectories of all vessels are then combined to obtain the vessel AIS trajectory set.

[0033] S2. Use lidar to acquire three-dimensional point cloud images of all ships in the current section of the channel, and determine the set of lidar trajectories of ships based on the three-dimensional point cloud images;

[0034] LiDAR sensors are deployed at fixed monitoring points along both sides of the current channel boundary. The LiDAR sensors scan the current channel at a fixed frequency (e.g., 10Hz) to obtain a 3D point cloud image of all vessels. Each 3D point cloud image consists of multiple points, each containing coordinate information (x, y, z) and reflection intensity. The latitude and longitude of each point are projected onto the image. Continuous trajectories of individual vessels are extracted from multiple frames of 3D point cloud images with similar shapes. The collection of all vessel continuous trajectories yields a set of vessel LiDAR trajectories.

[0035] When multiple vessels pass through the current section of the channel, complex situations such as a vessel not activating AIS, small fishing boats without AIS, or multiple vessels intersecting or overtaking may occur, causing the side view captured by the lidar at the fixed monitoring point to be unable to match the vessel with the AIS. In such cases, drones are needed to assist in tracking.

[0036] S3. Use UAVs to acquire two-dimensional images of all ships in the current section of the waterway, and determine the set of ship UAV trajectories based on the two-dimensional images;

[0037] In one optional implementation, Figure 2 This is a flowchart of determining the set of ship-based unmanned aerial vehicle (UAV) trajectories provided in an embodiment of the present invention; as shown below. Figure 2 As shown, S3 includes:

[0038] S31. Perform target detection on each frame of the two-dimensional image to obtain the first bounding box of all ships and the ship type;

[0039] S32. In the first bounding box of the current ship in two adjacent frames, the sparse optical flow algorithm is used to extract feature points, and the motion speed of the current ship in the two adjacent frames is calculated based on the extracted feature points.

[0040] S33. Predict the first bounding box of the current ship in the next frame based on the first bounding box of the current ship in the previous frame and the motion speed of the current ship in the next two frames.

[0041] S34. Update the first bounding box of the current ship in the next frame of the two adjacent frames according to the current ship's direction of motion.

[0042] S35. Based on the position of the updated first vessel outer frame relative to the edge of the current channel segment, and the latitude and longitude conversion of the UAV, obtain the bow latitude and longitude, stern latitude and longitude, length and width of the current vessel.

[0043] S36. The set of the first ship bounding boxes after updating all frames of all ships is the set of ship UAV trajectories.

[0044] Specifically, the optical camera on the drone views the current section of the waterway from the air, acquiring a continuous sequence of two-dimensional images. Each frame of the two-dimensional image transmitted back by the drone is input into a pre-trained deep learning object detection model (such as the YOLO series, SSD, etc.). The model outputs the first bounding boxes (initial detection boxes) of all ships in the image, as well as the ship type corresponding to each bounding box (such as cargo ship, fishing boat, etc.).

[0045] To estimate the ship's speed, optical flow analysis is performed on the same ship between adjacent frames (frame t and frame t+1): for a first ship bounding box in frame t, a set of points of interest (i.e., feature points) with significant characteristics are extracted within its region using a sparse optical flow algorithm; and for the corresponding first ship bounding box in frame t+1, a set of points of interest (i.e., feature points) with significant characteristics are extracted within its region using the same sparse optical flow algorithm. The speed of the current feature point is calculated based on the current feature points extracted in frames t and t+1 (i.e., based on the pixel displacement of the feature point and the phase difference between the two frames); the speeds of all feature points are obtained through the above method; the speeds of all feature points are filtered according to a preset threshold, and the average speed of the filtered feature points is used to obtain the speed of the current ship in the two adjacent frames, and the corresponding direction of motion is obtained.

[0046] Predict the position of the first outer frame of the current ship in frame t+1 based on the position of the first outer frame of the current ship in frame t and the speed of the current ship in the two adjacent frames.

[0047] The midpoint of the front frame of the first outer frame of the current ship in the predicted t+1 frame is taken as the bow position, and the midpoint of the back frame of the first outer frame of the current ship in the predicted t+1 frame is taken as the stern position. The line connecting the bow and stern positions is rotated to align with the direction of motion to obtain the updated first outer frame of the current ship.

[0048] Based on the position of the updated first ship's outer frame relative to the current channel edge, as well as the latitude and longitude of the UAV, the flight altitude of the UAV, the focal length of the optical camera in the UAV, and the size conversion of the optical camera sensor, the current ship's true bow latitude and longitude, stern latitude and longitude, length, and width are obtained.

[0049] The continuous trajectory of the current ship is extracted from the updated first ship bounding box in multiple frames of two-dimensional images, and the continuous trajectories of all ships are combined to obtain the ship-UAV trajectory set.

[0050] S4. Combine the ship's lidar trajectory set, ship's drone trajectory set, and ship's AIS trajectory set with the reference centerline for multi-source matching. If the matching is successful, use drones to track the ship and detect violations.

[0051] In one optional implementation, Figure 3 This is a flowchart of multi-source matching provided in an embodiment of the present invention, such as... Figure 3 As shown, S4 includes:

[0052] S41. Match the current ship lidar trajectory in the ship lidar trajectory set with the UAV trajectory of each ship in the ship UAV trajectory set. If the match is successful, proceed to S42.

[0053] In an optional implementation, S41 includes:

[0054] S411. Select the current frame from all images captured by the lidar, perform target bounding box selection on the 3D point cloud image of the current frame, and obtain the second ship bounding box of all ships; at the same time, select a frame that matches the lidar time from all images captured by the UAV, and obtain the updated first ship bounding box of all ships in this frame.

[0055] S412. In the current frame, generate a matching score for any two ship bounding boxes by combining them with the reference centerline.

[0056] S413. In frames 0 to k, the optimal pairwise ship bounding box matching results are determined based on the matching scores using the Hungarian algorithm among multiple ship bounding boxes.

[0057] S414. If the matching scores of the two current ship bounding boxes are both greater than 0 in consecutive frames j to k, then the two current ship bounding boxes are determined to be successfully matched, and the current ship's lidar trajectory and the current ship's UAV trajectory are successfully matched; otherwise, the matching is recalculated in frames 1 to k+1 until a match is successful or the ship is determined to be a non-ship.

[0058] Specifically, the set of ship lidar trajectories P LiDAR Current ship lidar trajectory p LiDAR , and the set of ship-drone trajectories P UAV The drone trajectory of each ship in the middle UAV Perform a match;

[0059] Take one frame from all the 3D point cloud images captured by the lidar, and perform target bounding box selection on the current frame of the 3D point cloud image to obtain the second ship bounding box of all ships; take two as an example: box A and box B; correspondingly, take one frame from all the images captured by the UAV that is the same or close to the lidar time, and perform the above S3 process on the current frame of the 2D image to obtain the updated first ship bounding box of all ships; take two as an example: box C and box D.

[0060] Taking boxes A and C as examples, calculate the matching score between the two boxes;

[0061] ① Select the line segment d that is closest to the center of frame A on the reference center line. i d i+1 ∈P R This line segment is the centerline of the corresponding flight segment. Draw d. i d i+1 perpendicular bisector I i In box A, in I i Laser points 1 meter to the left and right, in I i The projection range on is d LiDAR Frame C and perpendicular line I i The range of intersection is d UAV d LiDAR and d UAV The overlap length is denoted as d. cover According to d LiDAR and d UAV Calculate the first score .

[0062] ② Calculation box A and perpendicular line I i The distance c from the lower intersection point to the electronic fence n And calculate the frame B and vertical line I of the current frame captured by the lidar. i The distance from the lower intersection point of the two points to the electronic fence is taken as the minimum distance. nmin If there are 5 bounding boxes in the current frame captured by the LiDAR, then the smallest distance among the 5 is calculated as c. nmin According to c nmin and c n Calculate the second score .

[0063] ③Based on the distance between the bow positions of box A and box C The distance between the stern positions of box A and box C The distance between the midpoint of the line connecting the bow and stern of ship A (box A) and the midpoint of the line connecting the bow and stern of ship C (box C). and the preset first weight (For example: The weight is 0.2. The weight is 0.1. With a weight of 0.7, the third score is calculated. , where A is the preset first threshold.

[0064] ④ Based on the difference between the sailing directions in box A and box C The fourth score was calculated. , where B is the preset second threshold.

[0065] The matching scores for boxes A and C are ,in, As the preset second weight, if When it is less than the preset third threshold, record It is 0.

[0066] Calculate the matching scores of boxes A and C in frames 0-k, i.e., all... The scores are summed and averaged. Simultaneously, the matching scores of frames B and D in frames 0-k are calculated using the method described above. Then, the Hungarian algorithm is used to calculate the global matching score, which involves summing the matching scores of frames A and C, and frames B and D to obtain the comprehensive score for the first type of matching; summing the matching scores of frames A and D, and frames C and B to obtain the comprehensive score for the second type of matching; summing the matching scores of frames A and B, and frames C and D to obtain the comprehensive score for the third type of matching; and finally, the matching method with the highest score (e.g., the first type) is selected. Furthermore, for the matching of frames A and C, and frames B and D, the scores in consecutive frames jk are averaged. A match is considered successful if all values ​​are greater than 0. If boxes A and C match, then a match is considered successful in consecutive JK frames. If all values ​​are greater than 0 (where j is greater than 0 and less than k), then boxes A and C are successfully matched, and no further matching of subsequent frames is needed. The current ship's lidar trajectory and the current ship's UAV trajectory are successfully matched. If boxes B and D are matched, then within consecutive jk frames... If any frame is equal to 0, then frames B and D continue matching from frame 1 to frame k+1 (the matching process is the same as that from frame 0 to frame k) until the match is successful or unsuccessful. If successful, proceed to S42. If unsuccessful, then frames B and D continue matching from frame 2 to frame k+2, and so on, until all frames are matched. Check if the match is successful or unsuccessful. If successful, proceed to S42.

[0067] S42. Match the current ship lidar trajectory in the ship lidar trajectory set with the AIS trajectory of each ship in the ship AIS trajectory set. If the match is successful, proceed to S43.

[0068] Specifically, in the above embodiment, if box A and box C are successfully matched, the lidar trajectory of box A (the current ship) is matched with the AIS trajectory of each ship in the ship AIS trajectory set. Specifically, the latitude and longitude obtained by lidar projection can be matched with the latitude and longitude in the AIS data. If the match is successful, proceed to S43.

[0069] S43. Determine whether the current ship type in the successfully matched current ship AIS trajectory is consistent with the current ship type in the successfully matched current ship UAV trajectory. If they are consistent, the match is successful; otherwise, proceed to S44.

[0070] S44. Match the successfully matched current vessel AIS trajectory with the successfully matched current vessel drone trajectory. If the match is successful, use the drone to conduct subsequent tracking and violation detection of the current vessel.

[0071] In an optional implementation, S44 includes:

[0072] S441. In frames 0 to k, take the bow position of the first ship bounding box of the currently matched ship in each frame, and form the UAV trajectory polyline segment m. i On the currently matched AIS trajectory of the vessel, take k trajectory points at the corresponding time to form an AIS trajectory polyline segment n. i ;

[0073] S442. According to the ship's length, m i The bow position is translated point by point along the direction of the ship's travel with a preset step size to obtain multiple translation positions;

[0074] S443, Connect each translation position with n i For the corresponding trajectory points, calculate all error values ​​using the least squares method, and record the minimum error value and the corresponding translation distance;

[0075] S444. If the minimum error value is within the preset range, the current ship's AIS trajectory is determined to be successfully matched with the current ship's UAV trajectory, and the UAV is used to track the current ship and detect violations. Otherwise, the calculation continues in frames 1 to k+1 until a match is successful.

[0076] Specifically, in the above embodiment, in frames 0-k, the bow position (i.e., the midpoint of the front frame) of the currently matched ship (frame C) in each frame is taken, and k points are drawn as a polyline segment m. iCorrespondingly, k points are selected on the successfully matched current ship's AIS trajectory and plotted as a polyline segment n. i (Specifically, the ship's position can be estimated based on the AIS launch time and ship speed, obtaining k positions corresponding to the UAV); based on the ship length obtained from S3 (e.g., 40m), m i The bow position is moved towards the stern in specified steps (e.g., 1m), resulting in 40 possible positions. Each position is then compared with n. i The error value is calculated using the least squares method, and the minimum error value s is selected. ais If the minimum error value s ais If the match is successful within the preset fourth threshold range, the minimum error value s is recorded. ais The corresponding translation distance. Conversely, the successfully matched current ship AIS trajectory is matched with the successfully matched current ship UAV trajectory for frames 1 to k+1 (the matching process is the same as for frames 0 to k), until a match is successful or unsuccessful. If successful, the UAV is used to further track the current ship and detect violations. If unsuccessful, the successfully matched current ship AIS trajectory is matched with the successfully matched current ship UAV trajectory for frames 2 to k+2, until all frames are matched, and the match is checked for success or failure. If successful, the UAV is used to further track the current ship and detect violations.

[0077] In an optional implementation, the following is further included after S4:

[0078] If the current ship drone trajectory has a matching result in the ship lidar trajectory set, then the latitude and longitude of the current ship in the lidar trajectory are used as the current ship's latitude and longitude; otherwise, the latitude and longitude of the current ship in the drone trajectory are used as the current ship's latitude and longitude.

[0079] The current ship's latitude and longitude are calibrated using the latitude and longitude data from AIS data, so that the drone can subsequently track the ship based on the calibrated latitude and longitude.

[0080] Specifically, if the current ship's UAV trajectory matches the ship's lidar trajectory set, then the ship's latitude and longitude in the lidar trajectory are used as the ship's latitude and longitude; otherwise, the ship's latitude and longitude in the UAV trajectory are used. The ship's latitude and longitude are then calibrated using AIS results (i.e., if a matching AIS result exists, the latitude and longitude in the AIS are used), and the UAV uses the calibrated latitude and longitude to subsequently track the ship.

[0081] The calibration method is performed according to the following procedure:

[0082] In frames 0-k, the current ship's latitude and longitude are translated according to the translation distance recorded in S44 for each frame. ,Will And the corresponding latitude and longitude in AIS ( The optimized latitude and longitude of the current ship are obtained by calibration using the following formula. :

[0083]

[0084] Where w is the weight, calculated using the following method:

[0085] In frames 0-L (L>k), each frame Smoothing is performed using Kalman filtering to obtain a smoothed result; the smoothed result is then calculated. And the corresponding latitude and longitude in AIS ( The distance to the mean value is calculated. If the mean value is within the preset fifth threshold range, then it is counted. ,otherwise .

[0086] The final optimized latitude and longitude of the current ship A sliding window is used for smoothing. The UAV then uses the smoothed result from the sliding window to continue tracking the current vessel.

[0087] This application first completes the spatial constraints of each data point by sampling the flight segment, and assists LiDAR and UAV in matching ships. Secondly, it provides an evaluation method for LiDAR and UAV ship matching. Finally, it uses AIS results to optimize ship motion trajectories, helping UAVs track ships. The method in this application considers that LiDAR scanning results at monitoring points along the channel are greatly affected by obstruction, making it difficult to complete AIS matching and ship tracking detection in complex multi-ship situations. UAVs are used to assist in counting ship data under overhead conditions. Considering the problem of large jitter in the estimated ship positions of UAVs equipped with monocular cameras, LiDAR is used to provide more accurate ship positioning. Both UAV and LiDAR data supplement the current problems of long reception intervals and low accuracy of AIS data, while also helping to filter out some problematic data.

[0088] Figure 4 This is a schematic diagram of the structure of a ship monitoring system that integrates UAV data according to an embodiment of the present invention; as shown below. Figure 4 As shown, the system includes:

[0089] The first determining unit 201 is used to acquire AIS data of all ships in the current section of the waterway and determine the set of ship AIS trajectories based on the ship codes in the AIS data.

[0090] In an optional implementation, the system further includes:

[0091] Divide the current waterway into multiple segments; establish an electronic fence for each segment;

[0092] Based on the latitude and longitude sampling points of both sides of each flight segment, calculate the corresponding midpoint and form the original centerline;

[0093] The original centerline is thinned using the DP algorithm to obtain the reference centerline.

[0094] The second determining unit 202 is used to acquire three-dimensional point cloud images of all ships in the current section of the waterway using lidar, and to determine the set of ship lidar trajectories based on the three-dimensional point cloud images.

[0095] The third determining unit 203 is used to acquire two-dimensional images of all ships in the current section of the waterway using a drone, and to determine the set of ship drone trajectories based on the two-dimensional images;

[0096] In one optional implementation, Figure 5 This is a schematic diagram of the structure of the third determining unit provided in an embodiment of the present invention; as shown below. Figure 5 As shown, the third determining unit 203 includes:

[0097] The detection subunit 2031 is used to perform target detection on each frame of two-dimensional image to obtain the first ship bounding box and ship type of all ships.

[0098] The computation subunit 2032 is used to extract feature points in the first ship bounding box of the current ship in two adjacent frames using a sparse optical flow algorithm, and to calculate the motion speed of the current ship in the two adjacent frames based on the extracted feature points.

[0099] Prediction subunit 2033 is used to predict the first ship bounding box of the current ship in the next frame based on the first ship bounding box of the current ship in the previous frame and the motion speed of the current ship in the next two frames.

[0100] The update subunit 2034 is used to update the first ship bounding box of the current ship in the next frame of the two adjacent frames according to the current ship's direction of motion.

[0101] The conversion subunit 2035 is used to obtain the bow latitude and longitude, stern latitude and longitude, length and width of the current ship based on the position of the updated first ship outer frame relative to the edge of the current channel segment and the latitude and longitude of the UAV.

[0102] Set subunit 2036 is used to obtain the set of ship UAV trajectories by updating the first ship bounding box set after updating all frames of all ships.

[0103] The multi-source matching unit 204 is used to perform multi-source matching by combining the set of ship lidar trajectory, ship UAV trajectory, and ship AIS trajectory with the reference centerline. If the matching is successful, the UAV is used to track the ship and detect violations.

[0104] In one optional implementation, Figure 6 This is a schematic diagram of the structure of the multi-source matching unit provided in an embodiment of the present invention; as shown below. Figure 6 As shown, the multi-source matching unit 204 includes:

[0105] The first matching subunit 2041 is used to match the current ship lidar trajectory in the ship lidar trajectory set with the UAV trajectory of each ship in the ship UAV trajectory set. If the match is successful, it will proceed to the second matching subunit 2042.

[0106] In an optional implementation, the first matching subunit 2041 is configured to:

[0107] Select the current frame from all images captured by the lidar, perform target bounding box selection on the 3D point cloud image of the current frame, and obtain the second bounding box of all ships; at the same time, select the frame that is closest to the lidar image from all images captured by the UAV, and obtain the updated first bounding box of all ships in this frame.

[0108] In the current frame, any two ship bounding boxes are combined with the reference centerline to generate the matching score of the two ship bounding boxes;

[0109] In frames 0 to k, the optimal pairwise ship bounding box matching results are determined based on the matching scores using the Hungarian algorithm among multiple ship bounding boxes;

[0110] If the matching scores of the two current ship bounding boxes are both greater than 0 in consecutive frames j to k, then the two current ship bounding boxes are considered to be successfully matched, and the current ship's lidar trajectory is successfully matched with the current ship's UAV trajectory; otherwise, the matching will continue in frames 1 to k+1 until a match is successful or the ship is determined to be a non-ship.

[0111] The second matching subunit 2042 is used to match the current ship lidar trajectory in the ship lidar trajectory set with the AIS trajectory of each ship in the ship AIS trajectory set. If the match is successful, it will proceed to the judgment subunit 2043.

[0112] Judgment subunit 2043 is used to determine whether the current ship type in the successfully matched current ship AIS trajectory is consistent with the current ship type in the successfully matched current ship UAV trajectory. If they are consistent, the match is successful; otherwise, proceed to the third matching subunit 2044.

[0113] The third matching subunit 2044 is used to match the successfully matched current ship AIS trajectory with the successfully matched current ship UAV trajectory. If the match is successful, the UAV is used to conduct subsequent tracking and violation detection of the current ship.

[0114] In an optional implementation, the third matching subunit 2044 is used for:

[0115] In frames 0 to k, the bow position of the first ship bounding box of the currently matched ship in each frame is taken to form the UAV trajectory polyline segment m. i On the currently matched AIS trajectory of the vessel, take k trajectory points at the corresponding time to form an AIS trajectory polyline segment n. i ;

[0116] According to the captain, m i The bow position is translated point by point along the direction of the ship's travel with a preset step size to obtain multiple translation positions;

[0117] Connect each translation position with n i For the corresponding trajectory points, calculate all error values ​​using the least squares method, and record the minimum error value and the corresponding translation distance;

[0118] If the minimum error value is within the preset range, the current ship's AIS trajectory is determined to be successfully matched with the current ship's UAV trajectory, and the UAV is used to track the current ship and detect violations. Otherwise, the calculation continues in frames 1 to k+1 until a match is successful.

[0119] In an optional implementation, the system further includes: a calibration unit, for:

[0120] If the current ship drone trajectory has a matching result in the ship lidar trajectory set, then the latitude and longitude of the current ship in the lidar trajectory are used as the current ship's latitude and longitude; otherwise, the latitude and longitude of the current ship in the drone trajectory are used as the current ship's latitude and longitude.

[0121] The current ship's latitude and longitude are calibrated using the latitude and longitude data from AIS data, so that the drone can subsequently track the ship based on the calibrated latitude and longitude.

[0122] The system of the present invention corresponds to the method described above, and the specific implementation of the system will not be repeated here.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of ship monitoring fusing drone data, characterized in that, The method comprises the following steps: S1, acquiring AIS data of all ships in the current section of the channel, and determining a ship AIS trajectory set according to a ship code in the AIS data; S2, acquiring three-dimensional point cloud images of all ships in the current section of the channel by using a laser radar, and determining a ship laser radar trajectory set according to the three-dimensional point cloud images; S3, acquiring two-dimensional images of all ships in the current section of the channel by using a drone, and determining a ship drone trajectory set according to the two-dimensional images; S4, combining the ship laser radar trajectory set, the ship drone trajectory set and the ship AIS trajectory set with a reference center line for multi-source matching, if the matching is successful, using the drone to track and detect the ship in the future; The S4 comprises the following steps: S41, matching a current ship laser radar trajectory in the ship laser radar trajectory set with a ship drone trajectory of each ship in the ship drone trajectory set, if the matching is successful, entering S42; S42, matching the current ship laser radar trajectory in the ship laser radar trajectory set with an AIS trajectory of each ship in the ship AIS trajectory set, if the matching is successful, entering S43; S43, judging whether a current ship type in the current ship AIS trajectory matched successfully is consistent with a current ship type of the current ship drone trajectory matched successfully, if yes, the matching is successful, otherwise, entering S44; S44, matching the current ship AIS trajectory matched successfully with the current ship drone trajectory matched successfully, if the matching is successful, using the drone to track and detect the current ship in the future.

2. The method of claim 1, wherein, Before the S1, the method comprises the following steps: dividing the current section of the channel into multiple navigation sections, and setting up an electronic fence for each navigation section; calculating corresponding midpoints according to latitude and longitude sampling points of two side boundaries of each navigation section, and forming an original center line; thinning the original center line based on a DP algorithm to obtain a reference center line.

3. The method of claim 2, wherein, The S3 comprises the following steps: performing target detection on each two-dimensional image to obtain a first ship bounding box and a ship type of all ships; extracting feature points from the first ship bounding box of the current ship in adjacent two frames by using a sparse optical flow algorithm, and calculating a motion speed of the current ship in the adjacent two frames according to the extracted feature points; predicting the first ship bounding box of the current ship in a later frame of the adjacent two frames according to the first ship bounding box of the current ship in an earlier frame of the adjacent two frames and the motion speed of the current ship in the adjacent two frames; updating the predicted first ship bounding box of the current ship in the later frame of the adjacent two frames according to a motion direction of the current ship; obtaining a bow latitude and longitude, a stern latitude and longitude, a ship length and a ship width of the current ship according to a position of the updated first ship bounding box of the current ship relative to edges of the current section of the channel and a latitude and longitude conversion of the drone; obtaining the ship drone trajectory set from a set of all updated first ship bounding boxes of all ships in all frames.

4. The method of claim 3, wherein, The method further comprises the following steps after the S4: If the current ship UAV track has a matching result in the ship laser radar track set, the latitude and longitude of the current ship in the laser radar track is used as the current ship latitude and longitude; otherwise, the latitude and longitude of the current ship in the UAV track is used as the current ship latitude and longitude; The current ship latitude and longitude is calibrated through the latitude and longitude in the AIS data, so that the UAV can track the current ship according to the calibrated latitude and longitude.

5. The method of claim 4, wherein, The S41 comprises: S411, selecting a current frame from all images captured by the laser radar, and performing target frame selection on the current frame three-dimensional point cloud image to obtain second ship bounding boxes of all ships; meanwhile, a frame matching the laser radar time is taken from all images captured by the UAV, and updated first ship bounding boxes of all ships are obtained in the frame; S412, generating a matching score of the two ship bounding boxes in the current frame in combination with the reference center line; S413, in 0 to k frames, based on the matching score, the Hungarian algorithm is used to determine the optimal two-ship bounding box matching result among the multiple ship bounding boxes; S414, if the matching scores of the current two ship bounding boxes in the continuous j to k frames are all greater than 0, it is determined that the current two ship bounding boxes are matched successfully, and the current ship laser radar track and the current ship UAV track are matched successfully; otherwise, the calculation is continued in the 1 to k+1 frames until the matching is successful.

6. The method of claim 5, wherein, The S44 comprises: S441, in 0 to k frames, taking the ship head position of the first ship bounding box of the current ship matched successfully in each frame to form a UAV track polyline segment mi; on the AIS track of the current ship matched successfully, taking k track points at the corresponding time to form an AIS track polyline segment ni; S442, according to the ship length, the ship head position of mi is translated point by point along the ship running direction at a preset step length to obtain multiple translation positions; S443, based on the least square method, the error value of each translation position and the corresponding track point of ni is calculated, and the minimum error value and the corresponding translation distance are recorded; S444, if the minimum error value is within a preset range, it is determined that the AIS track of the current ship matched successfully is matched successfully with the UAV track of the current ship matched successfully, and the UAV is used to track and detect the current ship subsequently; otherwise, the calculation is continued in the 1 to k+1 frames until the matching is successful.

7. A ship monitoring system fusing drone data, characterized by It comprises: A first determination unit is configured to acquire AIS data of all ships in a current section of a channel, and determine a ship AIS track set according to ship codes in the AIS data; A second determination unit is configured to acquire three-dimensional point cloud images of all ships in the current section of the channel by using a laser radar, and determine a ship laser radar track set according to the three-dimensional point cloud images; A third determination unit is configured to acquire two-dimensional images of all ships in the current section of the channel by using a UAV, and determine a ship UAV track set according to the two-dimensional images; The multi-source matching unit is configured to combine the ship laser radar track set, the ship unmanned aerial vehicle track set and the ship AIS track set with a reference center line for multi-source matching, and if the matching is successful, the unmanned aerial vehicle is used to track and detect the ship. The multi-source matching unit comprises: A first matching subunit configured to match a current ship laser radar track in the ship laser radar track set with each ship unmanned aerial vehicle track in the ship unmanned aerial vehicle track set, and if the matching is successful, the current ship laser radar track enters a second matching subunit; The second matching subunit is configured to match the current ship laser radar track in the ship laser radar track set with each ship AIS track in the ship AIS track set, and if the matching is successful, the current ship AIS track enters a judgment subunit; The judgment subunit is configured to judge whether the current ship type in the current ship AIS track matched successfully is consistent with the current ship type in the current ship unmanned aerial vehicle track matched successfully, and if the current ship type in the current ship AIS track matched successfully is consistent with the current ship type in the current ship unmanned aerial vehicle track matched successfully, the matching is successful, otherwise, the third matching subunit is entered; The third matching subunit is configured to match the current ship AIS track matched successfully with the current ship unmanned aerial vehicle track matched successfully, and if the matching is successful, the unmanned aerial vehicle is used to track and detect the current ship.

8. The system of claim 7, wherein, The third determination unit comprises: A detection subunit configured to perform target detection on each two-dimensional image to obtain a first ship bounding box and a ship type of all ships; A calculation subunit configured to extract feature points in the first ship bounding box of the current ship in adjacent two frames by using a sparse optical flow algorithm, and calculate a motion speed of the current ship in the adjacent two frames according to the extracted feature points; A prediction subunit configured to predict the first ship bounding box of the current ship in a later frame in the adjacent two frames according to the first ship bounding box of the current ship in an earlier frame in the adjacent two frames and the motion speed of the current ship in the adjacent two frames; An update subunit configured to update the predicted first ship bounding box of the current ship in the later frame in the adjacent two frames according to a motion direction of the current ship; A conversion subunit configured to convert the first ship bounding box of the current ship updated to a bow latitude and longitude, a stern latitude and longitude, a ship length and a ship width of the current ship according to a position of the first ship bounding box of the current ship relative to a current segment channel edge and a latitude and longitude of the unmanned aerial vehicle; A set subunit configured to set all updated first ship bounding boxes of all ships in all frames to obtain a ship unmanned aerial vehicle track set.

Images