Inland ship tracking system based on laser radar and video fusion

Abstract

The invention belongs to the technical field of inland waterway intelligent monitoring and ship traffic management, and mainly relates to an inland ship tracking system based on laser radar and video fusion, which performs time synchronization and space calibration through a space-time calibration algorithm to eliminate the deviation between the laser radar and video monitoring, so as to improve the accuracy of the inland ship tracking. Deep cooperation of the laser radar and video monitoring is realized, and accurate matching of data is ensured; an AI target locking algorithm and a multi-distance-level tracking optimization technology are adopted, so that the ship can be stably tracked in an all-weather manner; dynamic risk pre-judgment is carried out based on the trajectory prediction model, the future position of the ship is predicted in advance, and sufficient avoidance time is provided for collision early warning; monitoring errors caused by a floating platform are effectively solved through a hardware and software dual anti-shake compensation mechanism; according to the method, the stability and accuracy of ship monitoring are improved, and meanwhile, a high recognition rate can be kept under the atrocious weather condition.

Description

Technical Field

[0001] This invention belongs to the field of intelligent waterway management technology, and mainly relates to an inland waterway vessel tracking system based on lidar and video fusion. Background Technology

[0002] Inland waterways (especially the Yangtze River main channel) experience high vessel traffic and complex navigation environments (rapid currents, frequent fog, and numerous obstacles), making vessel tracking and monitoring a crucial element in ensuring navigation safety. Current vessel tracking technologies largely rely on single sensors or simple data overlay.

[0003] For example, in Chinese patent application CN202510429693.4, "A Method and Apparatus for Ship Tracking Based on Radar-AIS Optimized by Visual Features," this invention provides a method and apparatus for ship tracking based on radar-AIS optimized by visual features, relating to the field of target tracking technology. The method includes: guiding an optoelectronic turntable to the bearing of the target ship based on target information acquired by radar-AIS; identifying several current ships by analyzing the current image collected by the optoelectronic turntable; tracking the current ships and acquiring visual feature information and track information for each current ship; and determining the target ship from the current ships for tracking based on the target information, visual feature information, and track information. This scheme achieves accurate identification and tracking of ship targets, providing stable and reliable tracking results.

[0004] The aforementioned solution provides a relatively complete ship tracking method, but it still has some core shortcomings: First, lidar can only provide distance and azimuth data, and cannot obtain visual features such as the ship's appearance and name, making it difficult to achieve "target identification"; second, video surveillance is greatly affected by the environment, with low recognition rate in low visibility, and is prone to "losing track" when the ship is moving quickly; moreover, lidar and video surveillance lack collaborative work, resulting in data not being spatiotemporally calibrated and causing positioning deviations; at the same time, the system cannot make dynamic predictions based on historical trajectories, collision warnings are delayed, and there is insufficient time left for crew to avoid collisions; in addition, the existing technology has poor adaptability to floating platforms, and water currents reduce monitoring accuracy; in summary, the existing technology cannot effectively solve the four core problems of "all-weather accurate identification, low-latency collaborative tracking, dynamic risk prediction, and adaptability to floating platforms".

[0005] To solve the above problems, the present invention proposes an inland river ship tracking system based on lidar and video fusion. Through a multi-sensor deep collaboration architecture, the system establishes a unified spatio-temporal coordinate system to achieve millisecond-level linkage tracking between lidar and video. Through an AI target locking algorithm and multi-distance-level tracking optimization, it achieves all-weather stable tracking and identity recognition. Based on a trajectory prediction model, it warns of risks in advance and overcomes platform shaking through a dual hardware-software anti-shake compensation mechanism, ultimately systematically solving the four major technical problems of all-weather accurate recognition, low-latency collaborative tracking, dynamic risk prediction, and floating platform adaptation. Summary of the Invention

[0006] The present invention provides an inland river ship tracking system based on lidar and video fusion, aiming to solve the problems in the prior art such as the single perception ability of lidar and video monitoring, lack of collaborative mechanism, insufficient dynamic prediction ability, and poor adaptability to floating platforms.

[0007] To solve the above problems, the present invention adopts the following technologies to achieve:

[0008] Provide an inland river ship tracking system based on lidar and video fusion, including:

[0009] Hardware collaborative deployment module: Taking a floating navigation mark as the platform, standardizing and integrating lidar, video monitoring, anti-shake bracket, edge computing unit and auxiliary equipment;

[0010] Spatio-temporal calibration algorithm module: Eliminating the spatio-temporal deviation between lidar and video monitoring through time synchronization + space calibration to establish a unified coordinate system; Using distance calibration to eliminate the errors existing in lidar:

[0011] Linkage tracking strategy module: Linking multiple modules, designing a linkage process of lidar trigger - video automatic tracking - data fusion output, automatically tracking and monitoring ships and outputting fusion data;

[0012] Dynamic risk prediction and tracking optimization module: Based on the fusion data, realizing the prediction of the future position of the ship and the dynamic adjustment of the tracking strategy through a trajectory prediction model and a multi-distance-level tracking optimization method;

[0013] Floating platform anti-shake compensation module: Designing a dual compensation mechanism including hardware compensation and software compensation to reduce the monitoring error caused by the shaking of the navigation mark.

[0014] The beneficial effects of the present invention are:

[0015] 1. Through an innovative spatio-temporal dual-calibration fusion algorithm, the system effectively solves the technical problem of misalignment of multi-source sensor data, and significantly improves the accuracy of ship identity recognition and position matching;

[0016] 2. It realizes intelligent linkage tracking between lidar and video surveillance, which greatly shortens the response delay and significantly improves the tracking success rate of high-speed moving ships and the system's emergency response capability;

[0017] 3. By leveraging the complementary advantages of multiple sensors, the limitations of environmental adaptability of traditional monitoring systems have been overcome, enabling the system to maintain a high ship identification rate even under severe weather conditions, thus significantly improving the system's all-weather working capability.

[0018] 4. By adopting advanced trajectory prediction algorithms, accurate prediction of ship navigation trends is achieved, which greatly increases the lead time for collision warning and provides a more sufficient time window for avoidance decisions;

[0019] 5. The anti-shake compensation mechanism effectively overcomes the impact of floating platform sway on monitoring accuracy, ensuring the measurement stability and data reliability of the system under different hydrological conditions. Attached Figure Description

[0020] Figure 1 This is a system structure diagram of the present invention;

[0021] Figure 2 This is a schematic diagram of the system hardware deployment;

[0022] Figure 3 This is a flowchart of the spatiotemporal calibration algorithm;

[0023] Figure 4 It is a trajectory prediction and risk identification map;

[0024] Figure 5 This is a flowchart of the method of using the present invention. Detailed Implementation

[0025] To make the technical means, creative features, and achieved objectives and effects of this invention easier to understand, the invention is further described below with reference to specific embodiments. However, the following embodiments are merely preferred embodiments of this invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments described herein without creative effort are all within the protection scope of this invention. Unless otherwise specified, the experimental methods in the following embodiments are conventional methods, and the materials and reagents used in the following embodiments are commercially available unless otherwise specified.

[0026] Example 1:

[0027] like Figure 1 As shown, this embodiment provides an inland waterway vessel tracking system based on lidar and video fusion, specifically including the following modules:

[0028] Hardware collaborative deployment module: Based on the floating beacon platform, it standardizes and integrates lidar, video surveillance, image stabilization bracket, edge computing unit and auxiliary equipment;

[0029] Spatiotemporal calibration algorithm module: Eliminates spatiotemporal discrepancies between LiDAR and video surveillance through time synchronization and spatial calibration, establishing a unified coordinate system; uses distance calibration to eliminate errors present in the LiDAR.

[0030] Linked tracking strategy module: Links multiple modules to design a linked process of lidar triggering - automatic video tracking - data fusion output, automatically tracking and monitoring ships and outputting fused data;

[0031] Dynamic Risk Prediction and Tracking Optimization Module: Based on fused data, this module uses trajectory prediction models and multi-distance-level tracking optimization methods to predict the future position of ships and dynamically adjust tracking strategies.

[0032] Floating platform anti-shake compensation module: The design incorporates a dual compensation mechanism, including hardware and software compensation, to reduce monitoring errors caused by beacon sway.

[0033] Example 2:

[0034] The difference between this embodiment and embodiment 1 is that the hardware collaborative deployment module specifically includes: LiDAR, video surveillance, anti-shake bracket, edge computing unit and other auxiliary equipment;

[0035] Specifically, the core parameters of the lidar are: ranging range of 0.1-50m, accuracy of ±0.1m, and scanning frequency of 10Hz; the deployment requirements are: to be installed at the center of the top of the navigation mark, with the laser emission direction consistent with the centerline of the navigation mark, a horizontal scanning angle of 360°, and a vertical angle of -15° to +15° (covering the water surface to the superstructure of the ship).

[0036] The core parameters of the video surveillance are: model iDS-2DE4423IW-DE(C), 4 megapixels, 30x optical zoom; deployment requirements are: installed 0.5m to the right of the lidar, with the initial pan-tilt angle consistent with the lidar's reference direction (0° facing the bridge), supporting 360° horizontal rotation and -30° to +90° vertical rotation;

[0037] The core parameters of the anti-shake bracket are: a built-in shock-absorbing bracket with a gyroscope (shock absorption range ±5°); the deployment requirements are: the lidar and video surveillance share the same anti-shake bracket, the bottom of the bracket is rigidly connected to the navigation beacon cabinet, and the gyroscope compensates for navigation beacon sway in real time (response frequency 50Hz).

[0038] The core parameter requirements for the edge computing unit are: computing power Se9, supporting GPU acceleration (16 TOPS); deployment requirements are: installed in the aviation beacon power distribution cabinet, connected to the lidar and video surveillance via RJ45 gigabit network, to achieve real-time data processing (latency <100ms).

[0039] The spatiotemporal calibration algorithm module eliminates the spatiotemporal discrepancy between the LiDAR and video surveillance through time synchronization and spatial calibration, establishes a unified coordinate system, and further uses distance calibration to address the errors present in the LiDAR.

[0040] Specifically, such as Figure 3 As shown in the spatiotemporal calibration algorithm flowchart, time synchronization refers to: using the PPS (pulse per second) signal of the Beidou positioning system (standard equipment for navigation aids) as the time reference, and locking the sampling frequency of both the lidar and video surveillance to 10Hz to ensure that the timestamp error of the data acquisition of both is <1ms, so as to solve the problem of data misalignment;

[0041] Spatial calibration refers to both azimuth calibration and range calibration. Azimuth calibration involves setting up three calibration points with known coordinates (such as fixed buoys) around a navigational aid, and then using a lidar system to measure the azimuth of these calibration points. , , The video surveillance camera rotated to the calibration point and recorded the pan-tilt angle. , , The deviation compensation is calculated using the least squares method, and the formula is as follows:

[0042] ;

[0043] in, For the calibration point orientation angle, For the angle of the gimbal, For fixed compensation values, such as the measured values ​​in the Wuhan section ;

[0044] use Fixed compensation is applied to the measurement results of the lidar;

[0045] Distance calibration refers to: measuring the distance D of a ship using lidar, combining this distance with the pixel width W of the ship in the video footage, and then using a pre-trained distance-pixel mapping model (W=k×D+b, where k and b are statistical coefficients for ships on the Yangtze River) to back-verify the lidar ranging accuracy. The lidar ranging result is compared with the model output; if the deviation is >0.3m, it is automatically corrected. The deviation of the distance-pixel mapping model is calculated using the following formula:

[0046] ;

[0047] in, The distance value that needs to be corrected. These are predicted values ​​obtained based on a distance-pixel mapping model. This is the distance difference that needs to be corrected;

[0048] The calculation formula is:

[0049] ;

[0050] in, The slope in the distance-pixel mapping model. To calculate the pixel width deviation value;

[0051] Use the obtained distance value that needs to be corrected. Correct the ranging results of the lidar.

[0052] The linkage tracking strategy module links multiple modules and designs a linkage process of lidar triggering - automatic video tracking - data fusion output to achieve zero-delay ship tracking;

[0053] Specifically, the implementation steps of the linkage process are as follows:

[0054] Target triggering phase: Real-time laser radar scanning (10Hz). When a vessel is detected entering the 30-meter monitoring range (standard monitoring radius of the Yangtze River waterway), the distance to the vessel is obtained. Azimuth ,speed and ship point cloud features; if ship speed (After removing water and debris), a linkage signal is triggered, and a "tracking command" is sent to the video surveillance via the edge computing unit;

[0055] Automatic video tracking phase: After receiving instructions, the video surveillance system tracks the data based on the spatial calibration formula mentioned above. The system controls the gimbal to rotate to the target azimuth within 0.5 seconds; employs an AI target locking algorithm to automatically identify the ship's outline in the video footage (recognition rate: daytime > 95%, nighttime (with infrared enabled) > 85%), and dynamically adjusts the focus and angle of the gimbal to ensure the ship remains centered in the frame (tracking error < 5% of the frame width); the tracking strategy is modified based on the multi-distance tracking optimization method of the dynamic risk prediction and tracking optimization module; when the ship's distance is < 30 meters, the video surveillance activates the ship name recognition function based on the OCR algorithm, extracts the ship's name, MMSI code, and other identity information, and associates it with the distance and speed data from the lidar to generate a unified ship identity-dynamic parameter file;

[0056] Data fusion output stage: The data obtained by the edge computing unit is fused to form fused data: distance / speed of LiDAR + ship name / image of video. The fused data is pushed to the shore-based platform in real time via MQTT protocol and stored locally on the navigation mark (path / mnt / record / ) at a storage frequency of 1 time / second, and retains 30 days of historical data.

[0057] The dynamic risk prediction and tracking optimization module, based on fused data, uses a trajectory prediction model and a multi-distance-level tracking optimization method to predict the future position of the ship and dynamically adjust the tracking strategy.

[0058] Specifically, the trajectory prediction model refers to an algorithm that uses a "3-point extension line + heading angle correction" to predict the trajectory based on three consecutive sampling points from a lidar system. Location , , Calculate the ship's heading angle ;predict Position after seconds If the predicted location enters the 30-meter collision risk zone of the navigation mark, a warning will be triggered 5 seconds in advance. Figure 4 The trajectory prediction and risk identification diagram is shown below.

[0059] The multi-range tracking optimization method refers to dynamically adjusting the operating parameters of the lidar and video based on the distance D between the ship and the navigation mark, balancing accuracy and energy consumption, and dividing it into the following three distance ranges:

[0060] When 50≤D<30 meters, the lidar scans at 10Hz, the video monitoring switches to medium magnification (20x), and ship name recognition is activated;

[0061] When D < 30 meters, the laser radar scanning frequency is increased to 15Hz, the video surveillance uses high magnification (30x) to lock onto the ship, and the sound and light alarm plays a warning voice in sync.

[0062] The floating platform anti-shake compensation module reduces monitoring errors caused by beacon swaying through a dual compensation mechanism of hardware and software compensation.

[0063] Specifically, hardware compensation refers to: the built-in gyroscope of the anti-shake bracket collects the sway angle of the navigation mark in real time (sampling frequency 50Hz), and drives the bracket to rotate in the opposite direction through the motor (compensation response time <10ms), so as to control the sway amplitude of the lidar and video within ±0.5°;

[0064] Software compensation refers to: the edge computing unit receiving the gyroscope's sway angle data in real time. δ: horizontal sway angle, δ: vertical sway angle), coordinate correction is performed on the lidar point cloud data, and the correction formula is:

[0065] ;

[0066] Where x is the x-axis coordinate before correction, and y is the y-axis coordinate before correction. The horizontal sway angle;

[0067] The formula for correcting the vertical sway angle is:

[0068] ;

[0069] Where x is the x-axis coordinate before correction, and y is the y-axis coordinate before correction. The horizontal sway angle;

[0070] The corrections ensure that the point cloud data matches the actual ship position.

[0071] Example 3:

[0072] The difference between this embodiment and Embodiment 2 is that the construction process of the distance-pixel mapping model includes:

[0073] Pre-training phase: Construct a training set by selecting a batch of typical vessels with known true beams as calibration samples in the buoy monitoring waters. These samples should cover as many common vessel types in the waters as possible. These calibration vessels are then positioned at multiple different, known distance points. During the navigation, at each distance point, a pair of data is collected simultaneously: the precise distance value D measured by the lidar as the baseline, and the pixel width W corresponding to the ship in the video image automatically extracted by the image analysis algorithm; through this process, a large number of (distance D, pixel width W) data pairs covering different distances are collected, which constitute the training dataset of the model.

[0074] Based on the constructed training set, the least squares regression algorithm is used to fit the pre-defined model. The model formula is as follows:

[0075] ;

[0076] Where k is the slope and b is the intercept;

[0077] The goal of this algorithm is to find an optimal set of model parameters k (slope) and b (intercept) such that the model calculates the theoretical pixel width. Compared with the actual measured pixel width in the dataset The overall error between them is minimized;

[0078] By fitting the training set, a set of definite and optimal parameters k and b is obtained. This formula contains specific k and b. This refers to the distance-pixel mapping model that has been trained.

[0079] Example 4:

[0080] The difference between this embodiment and Embodiment 2 is that the ship distance is obtained... Azimuth ,speed And ship point cloud features, including:

[0081] The collected raw point cloud is preprocessed, and a filtering algorithm is used to effectively remove environmental noise such as water surface reflection, waves and rainwater to obtain clean point cloud data.

[0082] The Euclidean clustering algorithm is used to segment point clouds that are geographically close into independent clusters, thereby separating potential target objects; based on preset ship characteristics (such as three-dimensional size, height above the water surface, and location within the waterway), these clusters are judged and ship clusters are selected.

[0083] The centroid position of the obtained ship cluster is calculated, and then the precise distance D and azimuth of the ship relative to the navigation mark are obtained. ;

[0084] Finally, by combining the detection results of multiple frames of continuous tracking of the same ship, the current detection results are correlated with historical frames, and the real-time speed V is accurately estimated based on its position change.

[0085] Furthermore, the AI ​​target locking algorithm includes:

[0086] Video target tracking technology is employed, with algorithm selection tailored to different deployment environments. For open water scenarios (with minimal background interference and prominent targets): In such scenarios, ships typically occupy a small proportion of the frame, and the background is mainly a relatively simple water surface. The main challenges lie in optimizing computational resources and ensuring stable tracking of high-speed moving targets. Highly efficient trackers, such as discriminative correlation filtering (e.g., the KCF algorithm), are used. This algorithm performs calculations in the frequency domain, is extremely fast, meets the computational constraints of edge devices, and exhibits excellent tracking smoothness for target motion without complex backgrounds.

[0087] For complex background scenes (such as bridge areas, urban shorelines, and mountain foregrounds): In such scenes, the background contains a large number of static or dynamic interference objects (such as buildings, trees, and vehicles) that are similar to the texture and color of the ship, which can easily cause the tracker to drift; in this case, a discriminative tracker based on deep learning (such as a SiamRPN twin network architecture) should be enabled.

[0088] By extracting high-level semantic features through deep convolutional neural networks, it is possible to better distinguish target ships from visually similar background interference objects, thus enhancing anti-interference capabilities.

[0089] Furthermore, when the lidar detects multiple ships simultaneously, the system assigns an independent and unique temporary tracking ID to each target and establishes its own tracking thread. It records the target's real-time position, speed, and heading, maintains its historical trajectory points, video appearance features (such as HOG features or deep learning feature vectors of the ship's outline), and final identity information (ship name, MMSI code). When the predicted trajectories of two or more targets are close, intersect, or physically occluded, the system continuously extracts and compares the appearance features of the targets in the video.

[0090] Furthermore, when multiple vessels are present within the monitoring range and their distribution directions exceed two, making it impossible for a single video surveillance system to simultaneously cover all targets, the system continuously records and tracks the position, speed, and heading data of each vessel using LiDAR. This ensures that even if the video surveillance fails to capture a particular target in real time, the system does not lose track of any vessel at the global level. For video surveillance resources, a periodic switching strategy is adopted. That is, according to a certain time period, the orientation of the video surveillance is adjusted based on the real-time target data provided by the LiDAR, thereby circumventing and acquiring monitoring videos of different vessels. During this process, if the trajectory prediction model determines that a vessel poses a collision risk, the system will interrupt the periodic switching rule and instruct the video surveillance to continuously lock onto the high-risk target, implement uninterrupted tracking and monitoring, and trigger corresponding alarm commands.

[0091] The ship name recognition function includes: when the distance to the ship continuously measured by the lidar is less than 30 meters, the video AI target locking algorithm has stably tracked the target ship, the ship name recognition function is activated, and the video surveillance captures a high-definition image of the target; at the same time, the AIS receiver continuously receives the signal, and the edge computing unit associates and calculates the MMSI code and its associated official ship name in the AIS target closest to the target's location based on the target's bearing provided by the lidar.

[0092] Perform ship name region detection and image enhancement on the captured video frames; use OCR recognition algorithm to recognize the text content in the image, and output the recognized text content and its confidence level; output AIS ship name and MMSI code;

[0093] The text content extracted by OCR is compared with the ship name in AIS. If the ship name matches, the ship name is output as the name of the ship. If there is a problem with the ship name extracted by OCR (low confidence) or extraction fails, the ship name in AIS is used as the name of the recognized ship. If there are problems with both OCR extraction and AIS reception, a message is sent to the designated back-end personnel to request manual intervention.

[0094] In practical applications, since the solution uses AIS to receive signals and OCR to extract ship names, in some cases, multiple ships may enter the ship name recognition range at relatively close times and distances. In this situation, AIS channel conflicts and close proximity may occur, and ship name information may be obscured. In this case, the AIS receiver continuously outputs a list containing MMSI, ship name, position, and heading / speed. The lidar provides a target queue with precise relative positions (range D, azimuth α) and contour sizes. The position, heading / speed matching method is used to match the ship name and signal, thereby achieving AIS positioning. OCR continuously identifies the ship, and when text information is present in the acquired image, it is immediately extracted and matched with the position, heading / speed to determine the specific ship name.

[0095] Example 5:

[0096] like Figure 5 As shown, this embodiment provides a method for using an inland waterway vessel tracking system based on lidar and video fusion, specifically including the following steps:

[0097] Step S1: Deploy the hardware collaboration module of the system in the waterway, combine it with the spatiotemporal calibration algorithm module to eliminate the time deviation of multi-source data, use the floating platform anti-shake compensation module to make up for the error caused by the swaying of navigation marks, and collect multimodal data of ships in the waterway;

[0098] Specifically, floating navigation marks are deployed at key locations in the waterway, such as bridge areas and junctions. These navigation marks include: lidar, video surveillance equipment, anti-shake brackets, edge computing units, and other auxiliary equipment including various sensors, speakers, power distribution cabinets, and gyroscopes.

[0099] After the floating beacon is deployed, the system is activated, and the spatiotemporal calibration algorithm module automatically starts to eliminate the discrepancies between the lidar and video surveillance. At the time level, the PPS (pulse per second) signal from the BeiDou positioning system (standard equipment on the beacon) is used as the time reference to lock the sampling frequencies of the lidar and video surveillance, ensuring that they have the same data acquisition frequency and keeping the timestamp error to a minimum. At the spatial level, three calibration points with known coordinates are set around the beacon. The lidar is used to measure the azimuth angle of each of the three calibration points, and the gimbal angle of the video surveillance camera rotating to the calibration point is recorded. The results are then analyzed using the least squares method. Calculate a fixed compensation value and use it to compensate for the measurement results of the lidar, thereby achieving spatial calibration;

[0100] Upon system startup, the dual compensation mechanism of the floating platform anti-shake compensation module reduces the shaking error of the video images acquired by the hardware collaboration module. At the hardware level, the gyroscope built into the anti-shake bracket senses the sway angle of the navigation beacon caused by water flow and wind waves in real time at a frequency of 50Hz. The motor drives the anti-shake bracket to perform instantaneous reverse movement, suppressing the shaking amplitude of the sensor platform to within ±0.5°. At the software level, the edge computing unit receives the sway angle data from the gyroscope in real time and performs coordinate correction on the lidar point cloud data to reduce the impact of shaking on the acquired images.

[0101] The final lidar measurement results and video monitoring results are consistent in time and space, and have high stability.

[0102] Step S2: The linkage tracking strategy module analyzes the video scan images and drives the hardware coordination module to monitor the movement of ships within the range;

[0103] Specifically, the linkage tracking strategy module analyzes the LiDAR detection results. When the LiDAR detects a ship entering its 50-meter detection range, it extracts the ship's distance. Azimuth ,speed Point cloud features; to eliminate the influence of water flow, a speed threshold of 0.5 m / s is set. If the speed exceeds the threshold, a linkage signal is triggered, and instructions are sent to the video surveillance via the edge computing unit.

[0104] Subsequently, the linkage tracking strategy module calculates the movement angle based on the real-time ship point cloud characteristics and the actual data from the PTZ, controls the PTZ of the video monitoring equipment to turn towards the target, uses an AI target locking algorithm to identify the ship's outline, and adjusts the PTZ position and angle in real time to keep the ship always in the center of the image, thus achieving ship monitoring; at the same time, the fused data mainly includes radar distance, speed, image and other content, forming structured fused data.

[0105] Step S3: During the monitoring process, the dynamic risk prediction and tracking optimization module analyzes the image results, monitors the risks in real time, and adjusts the tracking strategy of the linkage tracking strategy module; at the same time, the spatiotemporal calibration algorithm module analyzes the monitoring data in real time to eliminate hardware deviations.

[0106] Specifically, during the real-time monitoring of the vessel by the linkage tracking strategy module, the dynamic risk prediction and tracking optimization module is simultaneously invoked to adjust the hardware operating parameters using a multi-distance-level tracking optimization method. The operation of the hardware equipment and the linkage tracking strategy module are adjusted according to the changes in distance, which are divided into long-distance, medium-distance, and short-distance levels. At medium distance, the linkage tracking strategy module is activated to start the vessel name recognition function, which identifies the vessel's name based on the OCR algorithm and MMSI signal. At short distance, the audible and visual alarm is activated to play a warning voice.

[0107] Meanwhile, the dynamic risk prediction and tracking optimization module analyzes the fused data in real time and uses a trajectory prediction model to predict the ship's position 10 seconds later. If the predicted position enters the 30-meter collision risk zone of the navigation mark, an early warning is triggered 5 seconds in advance, an alarm is automatically triggered, and broadcast information is sent to the corresponding ship.

[0108] In addition, the linkage tracking strategy module will periodically activate the distance calibration function of the spatiotemporal calibration algorithm module, using a pre-trained distance-pixel mapping model to verify the ranging results of the LiDAR. When the deviation exceeds a certain level, automatic correction is triggered.

[0109] Step S4: The linkage tracking strategy module integrates and uploads the data acquired across the entire tracking process and stores it locally, completing a full tracking task;

[0110] Specifically, through the collaborative work of multiple modules, stable ship data information in the waterway is collected. During the collection process, the linkage tracking strategy module integrates the obtained data into a structured data format every second, stores it locally on the navigation mark, and uploads the integrated data to the cloud platform in real time via the MQTT protocol until the ship leaves the monitoring range of the navigation mark. The stored data is retained for 30 days, and indexes such as time and ship name are created for easy querying.

[0111] The application effects of the inland waterway vessel tracking system in this invention in real-world scenarios are as follows:

[0112] Actual measurements taken on the white buoy at the Wuqiao Bridge in Wuhan showed that, through BeiDou PPS time synchronization (error < 1ms) and least squares spatial calibration (deviation < 2.3°), and after aligning the lidar and video data, the ship position display deviation decreased from 10° to within 0.5°, and the target identification confirmation rate (ship name + position matching) increased to 98%.

[0113] Through the linkage process of LiDAR triggering and automatic video tracking, the gimbal turning response time has been reduced from the current 5 seconds to 0.5 seconds without manual intervention. Combined with the AI ​​target locking algorithm, the success rate of ship tracking (especially fast ships with speeds > 5 m / s) has increased from 60% to 95%. In the Shanghai section, LiDAR successfully tracked a passenger ship that suddenly turned, and the video did not lose track of it throughout the entire process, which bought 8 seconds of avoidance time for collision warning.

[0114] Leveraging the all-weather advantages of lidar and the visualization advantages of video, the ship identification rate remains above 85% even in extreme environments such as nighttime (with infrared enabled) and foggy days with visibility of 5 meters, representing a 55 percentage point improvement over single video surveillance. In a foggy test in the Wuhan section in February 2025, the fusion system successfully monitored 12 passing ships with no missed detections, while traditional video surveillance only identified 3 ships.

[0115] Based on the trajectory prediction model of "3-point extension line + heading angle correction", the system can predict the ship's position 10 seconds in advance, increasing the collision warning lead time from the current 2 seconds (after the ship enters the 30-meter risk zone) to 7 seconds, and increasing the crew's decision-making time for avoidance by 250%. After the application of this system in the Tianxingzhou Bridge section of the Yangtze River, the number of close approach incidents (<50 meters) has decreased from 15 per month to 3, reducing the collision risk by 80%.

[0116] The dual compensation mechanism of hardware image stabilization bracket and software coordinate correction reduces the lidar ranging error caused by beacon sway from 2 meters to 0.3 meters, and the video image jitter amplitude is less than 1%, ensuring that the monitoring accuracy does not decrease during the Yangtze River flood season (when the rapid water flow exacerbates the swaying of the beacon). In the March flood season test in the Wuhan section, the ranging error of the fusion system was stable within 0.5 meters, meeting the accuracy requirements for inland river collision early warning.

[0117] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An inland waterway vessel tracking system based on lidar and video fusion, characterized in that, include: Hardware collaborative deployment module: Based on the floating beacon platform, it standardizes and integrates lidar, video surveillance, image stabilization bracket, edge computing unit and auxiliary equipment; Spatiotemporal calibration algorithm module: Eliminates spatiotemporal discrepancies between LiDAR and video surveillance through time synchronization and spatial calibration, establishing a unified coordinate system; uses distance calibration to eliminate errors present in the LiDAR. Linked tracking strategy module: Links multiple modules to design a linked process of lidar triggering - automatic video tracking - data fusion output, automatically tracking and monitoring ships and outputting fused data; Dynamic Risk Prediction and Tracking Optimization Module: Based on fused data, this module uses trajectory prediction models and multi-distance-level tracking optimization methods to predict the future position of ships and dynamically adjust tracking strategies. Floating platform anti-shake compensation module: The design incorporates a dual compensation mechanism, including hardware and software compensation, to reduce monitoring errors caused by beacon sway.

2. The inland waterway vessel tracking system according to claim 1, characterized in that: The hardware collaborative deployment module specifically includes: The hardware collaborative deployment module uses floating beacons as a platform and standardizes the integration of lidar, video surveillance, image stabilization brackets, edge computing units, and auxiliary equipment. The core parameters of the lidar involved are: ranging range 0.1-50m, accuracy ±0.1m, scanning frequency 10Hz; deployment requirements are: installed at the center of the top of the beacon, with the laser emission direction consistent with the central axis of the beacon, horizontal scanning angle 360°, and vertical angle -15°~+15°. The core parameters of the video surveillance are: an image resolution of no less than 4 million pixels and an optical zoom capability of no less than 30 times; the deployment requirements are: to be installed 0.5m to the right of the lidar, with the initial pan-tilt angle consistent with the lidar's reference direction, and to support 360° horizontal rotation and -30° to +90° vertical rotation. The core parameters of the anti-shake bracket are: a shock-absorbing bracket with a built-in gyroscope; the deployment requirements are: the lidar and video surveillance share the same anti-shake bracket, the bottom of the bracket is rigidly connected to the navigation beacon cabinet, and the gyroscope compensates for the swaying of the navigation beacon in real time. The core parameter requirements for the edge computing unit are: Se9 computing power, supporting GPU acceleration; deployment requirements are: installed in the aviation beacon power distribution cabinet, connected to the lidar and video surveillance via RJ45 gigabit network to achieve real-time data processing.

3. The inland waterway vessel tracking system according to claim 1, characterized in that: The spatiotemporal calibration algorithm module specifically includes: Includes functions such as time synchronization, spatial calibration, and distance calibration. Time synchronization refers to using the PPS signal of the BeiDou positioning system as a time reference to ensure that the sampling frequency of the lidar and video surveillance are consistent, thereby reducing errors and solving the problem of data misalignment. Spatial calibration refers to both azimuth calibration and range calibration. Azimuth calibration involves setting up three calibration points with known coordinates within the beacon area, and then using a lidar system to measure the azimuth of these calibration points. , , The video surveillance camera rotated to the calibration point and recorded the pan-tilt angle. , , The deviation compensation is calculated using the least squares method, and the formula is as follows: ； in, For the calibration point orientation angle, For the angle of the gimbal, It is a fixed compensation value; use Fixed compensation is applied to the measurement results of the lidar; Distance calibration refers to: measuring the distance D of a ship using a lidar, combining this distance with the pixel width W of the ship in the video footage, and then using a pre-trained distance-pixel mapping model to reverse-engineer the lidar ranging accuracy. The lidar ranging result is compared with the model output; if the deviation is greater than 0.3m, automatic correction is performed. The deviation of the distance-pixel mapping model is calculated using the following formula: ； in, The distance value that needs to be corrected. These are predicted values ​​obtained based on a distance-pixel mapping model. This is the distance difference that needs to be corrected; The calculation formula is: ； in, The slope in the distance-pixel mapping model. To calculate the pixel width deviation value; Use the obtained distance value that needs correction Correct the ranging results of the lidar.

4. The inland waterway vessel tracking system according to claim 3, characterized in that: The specific construction process of the distance-pixel mapping model includes: Pre-training phase: Construct a training set by selecting a batch of typical vessels with known true beams as calibration samples in the buoy monitoring waters; then, place these calibration vessels at multiple different, known distance points. During the voyage, at each distance point, a pair of data is collected simultaneously: the precise distance value D measured by the lidar as the baseline, and the pixel width W corresponding to the ship in the video image automatically extracted by the image analysis algorithm; data pairs at different distances are collected to form the training dataset for the model. Based on the constructed training set, the least squares regression algorithm is used to fit the pre-defined model. The model formula is as follows: ； Where k is the slope and b is the intercept; The goal of this algorithm is to find an optimal set of model parameters k and b such that the model calculates the theoretical pixel width. Compared with the actual measured pixel width in the dataset The overall error between them is minimized; By fitting the training set, a set of definite and optimal parameters k and b is obtained. This formula contains specific k and b. This yields a pre-trained distance-pixel mapping model.

5. The inland waterway vessel tracking system according to claim 1, characterized in that: The linkage tracking strategy module specifically includes: Design a linkage process for LiDAR triggering, automatic video tracking, and data fusion output; Target triggering phase: The lidar scans in real time. When a ship is detected to enter the 50-meter monitoring range, it acquires the ship's distance D, azimuth α, speed V, and ship point cloud features. Based on the preset speed conditions, a linkage signal is triggered, and a tracking command is sent to the video surveillance via the edge computing unit. Automatic video tracking phase: After receiving instructions, the video surveillance system tracks the video based on the spatial calibration formula. The system controls the gimbal to rotate to the target azimuth within 0.5 seconds; employs an AI target locking algorithm to automatically identify the ship's outline in the video footage and dynamically adjusts the focus and angle of the gimbal to ensure the ship remains centered in the frame; the tracking strategy is modified based on the multi-distance tracking optimization method of the dynamic risk prediction and tracking optimization module; when the ship's distance is less than 30 meters, the video surveillance activates the ship name recognition function based on the OCR algorithm to extract the ship's name, MMSI code, and other identity information, which is then correlated with the distance and speed data from the lidar to generate a unified file of ship identity and dynamic parameters. Data fusion output stage: The data obtained by the edge computing unit is fused to form fused data: distance / speed of LiDAR + ship name / image of video. The fused data is pushed to the shore-based platform in real time via MQTT protocol and stored locally on the navigation mark at a frequency of 1 time / second, retaining 30 days of historical data.

6. The inland waterway vessel tracking system according to claim 5, characterized in that: The extraction of point cloud features of the ship specifically includes: The collected raw point cloud is preprocessed, and a filtering algorithm is used to effectively remove environmental noise such as water surface reflection, waves and rainwater to obtain clean point cloud data. The Euclidean clustering algorithm is used to segment point clouds that are geographically close into each other into independent clusters, thereby separating potential target objects; these clusters are then identified based on preset ship features, and ship clusters are selected. The centroid position of the obtained ship cluster is calculated, and then the precise distance D and azimuth of the ship relative to the navigation mark are obtained. ; Finally, by combining the detection results of multiple frames of continuous tracking of the same ship, the current detection results are correlated with historical frames, and the real-time speed V is accurately estimated based on its position change.

7. The inland waterway vessel tracking system according to claim 5, characterized in that: The AI ​​target locking algorithm specifically includes: The AI ​​target locking algorithm uses video target tracking technology and selects the algorithm according to the different deployment environments. For open water scenes, it uses a high-efficiency tracker represented by discriminative correlation filtering. For complex background scenarios: a deep learning-based discriminative tracker is enabled; high-level semantic features are extracted through deep convolutional neural networks to better distinguish the target ship from visually similar background distractions.

8. The inland waterway vessel tracking system according to claim 5, characterized in that: The ship name recognition function specifically includes: When the distance to a ship continuously measured by the lidar is less than 30 meters, the ship name recognition function is activated, and the video surveillance captures a high-definition image of the target. At the same time, the AIS receiver continuously receives signals, and the edge computing unit associates and calculates the MMSI code and its associated official ship name in the AIS target closest to the target's bearing based on the target's bearing provided by the lidar. Perform ship name region detection and image enhancement on the captured video frames; use OCR recognition algorithm to recognize the text content in the image, and output the recognized text content and its confidence level; output AIS ship name and MMSI code; The text content extracted by OCR is compared with the ship name in AIS. If the ship name matches, the ship name is output as the name of the ship. If the ship name extracted by OCR is abnormal, the ship name in AIS is used as the name of the recognized ship. If there are problems with both OCR extraction and AIS reception, a message is sent to the designated back-end personnel to request manual intervention.

9. The inland waterway vessel tracking system according to claim 1, characterized in that: The dynamic risk prediction and tracking optimization module specifically includes: Based on fused data, the future position of ships can be predicted and the tracking strategy can be dynamically adjusted through trajectory prediction models and multi-distance-level tracking optimization methods. The trajectory prediction model refers to an algorithm that uses "3-point extension line + heading angle correction" based on three consecutive sampling points from a lidar system. Location , , Calculate the ship's heading angle ;predict Position after seconds If the predicted location enters the 30-meter collision risk zone of the navigation mark, a warning will be triggered 5 seconds in advance. The multi-range tracking optimization method refers to dynamically adjusting the operating parameters of the lidar and video based on the distance D between the ship and the navigation mark, balancing accuracy and energy consumption, and dividing it into the following three distance ranges: When 50≤D<30 meters, the lidar scans at 10Hz, the video monitoring switches to medium magnification, and ship name recognition is activated; When D < 30 meters, the lidar scanning frequency is increased to 15Hz, the video surveillance uses high magnification to lock onto the ship, and the sound and light alarm plays a warning voice in sync.

10. The inland waterway vessel tracking system according to claim 1, characterized in that: The floating platform anti-shake compensation module specifically includes: The design incorporates a dual compensation mechanism, including both hardware and software compensation, to reduce monitoring errors caused by beacon swaying. Hardware compensation refers to the following: the gyroscope built into the anti-shake bracket collects the sway angle of the navigation mark in real time, and drives the bracket to rotate in the opposite direction through the motor, so as to control the sway amplitude of the lidar and video within ±0.5°. Software compensation refers to the correction formula for the horizontal sway angle: ； Where x is the x-axis coordinate before correction, and y is the y-axis coordinate before correction. The horizontal sway angle; The formula for correcting the vertical sway angle is: ； Where x is the x-axis coordinate before correction, and y is the y-axis coordinate before correction. The vertical sway angle; The corrections ensure that the point cloud data matches the actual ship position.

Images