Inland waterway ship traffic monitoring method, device and electronic equipment
By fusing data from shore-based cameras and automatic identification systems, and using YOLO11 and Bot-SORT algorithms to generate bird's-eye view, the problem of insufficient multi-source data fusion in inland waterway vessel traffic monitoring has been solved. This has enabled high-precision, real-time monitoring of vessel movement status and improved the integrity and reliability of monitoring data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI SHIP & SHIPPING RES INST CO LTD
- Filing Date
- 2026-01-19
- Publication Date
- 2026-06-05
AI Technical Summary
In the monitoring of inland waterway vessel traffic, there is insufficient fusion of multi-source data, resulting in low monitoring accuracy and poor reliability. In particular, it is difficult to meet the requirements of high-precision and real-time monitoring in complex water environments, and there is a lack of in-depth fusion analysis of multi-dimensional information such as vessel type and spatial location.
By using shore-based cameras for ship target detection and multi-target tracking, visual trajectories are generated. Depth estimation and cross-modal identity association are performed by combining data from an automatic identification system. The YOLO11 framework and Bot-SORT algorithm are used to enhance recognition capabilities, construct a bird's-eye view, and achieve cross-modal trajectory matching and information fusion.
It improves the integrity and reliability of monitoring data, ensures the integrity and stability of video trajectory, compensates for data loss caused by equipment failure or human shutdown, and achieves high-precision monitoring of ship motion status.
Smart Images
Figure CN122157178A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent shipping technology, and in particular to a method, device and electronic equipment for monitoring ship traffic on inland waterways. Background Technology
[0002] With the rapid development of inland waterway shipping, the density of vessel traffic has increased significantly, and the complexity of waterway traffic flow has risen. To ensure navigation safety and improve regulatory efficiency, shore-based cameras and Automatic Identification Systems (AIS) are widely used for vessel monitoring. While shore-based cameras can capture vessel images, their image quality is significantly reduced under adverse conditions such as fog, haze, rain, and snow due to environmental factors like weather and lighting. This makes it difficult to maintain a stable target recognition rate. Furthermore, overexposure or shadows are common under strong light or backlighting conditions, making it difficult to capture vessel features. Simultaneously, the two-dimensional image information provided by visual sensors lacks depth support, making it impossible to determine the actual geographical location of the vessel. AIS systems can provide information such as vessel identity, location, speed, and heading, but in the complex waterways of inland waterways, data loss often occurs due to equipment malfunction or human intervention, increasing the difficulty of regulatory implementation. Currently, academia and industry have proposed methods based on single data sources or shallow fusion, such as using visual recognition or AIS monitoring alone, or conducting preliminary hybrid fusion. However, these methods still struggle to meet the high-precision, real-time monitoring requirements in complex aquatic environments. In particular, they lack in-depth fusion analysis of multi-dimensional information such as vessel type and spatial location, failing to achieve spatial modeling of vessel motion and cross-modal identity association, resulting in incomplete and unreliable monitoring data. Therefore, inland waterway vessel traffic monitoring suffers from low monitoring accuracy and poor reliability due to insufficient multi-source data fusion. Summary of the Invention
[0003] Therefore, it is necessary to provide a method, device, and electronic equipment for monitoring inland waterway vessel traffic, which addresses the problems of low monitoring accuracy and poor reliability caused by insufficient fusion of multi-source data in inland waterway vessel traffic monitoring.
[0004] This invention provides a method for monitoring vessel traffic in inland waterways, the method comprising: Ship target detection is performed based on video data collected by shore-based cameras to obtain the ship's location and type information; Multi-target tracking is performed on detected ships to generate visual trajectories of the ships, and the ship positions are predicted using historical motion information in the event of occlusion to maintain tracking continuity. Depth estimation is performed based on the video data, two-dimensional image information is converted into three-dimensional spatial information, and a bird's-eye view is constructed to generate the bird's-eye view trajectory of the ship. Acquire and process Automatic Identification System (AIS) data to generate the vessel's AIS trajectory; The bird's-eye view trajectory is matched with the trajectory of the automatic recognition system, and cross-modal identity association is achieved based on trajectory similarity calculation; Output the matching results and display the fused ship motion status information on the monitoring interface.
[0005] In one embodiment, the ship target detection employs a detector based on the YOLO11 framework, wherein an MLDA module is introduced into the C3k2 module. The MLDA module is based on Mamba's forget gate mechanism and modular structure design to enhance the perception of ship structural features.
[0006] In one embodiment, the multi-target tracking employs the Bot-SORT algorithm, combined with an anti-occlusion mechanism. In the event of occlusion, the ship's position is predicted by the motion patterns of the historical automatic identification system trajectory or the bird's-eye view trajectory, while maintaining the appearance characteristics before occlusion to ensure ID consistency.
[0007] In one embodiment, the anti-occlusion mechanism includes: The system determines the occlusion area, calculates the ratio of the occlusion area to the minimum area of all candidate boxes, removes the detection box when it exceeds a preset threshold, and calculates the velocity component based on historical trajectory points to predict the position of the occluded target.
[0008] In one embodiment, the depth estimation uses the Depth Anything V2 model, and the two-dimensional pixel coordinates are back-projected to three-dimensional spatial coordinates through the camera intrinsic parameter matrix. Then, the ship's heading angle is calculated by combining the line-of-sight angle and the local correction angle to construct a bird's-eye view.
[0009] In one embodiment, matching the bird's-eye view trajectory with the trajectory of the automatic recognition system includes: Candidate pairs are selected based on ship type consistency. Calculate the Euclidean distance between the endpoints of the trajectory for preliminary screening; The dynamic time warping algorithm is used to calculate trajectory similarity.
[0010] In one embodiment, calculating trajectory similarity includes: A directional consistency constraint is applied, and the similarity is corrected based on the angle between the first and last vectors of the trajectory. When the angle exceeds the angle threshold, a penalty factor is added to exclude mismatches with opposite directions.
[0011] In one embodiment, matching the bird's-eye view trajectory with the trajectory of the automatic recognition system further includes: The Hungarian algorithm is used to process the similarity matrix to obtain the optimal matching set between the bird's-eye view trajectory and the trajectory of the automatic recognition system.
[0012] The present invention also provides an inland waterway vessel traffic monitoring device, the device comprising: The ship information acquisition module is used to detect ship targets based on video data collected by shore-based cameras and obtain the ship's location and type information. The ship position tracking module is used to perform multi-target tracking on detected ships, generate the ship's visual trajectory, and predict the ship's position using historical motion information in the event of occlusion to maintain tracking continuity. The bird's-eye view trajectory generation module is used to perform depth estimation based on the video data, convert two-dimensional image information into three-dimensional spatial information, construct a bird's-eye view, and generate the bird's-eye view trajectory of the ship. Automatic Identification System Trajectory Generation Module: This module is used to acquire and process Automatic Identification System data to generate the Automatic Identification System trajectory of a vessel. The ship trajectory matching and association module is used to match the trajectory of the bird's-eye view with the trajectory of the automatic identification system, and realize cross-modal identity association based on trajectory similarity calculation; The matching result output module is used to output the matching results and display the fused ship motion status information in the monitoring interface.
[0013] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the inland waterway vessel traffic monitoring method as described above.
[0014] The aforementioned inland waterway vessel traffic monitoring methods, devices, and electronic equipment detect vessels by using video data collected from shore-based cameras to obtain their location and type information. They then perform multi-target tracking to generate visual trajectories for the detected vessels. In cases of occlusion, historical motion information is used to predict vessel positions to maintain tracking continuity, thus ensuring the integrity and stability of the video trajectory. By performing depth estimation based on video data, two-dimensional image information is converted into three-dimensional spatial information, and a bird's-eye view is constructed to generate the vessel's bird's-eye view trajectory, solving the problem of visual sensors lacking depth support and unable to determine the vessel's actual geographical location. Automatic identification system (AIS) data is acquired and processed to generate the vessel's AIS trajectory, compensating for data loss due to equipment failure or human intervention. By matching the bird's-eye view trajectory with the AIS trajectory, cross-modal identity association is achieved based on trajectory similarity calculation. The matching result is then output and displayed on the monitoring interface as fused vessel motion status information. This achieves deep fusion of shore-based video and AIS information, overcoming the technical problems of low monitoring accuracy and poor reliability caused by insufficient multi-source data fusion, and improving the integrity and reliability of monitoring data. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0016] Figure 1 Here is a flowchart of an inland waterway vessel traffic monitoring method according to one embodiment; Figure 2 This is a schematic diagram of an inland waterway vessel traffic monitoring method according to one embodiment; Figure 3 This is a structural diagram of a YOLO11-MLDA detector according to one embodiment; Figure 4 A flowchart illustrating a ship target tracking process with an occlusion handling mechanism, as shown in one embodiment. Figure 5 This is a schematic diagram of an inland waterway vessel traffic monitoring device according to one embodiment; Figure 6 This is an internal structural diagram of an electronic device according to one embodiment. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] The following is combined with Figures 1-6 This invention describes an inland waterway vessel traffic monitoring method, apparatus, and electronic equipment.
[0019] like Figure 1 and Figure 2 As shown, in one embodiment, a method for monitoring vessel traffic on inland waterways includes the following steps: Step S110: Detect ship targets based on video data collected by shore-based cameras to obtain the ship's location and type information.
[0020] A detector based on the YOLO11 framework is adopted, and an MLDA module is introduced into the C3k2 module. This MLDA module is based on the forget gate mechanism and modular structure design of Mamba. By introducing LDConv to replace the traditional convolution module Conv, a more spatially adaptable MLDA module is constructed, which enhances the perception of ship structural features, improves the recognition accuracy of multiple types of ships in complex water environments, overcomes the image quality degradation caused by weather and lighting factors, and achieves high-precision detection of ship targets.
[0021] During the detection process, the input layer receives video image frames captured by a shore-based camera, typically with a resolution of 1920×1080, using an advanced YOLO11-MLDA detector. (See YOLO11-MLDA structure for details.) Figure 3 The main structural difference compared to YOLO11 lies in the C3k2 module, where the Bottleneck module is replaced with the MLDA module, which offers advantages over Mamba. The detector outputs a set of ship candidate boxes. Where L is the number of detected targets, and each target box can be represented as , , These represent the pixel coordinates of the top-left and bottom-right corners of the target bounding box, respectively, providing initial observations for subsequent processing. Step S120: Perform multi-target tracking on the detected vessel, generate the vessel's visual trajectory, and predict the vessel's position using historical motion information in case of occlusion to maintain tracking continuity.
[0022] Multi-target tracking employs the Bot-SORT algorithm, combined with an anti-occlusion mechanism. In cases of occlusion, the ship's position is predicted based on the motion patterns of historical automatic identification system trajectories or bird's-eye view trajectories, while preserving the ship's appearance characteristics before occlusion to maintain ID consistency. See also Figure 4 The anti-occlusion mechanism includes: determining the occlusion area (OAR) and calculating the area of the occlusion area. The minimum area of all candidate boxes When the ratio exceeds a preset threshold Remove the detection frame at the time, that is Remove the detection frame at any time, among which For the first Area of each candidate box The number of bounding boxes where occlusion occurs. A preset threshold is set to resist occlusion, and the velocity component is calculated based on historical trajectory points to predict the position of the occluded target.
[0023] When the above conditions are met, the smallest bounding rectangle (AR) is stored as the occlusion region in the OAR set, and all detection boxes located within this region are deleted to avoid false detections caused by ship overlap. For the removed occluded target boxes, their current positions are predicted using prior information. When the target has AIS data, the motion velocity in the pixel coordinate system is calculated using the AIS trajectory. Let the difference in the projected coordinates of the AIS at two adjacent moments be... , The predicted bounding box for the current frame of the video is: in, These are the coordinates of the target bounding box at the previous moment. When AIS information is lacking, the motion patterns of the visual BEV trajectory are used for estimation. Let the time interval be... The historical trajectory points within are Its velocity components are: This leads to the predicted bounding box location: .
[0024] For the appearance features of occluded targets, the embedding vector before occlusion is directly inherited to ensure the stability of the ID during occlusion. Finally, the normally detected target boxes and predicted boxes are input together into Bot-SORT for joint tracking, outputting a complete video trajectory sequence. This solves the occlusion problem caused by ship intersections or overlaps. By comparing the occluded area with the area of the smallest candidate box, it ensures that only relatively large occlusions are triggered for removal, avoiding the false removal of non-occluded targets. Historical motion information is used to maintain tracking continuity, thereby reducing ID drift and trajectory breaks, and improving tracking robustness and trajectory integrity.
[0025] Step S130: Depth estimation is performed based on video data, two-dimensional image information is converted into three-dimensional spatial information, and a bird's-eye view is constructed to generate the bird's-eye view trajectory of the ship.
[0026] Depth estimation employs the Depth Anything V2 model, and the two-dimensional pixel coordinates are back-projected to three-dimensional spatial coordinates through the camera intrinsic parameter matrix. Then, the ship's heading angle is calculated by combining the line-of-sight angle and local correction angle to construct a bird's-eye view. Through monocular depth estimation, camera parameter mapping, and spatial projection, the output ship trajectory is endowed with three-dimensional spatial attributes, generating a top-down BEV view. This addresses the problems of traditional shore-based cameras lacking depth information and having weak spatial modeling capabilities, providing spatial data support for subsequent trajectory matching.
[0027] 3D bounding box estimation aims to assign precise 3D spatial attributes, including position, size, and orientation, to each 2D detected target. First, it utilizes normalized depth values provided by the monocular depth estimation algorithm DepthAnythingV2. The system first linearly maps it to the physical distance range captured by the shore-based camera (e.g., to ): Then, the pixels in the two-dimensional image Its physical depth Z is obtained using the camera intrinsic parameter matrix. By back-projecting it, we can obtain its three-dimensional coordinates in the camera coordinate system. The calculation formula is as follows: In the above expressions, For pixel coordinates, Principal point coordinates It is the focal length.
[0028] To estimate the ship's heading in an image, a heading angle inference method based on a combination of line-of-sight angle and local correction angle is applied. Using a monocular image as input, and combining the target's geometric features and spatial position in the image, the ship's heading angle is approximately estimated. It is decomposed into the sum of the viewing angle and a local correction angle heuristically estimated based on the aspect ratio of the 2D detection box: Among them, the line of sight angle The direction from the camera's optical center to the target's three-dimensional position is determined by the following formula: in, Here are the horizontal spatial coordinates of the target in the camera coordinate system, reflecting the target's left and right offset relative to the camera. This is the target's forward spatial coordinate in the camera coordinate system, equivalent to the physical distance obtained from depth estimation. This angle reflects the target's orientation in space relative to the camera. Local correction angle. Then, heuristic estimation is performed based on the shape and position of the target's two-dimensional detection box in the image, with the following specific rules: in, The aspect ratio of the detection frame, The x-coordinate of the center of the detection frame. The x-coordinate of the principal point in the image is given. This method can approximately recover the ship's heading at the image level without additional sensors, providing directional constraints for subsequent trajectory matching, identity fusion, and anomaly detection.
[0029] The Depth Anything V2 model is used to estimate the depth of images captured by shore-based cameras. By adding camera intrinsic and extrinsic parameters, more accurate spatial distance information between the ship and the background is obtained. The depth information obtained is used to construct the BEV (Body Elevation Vehicle) to provide basic data for subsequent spatial modeling.
[0030] The 3D state information is fed into the BEV module to generate a bird's-eye view. This module projects targets in the 3D world onto a unified, God-like 2D plane. A 3D point in the camera coordinate system is mapped to BEV image coordinates: in, For the width and height of the BEV image, Used for the lateral coordinate range when projecting BEV, defining the left and right boundaries. The forward coordinate range used for BEV projection defines the forward and backward boundaries. After projection, the system uses different geometric primitives for symbolic rendering based on the target category, ultimately generating a bird's-eye view that clearly and intuitively reflects the spatial layout and relative distances between targets in the scene. By mapping the obtained 3D coordinates and heading angles onto the BEV plane, a view including the ship's position and heading is constructed, compensating for the lack of depth information from shore-based cameras, realizing ship spatial modeling, providing the ship's actual geographical location and heading information, and supporting subsequent trajectory matching.
[0031] Step S140: Acquire and process Automatic Identification System (AIS) data to generate the vessel's AIS trajectory.
[0032] The data from the automatic identification system is cleaned and its coordinates are transformed to unify it into a pixel coordinate system, ensuring the consistency between AIS data and visual data, solving the problem of missing or erroneous AIS data, and providing reliable input for trajectory matching.
[0033] Step S150: Match the bird's-eye view trajectory with the trajectory of the automatic recognition system, and realize cross-modal identity association based on trajectory similarity calculation.
[0034] Candidate pairs are selected based on ship type consistency, meaning that each AIS trajectory can only be matched with BEV trajectories of the same type of ship. The Euclidean distance between the trajectory endpoints is then calculated for initial screening, followed by dynamic time warping to calculate trajectory similarity. Trajectory similarity calculation includes directional consistency constraints and similarity correction based on the angle between the trajectory's beginning and ending vectors. When the angle exceeds a threshold, a penalty factor is added to eliminate mismatches with opposite directions. Simultaneously, the Hungarian algorithm is used to process the similarity matrix to obtain the optimal matching set between the bird's-eye view trajectory and the AIS trajectory.
[0035] In AIS trajectory set BEV ship trajectory set In the diagram, I represents the number of AIS targets existing at the current moment, and the i-th target is... The AIS trajectory is as follows: The endpoint of each trajectory is represented as: , For pixel coordinates transformed from AIS to a planar coordinate system, the BEV ship trajectory similarly possesses the above properties. The BEV trajectory is as follows: The endpoint of each trajectory is represented as: For each AIS track, the type of vessel is clearly identified. For each BEV track, the vessel type detected by the YOLOv11-MLDA detector determines the BEV track type. To reduce computational complexity, the matching candidates for each AIS track can only be BEV tracks of the same vessel type. To further reduce computational complexity, for all AIS tracks... With BEV trajectory First, calculate the Euclidean distance between the nearest time point and the endpoint: Among them, threshold Setting the distance to half the image width, if the end positions of two trajectories are more than half a field of view apart, the probability that they belong to the same ship is extremely low. This process can significantly reduce the number of candidate pairs that require complex calculations later, thereby improving real-time performance.
[0036] After selecting the corresponding Euclidean distances, the global matching cost needs to be calculated, which involves introducing a cumulative cost matrix. It is defined as the distance from the starting point (1.1) to the position ( , Minimum cumulative cost: in, Represents trajectory points and The cumulative cost matrix during alignment avoids the exponential computation of exhaustively searching all paths through dynamic programming, allowing trajectory alignment to be completed in polynomial time and yielding the minimum cumulative cost between two trajectories. Each pair of trajectories yields a set. Ultimately, multiple groups were obtained. Final cost Then, directional consistency constraints are applied to the obtained results, because spatial alignment alone may lead to incorrect matching with similar shapes but opposite directions. Therefore, directional consistency correction is introduced.
[0037] Define the start and end vectors of the AIS and BEV trajectories as follows: Calculate the angle between the two. Final trajectory similarity: ,when Smaller, correction factor This indicates that the two trajectories are in the same direction and therefore are... reserve, When the value is large, the correction factor increases significantly, thus penalizing incorrect matches considerably. Construct a similarity matrix The Hungarian matching method is run on the similarity matrix to obtain the optimal matching set. Each AIS trajectory is matched with the globally optimal BEV trajectory, thereby achieving spatial behavior alignment of heterogeneous trajectories. This breaks through the dependence of traditional trajectory matching on time synchronization. The technical effect is to improve the accuracy and real-time performance of cross-modal identity association and reduce false matching.
[0038] Step S160: Output the matching results and display the fused ship motion status information in the monitoring interface. The integrated Automatic Identification System (AIS) information, such as MMSI and call sign, is displayed on the monitoring interface, providing regulators with intuitive vessel identity and movement data, enhancing the semantic integrity and regulatory credibility of the monitoring data, and reducing the difficulty of regulation.
[0039] This embodiment of the inland waterway vessel traffic monitoring method detects vessels based on video data collected by shore-based cameras, obtaining their location and type information. It then performs multi-target tracking to generate visual trajectories for the detected vessels. In cases of occlusion, historical motion information is used to predict vessel positions to maintain tracking continuity, ensuring the integrity and stability of the video trajectory. Depth estimation based on video data converts two-dimensional image information into three-dimensional spatial information, constructing a bird's-eye view to generate the vessel's bird's-eye view trajectory, solving the problem of visual sensors lacking depth support and unable to determine the vessel's actual geographical location. Automatic identification system (AIS) data is acquired and processed to generate the AIS trajectory, compensating for data loss due to equipment failure or manual shutdown. By matching the bird's-eye view trajectory with the AIS trajectory and calculating trajectory similarity, cross-modal identity association is achieved. The matching result is then output and displayed on the monitoring interface as fused vessel motion status information. This achieves deep fusion of shore-based video and AIS information, overcoming the technical problems of low monitoring accuracy and poor reliability caused by insufficient multi-source data fusion, and improving the integrity and reliability of monitoring data.
[0040] The inland waterway vessel traffic monitoring device provided by the present invention is described below. The inland waterway vessel traffic monitoring device described below can be referred to in correspondence with the inland waterway vessel traffic monitoring method described above.
[0041] like Figure 5 As shown, in one embodiment, an inland waterway vessel traffic monitoring device includes a vessel information acquisition module 510, a vessel position tracking module 520, a bird's-eye view trajectory generation module 530, an automatic identification system trajectory generation module 540, a vessel trajectory matching and association module 550, and a matching result output module 560.
[0042] The ship information acquisition module 510 is used to detect ship targets based on video data collected by shore-based cameras and to obtain the ship's location and type information. The ship position tracking module 520 is used to perform multi-target tracking on detected ships, generate the visual trajectory of the ships, and predict the ship position through historical motion information in the event of occlusion to maintain tracking continuity. The bird's-eye view trajectory generation module 530 is used to perform depth estimation based on the video data, convert two-dimensional image information into three-dimensional spatial information, construct a bird's-eye view, and generate the bird's-eye view trajectory of the ship. The Automatic Identification System (AIS) trajectory generation module 540 is used to acquire and process AIS data to generate the AIS trajectory of the vessel.
[0043] The ship trajectory matching and association module 550 is used to match the bird's-eye view trajectory with the trajectory of the automatic identification system, and realize cross-modal identity association based on trajectory similarity calculation. The matching result output module 560 is used to output the matching results and display the fused ship motion status information in the monitoring interface.
[0044] Figure 6 This example illustrates a schematic diagram of the physical structure of an electronic device, which can be a smart terminal. Its internal structure diagram can be as follows: Figure 6 As shown. The electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an inland waterway vessel traffic monitoring method, which includes: Ship target detection is performed based on video data collected by shore-based cameras to obtain the ship's location and type information; Multi-target tracking is performed on detected ships to generate visual trajectories of the ships, and the ship positions are predicted using historical motion information in the event of occlusion to maintain tracking continuity. Depth estimation is performed based on the video data, two-dimensional image information is converted into three-dimensional spatial information, and a bird's-eye view is constructed to generate the bird's-eye view trajectory of the ship. Acquire and process Automatic Identification System (AIS) data to generate the vessel's AIS trajectory; The bird's-eye view trajectory is matched with the trajectory of the automatic recognition system, and cross-modal identity association is achieved based on trajectory similarity calculation; Output the matching results and display the fused ship motion status information on the monitoring interface.
[0045] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device to which the present invention is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0046] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements a method for monitoring vessel traffic in inland waterways, the method comprising: Ship target detection is performed based on video data collected by shore-based cameras to obtain the ship's location and type information; Multi-target tracking is performed on detected ships to generate visual trajectories of the ships, and the ship positions are predicted using historical motion information in the event of occlusion to maintain tracking continuity. Depth estimation is performed based on the video data, two-dimensional image information is converted into three-dimensional spatial information, and a bird's-eye view is constructed to generate the bird's-eye view trajectory of the ship. Acquire and process Automatic Identification System (AIS) data to generate the vessel's AIS trajectory; The bird's-eye view trajectory is matched with the trajectory of the automatic recognition system, and cross-modal identity association is achieved based on trajectory similarity calculation; Output the matching results and display the fused ship motion status information on the monitoring interface.
[0047] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, it implements a method for monitoring vessel traffic in inland waterways, the method comprising: Ship target detection is performed based on video data collected by shore-based cameras to obtain the ship's location and type information; Multi-target tracking is performed on detected ships to generate visual trajectories of the ships, and the ship positions are predicted using historical motion information in the event of occlusion to maintain tracking continuity. Depth estimation is performed based on the video data, two-dimensional image information is converted into three-dimensional spatial information, and a bird's-eye view is constructed to generate the bird's-eye view trajectory of the ship. Acquire and process Automatic Identification System (AIS) data to generate the vessel's AIS trajectory; The bird's-eye view trajectory is matched with the trajectory of the automatic recognition system, and cross-modal identity association is achieved based on trajectory similarity calculation; Output the matching results and display the fused ship motion status information on the monitoring interface.
[0048] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.
[0049] By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0050] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0051] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.
Claims
1. A method for monitoring vessel traffic on inland waterways, characterized in that, The method includes: Ship target detection is performed based on video data collected by shore-based cameras to obtain the ship's location and type information; Multi-target tracking is performed on detected ships to generate visual trajectories of the ships, and the ship positions are predicted using historical motion information in the event of occlusion to maintain tracking continuity. Depth estimation is performed based on the video data, two-dimensional image information is converted into three-dimensional spatial information, and a bird's-eye view is constructed to generate the bird's-eye view trajectory of the ship. Acquire and process Automatic Identification System (AIS) data to generate the vessel's AIS trajectory; The bird's-eye view trajectory is matched with the trajectory of the automatic recognition system, and cross-modal identity association is achieved based on trajectory similarity calculation; Output the matching results and display the fused ship motion status information on the monitoring interface.
2. The inland waterway vessel traffic monitoring method according to claim 1, characterized in that, The ship target detection employs a detector based on the YOLO11 framework, in which an MLDA module is introduced into the C3k2 module. The MLDA module is based on Mamba's forget gate mechanism and modular structure design to enhance the perception of ship structural features.
3. The inland waterway vessel traffic monitoring method according to claim 1, characterized in that, The multi-target tracking employs the Bot-SORT algorithm, combined with an anti-occlusion mechanism. In the event of occlusion, the ship's position is predicted by the motion patterns of the historical automatic identification system trajectory or bird's-eye view trajectory, while maintaining the appearance characteristics before occlusion to ensure ID consistency.
4. The inland waterway vessel traffic monitoring method according to claim 3, characterized in that, The anti-occlusion mechanism includes: The system determines the occlusion area, calculates the ratio of the occlusion area to the minimum area of all candidate boxes, removes the detection box when it exceeds a preset threshold, and calculates the velocity component based on historical trajectory points to predict the position of the occluded target.
5. The inland waterway vessel traffic monitoring method according to claim 1, characterized in that, The depth estimation uses the Depth Anything V2 model, and the two-dimensional pixel coordinates are back-projected to three-dimensional spatial coordinates through the camera intrinsic parameter matrix. Then, the ship's heading angle is calculated by combining the line-of-sight angle and the local correction angle to construct a bird's-eye view.
6. The inland waterway vessel traffic monitoring method according to claim 1, characterized in that, The matching of the bird's-eye view trajectory with the trajectory of the automatic recognition system includes: Candidate pairs are selected based on ship type consistency. Calculate the Euclidean distance between the endpoints of the trajectory for preliminary screening; The dynamic time warping algorithm is used to calculate trajectory similarity.
7. The inland waterway vessel traffic monitoring method according to claim 6, characterized in that, The calculation of trajectory similarity includes: A directional consistency constraint is applied, and the similarity is corrected based on the angle between the first and last vectors of the trajectory. When the angle exceeds the angle threshold, a penalty factor is added to exclude mismatches with opposite directions.
8. The inland waterway vessel traffic monitoring method according to claim 6, characterized in that, The matching of the bird's-eye view trajectory with the trajectory of the automatic recognition system also includes: The Hungarian algorithm is used to process the similarity matrix to obtain the optimal matching set between the bird's-eye view trajectory and the trajectory of the automatic recognition system.
9. A vessel traffic monitoring device for inland waterways, characterized in that, The device includes: The ship information acquisition module is used to detect ship targets based on video data collected by shore-based cameras and obtain the ship's location and type information. The ship position tracking module is used to perform multi-target tracking on detected ships, generate the ship's visual trajectory, and predict the ship's position using historical motion information in the event of occlusion to maintain tracking continuity. The bird's-eye view trajectory generation module is used to perform depth estimation based on the video data, convert two-dimensional image information into three-dimensional spatial information, construct a bird's-eye view, and generate the bird's-eye view trajectory of the ship. Automatic Identification System Trajectory Generation Module: This module is used to acquire and process Automatic Identification System data to generate the Automatic Identification System trajectory of a vessel. The ship trajectory matching and association module is used to match the trajectory of the bird's-eye view with the trajectory of the automatic identification system, and realize cross-modal identity association based on trajectory similarity calculation; The matching result output module is used to output the matching results and display the fused ship motion status information in the monitoring interface.
10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the inland waterway vessel traffic monitoring method according to any one of claims 1 to 8.