A BEV vision fusion method and system for intelligent supervision of inland waterways

Abstract

This invention discloses a BEV (Bird's Eye View) visual fusion method and system for intelligent inland waterway monitoring. The method includes: generating a BEV using shore-based camera calibration and inverse perspective mapping (IPM); projecting electronic chart (ENC) data onto the same plane coordinates and performing online calibration using shoreline, fixed navigation marks, and other elements as references to suppress scale deviations caused by extrinsic parameter drift and water level changes; performing time alignment and forward extrapolation based on historical changes on Automatic Identification Information (AIS) data; combining this with vessel targets detected in the BEV view; using a matching range that automatically adjusts according to the scene to achieve spatiotemporal correspondence; and performing consistency checks based on the waterway centerline and navigation boundaries; and explicitly marking and alerting targets that do not correspond or exhibit anomalies. The method and system can operate in real-time on shore, enabling unified display and reliable identification of multi-source data. It effectively reduces mismatches and missed detections under conditions of dense vessel traffic, AIS message delays, and obstruction, improving the accuracy and availability of waterway traffic monitoring. The attached figure is a schematic diagram of the method flow of this invention.

Description

Technical Field

[0001] This invention belongs to the field of inland waterway traffic condition monitoring technology, and in particular relates to the unified geographic reference fusion and matching discrimination of visual BEV reconstruction and electronic chart / AIS data. Background Technology

[0002] In current ship navigation and waterway monitoring systems, Electronic Navigational Charts (ENCs) serve as the core information platform, widely used to display static elements such as channel structure, water depth information, and fixed navigation marks. They also assist in the dynamic monitoring of vessels by receiving data from Automatic Identification System (AIS). However, the data fusion mechanism based on ENCs and AIS still has significant shortcomings in intelligent waterway monitoring. Firstly, AIS data is not mandatory for all vessels; some small vessels or illegal vessels are not equipped with AIS or have actively turned it off, resulting in blind spots in waterway monitoring. Secondly, due to the inherent lag in AIS transmission, it is difficult to meet the requirements of high-frequency, real-time dynamic monitoring, especially in areas with dense vessel traffic, complex currents, or potential emergencies. Traditional AIS+ENC-based monitoring systems exhibit significant shortcomings in terms of perception coverage and data timeliness.

[0003] Meanwhile, although waterway regulatory authorities are currently equipped with optical cameras or closed-circuit monitoring systems to improve their ability to perceive the waterway environment, traditional two-dimensional video surveillance lacks a spatial coordinate alignment mechanism with electronic nautical charts. Ships or obstacles in the images are difficult to accurately map to the geographical coordinate system of the nautical chart. Furthermore, video images are easily affected by lighting, occlusion, and viewing angle limitations, often failing to capture all surface targets comprehensively, creating blind spots in visual perception and further limiting their practical value in waterway situation identification and target tracking.

[0004] In recent years, Bird's Eye View (BEV) perception technology has rapidly developed in intelligent ships and automated navigation systems. The BEV method, through geometric transformation or multi-view fusion, converts images from cameras into a top view at a unified scale in real time, effectively reconstructing the spatial layout around the ship and possessing significant dynamic target visualization capabilities. Crucially, the BEV view can directly display the position and dynamic trajectory of ships, buoys, and moving obstacles without AIS equipment activated, thereby compensating for the incompleteness and latency issues of AIS perception and providing broader coverage and higher real-time image-level information support for waterway supervision.

[0005] However, there is currently a lack of effective integration mechanisms between BEV (Battery Elevator) perception results and electronic chart systems. While BEVs offer high real-time and spatial continuity perception capabilities, they cannot reflect authoritative channel information such as water depth, navigation mark types, and channel classifications. Electronic charts, while possessing a channel knowledge system, cannot proactively perceive the dynamics of vessels and the status of targets within the current channel. Although both systems have their advantages in perception dimensions and information authority, they have failed to form a complementary and unified regulatory platform. Summary of the Invention

[0006] To address the problems existing in current technologies, this invention discloses a BEV (Board of Vehicles) visual fusion method for intelligent monitoring of inland waterways. This method solves the problems of existing electronic chart systems' inability to monitor vessels without activated AIS (Automatic Information System), insufficient dynamic perception capabilities, and poor spatial correlation of video images. It improves vessel identification and traffic situation monitoring capabilities in complex waterways, and specifically includes the following steps:

[0007] Deploy shore-based camera systems and acquire video images of ship targets. Pre-calibrate the internal and external parameters of the cameras, establish a transformation model from image pixel coordinates to real-world coordinates, and use the IPM algorithm to perform geometric back-projection on the two-dimensional images to convert them into BEV images with a spatially consistent bird's-eye view.

[0008] Real-time channel information is obtained through electronic nautical charts, and the chart data is unified to a geographic reference frame consistent with BEV images through coordinate projection transformation.

[0009] Spatial alignment and overlay rendering are performed on BEV images and ENC data. Chart elements are directly drawn on BEV images through layer overlay, ensuring that the target detection results have a consistent spatial reference with the actual channel structure.

[0010] A layer fusion mechanism is used to spatially compare the ship targets in the BEV image with the ship position information in the electronic chart, thereby identifying the AIS status and determining whether the ship target information in the BEV image matches the AIS ship information provided by the electronic chart, and then explicitly distinguishing and marking it in the visualization interface.

[0011] Furthermore, when pre-calibrating the camera's internal and external parameters: first, determine the mapping relationship between pixel coordinates in the video image and physical space coordinates, and then calibrate the camera's internal and external parameters, including the intrinsic parameter matrix. Represented as:

[0012]

[0013] in, For pixel focal length, The matrix is ​​obtained by using the checkerboard calibration method, with the main point positions as the reference points.

[0014] The camera's extrinsic parameters in the world coordinate system are rotation matrices. With translation vector The merger is represented as:

[0015]

[0016] in, is the scale factor.

[0017] Furthermore, the tilted image is transformed into a bird's-eye view from above for ground target geometry reconstruction. For any pixel in the image... First, back-project to the camera coordinate system:

[0018]

[0019] in To assume ground depth, the camera coordinates are transformed to world coordinates:

[0020]

[0021] in, , From the extrinsic parameter calibration, during the BEV image generation process, the processing method sets the spatial resolution of the top view, and... The coordinates are projected onto a planar image grid and then interpolated and resampled to generate a BEV image.

[0022] Furthermore, when unifying nautical chart data to a geographic reference frame consistent with BEV imagery: load the electronic nautical chart for the corresponding waters, where the electronic nautical chart includes multiple layer features, each with spatial coordinate attributes, and the original coordinates are latitude and longitude. This means that a projection transformation function is used to uniformly convert latitude and longitude to UTM coordinates, thereby controlling image alignment.

[0023]

[0024] This algorithm is used to control spatial points in BEV images. The points of the ENC data layer coincide with the points projected onto the same plane.

[0025] Furthermore, a layer fusion mechanism is employed to spatially compare the vessel targets in the BEV image with the vessel position information in the electronic nautical chart, thereby identifying the AIS status and explicitly distinguishing and marking it in the visualization interface.

[0026]

[0027] in Represents the position of a ship on an electronic nautical chart. Represents the ship's position in the BEV image. Represents the search radius. For each BEV image frame, the system determines whether the detected ship target matches the AIS ship information provided by the electronic chart in a unified coordinate system. If the ship target exists in the BEV image but is not displayed in the ENC data layer, the ship target is marked in the image. Conversely, if the ship target can be identified and matched by AIS in the electronic chart, it is recorded as a successful match.

[0028] A BEV vision fusion system for intelligent monitoring of inland waterways includes:

[0029] The perception data acquisition module collects video images of ship targets, as well as information on the identity, speed, and heading of nearby ships, by setting up camera equipment at key nodes in the waterway.

[0030] The data preprocessing module acquires real-time channel information from electronic nautical charts and unifies the chart data to a geographic reference frame consistent with the BEV image through coordinate projection transformation.

[0031] The BEV image generation module establishes a transformation model from image pixel coordinates to real-world coordinates, and uses the IPM algorithm to perform geometric back projection on the two-dimensional image, converting it into a BEV image with a bird's-eye view that has spatial consistency.

[0032] The electronic chart analysis and coordinate unification module performs spatial alignment and overlay rendering of BEV images and ENC data. It directly draws chart elements on BEV images through layer overlay and controls the target detection results to have a consistent spatial reference with the actual channel structure.

[0033] The layer fusion and spatial relationship recognition module uses a layer fusion mechanism to spatially compare the ship targets in the BEV image with the ship position information in the electronic chart, thereby identifying the AIS status and determining whether the ship target information in the BEV image matches the AIS ship information provided by the electronic chart, and then explicitly distinguishing and marking it in the visualization interface.

[0034] By adopting the above technical solution, this invention discloses a BEV visual fusion method and system for intelligent supervision of inland waterways. The method establishes a unified geographic coordinate system to achieve spatial alignment and visual overlay based on BEV images and ENC vector maps, thereby enabling comprehensive identification of vessels with and without AIS activated in the waterway. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of the method of the present invention.

[0037] Figure 2 This is a schematic diagram illustrating the process of combining the BEV perspective of the waterway with electronic charts in the method of this invention.

[0038] Figure 3 This is a structural diagram of a BEV vision fusion system for intelligent monitoring of inland waterways.

[0039] Figure 4 shows a comparison between the BEV projection and the original camera view.

[0040] Figure 5 The AIS matching interface is displayed. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0042] like Figure 1 The BEV visual fusion method shown here for intelligent supervision of inland waterways specifically includes the following steps:

[0043] First, video images are acquired by deploying a shore-based camera system. By pre-calibrating the intrinsic and extrinsic parameters of the camera, a transformation model from image pixel coordinates to real-world coordinates is established. The IPM algorithm is then used to geometrically back-project the two-dimensional image, converting it into a spatially consistent bird's-eye view BEV image.

[0044] Secondly, the latest navigation information is obtained through electronic nautical charts. Then, through coordinate projection transformation, the chart data is unified to a geographic reference frame consistent with the BEV imagery.

[0045] Then, spatial alignment and overlay rendering are performed on the BEV image and ENC data. By overlaying layers, nautical chart features are directly drawn on the BEV image, so that the target detection results have a consistent spatial reference with the actual channel structure.

[0046] Finally, AIS data can be combined to identify and distinguish targets with AIS enabled, further improving the accuracy and completeness of target recognition.

[0047] Furthermore, in the specific process, this method uses inverse perspective mapping (IPM) to convert the two-dimensional images captured by the camera into BEV images from a top-down perspective, and spatially fuses them with electronic chart (ENC) data to overlay and display the channel structure and real-time dynamic information in a unified geographic coordinate system.

[0048] This invention discloses a BEV vision fusion system for intelligent monitoring of inland waterways, comprising:

[0049] The perception data acquisition module is used for comprehensive monitoring of the dynamic environment of vessels in the waterway. At the hardware level, camera equipment is installed at key nodes in the waterway. Simultaneously, the system receives dynamic information such as the identity, speed, and heading of nearby vessels via the AIS module. Multi-source sensor data is processed through time synchronization to form structured data packets, providing comprehensive environmental information for subsequent image transformation and coordinate fusion.

[0050] The data preprocessing module, in order to realize the mapping relationship between pixel coordinates and physical space coordinates, needs to complete the calibration of the camera's intrinsic and extrinsic parameters. Among these, the intrinsic parameter matrix... Represented as:

[0051] in, For pixel focal length, The principal point position is given. This matrix can be obtained using the checkerboard calibration method. The camera's extrinsic parameter in the world coordinate system is the rotation matrix. With translation vector The merger is represented as:

[0052]

[0053] in, is the scale factor.

[0054] The BEV image generation module uses inverse perspective mapping to transform an image viewed from a tilted angle into a bird's-eye view, commonly used for geometric reconstruction of ground targets. For any pixel in the image... First, project the image back onto the camera coordinate system:

[0055]

[0056] in Assuming ground depth. Then, the camera coordinates are transformed to world coordinates:

[0057]

[0058] in, , From external parameter calibration. Subsequently, the system sets the top-down resolution in the BEV image, and... The coordinates are projected onto a planar image grid and then interpolated and resampled to generate a BEV image.

[0059] The Electronic Chart Analysis and Coordinate Unification module loads the electronic nautical chart (ENC) for the corresponding waters. This data contains multiple layers of features, each with spatial coordinate attributes. The original coordinates are latitude and longitude. To achieve image alignment, a projection transformation function is used to uniformly convert latitude and longitude to UTM coordinates.

[0060] This transformation ensures that spatial points in the BEV image are preserved. The points on the ENC layer coincide with the points projected onto the same plane.

[0061] After completing the BEV image generation and electronic chart coordinate unification, the layer fusion and spatial relationship recognition module of this invention further constructs a layer fusion mechanism to spatially compare the ship targets in the BEV image with the ship position information in the electronic chart, thereby identifying the AIS status and explicitly distinguishing and marking it in the visualization interface.

[0062]

[0063] in Represents the position of a ship on an electronic nautical chart. The positions of the ships in the BEV image have been converted to the UTM coordinate system in the previous step. This represents the search radius.

[0064] The core logic of this method is as follows: for each BEV image, the detected ship targets are determined in a unified coordinate system to determine whether they match the AIS ship information provided by the electronic chart. If there are ship targets visible in the BEV but not displayed in the ENC layer, they are marked in red in the image; otherwise, if the target can be identified and matched by AIS in the electronic chart, it is displayed in green.

[0065] Example:

[0066] To achieve the fusion of BEV image generation and electronic charts, this invention deploys a shore-based camera system at key waterway nodes. The system utilizes wide-angle high-definition cameras with a resolution of 1920×1080, installed at high points along the riverbank to cover the main navigable channel. The shooting angle is set at a 30° overhead view to ensure that the images fully represent the vessel and surrounding water surface information. An SSH remote tunnel is also provided for data transmission to the local machine.

[0067] Furthermore, regarding the transformation of the BEV (Browser-Electrical Vehicle) perspective in a waterway, this invention converts the oblique-view image of the waterway captured by the camera into a bird's-eye view through inverse perspective mapping, eliminating perspective effects and ensuring that objects such as ships and waterway boundaries maintain their real-world geometric proportions from a top-down perspective. The generation process of the BEV relies on precise camera parameters and a geometric transformation model.

[0068] This experiment calibrated the camera's intrinsic parameters using Zhang Zhengyou's checkerboard calibration method, obtaining camera intrinsic parameters including focal length. Pixels, principal point coordinates , corresponding to the center of a 1920×1080 resolution image.

[0069] Subsequently, using the Homography matrix, since the ground can be approximated as a plane, a homography transformation matrix from the world coordinate system to the image coordinate system is constructed using the camera's extrinsic parameters. This matrix consists of the camera's intrinsic parameters and camera height. This is determined jointly. By calculating the inverse of the Homography matrix, image pixels can be mapped to the BEV plane in the world coordinate system. After determining the Homography matrix, perspective transformation (warping) is performed on the input image, reprojecting the image from the oblique viewpoint to the top viewpoint.

[0070] Regarding the fusion display of electronic charts and BEV (Body Vehicle) data, to achieve spatial consistency between BEV charts and nautical charts, the system unifies the coordinate systems through a triple coordinate transformation. After converting the channel monitoring video to a BEV perspective, the ship's position information is mapped to the world coordinate system. The perceived results from the BEV image are then overlaid on the nautical chart as a transparent layer, achieving a unified display of static geographic information and dynamic perceived data. Detected targets in the BEV chart are mapped to their corresponding latitude and longitude positions, and their type, direction, and other key attributes are marked with icons and colors.

[0071] This invention discloses a BEV (Boat Vehicle) visual fusion method for intelligent monitoring of inland waterways, which fully combines the real-time nature of visual perception with the authority of nautical chart data, significantly improving the perception capability and decision-making efficiency of ship navigation systems in dynamic environments. Compared with existing technologies, this method has the following significant advantages:

[0072] This invention effectively overcomes the limitation of traditional electronic nautical charts in failing to perceive real-time dynamic targets (such as other vessels, buoys, and obstacles) by introducing bird's-eye view (BEV) generated based on inverse perspective mapping (IPM). The system can generate BEV views aligned with geographic coordinates in real time, enabling dynamic monitoring of the ship's surrounding environment and significantly improving the system's perception capabilities and response speed in complex waterways and maritime junctions.

[0073] This invention unifies the coordinate alignment of video images, radar data, AIS information, and vector layers in electronic nautical charts, solving the problem of spatial mapping between sensor images and nautical chart data. This is achieved by converting BEV images into data that are consistent with electronic nautical charts. Figure 1 The consistent latitude and longitude coordinate system ensures that all information can be seamlessly overlaid on the same display interface, improving the information integrity and scene consistency of the navigation system.

[0074] This invention overcomes the limitations of traditional electronic nautical charts, which can only display static information, and constructs a novel visualization system that integrates dynamic visual perception with chart data. This method can be widely applied in inland waterway supervision, remote monitoring of unmanned vessels, and intelligent navigation mark management, demonstrating good versatility, scalability, and promising industrial application prospects. By using IPM instead of traditional deep learning reconstruction methods, it reduces reliance on computing resources and offers stronger real-time performance and deployment flexibility. The method is computationally simple and has clearly defined parameters, allowing for rapid deployment on embedded platforms or intelligent navigation mark terminals, which is beneficial for building low-power, high-response intelligent navigation assistance systems.

[0075] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A BEV visual fusion method for intelligent monitoring of inland waterways, characterized by direct fusion of aerial images and nautical chart information under the same geographic reference and AIS status recognition, wherein... include: A shore-based camera monitoring system is deployed to collect video images of ship targets. The internal and external parameters of the cameras are calibrated, a transformation model from image pixel coordinates to real-world plane coordinates is established, and a bird's-eye view BEV image with spatial consistency is generated using inverse perspective mapping. Acquire channel elements and related data from electronic nautical charts, unify the chart data to a plane coordinate system consistent with the BEV image through coordinate projection transformation, and complete the spatial alignment of the chart data and the BEV image with shoreline and fixed navigation mark reference elements, and estimate camera extrinsic drift and registration error covariance online. The BEV image and the electronic chart AIS data are overlaid and rendered in a unified coordinate system. Kinematic prediction, clock alignment and time delay compensation are performed on the data from AIS to obtain the predicted position and covariance, so that the target detection result has a consistent spatial reference with the actual channel structure. Based on the layer fusion results, the ship targets in the BEV image are spatially compared with the ship position information provided by the electronic chart. The distance threshold judgment is performed on the ship target positions detected by BEV and the AIS predicted positions to determine whether they match each other, thereby identifying the corresponding AIS status. Matched and unmatched targets are explicitly distinguished and marked in the visualization interface.

2. The BEV visual fusion method for intelligent supervision of inland waterways according to claim 1, characterized in that: When pre-calibrating the camera's internal and external parameters: First, determine the mapping relationship between pixel coordinates in the video image and physical space coordinates, and then calibrate the camera's internal and external parameters. The internal parameter matrix... Represented as: in, For pixel focal length, The matrix is ​​obtained by using the checkerboard calibration method, with the main point positions as the reference points. The camera's extrinsic parameters in the world coordinate system are rotation matrices. With translation vector The merger is represented as: in, is the scale factor.

3. The BEV visual fusion method for intelligent supervision of inland waterways according to claim 1, characterized in that: Transforming an image from a tilted perspective into a bird's-eye view for ground target geometry reconstruction, for any pixel in the image... First, back-project to the camera coordinate system: in To assume ground depth, the camera coordinates are transformed to world coordinates: in, , From the extrinsic parameter calibration, during the BEV image generation process, the processing method sets the spatial resolution of the top view, and... The coordinates are projected onto a planar image grid and then interpolated and resampled to generate a BEV image.

4. The BEV visual fusion method for intelligent supervision of inland waterways according to claim 1, characterized in that: When unifying nautical chart data to a georeferenced frame consistent with BEV imagery: load the electronic nautical chart for the corresponding waters, which includes multiple layer features, each with spatial coordinate attributes, using latitude and longitude as the original coordinates. This means that a projection transformation function is used to uniformly convert latitude and longitude to UTM coordinates, thereby controlling image alignment. This algorithm is used to control spatial points in BEV images. The points of the ENC data layer coincide with the points projected onto the same plane.

5. The BEV visual fusion method for intelligent supervision of inland waterways according to claim 1, characterized in that: A layer fusion mechanism is used to spatially compare vessel targets in BEV images with vessel position information in electronic charts, thereby identifying AIS status and explicitly distinguishing and marking them in the visualization interface. in Represents the position of a ship on an electronic nautical chart. Represents the ship's position in the BEV image. Represents the search radius. For each BEV image frame, the system determines whether the detected ship target matches the AIS ship information provided by the electronic chart in a unified coordinate system. If the ship target exists in the BEV image but is not displayed in the ENC data layer, the ship target is marked in the image. Conversely, if the ship target can be identified and matched by AIS in the electronic chart, it is recorded as a successful match.

6. A BEV vision fusion system for intelligent monitoring of inland waterways, characterized in that... include: The perception data acquisition module collects video images of ship targets, as well as information on the identity, speed, and heading of nearby ships, by setting up camera equipment at key nodes in the waterway. The data preprocessing module acquires real-time channel information from electronic nautical charts and unifies the chart data to a geographic reference frame consistent with the BEV image through coordinate projection transformation. The BEV image generation module establishes a transformation model from image pixel coordinates to real-world coordinates, and uses the IPM algorithm to perform geometric back projection on the two-dimensional image, converting it into a BEV image with a bird's-eye view that has spatial consistency. The electronic chart analysis and coordinate unification module performs spatial alignment and overlay rendering of BEV images and ENC data. It directly draws chart elements on BEV images through layer overlay and controls the target detection results to have a consistent spatial reference with the actual channel structure. The layer fusion and spatial relationship recognition module uses a layer fusion mechanism to spatially compare the ship targets in the BEV image with the ship position information in the electronic chart, thereby identifying the AIS status and determining whether the ship target information in the BEV image matches the AIS ship information provided by the electronic chart, and then explicitly distinguishing and marking it in the visualization interface.

Images