A Bridge Collision Avoidance Early Warning Method Based on Fusion of Texture Enhancement Trajectory and Visual SLAM

Abstract

This invention provides a bridge collision avoidance early warning method that integrates texture-enhanced trajectory and visual SLAM, relating to the field of intelligent ship collision avoidance technology. The method includes: acquiring and preprocessing ship monitoring image data in severe weather; constructing a multi-scale texture enhancement module to enhance the preprocessed images; constructing a feature fusion model incorporating water texture attention weights to obtain ship feature images; performing ship 3D target detection based on adaptive ship attitude adjustment to obtain ship 3D bounding boxes; performing multi-target ship trajectory tracking and on-site deployment; based on the multi-target ship trajectories, performing real-time trajectory detection and dynamic matching with electronic fences to determine the real-time regional status of the ship and the trajectory risk within a preset future timeframe; and setting up a multi-level early warning mechanism, clarifying the triggering conditions for each early warning level, and determining the linkage response mode. This invention can improve the accuracy of ship target detection and trajectory recognition, reduce collision risks, and achieve shipping safety.

Description

Technical Field

[0001] This invention relates to the field of intelligent collision avoidance technology for ships, specifically to a bridge collision avoidance early warning method that fuses texture-enhanced trajectory with visual SLAM. Background Technology

[0002] With the booming development of global trade and the continuous increase in the number of ships, accurate tracking of their navigation trajectories is crucial for ensuring navigational safety and improving transportation efficiency. Bridges, as key infrastructure in waterway transportation, span rivers, lakes, and seas, connecting different regions; their safety is essential for the stable operation of the entire transportation network.

[0003] In recent years, ship-bridge collisions have occurred frequently, threatening navigation safety and bridge stability. Traditional methods for ship target detection, trajectory recognition, and collision warning have several shortcomings. Existing 3D target detection methods often use fixed-axis aligned bounding boxes, which cannot dynamically adjust to the ship's pitch, roll, and yaw movements in real time. This leads to mismatches between the bounding boxes and the actual ship outline, failing to provide reliable spatial coordinate data. In adverse weather conditions such as heavy rain, fog, and strong backlighting, the contrast between the ship and the water background is low, and there is significant noise from rain, snow, and fog. Traditional detection methods struggle to accurately extract ship areas and key features, resulting in low detection rates and difficulties in trajectory recognition. Traditional bridge-waterway electronic fences are often fixed-area divisions, unable to adapt to changes in the water environment such as water level fluctuations. Furthermore, traditional detection methods lack deep integration with real-time ship trajectories, making it difficult to predict ship deviation risks in advance, resulting in delayed warning responses or high false alarm rates. Once a ship-bridge collision occurs, it will not only cause serious damage to the bridge structure, leading to traffic disruption and huge economic losses, but may also trigger a series of catastrophic consequences such as ship sinking, casualties, and environmental pollution.

[0004] Therefore, improving the accuracy of ship target detection and trajectory recognition, shortening bridge collision warning time, and reducing collision risk are crucial for achieving shipping safety. Summary of the Invention

[0005] Traditional ship collision avoidance and early warning methods in existing technologies suffer from several technical problems: the marker frame cannot adapt to the real-time attitude of the ship, the detection accuracy is low in severe weather, the electronic fence is not adapted to changes in the aquatic environment, and the fusion of detection and trajectory is insufficient, leading to delayed warnings or high false alarm rates, making it difficult to ensure navigation safety in bridge-adjacent waters. This invention proposes a collision avoidance and early warning method for bridge-adjacent waters based on the fusion of texture-enhanced trajectory and visual SLAM to ensure the navigation safety of ships in bridge-adjacent waters.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A bridge collision avoidance warning method that fuses texture-enhanced trajectories with visual SLAM, the method comprising:

[0008] Collect and preprocess ship monitoring image data in severe weather;

[0009] A multi-scale texture enhancement module is constructed to perform image enhancement processing on the preprocessed image;

[0010] Based on the enhanced image, a feature fusion model incorporating water texture attention weights is constructed to obtain ship feature images;

[0011] Based on the ship feature image, perform three-dimensional target detection of the ship to obtain the three-dimensional bounding box of the ship;

[0012] Based on the three-dimensional ship identification frame, perform multi-target ship trajectory tracking and on-site deployment;

[0013] Based on the multi-target trajectory of the vessel, real-time trajectory detection of the vessel is performed and dynamic matching with the electronic fence is carried out to determine the real-time regional status of the vessel and the trajectory risk within a preset time period in the future.

[0014] Based on the regional status assessment results and trajectory risk prediction results, a multi-level early warning mechanism is set up, the triggering conditions for each early warning level are clarified, and the corresponding linkage response methods are determined.

[0015] Compared with the prior art, the beneficial effects of the present invention are:

[0016] 1. This invention effectively resists interference from sea conditions and lighting by detecting key feature points, constructing an attitude parameter calculation model, and designing an attitude smoothing filtering algorithm. It achieves accuracy and stability in ship 3D detection under complex scenarios, ensuring a high degree of matching between the 3D bounding box and the actual attitude of the ship.

[0017] 2. The innovative multi-scale texture enhancement and feature fusion technology of this invention, through adaptive histogram equalization and the introduction of water texture attention weights, can enhance the texture difference between ships and water background, effectively filter extreme weather noise, and significantly improve the ship target detection rate and trajectory tracking accuracy under severe weather conditions.

[0018] 3. This invention introduces Simultaneous Localization and Mapping (SLAM) technology into bridge collision avoidance scenarios. It constructs a high-precision 3D waterway map using a mobile vision device and combines it with historical vessel trajectory data to form a dynamic electronic fence. This achieves dynamic matching between the electronic fence and the real-time vessel trajectory, solving the compatibility problem of traditional fixed fences and ensuring timely and targeted early warnings.

[0019] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0020] Figure 1 This is a flowchart of a bridge collision avoidance and early warning method based on the fusion of texture enhancement trajectory and visual SLAM according to the present invention.

[0021] Figure 2 This is a framework diagram for detecting and screening key feature points of ships according to the present invention;

[0022] Figure 3 This is a dynamic adjustment framework diagram of a three-dimensional marker box based on attitude parameters according to the present invention;

[0023] Figure 4 This is a structural diagram of the water texture attention module according to the present invention;

[0024] Figure 5 This is a visual SLAM system framework diagram according to the present invention;

[0025] Figure 6 This is a framework diagram of dynamic electronic fence matching for real-time ship trajectory according to the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings, so as to more clearly understand the purpose, features and advantages of this invention. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of this invention, but are only for illustrating the essential spirit of the technical solutions of this invention. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0027] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0028] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0029] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to mean “and / or” unless otherwise expressly stated herein.

[0030] In the following description, in order to clearly demonstrate the structure and working method of the present invention, a number of directional terms will be used. However, terms such as "front", "back", "left", "right", "outside", "inside", "outward", "inward", "up", and "down" should be understood as convenient terms and not as limiting terms.

[0031] The implementation details of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0032] The purpose of this invention is to address the shortcomings of the aforementioned background technology by providing a bridge-water collision avoidance and early warning method based on the fusion of texture-enhanced trajectory and visual SLAM, such as... Figure 1 As shown.

[0033] This invention addresses the challenges of low contrast between ships and the water background, making trajectory recognition difficult under adverse weather conditions such as heavy rain, dense fog, and strong backlighting. It proposes a ship trajectory recognition method based on enhanced water texture features. Furthermore, to solve the problem that traditional fixed electronic fences cannot adapt to changes in the water environment (such as water level fluctuations), a dynamic electronic fence early warning method for bridge waterways, integrating texture-enhanced trajectories and visual SLAM, is proposed, as detailed below:

[0034] Step 1: Collect and preprocess ship monitoring image data in severe weather;

[0035] Specifically, 1) High-definition industrial cameras deployed above the bridge navigation area are used to collect image data of ships under different adverse weather conditions (heavy rain, heavy fog, strong backlight, light snow), covering different water environments such as inland rivers and coastal areas. Ship types include cargo ships, passenger ships, and fishing boats. The driving status covers constant speed, turning, acceleration, and deceleration to ensure the diversity and representativeness of the data samples. At the same time, weather parameters (visibility, rainfall, light intensity) are recorded at the time of collection to provide label basis for subsequent model adaptation.

[0036] 2) The collected raw image data is preprocessed. First, image cropping technology is used to remove invalid edge areas (such as buildings on the shore) and retain the core water monitoring area. Then, image size is standardized by scaling all images to a uniform pixel size to adapt to the input specifications of the subsequent model. Finally, data augmentation techniques such as random flipping, rotation (±10°), and brightness fine-tuning are used to expand the dataset and avoid model overfitting.

[0037] Step 2: Construct a multi-scale texture enhancement module to perform image enhancement processing;

[0038] A multi-scale texture enhancement module is constructed, which includes an adaptive histogram equalization unit, an edge-preserving filtering unit, and a multi-scale feature fusion unit. The three units are connected in series to achieve progressive enhancement from noise removal to texture enhancement.

[0039] Specifically, 1) To address the issue of low image contrast under adverse weather conditions, an Adaptive Histogram Equalization (CLAHE) algorithm is used to enhance texture differences in the preprocessed image. The CLAHE algorithm's clipping limit parameter is set to 2.0, dividing the image into rectangular pixel blocks. Histogram equalization is then performed on each pixel block individually to avoid the loss of local details caused by global equalization. The algorithm focuses on enhancing the grayscale difference between the ship's outline and the water background, resulting in clearer ship edge textures.

[0040] 2) Considering the significant noise in images during heavy rain and light snow, a guided filter is used as an edge-preserving filtering algorithm to denoise the equalized image. The guided filter's radius parameter is set to 6, and its standard deviation parameter to 0.01. This effectively filters rain and snow noise and fog blurring interference while preserving key texture features such as ship edges and railings to the greatest extent possible, thus solving the edge blurring problem caused by traditional filtering algorithms.

[0041] 3) The denoised image is input into three convolutional kernels of different scales (3×3, 5×5, 7×7) for feature extraction to obtain multi-scale texture feature maps; the channel dimension of each scale feature map is compressed by a 1×1 convolutional kernel to unify the feature dimension to 256 dimensions; finally, the feature is fused by weighted summation, and the weight coefficients are obtained by adaptive learning through training data to achieve complementarity of ship texture features at different scales and improve the stability of ship target recognition in complex scenes.

[0042] Step 3: Construct a feature fusion model that incorporates water texture attention weights;

[0043] Specifically, such as Figure 4 As shown, 1) the improved ResNet-50 is selected as the basic feature extraction network, and the enhanced image obtained in step 2 is used. Inputting the network, multi-level feature maps (C1-C5) are extracted through forward propagation of residual units: shallow feature maps (C1-C2) focus on detailed features such as ship edges and textures, while deep feature maps (C3-C5) focus on the overall outline and semantic features of the ship. The expression for extracting multi-level features through forward propagation of residual units is as follows:

[0044]

[0045] in, For residual mapping (including convolution, BN, and ReLU operations), x is the input feature of the residual unit, and y is the output feature of the residual unit.

[0046] 2) Construct a water texture attention module. Input the deep feature map C5 and obtain the channel-level feature vector through global average pooling. Use a 2-layer fully connected network (128 hidden layer neurons) to perform dimensionality transformation and non-linear mapping on the feature vector and output an attention weight vector equal to the number of channels. Multiply this weight vector with the shallow feature map C2 channel by channel to strengthen the feature weight at the boundary between the ship and the water area, suppress the feature response of the pure water background area, and make the model focus on the area where the ship target is located.

[0047] Specifically, the input deep feature map C5 is used to obtain the channel feature vector v through global average pooling (GAP). R 2048 :

[0048]

[0049] In the formula, Let H and W be the feature values ​​of the c-th channel (c=1,...,2048) of C5 at (x,y), and H and W be the height and width of C5, respectively. The feature vector is transformed through a two-layer fully connected network (FC) to output attention weights. R 2048 :

[0050]

[0051] in, Mapping 2048 dimensions to 128 dimensions Map back to 2048 dimensions. For the Sigmoid function;

[0052] Weight Channel-wise weighting with shallow feature map C2 enhances boundary features:

[0053]

[0054] In the formula, For weight The size is adaptively adjusted (by interpolating to match the C2 size). This is for element-wise multiplication.

[0055] 3) The attention-weighted shallow feature map Cross-layer fusion is performed with the deep feature map C5, and upsampling technology is used to adjust the size of the deep feature map C5 to match that of the shallow feature map. The feature fusion is achieved by adding elements together; the fused feature map is then input into a 1×1 convolution kernel for channel integration to obtain the final ship feature image, which takes into account both detailed texture and global semantic information.

[0056] Specifically, the deep feature map C5 is upsampled to match its instance size. Subsequently with Combine according to the following formula:

[0057]

[0058] in, For bilinear interpolation upsampling, Used to integrate channels up to 128 dimensions.

[0059] Step 4: Perform three-dimensional target detection of the ship;

[0060] Specifically, 1) the final ship feature image obtained in step 3 is preprocessed again to eliminate interference from environmental noise such as sea state and lighting conditions on subsequent detection. This includes:

[0061] A) The final ship feature image is then denoised using an improved bilateral filtering algorithm. This algorithm considers the similarity between the spatial domain and the grayscale domain, effectively removing wave reflections and fog noise while preserving ship edge details. The filtering formula is as follows:

[0062]

[0063] Where g(i,j) is the pixel value of the denoised image at coordinate (i,j), and f(k,l) is the pixel value of the image at coordinate (k,l). For spatial domain standard deviation, C(i,j) is the standard deviation of the grayscale value range, and C(i,j) is the normalization coefficient, ensuring that the output pixel values ​​are within a reasonable range.

[0064] B) For scenes with uneven lighting, such as backlighting and strong light, an adaptive histogram equalization (CLAHE) algorithm is used to adjust image contrast and enhance the distinction between the ship and the background. This is achieved by dividing the image into several sub-blocks, performing histogram equalization on each sub-block, and setting a contrast threshold. To avoid excessive noise amplification, the formula is as follows:

[0065]

[0066] in, This is the histogram of the equalized sub-blocks. This is the original sub-block histogram. This represents the maximum frequency in the histogram.

[0067] C) Use the YOLOv8 lightweight network to perform preliminary ship detection on the processed image and output the ship's two-dimensional bounding box. (in The coordinates of the top left corner (The coordinates are the bottom right corner). Based on this bounding box, expand outward by 10%~20% to form the ROI region, eliminate background interference, and focus on the main body area of ​​the ship for subsequent key point detection.

[0068] 2) A ship key feature point detection model is constructed based on an improved HRNet (high-resolution network) to accurately locate the core feature points related to ship attitude, such as... Figure 2 As shown, the specific operation is as follows:

[0069] A) Based on the characteristics of the ship's structure, define 8 core attitude-related feature points of the ship (the port end of the bow). Starboard end of the bow port end of the stern Starboard end of the stern Chimney top point Midpoint of the left edge of the deck Midpoint of the right edge of the deck Lowest point of the ship's side (This covers the key related parts of the ship's pitch, roll, and yaw attitudes.)

[0070] B) Based on the original HRNet, a focus mechanism (CBAM) is introduced to enhance the feature response of feature point regions. Simultaneously, Focal Loss is used to address the uneven distribution of ship feature points (some feature points, such as the top of the chimney, have a small proportion). The loss function formula is as follows:

[0071]

[0072] in, Let be the weight coefficient of the i-th feature point. To predict the coordinates of feature points, To label the coordinates of the actual feature points.

[0073] C) Input the ROI region extracted in step 1) into the trained improved HRNet model, and output the initial coordinates of the 8 core feature points. (Pixel coordinate system); The RANSAC algorithm is used to filter the initial feature points, removing outliers caused by occlusion and noise, and retaining the interior point set. The selection criteria are as follows: calculate the distance from each feature point to the main outline of the ship, and when the distance is greater than the set threshold δ (empirical value is 5~8 pixels), it is judged as an abnormal point and removed.

[0074] 3) Based on the selected set of key feature points, construct an attitude parameter calculation model to accurately estimate the ship's pitch angle. Roll angle Yaw angle The scale parameters (length L, width W, height H) are as follows:

[0075] A) Obtain the intrinsic parameter matrix through camera calibration Convert the pixel coordinates of feature points into three-dimensional coordinates in the camera coordinate system. The conversion formula is as follows:

[0076]

[0077] in, The pixel coordinates of the feature point , , The focal lengths are in the x and y directions. The coordinates of the main point.

[0078] B) Attitude angle estimation:

[0079] (1) Pitch angle Define the line connecting the midpoints of the deck edges. The horizontal angle is calculated using the vector dot product:

[0080]

[0081] When the bow rises It is positive when it sinks, and negative when it sinks.

[0082] (2) Roll angle Define the lowest point of the ship's side. Line connecting the midpoints of the deck edges The angle corresponding to the ratio of the vertical distance to the standard width of the ship is the roll angle, and the formula is as follows:

[0083]

[0084] in, for arrive vertical distance, The standard width of the vessel (a value obtained by matching from the vessel type knowledge base).

[0085] (3) Yaw angle Define the bow direction vector. The angle between the camera and the X-axis of the camera coordinate system is given by the following formula:

[0086]

[0087] in, Let X be the component of the ship's longitudinal axis vector on the X-axis of the camera coordinate system. This represents the component of the ship's longitudinal axis vector on the Z-axis of the camera coordinate system.

[0088] C) Calculate the real-time dimensions of the ship based on the distance between feature points, using the following formula:

[0089]

[0090]

[0091]

[0092] in, It represents the length of the ship (the distance from the port end of the bow to the port end of the stern). Represents the width (distance between the left and right ends of the bow). It represents the height (the distance from the top of the chimney to the lowest point of the ship's side).

[0093] 4) The attitude angle estimated in step 3) Based on the scale parameters (L, W, H), a three-dimensional marker frame dynamic adjustment model is constructed to achieve precise matching between the marker frame and the actual attitude of the ship, as detailed below:

[0094] A) Based on the ship's center point Construct an initial axis-aligned 3D bounding box with the origin as the reference point. As the origin, such as Figure 3 As shown, the coordinates of its 8 vertices are:

[0095] k=1,2,...,8.

[0096] B) Based on the estimated pitch angle Roll angle Yaw angle Construct a three-dimensional rotation matrix This achieves the initial alignment of the bounding box. The rotation matrix is ​​constructed using the ZYX Euler angles order, as shown in the following formula:

[0097]

[0098] The rotation matrices for each axis are as follows:

[0099]

[0100] C) The coordinates of the 8 vertices of the initial frame are obtained through a rotation matrix. Rotate the vertex to obtain the coordinates after attitude adjustment. This ultimately forms a three-dimensional identification frame that matches the ship's real-time attitude. .

[0101] 5) To eliminate ship attitude jitter caused by sea wave turbulence, an adaptive weighted smoothing filter algorithm is designed to optimize the adjusted 3D bounding box, ensuring its stability, as follows:

[0102] A) A sliding window mechanism is used, selecting the vertex coordinates of the 3D bounding boxes in the current frame and the previous N frames (N is 5~10, adjusted according to real-time requirements) as input, and achieving smoothing through adaptive weight allocation. The weight calculation formula is as follows:

[0103]

[0104] in, Let be the weight of frame t. The pose error of the bounding box between frame t and the previous frame is calculated using the mean square error of the vertex coordinates. The attenuation coefficient is the factor that determines the weight of the frame. The smaller the attitude error, the greater the weight, thus increasing the influence of the stabilized frame.

[0105] B) Perform a weighted sum of the vertex coordinates of the identifier boxes in each frame within the sliding window to obtain the optimized vertex coordinates. The final output is a stable and accurate three-dimensional ship identification frame. .

[0106] C) Detection result optimization: Set the confidence threshold to 0.5, perform non-maximum suppression (NMS) on the output attitude-adaptive 3D bounding boxes to remove overlapping boxes, and obtain accurate 3D target detection results of ships in a single frame image.

[0107] Step 5: Multi-target tracking and on-site deployment of ships;

[0108] Multi-target trajectory association: Using Kalman filtering and the Hungarian algorithm, based on the center coordinates of the attitude-adaptive 3D bounding box output in step 4, the position and motion state (velocity, acceleration) of the ship in the current frame are first predicted by Kalman filtering; then, the intersection-over-union ratio (IoU) between the predicted position and the detected position in the current frame is calculated as a similarity metric to achieve target ID matching. Finally, the optimal matching of ship targets between consecutive frames is achieved using the Hungarian algorithm, assigning a unique ID number to each ship.

[0109] Trajectory fitting: Based on the center coordinates of the ship in consecutive frames, a polynomial fitting algorithm is used to generate the ship's trajectory.

[0110] On-site deployment: The fused model is deployed to the edge computing device (NVIDIA Jetson Xavier NX) at the bridge site to receive image data collected by the monitoring camera in real time. Through the process of steps 2-4, real-time identification of ship trajectories under severe weather conditions is achieved.

[0111] Step 6: Perform real-time vessel trajectory detection and dynamic fence matching;

[0112] Specifically, 1) Based on the multi-target trajectory results of ships output in step 5—including the unique ID of each ship and the time series sequence of real-time trajectory 3D coordinates generated by polynomial fitting—a 3D map of the bridge waterway is constructed. The trajectory coordinates are accurately mapped from the camera coordinate system to the local coordinate system of the map through the pre-calibrated camera extrinsic parameters, ensuring the spatial consistency between the trajectory and the 3D map. The Kalman filter algorithm is used to further smooth and optimize the mapped ship 3D coordinate data, accurately predict the ship's position and motion state (velocity, acceleration) at the next moment, and finally obtain the real-time ship driving trajectory that is completely adapted to the 3D map.

[0113] The construction of a 3D map of the bridge and the waterway specifically includes:

[0114] A) Conduct visual data acquisition and preprocessing for bridge waterways, including:

[0115] a) A mobile UAV platform equipped with a high-resolution RGB-D camera (1920×1080 resolution, depth measurement range 0.5-10m) was used to conduct panoramic scanning and data collection of the water area surrounding the bridge. The data collection path covered 500m of water area upstream and downstream of the bridge. The route was planned using a combination of "lateral scanning + longitudinal cruising". The lateral scanning interval was set at 5m, and the longitudinal cruising route was set along the centerline of the waterway and one of the two sides. During the data collection, the camera pose information (acquired through the camera's built-in IMU sensor) and environmental parameters (water level, wind speed, and water flow velocity, acquired in real time through portable sensors) were recorded simultaneously.

[0116] The acquired RGB and depth images are synchronized and aligned. Image distortion correction is performed based on the camera intrinsic parameter matrix (obtained in advance using the Zhang Zhengyou calibration method). The correction formula is as follows:

[0117]

[0118] in, These are the coordinates of the distorted pixels. For the corrected coordinates, Let these be the coordinates of the camera's principal point. This refers to the camera's focal length.

[0119] b) Noise removal is performed on the depth image using a bilateral filtering algorithm. The filter kernel size is set to 5×5, and the spatial standard deviation and gray value standard deviation are both set to 1.0. Finally, the RGB image, depth image, and IMU data are associated and bound according to the acquisition timestamp to form a standardized dataset.

[0120] B) Constructing a 3D map of the bridge and waterway based on a visual SLAM framework;

[0121] like Figure 5 As shown, a feature-point-based visual SLAM framework is adopted, which mainly includes four modules: front-end visual odometry, back-end optimization, loop closure detection, and 3D map construction. The overall process is "image feature extraction - feature matching - pose estimation - back-end optimization - map generation", specifically including:

[0122] a) The SIFT algorithm is used to extract feature points from adjacent RGB images and construct feature descriptors. Feature points are matched using the FLANN matcher and mismatched points are removed using the RANSAC algorithm (the number of iterations is set to 1000 and the inlier threshold is set to 2.0 pixels). Based on the matched feature point pairs, the coordinates of three-dimensional points are calculated using depth image data. The relative poses of the cameras in adjacent frames (rotation matrix R, translation vector t) are solved using the PnP algorithm, and the feature points are tracked using optical flow to optimize the continuity of inter-frame pose estimation, thereby obtaining the initial trajectory of the camera and the sparse three-dimensional point cloud.

[0123] b) Input the initial pose and sparse point cloud obtained from the front end into the graph optimization-based backend system (using the general graph optimization g2o framework) to construct a pose graph (nodes represent camera poses, and edges represent inter-frame relative pose constraints); simultaneously, start the loop closure detection module, use the bag-of-words (BoW) model to calculate the similarity between the current frame and historical frames, and determine potential loop closures when the similarity is greater than 0.7, add loop closure constraints by calculating the relative poses between the loop frames; based on the pose graph and various constraints, use the LM algorithm to optimize the pose graph, minimize the global reprojection error, and obtain high-precision camera trajectories and optimized sparse 3D point clouds.

[0124] c) Based on the optimized camera trajectory and sparse point cloud, the Poisson reconstruction algorithm is used to densify the sparse point cloud to generate a dense 3D point cloud map of the bridge water area, which includes key topographic features such as bridge piers, channel boundaries, and underwater shoals. To adapt to changes in the water environment, a dynamic map update mechanism is established: every 7 days, mobile vision equipment is used to re-collect key areas of the water area (around bridge piers and channel turning points), and the newly collected point cloud is registered and fused with the original map using the ICP algorithm to update the 3D data of key feature areas such as water level changes and shoal displacement in the map.

[0125] 2) such as Figure 6As shown, the electronic fence parameters (combined with parameters automatically adjusted based on the current water level) are retrieved from the database in real time to construct a three-dimensional spatial model of the fence; for each coordinate point (x) on the real-time trajectory of the vessel... t ,y t ,z t (t is a timestamp), the point-to-polygon distance algorithm is used to determine the region to which the point belongs: the shortest distance d from the point to the boundary of the safe zone is calculated. If d ≥ 0 and the point is inside the safe zone, it is considered to be in a safe state; if d < 0 and d ≤ 15m (the width of the warning zone), it is considered to be in a warning state; if d > 15m or the point enters the boundary of the danger zone, it is considered to be in a danger state. At the same time, based on the ship's current speed v and direction of travel θ, the set of trajectory points in the next 3 seconds is predicted. If the predicted trajectory points enter the warning or danger zone, a warning is triggered in advance.

[0126] The electronic fence delineation process in the database includes:

[0127] A) Collect nearly one year of ship AIS trajectory data and machine vision detection trajectory data in the bridge waterway, and filter valid trajectories (excluding those with a dwell time exceeding one hour or abnormal speed); convert the geographic coordinates (latitude and longitude) of the trajectory data into a coordinate transformation algorithm (such as Gauss-Kruger projection) that corresponds to three-dimensional coordinates. Figure 1 A local coordinate system is established; the Douglas-Peucker algorithm is used to simplify the trajectory, retaining key turning points and driving nodes to reduce data redundancy.

[0128] B) Based on the preprocessed historical ship trajectory data, the DBSCAN clustering algorithm is used to perform cluster analysis on the waterway navigation area. The clustering radius ε=10m and the minimum number of cluster points MinPts=50 are set. The high-frequency ship navigation area (i.e., the core waterway area) is obtained through clustering. Combined with the terrain obstacles (such as bridge piers and shoals) in the 3D map, the boundary of the dangerous area where ships are prohibited from entering is determined. The area between the core waterway area and the dangerous area is set as the buffer warning area.

[0129] C) Mark the three-dimensional boundaries of the safe zone, warning zone, and danger zone in the three-dimensional map to form a three-dimensional electronic fence. The definitions of each zone are as follows: (1) Safe zone: the core waterway area obtained by clustering, with a minimum distance of 20m from the boundary to obstacles on both sides of the waterway (such as the shore and shoals); (2) Warning zone: the buffer zone between the safe zone and the danger zone, with a width of 15m, and dynamically adjusted according to the water level; (3) Danger zone: the area within 10m around the bridge pier, the area where the top elevation of the shoal is higher than the minimum navigable water level, and other prohibited navigation areas. At the same time, establish a dynamic adjustment model for the fence, and adjust the vertical coordinates of the fence according to the real-time water level height h. The adjustment formula is:

[0130]

[0131] in To adjust the vertical coordinates of the fence, Reference water level The vertical coordinates of the fence below.

[0132] D) Store the three-dimensional boundary parameters (vertices coordinate set) of the divided dynamic electronic fence and the adjustment model in the database, and associate and bind them with the three-dimensional map to support real-time calling and updating of fence parameters according to the needs of water level, waterway adjustment, etc.

[0133] Step 7: Set up multi-level early warnings and determine the triggering conditions and linkage response methods.

[0134] Specifically, 1) A three-level early warning mechanism is set up, as follows: A) Level 1 warning (reminder level): The vessel enters the warning area, or is predicted to enter the warning area in 3 seconds. The triggering condition is that the distance between the trajectory point and the boundary of the safe area is 0 < d ≤ 10m; B) Level 2 warning (alert level): The vessel is less than 5m from the boundary of the danger area, or is predicted to enter the danger area in 3 seconds. The triggering condition is that 10m < d ≤ 15m or the predicted trajectory touches the boundary of the danger area; C) Level 3 warning (emergency level): The vessel enters the danger area, or is predicted to collide with obstacles such as bridge piers in 1 second. The triggering condition is that d > 15m or the vessel has already entered the danger area.

[0135] 2) Different warning levels correspond to different response methods: A) Level 1 warning: Low-frequency warning sounds are emitted through the audible and visual alarms on both sides of the bridge, and a reminder message is broadcast on the ship's VHF communication channel at the same time; B) Level 2 warning: The volume and frequency of the audible and visual alarms are increased, a warning signal is sent to the bridge control center, and the ship's ID, location and travel trajectory are displayed; C) Level 3 warning: The highest level audible and visual alarm is activated, and an emergency navigation prohibition order is sent in conjunction with the waterway traffic control department. At the same time, a mandatory avoidance signal is sent to the target ship through the ship's AIS system. If necessary, the bridge anti-collision device (anti-collision buffer pad) is activated to prepare for protection.

[0136] 3) Automatically record the time, vessel information, warning level, triggering reason and response measures for each warning and store them in the database; regularly review and analyze the warning data, optimize the area division parameters of the electronic fence and the warning triggering threshold, and improve the accuracy and reliability of the warning system.

[0137] This invention proposes a bridge collision avoidance early warning method that integrates texture-enhanced trajectory and visual SLAM. Addressing the problems of traditional methods, such as the inability of bounding boxes to adapt to real-time ship attitude, low detection accuracy in adverse weather conditions, the inability of electronic fences to adapt to changes in the aquatic environment, and insufficient fusion of detection and trajectory, this method acquires and preprocesses ship monitoring images in adverse weather conditions. It constructs a multi-scale texture enhancement module and a feature fusion model incorporating water texture attention weights to perform 3D ship target detection. Combined with Kalman filtering and the Hungarian algorithm, it performs multi-target trajectory tracking and on-site deployment. Using visual SLAM technology, it constructs a high-precision 3D water map and forms a dynamic electronic fence, achieving dynamic matching between the ship's real-time trajectory and the fence. Simultaneously, it sets up three levels of early warning and corresponding linkage response modes. This method effectively resists interference from sea conditions and lighting, improves the ship target detection rate and trajectory tracking accuracy in adverse weather conditions, solves the adaptability problem of traditional fixed fences, ensures the timeliness, relevance, and accuracy of early warnings, reduces the risk of ship-bridge collisions, and ensures navigation safety in bridge-adjacent waters.

[0138] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention. Such modifications or substitutions should all fall within the scope of the invention, or any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. A bridge collision avoidance early warning method that fuses texture-enhanced trajectories with visual SLAM, characterized in that, The method includes: The system collects and preprocesses ship monitoring image data during severe weather; constructs a multi-scale texture enhancement module to enhance the preprocessed images; based on the enhanced images, it constructs a feature fusion model incorporating water texture attention weights to obtain ship feature images; based on the ship feature images, it performs 3D target detection of ships to obtain 3D bounding boxes; based on the 3D bounding boxes, it performs multi-target trajectory tracking and on-site deployment of ships; based on the multi-target trajectories, it performs real-time trajectory detection and dynamic matching with electronic fences to determine the real-time regional status of ships and trajectory risks within a preset time period; based on the regional status determination results and trajectory risk prediction results, it sets up a multi-level early warning mechanism, clarifies the triggering conditions for each early warning level, and determines the corresponding linkage response methods. Specifically, the process of constructing a feature fusion model that incorporates water texture attention weights based on the enhanced image to obtain a ship feature image includes: 1) An improved ResNet-50 was selected as the basic feature extraction network to enhance the image. Inputting this network, multi-level feature maps C1-C5 are extracted through forward propagation of residual units: shallow feature maps C1-C2 focus on ship detail features, while deep feature maps C3-C5 focus on the overall ship outline and semantic features; the multi-level feature expressions extracted through forward propagation of residual units are as follows: In the formula, For residual mapping, x The input features of the residual unit, y The output characteristics of the residual unit; 2) Construct a water texture attention module. Input the deep feature map C5 and obtain channel-level feature vectors through global average pooling. Use a 2-layer fully connected network to perform dimensionality transformation and non-linear mapping on the feature vectors, outputting an attention weight vector equal to the number of channels. Multiply this weight vector with the shallow feature map C2 channel by channel to strengthen the feature weights at the boundary between the ship and the water area, suppress the feature response of the pure water background area, and make the model focus on the area where the ship target is located. Among them, the input deep feature map C5 obtains the channel feature vector v through global average pooling. R 2048 : In the formula, Let be the eigenvalue of the c-th channel of C5 at (x,y), where c=1,...,2048; H, W The height and width of C5 are given; the feature vector is transformed through a 2-layer fully connected network, and the attention weights are output. R 2048 : in, Mapping 2048 dimensions to 128 dimensions Map back to 2048 dimensions. For the Sigmoid function; Weight Channel-wise weighting with shallow feature map C2 enhances boundary features: In the formula, For weight The size adapts to self-adjustment. For element-wise multiplication; 3) The attention-weighted shallow feature map Cross-layer fusion is performed with the deep feature map C5, and upsampling technology is used to adjust the size of the deep feature map C5 to match that of the shallow feature map. Consistent feature fusion is achieved through element-wise addition; the fused feature map is then input into a 1×1 convolution kernel for channel integration to obtain the final ship feature image; in this process, the deep feature map C5 is upsampled to match its instance size. Subsequently with Combine according to the following formula: In the formula, For bilinear interpolation upsampling, Used to integrate channels up to 128 dimensions.

2. The method according to claim 1, characterized in that, The process of detecting three-dimensional targets of ships based on ship feature images to obtain three-dimensional ship bounding boxes specifically includes: The obtained ship feature images are preprocessed again to eliminate the interference of environmental noise on subsequent detection, including: A) The ship feature image is denoised using an improved bilateral filtering algorithm. This algorithm considers the similarity between the spatial domain and the grayscale domain, effectively removing wave reflections and fog noise while preserving ship edge details. The filtering formula is as follows: Where g(i,j) is the pixel value of the denoised image at coordinate (i,j), and f(k,l) is the pixel value of the image at coordinate (k,l). For spatial domain standard deviation, C(i,j) is the standard deviation of the grayscale range, and C(i,j) is the normalization coefficient, ensuring that the output pixel values ​​are within a reasonable range. B) For scenes with uneven lighting, an adaptive histogram equalization algorithm is used to adjust image contrast and enhance the distinction between the ship and the background. This is achieved by dividing the image into several sub-blocks, performing histogram equalization on each sub-block, and setting a contrast threshold. To avoid excessive noise amplification, the formula is as follows: in, This is the histogram of the equalized sub-blocks. This is the original sub-block histogram. The maximum frequency in the histogram; C) Use the YOLOv8 lightweight network to perform preliminary ship detection on the processed image and output the ship's two-dimensional bounding box. ,in The coordinates of the top left corner Using the coordinates of the lower right corner as a base, expand outward by 10%~20% to form the ROI region, eliminate background interference, and focus on the main body area of ​​the ship for subsequent key point detection.

3. The method according to claim 2, characterized in that, The process of detecting three-dimensional targets of ships based on ship feature images to obtain three-dimensional bounding boxes of ships also includes: constructing a ship key feature point detection model based on an improved HRNet to locate the core feature points associated with the ship's attitude. The specific operations are as follows: A) Based on the characteristics of the ship's structure, define 8 core attitude-related feature points of the ship, namely the port bow endpoint. Starboard end of the bow port end of the stern Starboard end of the stern Chimney top point Midpoint of the left edge of the deck Midpoint of the right edge of the deck Lowest point of the ship's side It covers the key related parts of the ship's pitch, roll, and yaw attitudes; B) Based on the original HRNet, an attention mechanism is introduced to enhance the feature response of the feature point region. At the same time, FocalLoss is used to solve the problem of uneven distribution of ship feature points. The loss function formula is as follows: In the formula, Let be the weight coefficient of the i-th feature point. To predict the coordinates of feature points, To label the coordinates of the actual feature points; C) Input the extracted ROI region into the trained improved HRNet model, and output the initial coordinates of the 8 core feature points. The RANSAC algorithm is used to filter the initial feature points, removing outliers and retaining the interior point set. The screening criteria are as follows: calculate the distance from each feature point to the main outline of the ship, and when the distance is greater than the set threshold δ, it is judged as an abnormal point and removed.

4. The method according to claim 3, characterized in that, The process of detecting three-dimensional targets of a ship based on its feature images to obtain a three-dimensional bounding box for the ship also includes: constructing an attitude parameter calculation model based on a set of in-points of selected key feature points to estimate the ship's pitch angle. Roll angle Yaw angle And the scale parameters, as detailed below: A) Obtain the intrinsic parameter matrix through camera calibration Convert the pixel coordinates of feature points into three-dimensional coordinates in the camera coordinate system. The conversion formula is as follows: in, The pixel coordinates of the feature point , , The focal lengths are in the x and y directions. Principal point coordinates; B) Attitude angle estimation: (1) Pitch angle Define the line connecting the midpoints of the deck edges. The horizontal angle is calculated using the vector dot product: When the bow rises When it is positive, it becomes negative when it sinks; (2) Roll angle Define the lowest point of the ship's side. Line connecting the midpoints of the deck edges The angle corresponding to the ratio of the vertical distance to the standard width of the ship is the roll angle, and the formula is as follows: in, for arrive vertical distance, Standard beam for ships; (3) Yaw angle Define the bow direction vector. The angle between the camera and the X-axis of the camera coordinate system is given by the following formula: in, Let X be the component of the ship's longitudinal axis vector on the X-axis of the camera coordinate system. This represents the component of the ship's longitudinal axis vector on the Z-axis of the camera coordinate system. C) Calculate the real-time dimensions of the ship based on the distance between feature points, using the following formula: in, Represents the length of the ship. Represents width, Represents altitude.

5. The method according to claim 4, characterized in that, The process of detecting three-dimensional targets of a ship based on its feature images to obtain a three-dimensional bounding box for the ship also includes: using estimated attitude angles and scale parameters... L , W , H Based on this, a three-dimensional dynamically adjustable model of the marker frame is constructed to achieve matching between the marker frame and the actual attitude of the ship, as detailed below: A) Based on the ship's center point Construct an initial axis-aligned 3D bounding box with the origin as the reference point. Let the origin be the coordinates of its 8 vertices: ,k=1,2,...,8； B) Based on the estimated pitch angle Roll angle Yaw angle Construct a three-dimensional rotation matrix The three-dimensional rotation matrix is ​​constructed using the ZYX Euler angles order, as shown in the following formula: The rotation matrices for each axis are as follows: C) The coordinates of the 8 vertices of the initial frame are obtained through a rotation matrix. Rotate the vertex to obtain the coordinates after attitude adjustment. This ultimately forms a three-dimensional identification frame that matches the ship's real-time attitude. .

6. The method according to claim 5, characterized in that, The process of detecting three-dimensional targets of ships based on ship feature images to obtain three-dimensional bounding boxes also includes: designing an adaptive weighted smoothing filter algorithm to optimize the adjusted three-dimensional bounding boxes in order to eliminate ship attitude jitter and ensure the stability of the bounding boxes, as detailed below: A) A sliding window mechanism is used, selecting the vertex coordinates of the 3D bounding boxes in the current frame and the previous N frames as input. Smoothing is achieved through adaptive weight allocation, where N is a natural number. The weight calculation formula is as follows: In the formula, Let be the weight of frame t. Let be the pose error of the bounding box between frame t and the previous frame. As the attenuation coefficient, the smaller the attitude error, the greater the weight, thus increasing the influence of the stable frame; B) Perform a weighted sum of the vertex coordinates of the identifier boxes in each frame within the sliding window to obtain the optimized vertex coordinates. The final output is a three-dimensional identification frame of the ship. ; C) Detection result optimization: Set the confidence threshold to 0.5, perform non-maximum suppression processing on the output attitude-adaptive 3D bounding boxes, remove overlapping boxes, and obtain accurate 3D target detection results of ships in a single frame image.

7. The method according to claim 6, characterized in that, The process of detecting and dynamically matching a ship's real-time trajectory with an electronic fence based on its multi-target trajectory, and determining the ship's real-time regional status and trajectory risk within a preset timeframe, specifically includes: Based on the real-time trajectory 3D coordinate time series generated by fitting the multi-target trajectory results of ships, a 3D map of the bridge waterway is constructed. The trajectory coordinates are accurately mapped from the camera coordinate system to the local coordinate system of the map through the pre-calibrated camera extrinsic parameters, ensuring the spatial consistency between the trajectory and the 3D map. The Kalman filter algorithm is used to further smooth and optimize the mapped ship 3D coordinate data, accurately predict the ship's position and motion state at the next moment, and finally obtain the real-time ship driving trajectory that is completely adapted to the 3D map. The system retrieves electronic fence parameters from the database in real time to construct a three-dimensional spatial model of the electronic fence; it also analyzes the coordinates (x, y, z) of each point on the ship's real-time trajectory. t ,y t ,z t The system uses a point-to-polygon distance algorithm to determine the region to which a point belongs: it calculates the shortest distance d from the point to the boundary of the safe zone. If d ≥ 0 and the point is inside the safe zone, it is considered to be in a safe state; if d < 0 and d ≤ the width of the warning zone, it is considered to be in a warning state; if d > the width of the warning zone or the point enters the boundary of the danger zone, it is considered to be in a danger state. At the same time, based on the ship's current speed v and direction of travel θ, it predicts the set of trajectory points in the next 3 seconds. If the predicted trajectory points enter the warning or danger zone, the warning is triggered in advance.

8. The method according to claim 7, characterized in that, The electronic fence delineation process in the database includes: A) Collect AIS trajectory data and machine vision detection trajectory data of ships in the bridge waterway within one year, and screen the valid trajectories; convert the geographic coordinates of the trajectory data into a local coordinate system consistent with the 3D map through a coordinate transformation algorithm; use the Douglas-Peucker algorithm to simplify the trajectory, retain key turning points and driving nodes, and reduce data redundancy. B) Based on the preprocessed historical ship trajectory data, the DBSCAN clustering algorithm is used to perform cluster analysis on the waterway operation area; the clustering radius and minimum number of cluster points are set, and the high-frequency ship operation area is obtained through clustering; combined with the terrain obstacles in the 3D map, the boundary of the dangerous area where ships are prohibited from entering is determined, and the area between the core waterway area and the dangerous area is set as the buffer warning area. C) Mark the three-dimensional boundaries of the safe zone, warning zone, and danger zone in the three-dimensional map to form a three-dimensional electronic fence; the definitions of each zone are as follows: (1) Safe zone: the core waterway area obtained by clustering, with a minimum distance of 20m from the boundary to the obstacles on both sides of the waterway; (2) Warning zone: the buffer zone between the safe zone and the danger zone, with a width of 15m, and dynamically adjusted according to the water level; (3) Danger zone: the area within 10m around the bridge pier, the area where the top elevation of the shoal is higher than the minimum navigable water level, and other prohibited navigation areas; at the same time, establish a dynamic adjustment model for the electronic fence, and adjust the vertical coordinates of the electronic fence according to the real-time water level height h, with the adjustment formula as follows: in To adjust the vertical coordinates of the electronic fence, Reference water level The vertical coordinates of the electronic fence below; D) Store the 3D boundary parameters and adjustment model of the divided dynamic electronic fence in the database, and associate and bind them with the 3D map to support real-time calling and updating of electronic fence parameters as needed.

9. The method according to claim 8, characterized in that, Based on the regional status assessment results and trajectory risk prediction results, a multi-level early warning mechanism is established, clarifying the triggering conditions for each early warning level and determining the corresponding linkage response methods, specifically including: 1) A three-level early warning mechanism is set up as follows: A) Level 1 warning, which is the reminder level: the vessel enters the warning area, or is predicted to enter the warning area in 3 seconds. The triggering condition is that the distance between the trajectory point and the boundary of the safe area is 0 < d ≤ 10m; B) Level 2 warning, which is the alert level: the vessel is less than 5m from the boundary of the danger area, or is predicted to enter the danger area in 3 seconds. The triggering condition is that 10m < d ≤ 15m or the predicted trajectory touches the boundary of the danger area; C) Level 3 warning, which is the emergency level: the vessel enters the danger area, or is predicted to collide with an obstacle in 1 second. The triggering condition is that d > 15m or the vessel has already entered the danger area. 2) Different warning levels correspond to different response methods: A) Level 1 warning: Low-frequency warning sounds are emitted through the audible and visual alarms on both sides of the bridge, and a reminder message is broadcast on the ship's VHF communication channel at the same time; B) Level 2 warning: The volume and frequency of the audible and visual alarms are increased, a warning signal is sent to the bridge control center, and the ship's ID, location, and travel trajectory are displayed; C) Level 3 warning: The highest level audible and visual alarm is activated, and an emergency navigation prohibition order is sent in conjunction with the waterway traffic control department. At the same time, a mandatory avoidance signal is sent to the target ship through the ship's AIS system. 3) Automatically record the time, vessel information, warning level, triggering reason and response measures for each warning and store them in the database; regularly analyze the warning data, optimize the area division parameters of the electronic fence and the warning triggering threshold, and improve the accuracy and reliability of the warning system.

Images