Bridge anti-collision early warning method based on fusion of texture enhancement track and visual SLAM (Simultaneous Localization and Mapping)

Abstract

The invention provides a bridge anti-collision early warning method based on texture enhancement track and visual SLAM fusion, and relates to the technical field of ship intelligent anti-collision. Comprising the following steps: collecting and preprocessing ship monitoring image data in severe weather; constructing a multi-scale texture enhancement module, and performing image enhancement processing on the preprocessed image; constructing a feature fusion model which introduces a water area texture attention weight to obtain a ship feature image; performing ship three-dimensional target detection based on ship attitude adaptive adjustment to obtain a ship three-dimensional identification frame; performing ship multi-target trajectory tracking and field deployment; according to the multi-target trajectory of the ship, performing real-time trajectory detection and electronic fence dynamic matching, and judging the real-time regional state of the ship and the trajectory risk in the future preset time; and setting a multi-level early warning mechanism, determining the triggering condition of each early warning level, and determining a linkage response mode. According to the invention, the accuracy of ship target detection and track identification can be improved, the collision risk is reduced, and the shipping safety is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent collision avoidance technology for ships, specifically to a bridge collision avoidance early warning method that fuses texture-enhanced trajectory with visual SLAM. Background Technology

[0002] With the booming development of global trade and the continuous increase in the number of ships, accurate tracking of their navigation trajectories is crucial for ensuring navigational safety and improving transportation efficiency. Bridges, as key infrastructure in waterway transportation, span rivers, lakes, and seas, connecting different regions; their safety is essential for the stable operation of the entire transportation network.

[0003] In recent years, ship-bridge collisions have occurred frequently, threatening navigation safety and bridge stability. Traditional methods for ship target detection, trajectory recognition, and collision warning have several shortcomings. Existing 3D target detection methods often use fixed-axis aligned bounding boxes, which cannot dynamically adjust to the ship's pitch, roll, and yaw movements in real time. This leads to mismatches between the bounding boxes and the actual ship outline, failing to provide reliable spatial coordinate data. In adverse weather conditions such as heavy rain, fog, and strong backlighting, the contrast between the ship and the water background is low, and there is significant noise from rain, snow, and fog. Traditional detection methods struggle to accurately extract ship areas and key features, resulting in low detection rates and difficulties in trajectory recognition. Traditional bridge-waterway electronic fences are often fixed-area divisions, unable to adapt to changes in the water environment such as water level fluctuations. Furthermore, traditional detection methods lack deep integration with real-time ship trajectories, making it difficult to predict ship deviation risks in advance, resulting in delayed warning responses or high false alarm rates. Once a collision between a ship and a bridge occurs, it will not only cause serious damage to the bridge structure, leading to traffic disruption and huge economic losses, but may also trigger a series of catastrophic consequences such as ship sinking, casualties, and environmental pollution.

[0004] Therefore, improving the accuracy of ship target detection and trajectory recognition, shortening bridge collision warning time, and reducing collision risk are crucial for achieving shipping safety. Summary of the Invention

[0005] Traditional ship collision avoidance and early warning methods in existing technologies suffer from several technical problems: the marker frame cannot adapt to the real-time attitude of the ship, the detection accuracy is low in severe weather, the electronic fence is not adapted to changes in the aquatic environment, and the fusion of detection and trajectory is insufficient, leading to delayed warnings or high false alarm rates, making it difficult to ensure navigation safety in bridge-adjacent waters. This invention proposes a collision avoidance and early warning method for bridge-adjacent waters based on the fusion of texture-enhanced trajectory and visual SLAM to ensure the navigation safety of ships in bridge-adjacent waters.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A bridge collision avoidance warning method that fuses texture-enhanced trajectories with visual SLAM, the method comprising: Collect and preprocess ship monitoring image data in severe weather; A multi-scale texture enhancement module is constructed to perform image enhancement processing on the preprocessed image; Based on the enhanced image, a feature fusion model incorporating water texture attention weights is constructed to obtain ship feature images; Based on the ship feature image, perform three-dimensional target detection of the ship to obtain the three-dimensional bounding box of the ship; Based on the three-dimensional ship identification frame, perform multi-target ship trajectory tracking and on-site deployment; Based on the multi-target trajectory of the vessel, real-time trajectory detection of the vessel is performed and dynamic matching with the electronic fence is carried out to determine the real-time regional status of the vessel and the trajectory risk within a preset time period in the future. Based on the regional status assessment results and trajectory risk prediction results, a multi-level early warning mechanism is set up, the triggering conditions for each early warning level are clarified, and the corresponding linkage response methods are determined.

[0007] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention, by detecting key feature points, constructing an attitude parameter calculation model, and designing an attitude smoothing filtering algorithm, can effectively resist the interference of sea conditions and lighting, achieve the accuracy and stability of ship 3D detection in complex scenarios, and ensure a high degree of matching between the 3D bounding box and the actual attitude of the ship.

[0008] 2. The innovative multi-scale texture enhancement and feature fusion technology of this invention, through adaptive histogram equalization and the introduction of water texture attention weights, can enhance the texture difference between ships and water background, effectively filter extreme weather noise, and significantly improve the ship target detection rate and trajectory tracking accuracy under severe weather conditions.

[0009] 3. This invention introduces Simultaneous Localization and Mapping (SLAM) technology into bridge collision avoidance scenarios. It constructs a high-precision 3D waterway map using a mobile vision device and combines it with historical vessel trajectory data to form a dynamic electronic fence. This achieves dynamic matching between the electronic fence and the real-time vessel trajectory, solving the compatibility problem of traditional fixed fences and ensuring timely and targeted early warnings.

[0010] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0011] Figure 1 This is a flowchart of a bridge collision avoidance and early warning method based on the fusion of texture enhancement trajectory and visual SLAM according to the present invention. Figure 2This is a framework diagram for detecting and screening key feature points of ships according to the present invention; Figure 3 This is a dynamic adjustment framework diagram of a three-dimensional marker box based on attitude parameters according to the present invention; Figure 4 This is a structural diagram of the water texture attention module according to the present invention; Figure 5 This is a visual SLAM system framework diagram according to the present invention; Figure 6 This is a framework diagram of dynamic electronic fence matching for real-time ship trajectory according to the present invention. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings, so as to more clearly understand the purpose, features and advantages of this invention. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of this invention, but are only for illustrating the essential spirit of the technical solutions of this invention. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0013] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0014] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0015] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to mean “and / or” unless otherwise expressly stated herein.

[0016] In the following description, in order to clearly demonstrate the structure and working method of the present invention, a number of directional terms will be used. However, terms such as "front", "back", "left", "right", "outside", "inside", "outward", "inward", "up", and "down" should be understood as convenient terms and not as limiting terms.

[0017] The implementation details of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0018] The purpose of this invention is to address the shortcomings of the aforementioned background technology by providing a bridge-water collision avoidance and early warning method based on the fusion of texture-enhanced trajectory and visual SLAM, such as... Figure 1 As shown.

[0019] This invention addresses the challenges of low contrast between ships and the water background, making trajectory recognition difficult under adverse weather conditions such as heavy rain, dense fog, and strong backlighting. It proposes a ship trajectory recognition method based on enhanced water texture features. Furthermore, to solve the problem that traditional fixed electronic fences cannot adapt to changes in the water environment (such as water level fluctuations), a dynamic electronic fence early warning method for bridge waterways, integrating texture-enhanced trajectories and visual SLAM, is proposed, as detailed below: Step 1: Collect and preprocess ship monitoring image data in severe weather; Specifically, 1) High-definition industrial cameras deployed above the bridge navigation area are used to collect image data of ships under different adverse weather conditions (heavy rain, heavy fog, strong backlight, light snow), covering different water environments such as inland rivers and coastal areas. Ship types include cargo ships, passenger ships, and fishing boats. The driving status covers constant speed, turning, acceleration, and deceleration to ensure the diversity and representativeness of the data samples. At the same time, weather parameters (visibility, rainfall, light intensity) are recorded at the time of collection to provide label basis for subsequent model adaptation.

[0020] 2) The collected raw image data is preprocessed. First, image cropping technology is used to remove invalid edge areas (such as buildings on the shore) and retain the core water monitoring area. Then, image size is standardized by scaling all images to a uniform pixel size to adapt to the input specifications of the subsequent model. Finally, data augmentation techniques such as random flipping, rotation (±10°), and brightness fine-tuning are used to expand the dataset and avoid model overfitting.

[0021] Step 2: Construct a multi-scale texture enhancement module to perform image enhancement processing; A multi-scale texture enhancement module is constructed, which includes an adaptive histogram equalization unit, an edge-preserving filtering unit, and a multi-scale feature fusion unit. The three units are connected in series to achieve progressive enhancement from noise removal to texture enhancement.

[0022] Specifically, 1) To address the issue of low image contrast under adverse weather conditions, an Adaptive Histogram Equalization (CLAHE) algorithm is used to enhance texture differences in the preprocessed image. The CLAHE algorithm's clipping limit parameter is set to 2.0, dividing the image into rectangular pixel blocks. Histogram equalization is then performed on each pixel block individually to avoid the loss of local details caused by global equalization. The algorithm focuses on enhancing the grayscale difference between the ship's outline and the water background, resulting in clearer ship edge textures.

[0023] 2) Considering the significant noise in images during heavy rain and light snow, a guided filter is used as an edge-preserving filtering algorithm to denoise the equalized image. The guided filter's radius parameter is set to 6, and its standard deviation parameter to 0.01. This effectively filters rain and snow noise and fog blurring interference while preserving key texture features such as ship edges and railings to the greatest extent possible, thus solving the edge blurring problem caused by traditional filtering algorithms.

[0024] 3) The denoised image is input into three convolutional kernels of different scales (3×3, 5×5, 7×7) for feature extraction to obtain multi-scale texture feature maps; the channel dimension of each scale feature map is compressed by a 1×1 convolutional kernel to unify the feature dimension to 256 dimensions; finally, the feature is fused by weighted summation, and the weight coefficients are obtained by adaptive learning through training data to achieve complementarity of ship texture features at different scales and improve the stability of ship target recognition in complex scenes.

[0025] Step 3: Construct a feature fusion model that incorporates water texture attention weights; Specifically, such as Figure 4 As shown, 1) the improved ResNet-50 is selected as the basic feature extraction network, and the enhanced image obtained in step 2 is used. Inputting the network, multi-level feature maps (C1-C5) are extracted through forward propagation of residual units: shallow feature maps (C1-C2) focus on detailed features such as ship edges and textures, while deep feature maps (C3-C5) focus on the overall outline and semantic features of the ship. The expression for extracting multi-level features through forward propagation of residual units is as follows: in, For residual mapping (including convolution, BN, and ReLU operations), x is the input feature of the residual unit, and y is the output feature of the residual unit.

[0026] 2) Construct a water texture attention module. Input the deep feature map C5 and obtain the channel-level feature vector through global average pooling. Use a 2-layer fully connected network (128 hidden layer neurons) to perform dimensionality transformation and non-linear mapping on the feature vector and output an attention weight vector equal to the number of channels. Multiply this weight vector with the shallow feature map C2 channel by channel to strengthen the feature weight at the boundary between the ship and the water area and suppress the feature response of the pure water background area, so that the model focuses on the area where the ship target is located.

[0027] Specifically, the input deep feature map C5 is used to obtain the channel feature vector v through global average pooling (GAP). R 2048 : In the formula, Let H and W be the feature values ​​of the c-th channel (c=1,...,2048) of C5 at (x,y), and H and W be the height and width of C5, respectively. The feature vector is transformed through a two-layer fully connected network (FC) to output attention weights. R 2048 : in, Mapping 2048 dimensions to 128 dimensions Map back to 2048 dimensions. For the Sigmoid function; Weight Channel-wise weighting with shallow feature map C2 enhances boundary features: In the formula, For weight The size is adaptively adjusted (by interpolating to match the C2 size). This is for element-wise multiplication.

[0028] 3) Attention-weighted shallow feature maps Cross-layer fusion is performed with the deep feature map C5, and upsampling technology is used to adjust the size of the deep feature map C5 to match that of the shallow feature map. The feature fusion is achieved by adding elements together; the fused feature map is then input into a 1×1 convolution kernel for channel integration to obtain the final ship feature image, which takes into account both detailed texture and global semantic information.

[0029] Specifically, the deep feature map C5 is upsampled to match its instance size. Subsequently with Blend according to the following formula: in, For bilinear interpolation upsampling, Used to integrate channels up to 128 dimensions.

[0030] Step 4: Perform three-dimensional target detection of the ship; Specifically, 1) the final ship feature image obtained in step 3 is preprocessed again to eliminate interference from environmental noise such as sea state and lighting conditions on subsequent detection. This includes: A) The final ship feature image is then denoised using an improved bilateral filtering algorithm. This algorithm considers the similarity between the spatial domain and the grayscale domain, effectively removing wave reflections and fog noise while preserving ship edge details. The filtering formula is as follows: Where g(i,j) is the pixel value of the denoised image at coordinate (i,j), and f(k,l) is the pixel value of the image at coordinate (k,l). For spatial domain standard deviation, C(i,j) is the standard deviation of the grayscale value range, and C(i,j) is the normalization coefficient, ensuring that the output pixel values ​​are within a reasonable range.

[0031] B) For scenes with uneven lighting, such as backlighting and strong light, an adaptive histogram equalization (CLAHE) algorithm is used to adjust image contrast and enhance the distinction between the ship and the background. This is achieved by dividing the image into several sub-blocks, performing histogram equalization on each sub-block, and setting a contrast threshold. To avoid excessive noise amplification, the formula is as follows: in, This is the histogram of the equalized sub-blocks. This is the original sub-block histogram. This represents the maximum frequency in the histogram.

[0032] C) Use the YOLOv8 lightweight network to perform preliminary ship detection on the processed image and output the ship's two-dimensional bounding box. (in The coordinates of the top left corner (The coordinates are the bottom right corner). Based on this bounding box, expand outward by 10%~20% to form the ROI region, eliminate background interference, and focus on the main body area of ​​the ship for subsequent key point detection.

[0033] 2) A ship key feature point detection model is constructed based on an improved HRNet (high-resolution network) to accurately locate the core feature points related to ship attitude, such as... Figure 2 As shown, the specific operation is as follows: A) Based on the characteristics of the ship's structure, define 8 core attitude-related feature points of the ship (the port end of the bow). Starboard end of the bow The port end of the stern Starboard end of the stern Chimney top point Midpoint of the left edge of the deck Midpoint of the right edge of the deck Lowest point of the ship's side (This covers the key related parts of the ship's pitch, roll, and yaw attitudes.)

[0034] B) Based on the original HRNet, a focus mechanism (CBAM) is introduced to enhance the feature response of feature point regions. Simultaneously, Focal Loss is used to address the uneven distribution of ship feature points (some feature points, such as the top of the chimney, have a small proportion). The loss function formula is as follows: in, Let be the weight coefficient of the i-th feature point. To predict the coordinates of feature points, To label the coordinates of the actual feature points.

[0035] C) Input the ROI region extracted in step 1) into the trained improved HRNet model, and output the initial coordinates of the 8 core feature points. (Pixel coordinate system); The RANSAC algorithm is used to filter the initial feature points, removing outliers caused by occlusion and noise, and retaining the interior point set. The selection criteria are as follows: calculate the distance from each feature point to the main outline of the ship, and when the distance is greater than the set threshold δ (empirical value is 5~8 pixels), it is judged as an abnormal point and removed.

[0036] 3) Based on the selected set of key feature points, construct an attitude parameter calculation model to accurately estimate the ship's pitch angle. Roll angle Yaw angle The scale parameters (length L, width W, height H) are as follows: A) Obtain the intrinsic parameter matrix through camera calibration Convert the pixel coordinates of feature points into three-dimensional coordinates in the camera coordinate system. The conversion formula is as follows: in, These are the pixel coordinates of the feature point. , , The focal lengths are in the x and y directions. The coordinates of the main point.

[0037] B) Attitude angle estimation: (1) Pitch angle Define the line connecting the midpoints of the deck edges. The horizontal angle is calculated using the vector dot product: When the bow rises It is positive when it sinks, and negative when it sinks.

[0038] (2) Roll angle Define the lowest point of the ship's side. Line connecting the midpoints of the deck edges The angle corresponding to the ratio of the vertical distance to the standard width of the ship is the roll angle, and the formula is as follows: in, for arrive vertical distance, The standard width of the vessel (a value obtained by matching from the vessel type knowledge base).

[0039] (3) Yaw angle Define the bow direction vector. The angle between the camera and the X-axis of the camera coordinate system is given by the following formula: in, Let X be the component of the ship's longitudinal axis vector on the X-axis of the camera coordinate system. This represents the component of the ship's longitudinal axis vector on the Z-axis of the camera coordinate system. C) Calculate the real-time dimensions of the ship based on the distance between feature points, using the following formula: in, It represents the length of the ship (the distance from the port end of the bow to the port end of the stern). Represents the width (distance between the left and right ends of the bow). It represents the height (the distance from the top of the chimney to the lowest point of the ship's side).

[0040] 4) The attitude angle estimated in step 3) Based on the scale parameters (L, W, H), a three-dimensional marker frame dynamic adjustment model is constructed to achieve precise matching between the marker frame and the actual attitude of the ship, as detailed below: A) Based on the ship's center point Construct an initial axis-aligned 3D bounding box with the origin as the reference point. Taking the origin as an example, Figure 3 As shown, the coordinates of its 8 vertices are: k=1,2,...,8.

[0041] B) Based on the estimated pitch angle Roll angle Yaw angle Construct a three-dimensional rotation matrix This achieves the initial alignment of the bounding box. The rotation matrix is ​​constructed using the ZYX Euler angles order, as shown in the following formula: The rotation matrices for each axis are as follows: C) The coordinates of the 8 vertices of the initial frame are obtained through a rotation matrix. Rotate the vertex to obtain the coordinates after attitude adjustment. This ultimately forms a three-dimensional identification frame that matches the ship's real-time attitude. .

[0042] 5) To eliminate ship attitude jitter caused by sea wave turbulence, an adaptive weighted smoothing filter algorithm is designed to optimize the adjusted 3D bounding box, ensuring its stability, as follows: A) A sliding window mechanism is used, selecting the vertex coordinates of the 3D bounding boxes in the current frame and the previous N frames (N is 5~10, adjusted according to real-time requirements) as input, and achieving smoothing through adaptive weight allocation. The weight calculation formula is as follows: in, Let be the weight of frame t. The pose error of the bounding box between frame t and the previous frame is calculated using the mean square error of the vertex coordinates. The attenuation coefficient is the smaller the attitude error, the greater the weight, thus increasing the influence of the stable frame.

[0043] B) Perform a weighted sum of the vertex coordinates of the identifier boxes in each frame within the sliding window to obtain the optimized vertex coordinates. The final output is a stable and accurate three-dimensional ship identification frame. .

[0044] C) Detection result optimization: Set the confidence threshold to 0.5, perform non-maximum suppression (NMS) on the output attitude-adaptive 3D bounding boxes to remove overlapping boxes, and obtain accurate 3D target detection results of ships in a single frame image.

[0045] Step 5: Multi-target tracking and on-site deployment of ships; Multi-target trajectory association: Using Kalman filtering and the Hungarian algorithm, based on the center coordinates of the attitude-adaptive 3D bounding box output in step 4, the position and motion state (velocity, acceleration) of the ship in the current frame are first predicted by Kalman filtering; then, the intersection-over-union ratio (IoU) between the predicted position and the detected position in the current frame is calculated as a similarity metric to achieve target ID matching. Finally, the optimal matching of ship targets between consecutive frames is achieved using the Hungarian algorithm, assigning a unique ID number to each ship.

[0046] Trajectory fitting: Based on the center coordinates of the ship in consecutive frames, a polynomial fitting algorithm is used to generate the ship's trajectory.

[0047] On-site deployment: The fused model is deployed to the edge computing device (NVIDIA Jetson Xavier NX) at the bridge site to receive image data collected by the monitoring camera in real time. Through the process of steps 2-4, real-time identification of ship trajectories under severe weather conditions is achieved.

[0048] Step 6: Perform real-time ship trajectory detection and dynamic fence matching; Specifically, 1) Based on the multi-target trajectory results of ships output in step 5—including the unique ID of each ship and the time series sequence of real-time trajectory 3D coordinates generated by polynomial fitting—a 3D map of the bridge waterway is constructed. The trajectory coordinates are accurately mapped from the camera coordinate system to the local coordinate system of the map through the pre-calibrated camera extrinsic parameters, ensuring the spatial consistency between the trajectory and the 3D map. The Kalman filter algorithm is used to further smooth and optimize the mapped ship 3D coordinate data, accurately predict the ship's position and motion state (velocity, acceleration) at the next moment, and finally obtain the real-time ship driving trajectory that is completely adapted to the 3D map.

[0049] The construction of a 3D map of the bridge and the waterway specifically includes: A) Conduct visual data acquisition and preprocessing for bridge waterways, including: a) A mobile UAV platform equipped with a high-resolution RGB-D camera (1920×1080 resolution, depth measurement range 0.5-10m) was used to conduct panoramic scanning and data collection of the water area surrounding the bridge. The data collection path covered 500m of water area upstream and downstream of the bridge. The route was planned using a combination of "lateral scanning + longitudinal cruising". The lateral scanning interval was set at 5m, and the longitudinal cruising route was set along the centerline of the waterway and one of the two sides. During the data collection, the camera pose information (acquired through the camera's built-in IMU sensor) and environmental parameters (water level, wind speed, and water flow velocity, acquired in real time through portable sensors) were recorded simultaneously.

[0050] The acquired RGB and depth images are synchronized and aligned. Image distortion correction is performed based on the camera intrinsic parameter matrix (obtained in advance using the Zhang Zhengyou calibration method). The correction formula is as follows: in, These are the coordinates of the distorted pixels. For the corrected coordinates, Let these be the coordinates of the camera's principal point. This refers to the camera's focal length.

[0051] b) Noise removal is performed on the depth image using a bilateral filtering algorithm. The filter kernel size is set to 5×5, and the spatial standard deviation and gray value standard deviation are both set to 1.0. Finally, the RGB image, depth image, and IMU data are associated and bound according to the acquisition timestamp to form a standardized dataset.

[0052] B) Constructing a 3D map of the bridge and waterway based on a visual SLAM framework; like Figure 5 As shown, a feature-point-based visual SLAM framework is adopted, which mainly includes four modules: front-end visual odometry, back-end optimization, loop closure detection, and 3D map construction. The overall process is "image feature extraction - feature matching - pose estimation - back-end optimization - map generation", specifically including: a) The SIFT algorithm is used to extract feature points from adjacent RGB images and construct feature descriptors. Feature points are matched using the FLANN matcher and mismatched points are removed using the RANSAC algorithm (the number of iterations is set to 1000 and the inlier threshold is set to 2.0 pixels). Based on the matched feature point pairs, the coordinates of three-dimensional points are calculated using depth image data. The relative poses of the cameras in adjacent frames (rotation matrix R, translation vector t) are solved using the PnP algorithm, and the feature points are tracked using optical flow to optimize the continuity of inter-frame pose estimation, thereby obtaining the initial trajectory of the camera and the sparse three-dimensional point cloud.

[0053] b) Input the initial pose and sparse point cloud obtained from the front end into the graph optimization-based backend system (using the general graph optimization g2o framework) to construct a pose graph (nodes represent camera poses, and edges represent inter-frame relative pose constraints); simultaneously, start the loop closure detection module, use the bag-of-words (BoW) model to calculate the similarity between the current frame and historical frames, and determine potential loop closures when the similarity is greater than 0.7, add loop closure constraints by calculating the relative poses between the loop frames; based on the pose graph and various constraints, use the LM algorithm to optimize the pose graph, minimize the global reprojection error, and obtain high-precision camera trajectories and optimized sparse 3D point clouds.

[0054] c) Based on the optimized camera trajectory and sparse point cloud, the Poisson reconstruction algorithm is used to densify the sparse point cloud to generate a dense 3D point cloud map of the bridge water area, which includes key topographic features such as bridge piers, channel boundaries, and underwater shoals. To adapt to changes in the water environment, a dynamic map update mechanism is established: every 7 days, mobile vision equipment is used to re-collect key areas of the water area (around bridge piers and channel turning points), and the newly collected point cloud is registered and fused with the original map using the ICP algorithm to update the 3D data of key feature areas such as water level changes and shoal displacement in the map.

[0055] 2) such as Figure 6 As shown, the electronic fence parameters (combined with parameters automatically adjusted based on the current water level) are retrieved from the database in real time to construct a three-dimensional spatial model of the fence; for each coordinate point (x) on the real-time trajectory of the vessel... t ,y t ,z t (t is a timestamp), the point-to-polygon distance algorithm is used to determine the region to which the point belongs: the shortest distance d from the point to the boundary of the safe zone is calculated. If d ≥ 0 and the point is inside the safe zone, it is considered to be in a safe state; if d < 0 and d ≤ 15m (the width of the warning zone), it is considered to be in a warning state; if d > 15m or the point enters the boundary of the danger zone, it is considered to be in a danger state. At the same time, based on the ship's current speed v and direction of travel θ, the set of trajectory points in the next 3 seconds is predicted. If the predicted trajectory points enter the warning or danger zone, a warning is triggered in advance.

[0056] The electronic fence delineation process in the database includes: A) Collect nearly one year of ship AIS trajectory data and machine vision detection trajectory data in the bridge waterway, and filter valid trajectories (excluding those with a dwell time exceeding one hour or abnormal speed); convert the geographic coordinates (latitude and longitude) of the trajectory data into a coordinate transformation algorithm (such as Gauss-Kruger projection) that corresponds to three-dimensional coordinates. Figure 1 A local coordinate system is established; the Douglas-Peucker algorithm is used to simplify the trajectory, retaining key turning points and driving nodes to reduce data redundancy.

[0057] B) Based on the preprocessed historical ship trajectory data, the DBSCAN clustering algorithm is used to perform cluster analysis on the waterway navigation area. The clustering radius ε=10m and the minimum number of cluster points MinPts=50 are set. The high-frequency ship navigation area (i.e., the core waterway area) is obtained through clustering. Combined with the terrain obstacles (such as bridge piers and shoals) in the 3D map, the boundary of the dangerous area where ships are prohibited from entering is determined. The area between the core waterway area and the dangerous area is set as the buffer warning area.

[0058] C) Mark the three-dimensional boundaries of the safe zone, warning zone, and danger zone in the three-dimensional map to form a three-dimensional electronic fence. The definitions of each zone are as follows: (1) Safe zone: the core waterway area obtained by clustering, with a minimum distance of 20m from the boundary to obstacles on both sides of the waterway (such as the shore and shoals); (2) Warning zone: the buffer zone between the safe zone and the danger zone, with a width of 15m, and dynamically adjusted according to the water level; (3) Danger zone: the area within 10m around the bridge pier, the area where the top elevation of the shoal is higher than the minimum navigable water level, and other prohibited navigation areas. At the same time, establish a dynamic adjustment model for the fence, and adjust the vertical coordinates of the fence according to the real-time water level height h. The adjustment formula is: in To adjust the vertical coordinates of the fence, Reference water level The vertical coordinates of the fence below.

[0059] D) Store the three-dimensional boundary parameters (vertices coordinate set) of the divided dynamic electronic fence and the adjustment model in the database, and associate and bind them with the three-dimensional map to support real-time calling and updating of fence parameters according to the needs of water level, waterway adjustment, etc.

[0060] Step 7: Set up multi-level early warnings and determine the triggering conditions and linkage response methods.

[0061] Specifically, 1) A three-level early warning mechanism is set up, as follows: A) Level 1 warning (reminder level): The vessel enters the warning area, or is predicted to enter the warning area in 3 seconds. The triggering condition is that the distance between the trajectory point and the boundary of the safe area is 0 < d ≤ 10m; B) Level 2 warning (alert level): The vessel is less than 5m from the boundary of the danger area, or is predicted to enter the danger area in 3 seconds. The triggering condition is that 10m < d ≤ 15m or the predicted trajectory touches the boundary of the danger area; C) Level 3 warning (emergency level): The vessel enters the danger area, or is predicted to collide with obstacles such as bridge piers in 1 second. The triggering condition is that d > 15m or the vessel has already entered the danger area.

[0062] 2) Different warning levels correspond to different response methods: A) Level 1 warning: Low-frequency warning sounds are emitted through the audible and visual alarms on both sides of the bridge, and a reminder message is broadcast on the ship's VHF communication channel at the same time; B) Level 2 warning: The volume and frequency of the audible and visual alarms are increased, a warning signal is sent to the bridge control center, and the ship's ID, location and travel trajectory are displayed; C) Level 3 warning: The highest level audible and visual alarm is activated, and an emergency navigation prohibition order is sent in conjunction with the waterway traffic control department. At the same time, a mandatory avoidance signal is sent to the target ship through the ship's AIS system. If necessary, the bridge anti-collision device (anti-collision buffer pad) is activated to prepare for protection.

[0063] 3) Automatically record the time, vessel information, warning level, triggering reason and response measures for each warning and store them in the database; regularly review and analyze the warning data, optimize the area division parameters of the electronic fence and the warning triggering threshold, and improve the accuracy and reliability of the warning system.

[0064] This invention proposes a bridge collision avoidance early warning method that integrates texture-enhanced trajectory and visual SLAM. Addressing the problems of traditional methods, such as the inability of bounding boxes to adapt to real-time ship attitude, low detection accuracy in adverse weather conditions, the inability of electronic fences to adapt to changes in the aquatic environment, and insufficient fusion of detection and trajectory, this method acquires and preprocesses ship monitoring images in adverse weather conditions. It constructs a multi-scale texture enhancement module and a feature fusion model incorporating water texture attention weights to perform 3D ship target detection. Combined with Kalman filtering and the Hungarian algorithm, it performs multi-target trajectory tracking and on-site deployment. Using visual SLAM technology, it constructs a high-precision 3D water map and forms a dynamic electronic fence, achieving dynamic matching between the ship's real-time trajectory and the fence. Simultaneously, it sets up three levels of early warning and corresponding linkage response modes. This method effectively resists interference from sea conditions and lighting, improves the ship target detection rate and trajectory tracking accuracy in adverse weather conditions, solves the adaptability problem of traditional fixed fences, ensures the timeliness, relevance, and accuracy of early warnings, reduces the risk of ship-bridge collisions, and ensures navigation safety in bridge-adjacent waters.

[0065] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention. Such modifications or substitutions should all fall within the scope of the invention, or any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. A bridge collision warning method of fusing texture-enhanced trajectory and visual SLAM, characterized in that, The method comprises: Collect and preprocess the adverse weather ship monitoring image data; Construct a multi-scale texture enhancement module to perform image enhancement processing on the preprocessed image; Based on the enhanced image, a feature fusion model introducing water area texture attention weight is constructed to obtain a ship feature image; Based on the ship feature image, a ship three-dimensional target detection is performed to obtain a ship three-dimensional bounding box; Based on the ship three-dimensional bounding box, a ship multi-target trajectory tracking and field deployment are performed; According to the ship multi-target trajectory, a ship real-time trajectory detection and electronic fence dynamic matching are performed to determine the ship real-time area state and the trajectory risk in the future preset time; According to the area state determination result and the trajectory risk prediction result, a multi-level early warning mechanism is set, the trigger conditions of each early warning level are determined, and the corresponding linkage response mode is determined.

2. The method of claim 1, wherein, Based on the enhanced image, a feature fusion model introducing water area texture attention weight is constructed to obtain a ship feature image, which specifically comprises: 1) Select the improved ResNet-50 as the basic feature extraction network, and input the enhanced image Input the network, and extract multi-level feature maps C1-C5 through forward propagation of the residual unit: among them, shallow feature maps C1-C2 focus on ship detail features, and deep feature maps C3-C5 focus on ship overall contour and semantic features; wherein, the multi-level feature expression extracted through forward propagation of the residual unit is as follows: wherein, is a residual mapping, x is an input feature of the residual unit, y is an output feature of the residual unit; 2) Construct a water area texture attention module, input deep feature map C5, get channel level feature vector through global average pooling; use 2 layers of fully connected network to convert the dimension of the feature vector and nonlinear mapping, output an attention weight vector equal to the number of channels; multiply the weight vector with the shallow feature map C2 channel by channel to strengthen the feature weight at the boundary between the ship and the water area, suppress the feature response of the pure water area background, and make the model focus on the area where the ship target is located; wherein, input deep feature map C5, get channel feature vector v R 2048 : wherein C5c(x, y) is the eigenvalue of the cth channel of C5 at (x, y), where c = 1,..., 2048; H, W C5h and C5w are the height and width of C5; convert the feature vector through a 2-layer fully connected network, output attention weight R 2048 : wherein, mapping from 2048 dimensions to 128 dimensions, mapping back to 2048 dimensions, is a sigmoid function; weighting the weights with the shallow feature maps C2, the boundary features are strengthened: wherein is a weight is a dimensionally adaptive adjustment, is an element-wise multiplication; 3) the attention-weighted shallow feature map is fused with the deep feature map C5, and the size of the deep feature map C5 is adjusted to be consistent with the shallow feature map by using an up-sampling technique, and the feature fusion is realized by element addition; the fused feature map is then input into a 1x1 convolution kernel for channel integration to obtain the final ship feature image; wherein the deep feature map C5 is up-sampled to match the instance size, and then fused with the following formula: wherein is bilinear interpolation up-sampling, for integrating channels to 128 dimensions.

3. The method of claim 2, wherein, Based on the ship feature image, a ship three-dimensional target detection is performed to obtain a ship three-dimensional bounding box, which specifically comprises: The obtained ship feature image is preprocessed again to eliminate environmental noise interference on subsequent detection, including: A) The ship feature image is denoised by using an improved bilateral filtering algorithm, which considers the similarity of spatial domain and gray value domain, can effectively remove sea wave reflection and fog noise, and can also retain ship edge details, and the filtering formula is as follows: where g(i,j) is the pixel value of the image after denoising at coordinate (i,j), f(k,l) is the pixel value of the image at coordinate (k,l), is the spatial domain standard deviation, is the gray value range standard deviation, and C(i,j) is a normalization coefficient, which ensures that the output pixel value is within a reasonable range. B) For the uneven illumination scene, the adaptive histogram equalization algorithm is used to adjust the image contrast, enhance the discrimination of the ship and the background; by dividing the image into several sub-blocks, histogram equalization is performed on each sub-block, and a limited contrast threshold is set Avoiding over-enhancing noise, the formula is as follows: wherein, is the equalized sub-block histogram, is the original sub-block histogram, is the histogram maximum frequency; C) The YOLOv8 lightweight network is used for preliminary detection of the processed image, and a two-dimensional boundary box of the ship is output wherein is the upper left corner coordinate, is the lower right corner coordinate, and the ROI region is formed by expanding 10%~20% outward on the basis of the boundary box, excluding background interference and focusing on the main body region of the ship for subsequent key point detection.

4. The method of claim 3, wherein, Based on the ship feature image, a ship three-dimensional target detection is performed to obtain a ship three-dimensional bounding box, which further comprises constructing a ship key feature point detection model based on an improved HRNet to locate the core feature points related to the ship posture, and the specific operation is as follows: A) According to the characteristics of the ship structure, 8 core attitude-related feature points of the ship are defined, which are the left end point of the bow , the right end point of the bow , the left end point of the stern , the right end point of the stern , the top end point of the funnel , the middle point of the left edge of the deck , the middle point of the right edge of the deck , and the lowest point of the ship side , covering the key related parts of the ship's pitch, roll and yaw attitude; B) On the basis of the original HRNet, an attention mechanism is introduced to enhance the feature response of the feature point area, and FocalLoss is used to solve the uneven distribution problem of ship feature points, wherein the loss function formula is as follows: In the formula, is the weight coefficient of the i-th feature point, is the predicted feature point coordinate, is the labeled real feature point coordinate; C) input the extracted ROI region into the trained improved HRNet model, output the initial coordinates of 8 core feature points ; adopt RANSAC algorithm to screen the initial feature points, eliminate abnormal points and retain the inlier set ; the screening standard is: calculate the distance from each feature point to the main contour of the ship, and when the distance is greater than the set threshold δ, it is determined as an abnormal point and eliminated.

5. The method of claim 4, wherein, The ship three-dimensional target detection based on the ship feature image obtains a ship three-dimensional identification frame, and further includes: constructing a pose parameter calculation model based on the screened key feature point in-point set, estimating the pitch angle , roll angle , yaw angle and scale parameter of the ship, specifically as follows: A) Obtain the intrinsic matrix by camera calibration Convert the pixel coordinates of the feature points into three-dimensional coordinates in the camera coordinate system The conversion formula is as follows: wherein, is a feature point pixel coordinate, , , is an x, y direction focal length, is a principal point coordinate; B) Posture angle estimation: (1) Pitch angle : defines the angle between the horizontal and the line connecting the midpoints of the deck edges the angle in horizontal direction, calculated by the vector dot product: When the bow rises Positive when the bow rises, negative when the bow falls. (2) Roll angle : Definition of the roll angle : Definition of the roll angle The roll angle is the angle corresponding to the ratio of the vertical distance from the lowest point of the ship's side to the midpoint of the deck edge to the standard width of the ship. The formula is as follows: wherein is to the vertical distance, is the standard width of the ship; (3) yaw angle : define the bow direction vector with the X axis of the camera coordinate system, the formula is as follows: wherein is the component of the vessel longitudinal axis vector on the X axis of the camera coordinate system, is the component of the vessel longitudinal axis vector on the Z axis of the camera coordinate system; C) Calculate the real-time scale of the ship based on the distance between the feature points, and the formula is as follows: wherein, represents the length of the ship, represents the width, represents the height.

6. The method of claim 5, wherein, The ship three-dimensional target detection based on the ship feature image obtains a ship three-dimensional identification frame, and further comprises the following steps of: taking the estimated attitude angle and scale parameter L , W , H as the basis to construct a three-dimensional identification frame dynamic adjustment model, so as to realize the matching of the identification frame and the actual attitude of the ship, and the specific steps are as follows: A) Construct an initial axis-aligned 3D bounding box with the center point of the ship as the origin as the origin, with 8 vertices coordinates: as the origin, with 8 vertices coordinates: ,k=1,2,...,8； B) Based on the estimated pitch angle , roll angle , yaw angle , construct a three-dimensional rotation matrix , the three-dimensional rotation matrix is constructed in the Z-Y-X Euler angle order, and the formula is as follows: Wherein the rotation matrix of each axis is respectively: C) 8 vertex coordinates of the initial identification frame are rotated by a rotation matrix Rotation is performed to obtain the pose-adjusted vertex coordinates Finally, a three-dimensional identification frame matching the real-time pose of the ship is formed .

7. The method of claim 6, wherein, Based on the ship feature image, a ship three-dimensional target detection is performed to obtain a ship three-dimensional bounding box, which further comprises designing a self-adaptive weighted smoothing filter algorithm to optimize the adjusted three-dimensional bounding box to ensure the stability of the bounding box, and the specific operation is as follows: A) A sliding window mechanism is adopted to select the three-dimensional bounding box vertex coordinates of the current frame and the previous N frames as input, and the smoothing processing is realized through self-adaptive weight distribution, wherein N is a natural number, and the weight calculation formula is as follows: In the formula, is the weight of the t-th frame, is the pose error of the t-th frame and the previous frame identification frame, is the decay coefficient, the smaller the pose error, the greater the weight, and the influence of the stable frame is improved; B) performing weighted summation on the top vertex coordinates of the identification frame of each frame in the sliding window to obtain optimized top vertex coordinates , and finally outputting a three-dimensional identification frame of the ship ; C) Detection result optimization: Set the confidence threshold to 0.5, and perform non-maximum suppression processing on the output adaptive three-dimensional bounding box of the posture to remove the overlapping boxes, and obtain the accurate three-dimensional target detection result of the ship in the single frame image.

8. The method of claim 7, wherein, Based on the ship multi-target trajectory, a ship real-time trajectory detection and electronic fence dynamic matching are performed to determine the ship real-time area state and the trajectory risk in the future preset time, which specifically comprises: Based on the real-time trajectory three-dimensional coordinate time sequence fitted by the ship multi-target trajectory result, the three-dimensional map of the bridge water area is constructed, the precise mapping of the trajectory coordinates from the camera coordinate system to the local coordinate system of the map is completed through the pre-calibrated camera external parameters, and the spatial consistency of the trajectory and the three-dimensional map is ensured; the ship three-dimensional coordinate data after mapping is further smoothed and optimized by using Kalman filtering algorithm, the position and motion state of the ship at the next moment are accurately predicted, and finally the ship real-time driving trajectory completely adapted to the three-dimensional map is obtained; Real-time call electronic fence parameters in the database, build electronic fence three-dimensional space model; for each coordinate point (x t ,y t ,z t ) on the real-time trajectory of the ship, using point to polygon distance algorithm to determine its belonging area: calculate the shortest distance d of the point to the boundary of the safe area, if d≥0 and in the safe area, it is determined as safe state; if d<0 and d≤warning area width, it is determined as warning state; if d>warning area width or into the boundary of the dangerous area, it is determined as dangerous state; at the same time, based on the current speed v and the travel direction θ of the ship, predict the trajectory point set within 3 seconds in the future, if the predicted trajectory point enters the warning or dangerous area, trigger the warning in advance.

9. The method of claim 8, wherein, The electronic fence division process in the database comprises: A) Collecting ship AIS trajectory data and machine vision detection trajectory data in the bridge water area within 1 year and screening effective trajectories; converting the geographic coordinates of the trajectory data into a local coordinate system consistent with the three-dimensional map through a coordinate conversion algorithm; simplifying the trajectory by using Douglas-Peucker algorithm, retaining key turning points and driving nodes, and reducing data redundancy; B) Based on the preprocessed ship historical trajectory data, the DBSCAN clustering algorithm is used for clustering analysis of the channel driving area; the clustering radius, the minimum cluster point number are set, and the high-frequency driving area of the ship is obtained through clustering; in combination with the terrain obstacles in the three-dimensional map, the boundary of the dangerous area which the ship is prohibited to enter is determined, and the area between the core channel area and the dangerous area is set as the buffer warning area; C) The three-dimensional boundaries of the safe area, the warning area and the dangerous area are marked in the three-dimensional map to form a three-dimensional electronic fence; wherein the areas are defined as follows: (1) the safe area is the core channel area obtained by clustering, and the minimum distance between the boundary and the obstacles on both sides of the channel is not less than 20m; (2) the warning area is the buffer zone between the safe area and the dangerous area, and the width is set to 15m and dynamically adjusted according to the water level; (3) the dangerous area is the area within 10m around the pier, the area with the top elevation of the shoal higher than the minimum navigable water level and other prohibited navigation areas; at the same time, a dynamic adjustment model of the electronic fence is established, and the vertical direction coordinate of the electronic fence is adjusted according to the real-time water level h, and the adjustment formula is: wherein is the adjusted e-fence vertical coordinate, is the reference water level e-fence vertical coordinate under the reference water level; D) The three-dimensional boundary parameters of the divided dynamic electronic fence and the adjustment model are stored in the database, associated and bound with the three-dimensional map, and the electronic fence parameters can be called and updated in real time according to the requirements.

10. The method of claim 9, wherein, According to the area state judgment result and the trajectory risk prediction result, a multi-level warning mechanism is set, the trigger conditions of each warning level are determined, and the corresponding linkage response mode is determined, which specifically comprises: 1) A three-level warning mechanism is set, which is specifically as follows: A) Level 1 warning is a reminding level: the ship enters the warning area or is predicted to enter the warning area after 3 seconds, the trigger condition is that the distance between the trajectory point and the safe area boundary is 0 2) Different response modes corresponding to different warning levels: A) First-level warning: low-frequency prompt sound is emitted through the sound and light alarm on both sides of the bridge, and warning information is broadcast on the ship VHF communication channel; B) Second-level warning: increase the volume and frequency of the sound and light alarm, send a warning signal to the bridge control center, display the ship ID, location and travel trajectory; C) Third-level warning: start the highest level of sound and light alarm, send an emergency navigation prohibition instruction to the channel traffic control department, and send a forced avoidance signal to the target ship through the ship AIS system; 3) Automatically record the time, ship information, warning level, trigger reason and response measures of each warning, and store them in the database; regularly analyze the warning data to optimize the area division parameters and warning trigger threshold of the electronic fence, and improve the accuracy and reliability of the warning system.

Images