A method and system for detecting navigable areas of a river for an unmanned ship

Abstract

The application discloses a kind of river navigable area detection methods for unmanned ship, comprising the following steps: obtaining river image data and laser radar point cloud data and processing to obtain image domain water bank boundary line, point cloud domain water bank boundary line;Image domain water bank boundary line, point cloud domain water bank boundary line are fused by cross-modal competition fusion strategy and are fused to obtain fused water bank boundary line;Based on the fusion water bank boundary line and the set dynamic safety distance mark channel water surface navigable area, the present application provides based on laser radar data segmentation, cluster section collaborative density constraint, double-end elevation sliding window and filtering processing to obtain water bank edge feature point, and propose the evaluation function of periodic time cumulative error self-adapting selection optimal parameter detection water bank line method, can effectively improve the precision and robustness of irregular shoreline detection, can reduce water surface clutter and noise interference, realize the stable, continuous, high-precision extraction of water bank boundary.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and intelligent water surface environment perception technology for unmanned vessels, specifically a method and system for detecting navigable areas in waterways for unmanned vessels based on multimodal fusion of vision and lidar. Background Technology

[0002] With the development of artificial intelligence, automatic control, and ship intelligence technologies, unmanned surface vessels (USVs) have been widely used in scenarios such as waterway inspection, river mapping, water surface cleaning, and water rescue. For USVs to achieve autonomous operation, they first need reliable autonomous navigation capabilities, and water environment perception and navigable area identification are key foundations. In inland rivers, lakes, and harbors, USVs need to determine the real-time positional relationships between the water surface, shoreline, and obstacles, and plan safe navigation paths accordingly. Currently, water surface perception mainly relies on optical cameras and lidar.

[0003] Visual optical camera-based methods typically extract water surface areas and shorelines through image processing or deep learning models. These methods can utilize color, texture, and edge information, but they are highly sensitive to lighting conditions. Under conditions of strong light reflection, water ripples, low nighttime illumination, or significant shoreline reflections, the water-shore boundaries in the image can easily become blurred, and reflections may be misidentified as real targets or water areas, leading to false positives, false negatives, or discontinuous boundaries, making it difficult to meet the requirements for stable operations in complex water environments.

[0004] LiDAR can utilize the spatial geometric information of point clouds for shoreline and obstacle identification, offering advantages such as independence from visible light and high ranging accuracy. However, in aquatic environments, absorption and specular reflection of the laser beam by the water surface lead to sparse or missing point clouds. Low riverbanks, gentle slopes, and shoreline vegetation further weaken the boundary features of point clouds, making it difficult for traditional thresholding, fitting, or clustering methods to reliably obtain continuous shorelines. Furthermore, point clouds lack color and texture information, limiting their ability to distinguish complex targets.

[0005] To compensate for the limitations of single sensors, existing technologies have begun to employ methods that fuse visual and lidar signals. However, some solutions still remain at the level of data overlay or result stitching, failing to fully consider the reliability differences of different sensors under varying water surface environments. For example, visual results are prone to inaccuracies under backlight, strong reflection, or low-light conditions, while radar results may also deviate under rain, fog, water vapor, or abnormal echo conditions. Without dynamic confidence evaluation and weight adjustment mechanisms, the fused shoreline and navigable area may still exhibit offsets, breaks, or misjudgments.

[0006] Therefore, how to comprehensively utilize the color, texture, and edge information of cameras and the spatial geometric information of lidar to stably extract continuous and accurate shorelines, and establish a multi-reset reliability fusion mechanism based on scene changes such as strong light, reflections, low light, rain and fog, low riverbanks, and complex vegetation, in order to improve the accuracy and robustness of unmanned vessel navigable area detection, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for detecting navigable areas in waterways for unmanned vessels, in order to solve the problems mentioned in the background art, improve the accuracy of unmanned vessels in judging navigable areas in water surface navigation, and enhance the reliability of their autonomous operation.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A method for detecting navigable areas in waterways for unmanned surface vessels includes the following steps: Acquire river channel image data and lidar point cloud data, and process them to obtain the water-shore boundary lines in the image domain and the water-shore boundary lines in the point cloud domain. A cross-modal competitive fusion strategy is used to fuse the waterfront boundary lines in the image domain and the point cloud domain to obtain the fused waterfront boundary line. Based on the integrated waterfront boundary line and the established dynamic safety distance markers, the waterway surface is marked as a navigable area.

[0009] In a preferred embodiment, the acquisition of the image domain waterfront boundary line includes the following steps: The model is based on the lightweight BiSeNetV2 network, and a joint loss function is constructed by introducing central difference convolution, adding boundary gradient consistency supervision loss and boundary continuity loss. The model training and river region segmentation are completed based on the joint loss function. Then, edge detection and edge fitting are performed based on the segmentation results to obtain the waterfront boundary line in the image domain.

[0010] In a preferred embodiment, obtaining the point cloud domain waterfront boundary line includes the following steps: Point cloud data is classified into clusters and segments. Then, the feature points of the shoreline and the equation of the shoreline boundary line of the point cloud domain are obtained by cluster and segment cooperative density constraint, double-ended elevation sliding window, smoothing coefficient feature extraction and distance filtering. The dual-end elevation sliding window feature extraction includes: projecting the three-dimensional point cloud onto the XOY plane, searching along the LiDAR scan line using a sliding window of a set size, arranging all points in each window in descending order of Z value, selecting the first and last m values ​​of the sequence as the maximum and minimum reference sets respectively, calculating the difference between the median and the mean of the two reference sets, and if both differences are within a preset threshold range, then adding the points in the window to the candidate set; An evaluation function based on the cumulative error over time (COT) is constructed, and the optimal threshold parameter is adaptively selected according to this evaluation function. Specifically, this includes: constructing a key parameter vector required for point cloud feature extraction; performing feature extraction and fitting a shoreline equation using the current parameters for each frame of point cloud data; evaluating the detection results using the COT cumulative error and calculating the shoreline evaluation function index; if the index is lower than a preset performance threshold, locally optimizing and updating the parameters until the optimal parameters are output for the next frame. The final shoreline boundary feature points and shoreline equations are obtained based on the optimal parameters.

[0011] In a preferred embodiment, the scan step size is set to... t The point cloud projection image of the river channel is horizontally scanned, and the integrity of the detected left and right boundary curve segments is determined. t The time detection results are divided into four categories: both left and right boundary points exist, only the left boundary point exists, only the right boundary point exists, and neither left nor right boundary point exists. For each scan line, calculate the distance error between the predicted point and the ground truth point according to its category; if the boundary does not exist, the error is recorded as zero; the average detection error of the frame is obtained by averaging the errors of all scan lines. Calculate the average detection error within period T, and take its reciprocal as the evaluation function index for the shoreline.

[0012] In a preferred embodiment, obtaining the fused waterfront boundary line includes the following steps: By calibrating the sensor, the coordinates of the lidar are aligned with the coordinates of the image, and the feature points of the lidar shoreline are projected onto the image plane. Data association and uncertainty modeling are used to calculate the confidence scores of the two sensors to determine the fusion weights, namely the confidence scores of the multidimensional perceptual quality of visual images and the multidimensional geometric quality of LiDAR point clouds. Based on a cross-modal feature point confidence competition mechanism, adaptive optimization is performed on candidate feature points for the shoreline; For the feature points retained after optimization, weighted least squares curve fitting is performed with their confidence levels as weights, and the equation of the fused curve is calculated. An improved particle filter algorithm is used for temporal tracking and calibration after the fusion curve fitting to prevent single-frame detection jumps.

[0013] In a preferred embodiment, the confidence level of multidimensional perceptual quality of visual images ,

[0014]

[0015] and These are the local neighborhood boundary smoothness index and the water surface highlight degradation index, respectively, for the boundary point i in the camera image. and These are the corresponding weights, calculated as follows: Waterfront Local Boundary Smoothness Index The value ranges from [0,1]. A larger value indicates a more continuous boundary. The calculation formula is as follows:

[0016] Represented by boundary feature points The set of m local points selected centered on the boundary point constitutes the neighborhood boundary point set. , This represents the number of boundary points within the neighborhood. The vertical distance from the j-th boundary point in the neighborhood to the locally fitted line; The local neighborhood scale normalization coefficient is taken as the diagonal length of the rectangle circumscribed at the boundary point of the neighborhood. Let the minimum and maximum values ​​of the x and y coordinates in the neighborhood point set be respectively... and ,but ; Localized Highlight Degradation Index The value ranges from [0,1]. A larger value indicates a stronger reflection from the water surface. The calculation formula is as follows:

[0017]

[0018] in, For the image in pixels ( x , y The brightness value at () location; Represents pixels ( x , y Is it a water surface highlight pixel? This represents the local water surface neighborhood centered at the i-th boundary point; T The threshold for determining highlights; This represents the total number of pixels in the local neighborhood water surface region. This represents the number of highlight pixels within a local neighborhood of the water surface. LiDAR point cloud multidimensional geometric quality confidence

[0019]

[0020]

[0021] and Let be the local effective feature point cloud density, neighborhood elevation stability, and distance attenuation index of the i-th laser point, respectively. , and These are the corresponding weights.

[0022] In a preferred embodiment, based on image confidence and LiDAR confidence, a cross-modal confidence competition rule is constructed for the matched visual feature points and LiDAR feature points to adaptively select reliable shoreline feature points and eliminate outliers. The competition rule is as follows:

[0023]

[0024] in, It is the spatial distance of the i-th matching point, representing spatial consistency. It is a spatial distance threshold. It is the confidence difference threshold. It is the minimum confidence threshold that meets the requirements. and These are lidar and camera visual boundary matching or sampling points, respectively. Indicates the relationship with the first i A visual point matched with a laser point; prediction hold means that when the two conflict and their confidence levels are close or the prediction hold candidate ratio exceeds the threshold, it indicates that the current observation is unreliable. In this case, the predicted shoreline from the previous moment is used to maintain the shoreline in order to avoid abrupt changes in the shoreline.

[0025] In a preferred embodiment, the improved particle filter algorithm includes the following steps: Initialization: In the first frame image, an augmented state vector is constructed using the initial shoreline curve parameters and their rate of change obtained by multi-sensor fusion. The augmented state vector includes curve parameters and parameter change rates. The particle set is initialized by sampling with a Gaussian distribution. Calculate the observation confidence gating factor: Based on the average confidence of visual detection, the average confidence of lidar detection, the cross-modal consistency confidence, and the prediction preservation flag of the current frame, calculate the observation confidence gating factor used to quantify the reliability of the fused detection results in the current frame.

[0026] Here are the weighting coefficients, and the min and max functions represent taking the minimum and maximum values, respectively. The average confidence level of the visual detection results in the current frame; This represents the average confidence level of the lidar detection results in the current frame. The cross-modal consistency confidence score is calculated based on the matching distance between the visual boundary point and the lidar boundary point. To predict the retention flag, the predicted retention flag is determined based on the degree of conflict and the degree of confidence similarity between the visual and lidar signals; Particle state prediction: Each particle is predicted according to the state transition model, wherein the process noise covariance matrix is ​​adaptively adjusted according to the observation confidence gate factor. When the observation confidence gate factor is large, the process noise is reduced, and when the observation confidence gate factor is small, the process noise is increased. Particle weights are calculated based on gating error: The observation error of a particle relative to the current frame's fusion curve and the prediction error relative to the historical prediction curve from the previous time step are calculated separately. The observation error and the historical prediction curve error are then gated and weighted using the observation reliability gating factor to obtain the gating error. Particle weights are calculated and normalized based on this gating error. When the fusion result is reliable, the reliability gating factor is... The particle weight is mainly determined by the degree of error between the historical prediction curve and the current fusion curve, and vice versa. It is determined solely by the degree of error compared to historical prediction curves; Adaptive particle number adjustment: The number of particles in the current frame is dynamically adjusted according to the observation confidence gating factor. When the observation confidence gating factor is large, the number of particles is reduced to improve computational efficiency, and when the observation confidence gating factor is small, the number of particles is increased to improve tracking accuracy. Gated elite particle resampling: Elite particles with high weights are directly retained in the new particle set. The remaining non-elite particles are generated by copying from all particles through system resampling. An adaptive random perturbation is added to the copied non-elite particles. The variance of the adaptive random perturbation is controlled by the observation reliability gating factor. The perturbation is reduced when the observation is reliable and increased when the observation is degraded. State estimation and gated output: High-quality particles are selected by weight threshold, and the high-quality particles are weighted and fused to obtain the current frame estimation result. Then, the observation confidence gate factor is used to gate and smooth the current frame estimation result and the historical prediction result of the previous moment to obtain the calibrated shoreline curve parameters of the current frame.

[0027] In a preferred embodiment, the marked navigable waterway area includes the following steps: Determine the coordinate range of the navigable area based on the equations of the left and right shoreline boundaries; A dynamic safety distance function is constructed. The dynamic safety distance function is based on the basic safety distance and is modified by weighting the shoreline stiffness coefficient, ecological sensitivity and dynamic constraint terms. The shoreline stiffness coefficient is used to characterize the hardness of the shoreline material, the ecological sensitivity is used to characterize the ecological fragility of the aquatic environment, and the dynamic constraint terms are determined by the product of the shoreline curvature and the square of the unmanned vessel speed. Based on the dynamic safety distance function, the left and right shoreline boundary lines are offset along the direction of the normal vector pointing into the river channel to generate the left and right safety boundary curves. Determine water depth safety by combining water depth measurement data; Based on the left and right safety boundary curves and water depth safety, an analytical expression for the navigable area is determined.

[0028] A system for detecting navigable waterways for unmanned vessels, comprising hardware modules and software modules; The hardware module includes the hull of the unmanned vessel, as well as an industrial control computer, lidar, camera or video camera, small multibeam echo sounder, network communication module, drive control board and positioning equipment installed on the unmanned vessel; The software module runs on the industrial control computer of the computing processing unit and includes a shoreline detection module and a navigable area judgment module. The shoreline detection module includes a camera image detection module, a lidar point cloud detection module, and a fusion detection module. The lidar point cloud detection module includes a point cloud feature processing module and a shoreline detection evaluation module.

[0029] The technical effects and advantages of this invention are as follows: 1. This invention provides a method for detecting shorelines using the lightweight BiSeNet V2 network based on central difference convolution and multi-constraint boundary supervised loss, which effectively improves the detection accuracy under water surface reflection interference environment; 2. This invention provides a method for obtaining shoreline feature points based on lidar data segmentation, cluster-segment collaborative density constraint, double-ended elevation sliding window and filtering. It also proposes a method for detecting shorelines by adaptively selecting optimal parameters based on the evaluation function of periodic time cumulative error. This method can effectively improve the accuracy and robustness of irregular shoreline detection, reduce water surface clutter and noise interference, and achieve stable, continuous and high-precision extraction of shoreline boundaries. 3. This invention constructs a visual degradation confidence model, a point cloud multidimensional geometric quality confidence model, and a cross-modal confidence competition rule. Based on the confidence level of different sensors in a local area, it adaptively generates a fused shoreline, which can effectively solve the problem of inaccurate detection by a single sensor in complex water environments, such as strong light reflection, water reflection, and low light environments.

[0030] 4. This invention constructs an observation confidence gating factor based on cross-modal confidence competition rules, and uniformly adjusts the observation update, prediction retention, particle number, and resampling perturbation intensity of the particle filter according to the gating factor. This enables the algorithm to track quickly when the observation is reliable and to stably maintain the prediction when the observation degrades. This method can effectively adapt to the nonlinear and variable characteristics of river boundary lines, suppress tracking drift caused by complex noise, improve computational efficiency while ensuring edge recognition accuracy, and achieve robust, real-time, and accurate tracking of complex river boundary lines.

[0031] 5. This invention constructs a dynamic safety distance judgment function based on shoreline attribute quantification and dynamic constraints, and dynamically calculates the navigable area of ​​the river surface based on the shoreline equation, providing an effective guarantee for the judgment of the safe driving range of unmanned vessels in complex and ever-changing river waters. Attached Figure Description

[0032] Figure 1 This is the main flowchart of the method for detecting navigable areas of unmanned vessels according to the present invention; Figure 2 This is a diagram of the image domain shoreline detection architecture based on the improved BiSeNet V2 of this invention; Figure 3 This is a flowchart of point cloud domain data processing and adaptive parameter updating of the shoreline according to the present invention. Figure 4 This is a schematic diagram of the boundary classification for evaluating the detection results based on the cumulative error of the periodic time in this invention. Figure 5 Flowchart for calibrating the river boundary line model using the improved particle filter algorithm of this invention; Figure 6 This is a flowchart of the navigable area determination method of the present invention; Figure 7 This is a schematic diagram of the coordinate system of the unmanned vessel of the present invention; Figure 8 This is a schematic diagram of the navigable area calculation symbol of the present invention; Figure 9 This is a structural diagram of the unmanned vessel river surface navigable area detection system of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0034] A method for detecting navigable areas in waterways for unmanned surface vessels includes the following steps: River channel image data and lidar point cloud data are acquired and processed to obtain the water-shore boundary lines in the image domain and the water-shore boundary lines in the point cloud domain.

[0035] Specifically, obtaining the waterfront boundary line in the image domain includes the following steps: The model is based on the lightweight BiSeNetV2 network, and a joint loss function is constructed by introducing central difference convolution, adding boundary gradient consistency supervision loss and boundary continuity loss. The model training and river region segmentation are completed based on the joint loss function. Then, edge detection and edge fitting are performed based on the segmentation results to obtain the waterfront boundary line in the image domain.

[0036] Obtaining the waterfront boundary line of a point cloud region includes the following steps: Point cloud data is classified into clusters and segments. Then, the feature points of the shoreline and the equation of the shoreline boundary line of the point cloud domain are obtained by cluster and segment cooperative density constraint, double-ended elevation sliding window, smoothing coefficient feature extraction and distance filtering. The dual-end elevation sliding window feature extraction includes: projecting the 3D point cloud onto the XOY plane, searching along the LiDAR scan line using a sliding window of a set size, arranging all points in each window in descending order of Z value, selecting the first and last m values ​​of the sequence as the maximum and minimum reference sets respectively, calculating the difference between the median and the mean of the two reference sets, and if both differences are within a preset threshold range, then adding the points in the window to the candidate set; An evaluation function based on the cumulative error over time is constructed, and the optimal threshold parameter is adaptively selected according to the evaluation function. Specifically, this includes: constructing the key parameter vector required in the point cloud feature extraction process; for each frame of point cloud data, using the current parameters to extract features and fit the shoreline equation; evaluating the detection results through the cumulative error over time and calculating the shoreline evaluation function index; if the index is lower than the preset performance threshold, the parameters are locally optimized and updated until the optimal parameters are output for the next frame.

[0037] The specific calculation process of the shoreline evaluation function index is as follows: With the set scan step size t The point cloud projection image of the river channel is horizontally scanned, and the integrity of the detected left and right boundary curve segments is determined. t The time detection results are divided into four categories: both left and right boundary points exist, only the left boundary point exists, only the right boundary point exists, and neither left nor right boundary point exists. For each scan line, calculate the distance error between the predicted point and the ground truth point according to its category; if the boundary does not exist, the error is recorded as zero; the average detection error of the frame is obtained by averaging the errors of all scan lines. Calculate the average detection error within period T, and take its reciprocal as the evaluation function index for the shoreline.

[0038] The final shoreline boundary feature points and shoreline equations are obtained based on the optimal parameters.

[0039] A cross-modal competitive fusion strategy is used to fuse the waterfront boundary lines in the image domain and the point cloud domain to obtain a fused waterfront boundary line.

[0040] The acquisition of the integrated waterfront boundary line includes the following steps: By calibrating the sensor, the coordinates of the lidar are aligned with the coordinates of the image, and the feature points of the lidar shoreline are projected onto the image plane. Data association and uncertainty modeling are used to calculate the confidence scores of the two sensors to determine the fusion weights, namely the confidence scores of the multidimensional perceptual quality of visual images and the multidimensional geometric quality of LiDAR point clouds. Confidence of multidimensional perceptual quality of visual images ,

[0041]

[0042] and These are the local neighborhood boundary smoothness index and the water surface highlight degradation index, respectively, for the boundary point i in the camera image. and These are the corresponding weights, calculated as follows: Waterfront Local Boundary Smoothness Index The value ranges from [0,1]. A larger value indicates a more continuous boundary. The calculation formula is as follows:

[0043] Represented by boundary feature points The set of m local points selected centered on the boundary point constitutes the neighborhood boundary point set. , This represents the number of boundary points within the neighborhood. The vertical distance from the j-th boundary point in the neighborhood to the locally fitted line; The local neighborhood scale normalization coefficient is taken as the diagonal length of the rectangle circumscribed at the boundary point of the neighborhood. Let the minimum and maximum values ​​of the x and y coordinates in the neighborhood point set be respectively... and ,but ; Localized Highlight Degradation Index The value ranges from [0,1]. A larger value indicates a stronger reflection from the water surface. The calculation formula is as follows:

[0044]

[0045] in, For the image in pixels ( x , y The brightness value at () location; Represents pixels ( x , y Is it a water surface highlight pixel? This represents the local water surface neighborhood centered at the i-th boundary point; T The threshold for determining highlights; This represents the total number of pixels in the local neighborhood water surface region. This represents the number of highlight pixels within a local neighborhood of the water surface. LiDAR point cloud multidimensional geometric quality confidence

[0046]

[0047]

[0048] and Let be the local effective feature point cloud density, neighborhood elevation stability, and distance attenuation index of the i-th laser point, respectively. , and These are the corresponding weights.

[0049] Based on a cross-modal feature point confidence competition mechanism, adaptive optimization is performed on candidate feature points for the shoreline; Based on image and LiDAR confidence levels, a cross-modal confidence competition rule is constructed for the matched visual and LiDAR feature points. This rule adaptively selects reliable shoreline feature points and eliminates outliers. The competition rule is as follows:

[0050]

[0051] in, It is the spatial distance of the i-th matching point, representing spatial consistency. It is a spatial distance threshold. It is the confidence difference threshold. It is the minimum confidence threshold that meets the requirements. and These are lidar and camera visual boundary matching or sampling points, respectively. Indicates the relationship with the first iA visual point matched with a laser point; prediction hold means that when the two conflict and their confidence levels are close or the prediction hold candidate ratio exceeds the threshold, it indicates that the current observation is unreliable. In this case, the predicted shoreline from the previous moment is used to maintain the shoreline in order to avoid abrupt changes in the shoreline.

[0052] For the feature points retained after optimization, weighted least squares curve fitting is performed with their confidence levels as weights, and the equation of the fused curve is calculated. An improved particle filter algorithm is used for temporal tracking and calibration after the fusion curve fitting to prevent single-frame detection jumps.

[0053] The improved particle filter algorithm includes the following steps: Initialization: In the first frame image, the augmented state vector is constructed using the initial shoreline curve parameters and their change rates obtained by multi-sensor fusion. The augmented state vector contains the curve parameters and the parameter change rates. The particle set is initialized by sampling through Gaussian distribution. Calculate the observation confidence gating factor: Based on the average confidence of visual detection, the average confidence of lidar detection, the cross-modal consistency confidence, and the prediction preservation flag of the current frame, calculate the observation confidence gating factor used to quantify the reliability of the fused detection results in the current frame.

[0054] Here are the weighting coefficients, and the min and max functions represent taking the minimum and maximum values, respectively. The average confidence level of the visual detection results in the current frame; This represents the average confidence level of the lidar detection results in the current frame. The cross-modal consistency confidence score is calculated based on the matching distance between the visual boundary point and the lidar boundary point. To predict the retention flag, the predicted retention flag is determined based on the degree of conflict and the degree of confidence similarity between the visual and lidar signals; Particle state prediction: Each particle is predicted according to the state transition model, where the process noise covariance matrix is ​​adaptively adjusted according to the observation confidence gate factor. When the observation confidence gate factor is large, the process noise is reduced, and when the observation confidence gate factor is small, the process noise is increased. Particle weights are calculated based on gating error: The observation error of a particle relative to the current frame's fusion curve and the prediction error relative to the historical prediction curve from the previous time step are calculated separately. A gating weighted fusion of the observation error of the fusion curve and the historical prediction curve error is obtained using an observation reliability gating factor. The particle weights are then calculated and normalized based on the gating error. When the fusion result is reliable, the reliability gating factor is used. The particle weight is mainly determined by the degree of error between the historical prediction curve and the current fusion curve, and vice versa. It is determined solely by the degree of error compared to historical prediction curves; Adaptive particle number adjustment: The number of particles in the current frame is dynamically adjusted based on the observation confidence gating factor. When the observation confidence gating factor is large, the number of particles is reduced to improve computational efficiency, and when the observation confidence gating factor is small, the number of particles is increased to improve tracking accuracy. Gated elite particle resampling: Elite particles with high weights are directly retained in the new particle set. The remaining non-elite particles are generated by copying from all particles through system resampling. An adaptive random perturbation is added to the copied non-elite particles. The variance of the adaptive random perturbation is controlled by the observation reliability gating factor. The perturbation is reduced when the observation is reliable and increased when the observation is degraded. State estimation and gated output: High-quality particles are selected by weight threshold, and the high-quality particles are weighted and fused to obtain the current frame estimation result. Then, the observation confidence gate factor is used to gate and smooth the current frame estimation result and the historical prediction result of the previous moment to obtain the calibrated shoreline curve parameters of the current frame.

[0055] Based on the integrated waterfront boundary line and the established dynamic safety distance markers, the waterway surface is marked as a navigable area.

[0056] Specifically: Marking navigable areas on the waterway involves the following steps: Determine the coordinate range of the navigable area based on the equations of the left and right shoreline boundaries; A dynamic safety distance function is constructed, which is based on the basic safety distance and is weighted and corrected by integrating the shoreline stiffness coefficient, ecological sensitivity and dynamic constraint terms. The shoreline stiffness coefficient is used to characterize the hardness of the shoreline material, the ecological sensitivity is used to characterize the ecological fragility of the aquatic environment, and the dynamic constraint term is determined by the product of the shoreline curvature and the square of the unmanned vessel speed. Based on the dynamic safety distance function, the left and right shoreline boundary lines are offset along the direction of the normal vector pointing into the river channel to generate the left and right safety boundary curves. Determine water depth safety by combining water depth measurement data; Based on the left and right safety boundary curves and water depth safety, the analytical expression for the navigable area is determined.

[0057] A system for detecting navigable waterways for unmanned vessels, comprising hardware modules and software modules; The hardware module includes the hull of the unmanned vessel, as well as an industrial control computer, lidar, camera or video camera, small multibeam echo sounder, network communication module, drive control board and positioning equipment installed on the unmanned vessel; The software module runs on the industrial control computer of the computing processing unit and includes a shoreline detection module and a navigable area judgment module. The shoreline detection module includes a camera image detection module, a lidar point cloud detection module, and a fusion detection module. The lidar point cloud detection module includes a point cloud feature processing module and a shoreline detection evaluation module.

[0058] Example 1 A method for detecting navigable areas in waterways for unmanned surface vessels includes three stages, such as... Figure 1 As shown, the first stage involves independent and parallel detection of the shoreline using both camera and lidar modes. The second stage involves fusing and optimizing the detection results. The third stage involves calculating the navigable area of ​​the river surface based on the shoreline equation. The detailed steps are as follows: Step 1: In the first stage, image data is acquired through the camera of the unmanned vessel, and the BiSeNet V2 network is improved by designing central differential convolution and multi-constraint boundary supervision loss to segment the water surface area, and the image domain shoreline equation is obtained based on the segmentation results. like Figure 2 As shown, the specific steps include: Step 1.1, Data Collection and Preprocessing: Collect images of riverbank scenes with different light intensities, ripples, and shadow levels. Expand the sample data through water surface reflection enhancement, dynamic ripple texture synthesis, and lighting condition simulation to improve the diversity of training samples and the model's adaptability to complex aquatic environments. (1) Physical simulation of water surface reflection and reflection enhancement. For the water surface area of ​​the training image, virtual reflections are synthesized based on a specular reflection model or a water surface reflection model, and the reflection intensity coefficient is changed. ∈[0.2,0.9] and vertical offset Δy∈[ [5, +5] pixels, enabling the network to learn and recognize the reflection area, training the image after water surface reflection enhancement;

[0059]

[0060] in, For the original image in coordinates The pixel value at that location, i.e., the actual pixel value of the shoreline scene; These are the original pixel values ​​for the water surface area, used to preserve the underlying texture of the water itself; The height of the image is the number of pixels. Used to achieve vertical flip mapping, i.e., simulated reflection; The adaptive reflection blending coefficient is used to control the blending ratio between the reflection and the background color of the water surface, and its value is (0,1). To control the decay rate, the speed at which the reflection decays as the boundary distance increases; The vertical distance from the pixel to the labeled waterfront boundary.

[0061] (2) Synthesizing dynamic ripple textures. Dynamic water surface ripple textures are generated based on a multi-sine wave superposition model, producing different... Time and space Displacement of water surface ripples at the location of superimposed texture intensity :

[0062] in, Indicates the first Each sinusoidal wave component is used to simulate irregular textures on a real water surface by superimposing multiple waves; random wave numbers are set. ,amplitude Phase Generate diverse water surface textures to expand the training set; pixel space coordinate vector ; To simulate time parameters, different times are randomly sampled to generate diverse static frame images; No. The angular frequency (radians / second) of each wave component controls the rate of change of the texture in the time dimension; (3) Simulate changes in lighting conditions, simulate extreme lighting conditions such as morning fog, backlight, and shadow, and apply random Gamma transformation to the water surface area. , This is the Gamma correction factor. ∈[0.5,2.5], and superimpose local highlights (Gaussian bright spots) to simulate water surface mirror reflection flares.

[0063] Step 1.2: Based on the lightweight BiSeNetV2 network, network improvements and optimizations are made for the special environment of the water surface to make it more adaptable to the river water surface environment. Central Difference Convolution (CDC) is introduced into BiSeNet V2 to enhance the ability to distinguish the texture features of the waterfront edge and water surface ripples. Multi-constraint boundary supervised loss is constructed for training, boundary gradient consistency supervised loss is added to improve boundary sharpness, and boundary continuity loss is added to ensure the geometric stability of the shoreline. The specific improvement methods are as follows: (1) Introducing central difference convolution Traditional convolution aggregates pixel intensity (color). The trees on the riverbank and their reflections in the water have the same color. Traditional convolution is not good at distinguishing such similar color regions. Therefore, CDC is introduced to aggregate pixel gradients (texture changes) to distinguish the subtle ripples and flowing textures of the riverbank and objects in the water reflection. The corresponding original objects on the bank are usually stationary.

[0064] In the first and second layers of the spatial path, CDC (Central Differential Convolution) replaces the standard 3x3 convolution to calculate the difference in pixel intensity within the local neighborhood, highlighting edges and surface roughness, weakening smooth reflection areas, and enhancing gradients, strengthening waterfront lines, and suppressing reflective areas. Let the input feature map be... The output is For the location in the image For each pixel, enumerate its 3×3 neighborhood. The formulas for calculating two types of convolution, namely balanced central difference and standard convolution, are as follows:

[0065] in, For the receptive field of the convolution kernel, These are learnable convolutional weights; This is a hyperparameter used to balance texture information (water ripples) and intensity information (color). In this implementation example, it's set to 0.7, causing the network to focus more on the water ripple texture, thereby suppressing reflections. Capture local texture contrast and suppress reflection areas; (2) Add boundary gradient consistency loss The predicted segmentation map gradient should be consistent with the gradient of the ground truth label to ensure network alignment in the gradient domain. A boundary gradient consistency auxiliary loss is added to the original auxiliary loss. The Sobel gradient operator is used to extract the ground truth boundary map of the label image, performing boundary supervision on the network's output segmentation map. This forces the network to focus on edge details and effectively distinguish between river edges and surface clutter. Let the network predict the segmentation map as... The real label is Boundary auxiliary loss The formula is as follows:

[0066]

[0067] in, Representing boundary feature points, The number of boundary feature points, Represents the Sobel gradient operator; (3) Add boundary continuity loss The shoreline should typically appear as a continuous, smooth curve, without obvious breaks, voids, or jagged boundaries. To enhance the continuity of the predicted boundaries, a boundary continuity loss based on the Laplace operator is introduced to constrain local abrupt changes in the boundary probability map. The formula for calculating the boundary continuity loss is as follows:

[0068]

[0069] in, This is the boundary probability map predicted by the network. This is a Laplacian convolution kernel used to constrain local abrupt changes in the boundary probability map, thereby enhancing boundary continuity and smoothness. Here, H represents the convolution operator, and H and W represent the height and width of the boundary probability map, respectively.

[0070] Finally, construct the joint loss function. for:

[0071] The standard segmentation loss, such as cross-entropy loss or Dice loss, is used to supervise the output of the entire network; The original auxiliary loss of BiSeNet V2 is used to supervise the output of intermediate layers; This is the boundary gradient consistency loss, used to enhance the consistency between the predicted boundary and the true boundary in terms of gradient location; The added boundary continuity loss is used to constrain the local smoothness and continuity of the boundary probability map; , , These are the corresponding weight coefficients; Step 1.3, Model Training and Segmentation: Set the joint loss weight coefficients, train the network using the constructed dataset, then use the trained model to infer the predicted image and output the binary segmentation results (water surface area and non-water surface area). Step 1.4, edge detection: extract water surface pixels, select the largest connected component, apply the edge detection operator to the segmented water surface region mask to extract the water shore contour, and fit the boundary pixels with a least squares quadratic polynomial to extract the final image domain water shore boundary line.

[0072] Step 2: Acquire point cloud data using the lidar sensor mounted on the unmanned vessel. Preprocess the point cloud data, segment it, and extract feature points along the waterfront edge to obtain the three-dimensional shoreline equation for the point cloud domain, as shown below. Figure 3 It includes the following core steps: Step 2.1: Preprocess the lidar point cloud data, including outlier filtering and region of interest (ROI) extraction; The outlier filtering steps are as follows: Step 2.1.1: Construct a Gaussian distribution model, assuming the points in the point cloud data... i With the surroundings k The average distance between the points follows a Gaussian distribution model. Its mean is μ And the standard deviation of the Gaussian function is σ ; Step 2.1.2: Calculate the distance from point i to its neighboring point. Average distance of points ; Step 2.1.3: Calculate the mean of the dataset. and standard deviation ;

[0073]

[0074] Step 2.1.4, if calculation If a point is found to be noisy, it is considered a noise point and needs to be removed; otherwise, the point is retained. It is the threshold of the standard deviation multiple; Step 2.1.5: Repeat the above steps until the entire point cloud data set has been traversed.

[0075] The method for extracting the Region of Interest (ROI) is as follows: Assuming the coordinates of the point are... , , This refers to the number of point clouds. The point clouds of interest are extracted through point cloud filtering, resulting in a set of filtered point clouds. Represented as

[0076] in It is the distance between the current point cloud and the center of the unmanned vessel in the vertical direction over the water; This is the maximum width of the region of interest at the left and right boundaries of the water area. When the waterfront boundary is not visible, this value is set to a very large constant. It is the maximum value of the effective point cloud, which is determined based on the height of common targets in the water area.

[0077] Step 2.2: Classify the point cloud data into clusters and segments, and obtain the feature points of the waterfront edge and the waterfront boundary line equation of the point cloud domain through methods such as cluster-segment cooperative density constraint, double-ended elevation sliding window, smoothing coefficient feature extraction and distance filtering. Specifically, the following steps are included: Step 2.2.1: Extract features using the KD-Tree-based cluster-segment collaborative density constraint method, including cluster segmentation, determining whether the density features within clusters and segments satisfy the constraint conditions, and obtaining the feature point set. The sub-step for extracting the cluster segment cooperative density constraint is as follows: Step 2.2.1.1: Classify the point cloud into clusters and segments. First, use the KD-Tree algorithm to quickly classify the point cloud into different clusters. If adjacent points on the sphere... radius The following constraints must be met:

[0078] These points are then grouped into the same cluster. It is a radius threshold, and then these clusters are treated at a fixed length along the vertical direction of the waterfront. The line segment is divided into several line segments, dividing the entire cluster into multiple parts.

[0079] Step 2.2.1.2: Density constraints are performed through cluster segment collaboration. After segmentation, the number of points in each segment is calculated, and it is determined whether the density features within cluster k and segment i meet the threshold conditions. If the conditions are met, the points are added to the candidate point set. The specific density feature calculation and judgment methods are as follows:

[0080]

[0081]

[0082] in It is the average density within cluster k. and It is the density threshold. It is the point density of segment i. It is the number of segments within each cluster. It is a segment The number of points inside, Section Chief The actual length inside, and These are the maximum and minimum values ​​in the Y direction within each segment; Step 2.2.2. For point clouds that satisfy the above characteristics, continue to extract features using the double-ended elevation sliding window method, and extract elevation difference features through sliding window search to obtain the point set. The sub-steps for extracting the dual-end elevation sliding window feature are as follows: Step 2.2.2.1: First, map the point cloud data onto the two-dimensional plane XOY, and then determine the size of the search sliding window based on the actual width of the waterfront edge. and step length In the lidar scanning line The above point cloud uses size and step length A sliding window is used for searching. In each search, the Z values ​​of all points are sorted in descending order, and then the first and last points of the sequence are selected respectively. These values ​​serve as the maximum and minimum reference set.

[0083]

[0084] in, for value sequence first The set of the largest numbers, for value sequence after The set of the smallest numbers, Total points within each search window; Step 2.2.2.2: Then, the elevation differences in the search window, including the median difference and the mean difference, are added to the candidate set if they meet the following conditions:

[0085]

[0086]

[0087]

[0088] in, and These represent the difference in median and the difference in mean within the search window, respectively. and These are the threshold values.

[0089] If a point that meets the above conditions is found on the current scan line, it is added as a candidate point. The search on the current scan line is stopped, and the same search is performed on the next scan line. Step 2.2.3, in the window point set Search for the set of features that satisfy the smoothness coefficient threshold. For point Within a small neighborhood, there exist neighboring points that enable connection. The angle between the line and the two sides should be an arc or a straight line, and the smoothness coefficient feature should satisfy the following conditions:

[0090] Where v 1eft and v right They are respectively The direction vectors of the leftmost and rightmost points in the neighborhood. It is the angle threshold parameter.

[0091] Step 2.2.4 involves filtering out false points caused by interference from obstacles such as rocks and vegetation using distance filtering, and selecting the set of points closest to the unmanned vessel along the scan line to obtain accurate candidate points. First, candidate boundary points are divided into left and right candidate boundary points based on the vessel's direction of travel. Then, the left and right candidate boundary points are further subdivided, segmenting the feature points along the direction of travel (parallel to the shoreline). Within each segment, the point closest to the unmanned vessel or the x-axis is selected as a candidate point. For any point... The segment number i is determined by an integer that satisfies the following conditions:

[0092] in, The length of the segment is the segment length. This represents satisfying the relation. The filtered point set, after rounding up to the nearest integer, is represented as follows:

[0093] in Let j be the y-coordinate of the j-th point within the segment. Let i be the set of point clouds in segment i. Let i be the set of the nearest points in segment i.

[0094] Step 2.2.5: Fit the feature boundary points filtered in the above steps. Fit the curve on the two-dimensional projection plane using RANSAC. Specifically, take the quadratic curve equation as an example:

[0095] in, These are the fitting parameters.

[0096] Step 2.3: Construct an evaluation function based on the cumulative error over the periodic time, and adaptively select the optimal threshold parameter according to the evaluation function; The specific steps are as follows: Step 2.3.1: Construct the key parameter vector required for segmented feature extraction of LiDAR point cloud. ; Step 2.3.2, initialize parameter θ to Alternatively, load the optimal parameters obtained from the previous training step. As initial value; Step 2.3.3: For each new frame of lidar point cloud data, feature extraction is performed using the current parameter θ. The above steps are used to extract the candidate point set of the shoreline and fit the shoreline equation. Step 2.3.4: Evaluate the detection results by calculating the cumulative error over the periodic time, and calculate the evaluation function index of the shoreline on both sides of the river channel. ; The calculation process for the evaluation indicators is as follows: Figure 4 As shown, the shoreline may be a discontinuous curved segment, through... Horizontal scanning, when intersecting with the left and right boundary points of the river channel, presents four possibilities: both left and right boundaries exist, only the left boundary exists, only the right boundary exists, and neither left nor right boundary exists. Considering the cumulative error over time, the coordinate errors of the points are calculated for each of these four cases to evaluate the detection results of the river channel boundary line model. The evaluation steps are as follows: Step 2.3.4.1: For t The projection image of the river channel point cloud onto the XOY plane at any given time is obtained by using... Perform a horizontal scan with a scan step size of [missing information]. h ,Will Substitute into the detection model to calculate the corresponding left and right boundary points y coordinate and Based on the completeness of the detected left and right boundary curve segments, t The time-based detection results are divided into four categories, and both left and right boundary points exist. Only the left boundary point exists. Only the right boundary point exists. Neither the left nor right boundary points exist. Four categories of sets; Step 2.3.4.2: Calculation t Time-based river boundary detection error

[0097]

[0098]

[0099] in, yes t Number of horizontal scan points at any given time , and These are the true values ​​of the x-coordinates of the left and right boundary points. and For predicted values, It is the L1 norm. It represents rounding up; Step 2.3.4.3: Calculation Period T Average error of boundary point detection within time ,

[0100] Step 2.3.4.4: Take the reciprocal of the cumulative error over the period as the evaluation index. ; Step 2.3.5: Determine and adjust parameter values, and set performance thresholds. ,if If the current parameters meet the requirements, no adjustment is needed; proceed directly to the next frame. Otherwise, it indicates that the current scene may have changed, such as the appearance of a special curve or increased sensor noise, causing the current parameters to no longer be optimal. Proceed to the next step and trigger a local search. Step 2.3.6: Extract parameters from the current shoreline. Perform a small-scale grid search or random search within a small nearby area. For each parameter, add a small perturbation. If the index increases, continue in the same direction; otherwise, reverse the direction or reduce the step size. The parameter search formula is as follows:

[0101]

[0102] in, It is a parameter The i-th search update parameter, The current parameter The i One parameter, It is the first i The parameter is the first The search decay step size in the next iteration, It is the first i Initial search step size (parameters) It is the decay index. The symbolic function is defined as follows:

[0103] Step 2.3.7, rapid evaluation: For each candidate parameter, quickly calculate the feature extraction and evaluation metrics once. ; Step 2.3.8, update parameters if in local search If the metric is larger, update the current parameters and use the optimal parameters. Set as new parameter ; Step 2.3.9: Repeat the above steps until the maximum number of iterations is reached or the evaluation metric stops updating, then output the current optimal parameters. θ *

[0104] Step 2.4: Obtain the final shoreline boundary feature points and shoreline equations based on the optimal parameters obtained in the previous step; Step 3: In the second stage, the image domain shoreline equation and the point cloud domain three-dimensional shoreline equation are transformed to a unified coordinate system, and a visual degradation confidence model, a point cloud multidimensional geometric quality confidence model, and a cross-modal confidence competition rule are constructed. Based on the confidence level of different sensors in the local area, the fused shoreline is adaptively generated.

[0105] Step 3 includes the following sub-steps: Step 3.1: Align the two coordinate systems through sensor calibration and project the LiDAR shoreline feature points onto the image plane; Camera and LiDAR calibration employs a checkerboard calibration method, observing the checkerboard grid from multiple angles to calculate the transformation matrix between them. Assuming points... The position coordinates relative to the lidar and camera are respectively expressed as: and ,point The pixel coordinates projected onto the image plane are The mathematical transformation relationship between the lidar coordinate system and the pixel coordinate system is as follows:

[0106]

[0107] in It is a 3×3 matrix. It is a 3×1 vector. and Abbreviated as and , represent the rotation matrix and translation vector between the lidar and camera coordinates, respectively, and are the external parameters for joint calibration. and Representing the x-coordinates and ordinate Focal length on the axis, in pixels These are the coordinates of the optical center in the image coordinate system. These are the camera's internal parameters that reflect the camera's inherent properties.

[0108] The joint calibration of external parameters uses the chessboard as the target point, and calculates the 3D point cloud according to the above formula. Projected onto two-dimensional image points And calculate its relationship with the actual image points. The absolute error between them is solved by minimizing the total loss to obtain the parameters. and This addresses the collaborative calibration problem. The objective loss function is minimized as follows:

[0109] in It is the L2 norm. This is the number of markers.

[0110] Due to the influence of waves and ship vibrations, calibration errors may increase over time. The accuracy of the transformation matrix after joint calibration must be confirmed before camera and LiDAR fusion. The projection error is calculated using the following formula. If the projection error is large or the point cloud shows a significant offset in the image, the calibration results need to be corrected again until the error meets the requirements. The calculation formula is:

[0111] in These are the pixel coordinates of the projection point. These are the actual extracted pixel coordinates. This is the number of test points.

[0112] Step 3.2, Data Association and Uncertainty Modeling: Calculate the confidence scores of the two sensors to determine the fusion weights. The calculation steps are as follows: Step 3.2.1 Calculate the confidence score of the multidimensional perceptual quality of the camera visual image. ,

[0113]

[0114] and These are the local neighborhood boundary smoothness index and the water surface highlight degradation index, respectively, for the boundary point i in the camera image. and These are the corresponding weights, and the specific calculation method is as follows: (1) Local boundary smoothness index of waterfront, which describes the continuity of the waterfront boundary within the neighborhood of the feature point. The value is in the range of [0,1]. The larger the value, the more continuous the boundary. The calculation formula is as follows:

[0115] Represented by boundary feature points The set of m local points selected centered on the boundary point constitutes the neighborhood boundary point set. , This represents the number of boundary points within the neighborhood. The vertical distance from the j-th boundary point in the neighborhood to the locally fitted line; The local neighborhood scale normalization coefficient is taken as the diagonal length of the rectangle circumscribed at the boundary point of the neighborhood. Let the minimum and maximum values ​​of the x and y coordinates in the neighborhood point set be respectively... and ,but ; (2) Local light degradation index of water surface The degree of sunlight reflection on the water surface is quantified using image information. Highlight areas on the water surface are extracted using a grayscale threshold T. The ratio of the number of local highlight pixels to the total number of pixels in the water surface area is calculated, with a value ranging from [0,1]. The larger the value, the stronger the reflection on the water surface. The specific calculation formula is as follows:

[0116]

[0117] in, For the image in pixels ( x , y The brightness value at () location; Represents pixels ( x , y Is it a water surface highlight pixel? This represents the local water surface neighborhood centered at the i-th boundary point; T The highlight detection threshold can be adaptively set according to the statistical characteristics of image brightness, and is set to 220-245 in this embodiment; This represents the total number of pixels in the local neighborhood water surface region. This represents the number of highlight pixels within a local neighborhood of the water surface. Step 3.2.2 Calculate the confidence level of the multidimensional geometric quality of the lidar point cloud.

[0118]

[0119]

[0120] and Let be the local effective feature point cloud density, neighborhood elevation stability, and distance attenuation index of the i-th laser point, respectively. , and These are the corresponding weights, and the specific calculation method is as follows: (1) Effective local feature point cloud density To assess whether the number of point clouds extracted near the boundary position is sufficient, all points with a radius of r from the i-th point cloud are constructed as the local neighborhood of that position, and the local point cloud density is calculated:

[0121] in, For the first i The number of valid shoreline points within the neighborhood of each candidate point This represents the number of reference point clouds in the boundary neighborhood, determined based on the statistical mean of the number of effective shoreline neighborhood points in the training set. (2) Elevation stability index This measures the elevation stability of characteristic points along the shoreline, ranging from [0,1]. A larger value indicates greater stability.

[0122]

[0123] For sample points The elevation value of the k-th point within the local neighborhood; The average elevation within this neighborhood; The number of neighboring points represents the number of point clouds within a local area. The standard deviation of elevation within a local neighborhood of a feature point on the waterfront boundary measures the degree of local elevation undulation. It is a reference threshold for elevation stability, used to measure the magnitude of elevation fluctuations. A smaller value is used for the water boundary of a flat city, and a larger value is used for the water boundary with greater fluctuations.

[0124] (3) Distance decay index

[0125] The distance attenuation index is weighted based on the distance in the LiDAR point cloud. The greater the distance, the larger the lateral error due to angular resolution, and the lower the confidence level. The calculation formula is as follows:

[0126] in, This is the distance from the laser point to the sensor; The minimum no-attenuation distance threshold; The maximum effective distance; The attenuation coefficient is less than 100%. At that time, the index is 1, representing complete confidence; in Between these distances, the decay is exponential based on distance; when the distance exceeds... A time index of 0 indicates unreliability. Step 3.3: Based on the cross-modal feature point confidence competition mechanism, adaptive optimization is performed on the candidate feature points of the shoreline. Neighbor feature point matching is performed between visual points and laser points to obtain matching pairs. Then, based on the confidence levels of camera images and LiDAR, a cross-modal confidence competition rule is constructed to select reliable shoreline feature points and eliminate outliers. The rules are as follows:

[0127]

[0128] in, It is the spatial distance of the i-th matching point, representing spatial consistency. It is a spatial distance threshold. It is the confidence difference threshold. It is the minimum confidence threshold that meets the requirements. and These are lidar and camera visual boundary matching or sampling points, respectively. Indicates the relationship with the first i A visual point matched with a laser point; prediction hold means that when the two conflict and their confidence levels are close or the prediction hold candidate ratio exceeds the threshold, it indicates that the current observation is unreliable. In this case, the predicted shoreline from the previous moment is used to maintain the shoreline in order to avoid abrupt changes in the shoreline.

[0129] Step 3.4: For the selected and retained feature points, perform weighted least squares curve fitting with their confidence levels as weights, and calculate the equation of the fused curve. It is assumed that the local shoreline curve conforms to a quadratic polynomial model.

[0130] Set curve parameters Construct the cost function This makes the laser point set and visual point set The parameters of the fused curve are determined by minimizing the weighted sum of squared longitudinal residuals relative to the fitted curve.

[0131] in It is the confidence level of the laser point. The confidence scores for the camera's viewpoints are calculated as follows:

[0132]

[0133]

[0134] and For LiDAR and camera modal weights, and The mean confidence scores for the two modalities are given. and They are and The number of midpoints To ensure that the denominator is a very small number, to prevent the fraction from being zero; Solving in matrix form The optimal analytical solution for the parameters is:

[0135] Where W is and The weight matrix, variable parameter matrix X, and observation vector Y are composed of x and y coordinates, respectively.

[0136]

[0137] Step 3.5: After fitting the fusion curve, an improved particle filter algorithm is used for temporal tracking and calibration to prevent single-frame detection jumps; The improved particle filter algorithm utilizes the fused shoreline curve parameters, visual confidence, lidar confidence, and cross-modal confidence competition results obtained in the previous step to construct an observation confidence gating factor. Based on this observation confidence gating factor, the observation weights, prediction-preservation ratio, and particle perturbation intensity of the particle filter are adaptively adjusted. This improves the tracking and fused shoreline response speed when the detection results in the current frame are reliable, and suppresses shoreline jumps when reflections, occlusions, sparse point clouds, or cross-modal conflicts exist in the current frame. The specific steps are explained below: To improve the continuity and stability of the river boundary model, the detected model results were calibrated using an improved particle filter to track and calibrate the parameters of the shoreline model. To enhance robustness in complex aquatic environments, including those with reflections, occlusions, and intricate reflections, each particle represents a probabilistic state marked by the river channel boundary. The state vector is expanded to include not only curve parameters but also the rate of change of those parameters, enhancing the predictive ability for continuous river channel deformation and capturing dynamic changes in the river channel. An augmented state vector coupled with the rate of change of parameters is constructed, defined as...

[0138] Where, 1≤ i ≤ N is the particle index, and N is the number of particles. , , , respectively, represent the coefficients of the quadratic polynomial curve model of the river boundary line corresponding to particle i at time t. This indicates the rate of change of the corresponding coefficient. Figure 5 The flowchart for calibrating the river boundary model using the particle filter algorithm is described below, with key steps as follows: Step 3.5.1: Initialize sampling of the river channel boundary model particle set. Use the fusion curve parameters obtained in Step 3.4 above in the first frame image. As the initial position, the rate of change of the parameter Initialize to 0, then initialize the sample particle set with a Gaussian distribution. The initial state of the particle set is:

[0139] Where 1≤ i ≤ N0, It is the initial number of particles. t 0 represents the initial time and the mean. For the initial position parameters , The initial covariance matrix is ​​adaptively set according to the image resolution and curve parameter scale. The implementation example uses the following values:

[0140] Step 3.5.2, calculate the observation confidence gating factor. Based on the mean visual confidence score, mean lidar confidence score, and cross-modal competition results obtained in the previous step, calculate the observation confidence gating factor for the current frame:

[0141] Here are the weighting coefficients, and the min and max functions represent taking the minimum and maximum values, respectively. The average confidence level of the visual detection results in the current frame; This represents the average confidence level of the lidar detection results in the current frame. For cross-modal consistency confidence; To maintain the label for prediction; The cross-modal consistency confidence Calculate based on the matching distance between the visual boundary point and the lidar boundary point from the previous step:

[0142]

[0143]

[0144] in, Let be the local cross-modal consistency confidence of the i-th matching point pair. The number of matching points. The spatial distance threshold, For the i-th laser point cloud in the current frame t, match the visual point pair ( , Spatial distance.

[0145] The prediction hold flag Based on cross-modal conflict determination, for the i-th matching point pair, if the spatial distance between the visual boundary point and the lidar boundary point is greater than a threshold, and the confidence difference between the two is less than the confidence difference approaching the threshold, then the shoreline predicted at the previous moment needs to be used, and it is considered that this position needs to be predicted and preserved. The calculation formula is as follows:

[0146]

[0147]

[0148] in, To maintain the proportion for prediction of the current frame, Maintain the state for the local prediction of the i-th matching point pair. For the confidence level to be close to the threshold, To predict the retention ratio threshold.

[0149] Step 3.5.3, Particle State Prediction. Each particle's state is predicted according to the state transition model:

[0150]

[0151] Where A is the state transition matrix. The frame interval is set to 1 frame in the implementation example, with process noise... Follows a Gaussian distribution. , The noise covariance matrix for the current frame is a diagonal matrix, with its diagonal elements corresponding to the noise variance of each curve parameter and its rate of change. The noise matrix... Based on the observed reliable gating factor Adaptive adjustment:

[0152] in, and These represent the minimum and maximum perturbations when observations are reliable.

[0153] Step 3.5.4: Calculate particle weights based on the observation reliability gating factor. The particle weights are primarily determined by the gating errors of the current fusion curve and the historical prediction curves, avoiding shoreline jumps caused by anomaly detection in the current frame. Let the candidate curve generated by the i-th particle prediction be... Based on the current frame fusion curve obtained in step 3.4 above... The predicted curve at the previous moment They are represented as follows:

[0154]

[0155]

[0156] in, For the fusion curve parameters in step 3.4, The parameters for the predicted curve at the previous time step are used. The weight calculation process is as follows: (1) Select on the curve Using uniform sampling points, calculate the error of the particle relative to the current fused observation:

[0157] (2) Calculate the error of the particle relative to the historical prediction curve:

[0158] (3) Constructing the gated error:

[0159] (4) Calculate and normalize the particle weights:

[0160]

[0161] When the fusion result is reliable The particle weight is mainly determined by the degree of error between the historical prediction curve and the current fusion curve, and vice versa. It is determined solely by the degree of error compared to historical prediction curves; Step 3.5.5, Adaptively adjust the particle number. The particle number is dynamically adjusted based on the observation confidence gating factor. :

[0162] and These are the upper and lower limits of the particle number. This represents rounding down; when the current frame is reliable, the particle number is close to... To reduce the number of particles and improve computational efficiency; when the current frame is unreliable, the number of particles is close to Increase the number of particles to improve tracking accuracy.

[0163] when The process involves copying high-weight particles from the existing particles and adding noise to increase the particle count. This copying method is performed according to step 3.5.6 of the gated elite resampling method described below. Discard the one with the smallest weight. One particle.

[0164] Step 3.5.6, Gated Elite Resampling. To prevent particle degradation, an adaptive resampling method guided by gated elites is constructed. When the number of effective particles is too low or there are too many low-weight particles, i.e. or ) Resampling is performed at certain times, among which For count values, and Using a threshold as a reference, the resampling adaptively selects particles with higher weights to replace particles with lower weights. The detailed steps are as follows: Step 3.5.6.1: Sort the particles according to their normalized weights. ; Step 3.5.6.2, select The particles with high weight are designated as elite particles. ,in, The proportion coefficient of elite particles; thus, the set of elite particles is obtained. The elite particles are copied to a new particle set while maintaining their state, and the remaining particles are copied from this set through the following steps. One particle; Step 3.5.6.3, construct the cumulative distribution function ; Step 3.5.6.4: Generate uniform sampling points , ; Step 3.5.6.5, for each ( ), find satisfaction If the index k of the condition is used, then the particle is copied. ; Step 3.5.6.6: Add adaptive random perturbation to the non-elite particles obtained from the resampling. , The perturbation variance is also controlled by the observation confidence gating factor:

[0165] in, and These represent the minimum and maximum perturbation variances when the observations are reliable.

[0166] Step 3.5.6.7: Repeat steps 3.5.6.5 and 3.5.6.6 above until the remaining parts are copied. One particle; Step 3.5.6.8: The weights of all new particles are reinitialized. ; Step 3.5.7, State Estimation and Gated Output. By introducing a weighted threshold screening mechanism, high-quality particles are selected for weighted fusion, and low-weight particles are removed to suppress the interference of abnormal particles, thereby improving the robustness and stability of the river boundary line estimation. The calculation process of the state estimate values ​​of the model parameters is as follows:

[0167]

[0168] in This represents an indicator function used to filter out high-quality particles. The particle weight threshold is the final output parameter. Smooth the output based on the gating factor:

[0169] When the gated output determines that the current frame detection is reliable, the output result is biased towards updating to the current frame calibration result; when the current frame detection is unreliable or prediction preservation is required, the output result is biased towards retaining the historical prediction result; the final output result... The first three parameters are the quadratic polynomial curve parameters of the river boundary line after calibration at the current time. They are used as the final output of the current time t and as the historical prior calculation value for the next time.

[0170] Step 4: In the third stage, a dynamic safety distance judgment function based on shoreline attribute quantification and dynamic constraints is constructed, and the navigable area of ​​the river surface is calculated according to the shoreline equation and the dynamic safety distance. The flowchart for calculating the navigable area of ​​a river channel based on the shoreline equation is as follows: Figure 6 As shown, assuming the equations for the left and right shoreline segments detected by the camera and lidar fusion in the previous step are:

[0171]

[0172] like Figure 7 and Figure 8 As shown, the X-axis points forward of the unmanned vessel, the Y-axis points to the right side of the hull, and the right bank is to the right of the left bank, satisfying the condition... The core steps for determining navigable areas are as follows: Step 4.1: First, determine the forward X-coordinate range. The navigable area exists within the X-coordinate range jointly covered by the curves on both the left and right banks. Determine the value range of X:

[0173] Step 4.2: Construct a dynamic safety distance judgment function based on shoreline attribute quantification and dynamic constraints. Shoreline environmental attributes include stiffness coefficient and ecological sensitivity. Construct a dynamic distance for the physical shoreline to contract towards the water center, which is the minimum dynamic safety distance judgment function that the unmanned vessel needs to maintain with the left and right shores. , as follows:

[0174]

[0175] in, Based on the safe distance, , For the width of the boat, Based on the basic safety margin, , The curvature of the left and right boundary lines; , , These are the corresponding weighting coefficients; the core parameters are explained below: (1) Stiffness coefficient ∈(0,1] represents the shoreline's ability to resist deformation or the degree of damage after a collision. The larger S is, the harder the riverbank boundary material, the more serious the consequences of a collision, and the higher the safety distance requirement. In the example, the values ​​are concrete / rock S=1.0, soil S=0.6, and vegetation S=0.3. (2) Ecological sensitivity ( x )∈(0,1], representing the environmental vulnerability of the area. The more ecological protection area there is, the higher the safety distance requirement. In the example, the values ​​are E=1.0 for ecological protection area, E=0.5 for natural shoreline, and E=0.1 for artificial shoreline. (3) Dynamic constraints This means that the faster the speed and the sharper the curve, the greater the required safety margin. This is the current speed of the unmanned vessel. Step 4.3: Generate safety boundary curves based on shoreline equations. Calculate the safety boundary curve equations for the left and right banks using the dynamic safety distance judgment function. and The parametric equation is expressed as follows:

[0176]

[0177] in, , and The unit normal vector pointing into the river channel x-axis and y-axis components:

[0178]

[0179] Step 4.4: Calculate the channel centerline and the channel normal width; The channel width is calculated based on the centerline normal vector to avoid underestimating the true width in curved waterways, as the traditional lateral width calculation method might do.

[0180]

[0181] in, Here is the equation for the channel centerline, with the normal vector perpendicular to the tangent to the centerline. This is the correction factor from the horizontal width to the normal width; Step 4.5: Calculate the water depth safety function using water depth measurements from a small multibeam echo sounder to determine underwater navigability. Perform water depth safety assessments on any candidate point within the navigable boundary to avoid shoals. Define the water depth safety function. :

[0182] in, For point The measured water depth (m) is given by T, where T is the design draft of the unmanned vessel (m), and ΔT is the water depth margin, which can be adjusted for factors such as waves, attitude changes, and measurement errors. This refers to the amount of sinking a ship produces while sailing. v is the speed. This is an empirical coefficient. Captain; Step 4.6: Determine channel passability based on channel safety width and water depth safety. The judgment method is as follows:

[0183] Step 4.7, if the waterway is passable, Then construct a parsed navigable area expression for any point. The formula for determining whether a point belongs to the navigable area Ω is as follows:

[0184] Step 4.8, if Determine if the current speed allows passage at a reduced speed. If yes, return to step 4.2 to recalculate the passability; otherwise, output a warning message indicating impassability and wait for manual intervention. Step 4.9: Based on the navigable area obtained from the above steps, control the unmanned vessel to travel along the centerline of the navigable area or a safe planned path.

[0185] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0186] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting navigable areas in waterways for unmanned vessels, characterized in that, Includes the following steps: Acquire river channel image data and lidar point cloud data, and process them to obtain the water-shore boundary lines in the image domain and the water-shore boundary lines in the point cloud domain. A cross-modal competitive fusion strategy is used to fuse the waterfront boundary lines in the image domain and the point cloud domain to obtain the fused waterfront boundary line. Based on the integrated waterfront boundary line and the established dynamic safety distance markers, the waterway surface is marked as a navigable area.

2. The method for detecting navigable waterways for unmanned vessels according to claim 1, characterized in that, The acquisition of the image domain waterfront boundary line includes the following steps: The model is based on the lightweight BiSeNetV2 network, and a joint loss function is constructed by introducing central difference convolution, adding boundary gradient consistency supervision loss and boundary continuity loss. The model training and river region segmentation are completed based on the joint loss function. Then, edge detection and edge fitting are performed based on the segmentation results to obtain the waterfront boundary line in the image domain.

3. The method for detecting navigable waterways for unmanned vessels according to claim 1, characterized in that, The acquisition of the point cloud domain waterfront boundary line includes the following steps: The point cloud data is classified into clusters and segments. Then, the feature points of the shoreline and the shoreline boundary equation of the point cloud domain are obtained by cluster-segment cooperative density constraint, double-ended elevation sliding window, smoothing coefficient feature extraction, and distance filtering. The dual-end elevation sliding window feature extraction includes: projecting the three-dimensional point cloud onto the XOY plane, searching along the LiDAR scan line using a sliding window of a set size, arranging all points in each window in descending order of Z value, selecting the first and last m values ​​of the sequence as the maximum and minimum reference sets respectively, calculating the difference between the median and the mean of the two reference sets, and if both differences are within a preset threshold range, then adding the points in the window to the candidate set; An evaluation function based on the cumulative error over time is constructed, and the optimal threshold parameter is adaptively selected according to the evaluation function. Specifically, this includes: constructing the key parameter vector required in the point cloud feature extraction process; for each frame of point cloud data, performing feature extraction using the current parameters and fitting the shoreline equation; evaluating the detection results through the cumulative error over time and calculating the shoreline evaluation function index; if the index is lower than the preset performance threshold, the parameters are locally optimized and updated until the optimal parameters are output for the next frame. The final shoreline boundary feature points and shoreline equations are obtained based on the optimal parameters.

4. The method for detecting navigable waterways for unmanned vessels according to claim 3, characterized in that, With the set scan step size t The point cloud projection image of the river channel is horizontally scanned, and the integrity of the detected left and right boundary curve segments is determined. t The time-based detection results are divided into four categories: both left and right boundary points exist, only the left boundary point exists, only the right boundary point exists, and neither left nor right boundary point exists. For each scan line, calculate the distance error between the predicted point and the ground truth point according to its category; if the boundary does not exist, the error is recorded as zero; the average detection error of the frame is obtained by averaging the errors of all scan lines. Calculate the average detection error within period T, and take its reciprocal as the evaluation function index for the shoreline.

5. The method for detecting navigable waterways for unmanned vessels according to claim 1, characterized in that, The acquisition of the integrated waterfront boundary line includes the following steps: By calibrating the sensor, the coordinates of the lidar are aligned with the coordinates of the image, and the feature points of the lidar shoreline are projected onto the image plane. Data association and uncertainty modeling are used to calculate the confidence scores of the two sensors to determine the fusion weights, namely the confidence scores of the multidimensional perceptual quality of visual images and the multidimensional geometric quality of LiDAR point clouds. Based on a cross-modal feature point confidence competition mechanism, adaptive optimization is performed on candidate feature points of the shoreline; For the feature points retained after optimization, weighted least squares curve fitting is performed with their confidence levels as weights, and the equation of the fused curve is calculated. An improved particle filter algorithm is used for temporal tracking and calibration after the fusion curve fitting to prevent single-frame detection jumps.

6. A method for detecting navigable waterways for unmanned vessels according to claim 51, characterized in that, Confidence of multidimensional perceptual quality of visual images , and These are the local neighborhood boundary smoothness index and the water surface highlight degradation index, respectively, for the boundary point i in the camera image. and These are the corresponding weights, calculated as follows: Waterfront Local Boundary Smoothness Index The value ranges from [0,1]. A larger value indicates a more continuous boundary. The calculation formula is as follows: Represented by boundary feature points The set of m local points selected centered on the boundary point constitutes the neighborhood boundary point set. , This represents the number of boundary points within the neighborhood. The vertical distance from the j-th boundary point in the neighborhood to the locally fitted line; The local neighborhood scale normalization coefficient is taken as the diagonal length of the rectangle circumscribed at the boundary point of the neighborhood. Let the minimum and maximum values ​​of the x and y coordinates in the neighborhood point set be respectively... and ,but ; Localized Highlight Degradation Index The value ranges from [0,1]. A larger value indicates a stronger reflection from the water surface. The calculation formula is as follows: in, For the image in pixels ( x , y The brightness value at () location; Represents pixels ( x , y Is it a water surface highlight pixel? This represents the local water surface neighborhood centered at the i-th boundary point; T The threshold for determining highlights; This represents the total number of pixels in the local neighborhood water surface region. This represents the number of highlight pixels within a local neighborhood of the water surface. LiDAR point cloud multidimensional geometric quality confidence and Let be the local effective feature point cloud density, neighborhood elevation stability, and distance attenuation index of the i-th laser point, respectively. , and These are the corresponding weights.

7. A method for detecting navigable waterways for unmanned vessels according to claim 5, characterized in that, Based on image confidence and LiDAR confidence, a cross-modal confidence competition rule is constructed for the matched visual feature points and LiDAR feature points. This rule adaptively selects reliable shoreline feature points and eliminates outliers. The competition rule is as follows: in, It is the spatial distance of the i-th matching point, representing spatial consistency. It is a spatial distance threshold. It is the confidence difference threshold. It is the minimum confidence threshold that meets the requirements. and These are lidar and camera visual boundary matching or sampling points, respectively. Indicates the relationship with the first i A visual point matched with a laser point; prediction hold means that when the two conflict and their confidence levels are close or the prediction hold candidate ratio exceeds the threshold, it indicates that the current observation is unreliable. In this case, the predicted shoreline from the previous moment is used to hold the shoreline to avoid abrupt changes in the shoreline.

8. A method for detecting navigable waterways for unmanned vessels according to claim 5, characterized in that, The improved particle filter algorithm includes the following steps: Initialization: In the first frame image, an augmented state vector is constructed using the initial shoreline curve parameters and their rate of change obtained by multi-sensor fusion. The augmented state vector includes curve parameters and parameter change rates. The particle set is initialized by sampling with a Gaussian distribution. Calculate the observation confidence gating factor: Based on the average confidence of visual detection, the average confidence of lidar detection, the cross-modal consistency confidence, and the prediction preservation flag of the current frame, calculate the observation confidence gating factor used to quantify the reliability of the fused detection results in the current frame. Here are the weighting coefficients, and the min and max functions represent taking the minimum and maximum values, respectively. The average confidence level of the visual detection results in the current frame; This represents the average confidence level of the lidar detection results in the current frame. The cross-modal consistency confidence score is calculated based on the matching distance between the visual boundary point and the lidar boundary point. To predict the retention flag, the predicted retention flag is determined based on the degree of conflict and the degree of confidence similarity between the visual and lidar signals; Particle state prediction: Each particle is predicted according to the state transition model, wherein the process noise covariance matrix is ​​adaptively adjusted according to the observation confidence gate factor. When the observation confidence gate factor is large, the process noise is reduced, and when the observation confidence gate factor is small, the process noise is increased. Particle weights are calculated based on gating error: The observation error of a particle relative to the current frame's fusion curve and the prediction error relative to the historical prediction curve from the previous time step are calculated separately. The observation error and the historical prediction curve error are then fused using the observation reliability gating factor to obtain the gating error. The particle weights are calculated and normalized based on this gating error. When the fusion result is reliable, the reliability gating factor is... The particle weight is mainly determined by the degree of error between the historical prediction curve and the current fusion curve, and vice versa. It is determined solely by the degree of error compared to historical prediction curves; Adaptive particle number adjustment: The number of particles in the current frame is dynamically adjusted according to the observation confidence gating factor. When the observation confidence gating factor is large, the number of particles is reduced to improve computational efficiency, and when the observation confidence gating factor is small, the number of particles is increased to improve tracking accuracy. Gated elite particle resampling: Elite particles with high weights are directly retained in the new particle set. The remaining non-elite particles are generated by copying from all particles through system resampling. An adaptive random perturbation is added to the copied non-elite particles. The variance of the adaptive random perturbation is controlled by the observation reliability gating factor. The perturbation is reduced when the observation is reliable and increased when the observation is degraded. State estimation and gated output: High-quality particles are selected by weight threshold, and the high-quality particles are weighted and fused to obtain the current frame estimation result. Then, the observation confidence gate factor is used to gate and smooth the current frame estimation result and the historical prediction result of the previous moment to obtain the calibrated shoreline curve parameters of the current frame.

9. The method for detecting navigable waterways for unmanned vessels according to claim 1, characterized in that, The marked navigable waterway area includes the following steps: Determine the coordinate range of the navigable area based on the equations of the left and right shoreline boundaries; A dynamic safety distance function is constructed. The dynamic safety distance function is based on the basic safety distance and is modified by weighting the shoreline stiffness coefficient, ecological sensitivity and dynamic constraint terms. The shoreline stiffness coefficient is used to characterize the hardness of the shoreline material, the ecological sensitivity is used to characterize the ecological fragility of the aquatic environment, and the dynamic constraint terms are determined by the product of the shoreline curvature and the square of the unmanned vessel speed. Based on the dynamic safety distance function, the left and right shoreline boundary lines are offset along the direction of the normal vector pointing into the river channel to generate the left and right safety boundary curves. Determine water depth safety by combining water depth measurement data; Based on the left and right safety boundary curves and water depth safety, an analytical expression for the navigable area is determined.

10. A system for detecting navigable river areas for unmanned vessels, characterized in that, The method according to any one of claims 1 to 9 includes a hardware device module and a software module; The hardware module includes the hull of the unmanned vessel, as well as an industrial control computer, lidar, camera or video camera, small multibeam echo sounder, network communication module, drive control board and positioning equipment installed on the unmanned vessel; The software module runs on the industrial control computer of the computing processing unit and includes a shoreline detection module and a navigable area judgment module. The shoreline detection module includes a camera image detection module, a lidar point cloud detection module, and a fusion detection module. The lidar point cloud detection module includes a point cloud feature processing module and a shoreline detection evaluation module.

Images