A method for water surface target detection based on the fusion of lidar point cloud and camera image

Abstract

This invention discloses a method for detecting water surface targets based on the fusion of LiDAR point clouds and camera images. The method includes: jointly calibrating the camera and LiDAR to obtain a coordinate transformation matrix between the point cloud and the image, achieving information fusion; filtering out surface clutter from the LiDAR point cloud and enhancing the camera image by combining water surface imaging characteristics and imaging effects; obtaining 3D bounding boxes of water surface targets using a two-stage 3D point cloud clustering algorithm combined with point cloud attribute information; using the regions of interest projected onto the camera-acquired image from the clustering results as subsequent detection inputs; and using a 2D salient target detection algorithm to obtain the precise location of targets in each region of interest. This invention proposes a two-stage point cloud clustering algorithm, which first roughly locates the boundary using Euclidean distance clustering, and then determines the precise boundary of the water surface target based on differences in point cloud distribution and intensity values.

Description

Technical Field

[0001] This invention belongs to the field of water surface target detection and recognition, specifically relating to a water surface target detection method based on the fusion of lidar point cloud and camera image. Background Technology

[0002] Surface target detection and identification is an essential function for autonomous navigation of unmanned surface vessels (USVs), applicable to maritime search and rescue, surface topography mapping, and maritime security patrols. Faced with complex and ever-changing marine environments, USVs must immediately detect targets and make decisions, requiring real-time, accurate, and robust detection algorithms. Therefore, this invention focuses on proposing a fast and interference-resistant surface target detection algorithm.

[0003] Optical cameras and lidar are commonly used sensing devices on unmanned surface vessels (USVs). They produce rich image features, suitable for distinguishing target categories on the water surface; however, interference such as overexposure or reflection can affect image quality, reducing detection accuracy. Furthermore, USVs need to acquire target positions during navigation, and cameras cannot obtain target distance information. Lidar emits a laser beam and receives the echo beam reflected by the target. By measuring the beam's travel time and generating a 3D point cloud, it estimates the distance to the target. This process is almost unaffected by light or weather. A drawback of lidar is that the point cloud is sparse in space, making it difficult to accurately describe target shapes. Multi-sensor fusion methods can compensate for the shortcomings of single sensors, improve detection accuracy, and expand application scenarios. However, deep learning-based methods require large datasets, which are scarce on the water surface; moreover, the types of targets on the sea surface are unknown, and the presence of waves further obstructs the view, making it difficult to collect large amounts of suitable datasets under different sea conditions. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention discloses a method for detecting water surface targets based on the fusion of lidar point clouds and camera images.

[0005] The present invention aims to provide a method for detecting water surface targets based on the fusion of lidar point clouds and camera images. The method comprises the following steps:

[0006] S1: Perform joint calibration of the camera and LiDAR to obtain the coordinate transformation matrix between the point cloud and the image.

[0007] S2: Combining water surface imaging characteristics and imaging effects, water surface clutter is filtered out from the radar point cloud, and the camera images are enhanced through preprocessing.

[0008] S3: Using a two-stage clustering algorithm for 3D point clouds, combined with point cloud attribute information, the 3D bounding box of the water surface target is obtained.

[0009] S4: Based on the coordinate transformation relationship, the clustering results are projected onto the image captured by the camera to obtain the region of interest, which serves as the input for subsequent detection.

[0010] S5: Use a 2D salient target detection algorithm to obtain the specific location of the target in each region of interest.

[0011] Furthermore, step S1 includes the following steps:

[0012] S11: Joint calibration experiment design:

[0013] The coordinate systems of lidar and camera in the unmanned vessel perception system are independent of each other. In order to achieve multi-sensor fusion, it is necessary to determine the external transformation matrix parameters between lidar and camera. The transformation matrix can be solved by obtaining the center point and normal vector of the object under different sensor coordinate systems.

[0014] S12: Solving for reference object coordinates and normal vectors in 3D point clouds:

[0015] The RANSAC algorithm is used to fit the point cloud in this space, generating a plane that matches the calibration board. The four vertices of the plane are necessarily extrema. Extrema points and their neighborhoods are selected to form vertex point cloud clusters. The edges of the fitted plane are obtained by connecting two points in two adjacent point cloud clusters. The variance of the fitted edges is then calculated, iterating for each point in the vertex cluster until the edge with the smallest error is selected, thus obtaining the four vertices of the plane. For a known plane, the coordinates of the center point can be obtained by connecting the diagonals. The normal vector of the plane can also be obtained.

[0016] S13: Solving for the coordinates and normal vector of the reference object in a 2D image:

[0017] The distribution of feature points in the acquired image is calculated, and combined with the physical parameters of the calibration board and the camera intrinsic parameter matrix, a Perspective-n-Point (PnP) problem is constructed. Then, the distance from each feature point to the camera is calculated, allowing the determination of the point's coordinates in the camera coordinate system. Given the coordinates of each corner point, and combining this with the length and width of the calibration board, the coordinates of the four vertices and the center point of the calibration board are obtained. The vector perpendicular to the two diagonals and passing through the center point is the normal vector of the calibration plate.

[0018] S14: Based on the collected data samples, solve for the coordinate transformation matrix between the lidar and the camera:

[0019] By repeating the sample acquisition process multiple times, the set of center points O in both the radar and camera coordinate systems can be obtained. L O C and the set of normal vectors N L N C Then, the transformation matrix from the lidar to the camera is obtained. The rotation matrix R from the point cloud to the pixel is...LC This can be obtained from the following relationship:

[0020]

[0021] Similarly, the translation matrix T can be obtained. LC =mean(O C -O L ).

[0022] Furthermore, step S2 includes the following steps:

[0023] S21: Remove point clouds formed by reflections from the water surface:

[0024] Based on the imaging characteristics of water targets, all targets to be detected must be floating on the water surface. Therefore, the point cloud formed by the water surface represents the lowest point in the entire scene. The 10 points with the lowest height in the point cloud are selected to obtain the average height, and points within a certain range above this height are used as candidate points for the horizontal plane.

[0025] The point cloud formed by the water surface can be represented by a planar model ax + by + cz = d. Four points are randomly selected from the candidate points to construct a plane, obtaining the parameters of the planar model. Then, all candidate points are substituted into the planar model to obtain the distance from each point to the plane, and points with distances less than a threshold T are selected. sur Points within the specified range are designated as inliers, and their count is recorded. If the number of inliers in the current model is greater than that in the optimal model, the model parameters are updated, and the current model is adopted as the optimal model. This process is repeated until model parameters with the maximum number of inliers are obtained, or the number of iterations terminates. After fitting the horizontal plane, points in the point cloud that are less than ε away from the horizontal plane are considered water surface points and are removed from the original point cloud.

[0026] S22: Optical Image Enhancement:

[0027] To enhance optical images, a dehazing algorithm based on dark channel prior images is used, taking into account the characteristics of water surface images. In computer vision and computer graphics, the image model containing fog is represented as follows:

[0028] I(x)=J(x)t(x)+A(1-t(x)) (2)

[0029] Where I(x) is the acquired image, J(x) is the desired image after restoration, A is the global atmospheric light intensity, and t(x) is the refractive index of the propagation medium. After obtaining the global atmospheric light intensity A and the refractive index t(x) of the propagation medium, the enhanced image J(x) after dehazing can be obtained.

[0030] Furthermore, step S3 includes the following steps:

[0031] S31: Obtain 3D bounding boxes based on Euclidean clustering as input for the next stage of clustering:

[0032] Euclidean distance-based point cloud clustering is a density-based algorithm characterized by fast response and robustness. This algorithm calculates the distance between two points in space and compares it to a threshold to determine if they belong to the same target. The algorithm inputs are: cluster radius R, minimum number of points η in a single cluster, and the point cloud set P after filtering for water surfaces. sur The algorithm outputs the 3D bounding boxes and corresponding point clouds of the detected water surface targets, which are also the inputs for the two-stage detection.

[0033] (1) For non-water surface dataset P sur Each point in the cluster is numbered by index. During the clustering process, height information is not considered. All points are projected onto the XdY plane, i.e., Z-axis information is ignored, in order to reduce the sparsity of the point cloud. (2) Select an initial point p i ∈P sur This forms a non-empty neighborhood with a clustering radius r centered at that point. If the neighborhood is empty, delete the index of that point and repeat this step. (3) If the neighborhood is not empty, create cluster C. i , will p i All points within the selected neighborhood are assigned to this cluster. (4) Then, for each point in the selected neighborhood, a search is performed with a radius of r to obtain a new point cloud and the index is recorded in cluster C. i In the middle, repeat this process until no new points appear. (5) If cluster C i If the number of midpoints N < η, it is considered that the detected target contains too few points, which may be a false detection, and cluster C is deleted. i If N > η, then the detection is successful, and C is obtained based on the index. i The maximum and minimum values ​​in the (x,y,z) coordinates are used as the 3D bounding box for the detected target. (6) Finally, C i Internal index from P sur Delete the points and repeat step 2 until the index contains no more points. The algorithm output is the 3D bounding box and corresponding point cloud of the detected targets on the water surface, which is also the input for the two-stage detection.

[0034] S32: Based on the two-stage clustering, refine the results of the previous stage:

[0035] In the second stage of clustering, the 3D point cloud is first converted into a hash table, similar to pixels in a 2D image. An element in the hash table with rows (row) and columns (col) can be represented as ind = row * W + col, where W is the number of LiDAR scan lines. Furthermore, an adaptive clustering radius is employed, such as... Figure 4 As shown, p ip is the current center point. i+1 As neighboring points randomly distributed in space, the angle Δθ between the lines connecting them to the origin is the horizontal resolution of the lidar. i+1 The coordinates are uncertain. We obtain the adaptive clustering radius R by setting a parameter λ:

[0036]

[0037] The point cloud is converted into a hash table, and the second stage of clustering begins. A neighborhood search is performed, centered on an element in the table, until all points have been traversed. This invention uses a clustering radius that considers the distribution of the point cloud, expressed as follows:

[0038]

[0039] Where (Δx, Δy, Δz) represents the coordinate difference between two points, and a, b, and c are adjustment parameters. For adjacent points in the horizontal dimension of the table, due to angular resolution, there is always a fixed offset on the X-axis. In this case, parameter a is reduced, and parameters b and c are increased, ignoring the impact of the offset on the X-axis, assuming that the discontinuities are mainly caused by changes in other axes. For adjacent points in the vertical dimension, which are distributed along the Z-axis with a fixed angular resolution, c should be reduced, and a and b should be increased. For adjacent points with deviations in both the horizontal and vertical dimensions, their thresholds in a single dimension are calculated, and then the dot product of the vectors is calculated as the final threshold. The mutual distance between two points is compared with the threshold; if it is less than the threshold, the points are considered to belong to the same object. During the clustering process, there will be points that do not belong to any category, or points that belong to multiple categories. For these points, their intensity is compared with the average intensity value of each cluster, and they are classified into the cluster with the closest intensity value. Finally, the minimum bounding box of each point cloud cluster is obtained as the output.

[0040] Furthermore, step S4 includes the following steps:

[0041] S41: Project the point cloud detection results onto the image based on the transformation matrix.

[0042] By jointly calibrating and obtaining the translation vector and rotation matrix, the point (x) in radar coordinates can be transformed. L ,y L ,z L Transform the point (x) to the camera coordinate system C ,y C ,z C The transformation relationships are as follows:

[0043]

[0044] Where T LC Let R be the translation vector between the sensors, and R be the rotation matrix.LC It can be determined by Euler angle θ LC = (α,β,γ) represents the pitch angle, yaw angle, and roll angle, respectively, and can be expressed as:

[0045]

[0046] For the detected 3D bounding box, the points on its diagonal are transformed according to formula (8), and then combined with the camera's intrinsic parameter matrix, based on z C [u,v,1] T =K[x C ,y C ,z C ] T This allows us to obtain the pixel coordinates of the target's diagonal in the image. Here, K is a known camera intrinsic parameter matrix. To ensure the integrity of the target within the region of interest, the center point of the projected region remains unchanged, while the area is proportionally increased by 20%.

[0047] Furthermore, step S5 includes the following steps:

[0048] S51: Segment the target in the candidate region using a salient target detection algorithm:

[0049] This invention employs a global contrast saliency detection algorithm. In images, most areas of scenes such as water surfaces are in the low-frequency domain, with a smaller portion being in the high-frequency domain mixed with noise and reflections. The target is in the high-frequency range relative to the background. By filtering out high-frequency noise and low-frequency background, the saliency region with the largest connected area is found, which is the object to be detected. A single bandpass filter can be represented as:

[0050]

[0051] =G(x,y,σ1)-G(x,y,σ2)

[0052] Where σ1 and σ2 are the standard deviations, and σ1 > σ2, the bandpass width of the bandpass filter is determined by the ratio of σ1 to σ2. Let σ1 = ρσ, σ2 = σ. Each Gaussian bandpass filter performs one pass through the image, filtering out high-frequency noise and low-frequency background. The complete bandpass filter can be represented as:

[0053]

[0054] The above formula performs N Gaussian difference summations at different scales to filter noise and background at different frequencies in the image. Then, the original image and the Gaussian-filtered image are transformed using Lab space, and the Euclidean distance between them is calculated, as shown in the following expression:

[0055] S(x,y)=PI ave -IGau (x,y)P (12)

[0056] Where I ave I represents the arithmetic mean of the pixels in the Lab color space. Gau (x,y) is the result after Gaussian filtering in Lab space, and S(x,y) is the saliency score map. After extracting the most salient target on the image, a binarization operation is performed, and the region greater than the threshold is the water surface target.

[0057] The beneficial effects of this invention are as follows:

[0058] (1) The water surface target detection method proposed in this invention, based on the fusion of lidar point cloud and camera image, addresses the shortcomings of a single sensor in target detection by combining data from both sensors, which is necessary because cameras cannot acquire depth information and lidar has low resolution. This expands the application scenarios of unmanned vessels. By synchronizing timestamps and resolving external calibration, a point cloud-to-pixel correspondence is established, realizing the fusion of lidar and camera data.

[0059] (2) This invention combines a three-dimensional clustering algorithm with two-dimensional salient object detection. Since the types of objects on the water surface are unpredictable, this limits the implementation of deep learning-based methods. This invention proposes to obtain the target's 3D bounding box using a point cloud clustering algorithm, project it onto the image using a transformation matrix to obtain candidate regions, and use these candidate regions as input to perform salient detection to obtain the final precise location of the target.

[0060] (3) This invention proposes a novel two-stage clustering method that utilizes point cloud distribution and attribute information to improve detection performance. Drawing upon the concept of depth map-based point cloud clustering, this invention designs a two-stage point cloud target detection method. This method first roughly locates the boundary using Euclidean distance clustering, and then determines the accurate boundary based on differences in point cloud distribution and intensity values. Attached Figure Description

[0061] Figure 1 This is a flowchart of the present invention;

[0062] Figure 2 Images of the calibration board of this invention in different sensors;

[0063] Figure 3 This is a comparison image of the water surface point cloud before and after filtration according to the present invention;

[0064] Figure 4 This is the adaptive threshold calculation method of the present invention;

[0065] Figure 5 The images show the detection results of this invention on different targets on the water surface.

[0066] Figure 6 This is a diagram showing the detection effect of the present invention on different targets on the water surface under low light conditions;

[0067] Figure 7 This is a diagram illustrating the detection effect of the present invention on targets extremely close to the water surface;

[0068] Figure 8 A comparison of the PR curves of the two-stage clustering method proposed in this invention with other clustering algorithms;

[0069] Figure 9 A comparison of ROC curves between the two-stage clustering method proposed in this invention and other clustering algorithms;

[0070] Figure 10 This invention aims to improve the detection accuracy of the proposed detection method when facing challenging environments such as water surfaces.

[0071] Figure 11 This invention aims to improve the detection accuracy of the proposed detection method for common targets on different water surfaces. Detailed Implementation

[0072] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0073] refer to Figure 1 The flowchart of a water surface target detection method based on the fusion of lidar point cloud and camera image provided by the present invention includes the following steps:

[0074] S1: Perform joint calibration of the camera and LiDAR to obtain the coordinate transformation matrix between the point cloud and the image.

[0075] S2: Combining water surface imaging characteristics and imaging effects, water surface clutter is filtered out from the radar point cloud, and the camera images are enhanced through preprocessing.

[0076] S3: Using a two-stage clustering algorithm for 3D point clouds, combined with point cloud attribute information, the 3D bounding box of the water surface target is obtained.

[0077] S4: Based on the coordinate transformation relationship, the clustering results are projected onto the image captured by the camera to obtain the region of interest, which serves as the input for subsequent detection.

[0078] S5: Use a 2D salient target detection algorithm to obtain the specific location of the target in each region of interest.

[0079] Furthermore, step S1 includes the following steps:

[0080] S11: Joint calibration experiment design:

[0081] The coordinate systems of the lidar and camera in the unmanned vessel's perception system are independent. To achieve multi-sensor fusion, the external transformation matrix parameters between the lidar and camera need to be determined. The transformation matrix can be solved by obtaining the center point and normal vector of the calibration object in different sensor coordinate systems. Due to its obvious features and ease of identification, a checkerboard calibration board is used as a reference object in this invention.

[0082] S12: Solving for the coordinates and normal vector of a reference object in a 3D point cloud:

[0083] For joint calibration, it is necessary to fit the calibration board plane in space as accurately as possible. Therefore, a three-dimensional space was manually selected, making the calibration board the only target within this range and eliminating other interference. The point cloud image of the calibration board in space is shown below. Figure 2 As shown in Figure a, the point cloud in this space is fitted using the RANSAC algorithm to generate a plane that matches the calibration board. The four vertices of the plane are necessarily extreme points. Extreme points on the plane and their neighborhoods are selected to form vertex point cloud clusters. The edges of the fitted plane are obtained by connecting two points in two adjacent point cloud clusters. Then, the variance of the fitted edges is calculated, and this process is iterated for each point in the vertex cluster until the edge with the smallest error is selected, thus obtaining the four vertices of the plane. For a known plane, the coordinates of the center point can be obtained by connecting the diagonals. The normal vector of the plane can also be obtained.

[0084] S13: Solving for the coordinates and normal vector of the reference object in a 2D image:

[0085] Image of calibration board as follows Figure 2 As shown in Figure a, before obtaining the coordinates of the calibration board, its length and width, the number of corner points of the chessboard within the calibration board, the spacing between the chessboard corner points, and the camera intrinsic matrix are used as inputs. The OpenCV built-in function `findChessboardCorners` is used to determine the distribution of chessboard corner points in the obtained image samples. Combined with the chessboard corner point spacing and the camera intrinsic matrix, a Perspective-n-Point (PnP) problem is constructed. Using the OpenCV function `solvePnP`, the depth information of each chessboard corner point can be calculated, i.e., the coordinates of the point in the camera coordinate system. Given the corner point coordinates, and combining them with the length and width of the calibration board, the coordinates of the four vertices and the center point are obtained. The vector perpendicular to the two diagonals and passing through the center point is the normal vector of the calibration plate.

[0086] S14: Based on the collected data samples, solve for the coordinate transformation matrix between the lidar and the camera:

[0087] By repeating the sample acquisition process multiple times, the set of center points O in both the radar and camera coordinate systems can be obtained.L O C and the set of normal vectors N L N C Then, the transformation matrix from the lidar to the camera is obtained. The rotation matrix R from the point cloud to the pixel is... LC This can be obtained from the following relationship:

[0088]

[0089] Similarly, the translation matrix T can be obtained. LC =mean(O C -O L ).

[0090] Furthermore, step S2 includes the following steps:

[0091] S21: Remove point clouds formed by reflections from the water surface:

[0092] LiDAR emits lasers and receives the echoes from objects to generate point clouds. On water surfaces far from the unmanned vessel, the reflection intensity is low; while on closer water surfaces, the echo intensity is high, potentially leading to false detections. Based on the imaging characteristics of water targets, all targets to be detected must float on the water surface; therefore, the point cloud formed by the water surface represents the lowest point in the entire scene. The 10 lowest-height points in the point cloud are selected to obtain the average height, and points within a certain range above this height are used as candidate points for the horizontal plane.

[0093] The point cloud formed by the water surface can be represented by a planar model ax + by + cz = d. Four points are randomly selected from the candidate points to construct a plane, obtaining the parameters of the planar model. Then, all candidate points are substituted into the planar model to obtain the distance from each point to the plane, and points with distances less than a threshold T are selected. sur Points within the specified range are designated as inliers, and their count is recorded. If the number of inliers in the current model is greater than that in the optimal model, the model parameters are updated, and the current model is adopted as the optimal model. This process is repeated until model parameters with the maximum number of inliers are obtained, or the iteration count is reached. After fitting the horizontal plane, points in the point cloud that are less than ε away from the horizontal plane are considered water surface points and are removed from the original point cloud. In this invention, the distance threshold T to the fitted plane is... sur =0.05, ε=0.05. Figure 3 (a) Original water surface point cloud image, Figure 3 (b) is a point cloud image of the target on the water surface after filtering. By comparison, it can be found that the point cloud formed by the reflection of the water surface is basically filtered out, while the point cloud formed by the target is basically not destroyed.

[0094] S22: Optical Image Enhancement:

[0095] Due to the complex and variable water surface environment, uncontrollable environmental factors such as fog, overexposure, and water surface reflection can all degrade image quality, and blurred target images can also reduce detection accuracy. To address the characteristics of water surface images, this invention uses a dehazing algorithm based on dark channel prior images to enhance optical images. In computer vision and computer graphics, the image model containing fog is represented as follows:

[0096] I(x)=J(x)t(x)+A(1-t(x)) (2)

[0097] Where I(x) is the acquired image, J(x) is the desired image, A is the global atmospheric light intensity, and t(x) is the refractive index of the propagation medium. The concept of a dark channel is introduced here for subsequent value calculations. The dark channel is defined as performing two filtering operations on the RGB channels of the original image. The size of the filter template Ω(x) is n*n; in this invention, n = 15. The specific calculation method for the dark channel is as follows:

[0098]

[0099] After obtaining the global atmospheric light intensity A and the refractive index t(x) of the propagation medium, the enhanced image J(x) after defogging can be obtained.

[0100] 1. Calculate the atmospheric light intensity A: Select the brightest 0.1% pixel in the dark channel image of I(x), that is, select the most blurred part, and then map these points to the same pixel position in the original image I(x). Select the largest value in each of the three channels of these points in the original image I(x) as the atmospheric light value A.

[0101] 2. Calculate the refractive index t(x): After performing a dark channel transformation on formula (2), we have:

[0102]

[0103] According to the principle that the closer the image is to the desired image, the closer the dark channel value is to 0, that is...

[0104] After substituting the values, we can solve for the following:

[0105]

[0106] w is a commonly used adjustment parameter; in this invention, w = 0.98. The final image after dehazing and enhancement is then obtained. Where t0 is an adjustment parameter set to prevent the refractive index t(x) of the medium from being too small, resulting in overexposure. In this invention, t0 = 0.1.

[0107] Furthermore, step S3 includes the following steps:

[0108] S31: Obtain 3D bounding boxes based on Euclidean clustering as input for the next stage of clustering:

[0109] After preprocessing, the locations of obstacles on the water surface are determined using lidar point cloud data. Clustering-based methods can well meet the requirements of real-time detection. The Euclidean distance-based point cloud clustering method is a density-based algorithm with fast response and robustness.

[0110] The algorithm's inputs are the cluster radius R, the minimum number of points η in a single cluster, and the point cloud set P after filtering the water surface. sur The algorithm outputs the detected 3D bounding boxes of water surface targets. The main process of the clustering algorithm in this invention is as follows:

[0111] (1) For non-water surface dataset P sur Each point in the cluster is numbered by index. During the clustering process, height information is not considered. All points are projected onto the XdY plane, i.e., Z-axis information is ignored, in order to reduce the sparsity of the point cloud.

[0112] (2) Select an initial point p i ∈P sur This forms a non-empty neighborhood with a clustering radius r centered at that point. If the neighborhood is an empty set, delete the index of that point and repeat this step.

[0113] (3) If the neighborhood is not empty, then create cluster C. i , will p i All points within the neighborhood are assigned to this cluster.

[0114] (4) Then, for each point in the selected neighborhood, a search is performed with a radius of r to obtain a new point cloud and the index is recorded in cluster C. i Repeat this process until no new points appear.

[0115] (5) If cluster C i If the number of midpoints N < η, it is considered that the detected target contains too few points, which may be a false detection, and cluster C is deleted. i If N > η, then the detection is successful, and C is obtained based on the index. i The maximum and minimum values ​​in the (x,y,z) coordinates are used as the 3D bounding box for the detected target.

[0116] (6) Finally, C i Internal index from P sur Delete the points and repeat step 2 until the index contains no points. The algorithm outputs the 3D bounding boxes and corresponding point clouds of the detected targets on the water surface, which are also the inputs for the two-stage detection.

[0117] S32: Based on the two-stage clustering, refine the results of the previous stage:

[0118] In the second stage of clustering, the 3D point cloud is first converted into a hash table, similar to pixels in a 2D image. An element in the hash table with rows (row) and columns (col) can be represented as ind = row * W + col, where W is the number of LiDAR scan lines. Furthermore, an adaptive clustering radius is employed, such as... Figure 4 As shown, p i p is the current center point. i+1 As neighboring points randomly distributed in space, the angle Δθ between the lines connecting them to the origin is the horizontal resolution of the lidar. i+1 The coordinates are uncertain. We obtain the adaptive clustering radius R by setting a parameter λ:

[0119]

[0120] The point cloud is converted into a hash table, and the second stage of clustering begins. A neighborhood search is performed, centered on an element in the table, until all points have been traversed. This invention uses a clustering radius that considers the distribution of the point cloud, expressed as follows:

[0121]

[0122] Where (Δx, Δy, Δz) represents the coordinate difference between two points, and a, b, and c are adjustment parameters. For adjacent points in the horizontal dimension of the table, due to the angular resolution, there is always a fixed offset on the X-axis. In this case, parameter a is reduced, and parameters b and c are increased, ignoring the influence of the offset on the X-axis, assuming that the discontinuities are mainly caused by changes in other axes. For adjacent points in the vertical dimension, which are distributed along the Z-axis with a fixed angular resolution, c should be reduced, and a and b should be increased. For adjacent points with deviations in both the horizontal and vertical dimensions, their thresholds in a single dimension are calculated, and then the dot product of the vectors is calculated as the final threshold. The mutual distance between two points is compared with the threshold; if it is less than the threshold, the points are considered to belong to the same object. During the clustering process, there will be points that do not belong to any category, or points that belong to multiple categories. For these points, their intensity is compared with the average intensity value of each cluster, and they are classified into the cluster with the closest intensity value. Finally, the minimum bounding box of each point cloud cluster is obtained as the output. The parameters used in this invention are as follows:

[0123]

[0124] Furthermore, step S4 includes the following steps:

[0125] S41: Project the point cloud detection results onto the image based on the transformation matrix.

[0126] By jointly calibrating and obtaining the translation vector and rotation matrix, the point (x) in radar coordinates can be transformed. L ,y L ,z L Transform the point (x) to the camera coordinate system C ,y C ,z C The transformation relationships are as follows:

[0127]

[0128] Where T LC Let R be the translation vector between the sensors, and R be the rotation matrix. LC It can be determined by Euler angle θ LC = (α,β,γ) represents the pitch angle, yaw angle, and roll angle, respectively, and can be expressed as:

[0129]

[0130] For the detected 3D bounding box, the points on its diagonal are transformed according to formula (8), and then combined with the camera's intrinsic parameter matrix, based on z C [u,v,1] T =K[x C ,y C ,z C ] T This allows us to obtain the pixel coordinates of the target's diagonal in the image. Here, K is a known camera intrinsic parameter matrix. Considering potential errors, to prevent the projected region of interest from containing an incomplete target and affecting subsequent detection, we keep the center point of the projected region unchanged and proportionally increase its area by 20%.

[0131] Furthermore, step S5 includes the following steps:

[0132] S51: Segment the target in the candidate region using a salient target detection algorithm:

[0133] Unique regions in an image scene are more visually noticeable. Based on this principle, saliency detection algorithms introduce foreground-background contrast, distinguishing the foreground and background by their different saliency values. After obtaining candidate regions, the color contrast between the target and the background is particularly prominent; therefore, this invention uses a global contrast saliency detection algorithm. In images, most areas of scenes such as water surfaces are low-frequency components, with a smaller portion being high-frequency components mixed with noise and reflections. The target is a high-frequency component relative to the background. By filtering out high-frequency noise and low-frequency background, the saliency region with the largest connected area is found, which is the object to be detected. A single bandpass filter can be represented as:

[0134]

[0135] Where σ1 and σ2 are the standard deviations, and σ1 > σ2, the bandpass width of the bandpass filter is determined by the ratio of σ1 to σ2. Let σ1 = ρσ, σ2 = σ. Each Gaussian bandpass filter applied to an image completes one filtering of high-frequency noise and low-frequency background. The complete bandpass filter can be represented as:

[0136]

[0137] The above formula performs N Gaussian difference summations at different scales to filter noise and background at different frequencies in the image. Then, the original image and the Gaussian-filtered image are transformed in Lab space, and the Euclidean distance between them is calculated, as shown in the following expression:

[0138] S(x,y)=PI ave -I Gau (x,y)P (12)

[0139] Where I ave I represents the arithmetic mean of the pixels in the Lab color space. Gau (x,y) represents the result after Gaussian filtering in Lab space, and S(x,y) is the final saliency score map. After extracting the most salient target in the image, binarization is performed. Regions larger than a threshold are identified as water surface targets. In this invention, the Otsu's method in OpenCV is used to automatically divide the image into foreground and background, and a suitable grayscale threshold is automatically generated by calculating the pixel variance. Based on the obtained maximum connected region, the precise target bounding box position is obtained, thus completing the detection of water surface targets.

[0140] The preferred embodiments and principles of the present invention have been described in detail above. For those skilled in the art, there may be changes in the specific implementation based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting water surface targets based on the fusion of lidar point clouds and camera images, characterized in that, Includes the following steps: S1: Perform joint calibration of the camera and LiDAR to obtain the coordinate transformation matrix between the point cloud and the image; S2: Combining the imaging characteristics and effects of the water surface, the radar point cloud is clutter-filtered to remove the point cloud formed by water surface reflection, and the camera image is enhanced and preprocessed. S3: Using a two-stage clustering algorithm for 3D point clouds, combined with point cloud attribute information, the 3D bounding box of the water surface target is obtained; S31: Obtain 3D bounding boxes based on Euclidean clustering, perform rough localization, and use the detection results as input for the next stage of clustering: Euclidean distance-based point cloud clustering is a density-based algorithm with fast response and robustness. This algorithm calculates the distance between two points in space and compares it with a threshold to determine whether they belong to the same target. The algorithm input is: cluster radius Minimum number of points in a single cluster Point cloud aggregation after water surface filtration The algorithm outputs the 3D bounding boxes and corresponding point clouds of the detected water surface targets, which are also the inputs for the two-stage detection. S32: Based on the two-stage clustering, refine the results of the previous stage: In the second stage of clustering, the 3D point cloud is first converted into a hash table. In addition, an adaptive clustering radius is adopted. The neighborhood search is performed with an element in the table as the center until all points are traversed. This invention uses a clustering radius that considers the distribution of the point cloud. During the clustering process, there will be points that do not belong to any category or belong to multiple categories. For these points, their intensity is compared with the average intensity value of each cluster and they are classified into the cluster with the closest intensity value. Finally, the minimum bounding box of each point cloud cluster is obtained as the output. S4: Based on the coordinate transformation relationship, the clustering results are projected onto the image captured by the camera to obtain the region of interest, which serves as the input for subsequent detection; S5: Use a 2D salient target detection algorithm to obtain the specific location of the target in each region of interest.

2. The method for detecting water surface targets based on the fusion of lidar point clouds and camera images according to claim 1, characterized in that, Step S1 includes the following steps: S11: Joint calibration experiment design: The coordinate systems of lidar and camera in the unmanned vessel perception system are independent of each other. In order to achieve multi-sensor fusion, it is necessary to determine the external transformation matrix parameters between lidar and camera. The transformation matrix can be solved by obtaining the object center point and normal vector of the coordinate system of different sensors. S12: Solving for the coordinates and normal vector of a reference object in a 3D point cloud: A plane fitting algorithm based on point clouds is used to obtain the calibration plate plane in space. For a known plane, the coordinates of the center point can be obtained by connecting the diagonals. Similarly, the plane normal vector can be obtained. ; S13: Solving for reference object coordinates and normal vector in a 2D image: The distribution of feature points in the acquired image is calculated, and combined with the physical parameters of the calibration board and the camera intrinsic parameter matrix, a PnP problem is constructed. The distance from each feature point to the camera is calculated, thus obtaining the coordinates of the point in the camera coordinate system. Given the coordinates of each corner point, and combining this with the length and width of the calibration board, the coordinates of the four vertices and the center point of the calibration board are obtained. The vector perpendicular to the two diagonals and passing through the center point is the normal vector of the calibration plate. ; S14: Based on the collected data samples, solve for the coordinate transformation matrix between the lidar and the camera: By repeating the sample collection multiple times, the sets of center points in both the radar and camera coordinate systems can be obtained. and the set of normal vectors Then, the transformation matrix from the laser radar to the camera is obtained.

3. The method for detecting water surface targets based on the fusion of lidar point clouds and camera images according to claim 1, characterized in that, Step S2 includes the following steps: S21: Remove point clouds formed by reflections from the water surface: Point clouds formed by water surfaces can be represented by planar models. This involves: randomly selecting four points from the candidate points to construct a plane, obtaining the plane model parameters; then substituting all candidate points into the plane model to obtain the distance from each point to the plane, and selecting points with distances less than a threshold. The points are marked as interior points, and the number of interior points is counted. If the number of interior points in the current model is greater than the number of interior points in the optimal model, the model parameters are updated, and the current model is taken as the optimal model. The above process is repeated until the model parameters with the maximum number of interior points are obtained, or the number of iterations to terminate is reached. After the horizontal plane fitting is completed, the water surface points are deleted from the original point cloud. S22: Optical Image Enhancement: To address the characteristics of water surface images, a dehazing algorithm based on dark channel prior images is used to enhance optical images. In computer vision and computer graphics, the image model containing fog is represented as follows: (2) in, It is the acquired image. This is the image expected after restoration. It refers to the global atmospheric light intensity. The refractive index of the propagation medium is used to determine the global atmospheric light intensity. and the refractive index of the propagation medium After that, the enhanced image after dehazing can be obtained. .

4. The water surface target detection method based on the fusion of lidar point cloud and camera image as described in claim 1, characterized in that, Step S4 includes the following steps: S41: Project the point cloud detection results onto the image based on the transformation matrix: Through joint calibration, the translation vector and rotation matrix are obtained, thereby realizing the adjustment of the radar coordinate system. Transform to camera coordinate system The transformation relationship is as follows: (4) in Let be the translation vector between sensors, and be the rotation matrix. Euler angles This means that for the detected 3D bounding box, the points on its diagonal are transformed according to formula (4), and combined with the camera intrinsic parameter matrix. This yields the diagonal pixel coordinates of the target in the image.

5. The method for detecting water surface targets based on the fusion of lidar point clouds and camera images according to claim 1, characterized in that, Step S5 includes the following steps: S51: Segment the target in the candidate region using a salient target detection algorithm: After identifying the target candidate region, the color contrast between the target and the background is particularly prominent. The method uses a global contrast saliency detection algorithm. In the image, most areas of the water surface scene are low-frequency components, with a small portion being high-frequency components mixed with noise and reflections. The target is a high-frequency component relative to the background. By filtering out high-frequency noise and low-frequency background, the salient region with the largest connected area is found, which is the object to be detected. Subsequently, the original image and the Gaussian-filtered image are transformed using Lab space, and the Euclidean distance between them is calculated. The specific expression is as follows: (5) in This represents the arithmetic mean of the pixels in the Lab color space. The result is obtained after Gaussian slicing in Lab space. This is the final saliency score map. After extracting the most salient target in the image, a binarization operation is performed. The area greater than the threshold is the water surface target.

Images